Lots of nice answers here, some uninformed answers as well, but I think that most of them miss the actual point.
First thing that needs to be understood is that device called "charger" is nothing more than a dumb power supply. It has no intelligence beyond the SMPS controller, which tries to maintain output voltage as current requirements change.
The charger itself is inside of the telephone itself and its operation can not be easily affected by the user.
As others have mentioned, it's actually important to know how fast the phone can charge, that is to say what is actually the maximum current that the charger will take. The charger will monitor the actual battery state and it's up to it to do the "smart" part.
For fastest charging, the power supply needs to be able to provide as much current as the charger needs to charge the battery, plus all the current the phone itself needs to operate. That's why we have answer saying that turning phone off will speed up charging, since this way we minimize the power used by the telephone itself. It would be nice if we have some percents more than phone's maximum consumption, just to be safe.
Next step, and the one most important, is getting the charger actually to tell the telephone how much current it can produce.
As I previously mentioned, chargers are dumb, so there's nothing to actually talk to, and USB is basically being abused to carry high current. We do have some standards for charging over USB, but they were all bolted on the basic USB 2.0 standard as afterthoughts.
Then we have the "smart" telephone/tablet manufacturers that are unsatisfied with standard options and want to have more choice. That's where most of the difficulty comes from. We already have one answer mentioning the Charging port specification and it is there where the main issue is. The pre-standard ways of negotiating higher currents are still there and are still making problems. Apples iDevices still use their resistor dividers to set the maximum charging current and those specifications can change over time (compare previous link with this). Furthermore, other manufacturers are pushing their own standards. Here is a picture for Samsung for example.
So we need not only a high current charger, but we need a high current charger that will be recognized by the phone.
Unfortunately, with exception of using the newest and greatest OEM chargers, there's little that can guarantee that a a power supply will be correctly detected by the phone. Some power supplies may mention how are they wired on the inside, so that may make the decision a bit easier for the end user. Otherwise, all that's left is looking for reviews and experimentation.
Finally, there's the part of getting current from the power supply to the telephone. One big problem with USB micro cables is that they tend to be thin! This can be a considerable problem for charging phones. A USB 2.0 cable will have on the inside at least 4 wires, and to pack those wires in a thin cable, the wires themselves need to be very thin. Then we have a problem of resistance of those wires. Quite often, it can be neglected, but in case of USB power supplies and micro cables, we are reaching the point where cable itself may be dissipating non-negligible amount of power and is affecting the output voltage at the device connector. In general, if we have telephone whose charger allows fast charging and we have power supply that can provide enough current, cable may end up being the limiting factor. Phone's charger is not perfect and it needs a bit of head-room to charge. Also chargers will often lower charge current, if they detect that the power supply voltage is sagging. Some experiment results published here tell us that iDevices will ramp down charge current based on voltage.
So let's take a look at some numbers and cable marking. Here's a handy calculator for voltage drops. Let's assume that our charger will provide 5 V and can supply at least 2 A without any voltage droop. I'll also post results for calbe lengths of 3 and 6 feet.
Traditionally, the thinnest USB cables will use a 28 AWG twisted pair for data and two 28 AWG wires for power. Let's take a look at some results with that cable.
At 3 feet length, we have output voltage at the end of the cable of 4.22 V. With this, we're already out of USB specification. If we take 6 feet, that gives us voltage of 3.44 V at the output of the cable. Our device definitely won't be able to charge at high current using that cable!
I we move a bit up and use a cable that has 24 AWG power conductors, we'll have output voltage of 4.69 V at the end of the cable, which is just good enough, but some devices will ramp down their charge current with that. If we go to six feet, we'll have output voltage of 4.38 V, which is outside of standard.
If we move again up a bit to 20 AWG for power conductors, we'll have output voltage of 4.88 V at 3 feet length and this should be OK for charging, but iDevices will still ramp down their current. At six feet, output voltage will be 4.76 V, which is still OK.
I've heard that there are also USB cables that use 18 AWG wires for power conductors, so let's take at some results with that: 3 feet gives us output voltage of 4.923 V, which is quite good and at 6 feet, we have 4.85 V at the end of our cable.
This is also explanation why some "chargers" have nominal output voltage of 5.1 V, even though the standard calls for nominal output voltage of 5 V. With 5.1 V, our 3 feet 20AWG cable will have 4.98 V at its output, while our 24 AWG cable will have 4.79 V at its output.
There are even special charging USB cables. Their magic is very simple: They are short and may use thick conductors, minimizing losses in the cable itself.
Bonus: How to interpret USB calbe markings: Some cables may have inscription on them such as 28AWG/1P+24AWG/2C. First value is about the data conductors. We have one twisted pair of 28 AWG conductors for data. On USB 3.0 cable, we may see something like 28AWG/3P. Second value is for the power conductors. We have two (non-twisted) wires of 24 AWG. The higher the second value is, better is the cable for charging.
TL;DR: Determine maximum charge current for your phone, get charger that can provide that much or more and is wired for fast-charging your phone brand and get a short, thick USB cable for connecting phone to the charger.