This is a very basic guide.  It doesn't do much beyond collect my own thoughts as to a general stepwise phone-buying experience, but it can serve as a blunt reminder as to the resources available and things one should look out for prior to actually purchasing a phone.  You don't want to be stuck with an expensive phone that you absolutely detest, solely because you weren't careful. 

Make sure you look over more than just the reviews written by CNET or other sites; it should be obvious that they fail to put their devices through extended use.  Start by looking up the device in question on Amazon, or sifting through user reviews on CNET (or the countless other mobile devices review sites). 

After gathering preliminary ideas as to which phones you want to try, make sure to visit howardforums, ppcgeeks, or any affiliated phone websites (depending on the phone you are getting -- obviously if you aren't looking for a ppc, there's no reason to visit ppcgeeks).  While you'll find individuals touting their phones, you'll also find common complaints about major problems with the device -- whether it's carrier dependent (firmware/software) or a base hardware problem.  You'll also find solutions (device overclocking, feature unlocks, hardware reviews [e.g. best gps or bluetooth headset pairing], and so forth). 

Following all this, I suggest that you actually try the phone out for a couple of days -- run out and grab the phone (if it's with a major carrier, they generally allow returns given a set period of time) unless it's an independently sold device.  All the online reviews can't tell you just how comfortable the phone is in the hand, the clarity of calls for your specific use, and so forth. That said, don't listen too much to the store representative selling the product; half the time they'll tell you phones come with this or that feature when it doesn't -- make sure yourself.  If stereo Bluetooth is important to you, see how it performs under that condition; if memory is important, make sure you know the cap (2gb? unlockable 4gb? 8gb?) and the type of memory you can buy (SD? SDHC? miniSD?).  If the interface is important, check responsiveness (open and simultaneously run apps; see if the screen responsiveness falters).  If Java is important, make sure you didn't grab a BREW phone.  If GPS is important, make sure the device has unlocked built in GPS and the carrier supports it (or make sure you account for the additional costs of a paired GPS receiver).  Look into carrying options (e.g. if it's a fashion phone, you obviously want something that doesn't look like a paper bag) and other accessories (e.g. extended batteries, bluetooth sets, etc). 

If you're looking for phones on eBay, I've come to the realization that you need to go through the steps to ensure a safe (as you can get) purchase that will satisfy you.  First, I would generally search within North America for devices, given the prevalence of knockoff Asian products (see the dozens of guides on refurbished or fake phone hardware).  Second, I'd look at the available pictures of the device, and make sure (if it's not new) that its quality matches what I want.  For instance, if the screen is of concern to you, never assume it's in flawless condition -- always ask about the presence of scratches, scuffs, dead pixels, and/or other blemishes you're worried about.  If you don't ask, the seller really isn't responsible.  Furthermore, always pay via a credit card -- it's probably a larger monetary purchase, so cover yourself as best as you can. 

All in all, so long as you make sure to look over as many details as possible, you'll be fine.