If  you're anything like my you go to yardsales seeking merchandise.  I love hitting the books section, since they are consistently filled with books of some rarity for a pittance.  Telling what's valuable and what isn't is very tricky, and to get real good at it you've got to have years of experience and just outright memorize some books.  However, there are a few guidelines you can often use for a books value and demand.

1. Condition.  If a book is falling apart, written in, and just generally beaten up people won't find it nice to look at, which is what collectible books are bought exclusively for.
2. Author.  If it's a no-name, your books probably not going to have much value.  But, if you recognize the name and its early work, or just someone famous from history (for example, William Jennings Brian), it may be worth something.  The key here is its before they were famous.  A first-edition Harry Potter 3 in the US isn't valuable, but a first-edition Harry Potter 1 (UK First edition) will fetch about $10,000 at auction.  If an author ever uses an odd name then it probably has value.  This usually means aliases, but there are some notable exceptions.  Mark Twain's real name was Samuel Clemens, but his alias was used more than his real name.  Of course, any first editions of Mark Twains work will have value.
3. Age.  This is a tricky one.  Many people seem to think the older a book, the more valuable.  This couldn't be further from the truth.  Very old books, i.e. pre-18th century, will probably be valuable.  But something from the turn of the century won't necessarily be worth much.  While older books do tend to be more valuable, this is by no means a standard.  First editions, or better yet first printings, of famous works may have value.  A major exception is the Da Vinci Code.  The first edition had 70+ printings, but a first printing is a good find.
   
4. Signatures.  Anything signed by the author will probably be worth at least $10.
   
5. Oddities.  This one principle is probably my guiding light in buying, and so far it's done me a world of good.  The first book I ever purchased for the purpose of reselling was a first-edition from 1917.  It was titled "Washington: The Man of Action."  A children's novel, I was drawn to it because of how early it was written, yet it was elaborately and colorfully illustrated.  This was exactly what gave it its value.  It ended up selling on commission at a book fair in Maryland for over $300.  Just today I grabbed some books that were "Folk Tales of the Soviet Union."  It has this funny, campy typeset for the title, quite uncharacteristic of the USSR.  Turns out they're also valuable.

This guide is not definite by any means.  These are just tips I go by, and sometimes it really pays off.  But usually I get hit with very low-value items ($1-$5).  These are just the standards I go by at yardsales.  A great guide I use is ABEBooks.com.  Another factor should be what the people are asking.  If they want $5 for a book you're not sure about, don't grab it.  Is there a chance it's actually a treasure?  Sure, but if you buy everything that could be a treasure at high prices you'll just lose any possible profit.

Often, a combination of these factors is what's key.  A semi-old book, first edition, in good condition is more valuable than a very rare beaten up second edition book.  I hope this guide has been helpful, thanks for reading!