My personal top 3:

1. Reliability
2. Support - available to answer issues
3. Price

You'll notice that I put the price at the bottom of the list. I've paid $5/mo and I've paid $60/mo for regular blog hosting.

Support is what makes the big difference in hosting companies. I'm a tech guy, so everything website related, I can handle without an issue. But when I run into an issue that's server related, someone better be available to resolve the issue, and quickly. 24 hour (1 day) response, is usually not something I want. And depending on the issue, a 1-hour wait is too long too.

I'm not looking for 5 minute resolution times, but I am looking for fairly immediate responses that they're looking into it (and not a generic email response either).