IMHO there is a difference.
The load a server endures (that actually hosts websites) is certainly much more than the average desktop machine. Think about how much our computer idles in the grand scheme. Worst case scenario is you have an intense graphic design program up for a few hours. Despite not having the intense program, servers are serving content 24/7 to humans and other computers. Traffic varies, and the resulting heavy load of maybe backing up, processing large log files, and serving PHP scripts adds up. The difference being this happens often, and not just here and there.
Aside from the obvious issues like space (not in play quite as much anymore), hardware needs to be the best suited for its environment. In the case of a server, you have a demanding environment not experienced with a desktop computer. It makes sense to pay a little more and use the better hardware because it's in use much more often.