I think it depends on your audience as well. If you are dealing with a tech savvy group of users and your Analytics confirm this, then you might not have to worry so much about it from the user perspective.

That being said, you always want to worry about Google and everyone is right about Google liking quick loading sites.