I do think that hearing music on a site can make any visitor to annoy and the user tends to close the window. But Music is different from Sounds. My vote goes to sounds. When a user try to click a button(made of gif file), I use sounds so that they frequently use those type of buttons. And therefore this results in more use of your site.
So, use sounds, but dont use music to your site. This is the word from me.