I too would only worry about .com and .net but really I haven't had anything yet that I would even consider buying the others. I think that having others sidetrack as you put it can also be beneficial to you on occasion if they are a success too. However the other domain could potentially be used to hurt the rep of your main site.

I haven't give much thought to this in the past, so I am glad that it was brought up.