View Single Post


Old
  Post #5 (permalink)   11-07-2006, 07:42 AM
pmhoran
HD Master
 
pmhoran's Avatar
 
Join Date: Dec 2004
Location: Ontario Canada
Posts: 388

Status: pmhoran is offline
Thanks for the responses ... much appreciated.

Lesli ... definitely a good suggestion. One I may try as a last resort

Artashes ... I know spiders in smaller numbers don't use up that much bandwidth ... but hard to believe that when they get up in the 60 to 80 range that they don't have a visible impact on it. Would LOVE to have so the spiders don't register or show as Guests. The forum software I use, there is an add-on mod to show or hide Googlebots but all the others will still show. Google isn't a problem though. Do you think that many spiders at one time would slow down the server or forum response time??? I get some of the users (dialup ... never high speed) complaining of the forums being slow ... and possibly coincidentally ... it seems to be on days Yahoo has the 60 to 80 spiders running around the site.

Blue ... thanks for reminding me about the robots.txt page. Guess that falls into the "Daaaaaaah" category for me. I had TOTALLY forgotten about being able to use that. And I know from experience that Yahoo at least does pay attention to them ... because I've used a robots page on other sites. I will definitely have to refresh my memory on everything I can do with a page like that.