UPDATE ... for those interested

The other day I checked the stats on these Inktomi Slurp Yahoo spiders on the site I've been talking about ...

I just about croaked when I saw what I saw ... For the month of November Inktomi Slurp had 116,409 spiders visit the site and total bankwidth used up was 1.01 GB

CAN YOU BELIEVE IT???? No wonder my friends web host (who gives him the space N/C because its a support forum for people with a chronic illness) sent him an email asking why the bandwidth usage has been dramatically increasing these past months.

When I went on the site today ... there was me, one Googlebot & 56 spiders from Inktomi Slurp Yahoo.

So ... I created (finally) the robots.txt file. But ... because I wasn't sure if the info I was using to identify the spiders was correct ... I also went in to the Cpanel & used their IP Ban option to ban 2 entire blocks of IP's that the spiders from them have been using. Then went into the forum software and did an IP Ban on those same entire blocks.

Within a few minutes of finishing doing all that ... there was just me & the Googlebot left on the site. So ... guess its working.

The owner of the site ... after seeing how much bandwidth the Inktomi Slurp Yahoo spiders used last month decided he doesn't want them on his site at all. Not even for just a few days each month.

The stupid thing about it ... we could go on to any other search engine for any of the other spiders that visit and punch in some keywords & his site would come up listed. No site that I could find that I figure would use the Inktomi Slurp Yahoo spiders for their searches even listed the site within the first 10 or so pages. Sheesh !!!

Anyway ... guess the problem is solved now.

Also ... kind of lets everyone know just how much bandwidth SOME of these spiders CAN use. Which is ... obviously ... significant in our case.

Thanks again for the suggestions offered.
Peter

PS ... I never did receive a response from Inktomi Slurp Yahoo for the email I sent them.