Originally Posted by PeterShene
Hi guys i would like to thank you for your massive discussion with relevance to cloud fare and other cdns. However i have another issue i would like some advice on and quite desperately. My bandwidth is getting sapped by spiders , mostly back link profiles i presume as i recently entered the top three for a few of my queries , this is new ground for me .
I have done alot of research on it and seen people discuss the pros and cons of actually trying to implement something like this.
However i know you cant stop them all but i would least spend 30 mins to throw a few rocks in the road then let this guy have a clean run in what took me months to accomplish.
So ive done a few things used the robot txt file to dis allow bots as a first step this did not work.
I accessed the Apache server in questions ht.acess file and tried to block via rewrite conditions using user agent string to try redirect the bots.
obviously they dont identify themselves in the correct manner because they still get through .
I tried ip ranges but i dont want to do this as here i dont know what else im blocking out.
i have heard there is a php alternative to do this but foud nothing online.
One of my competitors though has manged to do this i cannot acces ny of his back links i tried just to see if he managed to block them and he did.
Am i missing something here? is there another way ?
This gives a good way to block these