Get Paid to Participate - up to $1 per post!     Twitter     Facebook     Google+
Hosting Discussion
 

Hosting Discussion > Web Hosting Forums > Hardware and Server Configuration > Blocking back link website crawlers / Methods
forgot password?



Reply


Old
  Post #1 (permalink)   09-24-2017, 06:48 AM
HD Newbie
 
Join Date: Aug 2017
Posts: 10

Status: PeterShene is offline
Hi guys i would like to thank you for your massive discussion with relevance to cloud fare and other cdns. However i have another issue i would like some advice on and quite desperately. My bandwidth is getting sapped by spiders , mostly back link profiles i presume as i recently entered the top three for a few of my queries , this is new ground for me .

I have done alot of research on it and seen people discuss the pros and cons of actually trying to implement something like this.

However i know you cant stop them all but i would least spend 30 mins to throw a few rocks in the road then let this guy have a clean run in what took me months to accomplish.

So ive done a few things used the robot txt file to dis allow bots as a first step this did not work.

I accessed the Apache server in questions ht.acess file and tried to block via rewrite conditions using user agent string to try redirect the bots.

obviously they dont identify themselves in the correct manner because they still get through .

I tried ip ranges but i dont want to do this as here i dont know what else im blocking out.

i have heard there is a php alternative to do this but foud nothing online.
One of my competitors though has manged to do this i cannot acces ny of his back links i tried just to see if he managed to block them and he did.

Am i missing something here? is there another way ?
__________________
Im a Novice Freelance Developer http://www.newperspectivestudio.co.za/ hoping to start an agency one day.
 
 
Reply

Thread Tools

New Post New Post   Old Post Old Post
Posting Rules:
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
vB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On
Sponsored By: