View Single Post

  Post #5 (permalink)   01-02-2018, 10:02 AM
HD Newbie
Join Date: Aug 2017
Posts: 31

Status: PeterShene is offline
Ahh i found a massive loophole, robot.txt request a bot to not access a certain page. Most good bots follow the rules set in rbt.txt.

What is written in rbt text however is not mandatory for a bot and thus most bad bots will ignore the request , and this is where you catch em with php, something called honey pot.

This is what all those worpress plugins are based on , as soon a bot requests or acts in a fashion not set out to your rules of your website , its identified as a bad bot and subsequently all its data is recorded and it is denied any request.

24x7server your link contains an MASSIVE list thanks so much

I do know quite a bit about this already but still learned a few things here just by skimming , ive bookmarked it seems like a very good article nice find and thankyou once again..
Web design east london

Last edited by PeterShene : 01-02-2018 at 10:05 AM.