Get Paid to Participate     Twitter     Facebook     Google+
Hosting Discussion
 

forgot password?


Reply


Old
  Post #1 (permalink)   09-20-2009, 07:12 PM
HD Master
 
Zagor's Avatar
 
Join Date: May 2009
Posts: 357

Status: Zagor is offline
Do you use robots.txt file on your web site? I don't have much experience with this. My question is what files and locations should I forbid and to what robots specifically, if any?
 
 
 


Old
  Post #2 (permalink)   09-20-2009, 11:24 PM
HD Addict
 
Join Date: Aug 2009
Location: India
Posts: 138
Send a message via MSN to Hosting India Send a message via Yahoo to Hosting India

Status: Hosting India is offline
no need to add any such files for robots, robots will automatically come to your site
__________________
India Web hosting Professional Web Hosting in India
Best Deal in Reseller Hosting , Reseller Hosting India
24x7 support with 99.9 % Uptime guarantee Dedicated Servers
 
 
 


Old
  Post #3 (permalink)   09-22-2009, 06:31 PM
HD Addict
 
Join Date: Feb 2008
Posts: 115

Status: thewebhostingdi is offline
You just need to develop your site & your content should be fresh & unique. Then you need to submit your site to major search engine so that Google crawler can crawl your site.
__________________
Windows Hosting Expert
ASP.NET 3.5 | SQL 2005 Database | US Based Hosting Company | 24 X 7 Support | Daily Backups | Uptime Guarantee | Affiliates - $50 Per Sale
| Web Hosting Directory
 
 
 


Old
  Post #4 (permalink)   09-22-2009, 11:49 PM
HD Addict
 
Join Date: Aug 2009
Location: India
Posts: 138
Send a message via MSN to Hosting India Send a message via Yahoo to Hosting India

Status: Hosting India is offline
For faster indexing of your site and effective search, you can bookmark your site to certain good social site and submit it in the several directories for getting backlinks. If the sites you submit to are dofollow sites , you invite Robots to crawl your site. as many time your pages are crawled by the bots, it get indexed by the google
__________________
India Web hosting Professional Web Hosting in India
Best Deal in Reseller Hosting , Reseller Hosting India
24x7 support with 99.9 % Uptime guarantee Dedicated Servers
 
 
 


Old
  Post #5 (permalink)   09-28-2009, 06:26 PM
HD Master
 
Zagor's Avatar
 
Join Date: May 2009
Posts: 357

Status: Zagor is offline
Thanks, but yet I asked and no one seem to answered about what SPECIFIC files and locations is common to forbid for search engine robot crawling?
 
 
 


Old
  Post #6 (permalink)   09-28-2009, 06:45 PM
HD Moderator
 
Join Date: Dec 2004
Posts: 1,725

Status: webfreak08 is offline
Do you have something you want to hide from search engines? If so, you should consider password protecting the information and using the noindex meta tag in addition to a robots.txt file.

Otherwise, there's no reason to use a robots.txt file.
 
 
 


Old
  Post #7 (permalink)   10-01-2009, 10:06 AM
HD Master
 
Zagor's Avatar
 
Join Date: May 2009
Posts: 357

Status: Zagor is offline
The reason why I asked this is because people are always telling me that certain files are of no use at all for search engines, like JavaScript, CSS, flash, XML. So why not forbid them from the start and avoid their indexing?
 
 
 


Old
  Post #8 (permalink)   10-01-2009, 03:11 PM
HD Moderator
 
Join Date: Dec 2004
Posts: 1,725

Status: webfreak08 is offline
Quote:
So why not forbid them from the start and avoid their indexing?
Did you read an article that told you to do this?

I'm pretty sure Google/Bing will be able to understand what it finds on your site and index it (or not) appropriately.

Again, there's no reason to worry about this stuff--just focus on building your website.
 
 
 


Old
  Post #9 (permalink)   10-05-2009, 04:33 AM
HD Master
 
Zagor's Avatar
 
Join Date: May 2009
Posts: 357

Status: Zagor is offline
Yes I read about it more than once. And for the record I never worried about this before, I was just asking for your opinions.
 
 
 
Reply

Thread Tools

New Post New Post   Old Post Old Post
Posting Rules:
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
vB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On
Sponsored By: