Top Hosting Forum

WebHosting Reviews Forum

Robot.txt in Google webmaster

General Webmaster Discussion @ Top Hosting Forum
Hosting Forum Description
General Webmaster Discussion at Top Hosting Forum, Welcome

Robot.txt in Google webmaster

Postby raianju » Sat Jan 24, 2015 10:36 am

what is use of Robot.txt in Google webmaster?

Ad: There You can have your forum site-wide banner/text ad for $19 monthly. Contact admin

hey... I am Anju Rai from Delhi. i am working retails sector and Study in MBA. i am interesting more types of article, blogs and books like Get my ex love back, Powerful vashikaran mantra for love and some differents.
Posts: 49
Joined: Wed Mar 05, 2014 11:17 am
Location: jaipur

Re: Robot.txt in Google webmaster

Postby hostingforum » Wed Feb 11, 2015 2:04 pm

This file located in website root folder and it can contain instructions for search engine robots / spiders like GoogleBot.

Example instructions on which pages / folders Bot should skip.
It can also have instruction on how frequently bot should browse the website. Like it should not load webpage less than each 3 seconds.

User avatar
Site Admin
Posts: 640
Joined: Fri May 20, 2011 7:58 pm
Location: Czech Republic

Return to General Webmaster Talk

Who is online

Users browsing this forum: No registered users and 0 guests