Robots.txt - It's Classification & Use

Robots.txt - It's Classification & Use

Robots.txt : It is an index made by "webmasters" is a situated of guidelines for the web "indexes robots" that file theb ags. That is called The "Robots Exclusion Protocol".

It incorporates "User-Agent and Disallow" Here is the importance of these statements as these are exceptionally vital part of robots.txt record :

Client Agent : It is utilized for the "web crawler robots" it gives the authorization to a specific or all robots to list the

pages. Here is the language structure :

Client Agent : ("*" implies all robots have authorization to slither all the files/pages)

Disallow : It denies access to robots to directories/files. Here is the grammar :

Disallow : /pictures/ (it denies access to pictures registry.)

A Basic Robots.txt Record Ought To Be : 

Client Agent: * 

Disallow: 

It implies we give authorization to all the robots to visit and the registries and documents.It is extremely simple to make this record. Open notepad "proof reader".

Embed these guidelines and spare it by name robots in content form. Put it in "top-level" registry and transfer it. Assume your site name is http://www.abctest.com/ then your robots

index's way ought to be here http://www.abctest.com/robots.txt

You can check your robots.txt index here :- Click Now