Search engine bots crawl and index each page they visit, but sometimes it may be possible that as a website owner you don't want the bots to index any specific page of your website. In such case, robots.txt proves helpful. It is a text file that you need to put in the head section of your website so as to guide the google or any other search engine bots which page to visit or index and which to avoid. It looks like this

User-agent: *

Disallow: /

Here, User-agent set to * means that the condition applies to all the bots whether google, yahoo,etc. If you want to restrict only a specific bot like google bot then you may write google instead of *. The Disallow set to / means that the bots are restricted to visit any of the pages of the website. However, if you want that only a specific page should not be visited/indexed by the bots, then you must write its url in place of / like Disallow: /xyz.php/

For multiple pages you need to write the above line for each page's url.

You can check the robots.txt of your website in this way

www.yourwebsite.com/robots.txt

Robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website

Robots.txt is a text file you put on your site to tell search robots which pages you would like them not to visit.And also used for SEO purpose.

www.ghrix.com

to set parameters for search engine bots.