When your site is indexed by the search engines, it is "crawled" by the search engine spiders - GoogleBot, Yahoo Slurp, Bingbot - in order to find all the content on your site, so that other people can find it.
But what if you've got sections of your website that you don't want indexed? The bots dumbly index whatever they can find - they don't know that, for example, those photos on the hidden part of your site are strictly friends and family only, or that there are certain pages in your website that you'd really rather not have popping up in the search engine listings or being archived by that pesky internet archive bot — like your long-expired special offers. In this lesson we look at robots.txt - telling the search engines what they can and cannot index.
Robots.txt is a small text document that lives in the root of your website and tells the "robots" visiting your website which pages they can and cannot access. When one of these "robots" visits your site, the first thing they do is go looking for the robots.txt file. They listen to your requests, and won't visit pages that you've disallowed.
Decide which areas of your website you want the spiders to index, and which ones you don't want them crawling through. And decide if there are any bots you would rather not have crawling through your site.
Open up your plaintext editor of choice, create a new, blank text file and save it as robots.txt, then write this information into the file:
To block all spiders from your entire website:
User-agent: * Disallow: /
To let all spiders see all content on your site:
User-agent: * Disallow:
To block certain directories:
User-agent: * Disallow: /cgi-bin/ Disallow: /tmp/ Disallow: /personal/ Disallow: /photos/staffchristmasparty/
To block a certain spider:
User-agent: Googlebot Disallow: /
To allow a certain spider, while blocking others:
User-agent: Googlebot Disallow: User-agent: * Disallow: /
Now simply save your file and upload it to your website.
If you've seen our lesson on creating XML sitemaps, you'll know that your robots.txt file is a really handy place to let the search engines know where that is.
All you have to do is leave a blank line after the last command in your robots.txt file, and then paste this little line:
If you've got more than one sitemap, you can enter more than one line.
Sitemap: <http://www.example.com/sitemap1.xml> Sitemap: <http://www.example.com/sitemap2.xml> Sitemap: <http://www.example.com/sitemap3.xml>
This way you don't need to specifically tell each and every search engine where they can find your sitemap. They'll see it as soon as they look for your robots.txt file, which every polite bot will do when they visit your site anyway.
"Polite" spiders, such as those belonging to the major search engines, are polite and won't index items you've listed in your robots.txt file. However, not all robots are polite (for example, from smaller search engines, or general data scraping bots), so they will collect any and all content anyway.
Don't try to use your robots.txt file to hide content on your site - the robots.txt file is able to be viewed by anybody, simply by typing www.yoursite.com/robots.txt into their browser, so anybody can see the things you've said you don't want indexed!
If there's content on your website that you really, really don't want anybody else seeing, your best bet is to password-protect that directory. There will usually be a tool to help you do this in your hosting control panel (cPanel or similar). Note that password-protecting your comment (if done right) will also prevent the "unpolite" bots from accessing the content
In this lesson we've looked at robots.txt - what it is, what it's used for, and how to create one. We've looked at certain things you can do with robots.txt including: