Lesson Robots.txt - Telling the search engines what they can and cannot index

So your site has made it into the search engines — you've got the Googlebot, Yahoo Slurp, MSNBot and a whole host of other spider-like creatures crawling through your site on a regular basis. People can now find your site by typing something into the search engines... although you probably won't be showing up on the front page for your best keywords just yet.

But what if you've got sections of your website that you don't want indexed? Remember that search engine spiders basically just wander around the internet, following links and inputting whatever they find into their giant databases. If there's a link to the page... the spiders will index it. They don't know that those photos are strictly friends and family only, or that there are certain pages in your website that you'd really rather not have popping up in the search engine listings or being archived by that pesky internet archive bot — like your long-expired special offers.

That is, of course, unless you tell them not to....

Enjoy this lesson? Save it, or Share it with the world!