Want to work your way up in SERPs? Get and share ideas on link building, on/off page optimizing, social SEO and more.

What do I put in the robots.txt file?

dmcqpu
Posts: 148
Joined: 28 Sep 12
Trust:

What do I put in the robots.txt file?

Hey,

I'd like to create a robots.txt file and upload it to my root directory.

I have a couple of questions about this however.

1. What do you recommend I include in my robots.txt file?
2. How do I upload the robots.txt file to my root directory?

Many thanks!
  • 0
Don't just build a website... Build a Web BUSINESS
http://incomeautopilot.info
 
Site Admin
mikeantiga
Posts: 792
Joined: 28 Mar 10
Trust:
Hi Alain,


There are two ways you can create a robots.txt for your site. First is by using an SEO plugin called Yoast WordPress SEO and second is by using a plugin called WP-FileManager.

If your site already has Yoast WordPress SEO plugin installed, then you should use this tool instead for creating a robots.txt file (Part A below).

If you do not have Yoast WordPress SEO installed yet, then you can just install this plugin and follow the relevant steps below (Part B).


A. Creating robots.txt via Yoast WordPress SEO plugin:


1. Go to SEO > Tools File Editor > Create robots.txt file. The plugin will automatically create a robots.txt file for you.

2. Edit your robots.txt file based on the suggested robots.txt file below. (See Part C).

3. Once done, access your robots.txt file using the URL format http://www.yourdomain.com/robots.txt and see if it now exists. Please note that you might need to clear your browser's cache for you to see the new robots.txt file on your end. If your browser still does not show the robots.txt, use another browser or another computer instead.



B. Creating robots.txt via WP-Filemanager plugin:

1. Create your own robots.txt file on your computer using Notepad. Follow the suggested format below (See Part C).

2. Upload the generated robots.txt to your website using a plugin called WP-FileManager.

2.1 To install WP-FileManager, go to plugins > Add New and Search for the WP-Filemanager plugin.

2.2 Install the plugin developed by Anant Shrivastava and Johannes Ries.

2.3 Once installed, go to FileManager (below Settings) > Configuration and enter the following in the "Filemanager Default Home location"

../

(that's two dots followed by a forward slash, no spaces before, between or after)

2.4 Make sure to also check all checkboxes provided on that page.

2.5 Leave the rest of the fields empty and then click "Save changes".


3. Upload the robots.txt file that you created using the FileManager plugin to the home directory of your website (same level as your wp-config.php file).

4. Once done, access your robots.txt file using the URL format http://www.yourdomain.com/robots.txt and see if it now exists. Please note that you might need to clear your browser's cache for you to see the new robots.txt file on your end. If your browser still does not show the robots.txt, use another browser or another computer instead.


C. Suggested robots.txt format (replace 'www.yourdomain.com' with your actual domain name):


User-agent: *
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content
Disallow: /author
Disallow: /wget/
Disallow: /httpd/
Disallow: /i/
Disallow: /f/
Disallow: /t/
Disallow: /c/
Disallow: /j/
Disallow: /page/
Disallow: /comments/

User-agent: Mediapartners-Google
Allow: /

User-agent: Adsbot-Google
Allow: /

User-agent: Googlebot-Image
Allow: /

User-agent: ia_archiver-web.archive.org
Disallow: /
  • 1
Want lessons on Affiliate Marketing, SEO and Content Creation? Get them here for FREE:
http://www.affilorama.com/lessons
 
dmcqpu
Posts: 148
Joined: 28 Sep 12
Trust:
Wow! Thank you so much!!! This is exactly what I was looking for.
  • 0
Don't just build a website... Build a Web BUSINESS
http://incomeautopilot.info
 
vunguyen52
Posts: 17
Joined: 12 Sep 15
Trust:
mikeantiga wrote:Hi Alain,


There are two ways you can create a robots.txt for your site. First is by using an SEO plugin called Yoast WordPress SEO and second is by using a plugin called WP-FileManager.

If your site already has Yoast WordPress SEO plugin installed, then you should use this tool instead for creating a robots.txt file (Part A below).

If you do not have Yoast WordPress SEO installed yet, then you can just install this plugin and follow the relevant steps below (Part B).


A. Creating robots.txt via Yoast WordPress SEO plugin:


1. Go to SEO > Tools File Editor > Create robots.txt file. The plugin will automatically create a robots.txt file for you.

2. Edit your robots.txt file based on the suggested robots.txt file below. (See Part C).

3. Once done, access your robots.txt file using the URL format http://www.yourdomain.com/robots.txt and see if it now exists. Please note that you might need to clear your browser's cache for you to see the new robots.txt file on your end. If your browser still does not show the robots.txt, use another browser or another computer instead.



B. Creating robots.txt via WP-Filemanager plugin:

1. Create your own robots.txt file on your computer using Notepad. Follow the suggested format below (See Part C).

2. Upload the generated robots.txt to your website using a plugin called WP-FileManager.

2.1 To install WP-FileManager, go to plugins > Add New and Search for the WP-Filemanager plugin.

2.2 Install the plugin developed by Anant Shrivastava and Johannes Ries.

2.3 Once installed, go to FileManager (below Settings) > Configuration and enter the following in the "Filemanager Default Home location"

../

(that's two dots followed by a forward slash, no spaces before, between or after)

2.4 Make sure to also check all checkboxes provided on that page.

2.5 Leave the rest of the fields empty and then click "Save changes".


3. Upload the robots.txt file that you created using the FileManager plugin to the home directory of your website (same level as your wp-config.php file).

4. Once done, access your robots.txt file using the URL format http://www.yourdomain.com/robots.txt and see if it now exists. Please note that you might need to clear your browser's cache for you to see the new robots.txt file on your end. If your browser still does not show the robots.txt, use another browser or another computer instead.


C. Suggested robots.txt format (replace 'www.yourdomain.com' with your actual domain name):


User-agent: *
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content
Disallow: /author
Disallow: /wget/
Disallow: /httpd/
Disallow: /i/
Disallow: /f/
Disallow: /t/
Disallow: /c/
Disallow: /j/
Disallow: /page/
Disallow: /comments/

User-agent: Mediapartners-Google
Allow: /

User-agent: Adsbot-Google
Allow: /

User-agent: Googlebot-Image
Allow: /

User-agent: ia_archiver-web.archive.org
Disallow: /

I'm looking for this. How lucky I am. Thank you for this information
  • 0
gisteam
Posts: 27
Joined: 21 Dec 15
Trust:
If you make a global command that you want to apply to a specific bot and you have other specific rules for that bot then you need to put those global commands in the section for that bot as well
User-agent: *
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content
Disallow: /author
Disallow: /wget/
Disallow: /httpd/
Disallow: /i/
Disallow: /f/
Disallow: /t/
Disallow: /c/
Disallow: /j/
Disallow: /page/
Disallow: /comments/
  • 0

This topic was started on Aug 07, 2015 and has been closed due to inactivity. If you want to discuss this topic further, please create a new forum topic.