Just want to chat? Talk some smack. If it doesn't fit into one of the areas below then you'll be posting here (Note: Please try and find the best section before posting here)

Missing robots.txt file

dmcqpu
Posts: 148
Joined: 28 Sep 12
Trust:

Missing robots.txt file

I just did a health check of my website languagehelper.net using AffiloTools.

I got this message:
Missing robots.txt file

How do I fix this?

Many thanks,
Alain
  • 0
Don't just build a website... Build a Web BUSINESS
http://incomeautopilot.info
 
Site Admin
aletta
Posts: 3719
Joined: 09 Jul 06
Trust:
Hey Alain,

A robots.txt file isn't essential to your website. You'd only really need one if you wanted to block robots from certain parts of your site. Otherwise the robots just assume that you're happy for them to wander through as they please.

So you can just ignore that warning if you like.

But one thing you might want to check is that you've got any duplicate pages on your site blocked from robots. If you're using WordPress, it'll usually generate a bunch of archive pages containing your article content. This can count as "duplicate content" to the search engines and can hurt your SEO.

You can block these with a robots.txt, but an easier way to do it if you're using the Yoast SEO plugin is to disable indexing of these pages on a category level. You can set the rules for this inside the Yoast SEO plugin (menu down the left of the page). You can set it to "noindex, follow" -- so the search engines will still go through the pages, they just won't index them and it won't count as duplicate content.
  • 0
Are your changes not showing up on your website? Try doing a hard refresh! http://www.refreshyourcache.com/en/cache/
 
dmcqpu
Posts: 148
Joined: 28 Sep 12
Trust:
Can you tell me more about how to do this? I have Yoast but I don't know how to do what you described above.
  • 0
Don't just build a website... Build a Web BUSINESS
http://incomeautopilot.info
 

This topic was started on Sep 25, 2014 and has been closed due to inactivity. If you want to discuss this topic further, please create a new forum topic.