Need a new web host or having a problem with a script? Tips and advice on the best way to design, build and maintain your most valuable asset - your website!

WordPress and Robot.txt - Help please

webmann
 
Posts: 1
Joined: 20 Aug 07
Location: New Brunswick, Canada
Trust:

WordPress and Robot.txt - Help please

Hello everyone, I am really glad to be here and Mark you rock.

I recently converted my web sites to WordPress blogs and started seeing my traffic increase from a few visitors a day to about 100 a day. But...

... I have been trying to optimized. That is for another post though.

for now I need to know about the robot.txt

I found I was getting links in search engine results to pages with the WP folders so I decided to disallow some in the robot txt file.

These are the folders I have disallowed.

Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /trackback
Disallow: /category/*/*
Disallow: */trackback
Disallow: */*/trackback
Disallow: */*/feed/*/
Disallow: */feed
Disallow: /*?*
Disallow: /*?
Disallow: /docs
Disallow: /docs/
Disallow: /docs/*
Allow: /wp-content/uploads

This was done based on things I have found on the net but I am not sure if perhaps I have done something wrong.

I did this about a week or a little more ago and have noticed that my se positioning for keywords in google have dropped there position by about a hundred or more.

Anyone have some helpful info for my? I thank you all in advance.
 

  • 0