A question many affiliates ask is how to get the search engines crawling your website in the first place. In this lesson we look at How to Get Your Website Indexed.
There are a couple of ways you can get the search engines to actually find your website:
Search engine spiders are just basically tireless wanderers. They wander around the internet following links and seeing what they discover. Therefore the most straightforward way to get your site indexed is to get a link from another site. There are a couple of caveats here: The site that links to you has to be indexed by the search engines itself, and the link can't have a "nofollow" tag on it (if you want to get into Google).
One good way to get the search engines crawling your site is to set up a Twitter profile.The search engines love Twitter and the spiders spend a lot of time hanging out there. Once you've set up your profile, try to find some active members of the Twitter community to "follow" — which is basically like adding yourself to their friends list. This provides a link back to your Twitter profile, and allows the search engine spiders to start crawling your Twitter profile. Now just post a link to your new site and you should see little search engine bot footprints all through your stats in no time.
Many of the search engines have "submit your site" pages where you can enter your site's URL and some other information as a suggestion that they include it in their index. It's not a guarantee that they will index it.
This used to be a good way of getting your site indexed back in the day, but these days all the major search engines find your site by the link method we talked about above, so there's not really much point.
A quick trick to check if your website has been indexed is to go to Google and enter site:yourdomain.com into the search box. Google will return all the results they have for your website. The same trick applies for Yahoo and MSN.
You can also access this same information (and a bunch of other stuff) by signing up for Google's Webmaster Tools. Once you verify your site by adding a snippet of code to your HTML or uploading a file, you'll see an option to see which pages have been indexed by Google. This is pretty much the same as doing the site:yourdomain.com trick, but Google can also tell you if they're having difficulty crawling any of your pages, and the nature of the difficulty. This can be pretty handy for checking the overall health of your site structure.
Getting the search engines to visit your site is one thing, but getting them to visit and index all the pages on your website is a different matter. When you check how many pages of your site are indexed, you may notice that some of them are missing.
A search engine spider may visit your site but fail to index some of your pages for a number of reasons:
It ran out of steam
If you don't have many links to your site just yet (and consequently little PageRank) you might find the search engine spiders "not bothering" to dig too deeply into your site.
You're using duplicate content
If the search engines think that the content on your page is pretty much the same as content on another page (this could be on your website or on a completely different website) then it probably won't show up in their listings.
Badly organized link structures
Remember that the search engines find pages on your site by following links. A page on your site with more links to it is going to be easier to find than a page with only one link to it, buried in the deep recesses of your site. If a page is important, make sure it's easily accessible.
Use unique content
Make sure all your content is at least 25% different to other content on the internet, and preferably at least 50% different. This is particularly true if you're using PLR content on your site — don't be lazy, reword it!
Get more links to your site
The more good links you have to your site, the more "energy" the search engine spiders will have to explore your pages. You can tell how strong your links are by looking at your PageRank. The higher your PageRank, the more chance that the spiders will go deeper into your site.
Get links to your internal pages
Rather than just having every link pointing at your home page, try getting links that point to your internal pages as well. This is called "deep linking" and it's a very useful strategy.
Think of it this way: instead of conducting an all-out assault on your front door (and only getting as far as your living room), deep-linking means the spiders will also try the back door, the laundry window, and the ranch slider. They'll see a lot more of your house as a result!
PageRank "sculpting" is where you design your site so that as much PageRank "flows" to your important pages as possible. This means that these pages are more likely to appear in the search engines.
It sounds like a complex and painful procedure, but once you get past the fancy words it's really just what we were talking about before: making sure that your important pages are well-linked and easily accessible, and that the spiders have enough "energy" to reach your important pages.
From the articles page there are links to a bunch of articles - this is the main content of the site. The owner of this site expects that most of his traffic will arrive through through these articles, because they've been optimized for very popular keywords.
As you can see from the darkness of the shading, these articles aren't getting much PageRank flowing through to them. The PageRank is being split between the links at the top, but because the "articles" page only gets one fifth of the available PageRank, each of it's children (the articles) only receive one-tenth of one-fifth.
Chances are, until the website owner starts getting a lot of good links (and a lot of PageRank) to his home page, some of his articles might not be indexed because there's simply not enough PageRank flowing through to them.
Of course you probably want these pages indexed eventually. So it's a good idea to link to them from one of your less-important pages. It's just a way of telling the spiders which pages to pay attention to first.
Now we should note here: Google claims that PageRank sculpting doesn't work, and their engine does not direct PageRank evenly across all the links on a page; however we have run tests which show that this is, in fact, the case, so it's better to play safe than sorry and apply this trick anyway.
To add even more punch to this strategy, the website owner should work on getting links to his article pages as well as his home page. This benefits the article page itself, as well as any other pages the article might link to. You can see here that the PageRank spreads out through all pages the article links to, increasing the chance of those articles being indexed.
Sitemaps help search engine spiders find pages on your website by listing every single page that you have. XML sitemaps can also include information like the importance of the page that can help the spiders know which pages they should pay most attention to. If you're having difficulty getting all your pages indexed, it may be worth creating a sitemap.
(For more information on how to do this, see our lesson on how to create a sitemap.)
In this lesson we've looked at how to get your page indexed, including:
We also looked at checking if the search engine has indexed your site and what to fix if it isn't getting indexed correctly, including: