Home Lessons & Videos SEO How to Get Your Website Indexed

How to Get Your Website Indexed

Affilorama members only


Watch a preview Join for free

How to Get Your Website Indexed

2008-10-22 21:54:49

A question many affiliates ask is how to get the search engines crawling your website in the first place. In this lesson we look at How to Get Your Website Indexed.

Getting Seen

There are a couple of ways you can get the search engines to actually find your website:

1) Get a link

Search engine spiders are just basically tireless wanderers. They wander around the internet following links and seeing what they discover. Therefore the most straightforward way to get your site indexed is to get a link from another site. There are a couple of caveats here: The site that links to you has to be indexed by the search engines itself, and the link can't have a "nofollow" tag on it (if you want to get into Google).

One good way to get the search engines crawling your site is to set up a Twitter profile.The search engines love Twitter and the spiders spend a lot of time hanging out there. Once you've set up your profile, try to find some active members of the Twitter community to "follow" — which is basically like adding yourself to their friends list. This provides a link back to your Twitter profile, and allows the search engine spiders to start crawling your Twitter profile. Now just post a link to your new site and you should see little search engine bot footprints all through your stats in no time.

2) Submit your site to the search engines

Many of the search engines have "submit your site" pages where you can enter your site's URL and some other information as a suggestion that they include it in their index. It's not a guarantee that they will index it.

This used to be a good way of getting your site indexed back in the day, but these days all the major search engines find your site by the link method we talked about above, so there's not really much point.

Are You Indexed?

A quick trick to check if your website has been indexed is to go to Google and enter site:yourdomain.com into the search box. Google will return all the results they have for your website. The same trick applies for Yahoo and MSN.

You can also access this same information (and a bunch of other stuff) by signing up for Google's Webmaster Tools. Once you verify your site by adding a snippet of code to your HTML or uploading a file, you'll see an option to see which pages have been indexed by Google. This is pretty much the same as doing the site:yourdomain.com trick, but Google can also tell you if they're having difficulty crawling any of your pages, and the nature of the difficulty. This can be pretty handy for checking the overall health of your site structure.

How to get ALL your pages indexed

Getting the search engines to visit your site is one thing, but getting them to visit and index all the pages on your website is a different matter. When you check how many pages of your site are indexed, you may notice that some of them are missing.

A search engine spider may visit your site but fail to index some of your pages for a number of reasons:

  • It ran out of steam
    If you don't have many links to your site just yet (and consequently little PageRank) you might find the search engine spiders "not bothering" to dig too deeply into your site.

  • You're using duplicate content

    If the search engines think that the content on your page is pretty much the same as content on another page (this could be on your website or on a completely different website) then it probably won't show up in their listings.

  • Badly organized link structures

    Remember that the search engines find pages on your site by following links. A page on your site with more links to it is going to be easier to find than a page with only one link to it, buried in the deep recesses of your site. If a page is important, make sure it's easily accessible.

Suggestions for getting your pages indexed:

  • Use unique content
    Make sure all your content is at least 25% different to other content on the internet, and preferably at least 50% different. This is particularly true if you're using PLR content on your site — don't be lazy, reword it!

  • Get more links to your site
    The more good links you have to your site, the more "energy" the search engine spiders will have to explore your pages. You can tell how strong your links are by looking at your PageRank. The higher your PageRank, the more chance that the spiders will go deeper into your site.

  • Get links to your internal pages
    Rather than just having every link pointing at your home page, try getting links that point to your internal pages as well. This is called "deep linking" and it's a very useful strategy.

    Think of it this way: instead of conducting an all-out assault on your front door (and only getting as far as your living room), deep-linking means the spiders will also try the back door, the laundry window, and the ranch slider. They'll see a lot more of your house as a result!

PageRank Sculpting

PageRank "sculpting" is where you design your site so that as much PageRank "flows" to your important pages as possible. This means that these pages are more likely to appear in the search engines.

It sounds like a complex and painful procedure, but once you get past the fancy words it's really just what we were talking about before: making sure that your important pages are well-linked and easily accessible, and that the spiders have enough "energy" to reach your important pages.

Here's an example of a site that hasn't been "PageRank sculpted". It has a lot of links arriving at its home page, and then from the home page you can access the usual range of pages: a links page, a privacy policy page, a "contact us" page, a reviews page and an articles page.

From the articles page there are links to a bunch of articles - this is the main content of the site. The owner of this site expects that most of his traffic will arrive through through these articles, because they've been optimized for very popular keywords.

As you can see from the darkness of the shading, these articles aren't getting much PageRank flowing through to them. The PageRank is being split between the links at the top, but because the "articles" page only gets one fifth of the available PageRank, each of it's children (the articles) only receive one-tenth of one-fifth.

Chances are, until the website owner starts getting a lot of good links (and a lot of PageRank) to his home page, some of his articles might not be indexed because there's simply not enough PageRank flowing through to them.

Tip 1: Use "nofollow"

Does the website owner really need all that PageRank to go to his privacy policy and contact us page? Not really. So a good idea is to either remove links to these pages from his main home page, or use "nofollow" in his links. "Nofollow" tells the search engines not to bother exploring through this link, and means they can direct their attention to your more important pages. By blocking off those three unimportant pages, the website owner can increase the amount of PageRank flowing to his more important pages: The articles page and the reviews page.

Of course you probably want these pages indexed eventually. So it's a good idea to link to them from one of your less-important pages. It's just a way of telling the spiders which pages to pay attention to first.

Now we should note here: Google claims that PageRank sculpting doesn't work, and their engine does not direct PageRank evenly across all the links on a page; however we have run tests which show that this is, in fact, the case, so it's better to play safe than sorry and apply this trick anyway.

Tip 2: Get deep links

To add even more punch to this strategy, the website owner should work on getting links to his article pages as well as his home page. This benefits the article page itself, as well as any other pages the article might link to. You can see here that the PageRank spreads out through all pages the article links to, increasing the chance of those articles being indexed.

Create a Sitemap

Sitemaps help search engine spiders find pages on your website by listing every single page that you have. XML sitemaps can also include information like the importance of the page that can help the spiders know which pages they should pay most attention to. If you're having difficulty getting all your pages indexed, it may be worth creating a sitemap.

(For more information on how to do this, see our lesson on how to create a sitemap.)

Lesson Summary

In this lesson we've looked at how to get your page indexed, including:

  • Get a link to your website
  • Submit your site to the search engines

We also looked at checking if the search engine has indexed your site and what to fix if it isn't getting indexed correctly, including:

  • Checking with "site:yourdomain.com"
  • Ensuring you have unique content
  • Getting more backlinks to your site
  • Deep linking
  • Creating a sitemap


Learn something new? Share it with your friends!


Questions & Comments + Add a comment


I was trying to find a lesson for those, who only begin working with SEO, but unfortunately did not find it. Can you please share a link with me?

Reply Scarlett
This is Scarlett.
I have recently joined the Digital Marketing field and working as an SEO specialist.
I am looking to improve the Topical Trust flow and SERP ranking for my website.
I need suggestions to help me improve this. Can you please suggest me a few?
Cecille Loorluis
Hi Scarlett,

There are several methods mentioned here: https://www.affilorama.com/search-engine-optimization

You may also check out Moz.com. They have great tutorials on anything and everything SEO.

All the best!
Reply nivetha
Log into Google Search Console.
Navigate to Crawl Fetch as Google.
Take the URL you'd like indexed and paste it into the search bar.
Click the Fetch button.
After Google had found the URL, click Submit to Index.
Search Engine Optimization

Don't forget to follow us!

More topics

Essential Resources Every New Affilia... New to affiliate marketing? Not sure what it's all about? Then let us get you started!
Affiliate Outsourcing No one person can do everything on their own. Learn how to reduce (or even eliminate!) your workload with outsourcing.
Site Building As an affiliate, chances are you'll end up building a website or two. Here's your basic quickstart!
Marketing Ideas Needing inspiration? Wanting to go outside the box? Take a look in here for your next few marketing ideas!
Market Research for Affiliate Marketers To be a successful affiliate you must learn how to research their market before dedicating your precious time.
Pay Per Click Pay-per-click is one major method affiliates use to get traffic. Learn what it is and how you can use it effectively!
Search Engine Optimization Search engine optimization (SEO) is a major way of getting traffic to your site at minimal cost! Learn how to put it ...
Content Creation This section is all about content: How to write it, how to get it written for you, and other useful goodies!