Article compare tool concern

30 Dec 09 9:32 am

Hi Mark,

I have a similar question as well.

I have re worded an article in the Affilorama Article compare tool and it say i have 89.79% different, which i was well chuffed with.

I wanted to double check this and found somebody on the forum mentioned Dupefree Pro which is free i downloaded it and copy and pasted the articles into it and it states i have 79.42 duplicate content found.

Any ideas??????

Mark

04 Mar 10 11:52 pm

Is there any answer to this? Mark?

22 Apr 10 2:47 am

There are many ways to try and compare text. Can do it word by word, character by character, sentence by sentence etc.

There are also some lingual algorithms out there like levenshtein ("http://en.wikipedia.org/wiki/Levenshtein_distance") which is what we used to use however we have changed and use something else (to much to go into detail).

However, its worth noting that the % and the yellow colored text block calculated separately. The yellow is just a couple of lines of code that breaks the text into groups of 3 words and then compares them based on that - so really it's only useful of seeing how well you have changed sentences and paragraphs.

Hope that makes some sort of sense :)

22 Apr 10 2:56 am

You can always count on me, Nick.

23 Apr 10 2:10 am

I made some tweaks and getting Aletta to test :)

29 Dec 10 3:35 am

The difference in the results all comes down to the algorithm used to compare the text. Basically ours uses a custom solution quite different to most other compare tools. At the time of development this was taken to be more accurate, but it is under review and it is likely at some stage we will migrate it to use the more common method based on Levenstein distance, which as far as I understand is the most common method of comparison.

http://en.wikipedia.org/wiki/Levenshtein_distance

So in short, we compare the text differently to most, wether this is more accurate or not is really still up for debate, so I can't definitely say which is right or wrong (or really, more accurate).

03 Jan 11 1:50 am

Centered wrote:Based on your post above, we can conclude that Google and Yahoo search engines now must based their algorithm on Levenstein distance, mustn't they?

You won't find any official statement from Google mentioning the exact algorithm they use simply because revealing this would undermine their process of detecting duplicate and plagiarized contents.

Google probably uses a custom or proprietary algorithm for comparing articles if we are to base this from Google's patent application on near duplicate content (Granted December 2009). Source: Google Patent Granted on Duplicate Content Detection in a Web Crawler System.

Centered wrote:...what about the articles that have been posted for some time that based their uniqueness on Affilorama's compare tools whose custom solution quite different to most other compare tools? Aren't they now endangered by duplicate content penalty by the search engines?

First, duplicate content penalty is still subject for debate. Majority of webmasters (and even Google itself) says that it is a myth. Google, for sure, does not want to display same or exact content fill up the first 10 or 15 of its search results. Hence, a website with a very similar content will likely be placed in the lower part of the results pages but this does not necessarily mean the site is being 'penalized' for having a similar or duplicate content.

Second, we have not yet encountered any website that suffered from low search ranking just by using our Article Compare tool, although we get inquiries from time to time about the varying results between our tool and other commercially available comparison tools. Again, our algorithm is still subject for review at this time.