Rup,

I totally agree, let the users decide how they like it, what the requirements are.

Others,

There is a fancier way to try and determine duplicates, using statistics and word densities.  Again, not saying this is the "right" way to do it, but just a more modern approach.

(long rant, but of interest to some I hope)

In this approach, each document is considered to be a "bag of words", and using statistical methods, the top N "most important" words are collected and stored as a "fingerprint" for this document.  At a later time, this word fingerprint can be compared with that of other documents, to see if they are the same or not.  In some cases, this is the only measure used.  In other cases, this method is only employed as a tie breaker.

Notice my lose use of N (the number of terms), "important", and "fingerprint".  There are different methods for each of those.  I'll give some examples here.

For N, you could go as low as 5, or perhaps as high as 100.  For the problem Avi mentions, where the title and summary are the same, but later parts of the document vary widely, you'd probably want a higher side count, say 100 (or more).  I agree with what Avi points out, substantially different documents with a similar title and extract, are not a "fringe" case.  Think of replies to emails, or comments to a new mailing list posting, where the original text is often left in place.  Or a set of legal contracts, where only the terms in the last few sections of the doc changes (like what the price is!).  In all these cases, it's the end of the doc which may have really important changes, but where the upper portion of the doc may give similar looking results.

But even in these case, if you set N to 100, or heck, 250, you're are more likely to see these changes.

For "important", (selecting the most "important" words to store) there are at least two methods I've heard of.  One method uses the TF/IDF mathematics.  TF being "term frequency", or how dense this word appears in the document.  IDF being "inverse doc frequency", meaning how RARE  the word is in the OTHER documents.  So in English, important words are the ones in that particular document that occur a lot, and/or words in that document that are rarely used in other documents.  Different formulas can be used to calculate TF and IDF, and two combine those two scores.

A more interesting approach is to use more direct Bayesian or PCA style mathematics to calculate, for each word in a document, how well this word discriminates this document from the others.  Various tests are run for each word (or short phrases), and words that have the biggest variance are chosen.

And finally, for "fingerprint", you can store just the words as an unsorted list.  Or have them sorted most important to least.  Or even store the words and some type of "factor" or "score".  And when you compare two fingerprints, you can decide how many words need to match, and whether or not they need to be in the same order.  And if you stored scores, how closely the scores need to be.

These calculations can be expensive to compute, so might be applied only in "tie breaker" scenarios.

Back to Avi's scenario, if you had a 10 page email, and then somebody replied at the very bottom with a 1/2 page response, you are still likely to see a difference in this type of a "fingerprint".  The words in that extra half page should jostle the fingerprint enough to matter.

A very interesting field (related to clustering and spam detection), and something we've actually had to review this with customers.

One "cheat" that can foil simpler versions of these methods is having random gibberish text, which still has valid English words, but that happens to meet (or vary, in the case of spam detection) the statistical tests.  So a human looking at the text can see it's random words, but a simple statistics engine is fooled.  This is one reason I favor using more than single words in fulltext analysis; although you can then defeat those by using random sentences, though that dataset takes longer to gather.

Hope this is of interest to at least a few other folks out there.  :-)

Mark




--
Mark Bennett / New Idea Engineering, Inc.
Office: 408-733-0387 / Cell: 408-829-6513