It’s been a while since I did a Webmaster Wednesdays post, and I thought it might be useful to help you understand what some have had figured out for about a year now – if you want better search engine results, you need to understand what Latent Semantic Indexing is, why it’s important, and how to apply it to get better search engine results.
So let’s start with definitions.
Regular keyword searches approach a document collection with a kind of accountant mentality: a document contains a given word or it doesn’t, with no middle ground. We create a result set by looking through each document in turn for certain keywords and phrases, tossing aside any documents that don’t contain them, and ordering the rest based on some ranking system. Each document stands alone in judgement before the search algorithm – there is no interdependence of any kind between documents, which are evaluated solely on their contents.
Latent semantic indexing adds an important step to the document indexing process. In addition to recording which keywords a document contains, the method examines the document collection as a whole, to see which other documents contain some of those same words. LSI considers documents that have many words in common to be semantically close, and ones with few words in common to be semantically distant. This simple method correlates surprisingly well with how a human being, looking at content, might classify a document collection. Although the LSI algorithm doesn’t understand anything about what the words mean, the patterns it notices can make it seem astonishingly intelligent.
