comparing two sentences of strings

how i want measure between sentences of words example 1) " the monkey wants to eat banana the dog want to run after cat" 2) " munkey to to eat banan the dogy wabt to rub cat"

it is result of some ocr so i need to measure 1. correct location 2. distance between words like monkey abd munkey , dog dogy

word can repeat so i want to measure relative good location what best method i was thinkink use ismemebr , or regexp ?

Answer 1

[Stephen23]

This is a very complicated topic, because it depends what you mean by "similar" words, how close they have to be to count as matches, and how you want to deal with conflicts or missing words.

The difference between any two words can be measured using a metric such as the Hamming distance, the Damerau-Levenshtein distance or some other string metric .

But matching them is much more difficult. Consider that each of your example sentences has a different number of words and some of them repeat in the second sentence: how do does the computer know to match both 'to''s (the second and third words of the second sentence) to one 'to' (the fourth word of the first sentence)? How different do the words have to be for them to be considered a match? How do we classify them if there are multiple different possible matches? Which match decides the "correct location" ?

Consider this example:

A = 'bob bib did'
B = 'bib bid dib'

How would you match these: Does 'bib' in B match 'bob' in the same position in A, or does it match 'bib' which has the same spelling? 'Does 'bid' match 'bib' or 'did', from each of which it differs by only one letter? Does another possible match affect another? How are you going to tell your computer this?

Answer 2

[Michael scheinfeild]

i was thinking use dtw dynamic time warping algorithm for total distance http://en.wikipedia.org/wiki/Dynamic_time_warping

but i think is good idea for hamming i will try

Answer 3

[Greg Dionne]

Search the internet for "edit distance". MATLAB has a variant that works on real signals, called EDR. It does insertion/deletion but not transposition (e.g. "eht" vs "the").