--- In catmt@yahoogroups.com, "Samuel Murray (Home)" <leuce@...> wrote:
>
> Samuel Murray (Home) wrote:
>
> > Bacchus BG wrote:
>
> >> I look for BLEU rates of the following language pairs:
> >> English-Spanish and Spanish- English for "Google translate" and
> >> "Bablefish".
>
> > You could do similar analysis by aligning the reference translations
> > into TMs, then reverse the TMs, and then performing CAT
translation on
> > the machine translation target text. This will not be a BLEU
analysis
> > but it will be a non-programmer's solution to evaluating machine
> > translation in a way which is remotely similar to BLEU (since it
would
> > result in a percentage count).
>
> Okay, I just did it, and here are the results.
>
> Samuel
>
> ===================================
>
> = A very simple statistical analysis of two MT systems =
>
> == Method ==
>
> 1. Get an exact translation of an English text, to be used as human
> translated reference text. I used Marina Soldati's sample translations
> at her ProZ.com profile at http://www.proz.com/profile/108251 (used
> without permission).
> 2. Align the reference translation and turn it into a TM.
> 3. Reverse the TM (i.e. switch source and target languages).
> 4. Get Google and Babelfish to translate the source text (note that
> Babelfish does not indicate paragraph breaks).
> 5. Use a CAT tool such as Wordfast to translate the MT translations
> back, using the reversed TM from the reference translation, and note
the
> sentence fuzzy match percentages (remember to use a fresh reversed TM
> for each test, as the CAT tool may adjust the TM during the first test).
>
> == Statistical result (numbers) ==
>
> === Google ===
>
> Raw match percentages: 0, 76, 69, 83, 0, 0, 66, 0, 64, 0, 0, 73, 0, 0,
> 0, 100, 80, 0, 88, 0, 0, 0, 0, 0, 0, 62, 0, 0, 0, 64, 0, 0
> Total of match percentages: 825
> Median (all): 0
> Average (all): 25
> Median (non-zero only): 73
> Average (non-zero only): 75
>
> === Babelfish ===
>
> Raw match percentages: 0, 74, 65, 60, 0, 0, 64, 0, 0, 0, 0, 0, 0, 0, 0,
> 62, 0, 0, 84, 0, 69, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 68
> Total of match percentages: 564
> Median (all): 0
> Average (all): 17
> Median (non-zero only): 66
> Average (non-zero only): 68
>
> == Comparative result of numbers ==
>
> === Matches above 60% ===
>
> Google gives 11 matches
> Babelfish gives 8 matches
>
> === One-on-one comparison ===
>
> These are the sentences for which both MT translations showed a 60%
> match or higher with the reference text.
>
> Google : Babelfish
> 76 : 74
> 69 : 65
> 83 : 60
> 66 : 64
> 80 : 62
> 88 : 84
>
> ==
>
> Samuel
>



Thank you very much Samuel. I'm quite content with that statistic
given in percentages.
It was pretty good response.