Samuel Murray (Home) wrote:

> Bacchus BG wrote:

>> I look for BLEU rates of the following language pairs:
>> English-Spanish and Spanish- English for "Google translate" and
>> "Bablefish".

> You could do similar analysis by aligning the reference translations
> into TMs, then reverse the TMs, and then performing CAT translation on
> the machine translation target text. This will not be a BLEU analysis
> but it will be a non-programmer's solution to evaluating machine
> translation in a way which is remotely similar to BLEU (since it would
> result in a percentage count).

Okay, I just did it, and here are the results.

Samuel

===================================

= A very simple statistical analysis of two MT systems =

== Method ==

1. Get an exact translation of an English text, to be used as human
translated reference text. I used Marina Soldati's sample translations
at her ProZ.com profile at http://www.proz.com/profile/108251 (used
without permission).
2. Align the reference translation and turn it into a TM.
3. Reverse the TM (i.e. switch source and target languages).
4. Get Google and Babelfish to translate the source text (note that
Babelfish does not indicate paragraph breaks).
5. Use a CAT tool such as Wordfast to translate the MT translations
back, using the reversed TM from the reference translation, and note the
sentence fuzzy match percentages (remember to use a fresh reversed TM
for each test, as the CAT tool may adjust the TM during the first test).

== Statistical result (numbers) ==

=== Google ===

Raw match percentages: 0, 76, 69, 83, 0, 0, 66, 0, 64, 0, 0, 73, 0, 0,
0, 100, 80, 0, 88, 0, 0, 0, 0, 0, 0, 62, 0, 0, 0, 64, 0, 0
Total of match percentages: 825
Median (all): 0
Average (all): 25
Median (non-zero only): 73
Average (non-zero only): 75

=== Babelfish ===

Raw match percentages: 0, 74, 65, 60, 0, 0, 64, 0, 0, 0, 0, 0, 0, 0, 0,
62, 0, 0, 84, 0, 69, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 68
Total of match percentages: 564
Median (all): 0
Average (all): 17
Median (non-zero only): 66
Average (non-zero only): 68

== Comparative result of numbers ==

=== Matches above 60% ===

Google gives 11 matches
Babelfish gives 8 matches

=== One-on-one comparison ===

These are the sentences for which both MT translations showed a 60%
match or higher with the reference text.

Google : Babelfish
76 : 74
69 : 65
83 : 60
66 : 64
80 : 62
88 : 84

==

Samuel