Methods of Testing Machine Translation

January 20th, 2015

Testing machine translation involves either a faulty automated process or human beings – which defeats the purpose.

Methods of Testing Machine Translation | One Hour Translation

Love it or hate it, machine translation is always going to be part of this business, whether as a suite of tools to assist the human language translation professionals, or as an overall goal of creating a fully automated translation system. And every few years a company forgets the lessons of the past and attempts to cut out the human factor and rely entirely on machine translation for their globalisation and localisation needs.

I don’t blame them. If it can ever be made reliable and robust, machine translation will be significantly cheaper, faster, and much more efficient than the current kind, performed as it is by variable and unpredictable humans.

Still, even the most successful machine translation solutions at the lowest levels requires a robust testing system to ensure that nothing crazy or embarrassing gets through, and therein lies the second challenge of relying on machines for your translation work: Testing the translations it produces. There are several different methods, but what’s curious is that they all fall into two basic categories: Automated, and human.

Machine Review of Machine Translation

In what’s known as “Round-Trip” review, the translation produced by a machine is then fed into a machine translator going the other way. So you take an English source and a machine produces a French target, then the French is fed into a different machine which in turn produces an English target. In theory, if your English target is close to the original, or at least readable and high-quality, your machine translation has passed the test!

In practice, this testing is meaningless, because it is actually testing two systems. If the second machine is calibrated poorly, you’ll get a bad result even if the first translation is actually very good. In other words, since there are multiple points of potential failure, you can never be sure, using this method, where the breakdown occurred.

Back to Humans

Which then brings us to the other basic method for reviewing machine translation – submitting it to a human or humans for review. I could tell you that studies have found that when reviewing translation work produced by a machine it pays to have more than one reviewer, because a single reviewer can bring prejudices or other weaknesses to the process. But what difference does it make? If you have to have your machine translation reviewed by humans, then you don’t really have a machine or automated translation process, do you?

This is just another reason why machine translation remains the stuff of science fiction. Even when we can get it to (mostly) work, it requires so much hand-holding and review and babysitting you don’t actually get the savings in terms of resources that are promised. Human translation – supported, certainly, by machines – still offers a superior product with much less cost and time investment, making it the only sensible choice unless you want to add a whole second layer of review bureaucracy.

You might also like:

A pivotal localization industry event, LocFromHome keeps localization professionals current on key industry issues. This year’s talks explored four pre-selected topics: Business, Productivity, Engineering, and Outlook. Within these topic areas, participants were able to explore global trends, discuss business strategies, and learn about useful productivity tools. All from the comfort of their homes. We’re especially proud that Nir Sabato, our Head of Strategy, was selected to present on How to Find Your LSP Identity. In case you missed the event, you can read the summary of Nir’s presentation in this post.