Testing machine translation involves either a faulty automated process or human beings – which defeats the purpose.
Love it or hate it, machine translation is always going to be part of this business, whether as a suite of tools to assist the human language translation professionals, or as an overall goal of creating a fully automated translation system. And every few years a company forgets the lessons of the past and attempts to cut out the human factor and rely entirely on machine translation for their globalisation and localisation needs.
I don’t blame them. If it can ever be made reliable and robust, machine translation will be significantly cheaper, faster, and much more efficient than the current kind, performed as it is by variable and unpredictable humans.
Still, even the most successful machine translation solutions at the lowest levels requires a robust testing system to ensure that nothing crazy or embarrassing gets through, and therein lies the second challenge of relying on machines for your translation work: Testing the translations it produces. There are several different methods, but what’s curious is that they all fall into two basic categories: Automated, and human.
Machine Review of Machine Translation
In what’s known as “Round-Trip” review, the translation produced by a machine is then fed into a machine translator going the other way. So you take an English source and a machine produces a French target, then the French is fed into a different machine which in turn produces an English target. In theory, if your English target is close to the original, or at least readable and high-quality, your machine translation has passed the test!
In practice, this testing is meaningless, because it is actually testing two systems. If the second machine is calibrated poorly, you’ll get a bad result even if the first translation is actually very good. In other words, since there are multiple points of potential failure, you can never be sure, using this method, where the breakdown occurred.
Back to Humans
Which then brings us to the other basic method for reviewing machine translation – submitting it to a human or humans for review. I could tell you that studies have found that when reviewing translation work produced by a machine it pays to have more than one reviewer, because a single reviewer can bring prejudices or other weaknesses to the process. But what difference does it make? If you have to have your machine translation reviewed by humans, then you don’t really have a machine or automated translation process, do you?
This is just another reason why machine translation remains the stuff of science fiction. Even when we can get it to (mostly) work, it requires so much hand-holding and review and babysitting you don’t actually get the savings in terms of resources that are promised. Human translation – supported, certainly, by machines – still offers a superior product with much less cost and time investment, making it the only sensible choice unless you want to add a whole second layer of review bureaucracy.