At first glance, machine translation appears to be the answer to every localization program manager’s dream of translating more content more quickly and more cost-efficiently. But it may not be fast, simple, or inexpensive to implement — and after all that, the results might not be exactly what you expected.
Before you jump in headfirst, make sure the whole effort will be worth your time and give you the outputs you need. Will machine translation meet your quality expectations for all languages, or only a few? What levels of editing will get you to the desired level of quality, and how does that affect your expectations of budget and time to market?
The best way to investigate reliable answers is to run a pilot. Here’s what a well-planned pilot will enable you to do in advance of your MT program investment.
- Define quality targets. In order to know what level of post-editing (PE) is required — light or full or something in the middle — you have to understand your quality targets. Do you really need human-level fluency and style for that technical manual, or do you merely need content that’s comprehensible?
- Assess viability of various content types. Some content is more difficult to complete with an MT PE process than others. Knowledge base articles and user manuals are ideal; marketing content is more tricky, but not impossible.
- Evaluate source quality. You’ve probably heard the adage “garbage in, garbage out,” and that’s certainly true with MT programs. Source content that is clear and consistent with standard sentence constructions will deliver better results. A pilot lets you evaluate whether you first need to clean up the source content with a pre-edit or quality automations.
- Test multiple engines. While a pilot doesn’t typically allow for a complete survey of engines, you can try several of them and learn which may be the best for your content or language. Do you need to look at multiple engines? Do you understand the type of engine you need to consider: rules-based, statistical, or hybrid? For example, if you’re targeting both Norwegian and Russian with both User Assistance documentation and marketing materials, you will likely need more than one engine.
- Understand MT evaluation methods. Do you know all the various ways to evaluate raw versus post-editing quality? Should you use BLEU, Meteor, TER and/or human evaluation? Do you understand what the scoring means? Understanding evaluation data helps with decisions on which approach to use.
- Examine raw MT quality. After choosing and training engines, measurement offers insight about the strength of the training data and the amount of engine customization required. You can then investigate test various pre- and post-processing steps and measure the impact.
- Compare productivities. Did MT reduce the total time required to complete your project? How long does it take to edit your raw MT output to the desired quality level? For lesser quality levels, MT is often significantly faster, but the editing may actually make the entire MT process take more time than human translation, depending on your source and quality objectives.
- Refine pricing. It’s likely that your language services provider (LSP) requested the chance to evaluate files and process before committing to pricing. That’s because MT pricing depends on many variables — overall process and other technologies involved, engine used, content type, quality target — that are impossible to nail down without thorough investigation.
- Estimate implementation costs. A pilot will reveal all the activities required to deploy and achieve an effective MT program, which may include technology investments or even process changes among your program stakeholders.
A pilot is the most accurate way to generate a clear picture of the cost, TAT and quality factors that will help you determine the financial feasibility of MT. Armed with these insights, not only can you decide whether MT’s potential gains are worth the upfront investment of time, resources, and ongoing effort, but you can also ensure that everyone’s expectations are aligned to reality.
Tune in for the next post in our MT series for key steps in setting up an effective pilot.