The quality measurement field has a saying: “What gets measured gets done.” There’s a lot of truth and complexity in this seemingly simple statement. It’s both a guide and a warning. If you measure the right thing, you will know that it is done. If you don’t measure it, you won’t know.
On the other hand, if you measure the wrong thing or if the data you are using is flawed, you might be getting the wrong things done. Intuition and general impressions are important, especially with complex high-level ideas, but there is something about metrics — specifically how you approach and evaluate your data — that places you in the driver’s seat where “going with your gut” does not.
Let’s take a look at the basic metrics of translation and localization so that you can start planning better and improving on the relationships that you have with your partners and clients.
Start with Data
In my 15 years of industry experience I have seen a lot of failure with data management. Considering the massive amount of information stored in financial programs, linguistic management tools, and computer aided translation tech, it comes as no surprise that a lot gets lost in all of the raw data. But another part of it is the tools that some choose to use. Excel, for example, is flexible, accessible and great for data modeling; but without the right controls it can make maintaining data integrity a nightmare.
Start with understanding what data you are collecting.
- Where is the data coming from? Is it manually collected or is it being pulled from an automated source?
- Is it normalized, i.e. is all the data in the correct fields and formatted the same way? For example, are date fields all stored as dates using the same format? One of the biggest problems is that most data fields are stored as text which cannot easily be filtered or sorted.
- How is the data you have skewed? All data is skewed, so it’s important to understand how.
- Is your data accurate? Is it precise? (They are not the same thing).
- Which data is actionable? What data will be useful for your diverse user sets?
When talking about data normalization, beyond the basic formatting, a helpful example might be to consider how a team of workers is spending their time. You want to see where they are spending the most time so you can work on investing your limited resources on improving their performance. You set up a timesheet and then quickly see that different people will classify their time very differently.
Some will code at different time intervals and frequency. Some will round to the nearest 2 hours and others may code it down to the minute. Some will fill it out several times a day (and will therefore be able to remember details), others will wait until the end of the week and only be able to recall generalities. The data set itself will only be as precise as the least precise data in that set, and the classifications will only be as useful as the definitions of what each class means. Ambiguous definitions of classifications means ambiguous data.
An audit of your data that identifies what it is, where it is from, and how it could be used will be helpful for making sense of that data with metrics.
Follow with Standards
While it is important to note that there aren’t really any standards that are more important than those that are unique to your business structure, there are nevertheless fairly common metrics in the translation and localization space that help clients and vendors improve on the performance of their localization programs.
Percentage of On-Time Deliveries
∑ total deliveries that are on-time or early / total deliveries
How does your localization program help you meet your time-to-market goals? What do you mean by “on time” specifically and how should those specifics be used by HR, production teams, reviewers, and others to guide their work?
Average Linguistic Quality Scores
∑ total weighted errors / total word count
Production doesn’t happen in a vacuum — in addition to differing degrees of performance within a translation/localization group, there is an issue with using multiple language vendors on the same project. How will you identify and track quality performance? How will you define what an error is and how severely to rate it? How will you make sure the person scoring the translation is following the rules and is catching all the errors? What will define your brand’s unique quality needs, and how will you score how vendors meet those needs?
∑ total passed tests / total tests; ∑ total accepted jobs / total jobs
These are almost as important as the linguistic quality scores — either it meets your expectations and can be accepted or it doesn’t. But what does your company mean by passing? Poorly defined targets here or unrealistic expectations will likely mean that you don’t achieve meaningful progress. Also note that an insufficient data set, for example of only a few deliveries per month, is not a statistically valid set to draw conclusions.
∑ total escalations / total products or deliveries per region
Imagine that your CEO has just called because customers have been loudly complaining online. Even though this is “unstructured data” it is nevertheless vital data on your localization program’s performance. How you measure it? What constitutes an escalation? How do you track this type of information? What is an acceptable number of customer complaints? How has that changed over time, per project, or per locale? How does this compare to the total number of products and deliveries in a region?
∑ weighted word count / (network days – holidays)
Whether word counts in translation processing or page counts for desktop publishing, throughput is measured by the number of units divided by the number of days needed to complete the task (give or take holiday scheduling). A “network day” is a business day and excludes weekends. From that you should usually also subtract holidays. Besides helping you understand the performance of your system, it helps create predictability in scheduling resources and deliveries.
And predictability is what this is all about. The above are guidelines to start with. With metrics that are specifically tailored to the way that you do business, you get away from reactive mode and move into localization success planning. In this way, based on a solid foundation of data awareness and an evaluation toolset, you can build, scale, and target robust localization programs that can fit your global market needs.