In the context of a holistic quality management program, marketing translation errors are remarkably easy to identify, monitor, and analyze for actionable insights. Evaluating transcreation, on the other hand, is nowhere near as tidy. It’s like finding errors in an apple by comparing it to a banana. Where do you even begin?
Evaluation of localization program quality is basically a matter of defining clear quality standards, setting KPIs, training linguistic reviewers on the proper use of scorecards, and monitoring errors over time. You’ll encounter some debate about whether an error is preferential or not, but the outcome of that decision gets recorded in a style guide or terminology database, and then the answer is black and white from there on out.
But transcreation is a creative process that adapts the original concept and creates entirely new content. As a result, there’s no direct comparison between the original campaign materials and the locale-specific output, which makes quality control a bit harder to manage.
Nevertheless, it is entirely possible and highly advisable to evaluate transcreation teams on several key points.
Project managers, talent managers, content leads, and any client-side roles who might work with locale-specific talent can provide insights on the individual performance of creative talent and reviewers. How effectively do they communicate? Do they chime in with clarifying questions? Do they propose boundary-pushing ideas proactively for team input before fleshing them out and risking possible rework? Do they meet deadlines and manage expectations? Given that transcreation is entirely about highly nuanced communication, it’s a giant red flag if individuals on your transcreation team are lousy communicators in the context of working with vendor and client teams.
Especially in the first few months of a transcreation engagement, you can get to know how the creative talent approaches your work by requesting “back translations” (where the new material is translated back into the language of the original creative brief) along with detailed discussions of the team’s creative choices. Reviewing such documentation across all target languages can give your localization program owner or brand specialist a sense of how well the vendor understands and communicates your brand priorities to the creative teams.
Effectiveness of Final Output
Client stakeholder feedback ought to serve as the primary measure of resource effectiveness. Feedback may come from brand teams, SMEs, client in-country stakeholders such as sales staff or channel partners, or even from the client’s customer feedback channels such as support staff or social media moderators.
However, this type of feedback can be tricky to interpret: is it preferential feedback, or does it really speak to a core problem? For example, if your ad targets a demographic of teenage boys, is a mid-career professional’s complaint about unfamiliar slang terms really valid?
How to Investigate Effectiveness
Once an effectiveness issue comes to light, it can be challenging to validate, let alone pinpoint the source of the issue — especially if the program is staffed with seasoned transcreation talent collaborating in teams. Here are several creative ways to investigate root causes when they are not immediately apparent.
- Objective corroboration. Examine sales data, site traffic data, etc. for evidence that the target market is not responding to the content.
- Extra reviews. A content specialist unaffiliated with the program team and/or a third-party editor may review a sampling of work performed across the program or by a specific writer to assess effectiveness.
- Market surveys. No one is more qualified to evaluate the quality of the content than the people to whom the content is targeted. Market surveys and focus groups can assess effectiveness, and the resulting data can be used to suggest the appropriate remedies.
- Blind A/B testing. Sometimes it’s helpful to engage an additional creative team to deliver an independent transcreation. A panel of content specialists can then compare samples from both teams and objectively evaluate effectiveness.
Resolving Transcreation Issues
Once you’ve done some investigation and determined there’s an effectiveness problem, firing the creative team is not necessarily the solution — especially if you started with seasoned transcreation professionals and invested a lot of time bringing the team up to speed.
Instead, share your findings with your vendor and collaborate on a root cause analysis. Then, ask for a plan to remedy the situation, including milestones and checkpoints to keep reassessing until you’re confident things have turned around.
Have you ever struggled with reviewer feedback on transcreation effectiveness? How did you investigate the problem and remedy the situation?