memoQ AGT (Adaptive Generative Translation) scored 82.2 on the chrF++ quality metric for segments with good translation memory matches, against 67.3 for the strongest competitor on the same segments. With no examples at all, it matches DeepL and beats Microsoft. See the full benchmark results.
AI translation quality evaluation is the process of measuring how accurate and usable machine-generated translations are by using automatic metrics, carefully built datasets, and human review. At memoQ, a translation management system and language intelligence platform, a research team of three runs this evaluation process continuously. Our research team includes László Laki, Senior NLP (natural language processing) Engineer; Engracia Batista, Junior NLP Engineer; and Zsanett Ferenczi, Contractor, NLP Engineer. They test machine translation (MT) engines, large language models (LLMs), and prompts on the same data, so every improvement to memoQ AGT rests on measurements that go beyond gut feeling.
Here is how the memoQ research team tests memoQ AGT and what the numbers show. László Laki, Senior NLP Engineer at memoQ, explains the method.
In this article
Machine translation quality evaluation is a way to score how close an automatic translation comes to what a skilled human translator would produce. It is hard to evaluate because there is no single correct translation, and the gold standard, human evaluation, is slow and expensive.
To judge quality properly, translators have to read, rate, or post-edit every output. That does not scale when new models arrive frequently, typically once a month. Without a repeatable method, teams fall back on gut feeling. A few sample sentences look good, so the model must be good. With a new release every month, that hunch gets expensive.
The technology also keeps changing. Statistical MT is gone. Neural machine translation (NMT) replaced it with models of millions of parameters. Today, LLMs with billions of parameters translate text, and everything on the market is neural. Each new release needs a fair, repeatable check against the last one. That is one of the jobs of the memoQ research team.
The team uses an in-house evaluation framework that runs the same test data through different MT systems, models, and prompts, then compares every result against a baseline. Besides measuring quality, they benchmark new market tools and models and prototype AI features for memoQ.
A typical evaluation run follows these steps:
Choose the test data. The team picks a dataset with source sentences and trusted reference translations.
Run the candidates. Each MT engine, LLM, or prompt variant translates the same sentences.
Score the output. Automatic metrics rate every sentence, then the system as a whole.
Compare with a baseline. Results appear as percentage differences from a reference system, next to established MT engines.
Read the sentences behind the numbers. Researchers check where a model gains or loses, then recommend whether it is ready for memoQ.
Because the framework is automatic, the team can test a new model, a new prompt, or a new technique within a short time, instead of waiting for a full human review round.
The team uses two families of automatic metrics, statistical and neural, and both of them are universal. They work on any MT system, not only on memoQ. Here is how the main approaches compare.
| Approach | Examples | What it measures | Trade-off |
|---|---|---|---|
| Statistical metrics | BLEU, chrF++ | Word and character overlap with a reference translation | Fast and widely used, but can miss good translations that use synonyms or different word orders |
| Neural metrics | COMET, MetricX | Meaning, using vector representations of the sentences | Closer to human judgment, but needs more computing power |
| Multidimensional quality metrics (MQM) | MQM-based scoring | Errors, classified by type | Closest to human evaluation among automatic methods; planned but not yet in the framework |
| Human evaluation | Translator ratings, post-editing distance | How much people need to fix | Most reliable, but slow and very expensive |
No single score tells the whole story. That is why the team reads several metrics side by side and keeps human review for the questions that matter most.
Datasets matter because a metric is only as trustworthy as the data it runs on, and building good data takes most of the research effort. Here is what goes into them:
Real translation work. The base dataset comes from memoQ's own documentation translations, plus customer data used with consent.
Translation memory (TM) splits. Each dataset is a TM collection divided into reference, test, and validation sets, so models are never scored on sentences they have seen.
Cleaning first. An automated evaluation metric is only as good as the data it relies on, so we put considerable effort into cleaning the data.
Error hunting. Researchers catch problems such as a source sentence and its reference translation written in different languages and fix them before scoring.
Terminology next. No reliable way exists to measure terminology translation accuracy, so the team plans to build one, along with an evaluation of terminology extraction.
Good data keeps the results honest. A model that scores well on clean, domain-relevant sentences is far more likely to perform well in a real memoQ project.
Measuring AI translation quality lets teams choose the right engine, spot weak releases, and prove real gains with numbers instead of impressions. The main benefits are:
Faster decisions. The team can test a new model or prompt soon after release, without waiting for a full human review round.
Fair comparisons. Every MT engine, LLM, and prompt is scored on the same data with the same metrics, so results line up.
Fewer surprises. A model that scores lower than its predecessor gets flagged before it reaches a project.
Evidence you can share. Benchmarks give customers and partners numbers to check, not only claims.
Applying the same approach has clear use cases. The research team compares MT engines on identical sentences, picks models and prompts for memoQ AGT, and tests new techniques such as adding translation memory and term base context. Any localization team can borrow the logic and score candidate engines on a sample of its own translation memory before committing to one.
memoQ AGT has improved on two fronts over the past year: stronger underlying models and better handling of formatting tags.
memoQ AGT pairs an LLM with your own linguistic assets. memoQ retrieves fuzzy matches from your translation memory and aligned documents in LiveDocs, plus hits from your term base, then passes them to the model as examples. The output follows your terminology and your past translations. The research team’s key finding is that base LLMs already translate general text very well. As László puts it, “the translation business is no longer really about whether we can produce a plain translation. Of course, the models can. It is about how well we can integrate customer-specific or domain-specific data into the translation environment.”
The measured gains came from two changes:
Newer models. Moving from GPT-3.5 to GPT-4.1, then to the latest GPT (5.6 Luna) model, brought a large quality jump at every step.
Tag handling. Formatting from Word and similar files is stored as XML tags inside the text. memoQ AGT did not handle it well before, and now it translates tagged text and keeps the formatting.
In a like-for-like test, memoQ AGT matches DeepL and beats Microsoft. With translation memory matches, it scores far above all of them on segments that have good TM matches.
In October 2026, the research team ran memoQ AGT, Google (its neural and LLM-based models), Microsoft Azure Translator (NMT), and DeepL on 34 test sets in 12 languages, in both directions. That equals 68 test pairs. The sets mix memoQ's own localization content, customer-style documentation, public research corpora, and general text. Every output was scored with chrF++, an automatic metric that compares a translation with a human reference. Competitors ran with default settings, without glossaries, custom models, or translation memories.
| System | Average chrF++ higher is better |
MetricX lower is better |
|---|---|---|
| memoQ AGT with TM matches TM matches from memoQ TMS |
82.2 | 1.84 |
| Google (LLM-based model) | 67.3 | 2.01 |
| Google (neural model) | 65.7 | 2.06 |
| memoQ AGT, no examples | 64.9 | 2.12 |
| DeepL | 63.7 | 2.06 |
| Microsoft (Azure Translator) | 62.4 | 2.33 |
With the average chrF++ over 68 test pairs, the higher the score, the better. All systems were scored on the same test pairs. Only memoQ AGT with TM matches received translation-memory context, up to five matches per segment.
Like-for-like, no examples. With every system translating without examples, memoQ AGT averages 64.9. That is 2.5 points ahead of Microsoft, 1.2 ahead of DeepL and 0.8 behind Google's neural model, a gap too small to tell apart from zero statistically. Google's LLM-based model leads by 2.4 points.
With TM matches. On segments that have good translation memory fuzzy matches, memoQ AGT receives up to five of them as context and averages 82.2. Google's LLM-based model, the strongest competitor, scores 67.3 on the same segments, and memoQ AGT with TM matches comes out ahead on 59 of 68 test pairs. The average lead over the strongest competitor on each pair is 13.6 points.
The second comparison is not like-for-like, because only memoQ AGT receives translation memory context, and it covers segments selected for good matches, not all content. That is the point. It shows what your own translation memory adds to a strong LLM. chrF++ is only one metric, and the team also tracks neural metrics such as COMET and MetricX.
memoQ, a complex language technology provider, goes beyond gut feeling and treats AI translation quality as something to measure, and builds its AI translation tools around the translation memories and term bases that linguists already trust.
This shows up in three ways. First, every model and prompt is tested on the same data before it reaches a project. Second, memoQ AGT uses your own assets as context, so quality improves where it matters most: your domain and your terminology. Third, the output stays in the hands of linguists, who review and refine it inside the translation management system.
The memoQ team is also widening what it measures. Plans include MQM-based scoring, terminology accuracy, chatbot evaluation, and customized model variants such as fine-tuned local LLM models.
Want to see how memoQ AGT works with your own translation memories and term bases? Check out here.