The build or buy trap in enterprise translation, and the last 20 percent that decides it

By Lee Konstanty, GM, arbitr

The finding. In 2026, an engineering team can stand up a working translation prototype in days. Run content through a capable model, wire up a queue, and the output looks fluent enough to demo. Call that the first 80 percent. The remaining 20 percent, deterministic terminology enforcement, measured quality, and human review sustained past year two, is where the economics come apart. The seed analysis from Nimdzi frames the trap precisely: the hidden complexities of production-grade language infrastructure only surface after a successful proof of concept, and the line between a working demo and a sustainable enterprise platform is often the difference between innovation and new, brittle technical debt. So the real build or buy question for high-stakes, domain-specific content is not "can we build it." A weekend prototype already proves you can. The question is whether you can sustain the last 20 percent, and what it costs to do so.

This article works through the evidence on that question: why prototypes stall before production, where the money actually goes, why generic models miss inside a specific domain, how translation quality gets measured, and why specialization tends to beat raw model scale for bounded, repeated work. Then it says plainly where arbitr sits in that answer.

Why the prototype is not the product

The distance between a fluent demo and a production system is larger than most teams budget for, and the base rates are not kind. RAND finds that by some estimates more than 80 percent of AI projects fail, roughly twice the failure rate of IT projects that do not involve AI (RAND). A 2025 BCG study of more than 1,250 companies found that 60 percent report little to no value from their AI investments, and only about 5 percent qualify as fully AI future-built (BCG, BCG). McKinsey's 2025 State of AI survey found that almost all organizations now use AI somewhere, yet most remain in the early stages of scaling it and capturing enterprise-level value (McKinsey). Gartner projected that at least 30 percent of generative AI projects would be abandoned after proof of concept by the end of 2025, citing weak data quality, rising costs, and unclear business value (Gartner).

These are general AI failure rates. Translation compounds them, because language-specific complexity sits on top of the usual deployment risk. Nimdzi's own research found that close to 30 percent of enterprise localization leaders cannot find a market solution that covers their full language-platform requirement, which is a genuine reason teams consider building (Nimdzi). The same research names the counterweight: reduced complexity, access to mature capabilities, and faster deployment are the top reasons teams decide to buy. This is not a binary. MIT Sloan frames it as three paths, buy, boost, or build, and notes that buying allows quick adoption without investing in development or fine-tuning, while building is reserved for where a proprietary advantage justifies the sustained cost (MIT Sloan). The disciplined version of the question is not build or buy. It is which layer to buy and which narrow layer to build.

Where the money actually goes

Build accounting usually gets the visible costs right and the invisible ones wrong, and the invisible ones are the larger number. The foundational result here is not about translation at all. In a widely cited analysis of production machine learning systems, a team of Google researchers argued that it is dangerous to treat a quick, working model as a finished asset, because the model code is only a small fraction of a real-world system; the surrounding infrastructure, data plumbing, monitoring, and ongoing maintenance dominate the true cost and accrue as hidden technical debt that compounds over time (NeurIPS). The compute line, the part that looks like every other infrastructure decision, is not where a build gets expensive. The engineering around it is.

That reframes the build spreadsheet. Once engineering time, infrastructure operation, hardware utilization, and the opportunity cost of the team's attention are all counted, organizations routinely discover that the true cost of running inference in house is a large multiple of the figure they budgeted, because the parts that do not look like a line item, above all the standing team that keeps the system current as models shift beneath it, are the parts that cost the most. A production translation pipeline is not a model. It is a system, and a system needs a standing engineering team, so over a multi-year horizon the people cost runs well above the infrastructure cost.

Translation has its own version of this. In-house localization never reaches a maintenance-free state. String extraction, content synchronization across a changing product, and the review workflow itself are continuous obligations rather than one-time builds, and because those hours are absorbed into the salaries of engineers who already work there, they rarely appear as a distinct budget line. Nimdzi makes the same point from the language-infrastructure side: the hidden complexities of production-grade language infrastructure only surface after a successful proof of concept, and what looked like a finished demo becomes brittle technical debt (Nimdzi).

The buy case follows directly from where the money goes. The translation model itself is close to a commodity. Nimdzi's cost analysis of machine translation options puts the annual model spend for ten million words a year at roughly 2,100 dollars for a generic engine, 6,000 for an adaptive engine, and 8,000 for a large language model based approach with prompting and management (Nimdzi). Those are not the figures that break a budget. The expensive part is everything wrapped around the model: the terminology discipline, the measured quality, the review workflow, and the standing team to keep all of it current. That is exactly the part a mature platform has already built and amortized across many customers, and exactly the part a build has to fund alone, indefinitely.

Why generic models miss inside a specific domain

The build case often rests on an assumption that a capable general model, prompted well, will translate a bank's disclosures or a medical device's instructions as competently as it drafts an email. The research on domain adaptation does not support that assumption. A survey of domain adaptation for neural machine translation states the core finding plainly: modern systems become extremely powerful given sufficient in-domain training data, but they struggle when translating text from a new domain with a distinct style or vocabulary, and naive fine-tuning risks overfitting and catastrophic forgetting of previously learned behavior. A dedicated study of that failure mode confirms it: adapting a model to a domain by fine-tuning routinely degrades performance on everything the model previously handled, so the adaptation itself is an engineering problem, not a one-line command (ACL Anthology).

Terminology is where the gap shows up first. An evaluation of terminology translation in adapted neural MT found that when a generic engine receives requests from a new domain full of specific technical terms, adapting the model to the domain significantly boosts the accuracy of translated terms relative to the generic baseline. The 2025 Conference on Machine Translation added a dedicated terminology task precisely because accurate, consistent handling of specialized terms is what high-stakes domains require and what generic systems handle least reliably (WMT25 terminology task). The pattern across the literature is that domain fit, not raw capability, is the variable that most affects usable quality on specialized content. A commodity model is a commodity precisely because it does not know your terminology.

There is also a failure mode that a fluent demo hides entirely. Apple's machine learning research reports that translation systems built on large language models carry a higher risk of hallucination than dedicated encoder-decoder models, and that fine-tuning on a hallucination-focused preference dataset reduced the hallucination rate by an average of 96 percent across five language pairs, and by 89 percent in a zero-shot setting on unseen languages (Apple). A comprehensive study across more than 100 language pairs found that models hallucinate primarily in low-resource directions and when translating out of English, where they can surface toxic patterns traceable to training data (ACL Anthology). A weekend prototype measures none of this, because the person building it is reading the fluent cases, not sampling the tail.

Quality you can actually measure

If the last 20 percent is quality, then quality has to be measurable, not asserted. This is a mature discipline with published standards, and a build has to reimplement all of it. ISO 5060:2024 gives guidance on evaluating translation output, human, post-edited, and unedited machine output alike, using an analytic approach based on error types and penalty points that produce an error score and a quality rating. The Multidimensional Quality Metrics framework, which underpins the annual Conference on Machine Translation shared tasks, provides the error typology and scoring models that convert counts of error type and severity into a numeric quality score against a specification. These frameworks exist because quality is otherwise an elastic impression that varies from reviewer to reviewer.

The measurement discipline also warns against declaring the problem solved. The organizers of the 2024 machine translation shared task titled their findings "the LLM era is here but MT is not solved yet" (WMT24), and their 2025 findings called on the field to stop evaluating on easy test sets because current systems mask their failures on realistic input (WMT25). The operational consequence is that quality at scale is a routing and review problem, not a one-time acceptance test. As volume grows past the point where one linguist can read every string before launch, the discipline becomes scoring content by risk, evaluating against a fixed rubric, and sending the uncertain cases to a human. A build has to construct that scoring and review apparatus in-house, and keep it current as the underlying models shift beneath it.

Why specialization tends to beat scale

The most useful recent shift in enterprise AI is the recognition that for bounded, repeated tasks, a smaller specialized model often outperforms a larger general one. Gartner projects that by 2027 organizations will use small, task-specific models three times more than general-purpose large language models (Gartner). The translation literature reaches the same conclusion from the inside. In the 2025 machine translation shared task, strong neural systems remained competitive on lexical scores while fine-tuned large language models showed marked superiority in semantic fidelity, and fine-tuning a model directly on the translation task beat using it to refine another system's output (WMT25 system paper). Fine-tuning a compact model on an organization's own translation memories is a documented and effective path, not a research fantasy (arXiv).

Specialization does not remove the human. It positions the human where the consequence is highest. An experience report from a global medical technology company, built on more than 11,000 ranked translations across five language pairs, describes human-in-the-loop validation as the critical control for responsible deployment, and records reviewer preferences between smaller and larger language model outputs under different configurations (ACL Anthology). The standards agree: ISO 18587:2017 sets requirements for full human post-editing of machine translation output and for post-editor competence, because for regulated content a qualified human in the loop is a requirement, not a preference. The defensible architecture is therefore consistent across the cost evidence, the quality evidence, and the regulatory evidence: a specialized model for the domain, a measured score on every segment, and a human approving what ships.

Where arbitr fits: the specialized model is the product

The evidence above proves that specialization and fine-tuning beat generic scale on high-stakes content. The direct consequence is the part most build-versus-buy pieces leave implicit: that specialized model is what arbitr is.

arbitr builds custom small and large language models, purpose-trained for specialist domains. They are fine-tuned from a foundation model trained on more than ten years of professional human translation across ten language pairs, German, French, Spanish, Italian, Dutch, Polish, Finnish, Portuguese, Chinese, and Japanese, each paired with English. From that foundation, arbitr trains domain-specialist models per industry and language pair, for example a Japanese to English model built for the financial sector, so a banking disclosure is not translated like marketing copy and clinical content is not translated like a product manual. This is the domain-adaptation and small-model research above, realized as a product rather than proposed as a paper. On specialist content, a domain-specialist model outperforms both generic machine translation and general-purpose large language models, and approaches the quality of a professional human translator. You do not have to assemble this. The right specialist is applied automatically for your industry, and where a domain needs its own model, arbitr builds one for you.

Quality is measured by a model, not by assertion. A dedicated neural quality-estimation model scores each translation for accuracy, fluency, and adequacy, trained on human linguist ratings; that score is arbitr's confidence score. Separately, arbitr's Confidence Agent runs automated checks on every segment before it reaches a person, flagging errors and inconsistencies, and segments below the configured confidence threshold are routed for reviewer approval, which is the risk-based routing the quality literature calls for. The reviewer is structurally inside the workflow, not an afterthought. Your team accepts, rejects, or edits terminology decisions and approves the final translation before it ships, and every reviewer decision and terminology accept or reject is logged with a timestamp and reviewer identity. That is the measured quality and the human in the loop that the last 20 percent demands, delivered rather than rebuilt.

Then the part that answers "why not just build it ourselves" for good. Every approved decision is promoted into Cortex, arbitr's customer-controlled, tenant-isolated organizational memory: a governed knowledge ledger that is the system of record for your translation decisions. Reviewer-approved terminology and corrections become durable, reusable institutional knowledge and are reused in future work rather than re-litigated string by string, so the platform you bought becomes more yours the longer you use it. This is the asymmetry that settles the decision. A build depreciates: the engine you stand up this quarter is technical debt next year, and the terminology your team resolves lives in scattered spreadsheets and people's memory. A bought platform with governed memory compounds: every reviewer-approved decision is reused on the next translation and stays with you. Buying compounds. Building decays. That is the wedge today. Translation today, content operations tomorrow.

The takeaway

The build or buy decision for enterprise translation is not settled by whether a demo is possible. It is settled by the last 20 percent: deterministic terminology, measured quality, and human review, sustained past year two. The research is consistent on three points. Most AI projects fail before they deliver value, and translation compounds that risk. Domain fit, not model size, is what most determines quality on specialized content. And for bounded, repeated work, a specialized model with a human in the loop is both more accurate and more sustainable than a larger general one operated alone. The defensible path is to buy the specialized model, the measured score, and the governed memory, and to spend your own engineering only where it differentiates the business. That is precisely the shape of arbitr.

References

1. Nimdzi, The Build or Buy Trap of Language AI Platforms. https://www.nimdzi.com/the-build-or-buy-trap-of-language-ai-platforms/
2. RAND, The Root Causes of Failure for Artificial Intelligence Projects. https://www.rand.org/pubs/research_reports/RRA2680-1.html
3. BCG, The Widening AI Value Gap (weekly brief). https://www.bcg.com/assets/2025/252-weekly-brief-the-widening-ai-value-gap.pdf
4. BCG, Are You Generating Value from AI? The Widening Gap. https://www.bcg.com/publications/2025/are-you-generating-value-from-ai-the-widening-gap
5. McKinsey, The State of AI: Global Survey 2025. https://www.mckinsey.com/capabilities/quantumblack/our-insights/the-state-of-ai
6. Gartner, 30% of Generative AI Projects Will Be Abandoned After Proof of Concept by End of 2025. https://www.gartner.com/en/newsroom/press-releases/2024-07-29-gartner-predicts-30-percent-of-generative-ai-projects-will-be-abandoned-after-proof-of-concept-by-end-of-2025
7. MIT Sloan, Buy, boost, or build? Choose your path to generative AI. https://mitsloan.mit.edu/ideas-made-to-matter/buy-boost-or-build-choose-your-path-to-generative-ai
8. Saunders, Domain Adaptation and Multi-Domain Adaptation for Neural Machine Translation: A Survey (JAIR). https://www.jair.org/index.php/jair/article/view/13566
9. Domain adapted machine translation: what does catastrophic forgetting forget and why? (ACL Anthology). https://aclanthology.org/2024.emnlp-main.704.pdf
10. Evaluation of Terminology Translation in Instance-Based Neural MT Adaptation (ACL Anthology). https://aclanthology.org/2018.eamt-main.15.pdf
11. Findings of the WMT25 Terminology Translation Task (ACL Anthology). https://aclanthology.org/2025.wmt-1.30/
12. ISO 5060:2024, Translation services, Evaluation of translation output, General guidance. https://www.iso.org/standard/80701.html
13. The Multi-Range Theory of Translation Quality Measurement (MQM scoring models). https://doi.org/10.48550/arxiv.2405.16969
14. WMT24, Findings of the General Machine Translation Shared Task. http://www2.statmt.org/wmt24/pdf/2024.wmt-1.1.pdf
15. WMT25, Findings of the General Machine Translation Shared Task. https://aclanthology.org/2025.wmt-1.22/
16. Apple Machine Learning Research, Mitigating Hallucinated Translations in Large Language Models. https://machinelearning.apple.com/research/mitigating-hallucinated-translations
17. Hallucinations in Large Multilingual Translation Models (ACL Anthology). https://p.rst.im/q/aclanthology.org/2023.tacl-1.85.pdf
18. Gartner, By 2027 Organizations Will Use Small, Task-Specific AI Models Three Times More Than General-Purpose LLMs. https://www.gartner.com/en/newsroom/press-releases/2025-04-09-gartner-predicts-by-2027-organizations-will-use-small-task-specific-ai-models-three-times-more-than-general-purpose-large-language-models
19. DLUT and GTCOM, LLM-Based Translation System for WMT25. https://www2.statmt.org/wmt25/pdf/2025.wmt-1.49.pdf
20. Fine-Tuning Large Language Models for In-House Translation (arXiv). https://arxiv.org/html/2409.03454v2
21. Implementing Machine Translation in a Regulated Industry (ACL Anthology). https://aclanthology.org/2025.emnlp-industry.117.pdf
22. ISO 18587:2017, Translation services, Post-editing of machine translation output, Requirements. https://www.iso.org/standard/62970.html
23. Sculley et al., Hidden Technical Debt in Machine Learning Systems (NeurIPS 2015). https://proceedings.neurips.cc/paper_files/paper/2015/file/86df7dcfd896fcaf2674f757a2463eba-Paper.pdf
24. Nimdzi, The Real Cost of Customized NMT. https://www.nimdzi.com/the-real-cost-of-customized-nmt/