Machine translation (MT) has long struggled to match the quality of human translation, primarily due to challenges in resolving linguistic ambiguity and expressing meaning fluently and accurately in the target language. Rich morphology in source and target languages further complicates the task. Deep learning's recent success in other fields prompted a shift towards neural machine translation (NMT) models, which leverage large training datasets and computational power. This research investigates whether NMT can surpass the previously assumed upper bound of human translation quality. The paper focuses on the development and evaluation of CUBBITT, a novel NMT system, to assess its performance against professional human translations, particularly examining its ability to preserve meaning (adequacy) and its fluency.

The paper reviews previous approaches to machine translation, including rule-based systems and statistical machine-based methods, highlighting their limitations. It then discusses the transformative impact of deep learning on MT, the rise of neural machine translation (NMT) models, and the challenges remaining in the field such as gender bias and unsupervised MT. The authors mention existing parallel corpora (CzEng, Opus) but emphasize the need for more training data, leading to the exploration of data augmentation techniques like backtranslation.

CUBBITT utilizes a Transformer encoder-decoder architecture with multi-head self-attention to capture relationships between sentence elements. A key innovation is the use of a novel "block backtranslation" (block-BT) training regime. In block-BT, the model trains on blocks of authentic parallel data alternating with blocks of synthetic data generated through backtranslation of monolingual target language data. This contrasts with the traditional "mixed backtranslation" (mix-BT) where synthetic and authentic data are randomly interleaved. The system also employs checkpoint averaging, combining the outputs of several checkpoints during training to enhance stability and performance. The authors evaluated different training regimes (block-BT vs. mix-BT, with and without checkpoint averaging) using BLEU scores. They then conducted human evaluations using a context-aware blind assessment, involving both professional and non-professional translators, focusing on adequacy, fluency, and overall quality. Finally, they performed a Translation Turing test to assess the indistinguishability of CUBBITT translations from human translations. The evaluation included analyses of different error types (addition, omission, shift of meaning, etc.) and the impact of cross-sentence context. The study also explored CUBBITT's performance across various document types (business, sports, etc.). The methodology was applied to English-Czech translation and extended to other language pairs (English-French, English-Spanish, English-Polish, Polish-English) to demonstrate generalizability.

CUBBITT significantly outperformed a professional translation agency in terms of translation adequacy in a context-aware evaluation. Non-professional evaluators rated CUBBITT as significantly better in adequacy than the human reference (p = 4.6e-8). While human translation was rated as more fluent, CUBBITT’s fluency was substantially higher than previous state-of-the-art systems. Professional translators also rated CUBBITT significantly better in adequacy (P = 1.4e-4) but rated human translation higher in fluency and overall quality. A Translation Turing test revealed that many participants struggled to distinguish CUBBITT translations from human translations. Analysis of error types showed CUBBITT made significantly fewer errors in adding, omitting, or shifting meaning. However, human translation performed better in fluency-related errors and ambiguous word resolution. The impact of cross-sentence context was highlighted, with CUBBITT making more errors in this area. The study showed a synergistic effect between block-BT and checkpoint averaging, leading to increased translation diversity and novel translations not seen in individual checkpoints. This improvement was consistent across multiple language pairs. The study also showed that document type can affect performance, with CUBBITT excelling in business and political news but performing less well in entertainment and sports.

The results challenge the long-held belief that human translation quality is unattainable for MT systems. CUBBITT's superior adequacy, even surpassing human translation in certain cases, demonstrates the potential of deep learning to replace humans in applications where conveying meaning is paramount. The context-aware evaluation methodology is crucial; previous methods overestimated MT performance by neglecting cross-sentence context. The findings highlight the importance of considering different evaluator preferences (professional vs. non-professional) and the impact of document type on translation quality. The study’s success with block backtranslation suggests the potential of this technique for other data organization methods and machine learning tasks beyond MT.

CUBBITT represents a significant advancement in machine translation, demonstrating that deep learning can achieve near-human or even above-human performance in specific aspects of translation (adequacy). The novel block backtranslation and checkpoint averaging techniques significantly improved translation quality. Future research could explore the generalizability of these techniques to other language pairs and domains and refine the system to further improve fluency and handle complex cross-sentence contexts more effectively. Improving the quality of reference translations and using multiple human references are also important for advancing the field.

The study focused primarily on English-to-Czech news translation, with limited exploration of other domains and languages although some generalizability was demonstrated with additional language pairs. The number of evaluated documents (53) limited strong conclusions about document-level effects. Although the context-aware evaluation was a significant improvement over previous methods, it is still limited by the subjective nature of human evaluation. Future work should address these limitations to ensure more robust and generalizable results.