Most companies offering translation products claim that their translation quality is superior to others. So we started a journey to explore how they perform in real life. This post starts with the test detailed below, but will be extended over time with additional tests, as evaluating translation products is not an easy nor a short task.

Disclaimer: This analysis is not the most comprehensive test of these tools but offers a glimpse into the progress of translation technology.

Test Overview

There are many ways to evaluate translations, but for an objective comparison, we chose to translate 50 different sentences from English to Spanish and then back into English using the same tool. By comparing the original and the twice-translated content, we aimed to assess how well each tool preserves meaning. We tested ChatGPT 4.0, Google Translate, and DeepL. The results reveal significant differences in their performance, particularly in handling idiomatic expressions, slang, and contextual nuances.

ChatGPT 4.0: The Clear Winner

ChatGPT 4.0 emerged as the best performer in our test, with only 9 changes between the original and double-translated content. The meaning was consistently preserved, with no significant errors. Most alterations involved single words with similar meanings, and the tool demonstrated impressive handling of slang and non-literal expressions. For instance, "That's the straw that broke the camel's back" remained almost intact, showing ChatGPT's strength in contextual understanding.

Google Translate: A Formal Approach

Google Translate had 28 changes, showing noticeable alterations in some sentences. The tool often failed to maintain the original expressions, opting for more formal language. For example, "That's the straw that broke the camel's back" became "That's the last straw," and "under the weather" was translated to "sick." Google Translate's tendency to avoid slang and informal expressions was a recurring theme, leading to a less accurate reflection of the original content.

DeepL: Struggling with Nuance

DeepL faced the most challenges, with 40 changes between the original and double-translated content. Pronoun errors were common, such as "She is as cool as a cucumber" being translated to "It's as cold as a cucumber," which altered both the meaning and the gender reference. Time-related phrases also proved difficult, with "Let's watch a movie tonight" becoming "We're going to watch a movie tonight," indicating a shift in intent and context. Overall, DeepL struggled with maintaining the original nuances.

So how do they compare?

For translating creative writing, marketing content, or any material where tone and context are paramount, ChatGPT 4.0 (or newer models) is likely the best choice. Google Translate might suffice for technical documents, straightforward content, or formal communications. DeepL, despite its challenges in this test, may still be helpful when accuracy in straightforward translations is the priority, though it may require additional editing.

The Future of Translation Technology

As AI continues to evolve, we can expect further improvements in translation technology. Tools like ChatGPT 4.0 are already setting new benchmarks for quality, but there's still room for growth, especially in refining the handling of idiomatic and culturally-specific content. The ongoing development of these tools will likely lead to more nuanced and accurate translations, making them increasingly valuable for both personal and professional use.

Conclusion

In this test, ChatGPT 4o proved to be the most reliable translation tool, with minimal changes and a strong grasp of context and idiomatic language. Google Translate, while decent, leaned towards a more formal and less expressive translation. DeepL, despite its reputation, lagged behind in preserving meaning, especially in nuanced and context-dependent sentences. These results suggest that while all three tools have their strengths, ChatGPT 4.0 currently offers the most accurate and context-aware translations in 2024.