Ask any non-native English speaker, and they’ll probably tell you that LLMs tend to perform much better in Shakespeare’s language than in their own
Sometimes, the difference is subtle. Sometimes, not so much. Sometimes, it’s downright dangerous, as shown in this 2023 Carnegie Mellon study, which found that non-English inputs could more easily bypass safety filters.
Now, Apple has co-authored a study proposing a new method that could close part of this gap.
As Apple explains it:
Current Large Language Models are predominantly designed with English as the primary language, and even the few that are multilingual tend to exhibit strong English-centric biases.
Much like speakers who might produce awkward expressions when learning a second language, LLMs often generate unnatural outputs in non-English languages, reflecting English-centric patterns in both vocabulary and grammar.
In other words, even when models generate Chinese or French, they still “think” in English. The result? Non-English outputs still follow English-like grammar and vocabulary patterns.
To test this, Apple researchers, alongside researchers from Inria Paris, École Polytechnique, and Sapienza University of Rome, introduced two new metrics:
- Lexical Naturalness: Does the model use vocabulary like a native speaker would?
- Syntactic Naturalness: Does it structure sentences in a way that matches native grammar?
They compared model outputs to native-written Wikipedia articles in Chinese, French, and English.
The results confirmed the bias. Even the Chinese-developed model Qwen underperformed in all languages, including Chinese. Meta’s Llama 3.1 was the most natural overall, but still trailed far behind human-level output.
Apple’s proposed fix
To close the gap, Apple trained a model to prefer natural-sounding outputs over awkward ones, using a pretty clever method: instead of manually collecting unnatural examples, they generated them automatically using back-translation.
A fluent human-written Chinese response would be translated to English, then back to Chinese, introducing subtle unnatural patterns known as “translationese.” These manipulated outputs served as negative examples, while the originals were used as preferred responses.
By training the model to prefer the more natural version, Apple was able to significantly improve both vocabulary choice and grammar, without degrading general performance in standard benchmarks.
FTC: We use income earning auto affiliate links. More.
