A Smart Computer

Rico Sennrich’s mother-in-law sends greetings to him on WeChat in Chinese ideograms. Sennrich taps the “Translate” button on his smartphone and receives a translation of the message. The computational linguist speaks a little Chinese himself. Languages interest him. He originally was an Anglicist, and when he long ago attended upper secondary school in St. Gallen, he chose the classical languages profile. What fascinates him most is how human and machine languages work.

Sennrich is a professor of computational linguistics and recently received the Test of Time Award from the Association for Computational Linguistics. The prize is awarded for innovations born from basic research that have had a lasting impact over the past 10 years since their inception.

Sennrich’s invention has had such a lasting impact. It makes everyday life with smartphones, laptops and desktop computers easier. While we use ChatGPT, Claude and all of the numerous AI assistants to generate and translate text as a taken-for-granted matter of course, Sennrich’s algorithms compute a torrent of bits and bytes in the background at breakneck speed. Exactly 10 years ago, in 2016, Sennrich and his team unveiled a new method that proved to be formative to most commercial neural language models.

With tokenization, i.e. the segmentation of language into morphemes, a technique was successfully introduced that enables generative pretrained transformer (GPT) models to process all of the words in a language instead of confining themselves to just the most common ones, as they used to do. Originally developed for machine translation, today tokenization is the bedrock underlying practically all large language models (LLMs).

Sennrich smiles impishly. From 2013 through 2015, he was a guest postdoctoral fellow at the University of Edinburgh on a Swiss National Science Foundation scholarship. When he afterwards applied for an advertised postdoc position, during the job interview he was asked what kind of project he would pursue if he could devote his scientific attention to artificial neural methods, which were still new at that time. He replied that he had an idea on how to advance from a machine translation system with a limited vocabulary to a system with unlimited translation capability. Sennrich got the job, and this year he reaped an award for that idea, which proved to be groundbreaking.

Sennrich jumps up from the conference table, sits down at his computer and types a long word into a blinking text field: Corona­pandemie­massnahmen­gegner:innen (a German compound noun that means “people opposed to coronavirus pandemic mitigation policies” in English). The computational linguist shows me how ChatGPT slices the word into tokens. The carved-up word glows in different-colored fragments. Compound words, which are particularly common in the German language, used to pose an enormous challenge for machine translation systems. Sentences like “Er hängte die Wäsche auf” (“he hung up the laundry”) or “Sie legte den Mantel ab” (“she took off the coat”) were also problematic because the systems were incapable of recognizing the detached verb particles “auf” (“up”) and “ab” (“off”).

Anything unknown or rare posed a problem for earlier machine translation systems. “Yet, the crucial information in a sentence often lies precisely in elements of that kind,” Sennrich explains. Take, for example, “Ich wohne in Hindelbank” (“I live in Hindelbank”). To a human it’s obvious that the last word in that sentence is the name of a place. For earlier translation systems, though, everything was readable except the unknown word at the end of the sentence.

Still facing his computer screen, Sennrich explains that “in the process of tokenization by means of byte pair encoding, frequently occurring character sequences are merged into a unit while rare sequences get broken up.” This enables neural networks to process even rare words. Previously, a computer had to learn each individual word in a language by heart, in a sense. Segmentation into morphemes now enables the system to translate even certain unknown words accurately by assembling them from known components, just like piecing together a puzzle.

A tiny robot with a ticking clock embedded in its belly stands next to Sennrich’s computer. The robot looks cute and a bit outdated. Sennrich grew up in St. Gallen in the 1990s. His parents ran a hairdressing business. Sennrich was a science fiction literature enthusiast as a young boy. One can easily imagine him reading books in his parents’ beauty salon amid wafting wisps of shorn hair and a flurry of blow-dryer action. When he was around 10 years old, his godfather gave him his old computer as a gift. Sennrich reflects for a moment. It probably ran on Windows 3.1 back in the day. The computer fascinated him. He played the common games of the time, such as Minesweeper and Solitaire. But he mostly enjoyed exploring the digital universe and playing with graphics programs. And soon he was writing his first simple code to gain an understanding of how a computer program works.

When he set off for Zurich to study at the university, he wavered between English linguistics and informatics. He opted for linguistics and then discovered computational linguistics at UZH, which he ultimately chose as a minor. “Computational linguistics allows you to test your theories more easily, and that intrigued me,” Sennrich says. For his first project in computational linguistics, he wrote a program that parses grammatical relationships between words in a sentence and visualizes them in a syntax tree. Sennrich specialized in natural language processing. He then wrote his dissertation on the subject of machine translation.

Sennrich actively experiences multilingualism also in his personal everyday life. He speaks English with his Chinese wife – they met in Edinburgh and she, too, is a linguist – and speaks Swiss German with their two-year-old son. He communicates with his in-laws in Chinese and English with the help of a variety of AI tools. Sennrich laughs. Many tourists whip out a translation app when they’re puzzling with a furrowed brow over a menu written in a foreign language. “AI enables us to communicate even in foreign languages that we don’t know,” Sennrich says, “creating a great opportunity to tear down barriers and forge mutual understanding across language and cultural borders.”

And what’s next on deck? There is potential for improvement with regard to minor languages. Sennrich’s research group has just found a way to enable AI to translate Romansh Grischun into the various Rhaeto-Romanic dialects. The translation of lengthy scientific papers into different languages also continues to pose difficulties for neural language models because the sophisticated context, after all, has to be correctly identified. “We are specifically training them on the context level so that texts become more coherent and nothing gets omitted,” Sennrich says. Structural text characteristics are to be incorporated as well. It will be possible to apply these elaborate algorithms also to other texts later on.

And there’s yet another problem that the clever computational linguist wants to tackle. The tokenization of languages produces uneven costs. English words consist of nothing but single-token units, in stark contrast to other languages like Romansh, for example, in which a word splits into four to five tokens. Costs are billed in hard currency per token. This means that English is cheaper than many other languages. Sennrich wants fairness between languages and has set about developing a technique that compresses even complex languages with lengthy token sequences per word.

The Association for Computational Linguistics, the body that awarded Sennrich the Test of Time accolade, will be keeping an eye on his smart algorithms. Who knows? Maybe it will give him call again 10 years from now.