SOTA machine translation at Unbabel

José and Ricardo joined Daniel at EMNLP 2022 to discuss state-of-the-art machine translation, the WMT shared tasks, and quality estimation. Among other things, they talk about Unbabel’s innovations in quality estimation including COMET, a neural framework for training multilingual machine translation (MT) evaluation models. Featuring: Ricardo Rei – X https://x.com/RicardoRei7José Souza – X https://x.com/accezzDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Unbabel https://unbabel.com/COMET https://unbabel.com/research/comet/The WMT workshop/ conference https://www.statmt.org/wmt22/EMNLP https://2022.emnlp.org/Upcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  We have started a project on this. It's to combine these systems, these quality estimation systems, with the machine translation itself. So that is something that we started working on this, but I believe that you can work on this for the next few years. And there is a lot of things that we can improve there. Yeah, that gets me really excited. I think it's a direction that it's going to be really nice.

• 0:22Speaker 01

  This is the Quality Aware Decoding Project. That is basically what I just mentioned about, what we have been talking about of having these quality predictions about the hypothesis translations. The idea behind this project that Ricardo is talking about is, what if we bring the quality estimation or comment already to inside the MT process? And then we can make the machine translation aware or more aware about its quality, having a signal from a different model. So this is what this project is about.

• 1:04Speaker 02

  Welcome to Practical AI, a weekly podcast making artificial intelligence practical, productive and accessible to everyone. Subscribe now if you haven't already. Head to practicalai.fm for all the ways. Special thanks to our partners at Fastly for delivering our shows super fast to wherever you listen. Check them out at fastly.com. And to our friends at fly.io. We deploy our app servers close to our users, and you can too. Learn more at fly.io.

• 1:39Daniel Whitenack?

  Welcome to another episode of Practical AI. This is Daniel Whitenack. I'm a data scientist with S I L International, and I'm joined this week by Ricardo Ray and Jose Souza from UnBabel here at E M N L P 2022 in Abu Dhabi. How are you doing, guys?

• 1:57Speaker 00

  Hi, we are fine. Hi, good.

• 1:59Daniel Whitenack?

  Yeah. How's E M N L P for you?

• 2:01Speaker 00

  So far, we have been mostly attending the W M T workshop.

• 2:05Daniel Whitenack?

  Yeah, and what's the W M T? What does that stand for?

• 2:08Speaker 00

  Right, WMT stands for Workshop on Statistical Translation.

• 2:13Speaker 01

  Workshop on Machine Translation.

• 2:14Speaker 00

  On Machine Translation, but it was, this is an historical acronym. Because it's actually now a conference. It's, I would say that it's the main conference of machine translation. And it has been happening for several years. And it's always co-located with the EMNLP. So it's nice because it's one of the biggest NLP conferences together with the biggest MT conference.

• 2:37Speaker 01

  It's mostly attended by researchers. So not so much by people in localization industry. But it's interesting to know what's happening in terms of research, the latest approaches and methodologies for evaluation as well, so.

• 2:53Daniel Whitenack?

  Yeah, and is that the industry that UnBabel is in? Could you just give people a little bit of an understanding of what UnBabel is?

• 3:00Speaker 01

  Sure, so UnBabel is a translation company. We provide translations, trying to unite the best of both worlds, which is using machine translation and professional translators to provide these translations. And the best of both worlds, because if you only rely on translators themselves, it's very difficult to scale this process of translation to different volumes of content. And that's why you use machine translation to speed up this process and then you use the translators to correct if necessary. And that's, I think, the biggest difference of UnBabel to other companies, which is we are the pioneers to use something called quality estimation

• 3:44Speaker 01

  to actually decide whether if we should post-edit or not the translations. And I guess we are big on also evaluation technology, evaluation. And I think Ricardo can talk about Comet.

• 3:58Speaker 00

  Yeah, like what Zed just explained about the difference between combining humans and MT. So if you have a mechanism that tells you that your machine translation output is perfect, then you don't need a human. But for you to do this, you clearly need a very reliable quality estimation system, a system that receives... gives that translation and is able to give you an accurate score for that translation. And that's why, uh, and Babbo has been focusing for so many years on specifically quality estimation and also evaluation. Evaluation, it's a little bit more general. It can also include things like metrics where you compare the translation output with a reference translation that you believe to be perfect.

• 4:43Speaker 00

  And it's what people typically use when training models and stuff like that. For the past few years, we have been developing a metric that is, uh, being widely adopted by the research community and also the industry, which is called Comet. Comet has been very successful in the last two years. And yeah, it was developed by us. Uh, we also developed a quality estimation framework that was also, uh, gained a lot of traction three years ago, I think.

• 5:13Speaker 01

  Yeah, 2019 it was.

• 5:14Speaker 00

  Yeah, called Open Kiwi, which is basically similar in this, in terms of the model approach and everything, but it does not rely on a reference. So it's what we use internally for performing quality estimation. Yeah, I think this sums up a little bit.

• 5:31Speaker 01

  That, that said, just one thing is that all of this is only possible because over the years, Am babel established some quality controls for the translations. And, uh, this is started by using something, a framework called MQM, which stands for Multi-Dimensional Quality Metric, which is basically a typology and then guidelines on how to use this typology with different phenomena that happens when people, when translation is made. That goes from accuracy, you know, whether the, the translations are adequate, if they're fluent, and then there's a whole taxonomy about that.

• 6:10Speaker 01

  So this kind of evaluation enabled us to accumulate data about the quality of translations over time that we can then use to train quality estimation or, uh, metric, uh, evaluation models.

• 6:24Daniel Whitenack?

  Yeah, so this seems different. I think some listeners probably in their experience with like modeling and other domains or with other data are probably familiar with like a confidence score or a probability. So this goes like way beyond that, right? So just to clarify, this is not like just a confidence score coming out of your model like of translation, but this is actually a metric that you're running on the output of your model. Is that right? Exactly.

• 6:50Speaker 00

  Yeah, exactly.

• 6:51Daniel Whitenack?

  Yeah. And so explain maybe Comet a little bit because that has like gained so much traction. What is maybe different about Comet? Another, you know, popular one I know for machine translation is called BLEU. So what distinguishes Comet as different from maybe that or like other metrics that are out there?

• 7:10Speaker 00

  So like you were saying, BLEU is a very well-known metric, but BLEU is a lexical metric. And this means that BLEU will take the empty output and it will compare with a reference that was created from a human. And usually the typical setup is that we only compare that empty output with a single reference. And as we might know, there are multiple ways to translate a specific sentence. So a lot of times BLEU will give a very low score for a very good translation because of that. Sometimes it also gives you a very high score for a very bad translation because of another aspect of BLEU. It's that it's

• 7:54Speaker 00

  gonna give the same weight to all words. So if you have a named entity that is not correctly translated, it's going to be like one word that is missing from being perfect and BLEU will give a very high score. If you miss like a punctuation, the score penalty will be exactly the same, although the errors are completely different in terms of severity.

• 8:16Speaker 01

  Just one thing to differentiate between, just to explain a little bit more BLEU is that the way that it looks at both the translation hypothesis and the references, looking at the each word and trying to understand if there is an overlap of each word with the reference. So. It does that for combinations of, for one word or for combinations of two, three and four words usually, which we call N-grams. So, and then it has a brevity penalty that is basically to penalize if the translation is too small, too, too, too short. So that's basically the rationale. And there is a class of metrics called like that, that I think we are calling lexical metrics.

• 8:56Speaker 00

  Yeah, lexical metrics.

• 8:57Speaker 01

  Yeah, so T E R, which is translation error rate, it's similar to that. C H R F, it's similar to that, but C H R F goes at the character level. So this is a class of things that is very different from Comet, I think.

• 9:11Speaker 00

  Yeah, Comet takes advantage of the representations coming for large language models, like XLM-Ruberta, we have been using XLM-Ruberta. And basically those representations allow you to compare words in an embedding space. So to words that might not be exactly the same, but have the exact same meaning, the Comet will use those representations to output a score. Now, the other thing that we add on top is that we train those representations to be more suitable for the specific task of machine translation evaluation. And I'm saying this because this is a very important difference from other metrics that have also been proposed,

• 9:53Speaker 00

  like BirdScore, where because of the fact that you don't have any fine tuning on top, if you use BirdScore and you, you say "I love you" or "I hate you", because love and hate will have similar embeddings, the score will be very high when in fact they are the complete opposites. So we start from a pre-training model, but then by training the model with some supervision from human labels on errors, the model learns that "I love you" or "I hate you", on, for this specific task, they are complete opposites. And I, I think that kind of splits apart Comet from all the metrics that were being proposed before,

• 10:36Speaker 00

  that either fall into the lexical category or into the embedding category.

• 10:41Daniel Whitenack?

  Yeah, that's great. And you also mentioned, just in passing, like, there was another kind of category of quality estimation that didn't require a reference. Could you talk about that a little bit?

• 10:51Speaker 00

  Yeah, so the idea is very similar to the idea of Comet. So the difference is that when you have access to a reference, which is the case of Comet, when you create the embeddings for the MT outputs, they will be perfectly aligned with the embeddings from the reference because they are on the exact same language. On quality estimation, you are comparing it directly to the source. So the embeddings will not align perfectly. And still, what happens is that during training, using human supervision, the model learns to what is correct and what is incorrect, only comparing the MT output directly with the source.

• 11:32Speaker 01

  So quality estimation serves a different kind of application than the metrics like BLU, CHRF and Comet, which is usually I want to know what is the quality of specific sentences or translations, given their source sentences. For Comet, usually what you're more interested, Comet or the other metrics, you're more interested in understanding the difference between models or MT systems. So you're evaluating at some sort of, trying to understand at some sort of test set level or evaluation set level, so that you can decide whether I go with model, MT model A, B or C. And then in quality estimation, it's basically to take decisions on the fly at real time, in which I cannot wait for someone to make a reference

• 12:16Speaker 01

  or a post addition and decide, okay, can I trust this translation? If I don't, should I throw it out? Is that better that I should do it from scratch? Or I can still give it to someone that can repurpose this and rephrase it, you know? So there's slightly different in the applications. But this is something that you can talk about trends. They start to, like, I think, Ricardo was teasing, to intersect themselves a bit.

• 12:43Speaker 00

  I would say that the metrics field, so the evaluation on the metrics side, was stuck with BLU for a long time.

• 12:51Speaker 01

  Quality estimation, on the other hand, was, I feel that there were more research and more innovation on that field. Actually, that was our motivation when we built Comet. We tried to replicate what was being done, the state of the art of what was being done on quality estimation. We tried to bring it to the metrics field. And now, the modeling approaches are very similar, but it was viewed as two completely different tasks for years. So, just to give insight, to add a bit of context on what Ricardo said about the progress in quality estimation. So, I did my PhD working on this kind of problem, and I finished like in 2015.

• 13:36Speaker 01

  So, I was working from 2012 until 2015 on problems around this. And the approaches back then, they were basically using feature-based approaches like classical machine learning. And with deep learning and access to embeddings and now large protein models, this very, very fast shifted to this kind of approaches. And the performance of these models, of these approaches also are much better than when I used to first work on this. So, the quality of these quality estimation models nowadays, they are very useful. You can actually do a lot of things with them, like I was saying.

• 14:14Speaker 01

  And yeah, I just wanted to compliment that because for me, I was not working on the field specifically this problem for, I don't know, three years, I guess. When I came back to ETL, I was like, whoa, now it's really up to everything, you know?

• 14:36Daniel Whitenack?

  Could you explain a little bit, so you mentioned how like in Comet or in these other models, you might be comparing like the embeddings of words. But words don't always map like one-to-one between languages. And sometimes I don't know if you're looking at sentences or other things. But could you describe like some of the, I guess, what are the main challenges looking forward that like aren't solved yet in terms of like next steps with quality estimation? And things that you're looking at now that you see as as open problems?

• 15:05Speaker 00

  Yeah, you actually touched a very nice, very nice point. It's, I wouldn't say that it's not that the words don't align very well, but sometimes what we see is that the embeddings themselves for certain specific words are not discriminative enough. And we have seen some, for instance, if you translate the sentence, "This apple costs 50 cents", you translate it to Portuguese and the translation needs, I'm not gonna say it in Portuguese, but pretend that I'm speaking Portuguese, the perfect translation would also be 50 cents. But for some reason, the M T might have hallucinated and say that it's 500 cents. So it's basically changing the price of an apple, and this is a critical error in much scenarios.

• 15:49Speaker 00

  But if you look at the embedding space of the 500 or the embedding of 50, it's gonna be very similar. And it's gonna be very hard for the neural network that is trying to differentiate these two things. It's gonna be a very hard task, because there is not enough signal. You also see the same thing with some named entities. And currently, there has been some work, some progress in trying to look at the quality estimation and metrics and try to figure out why they are not working for this kind of very specific phenomena. Actually, yesterday, we had a lot of presentations

• 16:30Speaker 00

  about challenge sets that tried to test metrics for this specific phenomena. So in W M T, we have several competitions, several what we call shared tasks. And inside the metrics shared task, where people are trying to compete to create better metrics, there was also a shared task that we call the challenge set subtask, where people submit examples that are challenging for metrics. And then the participants from the metrics task have to score those examples. And then we get the scores back to the developers of the challenges for them to analyze. And a lot of people looked into this and tried to make some suggestions for future work in how to improve metrics for this.

• 17:15Speaker 00

  So if you guys are interested in this, I take a look at the findings from the metrics task, because they are interesting findings and pointers for future work in this area.

• 17:27Speaker 01

  One of the problems of this model-based MT evaluation approaches is that, you know, first, they are based on the data that the pre-trained models were trained on. So there's everything there. There is bias, and there is the limited amount, or it can be a lot of data as well, but all the idiosyncrasies of that data are encoded in the pre-trained models. Then, when you fine-tune this for the specific tasks that they need to work on, namely quality estimation and MT evaluation, they also are limited in data in the sense that we have orders of magnitude less label data for this fine-tuning process. So this can have its biases, and it can have also, like, taking the example

• 18:11Speaker 01

  of Apple. For some reason, you've never seen Apple, the company, but you saw only for the fruit. So you start to, every time you see Apple, you translate that to the fruit, you know. You actually say that it's, if the model translates that to the fruit, the evaluation thing is going to say, "Yeah, it's fine." Because in the evaluation data that you used to train the model, you never saw, for some reason, the brand. So, and this is related to the named entity problem that Hikard was saying. So, I think one step, we are giving the first step as a community to understand that now. And, you know, really poke it and see, okay, there is a hole here. And now the next step is how to, you know,

• 18:53Speaker 01

  alleviate that problem. I don't think it's possible to alleviate completely solve, but we are for sure, we'll try to alleviate this for these models now. There is a, and there are a lot of complaints also, people, not complaints, but, you know, even us, when we are using different models, not only ours, we see that these models fall short sometimes. And this can be very bad in a commercial setting, or even in sensitive scenarios in which if you get two cents and the model that translated this to, I don't know, two million, that's not very nice, right? You might have some legal implications with that. So, yeah. I don't know, are there other, other open problems? I think, for me, one big problem is...

• 19:36Speaker 01

  Is that and this is also a trend that we see in the metrics and in the quality estimation task. Is that bigger models have better predictive power. So. People usually what they are doing is just throw more GPUs at it and and just train a bigger model. And this seems to be giving improvements as well. But the problem is that not every practitioner can actually use these models. Once they are trained, because they take, they need bigger and bigger GPUs, which are costlier, even at inference time. So we actually had a paper in EMT, the European Association for Machine Translation Conference. That was actually making comments smaller and it's like a diminutive in the name of the paper.

• 20:21Speaker 01

  The name of the model is Cometinho, which is a diminutive of comet, like Portuguese, very Portuguese way to say it. And it was also a first step towards that. But I think there is a lot to be done for all the other models and also for Comet.

• 20:35Speaker 00

  Yeah, definitely. I think Cometinho was just a first step into that direction. We. There is a lot of things that can be improved in distillation of these models, even the evaluation models like we did for Cometinho. And not just for evaluation, we have been focusing this podcast a little bit on evaluation. But on machine translation, you have the same problem. On machine translation, bigger models have been achieving impressive. Machine translation quality, but it's very hard for everyone to develop those models. And it's even harder for for people to deploy those models. We face this at Enbebo. We develop our own machine translation systems and we have seen this trend.

• 21:20Speaker 00

  We get improvements if we keep scaling our MTs, but then we have difficulties serving those MTs. And it's also, we know that not every company has the capacity to build such big models like big tech companies develop. So, yeah, it's not just in the evaluation side, but also in the machine translation side. It is like something that people should look forward to. It's without losing performance, how to make these things smaller and easier to deploy.

• 21:52Daniel Whitenack?

  Yeah, would you say on the, so on the model side specifically, like Jose, you mentioned sort of models getting bigger and bigger. Some people might have seen like nice gif fees about like an encoder decoder and one language coming in and one language coming out and transformer models. But what are some things others are exploring, maybe yourselves, that like are either different approaches, or you mentioned distillation and all these other things to make models smaller, but are there different architectures or techniques being explored? I think I saw one of your papers, something about like KNN, MT or something. I don't know if you can speak to that, but.

• 22:30Speaker 01

  Yeah, we just, at this moment, there is a poster on the usage of KNN MT for the chat shared task. So this is something called, I think this is broadly called dynamic adaptation. And one approach to that is doing KNN MT that rather than actually fully fine tuning one base model, like one of these large protein models, you actually just do some data retrieval approach in which you combine the contents of a data store that has relevant data for the use case that you're trying to serve with machine translation. And then at decoding time, when you are assembling the translation, using the translation probabilities

• 23:10Speaker 01

  of the model, you interpolate these probabilities with the probabilities of words or expressions contained in the, in the data store. So this way you avoid having to fully fine tune a model for each use case that you have. And this is something that we started to research and approach at a bubble. But I just must say that this doesn't solve the problem of the base model being big. You just avoid fine tuning it completely. So there's still the problem of, okay, how do I shrink or compress this model so that I can reliably and cheaply explore it for translation? And this is, like you said, distillation, quantization and other compressing techniques.

• 23:55Speaker 00

  Just to complement what Jose was saying about the key and nearest neighbor approach. Another very big advantage of this is that it's very easy to combine with translation memories, which we know that they are widely used in a translation industry. And this is a seamless way to basically take the MT and make the MT work with those translation memories. Because you can build this data store that will help the model to translate the content accordingly. So just to add that also, which I believe that it's very important for the localization industry in general.

• 24:32Daniel Whitenack?

  Great, yeah. Well, we've talked a lot about challenges, I guess, which is fun to talk about at a research conference for sure. What are some things just like generally about like the machine translation industry, or UnBabel, or other things that you make both of you sort of excited and, you know, optimistic about the future? What are some of those things that excite you? It doesn't have to be an MT, or, you know, things you've seen at this conference, or things that you're following that give you some encouragement and excitement about the future of the space we're working?

• 25:04Speaker 00

  Actually, I'm very passionate about evaluation in general. I think that shows up in my work, because I mostly work on evaluation. I've been getting very excited with the progress that we have been doing in evaluation. I think, like, we have started a project on this that we... It's to combine these systems, these quality estimation systems, with the machine translation itself. So that is something that we... We started working on this, but I believe that you can work on this for the next few years, and there is a lot of things that we can improve there. Yeah, that gets me really excited. I think it's a direction that it's going to be really nice.

• 25:45Speaker 01

  Yeah, this is the Quality Aware Decoding Project. That is basically what I just mentioned about, what we have been talking about of having these quality predictions about the hypothesis translations. The idea behind this project that Ricardo is talking about is, what if we bring the quality estimation or comment already to inside the MT process? And then we can make the machine translation aware or more aware about its quality, having a signal from a different model. So this is what this project is about. So we have a paper at NACO this year describing that. So yeah, this is pretty exciting. And I think in terms of more broad challenges,

• 26:28Speaker 01

  what I find interesting is that I don't believe that translation is solved. I think a few years ago, some people claimed that there was human parity between MT systems or some MT models and humans and translators. But then what it turned out that the actual translators that were used were not really professional translators. Like, I know English, right? But I'm not a native speaker and I cannot translate everything. So I'm not a subject matter expert on different topics. So I cannot actually, if you give me some chemistry content to translate into English from Portuguese, I cannot do it, right? So I think what's exciting is to see that the technology is allowing us to translate better and better.

• 27:13Speaker 01

  Maybe compared to me as a non-native speaker when I'm translating some content. But still, there are a lot of challenges to actually translate very well, very specific content that is, you know, requires very specific terminology and very specific way of actually building the sentences. And what is much better is actually the fluency that these machine translation models are giving nowadays. But what remains is still a challenge is that sometimes the translations, they look very good, but they are not on point. So they are not adequate. They are talking about something slightly different or completely different. So I think this is exciting. I mean, not everything is solved,

• 27:58Speaker 01

  but at the same time is encouraging, is encouraging in this sense.

• 28:01Daniel Whitenack?

  Yeah, great. Well, as we close out here, where can people find out more about Unbabel and specifically maybe some of this research that's going on? And also, you mentioned beforehand that Unbabel was possibly hiring as well. Where could people find out about that?

• 28:17Speaker 01

  Right, so we have our website, like unbabel.com, and we have our Twitter handle, like @Unbabble. You can follow our news from there. We have just put up a research blog in which we are going to be writing about our research. This is going to be possibly in the links in your info box. I don't know.

• 28:37Daniel Whitenack?

  Yeah, we'll put it in the show notes for sure.

• 28:39Speaker 01

  And yeah, we are also hiring soon. Like, we are starting to accept applications for the next year for research scientists in different levels and different geographies. So, so Unbabel, we didn't talk about it, but it was born in Portugal, in Lisbon. But now we have offices all around the world. We have offices in the West Coast, in the US, in the East Coast, London, you know, and some other places in Europe. And we are going to post this also to give an email for contact for people who are interested in other research that we're doing and other works. We don't, we have open positions not only for research scientists, but also for engineers and other positions that are not technical, so.

• 29:22Daniel Whitenack?

  Well, yeah, thank you, Jose. Thank you, Ricardo. Really appreciate you taking time. I know there's a lot of good posters around to see and all that. So thanks for taking time. Thanks, Daniel.

• 29:31Speaker 01

  Thank you.

• 29:40Speaker 02

  All right, that is our show for this week. If you dig it, don't forget to subscribe. Head to practicalai.fm for all the ways. And if Practical AI has benefited your life, pay it forward by sharing the show with a friend or a colleague. Word of mouth is the number one way people find shows like ours. Thanks again to Fastly for fronting our static assets, to fly.io for backing our dynamic requests, to Breakmaster Cylinder for the beats, and to you for listening. We appreciate you. That's all for now. We'll talk to you again on the next one.