| Téléchargement | - Voir la version finale : Data sampling and (In)stability in machine translation evaluation (PDF, 294 Kio)
|
|---|
| DOI | Trouver le DOI : https://doi.org/10.18653/v1/2023.findings-acl.826 |
|---|
| Auteur | Rechercher : Lo, Chi-Kiu1; Rechercher : Knowles, Rebecca1 |
|---|
| Affiliation | - Conseil national de recherches du Canada. Technologies numériques
|
|---|
| Format | Texte, Article |
|---|
| Conférence | The 61st Annual Meeting of the Association for Computational Linguistics (ACL’23), July 9-14, 2023, Toronto, Ontario, Canada |
|---|
| Résumé | We analyze the different data sampling approaches used in selecting data for human evaluation and ranking of machine translation systems at the highly influential Conference on Machine Translation (WMT). By using automatic evaluation metrics, we are able to focus on the impact of the data sampling procedure as separate from questions about human annotator consistency. We provide evidence that the latest data sampling approach used at WMT skews the annotated data toward shorter documents, not necessarily representative of the full test set. Lastly, we examine a new data sampling method that uses the available labour budget to sample data in a more representative manner, with the goals of improving representation of various document lengths in the sample and producing more stable rankings of system translation quality. |
|---|
| Date de publication | 2023-07-09 |
|---|
| Maison d’édition | Association for Computational Linguistics |
|---|
| Licence | |
|---|
| Dans | |
|---|
| Langue | anglais |
|---|
| Publications évaluées par des pairs | Oui |
|---|
| Exporter la notice | Exporter en format RIS |
|---|
| Signaler une correction | Signaler une correction (s'ouvre dans un nouvel onglet) |
|---|
| Identificateur de l’enregistrement | bceb07fa-0260-423d-91fc-7e7af550dc8b |
|---|
| Enregistrement créé | 2023-07-17 |
|---|
| Enregistrement modifié | 2023-11-02 |
|---|
