Interdependence of expert categories and automated metrics applied to evaluate translation quality
DOI:
https://doi.org/10.21638/spbu09.2022.107Abstract
The article evaluates the quality of translation; we consider the applied and pragmatic aspects of such evaluation in the conditions of the current rapid increase in the number of texts to be translated. The article summarizes a plethora of assessment principles, each having its merits and drawbacks, and examines the correlation between the categories of adequacy and equivalence as the basic assessment parameters. We conclude that equivalence is oriented towards the result of translation, whereas adequacy indicates whether translation as process corresponds to the communicative situation given. Adequacy is thus viewed as a discursive feature. The article also looks into the possibility of applying these assessment criteria to automated evaluation of translation quality, and compares the specific computer metrics or, algorithms: Cosine Similarity, Word Error Rate, BLEU, ROUGE, NIST, METEOR. We analyze the potential for applying these metrics to man-made rather than machine translated. The n-gram metric for evaluation of translation with explicit ordering Meteor is the most efficient algorithm; however, it can only be applied advantageously to formal standardized texts. The article points out that literary translation can hardly be evaluated on the basis of n-gram correspondences due to it characteristic feature: ample use of non-equivalent units even at the sentence level. The purely mathematical procedures overviewed in the article cannot serve as linguistic meta-models of assessment unless supplemented with syntactic and semantic algorithms.
Keywords:
evaluation of translation quality, adequacy, equivalence, machine translation, automated metrics for evaluation of translation, word embedding
Downloads
References
Аносова 2011 — Аносова А. А. Критерии оценки качества перевода поликодового текста. Научно-технические ведомости СПбГПУ. Гуманитарные и общественные науки. 2011, (3): 46–49.
Гарбовский 2004 — Гарбовский Н. К. Теория перевода. М.: Изд-во Моск. гос. ун-та, 2004.
Григорьев, Мартыненко 2019 — Григорьев Ю. Д., Мартыненко Г. Я. Комбинаторные варианты рифм в сонетах Рильке: квантитативно-типологический подход. Вестник НГУ. Серия: Лингвистика и межкультурная коммуникация. 2019, (1): 5–20.
Евтеев 2017 — Евтеев С. В. Перевод: эквивалентно — насколько возможно, и адекватно — насколько нужно. Вестник Брянского государственного университета. 2017, (1): 262–267.
Илюшкина 2017 — Илюшкина М. Ю. Теория перевода: основные понятия и проблемы. М.: Флинта, 2017.
Комиссаров 1990 — Комиссаров В. Н. Теория перевода (лингвистические аспекты). М.: Высшая школа, 1990.
Комиссаров 1999 — Комиссаров В. Н. Общая теория перевода. Проблемы переводоведения в освещении зарубежных ученых. М.: ЧеРо; Юрайт, 1999.
Левенштейн 1965 — Левенштейн В. И. Двоичные коды с исправлением выпадений, вставок и замещений символов. Доклады АН СССР. 1965, 163 (4): 845–848.
Митренина 2016 — Митренина О. В. Машинный перевод. В кн.: Прикладная и компьютерная лингвистика. Николаев И. С., Митренина О. В., Ландо Т. М. (ред.). М.: URSS, 2016. C. 156–189.
Мошкович 2012 — Мошкович В. В.Взаимосвязь и взаимовлияние адекватности и эквивалентности. Вестник ЮУрГГПУ. 2012, (10): 270–279.
Найда 1978 — Найда Ю. К науке переводить. Вопросы теории перевода в зарубежной лингвистике. М.: Междунар. отношения, 1978. С. 114–137.
Портер 2004 — Портер Л. Г. Количественные критерии адекватности поэтического перевода. Мир перевода. 2004, 1 (11): 39–54.
Прунч 2015 — Прунч Э. Пути развития западного переводоведения. От языковой асимметрии к политической. М.: Р. Валент, 2015.
Рецкер 1974 — Рецкер Я. И. Перевод и переводческая практика. М.: Междунар. отношения, 1974.
Сдобников 2007 — Сдобников В. В. Методические основы подготовки переводчиков: нижегородский опыт. Н. Новгород: Изд-во Нижегород. гос. лингвистич. ун-та им. Н. А. Добролюбова, 2007.
Сдобников, Петрова 2007 — Сдобников В. В., Петрова О. В. Теория перевода. М.: АСТ: Восток-Запад, 2007.
Скребнев 1984 — Скребнев Ю. М. Основы стилистики английского языка. Киев: Вища школа, 1984.
Финкель 2001 — Финкель А. М. О точности стихотворного перевода. Московский лингвистический журнал. 2001, 5 (2): 127–186.
Швейцер 1988 — Швейцер А. Д. Теория перевода: Статус, проблемы, аспекты. М.: Наука, 1988.
Banerjee, Lavie 2005 — Banerjee S., Lavie A. METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments. In: Proceedings of Workshop on Intrinsic and Extrinsic Evaluation Measures for MT and/or Summarization at the 43rd Annual Meeting of the Association of Computational Linguistics (ACL 2005). Ann Arbor, 2005. P. 65–72.
Brownlee 2017 — Brownlee J. A Gentle Introduction to the Bag-of-Words Model. In: Deep Learning for Natural Language Processing. https://machinelearningmastery.com/gentle-introduction-bag-words- model/ (accessed: 03.02.2020).
Catford 1965 — Catford J. C. A Linguistic Theory of Translation: An Essay in Applied Linguistics. Oxford: Oxford University Press, 1965.
Chesterman, Wagner 2002 — Chesterman A., Wagner E. Can Theory Help Translators? A Dialog between the Ivory Tower and the Wordface. Manchester: St. Jerome Publ., 2002.
Goldberg et al. 2017 — Goldberg Y., Levy O., Søgaard A. A Strong Baseline for Learning Cross-Lingual Word Embeddings from Sentence Alignments. In: Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics. Vol. 1. 2017. P. 765–774.
Denkowski, Lavie 2014 — Denkowski M., Lavie A. Meteor Universal: Language Specific Translation Evaluation for Any Target Language. In: Proceedings of the Ninth Workshop on Statistical Machine Translation. Baltimore, 2014. P. 376–380.
Marzal, Vidal 1993 — Marzal A., Vidal E. Computation of Normalized Edit Distance and Applications. IEEE Trans. Pattern Analysis and Machine Intelligence. 1993, 15 (9): 926–932.
Mikolov et al. 2013 — Mikolov T., Chen K., Corrado G., Dean J. Efficient Estimation of Word Representations in Vector Space. In: Proceedings of the International Conference on Learning Representations. 2013. P. 1–12.
Singhal 2001 — Singhal A. Modern Information Retrieval: A Brief Overview. IEEE Data Eng. Bulletin. 2001, (24): 35–43.
Olson, Delen 2008 — Olson D. L., Delen D. Advanced Data Mining Techniques. Berlin; Heidelberg: Springer-Verlag Berlin Heidelberg, 2008.
Papineni et al. 2002 — Papineni K., Roukos S., Ward T., Zhu W.-J. BLEU: a Method for Automatic Evaluation of Machine translation. Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL). Philadelphia, 2002. P. 311–318.
Reiss, Vermeer 1984 — Reiss K., Vermeer H. J. Grundlegung einer allgemeinen Translationstheorie. Tübingen: De Gruyter, 1984.
Snover et al. 2009 — Snover M., Madnani N., Dorr B., Schwarz R. Fluency, Adequacy, or HTER? Exploring Different Human Judgments with a Tunable MT Metric. In: Proceedings of the Fourth Workshop on Statistical Machine Translation Athens, Greece, March 30–31, 2009. Athens, 2009. P. 259–268.
Vuorikoski 2016 — Vuorikoski A.-R. Importance of being accurate. Trudy Sankt-Peterburgskogo gosudarstvennogo universiteta kul'tury. Kul'tura i delovoi inostrannyi iazyk. T. 214: materialy III mezhdunar. nauch.-prakt. konf. 12–13 marta 2015 g., St Peterburg. St Peterburg, 2016. P. 101–109.
Аносова 2011 — Anosova A. A. Criteria of Polycode Text Translation Quality. Nauchno-tekhnicheskie vedomosti SPbGPU. Gumanitarnye i obshchestvennye nauki. 2011, (3): 46–49. (In Russian)
Гарбовский 2004 — Garbovskii N. K.Theory of Translation. Moscow: Izdatel’stvo Moskovskogo gosudarstvennogo universiteta Publ., 2004. (In Russian)
Григорьев, Мартыненко 2019 — Grigoriev Yu. D., Martynenko G. Ya. Combinatorics of Rhyming Variants in Sonnets by R. M. Rilke: A Quantitative and Typological Approach. Vestnik Novosibirskogo gosudarstvennogo universiteta. Seriia: Lingvistika i mezhkul'turnaia kommunikatsiia. 2019, (1): 5–20. (In Russian)
Евтеев 2017 — Evteev S. V. Translation: Equivalency as Possible, Adequacy as Needed. Vestnik Brianskogo gosudarstvennogo universiteta. 2017, (1): 262–267. (In Russian)
Илюшкина 2017 — Ilyushkina M. Yu. Theory of Translation: Basic Concepts and Issues. Moscow: Flinta Publ., 2017. (In Russian)
Комиссаров 1990 — Komissarov V. N. Theory of Translation (Linguistic Aspects). Moscow: Vyshaia Shkola Publ., 1990. (In Russian)
Комиссаров 1999 — Komissarov V. N. General Theory of Translation. Issues of Translation Studies as Viewed by Foreign Scholars. Moscow: CheRo Publ.; Iurait Publ., 1999. (In Russian)
Левенштейн 1965 — Levenshtein V. I. Binary Codes with the Correction of Symbol Omissions, Insertions, and Substitutions. Doklady Akademii nauk SSSR. 1965, 163 (4): 845–848. (In Russian)
Митренина 2016 — Мitrenina О. V.Мitrenina О.V. Machine Translation. In: Prikladnaia i komp’iuternaia lingvistika. Moscow: URSS Publ., 2016. P. 156–189. (In Russian)
Мошкович 2012 — Moshkovich V.V. Interconnection and Mutual Influence Between Adequacy and Equivalence. Vestnik Iuzhno-Ural’skogo gosudarstvennogo gumanitarno-pedagogicheskogo universiteta, 2012, (10): 270–279. (In Russian)
Найда 1978 — Naida E. Toward a Science of Translating. In: Issues of Translation Theory in Foreign Linguistics. Moscow: Mezhdunarodnye otnosheniia Publ., 1978. (In Russian)
Портер 2004 — Porter L.G. Quantitative Criteria of Translation Adequacy. The World of Translation. 2004, 1 (11): 39–54. (In Russian)
Прунч 2015 — Prunč E. Ways of development of Western translation studies. From linguistic asymmetry to political. Moscow: R. Valent Publ., 2015. (In Russian)
Рецкер 1974 — Retsker Ya. I. Translation and Practice of Translation. Moscow: Mezhdunarodnye otnosheniia Publ., 1974. (In Russian)
Сдобников 2007 — Sdobnikov V. V. Metodological Basis for Teaching Translation: Nizhny Novgorod Experience. Nizhnii Novgorod: Izdatel'stvo Nizhegorodskogo gosudarstvennogo lingvisticheskogo universiteta Publ., 2007. (In Russian)
Сдобников, Петрова 2007 — Sdobnikov V. V., Petrova О. V. Theory of Translation. Мoscow: АSТ Publ., 2007. (In Russian)
Скребнев 1984 – Skrebnev Yu. М. Fundamentals of the Stylistics of the English Language. Kiev: Visha shkola, 1984. (In Russian)
Финкель 2001 — Finkel’ A.M. Towards the Precision of Verse Translation. Moskovskii lingvisticheskii zhurnal. 2001, 5 (2): 127–186. (In Russian)
Швейцер 1988 — Shveitser A. D. Theory of Translation: Status, Issues, Aspects. Moscow: Nauka Publ., 1974. (In Russian)
Banerjee, Lavie 2005 — Banerjee S., Lavie A. METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments. In: Proceedings of Workshop on Intrinsic and Extrinsic Evaluation Measures for MT and/or Summarization at the 43rd Annual Meeting of the Association of Computational Linguistics (ACL 2005). Ann Arbor, 2005. P. 65–72.
Brownlee 2017 — Brownlee J. A Gentle Introduction to the Bag-of-Words Model. In: Deep Learning for Natural Language Processing. https://machinelearningmastery.com/gentle-introduction-bag-words-model/ (accessed: 03.02.2020).
Catford 1965 — Catford J. C. A Linguistic Theory of Translation: An Essay in Applied Linguistics. Oxford: Oxford University Press, 1965.
Chesterman, Wagner 2002 — Chesterman A., Wagner E. Can Theory Help Translators? A Dialog between the Ivory Tower and the Wordface. Manchester: St. Jerome Publ., 2002.
Goldberg et al. 2017 — Goldberg Y., Levy O., Søgaard A. A Strong Baseline for Learning Cross-Lingual Word Embeddings from Sentence Alignments. In: Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics. 2017, (1): P. 765–774..
Denkowski, Lavie 2014 — Denkowski M., Lavie A. Meteor Universal: Language Specific Translation Evaluation for Any Target Language. In: Proceedings of the Ninth Workshop on Statistical Machine Translation. Baltimore, 2014. P. 376–380.
Marzal, Vidal 1993 — Marzal A., Vidal E. Computation of Normalized Edit Distance and Applications. IEEE Trans. Pattern Analysis and Machine Intelligence. 1993, 15 (9): 926–932.
Mikolov et al. 2013 — Mikolov T., Chen K., Corrado G., Dean J. Efficient Estimation of Word Representations in Vector Space. Proceedings of the International Conference on Learning Representations. 2013. P. 1–12.
Singhal 2001 — Singhal A. Modern Information Retrieval: A Brief Overview. IEEE Data Eng. Bulletin. 2001, (24): 35–43.
Olson, Delen 2008 — Olson D. L., Delen D. Advanced Data Mining Techniques. Berlin; Heidelberg: Springer-Verlag Berlin Heidelberg, 2008.
Papineni et al. 2002 — Papineni K., Roukos S., Ward T., Zhu W.-J. BLEU: a Method for Automatic Evaluation of Machine translation. In: Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL). Philadelphia, 2002. P. 311–318.
Reiss, Vermeer 1984 — Reiss K., Vermeer H. J. Grundlegung einer allgemeinen Translationstheorie. Tübingen: De Gruyter, 1984. 245 S.
Snover et al. 2009 — Snover M., Madnani N., Dorr B., Schwarz R. Fluency, Adequacy, or HTER? Exploring Different Human Judgments with a Tunable MT Metric. In: Proceedings of the Fourth Workshop on Statistical Machine Translation Athens, Greece, March 30–31, 2009. Athens, 2009. P. 259–268.
Vuorikoski 2016 — Vuorikoski A-R. Importance of being accurate. Trudy Sankt-Peterburgskogo gosudarstvennogo instituta kul’tury. Kul’tura i delovoi inostrannyi iazyk. Vol. 214: Materialy III mezhdunarodnoi nauchno-prakticheskoi konferentsii, 12–13 marta 2015, Sankt-Peterburg. St Petersburg, 2016. P. 101–109.
Downloads
Published
How to Cite
Issue
Section
License
Articles of "Vestnik of Saint Petersburg University. Language and Literature" are open access distributed under the terms of the License Agreement with Saint Petersburg State University, which permits to the authors unrestricted distribution and self-archiving free of charge.
