Coordinate structure constraint in the linguistic competence of large language models
DOI:
https://doi.org/10.21638/spbu09.2024.309Abstract
A syntactic island is a construction extraction from which leads to ungrammaticality. Island constraints are generally demonstrated through the impossibility of the A′-movement, e. g. wh-movement. Considering extraction from an island as ungrammatical is common to all native speakers. In terms of natural language understanding and generation, the competence of large language models (LLM) is almost indistinguishable from the human one. However, the difference between the grammatical constraints of the native speakers and LLM are still studied insufficiently. If the LLM grammar is set up similar to the human one, they will demonstrate high sensitivity to island constraints. The current study aims to compare the language competence of the native speakers and LLM based on the coordinate structure islands. The three dialogue systems — ChatGPT, YandexGPT and GigaChat — were examined via two tests. The first one investigates whether the model is able to give a semantically correct answer to the question with violation of island constraints. The second test directly accesses the grammaticality judgements. The results clearly show that the LLM language competence differs from the human one. The observed models regularly answer the questions violating island constraints correctly and consider them grammatical. YandexGPT turns out to be more consistent, while ChatGPT and GigaChat frequently give incorrect answers to the questions Яwhich they judge acceptable. The influence of the stimuli’s grammatical features depends on the model: the island sensitivity of ChatGPT and GigaChat is determined by the same features in contrast to YandexGPT. Thus, the results call into question the fact that LLM language competence is close to the human one.
Keywords:
large language models, natural language processing, Russian, syntactic island, syntax
Downloads
References
Литература
Герасимова и др. 2024 — Герасимова А. А., Лютикова Е. А., Паско Л. И. Языковая компетенция сквозь призму грамматической вариативности. Часть 1. Теоретические и методологические соображения. Вестник Московского университета. Серия 9. Филология. 2024, (4): 9–22.
Гращенков 2024 — Гращенков П. В. RuConst: Синтаксический корпус русского языка с разметкой по непосредственным составляющим. Вестник Московского университета. Серия 9. Филология. 2024, (3): 94–112.
Зализняк, Падучева 1979 — Зализняк А. А., Падучева Е. В. Синтаксические свойства местоимения который. В кн.: Категория определенности-неопределенности в славянских и балканских языках: сб. ст. Николаева Т. М. (отв. ред.). М.: Наука, 1979. С. 289–329.
Лютикова, Герасимова 2021 — Лютикова Е. А., Герасимова А. А. (ред.). Русские острова в свете экспериментальных данных. М.: Буки Веди, 2021.
Моргунова 2021 — Моргунова Е. В. Островные конструкции в русском языке. В кн.: Русские острова в свете экспериментальных данных. Лютикова Е. А., Герасимова А. А. (ред.). М.: Буки Веди, 2021. С. 35–55.
Baldwin 1896 — Baldwin M. J. A New Factor in Evolution. The American Naturalist. 1896, 30 (354): 441–451.
Boeckx 2012 — Boeckx C. Syntactic islands. Cambridge: Cambridge University Press, 2012.
Chomsky 2004 — Chomsky N. Beyond explanatory adequacy. In: Structures and Beyond: The Cartography of Syntactic Structures. Belletti A. (ed.). Oxford: Oxford University Press, 2004. P. 104–131.
Chomsky 2013 — Chomsky N. Problems of Projection. Lingua. 2013, (130): 33–49.
Cinque 1990 — Cinque G. Types of A’ Dependencies. Cambridge: MIT Press, 1990.
Evanson et. al. 2023 — Evanson L., Lakretz Y., King J.-R. Language acquisition: do children and language models follow similar learning stages? Findings of the Association for Computational Linguistics: ACL 2023. 2023: 12205–12218.
Fenogenova et al. 2023 — Fenogenova A., Shavrina T., Kukushkin A., Tikhonova M., Emelyanov A., Malykh V., Mikhailov V., Shevelev D., Artemova E. Russian SuperGLUE 1.1: Revising the Lessons not Learned by Russian NLP-models. In: Computational Linguistics and Intellectual Technologies: Proceedings of the International Conference “Dialogue 2021”. 2021. P. 267–277.
Grosu 1973 — Grosu A. On the nonunitary nature of the Coordiate Structure Constraint. Linguistic Inquiry. 1973, 4 (1): 88–92.
Krejci 2020 — Krejci B. Syntactic and semantic perspectives on first conjunct agreement in Russian. PhD thesis. Stanford: Stanford University, 2020.
Lake, Baroni 2023 — Lake B. M., Baroni M. Human-like systematic generalization through a meta-learning neural network. Nature. 2023, (623): 115–121.
Lakoff 1986 — Lakoff G. Frame semantic control of the Coordinate Structure Constraint. Proceedings of the Chicago Linguistic Society. 1986, (22): 152–167.
Leivada, Westergaard 2020 — Leivada E., Westergaard M. Acceptable ungrammatical sentences, unacceptable grammatical sentences, and the role of the cognitive parser. Frontiers in Psychology. 2020, (11): 364.
Ott 2014 — Ott D. Syntactic islands by Cedric Boeckx (review). Language. 2014, (90): 287–291. Pearl, Sprouse 2013 — Pearl L., Sprouse J. Syntactic islands and learning biases: Combining experimental syntax and computational modeling to investigate the language acquisition problem. Language Acquisition. 2013, 20 (1): 23–68.
Phillips 2013a — Phillips C. On the nature of island constraints. I: Language processing and reductionist accounts. In: Experimental syntax and island effects. Sprouse J., Hornstein N. (eds). Cambridge: Cambridge University Press, 2013. P. 64–108.
Phillips 2013b — Phillips C. On the nature of island constraints. II: Language processing and reductionist accounts. In: Experimental syntax and island effects. Sprouse J., Hornstein N. (eds). Cambridge: Cambridge University Press, 2013. P. 132–157.
Rankin et al. 2015 — Rankin T., Grosso S., Reiterer S. Effects of L1 co-activation on the processing of L2 morpho-syntax in German-speaking learners of English. In: Proceedings of the 13th Generative Approaches to Second Language Acquisition Conference (GASLA 2015). Stringer D. et al. (eds). 2015. P. 196–207.
Ross 1967 — Ross J. R. Constraints on variables in syntax. PhD thesis. Cambridge, Massachusetts: Massachusetts Institute of Technology, 1967.
Tomida, Utsumi 2013 — Tomida Y., Utsumi A. A connectionist model for acquisition of syntactic islands.
Procedia — Social and Behavioral Sciences. 2013, (97): 90–97.
Wang et al. 2019 — Wang A., Pruksachatkun Y., Nangia N., Singh A., Michael J., Hill F., Levy O., Bowman S. R. Superglue: A stickier benchmark for general-purpose language understanding systems. Advances in Neural Information Processing Systems. 2019. Р. 3261–3275.
Wilcox et al. 2018 — Wilcox E. G., Levy R., Takashi M., Futrell R. What do RNN language models learn about filler-gap dependencies? In: Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP. Brussels, 2018. P. 211–221.
Wilcox et al. 2022 — Wilcox E. G., Futrell R., Levy R. Using computational models to test syntactic learnability. Linguistic Inquiry. 2022, Special Collection: CogNet: 1–44.
Williams 1978 — Williams E. Across-the-board rule application. Linguistic Inquiry. 1978, 9 (1): 31–43.
References
Герасимова и др. 2024 — Gerasimova A. A., Lyutikova E. A., Pasko L. I. Linguistic Competence Through the Lens of Grammatical Variation. Part 1. Conceptual and Methodological Considerations. Vestnik Moskovskogo universiteta. Ser. 9. Filologiia. 2024, (4): 9–22. (In Russian)
Гращенков 2024 — Grashchenkov P. V. RuConst: A Treebank for Russian. Vestnik Moskovskogo universiteta. Ser. 9. Filologiia. 2024, (3): 94–112. (In Russian)
Зализняк, Падучева 1979 — Zalizniak A. A., Paducheva E. V. Syntactic properties of the pronoun kotoryj. In: Kategoriya opredelennosti-neopredelennosti v slavyanskih i balkanskih yazykah: sbornik statei. Nikolaeva T. M. (ed.). Мoscow: Nauka Publ., 1979. P. 289–329. (In Russian)
Лютикова, Герасимова 2021 — Russian islands in the light of experimental data. Liutikova E. A., Gerasimova A. A. (eds). Moscow: Buki Vedi Publ., 2021. (In Russian)
Моргунова 2021 — Morgunova E. V. Island constraints in Russian. In: Russian islands in the light of experimental data. Liutikova E. A., Gerasimova A. A. (eds). Moscow: Buki Vedi Publ., 2021. P. 35–55. (In Russian)
Baldwin 1896 — Baldwin M. J. A New Factor in Evolution. The American Naturalist. 1896, 30 (354): 441–451.
Boeckx 2012 — Boeckx C. Syntactic islands. Cambridge: Cambridge University Press, 2012.
Chomsky 2004 — Chomsky N. Beyond explanatory adequacy. In: Structures and Beyond: The Cartography of Syntactic Structures. Belletti A. (ed.). Oxford: Oxford University Press, 2004. P. 104–131.
Chomsky 2013 — Chomsky N. Problems of Projection. Lingua. 2013, (130): 33–49.
Cinque 1990 — Cinque G. Types of A’ Dependencies. Cambridge: MIT Press, 1990.
Evanson et. al. 2023 — Evanson L., Lakretz Y., King J.-R. Language acquisition: do children and language models follow similar learning stages? Findings of the Association for Computational Linguistics: ACL 2023. 2023: 12205–12218.
Fenogenova et al. 2023 — Fenogenova A., Shavrina T., Kukushkin A., Tikhonova M., Emelyanov A., Malykh V., Mikhailov V., Shevelev D., Artemova E. Russian SuperGLUE 1.1: Revising the Lessons not Learned by Russian NLP-models. In: Computational Linguistics and Intellectual Technologies: Proceedings of the International Conference “Dialogue 2021”. 2021. P. 267–277.
Grosu 1973 — Grosu A. On the nonunitary nature of the Coordiate Structure Constraint. Linguistic Inquiry. 1973, 4 (1): 88–92.
Krejci 2020 — Krejci B. Syntactic and semantic perspectives on first conjunct agreement in Russian. PhD thesis. Stanford: Stanford University, 2020.
Lake, Baroni 2023 — Lake B. M., Baroni M. Human-like systematic generalization through a meta-learning neural network. Nature. 2023, (623): 115–121.
Lakoff 1986 — Lakoff G. Frame semantic control of the Coordinate Structure Constraint. Proceedings of the Chicago Linguistic Society. 1986, (22): 152–167.
Leivada, Westergaard 2020 — Leivada E., Westergaard M. Acceptable ungrammatical sentences, unacceptable grammatical sentences, and the role of the cognitive parser. Frontiers in Psychology. 2020, (11): 364.
Ott 2014 — Ott D. Syntactic islands by Cedric Boeckx (review). Language. 2014, (90): 287–291. Pearl, Sprouse 2013 — Pearl L., Sprouse J. Syntactic islands and learning biases: Combining experimental syntax and computational modeling to investigate the language acquisition problem. Language Acquisition. 2013, 20 (1): 23–68.
Phillips 2013a — Phillips C. On the nature of island constraints. I: Language processing and reductionist accounts. In: Experimental syntax and island effects. Sprouse J., Hornstein N. (eds). Cambridge: Cambridge University Press, 2013. P. 64–108.
Phillips 2013b — Phillips C. On the nature of island constraints. II: Language processing and reductionist accounts. In: Experimental syntax and island effects. Sprouse J., Hornstein N. (eds). Cambridge: Cambridge University Press, 2013. P. 132–157.
Rankin et al. 2015 — Rankin T., Grosso S., Reiterer S. Effects of L1 co-activation on the processing of L2 morpho-syntax in German-speaking learners of English. In: Proceedings of the 13th Generative Approaches to Second Language Acquisition Conference (GASLA 2015). Stringer D. et al. (eds). 2015. P. 196–207.
Ross 1967 — Ross J. R. Constraints on variables in syntax. PhD thesis. Cambridge, Massachusetts: Massachusetts Institute of Technology, 1967.
Tomida, Utsumi 2013 — Tomida Y., Utsumi A. A connectionist model for acquisition of syntactic islands.
Procedia — Social and Behavioral Sciences. 2013, (97): 90–97.
Wang et al. 2019 — Wang A., Pruksachatkun Y., Nangia N., Singh A., Michael J., Hill F., Levy O., Bowman S. R. Superglue: A stickier benchmark for general-purpose language understanding systems. Advances in Neural Information Processing Systems. 2019. Р. 3261–3275.
Wilcox et al. 2018 — Wilcox E. G., Levy R., Takashi M., Futrell R. What do RNN language models learn about filler-gap dependencies? In: Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP. Brussels, 2018. P. 211–221.
Wilcox et al. 2022 — Wilcox E. G., Futrell R., Levy R. Using computational models to test syntactic learnability. Linguistic Inquiry. 2022, Special Collection: CogNet: 1–44.
Williams 1978 — Williams E. Across-the-board rule application. Linguistic Inquiry. 1978, 9 (1): 31–43.
Downloads
Published
How to Cite
Issue
Section
License
Articles of "Vestnik of Saint Petersburg University. Language and Literature" are open access distributed under the terms of the License Agreement with Saint Petersburg State University, which permits to the authors unrestricted distribution and self-archiving free of charge.
