Forrest Davis

2025

1. Discourse Sensitivity in Attraction Effects: The Interplay Between Language Model Size and Training Data

   Sanghee Kim, and Forrest Davis

   In Society for Computation in Lingusitics, 2025

   Abs Bib HTML PDF Code

   While work on the linguistic ability of language models (LMs) is driven by a variety of aims, one dominant motivation is using LMs to determine what linguistic knowledge can be learned from unstructured text. The current work aims to evaluate LMs on discourse sensitivity–the capability to distinguish between content that is more relevant and important to the discourse and that which is less so. We ground our evaluation of LMs by leveraging an existing psycholinguistics study on the number agreement attraction effect, one of the well-studied measures of human language comprehension. Based on human empirical findings on the modulation of the attraction effect by discourse, we establish three tests that LMs should pass if they demonstrate discourse sensitivity. A total of 25 models were evaluated that vary in (i) model size (small or large) and (ii) training type (dialogue-based, plain, and instruction-based). The models showed systematicity in discourse sensitivity, though in ways dissimilar to humans, either by over-relying on structural cues or overusing discourse cues. Notably, models that patterned most similarly to human performance were predominantly smaller and those trained on dialogue-targeted data. We discuss the implications of these findings and insights into human language processing.

   @inproceedings{kim-davis-2025-discourse-sensitivity,
     title = {{Discourse Sensitivity in Attraction Effects: The Interplay Between Language Model Size and Training Data}},
     author = {Kim, Sanghee and Davis, Forrest},
     booktitle = {Society for Computation in Lingusitics},
     year = {2025},
     doi = {https://doi.org/10.7275/scil.3156},
     volume = {8},
     number = {1},
     pages = {14},
   }

2023

1. Can Language Models Be Tricked by Language Illusions? Easier with Syntax, Harder with Semantics

   Yuhan Zhang, Edward Gibson, and Forrest Davis

   In Proceedings of the 27th Conference on Computational Natural Language Learning (CoNLL), Dec 2023

   Abs Bib HTML PDF Code

   Language models (LMs) have been argued to overlap substantially with human beings in grammaticality judgment tasks. But when humans systematically make errors in language processing, should we expect LMs to behave like cognitive models of language and mimic human behavior? We answer this question by investigating LMs’ more subtle judgments associated with “language illusions” – sentences that are vague in meaning, implausible, or ungrammatical but receive unexpectedly high acceptability judgments by humans. We looked at three illusions: the comparative illusion (e.g. “More people have been to Russia than I have”), the depth-charge illusion (e.g. “No head injury is too trivial to be ignored”), and the negative polarity item (NPI) illusion (e.g. “The hunter who no villager believed to be trustworthy will ever shoot a bear”). We found that probabilities represented by LMs were more likely to align with human judgments of being “tricked” by the NPI illusion which examines a structural dependency, compared to the comparative and the depth-charge illusions which require sophisticated semantic understanding. No single LM or metric yielded results that are entirely consistent with human behavior. Ultimately, we show that LMs are limited both in their construal as cognitive models of human language processing and in their capacity to recognize nuanced but critical information in complicated language materials.

   @inproceedings{zhang-etal-2023-language,
     title = {Can Language Models Be Tricked by Language Illusions? Easier with Syntax, Harder with Semantics},
     author = {Zhang, Yuhan and Gibson, Edward and Davis, Forrest},
     editor = {Jiang, Jing and Reitter, David and Deng, Shumin},
     booktitle = {Proceedings of the 27th Conference on Computational Natural Language Learning (CoNLL)},
     month = dec,
     year = {2023},
     address = {Singapore},
     publisher = {Association for Computational Linguistics},
     url = {https://aclanthology.org/2023.conll-1.1},
     doi = {10.18653/v1/2023.conll-1.1},
     pages = {1--14},
   }

2022

1. Incremental Processing of Principle B: Mismatches Between Neural Models and Humans

   Forrest Davis

   In Proceedings of the 26th Conference on Computational Natural Language Learning (CoNLL), Dec 2022

   Abs Bib HTML PDF Code

   Despite neural language models qualitatively capturing many human linguistic behaviors, recent work has demonstrated that they underestimate the true processing costs of ungrammatical structures. We extend these more fine-grained comparisons between humans and models by investigating the interaction between Principle B and coreference processing. While humans use Principle B to block certain structural positions from affecting their incremental processing, we find that GPT-based language models are influenced by ungrammatical positions. We conclude by relating the mismatch between neural models and humans to properties of training data and suggest that certain aspects of human processing behavior do not directly follow from linguistic data.

   @inproceedings{davis-2022-incremental,
     title = {Incremental Processing of Principle {B}: Mismatches Between Neural Models and Humans},
     author = {Davis, Forrest},
     booktitle = {Proceedings of the 26th Conference on Computational Natural Language Learning (CoNLL)},
     month = dec,
     year = {2022},
     address = {Abu Dhabi, United Arab Emirates (Hybrid)},
     publisher = {Association for Computational Linguistics},
     url = {https://aclanthology.org/2022.conll-1.11},
     doi = {10.18653/v1/2022.conll-1.11},
     pages = {144--156},
   }

2020

1. Recurrent Neural Network Language Models Always Learn English-Like Relative Clause Attachment

   Forrest Davis, and Marten van Schijndel

   In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Jul 2020

   Abs Bib HTML PDF Code

   A standard approach to evaluating language models analyzes how models assign probabilities to valid versus invalid syntactic constructions (i.e. is a grammatical sentence more probable than an ungrammatical sentence). Our work uses ambiguous relative clause attachment to extend such evaluations to cases of multiple simultaneous valid interpretations, where stark grammaticality differences are absent. We compare model performance in English and Spanish to show that non-linguistic biases in RNN LMs advantageously overlap with syntactic structure in English but not Spanish. Thus, English models may appear to acquire human-like syntactic preferences, while models trained on Spanish fail to acquire comparable human-like preferences. We conclude by relating these results to broader concerns about the relationship between comprehension (i.e. typical language model use cases) and production (which generates the training data for language models), suggesting that necessary linguistic biases are not present in the training signal at all.

   @inproceedings{davis-van-schijndel-2020-recurrent,
     title = {Recurrent Neural Network Language Models Always Learn {E}nglish-Like Relative Clause Attachment},
     author = {Davis, Forrest and {van Schijndel}, Marten},
     editor = {Jurafsky, Dan and Chai, Joyce and Schluter, Natalie and Tetreault, Joel},
     booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
     month = jul,
     year = {2020},
     address = {Online},
     publisher = {Association for Computational Linguistics},
     url = {https://aclanthology.org/2020.acl-main.179},
     doi = {10.18653/v1/2020.acl-main.179},
     pages = {1979--1990},
   }