Forrest Davis

2024

1. Training an NLP Scholar at a Small Liberal Arts College: A Backwards Designed Course Proposal

   Grusha Prasad, and Forrest Davis

   In Proceedings of the Sixth Workshop on Teaching NLP, Aug 2024

   Abs Bib HTML PDF Code

   The rapid growth in natural language processing (NLP) over the last couple yearshas generated student interest and excitement in learning more about the field. In this paper, we present two types of students that NLP courses might want to train. First, an “NLP engineer” who is able to flexibly design, build and apply new technologies in NLP for a wide range of tasks. Second, an “NLP scholar” who is able to pose, refine and answer questions in NLP and how it relates to the society, while also learning to effectively communicate these answers to a broader audience. While these two types of skills are not mutually exclusive — NLP engineers should be able to think critically, and NLP scholars should be able to build systems — we think that courses can differ in the balance of these skills. As educators at Small Liberal Arts Colleges, the strengths of our students and our institution favors an approach that is better suited to train NLP scholars. In this paper we articulate what kinds of skills an NLP scholar should have, and then adopt a backwards design to propose course components that can aid the acquisition of these skills.

   @inproceedings{prasad-davis-2024-training-nlp,
     title = {Training an {NLP} Scholar at a Small Liberal Arts College: A Backwards Designed Course Proposal},
     author = {Prasad, Grusha and Davis, Forrest},
     editor = {Al-azzawi, Sana and Biester, Laura and Kov{\'a}cs, Gy{\"o}rgy and Marasovi{\'c}, Ana and Mathur, Leena and Mieskes, Margot and Weissweiler, Leonie},
     booktitle = {Proceedings of the Sixth Workshop on Teaching NLP},
     month = aug,
     year = {2024},
     address = {Bangkok, Thailand},
     publisher = {Association for Computational Linguistics},
     url = {https://aclanthology.org/2024.teachingnlp-1.16},
     pages = {105--118},
   }

2023

1. Can Language Models Be Tricked by Language Illusions? Easier with Syntax, Harder with Semantics

   Yuhan Zhang, Edward Gibson, and Forrest Davis

   In Proceedings of the 27th Conference on Computational Natural Language Learning (CoNLL), Dec 2023

   Abs Bib HTML PDF Code

   Language models (LMs) have been argued to overlap substantially with human beings in grammaticality judgment tasks. But when humans systematically make errors in language processing, should we expect LMs to behave like cognitive models of language and mimic human behavior? We answer this question by investigating LMs’ more subtle judgments associated with “language illusions” – sentences that are vague in meaning, implausible, or ungrammatical but receive unexpectedly high acceptability judgments by humans. We looked at three illusions: the comparative illusion (e.g. “More people have been to Russia than I have”), the depth-charge illusion (e.g. “No head injury is too trivial to be ignored”), and the negative polarity item (NPI) illusion (e.g. “The hunter who no villager believed to be trustworthy will ever shoot a bear”). We found that probabilities represented by LMs were more likely to align with human judgments of being “tricked” by the NPI illusion which examines a structural dependency, compared to the comparative and the depth-charge illusions which require sophisticated semantic understanding. No single LM or metric yielded results that are entirely consistent with human behavior. Ultimately, we show that LMs are limited both in their construal as cognitive models of human language processing and in their capacity to recognize nuanced but critical information in complicated language materials.

   @inproceedings{zhang-etal-2023-language,
     title = {Can Language Models Be Tricked by Language Illusions? Easier with Syntax, Harder with Semantics},
     author = {Zhang, Yuhan and Gibson, Edward and Davis, Forrest},
     editor = {Jiang, Jing and Reitter, David and Deng, Shumin},
     booktitle = {Proceedings of the 27th Conference on Computational Natural Language Learning (CoNLL)},
     month = dec,
     year = {2023},
     address = {Singapore},
     publisher = {Association for Computational Linguistics},
     url = {https://aclanthology.org/2023.conll-1.1},
     doi = {10.18653/v1/2023.conll-1.1},
     pages = {1--14},
   }

2022

1. Incremental Processing of Principle B: Mismatches Between Neural Models and Humans

   Forrest Davis

   In Proceedings of the 26th Conference on Computational Natural Language Learning (CoNLL), Dec 2022

   Abs Bib HTML PDF Code

   Despite neural language models qualitatively capturing many human linguistic behaviors, recent work has demonstrated that they underestimate the true processing costs of ungrammatical structures. We extend these more fine-grained comparisons between humans and models by investigating the interaction between Principle B and coreference processing. While humans use Principle B to block certain structural positions from affecting their incremental processing, we find that GPT-based language models are influenced by ungrammatical positions. We conclude by relating the mismatch between neural models and humans to properties of training data and suggest that certain aspects of human processing behavior do not directly follow from linguistic data.

   @inproceedings{davis-2022-incremental,
     title = {Incremental Processing of Principle {B}: Mismatches Between Neural Models and Humans},
     author = {Davis, Forrest},
     booktitle = {Proceedings of the 26th Conference on Computational Natural Language Learning (CoNLL)},
     month = dec,
     year = {2022},
     address = {Abu Dhabi, United Arab Emirates (Hybrid)},
     publisher = {Association for Computational Linguistics},
     url = {https://aclanthology.org/2022.conll-1.11},
     doi = {10.18653/v1/2022.conll-1.11},
     pages = {144--156},
   }

2020

1. Recurrent Neural Network Language Models Always Learn English-Like Relative Clause Attachment

   Forrest Davis, and Marten van Schijndel

   In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Jul 2020

   Abs Bib HTML PDF Code

   A standard approach to evaluating language models analyzes how models assign probabilities to valid versus invalid syntactic constructions (i.e. is a grammatical sentence more probable than an ungrammatical sentence). Our work uses ambiguous relative clause attachment to extend such evaluations to cases of multiple simultaneous valid interpretations, where stark grammaticality differences are absent. We compare model performance in English and Spanish to show that non-linguistic biases in RNN LMs advantageously overlap with syntactic structure in English but not Spanish. Thus, English models may appear to acquire human-like syntactic preferences, while models trained on Spanish fail to acquire comparable human-like preferences. We conclude by relating these results to broader concerns about the relationship between comprehension (i.e. typical language model use cases) and production (which generates the training data for language models), suggesting that necessary linguistic biases are not present in the training signal at all.

   @inproceedings{davis-van-schijndel-2020-recurrent,
     title = {Recurrent Neural Network Language Models Always Learn {E}nglish-Like Relative Clause Attachment},
     author = {Davis, Forrest and {van Schijndel}, Marten},
     editor = {Jurafsky, Dan and Chai, Joyce and Schluter, Natalie and Tetreault, Joel},
     booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
     month = jul,
     year = {2020},
     address = {Online},
     publisher = {Association for Computational Linguistics},
     url = {https://aclanthology.org/2020.acl-main.179},
     doi = {10.18653/v1/2020.acl-main.179},
     pages = {1979--1990},
   }