Computational Seminar 2021

Neural networks have enabled amazing progress in natural language processing over the past decade. However, their linguistic representations have been repeatedly shown to simply exploit shallow statistical heuristics rather than actually learning the underlying linguistic patterns. This has sparked an active debate in natural language processing as to whether or not these models will ever be able to learn linguistic meaning, especially when most language models are trained solely on text data. In this seminar we will read and discuss a number of papers on cognitive theories of meaning, computational models of meaning, and statistical learning to look outside natural language processing for solutions to this dilemma.

Syllabus

pdf

Schedule

Week 1: Poverty of the Stimulus

Syllabus
Bender and Koller (2020). Climbing towards NLU
Davis & van Schijndel (2020). Recurrent Neural Network Language Models Always Learn English-Like Relative Clause Attachment
Bisk et al. (2020). Experience grounds language

Week 2: Variability in Meaning

Pavlick and Kwiatkowski (2019). Inherent Disagreements in Human Textual Inferences

Week 3: Tracking Events

Altmann & Ekves (2020). Events as intersecting object histories

Week 3: Tracking Entities

Kunz and Hardmeier (2019). Entity Decisions in Neural Language Modelling: Approaches and Problems
Clark et al. (2018). Neural Text Generation in Stories Using Entity Representations as Context

Week 4: Informativity-Driven Saliency

Rohde et al. (2021). What's new? A comprehension bias in favor of informativity
Michael: Ribeiro et al. (2020). Beyond Accuracy: Behavioral Testing of NLP Models with CheckList

Week 5: Disentangling Syntax and Semantics

Stengel-Eskin et al. (2020). Universal Decompositional Semantic Parsing
White et al. (2020). The Universal Decompositional Semantics Dataset and Decomp Toolkit
Chih-Chan: Romanov et al. (2019). Adversarial Decomposition of Text Representation

Week 6: Symbolic vs Subsymbolic (Part 1)

Feldman (2012). Symbolic representation of probabilistic worlds

Week 7: Discourse Influences (Part 1)

Lee and Kaiser (2021). Does hitting the window break it?: Investigating effects of discourse-level and verb-level information in guiding object state representations
Forrest: Bresnan (2021). Formal grammar, usage probabilities, and English tensed auxiliary contraction

Week 8: Discourse Influences (Part 2)

Clifton and Frazier (2018). Context Effects in Discourse: The Question Under Discussion
Isa: Mueller et al. (2020). Cross-Linguistic Syntactic Evaluation of Word Prediction Models

Week 9: Symbolic vs Subsymbolic (Part 2a)

Piantadosi (2020). The Computational Origin of Representation (Sections 1-3)
Joseph: Hengchen et al. (2021). Challenges for Computational Lexical Semantic Change

Week 10: Symbolic vs Subsymbolic (Part 2b)

Piantadosi (2020). The Computational Origin of Representation (Sections 4-8)
Kaelyn: Events in psycholinguistics and in language models

Week 11: Wellness day

Week 12: Meaning from Form Revisited

Merrill et al. (unpublished). Provable Limitations of Acquiring Meaning from Ungrounded Form: What will Future Language Models Understand?
Will: Rodriguez and Merlo (2020). Word associations and the distance properties of context-aware word embeddings

Week 13

Project presentations