The Emergence of a Grammatical Subsystem in Neural Language Models
Marco Baroni, Research Scientist at Facebook Artificial Intelligence (FAIR) Paris, and Catalan Institution for Research and Advanced Studies (ICREA) Barcelona)
Venue| Senior Common Room, Level 2 (2D17), Priory Road Complex
Date | **Wednesday 19 June 2019**
Time | 13:00
Sequence-processing neural networks, such as long-short-term-memory networks, have achieved astounding performance in many sophisticated natural language processing tasks, while being trained on the simple task of predicting the next word given the context in running text. We try to reach a better understanding of which linguistic knowledge the networks are effectively inducing, and how they are encoding it, by focusing on the task of long-distance number agreement (“the *girl* next to the buses *is*…”). We find that the networks developed a relatively sophisticated circuit of connected units that are sensitive to syntactic structure, and units that encode morphological features in a sparse manner. In my talk, I will summarize the evidence we were able to gather until now, as well as discussing ongoing work that tries to assess if predictions made by observing the characteristics of the emerging system are borne out with human subject. I will conclude by posing the yet-to-be-explored question of why such system should have emerged.
[Joint work with: Yair Lakretz, Dieuwke Hupkes, Theo Desbordes, Germán Kruszewski and Stan Dehaene]
All Welcome