BI 169 Andrea Martin: Neural Dynamics and Language

Andrea    00:00:03    I really am trying to explain the thing that I want to understand. Like that’s what I’m, even as the rise of large language models come, and I feel often that I’m, that I don’t, uh, always find, uh, a camp of people who, who see things the same way. And that can be good and sometimes challenging and can sometimes very sort of lonely <laugh>. Why can’t dynamical systems represent and compute there? It, it’s, we have take, we have, it’s a choice for us to believe that they cannot, it’s a choice for us to not to try to, you know, bend or break or reformulate how we talk about these systems.  

Paul    00:00:39    What is the dimensionality of, of language  

Speaker 0    00:00:42    <laugh>?  

Andrea    00:00:43    Oh, it depends on who you ask.  

Paul    00:00:44    Sorry. Isity one of those words for you. No, do we need  

Andrea    00:00:47    To, dimensionality is a, is a safe word. It’s a good word.  

Speaker 0    00:00:50    <laugh>.  

Paul    00:00:54    Hello, Intrepid mind explorers. This is brain inspired. I’m Paul. Today my guest is Andrea Martin, who is the research group leader in the Department of Language and Computation in neural systems at the Max Plunk Institute and the Donders Institute. Andrea is deeply interested in understanding how our biological brains process and represent language. To this end, she is developing a theoretical model of language, and the aim of the model is to account for the properties of language like its structure, its compositionally, um, its infinite express ability, uh, all while adhering to physiological data that, that we can measure from human brains. So her theoretical model of language, among other things, brings in the idea of low dimensional manifolds and neurodynamics along those manifolds. So we’ve discussed manifolds a lot on the podcast, but, uh, in short, there are kind of abstract structure in the space of possible neural population activities.  

Paul    00:01:56    The neural dynamics, uh, and that manifold structure defines the range of possible trajectories or pathways the neural dynamics can take over time. One of Andrea’s ideas, uh, is that manifolds might be a way for the brain to combine two properties of how we learn and use language, uh, and process language. One of those properties is the, um, statistical regularities found in language. So a given word for example, occurs more often near some words and less often near some other words. This statistical approach is the foundation of how large language models are trained. The other property of language, um, is the more formal structure of language, how it’s arranged and organized in such a way that gives it meaning to us. Um, perhaps these two properties, uh, of language can come together as a single trajectory, uh, along a neural manifold and our brain’s predictions about what’s coming next, uh, essentially guide and push neural activity along that manifold trajectory.  

Paul    00:02:57    So that’s a mouthful I know, but we hash it out more during the episode and she has a lot more ideas. And of course, we discuss many of them. And of course we discuss, uh, large language models more and how Andrea thinks of them with respect to biological, uh, cognition. We talk about modeling in general and what models do and don’t tell us, um, and much more. So as always, I hope that our discussion just wets your appetite to learn more about Andrea’s work. Uh, and you can find links in the show notes at brain inspired.co/podcast/ 1 6 9, where you can also learn how to support this podcast if your hearts are as generous as your minds are intrepid. Okay. Thank you for listening. Here’s Andrea. Andrea, you’ve been, uh, studying language for a long time, and, uh, why, first of all, why didn’t you study vision instead of language? No, I’m just kidding. I’m kidding. No, I,  

Andrea    00:03:50    I actually have an answer to that. Okay. All right. I actually started working, um, this first, I went to college for one year at Columbia University. Uh, and then I, I dropped out, but I worked in a, uh, a p 53 oncogene lab. And I, you know, as like a intern, one of my tasks was to be involved in like sacrificing the animals. And at the time, uh, I was a vegan. I’m not anymore <laugh>. And I found it really rough and I was like, I’m really interested in neuroscience. What can I study where I won’t ever have to use animal models? Cause I don’t have the stomach for it, even though I think the data are important. And that’s how I ended up with language. I mean, there are many other steps in between. I ended up doing some other stuff and I ended up studying cognitive science at Hampshire College where I could really focus exclusively, uh, on psych linguistics and cognitive science.  

Paul    00:04:35    So it really was a matter of, um, so what’s, it’s not language per se, that you were interested in, and I’m sure that you’ve become very interested in it. <laugh>  

Andrea    00:04:44    It was avoidance. No, it was, i, it, it was helped me sort of like crystallize, what was it about actually the human mind that I was so interested in was actually something that I couldn’t study in an animal model.  

Paul    00:04:54    Hmm.  

Andrea    00:04:54    That’s a different way to put it.  

Paul    00:04:56    And this is an unfair question because I know that there are many aspects of language that you’re interested in, but is there a particular aspect of language that, uh, that, uh, excites you that you’re, that you’re really focused in on?  

Andrea    00:05:10    I think it’s, for me, I mean, I think, you know, foreshadow of it, it’s that the idea that the language represents sort of the, the boundary computation for the brain. So I can talk more about this later, but the fact that you need statistical and quote unquote algebraic or variable like information, um, to account for our language behavior and the languages of the world, I think really sort of presses us to say, okay, what’s sort of the limits on what we can express in a neural system? And language gives us a really great, uh, readout for that. And so, following that line, then, I guess the most sort of interesting property of language that gets to that is really the notion of composition. That you are putting things together, uh, that are more than the sum of their parts, but that are a function of them, uh, and the rules used to combine them. And that’s really, um, that just seems to, to be endlessly giving <laugh> important insights about the human mind and about the brain.  

Paul    00:06:02    Um, so you use the term boundary computation, I believe there. So, and, and then you also use the term limits. I mean, do you think that language represents a, a bound on our cognitive, um, capacities?  

Andrea    00:06:14    Um, I think more what I mean there is at at least, uh, for what we know about, uh, language now at least, just talking about the world’s languages and what linguistics tells us, we already have a really hard problem, um, that I think, uh, neuroscience and cognitive science are like, you know, running <laugh> their, their, their best race to try to explain already, but already I think there’s a lot of stuff that isn’t sufficient or that’s, that’s sort of falling short. Uh, and that’s what I mean there. So the, the other way to, to look at it more formally is to say, look, if you really wanna, uh, push a system to be both statistical and algebraic, that that’s going to be the boundary condition on the things that you can compute, uh, for, for the brain.  

Paul    00:06:51    What, what does it mean to say that language is algebraic  

Andrea    00:06:54    <laugh>? That’s, that’s a good question. So I thought about this a lot, um, since you brought it up. Um, and I think the way that I typically try to like get to the problem or define it, uh, for people is to say that, to start thinking about it in terms of the fact that human language is shaped by its statistics, but it’s not def defined by them, right? So, um, language can break free from the statistics of itself. So frequency or probability of a sound of a morphine of a word or a phrase. Um, and if we couldn’t do that, you know, notably our behavior would be really, really different, right? So it’s easy to think of extreme or funny examples. Like we could only say, uh, fixed responses or would constantly be being stuck, uh, in local or minimum or maxima about what we’re talking about or what we’re seeing.  

Andrea    00:07:36    Uh, and we don’t do that. And I mean, there are many, uh, classical and better examples, uh, from the cognitive revolution fact that we can say things that we haven’t heard before, that we don’t have, uh, lookup tables of utterances that we’re, that we’re bound to. Um, but I think we kind of actually overlook the simple fact all the time, and that not being driven by statistics, well then what is the thing can that can explain, uh, our behavior and the way the systematicity of language is how they work? Uh, and that seems to be minimally some kind of system that has variables at its heart. And it’s from that notion that you don’t, you’re not stuck to a, a fixed value that is, you know mm-hmm. A sound or a word that you heard in a particular context that you only return, that you somehow, uh, come upon a system of rules where you can have variables and functions that then can be adapted to, you know, the conditions at hand, what you need so you can express the thing, uh, that’s needed in that situation.  

Paul    00:08:32    Where, where do those rules come from, <laugh>  

Andrea    00:08:35    Are built. That is a, that is a deep and contentious question. <laugh>,  

Paul    00:08:40    Let, let me back up because I, what I wanted to ask, you know, just, uh, off of what you were talking about was, you know, what, what came first? Statistics or rules, right? Or variables? Uh, maybe I will just ask that. Yeah.  

Andrea    00:08:52    I think that’s a good place to start. So, I mean, that’s, that’s already in itself as a, has got the, the hard question in it. So, which came first in order to, you know, sort of a simple way to talk about statistics is to say there, it’s counting something, right? But in order to be able to count something, you have to know what that thing is to recognize it and say, okay, you know, obviously our, the way that we used statistics now in natural engine language processing and in cognitive science is much more sophisticated. It’s not just simply counting one occurrence, but still there, there has to be some in practice labeling that’s going on, uh, uh, in the engineering sense, or, uh, there has to be some thing built into the system that tells you, okay, this is the thing that’s relevant to count, to calculate a statistic over.  

Andrea    00:09:33    So in that sense, I think you have to at least have some kind of, uh, pattern recognition that allows you to extract some in variance, um, across all the different situations that we, uh, encounter Language. I mean, language learning and acquisition is not my specialty. It’s a really hard problem too, <laugh> Mm. Um, but I think that it’s clear that there’s a, the interaction between, um, uh, you know, an inductive bias that we have for hierarchical, uh, relationships and patterns that, for looking for more than sequences, for doing more than predicting the next word. And that that can be bootstrapped over the information that we get from our environment. And we do lots of, I think, interesting internal comparison between representation and context to then over time, uh, define these rules. Hmm. So in some sense, you know, the rules actually are reflected in the statistics, um, but the statistics don’t, uh, from them. Just having the statistics alone won’t give you the rules. So there’s sort of asymmetry, uh, uh, between the sources of information. So,  

Paul    00:10:34    Okay. Yeah, these are all like big problems. <laugh>, I wanna just zoom out and, and back up maybe and ask how so, so language has always intimidated me. I mean, I, I remember I took a linguistics course and it was kind of fun, but just the idea. So I come from a non-human primate, neurophysiology Yeah. Background, um, at studying eye movements and decision making vision. Mm-hmm. <affirmative>, the vision sciences as, as everyone else is 90% or something. Uh, do you know what the percentage is? I don’t even know.  

Andrea    00:11:01    No, I don’t. I do know when I was a PhD student at nyu, it seemed like everybody else except for me, and like my three friends worked on vision. So <laugh>. Yeah. So my anecdote, it’s, there’s a big pop, a big population of vision scientists.  

Paul    00:11:12    Well, one of the reasons is because, um, the, you know, the musculature of the eye and making eye movements, all that is like really worked out Yeah. The circuit diagram and how all that works. Yeah. And I, you know, so that makes it a somewhat more tractable  

Andrea    00:11:26    That’s appealing. Yeah.  

Paul    00:11:28    It is appealing. Right. Um, and, and, you know, going back to my own fears about studying language, um, and I, you know, I’ve been focusing a little bit more on language recently, uh, as we were talking just a moment ago, off air, uh, due to these large language models. Yeah. Um, and, and the rise and the popularity of, of these. So, um, and you know, I’ve, I’ve had multiple people on the podcast talking about language, uh, but I think if I, you know, I’m not sure I, I feel like language. So here’s, here’s the question is Yes, I’m curious how your own view on language as an object of study mm-hmm. And just as a cognitive process, uh, has sort of evolved over time. Do, has it become, uh, uh, easier, uh, you know, seeming like a more, like an easier problem to tackle? Or has it become more wily and <laugh>?  

Andrea    00:12:14    Uh, no, I think it was always wily. So I do share with you that fear <laugh> of language or of the enormity of the problem and the, the endless, uh, complexity. And for me, it’s also the importance is like language is so multifaceted. So I really focus on, on what it means to have, uh, structured representation that’s linguistic, uh, in something like a neural system in Neurodynamics. And that’s, for me, that’s really core and key part of language. But there’s so many other parts of language that are, that are equally important and linguistic, and that are also harm hallmarks of, of the incredible human capacity of language itself. So, uh, that for me is like the source of a fear, cuz it can become so complex and there’s so many things that obviously matter to the brain. Yeah. Like when we’re talking to each other, you know, we can name a long list of factors that are gonna be really important, including our eye movements and joint attention and all these things that lead to, uh, mutual understanding and our ability to refer to things.  

Andrea    00:13:04    Right. Um, but I, I guess I’ve already sort of been comfortable with that <laugh> that that complexity or that chaos of what needs to be explained for a long time. Uh, and I think over the years, uh, I’ve gotten sort of more comfortable or more situated in Oh yeah, no, it really is. I really am trying to explain the thing that I want to understand. Like that’s what I’m, that, you know, even as the rise of large language models come, and I feel often that I’m, that I don’t, uh, always find, uh, a camp of people who, who see things the same way. And that can be good and sometimes challenging and can sometimes very sort of lonely <laugh>. But more and more I sort of think, and it’s like, no, this is really what, what I think is important, right. So that I don’t think that we get very far by trying to, uh, deny what linguistics tells us or escape what psychology tells us about, um, the importance of timing, for example, in, in cognitive processing.  

Andrea    00:13:56    And so, um, no matter sort of what comes in terms of developments in these incredible language model, uh, systems, I still don’t, I don’t think that necessarily negates all the other insights of, of, you know, of 50 years if not centuries of insight and linguistics and psychology. If anything, it sort of, uh, you know, emphasizes or, or reiterates how important it’s that we have a theory that can account for both types of information, so for structured information and statistics about it. Right. And so far, that’s really what I, what I keep seeing is just being absent or, or, or not being there. So we kind of as a field polarize and go back and forth. So we say, okay, no. Well, for a while we focused a lot, uh, on, on symbolic structures and representations. I’ll still, I think not, not as much as we could have or not in the way that would’ve been most effective <laugh>. Uh, and then, you know, there, there there’ve been different sort of periods in the history of cognitive science and in language where you have a lot of focus on, on the importance of statistics and statistical learning. And I don’t think one excludes the other by any means. And I’m just sort of hoping that more and more people, uh, come around to that point of view. Because that’s really the hard problem is explaining both or, or understanding why both are important to the brain, uh, for processing language.  

Paul    00:15:05    Okay. Right. But, but so your interest is really in how the brain does it, how the mind does it. Right.  

Andrea    00:15:10    Yeah. I don’t know if I would make a distinction between the brain and mind, because I think that’s also scary and difficult, but important. But yes, in both, I would say, if we wanna be dualist about it, let’s say both  

Paul    00:15:19    <laugh>. Yeah. Well, I, I just mean in contrast to, you know, the idea of multiple realizability, like if you can, like, can you, you know, maybe you can brute force this thing Yeah. Using just a statistical and associate up approach. Yeah. Um, and then structure comes along for the ride. Yeah. I mean, or do you disagree with that?  

Andrea    00:15:35    I, I would disagree with that because I mean, I think it’s gonna be really hard to show that structures there without, um, sort of, uh, theoretical or like, you know, choices that you make to the instantiation, um, to the particular model that says, you know, here’s what we think the ingredients of structure are. And there, I think there’s a lot of, um, stuff we could talk about in cognitive science about what those things might be. Waiting for them to emerge, I think is kind of a weird approach. I mean, maybe it’s possible, but is that really the approach that we want to take? And then if you sort of wait for structure to emerge, uh, from all the data and all the compute, and the question is what are the right benchmarks or tests? And I feel like that’s a huge can of worms and that we can all, we can work on that and fight about it for decades.  

Andrea    00:16:16    And I’m not really sure that that’s as efficient as saying, I wanna build these things already, uh, uh, into the model. Then of course, that makes the question, well, what are those things and how do we do, how do we agree on what is important? Hmm. Um, I think a lot of people do want to say, oh, yeah, I, no, the structure is emergent. It’ll be there. But I feel like that’s, you know, sometimes I wonder how, you know, how do we evaluate that? Is that really what is, what is the science behind that? How, if, if that’s the case, which I don’t think it is, if it were the case, but you still need to explain how did that happen? And, you know, what does that tell us about the human organism? Right.  

Paul    00:16:47    Right. <laugh>. Okay. So, so, uh, you said a lot there. Um, well, you know, I, I’ve seen multiple people saying that maybe, and we don’t, we’re not gonna go down the large language model, um, pathway quite yet. Okay. But, uh, just in terms of, you know, structure emerging Yes. From a purely statistical approach, well, you know, then there’s always the question of meaning in the large language models. And, um, multiple people I’ve had on the podcast have argued that, uh, and, and others have argued that, well, maybe you can learn meaning later. Um, so it’s the opposite direction, um, yeah. From the way that, that humans develop, right? Yeah. Where the, the meaning kind of comes along with the language, but maybe you can train up these large language models and that grounding isn’t even necessary for meaning.  

Andrea    00:17:32    I mean, maybe I wouldn’t, I wouldn’t exclude that possibility, but that seems to me like an engineering question. I mean, the main thing that I think the models are missing that you need to get at, meaning, uh, scary, scary topics is intention, is intentionality. Right. That’s what these, what what no <laugh> system has, and we don’t understand how humans have it. I mean, I, I think, I think that we have it, and I don’t wanna, I don’t wanna dispute that at all, but how we have it is already that’s, you know, that’s also a, a, a theoretical object worthy of study as well. Right. So the fact that our representations are about things, um, that’s difference.  

Paul    00:18:06    Yeah. I mean, the reason I brought that up is just because of the structure slash statistics, um, yes. Uh, question as well of, you know, whether structure can come along later, and does it matter the order that these things are learned? And, you know, do we need to pay attention to how the human brain and mind develops?  

Andrea    00:18:21    I mean, I think yeah, that, I think it depends on your goal. I mean, I, I think we need to, because I wanna come up, I wanna ex that’s what I wanna explain. That’s what I wanna study, is how, uh, yeah, the human mind and brain do something, uh, as interesting and powerful, uh, formally as language. Um, but I don’t, I think in terms of like the question, can you get a lang large language model to pass some benchmarks on meaning by first, you know, by adding stuff, grounding later. Sure. Why not? I mean, that’s, that’s the other thing. I mean, there’s like infinite ways to cut a cake. There’s infinite ways to arrive, probably at the end behavior that we, that we want. But is it following the path, uh, that we are now? I think that there’s actually, there’s a bit of duality.  

Andrea    00:18:59    There’s, who can say no, I think for a theory of how cognition or language arises in the human mind and brain, how it’s learned or how it’s implemented, or how it’s, uh, how it came about. Yeah. I, I think that you could, some, some people I wanna sort of leave room for other <laugh> other points of view. Some people might think that having something like, okay, this is the limit of what can be learned, whatever it is, we still, you know, that’s also a subject of debate, right? Right. Uh, from all of the data and, uh, from increasingly opaque training practices, um, you know, there’s lots of leakage. Now, now, you know, a lot of, we, we can’t really separate even between, maybe with g bt fort’s a bit different, but with chat chatt, b t the, the role of, of reinforcement learning, human feedback is likely quite large, right?  

Andrea    00:19:45    Mm-hmm. And we just don’t, we, we don’t know, you know? I mean, then, then I think we really have to turn to the, the, the open source, um, uh, large language models to, to, to study questions like that. But I’m not, I don’t see why in the limit that’s gonna tell me necessarily anything about how it works in the mind and brain. Um, I will say though, I’m one of the, I feel like, uh, several sort of, uh, I guess analogies I’ve been using to like think about this for myself, but also to, to communicate with other people, said, you know, in the history of science, right, the debate between geo centrism and helio centrism, um, you could actually obtain better model fits better prediction of celestial the position of celestial bodies and their path, uh, with geocentric models and models that p the earth is at the center of our, of our solar system.  

Andrea    00:20:33    And that’s because you can do that by adding more and more parameters, <laugh>, that essentially overfit the data. And so I don’t think it’s a coincidence that, you know, we have, in order to get the level of behavior that, that we expect now we need to have more parameters than training data points, because we’re really, you know, asking these systems to, uh, to, you know, return things that humans are satisfied with. And basically through, I think parameter interpolation, but not to ver verbatim reproduce the training data. And Right. In order to do that, you’re gonna need this, you know, extreme parameter space.  

Paul    00:21:11    We’re gonna come to your ambitious theoretical model. Okay. Eventually <laugh>, but studying language is necessarily a human endeavor. Well, um, okay. That’s could be argued, but, um, yeah, the vast, vast majority of studying language is, um, using humans, right? Yes. And, um, another, I guess, advantage of not using humans or studying something that’s not necessarily just a human, uh, cognitive function is, um, being able to be more invasive in your experiments. Yes. Um, so does the ability to acquire a limited range of data like e e g and fm r i, do you find that as a fundamental limiting aspect of your research? Or, well, I’ll just leave it at that.  

Andrea    00:21:54    Yeah. I mean, I do. I mean, although I see like, I mean, there, there’s, I feel like, you know, there’s such, you know, the ethical reasons are solid. So, I mean, it is what it is, right? <laugh>. Yeah. Yeah. Um, I, yes, I do feel it’s a limitation. Um, I’m not sure, I can’t really imagine a world where we fully got around that, but, um, I think it does certainly limit the, the, the rate at which we can map, you know, concepts and linguistics and cognitive science, uh, two concepts in neuroscience. Because like in, in my model, for example, I’m really am trying to think about things in a population coding level. Yeah. I just can’t get data that are gonna gimme that resolution.  

Paul    00:22:30    Right? Yeah. I mean, a large, a large part of what you deal with is, um, oscillations and you, you can measure, uh, oscillatory measures using e EEG and, and M e g mm-hmm. <affirmative>. Um, but then you’re even, you’re interested in the underlying neural population level, um, with, you know, with gain modulation inhibition. And so you, uh, maybe we can just talk about your model <laugh>. Uh,  

Andrea    00:22:53    Yeah. So, I mean, I guess one way to, to say that is, so I, I mean, I still, we, the readouts that we use are from, from M E G and more and more I talk about them as neurodynamics, because I wanna sidestep the, the debate about, you know, pure oscillators and Oh, okay. <laugh> and evoked responses. Cuz I think, you know, spoiler, it’s both. I don’t wanna get sort of sidetracked in trying to establish what, you know, again, a false dichotomy. So many things in our field are really a false dichotomy. It’s really both like <laugh> structural statistics. Yeah. Yeah. Um, I think that, you know, the evoked responses are really important. And sometimes I think they, they’re endogenously generated. So there’s state changes within, uh, you know, that are driven internally. Right? But that’s what recognizing a word is, uh, from speech or, uh, putting two words together, you’re, you’re driving that change, that, that state change, uh, internally.  

Andrea    00:23:35    But, um, for me, the point, the important thing about neurodynamics or population rhythmic activity or oscillations, is to remember what their, their, what ultimately their source is, uh, in the limit, right? It’s driven by neural ensembles. We might not understand the relationship to single unit activity, but single unit activity forming these different ensembles, uh, and, and synchronizing together, uh, at some level that is going to be reflected in what we can read off, uh, in our m e g signal. We just can’t exclude different interpretations of it. And that is very limiting. But it doesn’t mean that we should only theorize about the readout. It would be sort of like if I tried to come up with a theory of ency linguistics that was about trying to explain, uh, reaction time differences for the sake of reaction time and not trying to talk about the cognitive processes or the limits on eye movements of how, uh, eye movement control works, right?  

Andrea    00:24:25    Not taking that into account. So for me, it’s thinking of something, you know, trying to come up with a theory about, for example, actic structure building and oscillations that assumes any kind of one-to-one mapping between structure building and oscillation. As if oscillation is a, is a kind, it just doesn’t really, uh, float for me. So I try to think, okay, I wanna try to do this difficult abstract vehicle theoretical work. I’m at first gonna go to sort of this space where I know, given everything we know about the world, this is where the action has to be happening. Even if I can’t directly measure if myself, and then try to challenge myself as I develop the thel model and, you know, different instantiations of, of, uh, specification in a computational model, uh, and different empirical projects try to see, okay, can the basic tenants, the first principles of this model, um, what would they look like when expressed on the other end? You know, in m egen neural dynamics, which I can’t, you know, again, pin back one-to-one to a particular state of affairs, uh, uh, int interally, uh, or in on the population level. But still, I want them to be largely consistent with one another. Right? I don’t think that it’s, it has the same explanatory force to come up with this, uh, a theory of syntactic structure building that talks about the, the delta band or the theta ban as divorced from the things that are actually producing those signals. Mm-hmm.  

Paul    00:25:39    <affirmative>. So neural dynamics, I’m gonna have to, I’m gonna struggle to continue cuz I just wanna fall back on oscillations <laugh>  

Andrea    00:25:47    Just say oscillations. That’s absolutely fine. We just go with the agreement between you and I that we don’t, we’re not trying to talk about pure oscillators <laugh>.  

Paul    00:25:54    Right, right. Well, that’s the thing is it puts such a different flavor, uh, onto it and makes it less, it makes it more agreeable because you can talk about dynamics is such a general term. I know, right? Yes. And your model is, you know, talks about neural trajectories Yes. Along a manifold. Yes. So that’s dynamics as well. Yes. And you don’t have to talk about oscillations per se Yes. In that regard. Yeah. But, okay. So, uh, while I was gonna ask you like what the general, I know there’s no, never a consensus in science, but, um, what the, um, current, uh, current, uh, favorability, um, within the language, cognitive sciences, ang of language, uh, community of oscillations is, or neural dynamics, or you, you would answer them differently, right?  

Andrea    00:26:42    I would, I would. So neurodynamics I think is actually it in, you know, it, it includes more people and it includes more, um, periods of time, periods of time, uh, when, you know, like you couldn’t include event related potentially research in that. Right. Um, but on the other hand, I mean, I think, I think the terms should be flexibly used, but what I find <laugh>, this is a bit meta, uh, response, but sometimes you can use certain terms and they can be very, um, they can, people can get completely the wrong idea about what you mean, or they react so strongly to that term. Oh. That they don’t listen anymore to what your argument is <laugh>. And so I just wanted to, I wanna try to reach some the, you know, most, uh, uh, I guess like prepared listener or, or interested listener, not by sending people off already on Oh, but it can’t be a pure oscillator or it can’t, you know, it  

Paul    00:27:26    Is rhythms not better or,  

Andrea    00:27:28    I think rhythms is fine. I mean, it, it depends like, I would say for, again, sorry, my theoretical <laugh>, my laptop thing is shaking in. Oh, that’s okay. I think it depends on your, on what your interest is. So I think if you’re gonna use the term oscillations, um, then, then you have to be, you know, careful about it to be sort of neat <laugh>. Um, if you’re gonna, if, if, if you don’t want to make that kind of commitment yet cuz you don’t have maybe all the information you need like myself, then, then I fall back on terms like neural dynamics. But for me, I find I’m, I’m, I’m okay with all these terms. I don’t find them  

Paul    00:28:00    Okay.  

Andrea    00:28:00    Uh, that they’re, that they’re making important theoretical distinctions at the moment in  

Paul    00:28:04    Language sense. Many moons ago I had, uh, David Poppel on the show and, um, you know, you cite he had some of the early work linking rhythms linking neural dynamics too, like, uh, phone ees and, you know, at the syllable level Yes. And correlating, um, uh, different rhythms in EEG signal, for example, to, uh, the rhythms of the language. Yes. Um, and so, so I’m curious how, so back to my question, just how the, uh, field writ large sees this kind of approach that you have, you know, continued and, um, and we’ll come back to your work, uh, as well.  

Andrea    00:28:40    Well, I think there’s certain, I mean, there’s like multiple layers here. So you’ve got people who are, like I’ve referenced before, the debate between, you know, is it an oscillation or not sort of camp. And that can, that can be focused around speech and music processing, but it can also sort of be in all areas of neuroscience <laugh>. Um, then sort of within, um, sort of the oscillations and speech world, you have people who are really interested in speech processing, uh, or in how oscillations might, uh, shape perception psychophysics, um, who are sort of very focused on a different sort of level of, of explanation. I think. So I was recently at a, one of my favorite sort of workshops that gets together to talk about exactly these issues. And I was just sort of like, I was sort, how can we not be talking about language when we talk about the brand’s response to spoken language? Right. I mean, that’s really, I think, the behavioral goal. And I feel like the lar you know, the more, uh, I guess the more, I guess you would say it’s the more low dimensional behavioral goal of the brain, right? People are trying to settle upon, uh, interpretation of, of the speech stimulus, right? Mm-hmm. <affirmative>. And so that, that’s gonna be linguistic in nature. If you know the language, what,  

Paul    00:29:46    What’s the, what’s the alternative? Just talking, go  

Andrea    00:29:48    Ahead. No, but I don’t, I don’t think that people actually think deeply enough about this. So when you’ve, I think there are interesting and important things to be, um, uh, explained in terms of speech processing, but that there might be a lot of, again, dynamics or signals that are, that are, uh, going to sort of steer the brain that are more about more mesoscale objects like words or morphines, uh, or intonation or prosody or tear taking. These, all these different kinds of things might actually be driving more of the, of the brain’s, uh, systems or the trajectory where you are at, at a given moment than per se Oh, reading out individual phoning, or if you wanna talk about that, then, then you have to, you know, you, I think employ a different kind of close experimental technique where you’re really manipulating things on a very fine grain level.  

Andrea    00:30:33    Mm-hmm. Um, so I think, uh, the sort of world of oscillations and speech processing is really quite, quite focused on understanding, uh, the theater rhythm. And they have a lot of, um, uh, focus on things like decoding, right? Mm-hmm. <affirmative>, so you, so you mm-hmm. <affirmative>, you know, segmentation and ial. When I see that literature, I feel like it’s a bit, it’s a very important and inspirational literature, but for me, the difference is I wanna have sort of orthogonal evidence to, uh, instantiate a, a process. Like, if I wanna attribute, because, you know, as you know, neural signals are so messy, right? There’s so many things going on in their brain, and plus all the signals, noise problems, <laugh>. And then on top of that, it really wanna have a strong motivation for saying, okay, this difference that I find it’s really about segmentation or decoding.  

Andrea    00:31:18    I feel like that that requires a lot of sort of theoretical pre-gaming <laugh>, uh, and, and, and computational modeling, I think, or, you know, ways to develop, um, more specific claims about what those operations might be before you can at clearly attribute, uh, neural readout patterns to one or the other explanation. Um, but to go, to return to your question about the sort of the, the status of the field, if I had to sort of like summarize it in a high level way, you’ve got the focus really on, uh, on speech processing and oscillations, and then there’s a, a, a smaller, uh, but enthusiastic group of people working on, uh, uh, what would have been called sentence processing a few years ago in oscillations. And this actually has, uh, roots even decades ago, um, when people started looking in the time frequency domain, mostly with eeg, um, and looking at, uh, basically power modulations.  

Andrea    00:32:06    And now this has sort of, uh, uh, had a resurgence where we try to, you know, we do have different techniques to look at the relationship between power and phase of different bands, uh, and different linguistic annotations. And within that camp, there seems to be, uh, some people who are more focused on, uh, how much can be explained, uh, by having sort of a, a one-to-one mapping, either between syntactic structure, uh, or prosody, uh, or ris, uh, and, uh, neuro oscillation sort of modulations of them. Uh, and then I guess, yeah, I guess that’s, would be sort of the main, the main approach. And I, I sort of don’t really fall into any of that  

Paul    00:32:45    <laugh>. Right. Okay. Well, let’s talk a little bit more about your model then, but yeah, because as you were describing, uh, the way that the field currently sees this, it made me think of your model. And so you posit, um, I mean, there’s a lot of different aspects to your model, but <laugh>, but it all kind of begins with an internal, um, predictive sort of model mm-hmm. About what is coming speech wise, right? Yes. And, and that is like shaping, uh, the, the incoming sensory information That’s right. And pushing it along a manifold. That’s right. Yes. So in that sense, and so maybe you can elaborate a little, a little bit more on your model, but I’ll, I’ll seed it with this question. Like, in that sense, are the, uh, like theta are, are the rhythms, um, sort of a readout of that shaping process? Yes. Would you say <laugh>?  

Andrea    00:33:33    Yes, I would. Thank you. I’m so, it means so much to me that you, that you, that you got that <laugh>. Oh, that’s wonderful. Thank you. Yes, <laugh>. Um, yes, that’s exactly, that’s exactly right. So what I, when I, when I was working on this model, I really wanted to challenge myself, um, to try to capture just the minimal, you know, things I couldn’t give up about language <laugh> with what I felt was this very sort of, you know, abstract, low dimensional, and you could argue it’s high dimensional, but mm-hmm. <affirmative> sort of, you know, manifold approaches in neuroscience that were, that have been, that are only becoming more popular, but were sort of burgeoning a few years ago. Uh, and it took a lot of, you know, really dove into that to try and say, can, can I really make these things? You know, maybe they can’t ever meet in a one-to-one way.  

Andrea    00:34:17    Like I, like I’ve talked about, I don’t think there could be this one-to-one mapping, but at least how much do I have to break concepts in both fields to try to then fuse them together? Uh, and I mean, yes, I think, uh, it, it asks a lot of readers, so I, I, you know, that they have to sort of be willing to dive into both, uh, domains. Um, but for me, that was the goal. Cause I was, I was no longer sort of satisfied with coming up with, you know, box model, like theories that just didn’t have anything to say about the neural readouts that we were measuring. Um, and I didn’t want to work on trying to understand these neural readouts and giving up, you know, sort of my object of study or the lessons of, of linguistics and cognitive science. So that was my response to that problem.  

Paul    00:35:00    Well, can you, maybe, in your own words, can you describe your model <laugh>, like the, the, uh, broad,  

Andrea    00:35:05    I thought you did a great job, <laugh>.  

Paul    00:35:06    Oh, okay. Well, I, I want,  

Andrea    00:35:07    Yeah, no. So basically what I wanted to, to, to, um, capture with this model was really the idea that, um, that, uh, you have this unfolding trajectory where you can get a lot of the information that, um, typically would’ve been thought about in more traditional, um, cycl linguistics or cognitive science models as being sort of separate, right? So even, uh, I don’t, in no way deny the fact that syntax and semantics can be described orthogonally, right? So you can have separate formal linguistic fields and tools and methods for describing, uh, the systematicity of those information, uh, sources and language. But I think to the brain, um, even though it might the brain, I believe that we, and the brain knows, uh, rules that are formally dissociable, I think the actual representation of that information in real time and processing probably does not best serve the brain by being distinct.  

Andrea    00:35:57    And so, one way to bring that together is to think in a manifold space where you have, uh, populations that might be able to, uh, to decompose the source of in, of information like tics and semantics on different dimensions. But it’s really the, the act, the patterns of activity that they, uh, that you can project into that manifold space that are going to capture, uh, that information for language, right? So it’s kind of a way of having your cake and eating it too, so you don’t deny <laugh> that you can separate these sources of information, um, formally, but you have a way of expressing them, uh, in neural population activity, uh, that where they, where they, uh, impinge upon each other, right? Where you have to plot one in relation to the other, and that’s the key. Uh,  

Paul    00:36:37    So the manifolds are nested much as the structure of language.  

Andrea    00:36:41    Yes. Although, again, I don’t want to say that that’s a one-to-one, uh, relationship in the structure, although I, yeah, I tried to say that, that, that they’re not injective in the paper, but I think that, you know, discovering exactly how they’re nested and how much, uh, how they map onto each other, you know, how much different levels affect each other and how that’s expressed in the space time of the brain. Like where in the brain, and when I think that that’s all obviously up for debate. This is more a theoretical ex, you know, is this possible? What would this look like? Um, and just, just last month, I, uh, managed someone who was actually, who start, you know, starting to work on manifolds and speech processing, and I thought, this is really, really cool. I mean, they’re gonna, I’m hoping that they’re gonna solve all of the really hard <laugh> bottom up problems making manifolds of speech processing and brain. And then I can come back, you know, in 10 years and be like, oh, let me see if I can test my models on, on yours, <laugh>.  

Paul    00:37:29    Well, you, we talked, you know, I, I mentioned it’s a theoretical model, but yeah. So how’s it coming along? Where are you in the process of, you know, implementing? Yeah,  

Andrea    00:37:37    So the various, yeah, there are various, you know, it’s sort of various offshoots. I’m not sure that I gave, like, you know, all the, the best description of everything that the model claims, but, um, like maybe I can sort of, that can come out as I keep talking. But, so what, what I’ve been trying to do is sort of the, the court claim, right? Is that you’re gonna get relational structures, sort of the proto ingredients of something like a word or, or, or a syntactic structure, um, by looking at the, the dynamics between populations, right? This is how you can also preserve information. You can bind words together, uh, in models, uh, like Dora discovery of relations by analogy, that’s Alex Dumas’s, um, model. And he’s a good friend and collaborator of mine. And we’ve worked on this question together in the computational modeling space mm-hmm.  

Andrea    00:38:18    <affirmative>. And then, so my theoretical model sort of takes that as a core principle. So you’ve got this time-based binding, so you’ve got the neurodynamics that are, uh, giving you these relational structures. Uh, and then the question is, you know, what are the sort of broad strokes predictions of that <laugh> that we can measure that we don’t need to have, you know, data we can’t get, uh, in order to, to validate or not. Uh, and that a couple of, uh, students in my group have been working on projects that look at that. So basically the sort of most course grain claim is that there’s going to be more, uh, phase synchronization, uh, the more structure that you build. So the more that you have to coordinate these, these distributed populations together in order to establish relationships between them, the more they’re gonna be synchronized. That’s like a very, you know, broad  

Paul    00:39:01    Stroke <laugh> might, but might there be like a limit or a, a, a ceiling right, of synchronization that  

Andrea    00:39:06    Would, would, that’s the important thing. So one of the things that, that we are, that that comes back again. And so what’s the right baseline? So what, what is sort of, you know, the base rate of synchronization, uh, in the language network or in the brain in a given time, or when you’re listening to stories or when you’re in resting? So establishing that’s really important. Uh, we’re not totally there yet, but looking sort of within, uh, like within experiment condition contrast. So between, uh, phrases and sentences, for example, when we’ve meticulously made them the same length of in time, uh, physically and indistinguishable <laugh>, um, this is the work of fund by a recent PhD student, uh, graduate, uh, in my group, that we can show that, that when you make these stimulus so that they really are physically as similar as possible in the same amount of time, we can still detect a difference in synchronization and different measures related to synchronization, uh, in EEG between whether something is a phrase or a sentence. And this is possible in Dutch because we can, we can make phrases, uh, and sentences the same number of syllables.  

Paul    00:40:06    How, so going back to the, you know, just thinking about manifolds again. Yeah. I mean, is there a, so, you know, I’m thinking of these kind of nested manifolds, um, and the face synchronization, um, helping them communicate between each other. Is there like an upper limit to, or, or maybe I should say lower limit, like the lower, uh, lower limit to the dimensionality, right? As you know, because we think of manifolds as like a lower dimensional Yeah. Um, structure, um, lower dimensional from the, let’s say you have a a million neurons, right? Yeah. That would be a, a million dimensional Yeah. Um, uh, space. Yes. Uh, that then gets lowered, right? Um, and I, I relate the dimensionality to the abstraction of like a concept or meaning. Yes. So where’s the, where’s the bottom  

Andrea    00:40:53    <laugh>? Yeah, that’s a good question. Um, um, I hadn’t thought about the absolute, you know, what is the, what is the absolute bottom? I was thinking, you know, the important thing to remember with the manifolds that I talk about in my model is that, um, you know, there’s defined over a given time period in, in, in, you know, a moment in the brain, so to speak, or data set that you have. So, um, I don’t know if you mean sort of like the bottom for all representational states or for a given  

Paul    00:41:19    Well, I’m just thinking about intelligence and, and creating concepts and understanding, you know, the meaning of a concept, for example, or the meaning of a, uh, story. If we, if we  

Andrea    00:41:32    Wanna like, so maybe, so maybe this is, maybe this is a way to, to approach the question. So I was talking actually to Alex Demas, my friend and collaborator about this. So, um, you know, just because we, we, we think that, um, neurodynamics are important for representing structured relational symbolic information that doesn’t exclude the fact that, for example, single units can learn or, or encode, uh, you know, conjunctive codes, um, or that you have a localist node or a localist code, which is different from conjunctive coding, um, that, that would encode, uh, some kind of, uh, you know, necessary ingredient for relation or a particular binding or particular relationship between a value and a variable. So probably the system is gonna use both, right? So, you know, the famous grandmother cell and things like this, I don’t think that the existence of cells like that, you know, the debates, I don’t think that the existence of those, whether that’s, you know, real or not, uh, necessarily says that neurodynamics aren’t important. I think that neural dynamics being important for symbolic representation doesn’t exclude the fact that you have cells that, that encode this, right? So I think it’s a, uh, you know, a sparse and redundant system, if that makes  

Paul    00:42:37    Sense. Yeah. Well, I’m not thinking so much in terms of neurons and I, I, I apologize because Good. I think my question is ill formed. Okay. Um, but, but I, I think the issue is, I’m trying to picture these like state spaces Yes. And, and their relation to each other, right, okay. Within language. Right. Okay. And I wanna like, so then I necessarily for myself, because I’m not that bright, I, you know, okay. I’m trying to like see the, kinda like the shapes and how they relate to each other. Yeah. And then where’s the end of it, right? Where’s the, uh,  

Andrea    00:43:05    Yeah, so I was, so I was trying to think about what, what you might mean with your question that it could be like, what’s the,  

Paul    00:43:10    Sorry, making a better question.  

Andrea    00:43:12    Make it better. No, it’s on my end. I just don’t understand the question. No, that, uh, like I was gonna say that the sort of, you know, maybe one way to think about it is what the system tries to do again, because with these manifolds, you can do perceptual inference, right? That’s another big part of my model. Yeah.  

Paul    00:43:27    Talk about that for a moment. Yeah. Cause  

Andrea    00:43:28    Yeah, so you want to sort of basically leap, you know, as much evidence as you have, you want to take it as far as you can, and you want to, to leverage your knowledge about the hierarchical structure of language and your language experience, you wanna leverage both of those things, right? So you, you know, you, we know that we have explicit sensitivity even to, you know, speaker adaptation. So talking, talking to each other can change our vowel, uh, spaces. Um, and I think that, you know, we obviously want to use that information as much as we can, uh, but we don’t want to, you know, be ta taken over by statistics such that we end up in the wrong, uh, interpretation or not, not what’s being said. Right? Cuz I know in an extreme <laugh> an extreme situation is if, if, if we really were only just predicting the most predictable thing, we would only hear that thing, right? We wouldn’t hear what was actually said. Cuz people don’t always say the most predictable thing. In fact, that’s often how we get important new information. Um, so I guess that’s sort of the, the lower limit on conceptual representation, if it’s, if we’re not gonna talk about, about single neurons, then I would say is okay, when you have something that you really can’t associatively or structurally or relationally relate to anything else, you know, I guess then you’ve got, uh, you know, I guess a predicate without an argument or a,  

Paul    00:44:40    But I, but if, but my internal model would fill that in Yes, yes. My own That’s right. Interpretation’s,  

Andrea    00:44:45    Right? That’s right. Yeah. Yeah, yeah. Yeah. No, I mean, I’m, maybe learning new sensory learning or perceptual learning situations I think is like learning that mapping of some sensory stimulus to, you know, your internal model. And I guess my model would claim that that internal model is so biasing that it’s always gonna be warping <laugh>. Yeah. That, that sensory stimulus towards its state  

Paul    00:45:06    That happens as we get older, right? We don’t listen to what people are actually saying. That’s right. We listen to what we hear them That’s right. For ourselves and our own internal law. That’s right. Yeah. I was just, I mean, so, um, okay, going back to your model, um, and, and the original question there recently original question was, um, you know, kind of how far along you are, like the different aspects of the  

Andrea    00:45:29    Model, right? That’s right. Yes, yes. So, like I was saying, so the, these findings, so we keep finding, you know, again, the important that that, that when language comprehension is, is, uh, happening, we compare it to a bunch of, you know, carefully thought out controls. We just, we see again, you know, this synchronization is really important and also readouts that are related to synchronization. So coupling of various, uh, varieties. And through that we’re then trying to make, um, you know, more detailed <laugh> sort of movies of activation and time, uh, to spoken language comprehension. And then the rub becomes trying to understand those neurodynamics projected in space. So how do we constrain that in the right space? That’s a whole another issue with human brain imaging and that I’ve been working on, you know, starting to work on that in collaboration with others, trying to situate these, you know, dynamic signals in a space is already is a lot.  

Andrea    00:46:17    Um, but then, uh, beyond that, right, you then, uh, want to understand, well, how much of that signal and the dynamics of it is, is just, uh, you know, pushing a, uh, signal through a distributed network. So when you can do, when you can understand what’s being said to you, you know, you’re presumably recruiting lots and lots of distributed networks more so than if you can’t understand it, right? So even just that difference is probably gonna, uh, be important for understanding, for example, delta modulations, uh, in language. But, um, you know, so that, that’s sort of the empirical trajectory. We’re trying to also get a better understanding of what’s the right kind of annotations. So a lot of the annotations that we use to decompose our neural readouts are things like, um, you know, the syntactic structure as annotated, uh, uh, in a story, uh, an audiobook story, word onsets.  

Andrea    00:47:02    We can also use, um, a lot of natural language processing tools to get things like phony entropy, word entropy, ris such and such, and use them to see, you know, as, as we build these more and more, more, uh, specified or articulated models, how much more, uh, of the datas that explain. And that’s all well and good, but lately more in my thinking. I mean, there are many people in the group working on that, and it’s, it’s very important. But I think we’re all sort of coming to the conclusion together. We need to sort of break or rethink how we think about expressing linguistic representation in our annotations. Uh, and so that’s probably gonna come through a combination of more data-driven, uh, uh, decomposition of the neural readout. And that’s something that’s, that actually <laugh> machine learning and deep learning can help with. Oh,  

Paul    00:47:44    Yeah. But that using it as a tool  

Andrea    00:47:46    Yes. As a statistical model, right. It’s a way to decompose.  

Paul    00:47:49    Sorry, I just wanna clarify what, what that means, uh, rethinking how the, the annotations are represented.  

Andrea    00:47:54    Yes. So, so more and more we’re sort of questioning, um, like I said before, where I think that, that sometimes it doesn’t necessarily serve us to think about how syntax and semantics might be orthogonal, even though they’re, they are, you know, they, I don’t take away from any of their legitimacy as separate systems. I think the brain might have to do something more interesting to combine them together or be constrained by them in real time. Now, how would that actually work? So the way that we do it now in our analysis is we annotate things and we, by the way, don’t have any good semantic annotations. Really. That’s another thing that we’re Oh, okay. Thinking about and working on. Right. So the best thing that people tend to have are things like, uh, word embeddings or vector embeddings. That has a lot  

Paul    00:48:33    In, in the, in the state space. And that’s  

Andrea    00:48:35    How you gather their meaning. Yeah. And that does Okay. But that has a lot of assumptions in it that don’t necessarily have to be the case. So we’re also starting to explore, you know, what, what, what are the assumptions are, are they, are they going in the direction that we want or not there? But my point is there that already in the annotations are some of our, is a specification mm-hmm. <affirmative> or a bunch of our assumptions about how it works. And I think that, that, that, that, you know, we need that right now because we, we understand so little of the neural readout, and it’s gotten us, it’s, it’s really gotten us a foot in the door and, you know, but I think we, we need to also at the same time, push ourselves to think like, well, how can we, uh, change how we think about this represenation without denying any of the system facts, but not, uh, really biasing the view of what we see or putting blinders on because we, uh, you know, want to annotate our stimulus in this particular way, because that’s the way we’ve always done it.  

Paul    00:49:22    <laugh>. But is this a job for linguistics?  

Andrea    00:49:24    It is a job for linguistics. It’s where we need linguistics. Indeed.  

Paul    00:49:28    I thought. But the annotation have, has been set by linguistics. It  

Andrea    00:49:31    Has  

Paul    00:49:32    Lis in the past. So they, they might, they might be upset if you came back and say, Hey, we need to do this different. I mean,  

Andrea    00:49:37    <laugh>, I think it’s, it’s something where we work together, right? Where we say like, look, we, this is the property of your, uh, formal analysis that we wanna capture. But when we do it this way, we have a certain assumption about how that relates to the time, neural time series data that we don’t think is necessarily anything to do with your theory. Right. It’s just a part of how the annotation works. What do you think about that? And then we, we can, you know, it’s, you know, it’s not easy or fast work, but it’s important because I think it, it, it, uh, it creates another space where different expertise has come together to try to, you know, what is the essence of what we want to capture with models, right? And annotations are, are in a sense, uh, a model, right?  

Paul    00:50:12    So this would be using an argument from dynamics from time Yeah. To nudge a linguist to think more in terms of how time would constrain  

Andrea    00:50:24    Structure. And there are, I wanna make, there are examples of that are there already. So there are people who have, who have worked and thought about that. But it’s, it’s more thinking about then, how do you know at the end of the day, how do you get a matrix that then you relate to another matrix? <laugh>.  

Paul    00:50:35    Yeah. Yeah.  

Andrea    00:50:36    Unfortunately. Yeah. And you don’t wanna just use matrix multiplication. So <laugh>,  

Paul    00:50:41    Yeah. Okay. It’s the go-to <laugh>. Um, yeah, it was interesting. I, I’d never thought about, um, annotations in semantics. I mean, I, you know, the story is, it’s all in a state space, an embedded vector in a state space. But ha have you guys come up with, um, other potential?  

Andrea    00:50:57    There’s some students in the group working on it. It’s a hard problem. And I think there’s a reason it doesn’t exist at this point. Yeah. But it’s more, we’re focusing more, uh, and this is also inspired by the work of many, many people. Uh, that’s, you know, the, the sort of transition, transition moments when you have a change of state or something happens in the semantic world, not just the distributional representation of a word in a corpus. It’s something that’s actually unfolding in that particular, uh, phrase, sentence or discourse that’s important. And some other, you know, there many ways that this has been reflected to some degree in existing things. So, situation, models, discourse models, but coming up with, you know, sort of, uh, annotations that could reflect sort of the crucial insight of each, uh, traditional level of linguistic analysis, I think is important. You know, not everything is going to stick. Right. That it’s not clear to me. You know, how, what, again, what is the, the limit of this? What is the end? You know, when, when do you stop adding features, uh, or predictors? It’s not, yeah. You can’t, I don’t think you can play 20 questions with features <laugh> or model predictors and when either. Yeah. Um, so, but I think it is at least important, it is striking nonetheless, that we don’t seem to have a sort of standing go-to, uh, temp, temp, uh, temporally situated semantic annotation.  

Paul    00:52:12    Yeah. Well, time is so important for biological systems, right? Yes. I mean, well, yes. I didn’t mean that as a question, right.  

Andrea    00:52:20    Just wanna show. I have, I, I’m not, I did not, I have not read this book, but I have it on my desk. Okay. Cause it’s that important <laugh>.  

Paul    00:52:26    Okay, cool. I’ll have to look that up. The geometry of biological time.  

Andrea    00:52:29    I know you wanna be talking about being scared, <laugh>.  

Paul    00:52:31    Oh, that’s gonna be all about topology and, and  

Andrea    00:52:34    Yeah. Moving space system. Yeah. That’s the other thing. So yeah. Then it becomes all about topology. And that has lots of assumptions that I’m not, that I’m not totally sure Right. Uh, for this problem. But anyway.  

Paul    00:52:42    Well, you have to wear a lot of hats, right? Think that’s the other thing about language. Cuz you have to understand the li linguistic structure or you don’t have to <laugh>, but, um, I think it’s wise too, right? Yes. Um, yes. But, but I guess linguists, so I’m gonna come, I’m gonna come back to artificial systems here in a second. Okay. But, uh, the first thing I wanted to talk, ask you about then, linguists have traditionally discarded time cuz it’s of no importance when you’re looking, or that’s a que that’s a question. No,  

Andrea    00:53:09    That’s right. So I mean, so, okay, let’s, can we can, you know, return to the great David Ma and say, you know, they’re working on the computational levels, they have a computational level theory of what needs to be explained, and that does not have to have any bearing on time.  

Paul    00:53:22    Well, that’s a question. Is that true?  

Andrea    00:53:24    Um, for their brain? I don’t think it’s true. So that, that’s sort of more and more what my models are ending up being is  

Paul    00:53:30    That’s the implementation level, even for the computational  

Andrea    00:53:33    Level. Well, but that you get con that all of the levels actually end up constraining each other in practice. Right. So I, again, you know, it’s sort of like the spherical cow situation. Yes, there is value in having theories on those different levels that are divorced from each other. But for me, and trying to go after the question that I’m going, I see more and more how those levels have to have to influence each other. They have to constrain each other, uh, and time is one, one that can constrain all of them. And yes, how that impinges on the computational level, I think is probably the most fraught. Um,  

Paul    00:54:03    Mm-hmm.  

Andrea    00:54:04    Yeah.  

Paul    00:54:05    Well, you know, th so going back to the, just the importance of, of time. It, it’s everything, you know, between our communication, yours and mine, when you, you know, I’ve, I’ve often just kind of in my, uh, mind wanderings thinking about consciousness and, and being yes, <laugh>, um, and thinking, well, maybe the earth, maybe geological time is just on a different scale. Uh, we don’t have to bring like, awareness into this mm-hmm. <affirmative>, but it, it’s still a time scale. But then a, something like a touring machine or a, uh, I don’t know, a large language model or any computational model, you just can turn it off one day and then, and it’s, then it’s frozen, essentially. Um, and time is not of the essence in a touring machine, for example. No. Because there’s no necessity for it. So is that just a biological constraint or do we really think that like there’s something, uh, that temporal like are dynamics important for cognition? I mean, I think  

Andrea    00:54:59    Beyond biological, this is a very deep philosophical and yeah. Metaphysical question <laugh>. Um, but I, I think I, yeah, I think that we’re sort of dealing again on sort of the meso scale, right? So for our brains and our behavior time is really, really important. And if you wanna talk about, you know, what, what is time? Of course, it’s a question, a deep question beyond the scope of my expertise, but minimally, uh, it, it has a relational quality, right? So there’s some relative, uh, nature to it, right? Mm-hmm. <affirmative> that one thing happened before another, or the path back from one, one state to another is not accessible. And I think not taking to that, that into account would definitely hamper our understanding of how information processing works, uh, and how, and what neural neural time series data can represent or do represent, right?  

Paul    00:55:47    So in that regard, I guess like a large language model does have some ordering somehow, although it looks at everything all at once in one kinda computational step. Yes. Yes. So, but there is relational, a relational aspect,  

Andrea    00:55:59    Right? But that’s, that, that relational aspect is i, is fundamentally driven by what the goal of the system is. Like. So relational in accordance to what, right? So if you don’t have a, there’s sort of no, uh, impetus or bias or reason to, uh, or I could say the, the motivation or the bias or the impetus to have hierarchical structure is not as strong for predicting the next word. If that’s your only task in life as it is for clearly for the task that humans are solving with language, right? Or mm-hmm. <affirmative> that language, uh, you know, I hesitate to use the word evolved because it’s so fraught. But that language appeared <laugh> over time to solve  

Paul    00:56:37    Neural dynamics instead of oscillations appeared instead of involved. No, no. You just  

Andrea    00:56:42    Solicit. I just, well, cause I really, cause I could just go, I could, you know, we could have several of these interviews about all these other topics, some of which I’m not an expert on. And yeah, I, yeah, I try to avoid that, that I wanna give you the answers though, to your questions in a, in a timely fashion. No, I just, I think evolution has a, there’s a lot of assumptions about it that, that, that, that might hold for biological systems where the link to language has to be better fleshed out and more carefully, uh, contemplated in my opinion.  

Paul    00:57:07    Yeah. I mean, that’s  

Andrea    00:57:08    Fair. I’m not an expert. Yeah.  

Paul    00:57:09    Yeah. I guess the, my question really is just about like, uh, maybe I’m repeating myself, but how important is like the temporal aspect to cognition? Is it just a biological constraint? Or is there, you know, uh, are there so many different ways to skin a cat that we can, we will eventually, these large language models, even if they work on a completely different principle, will without having any time constraint mm-hmm. <affirmative>, uh, right. And there’s doesn’t need to be a manifold. Well, there doesn’t need to be.  

Andrea    00:57:43    So one way to think about this is, right. So, um, like I said before, you know, there infinite ways to, to cut a cake. I was, again, trying to avoid the skin. Skin, sorry. There’s infinite ways to skin of cats  

Paul    00:57:53    Dynamics, evolution, and cut a cake.  

Andrea    00:57:56    No evolution appeared, <laugh> appeared. Emergence. Sorry. Yes. Also bad. No, I’m just trying to not step in all of, all of the mine fields in one in  

Paul    00:58:04    One. I need to change that language myself. Yeah. So it’s the cat language  

Andrea    00:58:08    <laugh>. Anyway, okay. So the point is, is what it’s talking about. So the low, so with large language models, maybe what they have in common with the brain, and I don’t know, for me, this is not, uh, a deep insight, but you know, maybe for for others it is, is that, you know, there’s an unfolding time series. And in some way the, the language model is, you know, by having access to everything all at once. But to having this sort of, you know, the different aspects of the architecture, but also the, uh, the, you know, the mapping of the training, uh, procedure and everything that, that do force that kind of, you know, sequential of, of information processing mm-hmm. <affirmative> that, that is not, uh, not in common with how, uh, speech input comes into the brain, uh, at some level, right? But, uh, like we were talking about before with time, you know, I think there’s a lot of, a lot of people also thinking about this notion of brain time, right? That part of the reason that we have cognition, a way that we abstract away from sensory and perceptual representations is to get away from the vagaries of time to be able to do more with that information than just have it be a sort of a coic perceptual thing that’s then gone. We, you know, if we can have memory and we can bootstrap that and leverage all the other, uh, uh, things we can do with those representations once they’ve been taken away from their temporal, uh, extension, we can do a lot more with that information.  

Paul    00:59:22    But in the brain and in, in your proposed model, there’s still, so oscillations have a lot to do with the computational aspect, right? So there, they’re, in some sense, an implementation, although we just, uh, before we’re talking about how they’re more of a readout. Yes. Maybe you can clear this up for me more.  

Andrea    00:59:39    How do you see  

Paul    00:59:40    O oscillations in the role? Yeah,  

Andrea    00:59:41    No, it’s, I think it’s, so it’s again, one of these things where one, uh, characterization does not exclude the other. And I guess this is like coming back again and again for me, is that like every time one is presented with something that seems like it’s mutually exclusive or paradox, if you do a little bit of digging, you quickly find that it’s not, yeah. So, um, I think that it’s definitely the case that, uh, you know, when we talk about oscillations, that they can, um, be, uh, a readout, right? Of computation that you could describe on the population level, right? So do in the sort of most neutral terms. Um, but, uh, uh, certainly you could, uh, use, you know, use the oscillations as a way of talking about the computations on the population level that, as we talked about before, for humans, we can’t access.  

Andrea    01:00:25    But you can still imagine, also imagine a world where sometimes, you know, the computation that’s embodied in a particular population, it’s going on and it gets hit by a traveling wave of, you know, information that is gonna change the sort of the, however you wanna talk about it, the l f p, the base rate, whatever of, of pot potentiation of that, uh, you know, part of that micro circuit or even on a, on a larger level. And then is it part of the computation or not? Yes, but also not no, in the sense of that, you know, and then you’re saying, okay, it was like in the bja term of neural syntax that you’re reading something out on the other, you know, that you have a, a reader that sort of catches the information processing of the last circuit or whatever, right? So you’re not in that direct linkup situation anymore, but you can have your, uh, computation still be affected by it.  

Andrea    01:01:08    So I think sometimes it helps to, uh, to deconstruct a bit what we mean when we say, okay, are they doing computations or not? Is there actually a world where they’re only doing computation and, and they’re not, or where they’re not, where they’re purely epiphenomenal? And often the answer is always in between, right? Yeah. So the same thing with the, the invoked induced, uh, question, right? So, you know, clearly there’s gonna be evoked information that’s important. And clearly the role of oscillations, or something that happens to oscillations is they get combined <laugh> with, with evoked responses. Because at the, you know, we can talk about, um, it is still important to talk about, uh, specialization and encapsulation. And maybe to some degree modularity. Modularity is also one of those words for me, <laugh>. But, um, but I think we also can’t, can’t, can’t pretend like everything is, you know, independent or not, you know, connected spatially next to each other, part of the same organ. Uh, yeah.  

Paul    01:02:03    I mean, it’s, it’s amazing thinking that we can think, knowing that we can think at all, given, you know, there’s the circular causality aspect also mm-hmm. <affirmative>, because yes, part of the computation becomes part of the fun, the readout of the computation. Yes,  

Andrea    01:02:17    Indeed. And, but maybe that’s actually, that’s, that’s a feature, not a bug, right? Um, it’s only from our perspective as the measure measurement person, the experimenter or the modeler that we make this distinction. I, that’s another important thing to always remember with modeling that distinction between readout and computation might only be true at that particular instance, from our point of view, from the brain’s point of view. Or depending where you are in the brain or what <laugh> uh, you know, circuit or computation you’re interested in, uh, those answers can change.  

Paul    01:02:47    So what, how, where am I left then? Thinking about building models of language? Like, should I, are, are we, uh, forever going to be coming up short? Because I mean, a model is always wrong, right? Yes, yes. All models are wrong, but some are useful.  

Andrea    01:03:02    Yes. Exactly. Um, for me, I think, you know, this is also something that I think about a lot with my, uh, friend and collaborator, Olivia guest, um, where we talk about, you know, what, what can models, uh, tell us and what, how to do models adjudicate between theory and data. And that, uh, you know, the data are always right. And that’s not to be a data chauvinist, it’s just to be actually a proponent of theory building that we need to, the theories always need to be better. Uh, and what better actually means is, is is complicated and depends on sort of your goal and the task at hand. Right? Um, but, uh, lost my train of thought.  

Paul    01:03:36    So, so yeah, you’re alluding to, uh, a couple papers. Yes. Uh, the most recent was 2023 this year, uh, that are, is really specific about how to differentiate between levels, right? Theoretical levels, specification levels, implementation. Right. Uh, different aspects of how to  

Andrea    01:03:54    Figure something. Sorry, before I get this, can I just interrupt? Sure. Cause I wanna get back. Yeah. Cause you asked, now I remember what you were asking me is like, what, so what, you know, sort of what is the, where are we with, with building language models in the brain, and how frustrating is that? And while you were talking, I was thinking about, well, what maybe I, it helps to talk about where I, what I worry about with the manifold approach is really like, uh, so what, you know, when we wanna think about capturing a formal system like language in a neural manifold, and this is, um, these are ideas and things I’ve worried about together with several people in my, in my group, CAS K cart kuk, and also with Alex DMAs, um, that, you know, how can you, if we want to give a space to something like language, then, then we’re taking this, you know, this infinite, uh, means and putting it in a finite space, and isn’t that problematic?  

Andrea    01:04:38    And then we started to get, we started to think about this more and more, uh, and basically we, you know, have wanted to do this proof that we haven’t gotten around to yet trying to think about whether, you know, the space of ungrammatical, uh, things is more infinite or a bigger infinite of infinity, uh, than the space of grammatical, uh, utterances. And then can, and to understand the relationship of those, uh, infinite sets, uh, in relationship to, uh, the sets of, of, of what could be expressed <laugh> with neurodynamics in various manifold spaces. Oh, and I don’t know the answer to that and that, but that seems to me to be a clear, again, I think a lot about boundaries. If it’s the case, right, that the, that the orders of magnitude of infinity in those spaces are not, uh, reconcilable, then we have a problem. Right? So, uh, what’s your hunch? That’s for my own, my own brand of language modeling <laugh>. I can answer the question in response to large language models as well, if  

Paul    01:05:26    That’s what you meant. No, let’s let, let’s stick on your own branding of, um, modeling. So what, what’s your hunch to the answer?  

Andrea    01:05:33    Um, my hunch is that time, again, this time and space. So I don’t know, I, I don’t have a very strong intuition about the infinity of trajectories in, uh, different spaces and how, how they can, depending on that, of course, just completely on how they’re defined, right? Mm-hmm. <affirmative>. Uh, but time is a really important, um, aspect for approaching limits, uh, in those spaces. Uh, but my, my intuition about the grammatical versus ungrammatical structure thing is more sharp. And that is that the, the, you know, the space of ungrammatical strings or utterances is gonna be larger than grammatical ones, because even, uh, within, you know, this is a lot of interesting formal work exists in linguistics. You can really show that sort of the space of, uh, in the formal space, human languages still occupy a particular corner of that space. Mm-hmm. <affirmative>, they’re not actually really, um, that spanning the spa the space of formal systems. Yeah. Yeah. Uh, and yeah, and I think that’s actually important. And that can tell us something about the neural implementation of these kinds of formal systems,  

Paul    01:06:29    Because it’s like a limiting Yes.  

Andrea    01:06:32    <laugh>, because it can at least say, okay, if we can understand why those limits are there in that space, uh, can we then understand the analog in the, in the neural manifold space? Again, that gets trippy really fast mm-hmm. <affirmative>, but it’s at least a foot in the door. You know, it’s a way to tack down one part of the map, uh, to the other map, and then to the territory <laugh>, because otherwise it, it just, you know, everything. If you, you know, you are no longer pointing at each other, uh, in a meaningful way, I think. So that, that’s at least in our, in our space. So trying to, to, to sharpen the computational claims of how, of how a neural manifold could lead to functionally linguistic representations is one avenue. Trying to look in neural time series data and neural dynamics to really, uh, understand if the broad strokes predictions of the model about synchronization, uh, and, uh, and coupling, uh, are born out in a, in a basic way. Uh, and then trying to, you know, tease apart or break the ways we think about annotations and form linguistic representation to sort of brain them. How much can we, can we, uh, warp them or, or smooth them or, or change them, uh, to become, uh, more, uh, integrable with our, uh, neurodynamics  

Paul    01:07:38    Is part of the, so, so you mentioned earlier the problem is you have an infinite, um, uh, potential of grammatical Yes. Uh, structures. And then the one that we occupy in that state space, we’re in a very small corner.  

Andrea    01:07:53    Um, yeah, that’s, so it’s slightly different. There’s, like, there’s two, two important aspects here. One is that we, that any language has, its, you know, infinite productivity mm-hmm. <affirmative> basically through recursion. We can keep adding stuff forever. Nobody does that because nobody lives forever, unfortunately. So that is a, that’s a, a limit on a proof <laugh>. Um, but the o the other problem, the other observation is that the, the way that human languages work, they still only occupy one particular corner of the space of possible formal languages, the way the formal languages could work, right? Yeah. From which we we see in computing or mathematics. Right? Uh, and, but I think that fact can be, uh, capitalized upon more than we currently do, uh, in understanding, you know, that state space in trying to relate it to neural dynamics in some meeting, not, you know, not in a one-to-one way. Again, that’s almost never the right approach, but yeah. What are the constraints on that space? What are the constraints on the neural manifold space of expressive trajectories? Right? Uh, and then can we begin in that way to gradually link things together such that we, we have a principled way to decide what are the kind of annotations that we should relate to our, uh, to our neural data?  

Paul    01:08:54    Yeah. Okay. What is the dimensionality of of language <laugh>?  

Andrea    01:08:59    Oh, it depends on who you ask.  

Paul    01:09:00    Sorry, isity one of those words for you. No, do we need to,  

Andrea    01:09:03    Dimensionality is a, is a safe word. It’s a good word. <laugh>, dimensionality reduction. <laugh>.  

Paul    01:09:09    Well, yeah. I mean the, the reason, right. So a manifold is a reduced dimension space, yes. Dimensionality space. And then if we’re, like, comfortable with language on a manifold, it sounds dumber.  

Andrea    01:09:18    Yeah. No, but it, okay. It’s track. Okay. I don’t wanna go into the don’t. Tractability is also maybe a bit, bit, it’s more, it’s at least it’s more studyable for us. Yeah. Okay. And I think, like, again, like what I said, if we think that, yes, it’s not to say that there aren’t general signals that are tapping into all these different dimensions of language and of perceptual experience. It’s just what are the, you know, what’s the PCA of them? What is the thing that’s really, that we can attribute most of the, the, the trajectory, uh, to, uh, and it, and as much as that’s at least signal strength now that doesn’t in any way mean that the, that other information is not encoded or important to the brain. Mm-hmm. Yeah. Then we start being limited by what we can, how we can relate our, our, our models to the data, I think.  

Paul    01:09:59    Hmm. Okay. Um, should we go back to, uh, talking about just how you approach all the, there are just so many issues, uh, and, and recently you’ve written with Olivia guest. Yes. Um, kind of formalizing an approach. I forgot to that. Yeah. Is that, did you take on that project, uh, because just there are so many thorny issues for language specifically? Or is this just a cognitive modeling? Uh,  

Andrea    01:10:22    Yeah. For us it was really like, you know, we, we were working together and we had such a great time writing the first modeling paper and really, you know, really coalesced some of, some, many of the, uh, things that, that she and I think about in our, our own research programs, which are actually very different, but we think, we think a lot in the same ways. So we have a lot of the same sort of reactions to things. And we really ca you know, we have, we really had a sort of, I don’t know, a catalyzed way of thinking about modeling together. And it was just so much fun. And it really, I really felt that we were, um, so many things that we’ve been thinking about really came together. And that was an, a natural extension to that was then thinking about that for, um, for computational, uh, cognitive neuroscience.  

Andrea    01:10:55    And, and also the more, more and more we saw this sort of pattern of reasoning in the field where we think, you know, is that really what people wanna claim? Is that really like, what, what, what, what do we take away from that that can’t, that’s not totally internally consistent? And then we started to analyze it more and think about it more deeply. And then this sort of, you know, is it, it’s all started actually from a tweet thread of some observations and frustrations that I had <laugh>. And then Olivia came and, and, uh, worked as a, a postdoc for a year in my group before, uh, moving on to an assistant professorship in AI here at the university. And that one year we, you know, we just got so into this paper and, and banged out. And there, I mean, there’s so many things that can, that would continue from that. In thinking about these problems,  

Paul    01:11:33    What were the frustrations that you had? Um, I think that you felt compelled to tweet about  

Andrea    01:11:37    <laugh>. Yeah. Always a mistake, right? Never, never work out your emotions on social media. <laugh> in this case, I had a good outcome, <laugh>, but, uh, it was, it’s more that, that, that, um, I see more and more, and I don’t wanna like, make it personal or anything. Cause I don’t, I hold us all accountable. I think it’s really a sort of, uh, zeitgeist or way of thinking that we’re all, we all are complicit in, is that, you know, when you, when you fit a model to the data, uh, and you get a good fit, you know, you don’t conclude that the model then are is, you know, is, is the object of that you’re modeling. And that seemed to just start this sort of, it’s happening more and more. And maybe it’s particular with language, but I’ve also seen it envisioned, to be honest. Um, and I’m really still very interested in exploring what, what does that mean for people? And this sort of a meta science question, like when you say, okay, this model, cuz it’s very different from the sort of way that we learned cognitive modeling, the both of us in the, in, in, in the nineties and naughties. It’s just a different approach, right? Where you start to really sort of, things start to bleed together in a way that it just never as a modeler never considered  

Paul    01:12:40    Ex elaborate on that. Yeah. What do you mean they bleed together?  

Andrea    01:12:43    Well, I think it’s more sort of like the, the map and the territory. So prop, you know, this is the, the properties of the, uh, the map that you, that is your model of the territory. Don’t, they aren’t, they don’t extend the, the territory itself. So I think, you know, it’s in our, uh, paper, we have some great, uh, example with the digital clock, right? I don’t know if you remember this figure.  

Paul    01:13:04    It’s like, yeah, yeah. I was just, I was trying to pull up the paper just cuz I wanted to get the, the different levels, right. So this the 2016 one,  

Andrea    01:13:11    Uh, for my model or for the work? Work,  

Paul    01:13:13    I’m sorry. No, the 2020  

Andrea    01:13:14    23. That’s the paper with Olivia about, um, logical inference and Yeah.  

Paul    01:13:18    But that doesn’t have the, uh, one that has the, oh,  

Andrea    01:13:21    That’s the one from 2021. You mean the, uh, 2021? The, the modeling with the theory specification and, and  

Paul    01:13:27    Yeah. Implementation. Yeah. But we, we can talk about, so the 20 23 1 is, is complex  

Andrea    01:13:32    <laugh>, it’s <laugh>,  

Paul    01:13:34    It’s, um, yeah, no, um, remind us of the, the clock. Uh,  

Andrea    01:13:38    Yeah, so the clock, um, yeah, I have a slide of it here. I don’t know if it, if it’s helpful to share that or not.  

Paul    01:13:43    Well, the vast majority of the audience is, is audio.  

Andrea    01:13:46    Okay. So I’ll try my best to describe it. Um, so the whole, that whole paper is sort of based around what we, um, see as this fallacy and reasoning not necessarily directly in the empirical work. So we’re not making a claim about, you know, the work that people are doing is inductive reasoning. So we’re not trying to apply deductive rules to it. But what we’re we’re saying more is when you create a metatheoretical calculus about that talks that formalizes how, uh, models relate to theories and data that these kind of fallacies seem to pop up. And you see them again and again, we have this whole, you know, old, uh, sort of database of, of occurrences. Um, so when you we’re thinking about a digital clock and analog clock, or a clockwork clock is an analogy to, for example, large language models and the human brain for something, right?  

Andrea    01:14:30    So digital clocks display the time clockwork clocks display the time and require manual winding. Therefore, digital clocks require manual winding, right? So that’s clearly a fallacy. <laugh>, we see it easily when we talk about digital clocks and clockwork clocks. But when we talk about large language models in their brain, somehow this is alludes us. Somehow there’s this strong urge to, uh, you know, give the attributes of the map to the territory. So then we can translate this. So we do this, so artificial neural networks correlate with fm r i data, uh, brains correlate with fm r fm r i data and instantiate the biological me mechanisms for cognition. Therefore, artificial neural networks instantiate the biological mechanisms for cognition.  

Paul    01:15:12    So is this in response to like the, um, the recent work that compares the predictive, uh, responses in brain activity to the predictive, uh, correlates that to the predictive aspects in large language models?  

Andrea    01:15:24    Yeah, I think it’s a, that’s a, it’s actually a tradition that’s been going on for quite some time. Yeah, yeah. No. So then, then I, we’ve made an version of the syllogism that, that makes it more specific. Okay. So artificial neural networks produce grammatical strings. You can say large language models produce grammatical strings, humans produce grammatical strings and in instantiate the biological mechanisms for language, therefore large language models instantiate the biological mechanisms for language not gonna be the case. Right? Um,  

Paul    01:15:52    Yeah, not necessarily It could be the case.  

Andrea    01:15:54    No, it could be the case, but not necessarily that’s  

Paul    01:15:56    The, a logical fallacy to assume it is the case.  

Andrea    01:15:58    Yes. And the other, the other point is that we, and we talk about this in the paper, is there’s so many things that would disqualify these models, uh, as being, uh, models of cognition and Yeah. Or of language. But we, we tend to not, we don’t take, we don’t worry about that fact so much. Right.  

Paul    01:16:13    Can you list some of those?  

Andrea    01:16:14    Well, <laugh> well, you know, the parameters, the training procedures, the data Yeah, yeah. The, you know, complete lack of internal structure that has any interpretable <laugh>, uh, mapping to cognitive theory and that, but that’s all changeable, right? That could be changed. Doesn’t necessarily necessarily have to be the case. Um, but some of, you know, just to say briefly about it, cuz again, this is not my, my focus is that, you know, as you see as the performance, performance becomes more and more important, uh, the predictive quality for the brain is also going down, right? So that, that’s something that many people have, have noticed. So yeah, at the end of the day, I just, I don’t really know, you know, I think about this a lot because it’s so common, right? People really believe that there is something be behind it. And I wanna know, I wanna understand, uh, what that is because what would be, so, so I try to, to ask myself like, what would it mean if we take that, that key, uh, claim to its lemon? I wrote some notes for myself about this cause I was really trying to think about it. Hmm. Um, and I guess, you know, some of the things that,  

Paul    01:17:12    But the, the, the key, the key claim being that large language models are useful as,  

Andrea    01:17:17    No, so they’re different claims. So like, I think that most extreme version is like, large language models are the brain, which I think no one, everyone say, oh, no, I don’t mean that. I don’t mean that. Okay. Yeah. But like, I think another version of that is an, oh, well, uh, they’re a good model or good enough model of language in the brain or clinician in the brain, right? So I think that’s, that’s fine. Okay. And then we can talk about what, what, you know, what’s a valuable model. I think that’s all, it’s all very complicated and we’re thinking about, but, um, you know, of course. Wait, but  

Paul    01:17:40    What’s the, what’s, sorry, what’s the key though? The key claim that you mentioned is the good enough version?  

Andrea    01:17:44    Yeah. Or the people that, so, so what typically seems to happen, and this is what this paper with Olivia that we focus on, is like you, you, you find that a, an artificial neural network or a large language model can predict neural activity. Okay? Uh, and then, and then what do you conclude from that? Therefore, yeah. Therefore, and there’s all this range of conclusions. Like, oh. And so I guess maybe like a good summary, the sort of like middle of the road one is like, oh, then these models are good models of, you know, condition x, task Y, capacity Z in the brain. Mm-hmm. <affirmative> fine. Okay, let’s look, let’s go with that <laugh>. So then I was, when I was thinking about this, you know, so the, the price of modeling or the power of modeling is that you, you sort of exchanged, you, you ex exchange specificity or you simplify and leave out stuff and gloss over things in order to gain insight, uh, into the shared causal structures of the mechanisms between your model and the object of modeling or the phenomenon in the world, right?  

Andrea    01:18:36    Um, but in some way, when we do this, if we say that, you know, substituting, uh, models of cognition or theories of cognition and language with large language models, uh, is good enough, or if we base our theories on them in some level, I think we’re saying that predicting signals is more important than explaining them. And I think we have to think about if that’s what we really mean and what are the implications of that. Maybe people disagree with that. Maybe that’s not what they mean when they say that. Um, then I wanna know more what, what is it, what is meant by that? What is the good model? Is it, it’s a good model because it predicts it’s a good model because it behaves a certain way. That’s the other thing. So in some level, I think we’re saying that behavior is enough. So approximating function approximation is enough. And that seems like an engineering goal to me.  

Paul    01:19:26    But some, I think even some neuroscientists would be, well, neuroscientists concerned with behavior would be okay with that. Yeah. Um, if, if the answer is like, what is cognition, not what is biological cognition?  

Andrea    01:19:39    Yeah. No, I think this, this touches quickly and deeply back onto, you know, debates about computational and cognitive science. Yeah. Representationalism. Um, and again, I think, again, I think it’s both. So I, you know, why can’t dynamical systems represent and compute there? It, it’s, we have take, we have, it’s a choice for us to believe that they cannot mm-hmm. <affirmative>, it’s a choice for us to not to try to, you know, bend or break or reformulate how we talk about these systems both formally and conceptually in order to do that. And that’s what I think it’s the same with structure and statistics face the hard problem. Are they really mutually exclusive? Does one really mean we can’t do the other? When is when you saw,  

Paul    01:20:17    But we’re so much, we’re so much more comfortable thinking and talking in dichotomies in that. Is that a roadblock?  

Andrea    01:20:25    Um,  

Paul    01:20:26    Dichotomous  

Andrea    01:20:27    Thinking on, on some level? I don’t know. I just had the insight thought, oh, well maybe that’s actually like a fundamental part of, like, of cognition and perception in general is trying to give a deterministic perceptual interpretation to a probabilistic series of cues, right? But  

Paul    01:20:40    All from a probabilistic brain, the probabilistic brain is yes.  

Andrea    01:20:43    But then it’s, again, that’s that same tension between structure and statistics and between dichotomies and gradients and whatever, you know, you can create a gradient about from, between anything, right? Uh, is, is that, is again, that tension. So you’ve got, you know, this probabilistic cues and, but the, for, for the sake of behavior and for, you know, I guess some level it’s about practicality, determinism enters into this, the, the processing stream at some point, right? Mm-hmm. <affirmative>, um,  

Paul    01:21:10    And categorizing things is also one of the things we do well that has served us well. Right? So concepts, yes. Making concepts and indeed, so orthogon analyzing everything to its maximum degree. That’s, that’s right.  

Andrea    01:21:21    And I think that that clearly has a, a huge benefit, right? In terms of, of structuring the world, right? Creating an internal structured model. But it has, you know, it has, I think it has negative sides too mm-hmm. <affirmative>. Um, but, and there’s of course very interesting things and, you know, there’s fuzzy logic and vague con, you know, con concepts of how to be tru, you know, truly vague <laugh> Yeah. In logic. And I think, you know, these are sort of the limits of what we can think about, uh, formally as well, you know, on the, on the other side, right? But, so talking about more sort of practically, right? The, the urge to, I guess, or the something fundamental about the, the perceptual system has this categorical goal, right? Because so much of perception is that’s the whole point of it. So <laugh>, yeah.  

Paul    01:22:03    Yeah. Okay. Yeah. All right. Well, I, I, did, I, um, distracted you from, could you had, I think you had some, uh, you were, we were going back to your notes about the way that you think about this fundamental,  

Andrea    01:22:13    Right? So what, yeah. So I wanted to think, I was asking myself like, what, what’s really behind this, right? So, like I said before, you know, it’s what’s an, so what would it mean if a large language model was an adequate theory of X or Y in language mm-hmm. <affirmative> or in cognitive science, or in neuroscience. And, um, I’m just trying to think about the kinds of answers that you end up with. Um, and again, summarize those answers is like, don’t, maybe this is not doing justice to the range of things out there, but one of the sort of middle of the road thing is to say, oh, yeah, you know, large language models are a good or good enough, uh, model of language processing in the brain, or of language representation in the brain. And I want to know more, like, what do people actually mean by that?  

Andrea    01:22:47    Right? Because they don’t, if you say, well, so do you mean that like the brain is a large language model? They would say, no, and that’s reasonable. Yeah. No. Right? So then what <laugh>, what would it, what does it actually mean? So then I think charitably, um, I think what they mean is that there’s some important or meaningful similarity, uh, between systems that correlate, right? Mm-hmm. And this is again, a little bit what we talked about with the geocentric and heliocentric models, right? They correlate with each other, but they come from fundamentally different, uh, claims about the, the, the nature of the solar system. Um, but again, as Olivia guess, and I have discussed at length in our papers with multiple realizability, the infinite ways to skin a cat that we’ve no agreed to talk about <laugh>, infinite ways to cut a cake. Yeah. This correlation actually doesn’t tell us very much.  

Andrea    01:23:36    Um, maybe, maybe, you know, most charitable we could say, okay, well then there’s gotta be some broad strokes, low dimensional similarity, like I said about the time series, uh, or about the fact that, you know, it has all the language, data, text data ever produced. Yeah. And that language is what is underlying that the human mind humans created those data <laugh> like that, that’s in common. Um, yeah. But it doesn’t tell, it doesn’t for me. That’s not a, that’s not the kind, that’s not an explanation for how language works in the brain. And that’s just a different, it’s a different nature of, of, you know, meaning behind that clean. So that, you know, by, I think this needs to be, uh, analyzed and thought about more carefully. So what, what, you know, if you say large language models or good models of language, what does it actually mean? What are the things that that entails in a logical, uh, interpretation?  

Paul    01:24:27    Hmm. I mean, do you apply the same principle to all of the work and vision with convolutional neural networks being the quote unquote best models of Yes.  

Andrea    01:24:36    Yes, we would. Although I realize, you know, that’s a, that’s a much more, um, there’s alar much larger literature and a larger, um, sort of population and tradition of people working on that. And I don’t, you know,  

Paul    01:24:47    Would you say it’s more rigorous too, because the findings are correlated, uh, to single units, single neurons? You know,  

Andrea    01:24:55    I mean, I think so. I think in the limit, in like, sort of in the logical space, that’s not any different mm-hmm. <affirmative>. But there is something, uh, I would say inherently beautiful about seeing that correlation sort of extend down to every pin of the, you know, of the system, so to speak. And you see it in that way. And it is, like you said, for some behavioral neuroscientists enough to really be able to predict every subsequent data step. Right? And,  

Paul    01:25:18    And Jim DiCarlo would say, and control, like prediction and control is understanding  

Andrea    01:25:23    Yes, for me, that that’s engineering. But that’s, that’s okay. I mean, that’s, that’s what people think, right?  

Paul    01:25:28    E it’s e understanding, engineering understanding or something.  

Andrea    01:25:30    And when we want eye science <laugh>. Yeah. There you go. I, I think, no, I mean, I think, uh, I think it’s complicated. I think these, these debates are really worth having, and I’m very suspicious of myself, um, and others who have extreme exclusionary views where you say, you know, that, you know, right? Well, okay, it’s a, maybe, maybe that’s all we can achieve, right? Is prediction and control, but don’t you want more? Right? I, I just want more. And I think more is possible. It may not be in the, in the form that we expect where we think like, okay. And you know, an explanatory model of making it up sensory motor control, you know, is you have these boxes and you, you know, there’s just different forms, right? Mm-hmm. <affirmative> mm-hmm. <affirmative>  

Paul    01:26:13    Different  

Andrea    01:26:14    Levels and different levels. And I don’t think that we should, you know, every may, you know, I don’t think we then, that the solution is not to say that prediction and control are, is understanding. I think that’s  

Paul    01:26:28    But for you personally, because you, you dive into, uh, potential mechanisms in the brain underlying, you know how I  

Andrea    01:26:34    Try Yeah. <laugh>  

Paul    01:26:35    Well, yeah. I mean that in your theoretical Yes. Approach. I mean, does it matter how the brain do, like, for you personally, to have a satisfactory understanding of how language works in biological cognition in humans? Does it matter how the brain does it?  

Andrea    01:26:51    Yes. Is that a silly question? But I think, but I think there are two aspects to that. So do the particulars of how the brain, does it matter? Yes and no. Right? So I’m, I’m not able to get, I think maybe this is maybe the difference between the, the DeCarlo prediction and control and is understanding, and that’s enough is I had to accept long ago that I would never have perfect prediction or control, right. Use this different state of, you know, of, of organism and, and ethical situation. Yeah. You know, that, that I don’t think that I need to have, I don’t, I don’t wanna model, for example, a model of general relativity where I have to know the position of every, um, Adam in the universe to know whether my model is correct. Sure. That would give me prediction and control, but would give me no understanding of the principles of the system.  

Andrea    01:27:37    But I actually don’t, I mean, I, I would actually say, I mean, I dunno Jim DiCarlo, but I don’t think, I think that they, everyone is actually gaining more than just prediction and, uh, controls understanding. They’re actually doing inference when they write their papers and think about it. They are in their head trying to, uh, glean categories and first principles. They, that’s, that’s what the human mind does. That’s the human urge that underlies all science, and it has to be refined and, and, and honed and carefully carried out to become science. Uh, and, but I think it’s, I mean, I don’t think that, I feel like that’s, that somehow missing out on a core thing. It’s actually not actually accurately describing what people are doing. Even envision science to say that you’re just trying to get prediction and control, and that that’s understanding, people are still trying to derive some first principles based on that. Mm.  

Paul    01:28:25    You know, I, I’ve recently had, um, LA Pavlik on and, um, ga Lian, and there’s a lot of people who are one, just, you know, impressed with the large language models. Mm-hmm. Of course we all are, but also like probing them. Um, and I’m not sure where they land on, you know, whether it’s a good model for biological cognition, but then, you know, they’re, but, but they’re find, like Ellie Public is finding some symbol like representations, right? In the large language models. And so there are kind of being used to, to probe and there, there’s a lot of people excited about large language models as models of biological, uh, uh, cognition.  

Andrea    01:28:59    Well, so can, so can I jump in already? Yeah. So I would say there already, there’s a different approach. So there’s a tradition of having, of instantiating symbols and connections networks that goes back to the, the eighties and nineties, the work of John Humel and Alex doma and Keith Holyoke. Um, and there it’s sort of a different, you know, the idea is like, okay, what, what do we need to use to realize something that’s symbolic in the, in, you know, in, in a neural network? And the key thing, the key sort of insight of, of Alex do’s work is that, you know, what do I need? What is the series of algorithm ALG algorithms or transformations of information in the internal states of a settling neural network that, that I need, you know, to predicate something, right? Mm-hmm. <affirmative> to learn, uh, structured representations from unstructured data, right?  

Andrea    01:29:39    That’s the inside of, of Dora, which I could talk for a long time about <laugh>. Um, but, um, I’ve lost my train of thought. Oh, the symbol. So, so, so the other approach that you’re talking about is sort, well, oh, you know, symbols emerge. And then again, as I talked about at the beginning, then you have to do the hard work of like extensively testing this. So then you have to say, okay, what’s a symbol? What counts as being a truly symbolic representation? Yeah. And to some degree, uh, the, you know, previous work, especially in analogy in, in categories and concepts, literature, and in cognitive science has done this and has established some benchmarks for what that would be. Those are not currently being applied in natural language processing. I think, uh, maybe for historical reasons, or it’s taking a while for people to discover that literature again.  

Andrea    01:30:16    Um, but it’s, I think that’s just a very different approach. And it seems more sort of like open-ended, right? So like when you, you know, what if you just haven’t found again, what is that with the, the black swans and the white swans? You just haven’t <laugh>, you know, you just don’t know whether one day one’s gonna come along, right? Mm-hmm. <affirmative>. So maybe you just haven’t found the right test. And I think that’s a much, I would be much more sort of on shaky ground with that, rather than saying, okay, I’ve understood the principles of what I need to have a neural network, um, uh, realize something that’s functionally symbolic. Yes, it’s important to test that, what the representations are. Do those, do they meet those criteria? But then at least I have a theory about what caused that to occur. Otherwise, looking at emergent behavior, that’s not to deride that at all. I’m sure that has value. Um, you still don’t, in the end, you find those symbolic representations. Well, how did they get there? How do you know? How are they instantiated? Will they always be instantiated the same way? How much are they dependent on the properties of the data that were produced by systems that have symbolic representations? I e humans?  

Paul    01:31:10    Sure. I see I scratched an itch there, <laugh> or maybe caused a, uh, an itch, uh, with this, you know, specific to, to symbols. But I, you know, I was, yes, I was bringing it up as a, which I’m glad, I’m glad you went down that road, but I was, I was just bringing it up because, uh, the question was, you know, where do you land on what the usefulness of like a large language model in, in their current instantiation, are they useless, uh, to think about biological cognition? Like, you know, what do you see their role  

Andrea    01:31:37    As? Yeah, I mean, I think the links to biological cognition are most tenuous. I think the closest link is through the, the, the data which, you know, we don’t always know everything about. And the training regime, which we don’t always know everything about, um, but mostly through the data, because that is produced by the system we’re trying to study. Mm-hmm. But it’s like, through how many filters do you wanna study the system? <laugh>. Okay. So I guess large language models have two aspects that, that people would deem, um, important for studying, right? So they have the behavior that humans have, right? Yeah. So let’s, let’s say that they produce  

Paul    01:32:06    I Im impressive output,  

Andrea    01:32:07    Right? Let’s say that, um, we don’t, I mean that, so that, that’s the one thing that they have to study. They don’t have any of the other constraints that we have. So if we think maybe there’s some, you know, maybe that that is telling us what in the limit is learnable, um, from data that we don’t get exposed to as humans or as children. But I, I mean, I don’t wanna, I don I don’t think it’s useful or productive to sound completely negative about this, but I, I’m really just honestly saying my, my point of view, I just don’t see, uh, the links because they have not been constrained in a rigorous way. Um, I understand the urge to say, oh, yeah, it’s a great model organism for language, but it doesn’t, I don’t know. It’s a bit like saying, you know, a model organism for a mouse would be like some output of like all the mouse behaviors and like a space <laugh> with like, you know, how much of could you know about the, you know, what we know about maybe the mouse is, is too high up, but, but about, I don’t know.  

Andrea    01:33:15    What’s the, what’s the, isn’t it, um, see elegance that we have the full connect and wealth mm-hmm. <affirmative>, of course you can debate, debate about whether that’s actually even the right, uh, readout to be caring about. But I think it’s useful to think about for what other system, if you were to make a large language model of it, <laugh>, would you be satisfied studying that, that object?  

Paul    01:33:35    Well, I think the curiosity is just how well these deep learning models are performing without having been built generally, you know, like with the constraints of, uh,  

Andrea    01:33:46    Organism, but they’re specifically built to do this task, right, that they’re doing, and then they’re exquisitely tuned now with feedback, and they’re given the, basically the prerogative, impress us, give me a response that’s, that can’t be the training data, but that is a really good approximation of it. And of course, it, you know, that it, that it can do that at all is deeply impressive. But without, you know, we don’t know about the leakage. We don’t know about how much, you know, the training and the test data is completely blurred. We don’t know the role of human feedback. We know that there’s so many other things about the models that violate our beliefs about what a tenable system is. They’re, they’re ethically very fraught. I mean, that’s something that, that is not my expertise, but that I’m very aware of that there’s so many, um, ethical issues that come up with these. Yeah. Um, and that they really amplify power structures. There’s all sorts of complicated, important work about that. Um, but yeah, in the, I don’t know. I, I feel like I’m, I’m letting you down, not being an enthusiast, but I’m  

Paul    01:34:45    Just, why, why are you so negative, Andrea? It’s, no, this is great.  

Andrea    01:34:49    Why about manifold’s so negative <laugh>, large  

Paul    01:34:52    Language models and Ella, large language models have zero manifolds,  

Andrea    01:34:56    So Well, so, but that, I mean, you know, that’s a question. No, I  

Paul    01:34:59    Know. Yeah.  

Andrea    01:35:00    I don’t have,  

Paul    01:35:00    They don’t have, you can make anything into a  

Andrea    01:35:02    Manifold though. That’s one of the weaknesses of manifolds and gradients. Anything can be a gradient <laugh>. Right. Whenever I see gradient work, I’m always like, but anything could be gradient, but Yeah. Yeah. No, but I do think that it would be, you know, I thought about doing this. I think, I’m not sure that it’s somewhat, it’s worth it, but that you know, looking at the dynamics of large language males, they’re gonna be very, very different  

Paul    01:35:21    <laugh>. So you don’t feel like, um, so, so you are tempted to sort of probe, look more into large language models for your own negative reasons? Of course.  

Andrea    01:35:29    Yeah. But I don’t, I I’ve tried, you know, Alex and I did some work where we, you know, <laugh>, we, we, uh, made Dora the settling neural network model that can learn, um, predicates, uh, play, um, uh, pong after being trained on breakout and vice versa. And it can do that because it learns relational representations. Mm-hmm. And we did a whole series of stuff where we learned relational representations from the collabor data set and then play these games. And the way that the work just, it was just so difficult to communicate what we were trying to say. There’s such a, I feel like not, not being completely enthusiastic and on board with large language models, it’s already a quite difficult position to be in. Um, and I feel like my own, you know, my own work and my own case is not often, uh, served by Yeah. Being negative, but I also don’t, I can’t, and this is what I think, so I just try to say what I think and, and be, um, kind about it. Um,  

Paul    01:36:19    But do you feel like, uh, that that large language model world is encroaching on your definitely approach? You know? Yeah. Yeah. No,  

Andrea    01:36:27    I feel, I feel like, and I, you know, that the reason why, you know, I feel sort of that I will engage with them because I can’t not, they’re so, they’re everywhere. Wow. And I think they’re loud and, you know, when, when you, with, with, with manuscripts and for my, my students and my team, you know, I have, I have a responsibility and I want to put them in the best position mm-hmm. <affirmative> to grow their talent and go where they can go. And I’m not gonna hold them back, you know, because I <laugh> because of my own beliefs. Right. So I, I, you know, it’s a, it’s a, it’s an, an interactive, uh, process and I, you know, we always tend to end up on the same page. I mean, I’ve never had any like, you know, huge disagreements in my own team. It’s actually quite, I mean, there’s many ways to explain that, but my point is more, I don’t think it sets that, you know, already coming into a polarized debate. Um, and also, you know, I, I don’t know. I think maybe someone who’s different from me, if they were advocating my position might be received differently. I can never know about that. Mm-hmm. Um, so I try to just talk, you know, say what I think, say it in an accessible way, uh, not disparage people’s entire research programs <laugh>. Um, but yet say what I think needs to be said. And it’s not, you know, it’s not easy. I’ll say that. It’s not always clear what the right thing to do is.  

Paul    01:37:36    Yeah. But, so it also makes you more unique that you’re not walking down the path that every seem, everyone seems to be walking down, right? Yeah. In, in terms of, um, mentorships and guiding Yeah. Your students, I mean, is there, there’s value to, um, being unique, right? And, and just not fa not falling into the large language model  

Andrea    01:37:55    Trap, but from my experience, you know, again, I don’t, you know, go back and forth on this. It doesn’t necessarily, um, always go over, well, <laugh>, you know, you don’t always get heard or you get misinterpreted, or sometimes you sure. You know, you just sort of get excluded because it’s not, you know, how do you engage with that? Mm-hmm. <affirmative>, uh, these are all sort of the dynamics of all human activities, science, politics, life. Yeah. Yay. Yeah. Yay. Exactly. But I do feel like it’s important for my students to, um, it says there’s a bit of a learning process in that as well. So, you know, well, how do you defend what you really think? And how do you do it in a way that people will listen to some degree? Uh, how do you do it in a way that doesn’t exclude you completely <laugh> from everybody else? You  

Paul    01:38:32    Change oscillations to neural dynamics is one thing. <laugh>,  

Andrea    01:38:35    You don’t talk about evolution. You talk about appearances. No, I don’t advocate any of this. I’m just, you know, I’m figuring out as like, oh, it’s not, it’s not easy. But for example, like for funders, I’m very aware that, you know, um, talking about language models in the brain probably won’t be seen as timely or contemporary if you don’t have something to say about large language models. Oh,  

Paul    01:38:53    Yeah.  

Andrea    01:38:53    Um, so some can’t  

Paul    01:38:54    Just cast them  

Andrea    01:38:55    Off. No, no. And I feel like that’s also, that’s also, uh, sort of not, it is also a moment where language is in the public’s view a lot more. And so that’s actually good for language sciences Yeah. And for, and for neuroscience in the end too. So it’s a way of, of engaging with that and trying to make the best of it without, you know, compromising one’s principles. Um, and so, yeah, I just, I guess I won’t ever go on the record saying, I think that there’s, like, you know, something excited is gonna change everything. I think, again, for what I’ve said, you know, we, when we, when we say that these, these models are enough, we’re saying that behavior is more important. And it’s not even behavior in the sense of, of human behavior. I don’t even wanna make that equivalence that a certain, you know mm-hmm. <affirmative> sequence processing or grammatical strings is more important that are contextually licensed, or however you can make them more generous in your description, that that’s the most important thing, uh, about a model, not what it explains. And maybe from a prediction and controls enough that’s understanding. Maybe those two things align. Right.  

Paul    01:39:52    All right, Andrea, I’ve taken you far enough and we’ve, um, we’ve walked down a hierarchy of manifolds and we have, we didn’t even get to everything that  

Andrea    01:39:59    We, I know. I feel like we could have done, you know, there are many other manifolds we could have explored,  

Paul    01:40:02    A lot of manifolds. Um, and maybe another time, I’d love to have you back another time, but thank you. I’d love to hear that. Um, I appreciate you, um, figuring all this out for us and continued success doing that. Thank you. And, and thanks for taking the time.  

Andrea    01:40:13    Thank you. Thanks so much for reading my work. It was really, it was really wonderful to hear you talk about it, that you got it right away. It meant so much to me. Thank you.