Show Summary Details

Page of

PRINTED FROM OXFORD HANDBOOKS ONLINE (www.oxfordhandbooks.com). © Oxford University Press, 2018. All Rights Reserved. Under the terms of the licence agreement, an individual user may print out a PDF of a single chapter of a title in Oxford Handbooks Online for personal use (for details see Privacy Policy and Legal Notice).

Subscriber: null; date: 26 September 2018

Electrophysiological Research on Conversation and Discourse Processing

Abstract and Keywords

Research into the electrophysiology of language comprehension has essentially been “speakerless.” This has left three vital aspects of communication—it is social, pragmatic, and dynamic—severely underresearched. This chapter makes a case for the investigation of language users involved in active conversation and describes the problems and possibilities that accompany this choice of subject. It gives an overview of what is currently known about how the social, pragmatic, and dynamic puzzles of communication are solved by the brain, and it describes the well-filled toolbox of language-related event-related brain potential (ERP) components (e.g., Nref, N400, P600) that are at our disposal.

Keywords: Language, communication, conversation, ERPs, pragmatics, N400, P600, Nref

Research into the electrophysiology of language comprehension has essentially been “speakerless.” For more than 30 years, participants in language experiments have read isolated and unconnected sentences, word-by-word, from a computer screen. In the past decade or so, there has been increased, although certainly not overwhelming interest in presenting language stimuli auditorily, together with a subtle shift from studying isolated sentences toward looking at multisentence discourse or even minidialogues. However, in all of these studies, participants are, at best, detached observers of short, unrelated fragments of conversations between people they don’t know and don’t care about. A number of recent studies (e.g., Dumas, Nadel, Soussignan, Martinerie, & Garnero, 2010; Dumas, Chavez, Nadle, & Martinerie, 2012; Stephens, Silbert, & Hasson, 2010; Suda et al., 2010; for an overview, see Babiloni & Astolfi, 2012) took issue with that and started to investigate participants actively taking part in an interaction. Unfortunately, these studies had to restrict themselves to looking at very global measures of contiguity and synchronicity of processing; none of them takes into account what participants say to each other, when they say it, and how they say it.

This state of affairs is not as strange as it seems. For investigating brain activity through the analysis of event-related brain potentials (ERPs), the experimenter needs full control over the experimental stimuli: what they are, how they are realized, and exactly when they appear. As yet, there is no way to get this kind of control over utterances in free-running interactions (Schilbach et al., 2013; see also Van Berkum, 2012). However, as we argue later, there are ways to approximate real conversation while maintaining strict experimental control over content, form, and timing (e.g., the dialogue immersion paradigm; Hoeks, Schoot, Neijmeijer, & Brouwer, in preparation). But, in addition to this rather technical reason, there is also a more historical reason for the relative neglect of conversation as a research topic. Psycholinguistic investigations of language processing started out with (p. 366) the implicit adage that if you want to know more about language processing, you should start with its basic building blocks: words. Hence, psycholinguistics set off with the investigation of written words, often presented in isolation. It was soon recognized, however, that word recognition and meaning activation crucially depend on the sentential context in which the words occur and eventually also that the processing of a sentence can only be properly understood if the larger context of the sentence is taken into account. What we are seeing now is the beginning of a third paradigmatic shift in language processing research, one that carries us from investigating reader’s cognitive responses to a sentence or a text to having participants actively interact in conversation.

Why Active Conversation?

Once a participant enters into an actual conversation, three vital aspects of communication come to the fore that are severely underresearched: (1) communication is social—for instance, speakers and hearers use a variety of communication strategies in order not to damage the public image or “face” each puts forward. (2) Communication is pragmatic—communication is ultimately a means toward a goal, and precisely the appreciation of this goal-directedness makes our interlocutors try to make sense of what we say (or do). This “sense making” can sometimes be quite fast, but it often requires considerable inferencing because the words and sounds uttered by our conversational partners (or other nonverbal signals that they emit) are at best pieces of evidence (cf., Sperber & Wilson, 1986, 1995) or instructions to make us think, do, and feel things. Understanding an utterance, we assume, always requires inferencing, or pragmatic processing, on the basis of the communication at hand and of what is known about the conversational context, including the speaker. And (3) communication is dynamic; it is a process that must be monitored and managed (Clark & Brennan, 1991). Turns must be negotiated and signaled, and feedback is necessary to indicate that the listener understood what the speaker meant to communicate. This “backchannel information” can consist of head nodding, smiling, making small sounds such as “uhuh” or “hmhm,” but can also be fully linguistically expressed and take the form of occasionally completing an utterance of the speaker (with one or few words) or even with brief, but complete turns (“yes,” “that’s odd!,” “incredible!,” “Is that so?”). Leaving out such backchannel information may lead socially sensitive speakers to stop talking altogether (Kraut, Lewis, & Swezey, 1982).

It is important to note that these social, pragmatic, and dynamic dimensions are highly interrelated. For instance, frequency and timing of turn-taking in a conversation is regulated by social factors such as power distance and familiarity between the interlocutors. Also, diverging from socially appropriate turn-taking and backchannel scripts will lead to an increase in pragmatic processing aimed at ascertaining the possible meaning of these transgressions. Furthermore, the “indirectness” of an utterance, which is strongly determined by social rules of engagement (here, a specific set of rules is applicable, which we refer to as “linguistic politeness”), will affect the pragmatic processing necessary to uncover the intended meaning of an utterance. Despite the commonalities among the three dimensions, it is fruitful to distinguish them and study them separately.

It could be argued that there is no real need to study active conversation, with all its intricacies, because essentially every important process and representation will also be present when conversations are overheard. And, indeed, we all are very familiar with the role of “eavesdropper,” if not in real situations, then surely when reading a novel or watching a play or a movie. On each of these occasions, we observe other people interact and converse, and we can readily predict the feelings and mental representations of the protagonists. However, it is also intuitively clear that being an observer is something quite different from being a real actor. This is, of course, especially true for language production, but it is likely to be equally applicable to language comprehension. For one thing, you understand a conversation better if you are a real interactant than if you are merely overhearing (Schober & Clark, 1989). Another important concept is “involvement.” When we watch movies, we are motivated to understand what the characters mean by their utterances and behavior. We can also be emotionally involved, at least to some degree. However, if you are a real actor (instead of an observer), it becomes essential that you understand what the other person is saying because your face is at stake, which is clearly not the case when you, for instance, watch a conversation on television. Lots of things may go wrong during an actual conversation; you might not be able to find a good answer to a question; you stutter; your conversational partner is not very considerate, leading you to experience face loss, which, although often partial and temporary, can lead to (p. 367) several kinds of negatively valenced emotions (especially embarrassment, but also fear, guilt, sadness, etc.). In addition, as an actor, you may not reach your personal goals if you do not understand what the other person is saying and you cannot act on that appropriately. Hence, you will be trying your best to make sense of what the other person wants to communicate. Thus, it seems likely that the processes and representations involved in language use are different in degree and possibly also in type when we actively interact versus when we observe the interactions of others.

For these reasons, we propose that the main object of study for the electrophysiology of language use should be active conversation. We believe it is not enough to look at participants as observers of interaction—although that is certainly a considerable improvement over the word-by-word rapid serial visual presentation (RSVP) method that is still current—because active conversation may involve different processes and different representations. This conclusion is shared by Schilbach et al. (2013), who add: “The state of the art in neuroimaging provides severe limitations to studying free-running interactions using the full range of verbal and nonverbal channels. Most likely successful studies will need to identify and isolate salient communicative subsystems” (Schilbach et al., 2013, p. 405). Moving toward free-running interaction presents a formidable challenge to experimenters, and looking at specific methodological setups focusing on subsystems seems to be a necessary step. One approach to bring conversations under experimental control is through scripting. Hoeks et al. (in preparation) use what they call the dialogue immersion paradigm (DIP) to have participants take part in a conversation. What participants do not know is that the responses they get from their conversational partner are prerecorded, which permits reliable presentation of language stimuli with millisecond precision. Participant’s own contributions are scripted to match with these responses. Although this can hardly be seen as free-running interaction, it does permit the crucial involvement of the participant in what she thinks is a real exchange, in which the participant becomes sensitive to face-related issues, to pragmatics, and to the dynamics of backchannel information and turn-taking—the participant is immersed in the conversation. It is not impossible that advances in ERP recording techniques and innovations in experimental design will lead to interactions that are freer than the very restrictive DIP. However, for now, it seems to be a viable paradigm to study aspects of active conversation.

Why Event-Related Potentials?

The DIP was put forward as a solution to technical and methodological problems that arise when a researcher wants to use ERPs to investigate the neural correlates of language processing. But why would we want to use ERPs in the first place? In a recent overview, Van Berkum (2012) deplores the state of affairs in the electrophysiology of language and paints a gloomy picture of a field with lots of scattered findings and little consensus on what they mean. However, since he wrote his chapter, there have been new developments. Brouwer, Fitz, and Hoeks (2012), for instance, offer a parsimonious account of the two most prominent ERP components, the so-called N400 and P600, which we discuss in more detail later. Furthermore, Brouwer and Hoeks (2013) have combined this account of language-related ERP components into a core anatomical model of language processing. But let us start by describing ERPs and explain how they can inform theories of language processing.

The Electrophysiology of Language Processing

What Are Event-Related Potentials?

The human brain contains billions of interconnected neurons (or nerve cells), which are effectively tiny information processors. Through cooperation, networks of these information processors are able to carry out the complex computations underlying higher cognitive functions, such as navigating the physical and social world and satisfying goals. Electroencephalography (EEG)—the technique used to measure ERPs—provides a method to study these neural networks online (during real-time processing) by making use of the physiological properties of computation in the brain.

Cooperative computation among neurons requires interneuron communication, and, in the brain, this is handled by electrical and chemical signaling. A signal sent between neurons can be either excitatory or inhibitory. If a neuron is excited to a large enough degree, it will send out electrical signals, called action potentials, which, upon leaving the neuron are transformed into chemical signals that either excite or inhibit a next neuron. The excitation or inhibition of neurons is manifested by tiny voltage fluctuations (in the order of microvolts; micro =1 per million). If enough of these voltage fluctuations take place simultaneously and with the (p. 368) same polarity (i.e., positive or negative, relative to a neutral reference point), they can be measured using electrodes placed on the scalp, which is what is referred to as EEG. A single EEG recording typically reflects thousands of neural processes in parallel. Most of these processes may be involved in sensory, cognitive, and motor activities unrelated to language processing. When averaging over numerous similar EEG recordings time-locked to a specific (e.g., a linguistic) stimulus, this background activity is filtered out, and what is left reflects the neurophysiological activity as elicited by the stimulus. This stimulus-related activity is referred to as an event related brain potential or ERP.

A typical ERP signal is a temporal sequence of negative and positive voltage deflections (relative to a prestimulus baseline), which are called components. A single ERP component is taken to index the neural activity underlying a specific computational operation carried out in a specific neuronal ensemble (Luck, 2005; Näätänen & Picton, 1987). Components vary in polarity (i.e., they are more positive or more negative than electrodes chosen as a reference), amplitude, latency, duration, and distribution over the scalp, suggesting that different components reflect distinct functional processes, possibly carried out in distinct cortical regions. This means that if two stimuli are processed differently to some degree, this difference might be apparent in the ERP responses that they produce.

Kutas and Hillyard (1980) were the first to see the significance of ERPs for the study of language processing. They presented subjects with sentences containing a final word that rendered the overall sentence meaning anomalous (“He spread the warm bread with socks”) and contrasted the ERP signals evoked by these sentence-final words with those evoked by nonanomalous ones (“He spread the warm bread with butter”). This contrast revealed that, relative to controls, semantically anomalous sentence-final words produce an increase in the amplitude of the so-called N400 component, a negative deflection in the ERP signal that starts around 200–300 ms post-word onset and peaks at about 400 ms. This N400 effect has been interpreted to reflect increased semantic integrative processing for anomalous relative to nonanomalous sentence-final words. An overwhelming number of studies have since investigated the factors that modulate the presumed semantic processes underlying the N400 component (see Federmeier & Laszlo, 2009; Kutas & Federmeier, 2011, for overviews). The general picture that has emerged from this work is that N400 amplitude not only increases in response to semantically anomalous sentence final words, but is in fact evoked by every content word in a sentence, reflecting how well this word fits with its prior context (Kutas & Hillyard, 1984; Kutas, Lindamood, & Hillyard, 1984).

Van Berkum (2012), in his overview on the electrophysiology of discourse and conversation, focused primarily on the N400 component. However, we believe that we should direct our attention to the other main language component, the P600, which we think is more pertinent to the investigation of communication and higher cognitive processing. To appreciate this shift in focus, we first provide a brief introduction to the “semantic illusion” phenomenon, which has received a considerable amount of interest over the past decade (Bornkessel-Schlesewsky & Schlesewsky, 2008; Brouwer et al., 2012; Chow & Phillips, 2013; Hoeks, Stowe, & Doedens, 2004; Kim & Osterhout, 2005; Kolk, Chwilla, Van Herten, & Oor, 2003; Kuperberg, 2007; Kuperberg, Sitnikova, Caplan, & Holcomb, 2003; Nieuwland & Van Berkum, 2005; Stroud & Phillips, 2012; Van Herten, Kolk, & Chwilla, 2005; Van Herten Chwilla, & Kolk, 2006).

The Semantic Illusion Phenomenon

For a long time, N400 amplitude has been assumed to reflect semantic integration (Brown & Hagoort, 1993; Chwilla, Brown, & Hagoort, 1995; Hagoort, Baggio, & Willems; Hagoort & Van Berkum, 2007; Lotze, Tune, Schlesewsky, & Bornkessel-Schlesewsky, 2011; Osterhout & Holcomb, 1992; Wang, Bastiaansen, Yang, & Hagoort, 2009). On this view, the N400 component indexes the compositional, or combinatory processes involved in integrating the meaning of an incoming word with the semantic representation of its prior context. However, recently, a different perspective on the N400 component, the memory retrieval hypothesis, has gained influence (Kutas & Federmeier, 2000; Van Berkum, 2009). Under the retrieval view, N400 amplitude does not reflect integration (or any other kind of combinatorial processing) but rather the retrieval from long-term memory of the lexical and semantic knowledge associated with a word. Retrieval is facilitated if the (lexical and conceptual) knowledge associated with a word is preactivated by its prior context, leading to a reduction of N400 amplitude. Preactivation occurs through lexical priming (explaining N400 modulations in word pairs), as well as through (p. 369) message-level and script priming from the context (e.g., Ditman & Kuperberg, 2007; Otten & Van Berkum, 2007; St. George, Mannes, & Hoffman, 1994; Van Berkum, Hagoort, & Brown, 1999; Van Berkum, Zwitserlood, Brown, & Hagoort, 2003). According to the memory retrieval account, the N400 effect observed by Kutas and Hillyard (1980) reflects the fact that the conceptual knowledge associated with the nonanomalous sentence-final word butter is preactivated through both lexical priming (butter fits well with the words bread and spreading) and message-level or script priming (for instance, spreading and bread together easily evoke a “breakfast” scenario stored in long-term conceptual memory, in which things like butter, but also jam, eggs, coffee, etc. are very likely to occur), whereas the conceptual knowledge associated with the anomalous word socks is not preactivated. As a consequence of this difference in preactivation, the conceptual knowledge associated with butter is more easily retrieved (because it is preactivated) than that associated with socks.

It has been quite difficult to decide between these competing interpretations of the N400 component. However, a set of studies investigating so-called semantic illusion sentences eventually proved crucial for this decision. Hoeks et al. (2004), for instance, tested the processing of sentences in which two semantically plausible arguments for a verb appear in the wrong order (e.g., “De speer heeft de atleten geworpen”; lit: “The javelin has the athletes thrown”), rendering the overall sentence meaning semantically anomalous. Relative to a sentence in which the arguments appear in canonical order (e.g., “De speer werd door de atleten geworpen”; lit: “The javelin was by the athletes thrown”), the semantic integration and memory retrieval hypotheses make opposite predictions regarding N400 amplitude at the critical verb thrown. The semantic integration hypothesis predicts an N400 effect at the critical word thrown, reflecting increased difficulty in integrating its meaning with the preceding context. The memory retrieval hypothesis, conversely, predicts no N400 effect at the critical verb because, in both the reversal and the control condition, the conceptual knowledge associated with the critical verb thrown should be equally easy to retrieve; in both conditions, thrown is primed lexically through javelin and athletes, as well as through world knowledge (throwing a javelin is a normal thing to do for an athlete). Consistent with the memory retrieval hypothesis, Hoeks et al. observed no N400 effect at the critical verb.

Proponents of the semantic integration hypothesis have argued that the absence of an N400 effect can be explained by assuming that participants made a coherent semantic representation of the sentence without reference to its syntactic structure. That is, they assume that through an independent semantic analysis readers were effectively tricked into a semantic illusion leading them to believe that the stimuli made perfect sense. A few hundred milliseconds later, however, the processor did somehow find out that this interpretation is incorrect. This is reflected in the increase of P600 amplitude, a positive deflection in the ERP signal that, on average, reaches its maximum around 600 ms. In some of these “multistream” models—all of these models include an independent semantic analyzer and an algorithmic/syntactic stream and sometimes additional processing streams as well—the P600 effect is associated with effortful syntactic processing in order to make sense of the utterance (see Gouvea et al., 2010, for an overview of syntactic interpretations of the P600). Other multistream models view the P600 as indexing the resolution of a conflict between syntactic and semantic streams (Bornkessel-Schlesewsky & Schlesewsky, 2008; Kim & Osterhout, 2005; Kolk et al., 2003; Kos, Vosse, van dem Brink, & Hagoort, 2010; Kuperberg, 2007; van de Meerendonk, Kolk, Chwilla, & Vissers, 2009; van de Meerendonk, Kolk, Vissers, & Chwilla, 2010; Van Herten et al., 2005, 2006; Vissers, Chwilla, & Kolk, 2006, 2007; Vissers, Chwilla, Egger, & Chwilla, 2013; Ye and Zhou, 2008). However, in an extensive review, Brouwer et al. (2012) showed that none of the complex multistream models can actually explain all of the relevant data. They argue for a single stream architecture, the retrieval-integration account, which is based on different assumptions of the functional significance of the N400 and the P600.

Brouwer et al. argue that the retrieval account of the N400 component offers the most parsimonious explanation because the absence of an N400 effect in reversal anomalies can easily be explained as lexical and message-level priming; no special semantic stream is necessary (see Chow & Phillips, 2013; Stroud & Phillips, 2012, for similar arguments). However, if the N400 component does not reflect compositional or combinatorial semantic integration processes, then where in the ERP signal do these processes show up? Given that semantic integration is essential for the comprehension system, one would expect it to be reflected in the electrophysiology of language processing. Brouwer et al. suggest that these integrative semantic processes are (p. 370) indexed by the P600 component. They hypothesize that the P600 is not a single component but actually a family of late positivities that reflect the word-by-word construction, reorganization, or updating of a mental representation of what is being communicated—the “MRC.”

An MRC is very similar to a “mental model” (Johnson-Laird, 1983) or “situation model” (Kintsch, 1998; Zwaan & Radvansky, 1998). The difference with these models is only of degree: whereas mental models and situation models represent interpretations of all types of situations, be they linguistic or otherwise, an MRC is a representation of how a person interprets his or her current communicative situation. A communicative situation, such as a conversation, can be seen as containing a number of different aspects: a representation of the current utterance, a representation of the discourse/conversation until now, a representation of the “face” of self and of the other interactants, or a representation of the language user’s private representation of self and others, including their long-term and short-term (conversational) goals. All of these aspects are assumed to be represented in the current MRC of an individual involved in a conversation. And adapting this MRC leads to a increase in P600 amplitude.

Summarizing from the perspective of the retrieval-integration account, the P600 component is modulated by each word in a sentence as the lexical information activated by a word is integrated into the current mental representation, resulting in an updated representation of the input given thus far. The effort involved in this integrative processing is reflected in P600 amplitude; that is, the more effort it takes to arrive at a coherent MRC, the higher the amplitude of the P600 component (see Molinaro, Carreiras & Duñabeitia, 2012, for a clear example of such graded P600 modulations evoked by non-anomalous expressions). An important consequence of this view is that language comprehension proceeds in biphasic N400–P600 cycles. The retrieval of the conceptual information associated with a word modulates the N400 component, and the integration of this information with an unfolding MRC into an updated representation is reflected in P600 amplitude. Importantly, we propose that these two components are responsive to general cognitive processing, and that neither of them is language-specific: every meaningful stimulus will cause the retrieval of those features in long-term memory that are associated with that stimulus, and thus, every meaningful stimulus will evoke an N400. Likewise, creating a new representation of a (linguistic or nonlinguistic) situation, or any change in that representation, is hypothesized to lead to an increase in P600 amplitude.

The Brain Basis of the Retrieval-Integration Account

Retrieval-integration cycles provide a general and parsimonious account of the elicitation patterns of the N400 and the P600 component. This sheds light on the how and the when of comprehension, but not on the where, which is indispensable information for creating and supporting a viable model of language comprehension. Event-related potentials have a poor spatial resolution, which makes it very difficult to pinpoint where an observed ERP signal was generated in the brain. Hemodynamic-based (“blood-based”) neuroimaging techniques, by contrast, such as functional magnetic resonance imaging (fMRI) and positron emission tomography (PET), can be used to localize cognitive processing but have a poor temporal resolution. This might suggest that electrophysiological and hemodynamic methods provide complementary information, which we would ideally like to combine to arrive at a viable neurobiological model of language comprehension. However, due to the fundamentally different nature of electrophysiological and hemodynamic measurements, it is not immediately clear how they can be combined; that is, due to their differences in spatial and temporal resolution, it is often impossible to simply align them for a given experimental paradigm (cf. Lau, Phillips, & Poeppel, 2008).

Electrophysiological Research on Conversation and Discourse ProcessingClick to view larger

Fig. 23.1 Schematic overview of a typical retrieval-integration cycle in the left hemisphere. Depending on whether the linguistic input is spoken or written, an incoming word reaches the left posterior middle temporal gyrus (pMTG) via either the auditory cortex (ac) or the visual cortex (vc), respectively. The pMTG then retrieves the lexical information associated with a word from the association cortices, which generates the N400 component. Via the dorsal pathway (dp) or the ventral pathway (vp), the retrieved information is connected to the left inferior frontal gyrus (IFG), where it is integrated with its prior context (the prior mental representation of what is being communicated; MRC) into an updated representation of what is being communicated (an updated MRC). This integrative processing generates the P600 component. Finally, the updated MRC is connected back to the pMTG via one of the pathways, providing a context for upcoming words, upon which a new retrieval-integration cycle may start.

A more promising strategy, therefore, is to start from the processes assumed to be reflected in the different language-related ERP components and map these onto cortical areas or networks that could host them. On the basis of this “process alignment strategy,” Brouwer and Hoeks (2013) have recently proposed a minimal functional-anatomic mapping of the retrieval-integration account that focuses on the anatomical and computational epicenters (cf. Mesulam, 1990, 1998) or hubs (Buckner et al., 2009) of the language network. Building on several large-scale reviews on the cortical organization of the comprehension system (Bookheimer, 2002; Dronkers, Wilkins, Van Valin, Redfern, & Jaeger, 2004; Friederici, 2002, 2011; Hickok & Poeppel, 2004, 2007; Lau et al., 2008; Shalom & Poeppel, 2008; Turken & Dronkers, 2011; Vigneau et al., 2006), they identified the left posterior middle temporal gyrus (lpMTG; Brodmann area [BA] 21; the area represented in darker grey in Figure 23.1) as (p. 371) an epicenter that mediates the lexical retrieval processes underlying the N400 component. That is, the lpMTG is assumed to mediate the mapping of word form to conceptual representations, which are stored in a distributed manner across the association cortices (cf. Elman, 2004, 2009; Pulvermüller, 1999, 2001; Rogers & McClelland, 2004). As such, the focus of activity reflected in N400 amplitude is assumed to take place in the lpMTG, but the full range of activity reflected in the N400 component also includes the activation of conceptual features in the association cortices. The integrative processes underlying the P600 component, in turn, are assumed to be mediated by the left inferior frontal gyrus (lIFG; BA 44/45/47; the lighter grey area in Figure 23.1), which is a neuroarchitecturally complex, highly parcellated area (Amunts et al., 2010; Amunts & Zilles, 2012; Friederici, 2011). Brouwer and Hoeks suggest that this anatomical parcellation may underlie a fine-grained functional topology of the lIFG (see also Friederici, 2011; Hagoort, 2005), in which different parcels of the area may carry out different subprocesses involved in MRC construction, thereby unifying ongoing debates about the functional role of the lIFG (see Grodzinsky & Santi, 2008; Rogalsky & Hickok, 2011, for overviews). On this view, the mental representation of what is being communicated is maintained in (or its activity is coordinated by) the lIFG. Critically, linking the P600 to the lIFG suggests that characteristically different P600s in terms of electrophysiological properties like onset, duration, amplitude, and scalp distribution, which Brouwer et al. (2012) assume to reflect different subprocesses of MRC construction, may have an anatomical basis in different parcels of the lIFG. Like the lpMTG and the N400, the lIFG is assumed to serve as an epicenter that is host to the focus of activity reflected in P600 amplitude, but the full range of activity underlying the P600 component may include activity from adjacent areas as well.

The retrieval epicenter (lpMTG) and the integration epicenter (lIFG) are wired together by means of two white matter pathways, a dorsal pathway (including the classic arcuate fasciculus and the superior longitudinal fasciculus) and a ventral pathway (including the extreme fiber capsule system, the uncinate fasciculus, and the inferior and medial longitudinal fasciculi), the precise functions of which are still much debated (Baggio and Hagoort, 2011; Friederici, 2009, 2011; Hickok & Poeppel, 2004, 2007; Tyler et al., 2011; Weiller, Musso, Rijntjes, & (p. 372) Saur, 2009). Consequently, it is as yet not possible to choose which of these pathways supports the connection of information retrieved in the lpMTG to the lIFG for integration and which pathway serves to connect the updated MRC back from the lIFG to the lpMTG to provide a context for upcoming words. The presence of these pathways, however, indicates that there are at least two pathways supporting retrieval-integration cycles.

Putting this all together, we can walk through a typical functional-anatomic retrieval-integration cycle. An incoming word reaches the lpMTG via either the auditory cortex (ac; see Figure 23.1) or the visual cortex (vc), depending on whether the linguistic input is spoken or written. The lpMTG then mediates the retrieval of the lexical information associated with this word from the associations cortices, where it is stored in a distributed manner. This process generates the N400 component of the ERP signal, the amplitude of which corresponds to the ease of retrieval. The retrieved lexical information is linked up to the lIFG via either the dorsal (dp) or ventral pathway (vp), where it is integrated with its prior context (the prior MRC) into an updated representation of what is being communicated (an updated MRC). The extent of work required to get this updated representation of what is communicated is reflected in P600 amplitude. Meaning aspects of the updated representation are fed back from the lIFG to the lpMTG via the dorsal or ventral pathway, which provide a context for possible upcoming words on which a new retrieval-integration cycle may start. Figure 23.1 provides a schematic overview of a retrieval-integration cycle.

The functional-anatomic mapping of the retrieval-integration account proposed by Brouwer and Hoeks (2013) is not intended to be a full-fledged neurobiological model of language comprehension but the extendable core of the comprehension system (see Brouwer, 2014, for a proof of concept in terms of neurocomputational simulations). As such, it can serve as a starting point for research into the coupling of brain anatomy and electrophysiology, as a step toward a neurocognitive model of communication.

The Electrophysiology of Discourse and Conversation

Conversation is without doubt the most natural form of human interaction. Some have likened it to a dance, in which two partners together carry out a complex pattern of coordinated movement. However, this metaphor is perhaps a bit too friendly and also a bit misleading. The dance metaphor suggests that the two partners in the interaction are equal and that the interaction is fun. Although that may be true in some cases, it certainly is not in others. If conversation is to be considered a dance, it is one in which, often enough, one of the dancers is dominant and makes the other dancer go somewhere, instead of just giving a good show. But what is more, conversation isn’t fun—it’s serious business, an essential part of human social functioning.

Social Aspects of Communication

As biological beings, we humans are born with a set of goals, the primary goal being procreation (some would formulate it slightly differently: our genes have goals, namely to spread; we are “survival machines” for our genes; cf. Dawkins, 2006). To reach the goal of procreation, evolution has provided us with internal drives that keep us alive long enough to procreate and take care of our offspring. Brain research has suggested that these motivators are implemented in specific areas deep in the brain, such as the hypothalamus and other parts of what researchers have sometimes referred to as the limbic system. These motivators (hunger, thirst, sex drive, aggression, etc.) create a web of distal and proximal goals, hierarchically ordered in subgoals, sub-subgoals, and so forth. To reach these goals, we have to negotiate the physical world, but because we are also social animals, we encounter our conspecifics while trying to satisfy our goals, so we need to negotiate the social world as well. Furthermore, some of our goals lie entirely in the social realm, goals like being recognized and valued. What separates us from more solitary animals is that we are not alone in the world, we have a public, we are “seen,” every moment of the day, if we like it or not. And that is where communication comes in: we can get what we want only by interacting with our fellow human beings because they are everywhere.

The ubiquity of fellow human beings means that there is a pressure to adapt our behavior to certain social rules that have developed to minimize within-group competition while maintaining high chances of individual survival. Being a social animal often means that we have to inhibit our first, impulsive original plans and adapt them to suit the requirements of the social situation. These impulsive behaviors may involve impulses to mate with a certain individual, to grab food that is available, to drink, to hit someone who comes too close or who (p. 373) shows interest in your piece of food. The social rules that have been created to regulate these impulses to minimize conflict are sometimes highly complex and take a long time to learn (you need to be with children only a couple of minutes to appreciate the latter fact). Some of these social rules can be considered a sort of “baseline” and preclude, among other things, showing behavior in public that has to do with bodily functions (belching, farting, scratching, nose picking, grunting, etc., with the exception of eating), but also include rules regarding clothing, posture, hairdo, gait, bodily proximity, duration of eye contact, and more. The majority of social rules, however, seem in place to govern active interaction. The whole set of social rules regulating behavior, which varies from culture to culture, can be called “politeness,” subsuming “linguistic politeness.” Linguistic politeness rules specifically concern what to say and how to say it. Brown and Levinson (1987) argue that the rules for formulating, for instance, requests to another person are not random and haphazard but are centered around “directness.” As a general rule, extreme directness (“bald on record”), as in “Leave this room,” is only allowed under very specific circumstances, such as when there is an emergency or when the power distance between interactants is very high. In all other cases, more indirect utterances are required, in which requests are embedded in lexical material, in which reasons are given and the speaker apologizes (“I am really sorry, but could you please leave the room, we have to clean the floor”).

We have many social goals, such as getting what we want from people (through persuasion). However, one of our most important social goals is what is sometimes called “impression management” (Schlenker, 1980), a concept derived from the notion of “face” introduced by Erving Goffman (1959, 1967)—how to create the desired image of yourself in the public that is currently present. This impression can be said to have an upper limit, which is the representation we desire—that is, how we want to seem (to this end, we also use postures, clothes, and other accessories, such as cars, houses, etc.; see Goffman, 1959)—and a perhaps more general lower limit (that we want to avoid), which is the “neutral” baseline mentioned earlier. According to Goffman (1959), each time we are in a social situation, we choose a “stance” or “role,” for instance, one in which we are very successful in our job, our family is doing well, and we are happy, but also friendly and sympathetic—all of these are of course idealizations of our real mammalian selves. Effectively, we thus have at least two types of interactant representations that have to be taken into account in a given interaction— our own public face and the public face of the interlocutor—but also the two corresponding private representations of self and other at play that may overlap with the public face representations but that are certainly not identical.

Most people are surprisingly good at face management during conversation. This is no small feat because it requires handling at least four different representations (public face of the interactants, plus the private representations including the interactants’ goals), a representation of the current situation, and a set of complex social rules, including the specific and quite intricate rules for conversation. But we all know people who either seem to lack the social knowledge required to take the face of the interactant into account or who simply do not care too much about the consequences that face damage might have, either regarding their own face or the face of others. So there are individual differences in social sensitivity and social knowledge (which will most often be correlated) that can increase the chance of face-threatening situations. Sometimes face damage is the result of clumsiness in you yourself or your fellow interactant. However, it may also be intentional, motivated by aggression, anger, disappointment, and other emotions (cf. Bousfield, 2008; Culpeper, 1996). This illustrates that the rules of social engagement are essentially normative: they can be broken, although there will be consequences.

Despite the importance of the social aspects of language use we just discussed, there have been no electrophysiological studies on how social aspects of conversation are processed. There is only one study by Hoeks, Schoot, Taylor, and Brouwer (in preparation), who looked at participants “overhearing” visually presented minidialogues. One of the protagonists in the minidialogue asks the other person for a favor or asks a mere knowledge question. Social rules of conversation tell that a blunt “No,” without giving reasons or apologizing is not acceptable, even when people know each other really well (in the experiment, participants were told that the miniconversations were between two people who were in a relationship).

Request condition

partner 1: “Could you please put out the garbage tomorrow?”

partner 2 (blunt): “No.”

partner 2 (softened): “No, I’m sorry, I have to leave early”

Knowledge Question condition

partner 1: “Did you know that John will put out the garbage tomorrow?”

partner 2 (blunt): “No.”

partner 2 (softened): “No, I’m sorry, I didn’t know”

(p. 374) Hoeks et al. found a significant positivity for the blunt NO (“No.” with a period), as compared to the softened NO (“No,” with a comma, later followed by an apology) for both requests and knowledge questions. In the request condition, however, the positivity started earlier, had a broader scalp distribution, and was of a larger magnitude than in the knowledge question condition. These positivities were interpreted as P600 effects reflecting the reorganization of the MRC as a result of less than polite linguistic behavior. Part of the processing may have involved the updating of speaker representation—“this person saying ‘no’ is rude, I better be on my guard.”

To what extent this knowledge about the speaker is represented as an integral part of the developing discourse representation, or is relatively separate, is a matter for future research. We do know from fMRI research that the application of social rules most likely involves areas in the frontal cortex (i.e., medial prefrontal cortex, see Barbey & Grafman, 2011). In addition, studies have identified the bilateral temporal parietal junction (i.e., in both hemispheres) as an area that may be specifically involved when reasoning about the intentions of persons in interaction, which makes it a good candidate brain area supporting a separate speaker representation (Bara, Ciaramidaro, Walter, & Adenzato, 2011; Decety & Lamm, 2007). It will have to be established how a representation of speaker fits into the anatomical core network of language processing that we described earlier.

Returning to the experiment on politeness just described, part of the processing difficulty that was found for bluntly saying “No” may in some sense also be pragmatic; what did the person want to communicate by being blunt? That is, why did the person refrain from offering an apology or some kind of a motivation for the refusal? This kind of inference, where the intended meaning of the speaker is recovered, is further discussed in the next section.

Pragmatic Aspects of Communication

Only very few experiments on language comprehension featured real speakers. Generally, participants simply read isolated sentences or unrelated texts from a screen and never have to wonder what the speaker would actually mean by these sentences or texts because, of course, there was no one there to mean something. However, in real life, language is uttered with a goal—speakers want to get a certain message across—and listeners try their best to get at that message. Utterances that are fully unambiguous do not exist. This means that listeners will have to engage in effortful processing to get a grip on what the speaker meant to communicate. This processing can stop once the listener manages to create a coherent representation of the conversation so far. Coherence in conversations is achieved by three means: (1) establishing reference, (2) using information structural cues to identify the new information and how and where it fits in the larger discourse, and (3) using inference to establish what the speaker meant to communicate or when there are problems with (1) or (2). Precisely because we know that a speaker has a goal and therefore wants to be coherent, listeners operate on the basis of the evidence presented in the utterance, on the basis of the representation of the situation, and on the basis of the knowledge of the speaker (including what is known or might be surmised about his or her goals) to figure out what is meant.

Establishing Coherence: Reference

Every time a noun phrase (the baker), a name (John), or a pronoun (he) is encountered, the listener must determine the entity this expression refers to: its antecedent. Retrieving the conceptual features associated with this referring expression seems to be a prerequisite for finding the right antecedent. Full nouns (e.g., “baker”) will require a more extensive retrieval process than either names or pronouns because these latter are rather “knowledge lean.” For instance, in isolated sentences such as “John hit Mary,” names such as “John” and “Mary” contain very little information. Similar to pronouns, such names will hardly activate more features than those regarding number (singular vs. plural) and gender (grammatical and/or biological). However, if the name refers to a known figure, either from reality (“Bram Stoker”) or from fiction (“Count Dracula”), there is more information about this person in long-term memory that can be retrieved. Note that these retrieval processes can be expected to be especially extensive when nouns or “real” names are mentioned in a discourse for the first time; retrieval effort will be much reduced for repeated mentions. After this meaning activation, the processor can look for a matching antecedent.

(p. 375) Van Berkum, Brown, and Hagoort (1999) were the first to look at the electrophysiological correlates of processing reference. They presented participants with sentences that contained a singular definite noun phrase (NP), like “The dean told the lecturer…,” embedded in a story that had either introduced a single referent (one lecturer) or two equally plausible referents (two lecturers). Relative to the one-referent condition, the critical NP in the ambiguous, two-referents condition was found to produce a sustained negativity that was broadly distributed over the scalp but with a distinct frontal focus starting at about 200 ms post word onset. The increase in the amplitude of this sustained negativity has been labeled the Nref effect and has been taken to reflect the increased difficulty of establishing reference when there are multiple possible antecedents. This interpretation is supported by the finding that the Nref effect disappears if the ambiguity is eliminated before the critical NP, for instance, by having one of the referents leave the scene (Nieuwland, Otten, & Van Berkum, 2007).

Van Berkum (2009) suggests that the Nref effect is not a response to an anomaly, but “the brain’s natural inclination to immediately relate every shred of new information to what is known already” (p. 287). In line with that, we would like to hypothesize that each referring expression elicits an Nref, a negative deflection of the ERP signal that reflects the search for the right antecedent. We need more research to provide a detailed view of the Nref as a component, but we already know a number of things about the Nref as an effect (see Van Berkum, Koornneef, Otten, & Nieuwland, 2007, for review). For one, it is often broadly distributed over the scalp, although quite pronounced frontally. Also, the Nref effect often appears as a sustained negativity, although, as we argue below, it can sometimes be very difficult to judge whether we are dealing with a sustained effect or merely a shifted waveform that did not return to baseline. Furthermore, the extent of the Nref effect may depend on the ease with which the reference problem can be resolved. In the case of a global ambiguity (e.g., “Robert and Daniel were talking, and he…”), the processor may have a hard time finding the right antecedent for “he,” leading to sustained processing difficulty; in other cases, the negative shift may appear to be more “phasic.” And, finally, as far as latency is concerned, it has been claimed that the Nref appears no earlier than 300–400 ms post onset, but the difference waves presented in Van Berkum et al. (1999; p. 157, fig. 1b) suggest that the original Nref effect already starts around 200 ms after the onset of the ambiguous word. It is highly likely that the Nref as a component starts even earlier.

It seems plausible that the processor first activates salient knowledge about a given referring expression before being able to find the most plausible antecedent. As we argued above, this activation of relevant knowledge will be reflected in the amplitude of the N400 and referent resolution in the Nref. If we compare the latencies of the ERP components involved (i.e., the N400 for meaning activation and the Nref for establishing reference), these two processes seem, for a large part, to be running in parallel. Reference assignment thus does not seem to wait for meaning activation to have completed. This makes it difficult to separate the two components (and even raises the question whether they are all that different, except for the representational system—conceptual memory or current discourse representation—that is consulted).

Summarizing, we assume that as soon as referring expressions are encountered, their lexical semantic features will be activated, giving rise to an N400, and the right antecedent will be sought, giving rise to an Nref. We do expect considerable differences between nouns, names, and pronouns. As we pointed out earlier, nouns (and “real” names) will require the most extensive retrieval from long-term memory and will thus elicit the largest N400 amplitude. The associated Nref is most probably running in parallel with or slightly behind the N400 and will thus overlap with it. In contrast, names and pronouns have relatively little lexical and semantic information associated with them, so the retrieval stage will probably be rather restricted; thus, the processing of names and pronouns will most likely be most pronounced in the Nref component. These assumptions mesh with the retrieval-integration account of language processing presented earlier. The retrieval-integration account has identified two core systems that are both functionally and anatomically distinct: the semantic memory system containing all lexical and world knowledge, with a hub in the temporal lobe; and a working memory system involved in creating, adapting, and maintaining a representation of what is currently communicated, with a hub in the inferior frontal lobe. It makes sense that in the interpretation of nouns, both of these representations are accessed to a certain degree. The same is likely true for the interpretation of names and pronouns but, in this case, the balance is skewed toward accessing the situation model/MRC. We propose that searching the (p. 376) situation model leads to an Nref, whereas searching the semantic memory system leads to an N400. This is in line with Kotchoubey’s (2006) account of the relation between component polarity and cognitive processing. On that account, negative shifts (N400, Nref) reflect preparatory processing and search, whereas positive shifts (P600) reflect combinatory processing and synthesis. An fMRI study by Nieuwland, Petersson, and Van Berkum (2007) showing that referential processing leads to activation of frontal brain areas provides further support of this hypothesis. So, for content nouns (and “real” names), we expect both N400 and Nref; for pronouns and “empty” names, we expect an Nref and hardly any modulation of the N400. This is consistent with findings from a recent study by Taylor and colleagues (Taylor, 2013; Taylor, Stowe, Redeker & Hoeks, in preparation). They presented participants with spoken minidialogues like the following:

A: “Lisa lost the chess tournament”

B: “Oh?”

A: “The teacher congratulated her.”

Compared to dialogues in which Lisa won the tournament, there was no N400 effect on the pronoun, even though its antecedent was unexpected, because it is strange to congratulate Lisa if she just lost a tournament; a P600 effect was found instead.

To be clear, there have been reports of “N400 effects” for names and pronouns. Indeed, there is a comprehensive literature concerned with the so-called repeated name penalty. Ledoux, Gordon, Camblin, and Swaab (2007), for instance, found longer reading times for a repeated name in an eye-tracking study using sentences such as: “At the office, Daniel moved the cabinet because Daniel needed room for the desk.” Repeating the name (Daniel) to refer to the same antecedent is infelicitous and leads to processing difficulty (hence the “penalty”). In a replication of this experiment with ERPs reported in the same paper, Ledoux et al. found a negative shift for the repeated name as compared to a felicitous control sentence (“At the office, Daniel and Amanda moved the cabinet, because Daniel needed room for the desk”). They interpret this negativity as an N400 effect, but, in the light of our previous discussion, it is more likely to have been an Nref effect. Indeed, a close look at the data from Ledoux et al. reveals that the negativity is broadly distributed over the scalp and clearly visible at frontal electrodes, a pattern that is very similar to the repeated name penalty effect found in an earlier ERP experiment (Swaab, Camblin, & Gordon, 2004). The auditory version of the same experiment also produced a negative shift that was actually largest at frontal electrodes (Camblin, Ledoux, Boudewyn, Gordon, & Swaab, 2007). Thus, it seems that if a referring expression is “marked” in some way (using the full form “Daniel” when a reduced form, the pronoun “he,” is called for), it is more difficult to find the correct referent, leading to an increase in Nref amplitude. Interestingly, the Camblin et al. data show that if the name is not a marked form, as in the felicitous control sentence, the negativity is actually reduced as compared to a control sentence, indicating that finding the antecedent has become easier. This repetition effect on the Nref mirrors the well-known repetition effect on the N400 (Kutas & Federmeier, 2011).

Whether we can categorize a given effect as an Nref or N400 of course depends crucially on the estimated onset and scalp distribution observed. Unfortunately, though, onset and scalp distribution are also known to vary with materials used, mode of delivery (auditory versus visual), and the individual differences in functional and anatomical organization of language in the participants. We return to this issue in the discussion section because it is a general problem that affects the interpretation of other components and effects as well.

Until now, we have discussed cases in which the antecedents are present in the discourse model/MRC of the listener. When there is not yet an antecedent, the listener must create a brand new discourse entity. In case of definite referring expressions, the definiteness acts as a signal to listeners that they are supposed to have an antecedent for this expression (i.e., the “definiteness presupposition”; Russell, 1905). If that is not the case, listeners may have to accommodate the antecedent in the current discourse representation. Accommodation of discourse entities has been shown to lead to P600 effects. For instance, Burkhardt (2006) compared the following discourse fragments:

Tobias was having a chat with Nina. He said that the conductor was…

Tobias visited a conductor in Berlin. He said that the conductor was…

Only in the second fragment is the definite expression “the conductor” from the second sentence properly introduced in the first sentence, namely as an indefinite NP (“a conductor”). In the first discourse fragment, the conductor from (p. 377) the second sentence is not introduced, leading to a P600 effect, presumably due to accommodation.

Sometimes, the antecedent for a referring expression can only be identified through so-called reference transfer (cf., Nunberg, 1979). Schumacher (2011) examined the processing of mini-discourses in which the second sentence contains a semantic anomaly (e.g., “The doctor asks his assistant again who called that early. The assistant responds that the hepatitis had called that early”). In its literal sense, the critical NP “the hepatitis” is a poor agent for the verb “call,” rendering the sentence anomalous. However, this anomaly disappears if the reference of “the hepatitis” is transferred from its literal meaning (e.g., liver inflammation) to a referent that is contextually associated with it (e.g., a patient suffering from liver inflammation). Relative to a minidiscourse in which no transfer of reference was required (e.g., “The doctor asks his assistant again who called that early. The assistant responds that the therapist had called that early”), Schumacher found that the critical NP “the hepatitis” produced a P600 effect, which she interpreted as reflecting enriched composition.

There is also the possibility that antecedents are present, but none of them is available for the referring expression that is used. For instance, Van Berkum et al. (2007) review a number of studies looking at sentences such as “Anna shot at Linda as he…,” where there is no matching antecedent. Compared to the control sentence “David shot at Linda as he…,” the failure to find an antecedent led to a P600 effect. Thus, the failure to find a matching referent leads to immediate interpretation problems.

In conclusion then, we propose that each referring expression elicits an Nref from searching the discourse representation or MRC. Marked delivery, for instance by using a full NP where a pronoun is required, will lead to an increase in the Nref amplitude, reflecting the extra effort needed to find the correct antecedent. In addition to an Nref, all referring expressions will elicit an N400 due to the activation of lexical semantic information—contained in the conceptual memory system—although this is most pronounced for nouns and “real” names. If establishing reference turns out to be impossible or leads to an implausible interpretation, a P600 will ensue, reflecting the reorganization of the MRC. A P600 may also appear in the absence of serious problems like those mentioned above (i.e., persistent reference failure or semantic anomaly), namely, when a discourse entity needs to be accommodated or when the referring expression needs some “preprocessing” before the antecedent can be successfully identified.

Establishing Coherence: Information Structure

In discourse, and especially in conversation, coherence is achieved not only through referential links, but also via the information structure of an utterance, and more specifically through the information structural dependencies between an utterance and its preceding context. Information structure can be broadly described as the way the information contained in an utterance is segmented (cf. Chafe, 1979). In general terms, the informational content of an utterance can be divided into information that is backgrounded, also termed topic or “given,” and information that is in focus, also termed “comment” or “new.” The way in which the informational status of these different parts of the message is signaled varies per language. For instance, in languages like English and Dutch, prosody plays an important role in the marking of topic and focus: the topic is deaccented, and the focus part of the message receives a pitch accent. Other languages, such as Italian and French, predominantly use more structural means, such as clefting and word-order variations to indicate focus structure (Lambrecht, 1994). Marking the information structure of an utterance enables the listener to know which aspect of the message is meant as the “real” information and what part of the current discourse representation should serve as the “attachment site” for this information.

The importance of information structure becomes apparent if information structure cues are misplaced. Quite a few studies have looked at what happens if backgrounded information is treated as new or new information is treated as if it were already known (e.g., Bornkessel et al., 2003; Cowles, Kluender, Kutas, & Polinsky, 2007; Dimitrova, Stowe, Redeker, & Hoeks, 2012; Heim & Alter, 2006; Hruska & Alter, 2004; Hruska, Alter, Steinhauser, & Steube, 2001; Li, Hagoort, & Yang, 2008, Li, Yang, & Hagoort, 2008; Magne, Astésano, Lacheret-Dujour, Morel, Alter, & Besson, 2005; Toepel & Alter, 2004; Toepel, Pannekamp, & Alter, 2007; Wang, Hagoort, & Yang, 2009; Wang et al., 2011). For instance, Dimitrova, Stowe, Redeker, and Hoeks (2012) looked at spoken minidialogues in which, in some conditions, the topic of the target sentence sometimes received a pitch accent (superfluous accent), whereas in other conditions, the focus element sometimes remained unaccented (missing accent). (p. 378)

Q: Did the club give a bonus or a fine to the player?

#A: The club gave a bonus to the PLAYER. (missing accent on bonus)

A: The club gave a BONUS to the player.

Q: Did the club give a bonus to the player or to the trainer?

#A: The club gave a BONUS to the player. (superfluous accent on bonus)

A: The club gave a bonus to the PLAYER.

Both missing and superfluous accents were shown to lead to P600 effects, indicating problems in creating a coherent MRC. In addition, the superfluous accent condition also showed a long-lasting right-lateralized negativity. Importantly, in this study, and also other studies reviewed here, treating background information as new was operationalized by adding a pitch accent to the noun of the topical NP (here: “a bonus”). As we have seen in the studies on the repeated name penalty discussed earlier, marked delivery in the form of an overspecification (“Daniel” instead of “he”) leads to increased search in the situation model, causing an increase in Nref amplitude. Accenting a noun that should not be accented according to information structure rules could be seen as a marked delivery as well. This could explain why Dimitrova et al., but also other studies, found a negativity. Dimitrova et al. suggested that this negativity may have been an N400 effect, but we would like to entertain the possibility that the negativity was not an N400 effect, but an Nref effect, instigated by the marked delivery of the referring expression. If we look at the relevant waveforms in Dimitrova et al. (2012, fig. 5), we can see that the negativity starts around 200 ms post onset and goes on for quite some time (at some electrodes even until after 1,000 ms post onset) and is also present at frontal electrodes.

It is important to note that, in contrast to many of the earlier studies, Dimitrova et al. did not use a prosodic judgment task precisely because they hypothesized that such a task might interfere with the normal processing of the minidialogues. They were concerned that, instead of focusing on the meaning of the exchange, participants’ attention would be directed to categorizing the stimulus and making a decision as to whether it is prosodically and contextually well-formed. These kinds of task-induced cognitive processes involve controlled processing, decision making, and keeping the decision in memory until it is time to respond, plus the associated motor preparation for pushing the button. Thus, the actual comprehension processes of spoken minidialogues will be confounded with decision-related processing (see Magne et al., 2005, for similar reasoning).

If the application of a superfluous pitch accent on a referring expression should be seen as a form of marked delivery, giving rise to Nref effects rather than N400 effects, this should hold for accented nouns (see Dimitrova et al., 2012, for an overview) as well as for accented names and pronouns. The experiment by Taylor et al. (in preparation; Taylor, 2013) that we discussed in the section on reference included conditions in which the critical pronoun was accented.

A: “Lisa won the chess tournament”

B: “Oh?”

A: “The teacher congratulated HER.”

Taylor et al. found that accenting the pronoun in this case, as compared to the condition in which the pronoun was unaccented, led to a prolonged negativity with a frontal maximum (especially on the right). This is very likely also an Nref effect, indicating the increased effort expended to find the correct referent. So, in general, it seems that a mismatch between the expected and actual information structure leads to increases in P600 amplitude; in addition, superfluous marking gives rise to an increase in Nref amplitude.

An example of an information structural mismatch that does not involve marked delivery is a study by Hoeks, Stowe, Hendriks, and Brouwer (2013). They have recently looked at violations of projected information structure in question–answer pairs.

Q1: What happened?

A: The producer fired the actor.

Q2: What did the producer and the director do?

#A: The producer fired the actor.

In the context of question 1, answer A is perfectly fine. However, question 2 sets up an expectation of both the producer and the director to become topics in the answer, either together (e.g., “They went shopping”) or separately, as contrastive topics (“The producer fired the actor, and the director hired him again”). This expectation of contrastive topics is not met by the answer. At “actor,” Hoeks et al. found a positivity that started quite early (350 ms post onset), had a broad distribution, and became more frontal as time elapsed (from (p. 379) 600 ms onward). Despite this early onset, they interpreted this positivity as a P600 due to MRC reorganization to create a coherent representation, for instance, one including the computation of the pragmatic implication that the director did not do anything because he was powerless or uninterested. This latter kind of processing or inference is the third source of coherence in the mental representation of what is communicated.

Establishing Coherence: Inference

Speakers strive to deliver a coherent message and listeners to create a coherent representation from the input they get. The links between referring expressions and their antecedents provide a scaffolding for putting into place the new information that is conveyed by an utterance. Cues in the input that indicate where to find topic and focus are extremely helpful in that process. All the holes that are left in the representation are then filled by inferencing, to get at what the speaker might have wanted to say. A specific type of inference only occurs in conversational settings, and these are called conversational implicatures (Grice, 1975). These arise, for instance, when an answer does not match the requirements of the question regarding relevance (i.e., the answer should be relevant), quantity (i.e., the answer should contain just enough information, not more, not less), quality (i.e., the answer should be truthful), and manner (i.e., the answer should be clear and orderly) (see also Sperber & Wilson, 1986, 1995, who propose that “relevance” is the overarching concept). The listener will use inferencing to solve the problems that result from a mismatch, including the computation of the real implicature, or “What did the speaker mean to communicate by creating this mismatch in the first place?” The study by Hoeks et al. (2013) mentioned in the previous section can be considered an example of an exchange in which a conversational implicature could have been computed. In the mismatch condition, the answer violated the quantity requirements, possibly giving rise to inferencing about its true meaning.

Another type of conversational implicature can be seen in ironic utterances. Regel, Gunter, and Friederici (2011) looked at short texts presenting a mismatch between what the protagonist said and what he believed was actually true (i.e., a mismatch regarding quality requirements; see Spotorno, Cheylus, Van der Henst & Noveck, 2013, for a similar experiment). They presented participants with story fragments, for example, about someone hearing many mistakes in the performance of a Bach sonata. The person in question would then look at the orchestra playing the sonata and say to his or her conversational partner: “These artists are gifted.” In the control condition, the person would be listening in ecstasy and again say “These artists are gifted,” but this time be sincere. Regel et al. found that in the irony condition the critical verb produced a P600 effect relative to the control condition. This increase in P600 amplitude reflects that the hearer has to compute the ironic meaning to make his interpretation become meaningful. That is, in the irony condition, where the sonata is played with a lot of mistakes, the literal interpretation of the sentence “These artists are gifted” is at odds with the situation and does therefore not lead to an overall coherent and meaningful interpretation of the speaker’s utterance.

These two types of inference can be categorized under the particularized conversational implicatures because they critically depend on the specific context in which they occur. The other category is the one of generalized conversational implicatures that are more independent of specific aspects of the situation. One prime example of this second category is the so-called scalar implicature. This term refers to the speaker’s scaling of an utterance by using a weaker, less informative term on a scale (e.g., “some”) instead of a stronger, more informative one (e.g., “all”) to communicate that none of the stronger, more informative quantifications on that scale holds. It is not always clear what exactly speakers would like to communicate by using scalar implicatures. For instance, one can say “Some students are hard working” to mean that not all students are hard working. However, it would have been just as easy to use the utterance: “Not all students are hard working.” Maybe the use of scalar implicatures is related to the use of irony, in which the speaker says things he does not believe himself to draw attention to his opinion.

Certain types of scalar implicatures induce world knowledge violations, as in “Some people have lungs” (Nieuwland, Ditman, & Kuperberg, 2010), or “Some turtles have shells” (Noveck & Posada, 2003) because it can be assumed that all people have lungs and all turtles have shells. Under the retrieval-integration account, both the calculation of a scalar implicature and the possibly resulting contextual or world knowledge violations should lead to reorganization of the MRC and thus cause an increase in P600 amplitude. However, neither of the two studies mentioned above reported any statistics for the P600 window, so we cannot evaluate the MRC hypothesis on the basis of these results. In (p. 380) addition, Nieuwland et al. (2010) and Noveck and Posada (2003) found contradictory results regarding N400 amplitude, but because their materials were not matched on Cloze probability (either within or between experiments), these findings may not be very informative.

A recent study using sentence–picture verification did report analyses in the P600 time window (Hunt III, Politzer-Ahles, Gibson, Minai, & Fiorentino, 2012). Hunt III et al. used sentences such as “The student has cut some of the brownies” accompanied by pictures where (a) none of the brownies had been cut (false), (b) all of the brownies had been cut (underinformative), or (c) some of the brownies had been cut. They found a P600 for both false and underinformative conditions relative to the true condition, just as predicted by the MRC hypothesis. In addition, they found modulation of the N400, which matched the Cloze probabilities for the different sentence–picture pairings (e.g., when hearing “the student sliced some of the…,” a picture in which some brownies are sliced will preactivate “brownies,” a picture without any sliced brownie will not, and a picture in which all brownies are sliced will be somewhere in between).

A final category of inferences are the well-known bridging inferences. Bridging inferences are said to occur when information from the current sentence has to be integrated with the previous context, as in “Horace got some picnic supplies out of the car. The beer was warm” (Clark & Haviland, 1977). Here, readers must infer that beer was included in the picnic supplies, otherwise they would not be able to understand the meaning of the two sentences. In terms of ERP effects, we would predict that the NP “the beer” would evoke an increased Nref because “the beer” is a definite expression that does not yet have an antecedent; also, an N400 will be evoked because semantic features of beer must be activated; and, in addition, a P600 will ensue, both as a consequence of the introduction of a new discourse entity and as a reflection of the inference processes that eventually led to the assumption of beer being part of the picnic supplies. To our knowledge, this specific kind of bridging inference has not yet been investigated with ERPs, but there is a related experiment by Burkhardt (2007) that may be pertinent to this question. Burkhardt looked at minidiscourses such as

Yesterday a Ph.D. student was {shot / killed / found dead} downtown.

The press reported that the pistol

By using the verb “shot” in the preceding context sentence, the definite NP “the pistol” is not a completely new discourse entity because it has already been implied as an instrument of shot. The formulation “found dead” by contrast, does not provide any such introduction because it does not imply an action or an instrument for that action, so it must be inferred that the pistol had something to do with the student’s death. The verb “killed” seems to be somewhere in between because it indicates that the student was murdered, but doesn’t reveal the precise instrument. The P600 effects that were found are consistent with this interpretation: a P600 effect was found for “found dead,” as well as for “killed,” relative to “shot.” Burkhardt did not find effects on the N400 or report effects on any other component. This may mean that the search for an antecedent and the preactivation of the word “pistol” was more or less the same for the three conditions.

In summary, then, listeners strive to have a coherent representation of what is communicated. All three central ways to achieve coherence: establishing reference, creating information structural links, and inferencing, can be investigated with ERPs. Until now, participant were made—at best—to eavesdrop on small and relatively uninvolving texts or, in a few cases, conversations. The presence of a real speaker may change the relative importance of the three cohesive devices but also change the processing more radically as more representations come into play, among which is the representation of the speaker: who is he or she, and, by extension, what does he or she mean by articulating a given utterance? Some researchers have been skeptical about the importance of the representation of our conversational partners for language use (see, e.g., Barr, this volume; Keysar, 2007). They claim that language use, especially language production, is first and foremost rather egocentric and automatic and that any audience design that people manage to do eventually is effortful and is the first to go when task demands increase. This may to some extent be true, but it is of course obvious that we do adapt to the person that is before us because our personal and social goals in each situation differ with the person (or persons) we are encountering: is it our boss, our child, our husband or wife, our colleague, our friend, our neighbor, a stranger at a bus stop? Every single person and every encounter will be “assessed” in terms of your personal and social goals at that time, in relation to what you know about them, ranging from superficially visible or inferable characteristics to memories of shared events. All these (p. 381) example persons stand in different (hierarchical) social relations to us and also differ in familiarity. What we want from our boss is different from what we want from our children or our students. And that is also different from what we want from our colleagues when we meet them at the coffee machine. So, in what we say and how we say it, we adapt constantly, although on a lower linguistic level—we may be less proficient in tailoring our contributions and frequently resort to automatized routines. It is very likely that the same holds for language comprehension: interpretation is guided by what we know about our interlocutors. A nice illustration of that fact is seen in a study by Van Berkum, Van den Brink, Tesink, Kos, and Hagoort (2008). Participants listening to sentences such as “I have a large tattoo on my back” spoken with an upper class accent showed an increased N400 amplitude to the critical word “tattoo” as compared to the same sentence uttered in a low-class accent. This finding shows that listeners take the characteristics of the speaker into account and that they are able to do so very quickly. So, we would like to conclude that speakers matter, although perhaps not always on all levels of language use.

Dynamic Aspects of Communication

When the speaker is present, the language situation changes from static to dynamic. We have to negotiate turns and the frequency of turns. We have to establish whether it is allowed to interrupt the speaker or whether we let ourselves be interrupted. We have to give and receive backchannel information and decide on the sort of feedback and the timing of it. Kraut et al. (1982) showed that feedback serves to regulate the content of speech. It also has an effect on more structural aspects, such as the form of referring expressions, the occurrence of ellipsis, and the like. But backchannel information also indicates that the listener is interested and committed to engaging in the conversation. Furthermore, feedback reflects the state of the relationship between speaker and listener: smiles and friendly nods signal a positive relation. Unfortunately, no single study on electrophysiology of language seems to have looked at these aspects of human interaction. The DIP may play a role in filling this gap.

Conclusion

With ERPs we have a valid and reliable toolkit at our disposal to study language use in discourse and conversation. We have identified three components that are linked to three basic linguistic operations: establishing reference, meaning activation, and integration. The Nref is sensitive to referential processing—searching for the antecedent in the discourse representation or MRC. The N400 is sensitive to preactivation and reflects the retrieval of semantic (and lexical) features of incoming stimuli. And the P600 reflects the construction or revision of an MRC. In the meantime, there are still some problems that need to be solved. For instance, one important problem is how we can design experimental methods that allow us to measure ERPs in active conversation. A first step may be the DIP, in which participants are under the illusion that they are talking via an intercom to a person in another room, although the utterances of this person are in fact prerecorded (Hoeks et al., 2013). To make this work, the participant’s own contributions are also scripted, so that they match the responses of their conversational partner. The DIP provides a means, be it a rather restrictive one, to study the social, pragmatic, and dynamic aspects of communication that come into play when participants are engaged in an actual conversation.

Another issue is a bit more technical. Until now, much research has focused on the N400 and how it can be modulated by the context. In this chapter, we argued that the P600 is the most interesting component for the investigation of discourse and dialogue. However, the processes reflected in N400 and P600 amplitude may overlap in time. Because ERPs are additive, this means that the amplitude of any P600 depends on the amplitude of the preceding N400. In other words, a specific P600 waveform of a given magnitude will turn out more positive if for any reason the preceding N400 is more positive, and P600 amplitude will turn out less positive if the preceding N400 is less positive. Thus, P600 amplitude is modulated by N400 amplitude, and possible P600 effects may become obfuscated. As a consequence, we may conclude that a given contrast between an experimental and a control condition only produces an N400 effect and no P600 effect, whereas, in reality, there is a difference in P600 amplitude. This issue has been noted before in the literature (e.g., Hagoort, 2003; Kutas, Van Petten, & Kluender, 2006) but has of yet not led to a change in how ERPs are analyzed. One reason for this is probably that the P600 has often been seen as an effect that is only generated when something goes wrong during processing and not as a component that is always present in the ERP signal. Conversely, component overlap is, in fact, the reason that the P600 has been taken to be an effect because its overlap with the (p. 382) N400 component fosters the impression that it is sometimes present (e.g., no preceding N400 effect or rather large P600 effect) and sometimes absent (e.g., large N400 effect or rather small P600 effect). It is not easy to find a good way to estimate “true” P600 amplitude that takes into account preceding differences in the ERP signal. One approach to this would be to baseline on the N400 time window (cf. Hagoort, 2003). If we can find a way to adjust for such prior differences between conditions, we might explain why a P600 effect sometimes appears to be absent where it is in fact expected (e.g., see Van Petten & Luka, 2012).

Not only component overlap, but also other differences in the ERP signal preceding the target word are to be reckoned with, especially where utterances appear in a context, as is the typical case in discourse and conversation. Every change in the context may affect the expectations that listeners have about the information contained in the upcoming utterances. In addition, in spoken language, especially when languages are used in which information structure is indicated by means of prosody, prosodic cues are sometimes available well in advance of the target word. For instance, prosodic marking of topic and focus is not strictly local, but has an effect on the entire prosodic contour of an utterance. This is also a problem that we need to take seriously and one for we which we do not yet have a satisfactory solution. Further research in this area is definitely necessary.

Addressing these issues is critical for the successful investigation of the internal functional structure of the P600. Brouwer et al. (2012) suggested that the P600 may not reflect one single process, but may subsume many different subprocesses, which can be differentiated and labeled on the basis of differences in onset, duration, and scalp distribution. In the model proposed by Brouwer and Hoeks (2013), the P600 component is generated in the left lIFG (BA 44/45/47). They argued that the complex neuroarchitecture of this area supports a fine-grained functional topology, hosting different subprocesses of MRC construction and reorganization. As such, different P600s in terms of scalp distribution and other electrophysiological properties like onset, duration, and amplitude arise because they are generated in different, but potentially overlapping subareas of the lIFG. However, the functional and anatomical neurocognitive organization that underlies the scalp distribution of an effect is different for each individual brain. Thus, the derivation of a sound functional categorization of P600 is not as straightforward as it might seem. Possibly, this issue may be circumvented by comparing anatomy and ERP components on an individual, within-participant level and not (only) by looking at between-participants data.

Once we know more about what the different P600 subtypes mean (and by which brain areas they are subserved), we may start to answer questions regarding the nature of the representations involved in conversation. We have suggested that successful conversation requires an interactant to manage a number of representations: two representations for himself: his private goals and knowledge and his public “face,” two representations (public and private) of his main conversational partner, a representation of the content or the upshot of the conversation as a whole, and a representation of the more situational aspects of the conversational setting. The question is what these representations look like, how they interact, and how they are accessed. Are they all part of a single representation, or are they stored and maintained somewhat separately, perhaps in totally different brain areas? It would make sense to separate private and public representations physically, to prevent leakage, although in the brain, distance doesn’t seem to matter much.

In conclusion, then, if we want to arrive at a plausible neurocognitive model of communication, we cannot afford to neglect the social, pragmatic, and dynamic aspects of language use. The actual or perceived presence of an interactant may crucially shape the representations and the processes involved in communication. Thus, we should turn to investigating active conversation.

Acknowledgments

We would like to thank Noortje Venhuizen for comments on an earlier version of this manuscript. Writing of this chapter was in part supported by the Netherlands Organization for Scientific Research (NWO) PGW grant 10–26.

References

Amunts, K., Lenzen, M., Friederici, A. D., Schleicher, A., Moroson, P., Palomero-Gallagher, N., & Zilles, K. (2010). Broca’s region: Novel organizational principles and multiple receptor mapping. PLoS Biology, 8(9).Find this resource:

    Amunts, K., & Zilles, K. (2012). Architecture and organizational principles of Broca’s area. Trends in Cognitive Sciences, 16(8), 418–426.Find this resource:

      Babiloni, F., & Astolfi, L. (2012). Social neurosciences and hyperscanning techniques: Past, present and future. Neuroscience (p. 383) & Biobehavioral Reviews. http://www.sciencedirect.com/science/article/pii/S0149763412001194

      Baggio, G., & Hagoort, P. (2011). The balance between memory and unification in semantics: A dynamic account of the N400. Language and Cognitive Processes, 26, 1338–1367.Find this resource:

        Bara, B. G., Ciaramidaro, A., Walter, H., & Adenzato, M. (2011). Intentional minds: A philosophical analysis of intention tested through fMRI experiments involving people with schizophrenia, people with autism, and healthy individuals. Frontiers in Human Neuroscience, 5(7), 1–11.Find this resource:

          Barbey, A. K., & Grafman, J. (2011). An integrative cognitive neuroscience theory of social reasoning and moral judgment. Wiley Interdisciplinary Review: Cognitive Science, 2 (1), 55–67.Find this resource:

            Barr, D. J. (this volume). Perspective taking and its impostors in language use: Four patterns of deception. In Thomas Holtgraves (Ed.), The Oxford Handbook of Language and Social Psychology (pp.). New York: Oxford University Press.Find this resource:

              Bookheimer, S. (2002). Functional MRI of language: New approaches to understanding the cortical organization of semantic processing. Annual Review of Neuroscience, 25, 151–188.Find this resource:

                Bornkessel, I., Schlesewsky, M., & Friederici, A. D. (2003). Contextual information modulates initial processes of syntactic integration: The role of inter- versus intrasentential predictions. Journal of Experimental Psychology: Learning, Memory, and Cognition, 29(5), 871–882.Find this resource:

                  Bornkessel-Schlesewsky, I., & Schlesewsky, M. (2008). An alternative perspective on semantic P600 effects in language comprehension. Brain Research Reviews, 59, 55–73.Find this resource:

                    Bousfield, D. (2008). Impoliteness in interaction. Amsterdam/Philadelphia: John Benjamins.Find this resource:

                      Brouwer, H. (2014). The Electrophysiology of Language Comprehension: A Neurocomputational Model. PhD thesis, University of Groningen.Find this resource:

                        Brouwer, H., Fitz, H., & Hoeks, J. C. J. (2012). Getting real about Semantic Illusions: Rethinking the functional role of the P600 in language comprehension. Brain Research, 1446, 127–143.Find this resource:

                          Brouwer, H., & Hoeks, J. C. J. (2013). A time and place for language comprehension: Mapping the N400 and the P600 to a minimal cortical network. Frontiers in Human Neuroscience, 7(758), 1–12.Find this resource:

                            Brown, C. M., & Hagoort, P. (1993). The processing nature of the N400: Evidence from masked priming. Journal of Cognitive Neuroscience, 5, 34–44.Find this resource:

                              Brown, P., & Levinson, S. C. (1987). Politeness: Some universals in language usage. Cambridge: Cambridge University Press.Find this resource:

                                Buckner, R. L., Sepulcre, J., Talukdar, T., Krienen, F. M., Liu, H., Hedden, T., Andrews-Hanna, J. R., Sperling, R. A., & Johnson, K. A. (2009). Cortical hubs revealed by intrinsic functional connectivity: Mapping, assessment, of stability, and relation to Alzheimer’s disease. The Journal of Neuroscience, 29, 1860–1873.Find this resource:

                                  Burkhardt, P. (2006). Inferential bridging relations reveal distinct neural mechanisms: Evidence from event-related potentials. Brain and Language, 98(2), 159–168.Find this resource:

                                    Burkhardt, P. (2007). The P600 reflects cost of new information in discourse memory. NeuroReport, 18(17), 1851–1854.Find this resource:

                                      Camblin, C. C., Ledoux, K., Boudewyn, M., Gordon, P. C., & Swaab, T. Y. (2007). Processing new and repeated names: Effects of coreference on repetition priming with speech and fast RSVP. Brain Research, 1146, 172–184.Find this resource:

                                        Chafe, W. L. (1979). Givenness, contrastiveness, definiteness, subjects, topics and point of view. In C. N. Li, C. N. (Ed.), Subject and topic (pp. 27–55). New York: Academic Press.Find this resource:

                                          Chwilla, D. J., Brown, C. M., & Hagoort, P. (1995). The N400 as a function of the level of processing. Psychophysiology, 32, 274–285.Find this resource:

                                            Chow, W.-Y., & Phillips, C. (2013). No semantic illusion in the “semantic P600” phenomenon: ERP evidence from Mandarin Chinese. Brain Research, 1506, 76–93.Find this resource:

                                              Clark, H. H., & Brennan, S. E. (1991). Grounding in communication. Perspectives on Socially Shared Cognition, 13, 127–149.Find this resource:

                                                Clark, H. H., & Haviland, S. E. (1977). Comprehension and the given-new contract. In R. O. Freedle (Ed.), Discourse processes, Vol. 1: Advances in research and theory (pp. 1–40). Norwood, NJ: Ablex.Find this resource:

                                                  Cowles, H. W., Kluender, R., Kutas, M., & Polinsky, M. (2007). Violations of information structure: An electrophysiological study of answers to wh-questions. Brain and Language, 103(3), 228–242.Find this resource:

                                                    Culpeper, J. (1996). Towards an anatomy of impoliteness. Journal of Pragmatics, 25(3), 349–367.Find this resource:

                                                      Dawkins, R. (2006). The selfish gene (30th anniversary edition). New York: Oxford University Press.Find this resource:

                                                        Decety, J., & Lamm, C. (2007). The role of the right temporoparietal junction in social interaction: How low-level computational processes contribute to meta-cognition. The Neuroscientist, 13(6), 580–593.Find this resource:

                                                          Dimitrova, D. V., Stowe, L. A., Redeker, G., & Hoeks, J. C. J. (2012). Less is not more: Neural responses to missing and superfluous accents in context. Journal of Cognitive Neuroscience, 24(12), 2400–2418.Find this resource:

                                                            Ditman, T., & Kuperberg, G. R. (2007). The time course of building discourse coherence in schizophrenia: An ERP investigation. Psychophysiology, 44, 991–1001.Find this resource:

                                                              Dronkers, N. F., Wilkins, D. P., Van Valin, R. D., Jr. Redfern, B. B., & Jaeger, J. J. (2004). Lesion analysis of the brain areas involved in language comprehension. Cognition, 92, 145–177.Find this resource:

                                                                Dumas, G., Chavez, M., Nadel, J., & Martinerie, J. (2012). Anatomical connectivity influences both intra- and inter-brain synchronizations. PLoS ONE, 7(5), 1–11.Find this resource:

                                                                  Dumas, G., Nadel, J., Soussignan, R., Martinerie, J., & Garnero, L. (2010). Inter-brain synchronization during social interaction. PLoS ONE, 5(8), 1–10.Find this resource:

                                                                    Elman, J. L. (2004). An alternative view of the mental lexicon. Trends in Cognitive Sciences, 8, 301–306.Find this resource:

                                                                      Elman, J. L. (2009). On the meaning of words and dinosaur bones: Lexical knowledge without a lexicon. Cognitive Science, 33, 547–582.Find this resource:

                                                                        Federmeier, K. D., & Laszlo, S. (2009). Time for meaning: Electrophysiology provides insights into the dynamics of representation and processing in semantic memory. Psychology of Learning and Motivation, 51, 1–44.Find this resource:

                                                                          Friederici, A. D. (2002). Towards a neural basis of auditory sentence processing. Trends in Cognitive Sciences, 6, 78–84.Find this resource:

                                                                            Friederici, A. D. (2009). Pathways to language: Fiber tracts in the human brain. Trends in Cognitive Sciences, 13, 175–181.Find this resource:

                                                                              Friederici, A. D. (2011). The brain basis of language processing: From structure to function. Physiological Reviews, 91, 1357–1392.Find this resource:

                                                                                Goffman, E. (1959). The presentation of self in everyday life. New York: Doubleday.Find this resource:

                                                                                  (p. 384) Goffman, E. (1967). Interaction ritual: Essays on face-to-face behavior. New York: Pantheon.Find this resource:

                                                                                    Gouvea, A. C., Phillips, C., Kazanina, N., & Poeppel, D. (2010). The linguistic processes underlying the P600. Language and Cognitive Processes, 25(2), 149–188.Find this resource:

                                                                                      Grice, H. P. (1975). Logic and conversation. In P. Cole & J. L. Morgan (Eds.), Syntax and semantics, Vol. 3: Speech acts (pp. 41–58). New York: Academic Press.Find this resource:

                                                                                        Grodzinsky, Y., & Santi, A. (2008). The battle for Broca’s region. Trends in Cognitive Sciences, 12, 474–480.Find this resource:

                                                                                          Hagoort, P. (2003). Interplay between syntax and semantics during sentence comprehension: ERP effects of combining syntactic and semantic violations. Journal of Cognitive Neuroscience, 15, 883–899.Find this resource:

                                                                                            Hagoort, P. (2005). On Broca, brain, and binding: A new framework. Trends in Cognitive Sciences, 6, 416–423.Find this resource:

                                                                                              Hagoort, P., Baggio, G., & Willems, R. M. (2009). Semantic unification. In: Gazzaniga, M. S. (Ed.), The cognitive neurosciences (4th ed., pp. 819–836). Cambridge, MA: MIT Press.Find this resource:

                                                                                                Hagoort, P., & Van Berkum, J. J. A. (2007). Beyond the sentence given. Philosophical Transaction of the Royal Society, Series B: Biological Sciences, 362, 801–811.Find this resource:

                                                                                                  Heim, S., & Alter, K. (2006). Prosodic pitch accents in language comprehension and production: ERP data and acoustic analysis. Acta Neurobiologiae Experimentalis, 66, 55–68.Find this resource:

                                                                                                    Hickok, G., & Poeppel, D. (2004). Dorsal and ventral streams: A framework for understanding aspects of the functional anatomy of language. Cognition, 92, 67–99.Find this resource:

                                                                                                      Hickok, G., & Poeppel, D. (2007). The cortical organization of speech processing. Nature Reviews Neuroscience, 8, 393–402.Find this resource:

                                                                                                        Hoeks, J. C. J. Stowe, L. A., Hendriks, P., & Brouwer, H. (2013). Questions left unanswered: How the brain responds to missing information. PLoS ONE, 8(10), 1–9.Find this resource:

                                                                                                          Hoeks, J. C. J., Schoot, L., Taylor, R. C., & Brouwer, H. (in preparation). How to say “no” politely. An ERP investigation. Manuscript in preparation.Find this resource:

                                                                                                            Hoeks, J. C. J., Schoot, L., Neijmeijer, M. I. A., & Brouwer, H. (in preparation). The Dialogue Immersion Paradigm (DIP) as a tool to investigate active conversation. Manuscript in preparation.Find this resource:

                                                                                                              Hoeks, J. C. J., Stowe, L. A., & Doedens, G. (2004). Seeing words in context: The interaction of lexical and sentence level information during reading. Cognitive Brain Research, 19, 59–73.Find this resource:

                                                                                                                Hruska, C., & Alter, K. (2004). Prosody in dialogues and single sentences: How prosody can influence speech perception. In A. Steube (Ed.), Information structure: Theoretical and empirical aspects (pp. 211–226). Berlin: Walter de Gruyter.Find this resource:

                                                                                                                  Hruska, C., Alter, K., Steinhauer, K., & Steube, A. (2001). Misleading dialogs: Human’s brain reaction to prosodic information. In C. Cave, I. Guaitella, & S. Santi (Eds.), Orality and gestures (pp. 425–430). Paris: L’Hartmattan.Find this resource:

                                                                                                                    Hunt III, L., Politzer-Ahlers, S., Gibson, L., Minai, U., & Fiorentino, R. (2012). Pragmatic inferences modulate N400 during sentence comprehension: Evidence from picture-sentence verification. Neuroscience Letters, 534, 246–251.Find this resource:

                                                                                                                      Johnson-Laird, P. N. (1983). Mental models. Cambridge, UK: Cambridge University Press.Find this resource:

                                                                                                                        Keysar, B. (2007). Communication and miscommunication: The role of egocentric processes. Intercultural Pragmatics, 4(1), 71–84.Find this resource:

                                                                                                                          Kim, A., & Osterhout, L. (2005). The independence of combinatory semantic processing: Evidence from event-related potentials. Journal of Memory and Language, 52, 205–225.Find this resource:

                                                                                                                            Kintsch, W. (1998). Comprehension: A paradigm for cognition. Cambridge, UK: Cambridge University Press.Find this resource:

                                                                                                                              Kolk, H. H. J., Chwilla, D. J., Van Herten, M., & Oor, P. J. W. (2003). Structure and limited capacity in verbal working memory: A study with event-related potentials. Brain and Language, 85, 1–36.Find this resource:

                                                                                                                                Kos, M., Vosse, T., van den Brink, D., & Hagoort, P. (2010). About edible restaurants: Conflicts between syntax and semantics as revealed by ERPs. Frontiers in Psychology, 1, 1–11.Find this resource:

                                                                                                                                  Kotchoubey, B. (2006). Event-related potentials, cognition, and behavior: A biological approach. Neuroscience and Biobehavioral Reviews, 30, 42–56.Find this resource:

                                                                                                                                    Kraut, R. E., Lewis, S. H., & Swezey, L. W. (1982). Listener responsiveness and the coordination of conversation. Journal of Personality and Social Psychology, 43(4), 718–736.Find this resource:

                                                                                                                                      Kuperberg, G. R. (2007). Neural mechanisms of language comprehension: Challenges to syntax. Brain Research, 1146, 23–49.Find this resource:

                                                                                                                                        Kuperberg, G. R., Sitnikova, T., Caplan, D. N., & Holcomb, P. J. (2003). Electrophysiological distinctions in processing conceptual relationships within simple sentences. Cognitive Brain Research, 17, 117–129.Find this resource:

                                                                                                                                          Kutas, M., & Federmeier, K. D. (2000). Electrophysiology reveals semantic memory usage in language comprehension. Trends in Cognitive Sciences, 4, 463–470.Find this resource:

                                                                                                                                            Kutas, M., & Federmeier, K. D. (2011). Thirty years and counting: Finding meaning in the N400 component of the event related brain potential (ERP). Annual Review of Psychology, 62, 621–647.Find this resource:

                                                                                                                                              Kutas, M., & Hillyard, S. A. (1980). Reading senseless sentences: Brain potentials reflect semantic incongruity. Science, 207, 203–205.Find this resource:

                                                                                                                                                Kutas, M., & Hillyard, S. A. (1984). Brain potentials during reading reflect word expectancy and semantic association. Nature, 307, 539–550.Find this resource:

                                                                                                                                                  Kutas, M., & Lindamood, T. E., & Hillyard, S. A. (1984). Word expectancy and event-related brain potentials during sentence processing. In S. Kornblum & J. Requin (Eds.), Preparatory states and processing (pp. 217–317), Hillsdale, NJ: Erlbaum.Find this resource:

                                                                                                                                                    Kutas, M., Van Petten, C., & Kluender, R. (2006). Psycholinguistics electrified II: 1994–2005. In M. J. Traxler & M. A. Gernsbacher (Eds.), Handbook of psycholinguistics (2nd ed., pp 659–724). New York: Elsevier.Find this resource:

                                                                                                                                                      Lambrecht, K. (1994). Information structure and sentence form. Cambridge, UK: Cambridge University Press.Find this resource:

                                                                                                                                                        Lau, E. F., Phillips, C., & Poeppel, D. (2008). A cortical network for semantics: (de)constructing the N400. Nature Reviews Neuroscience, 9, 920–933.Find this resource:

                                                                                                                                                          Ledoux, K., Gordon, P. C., Camblin, C. C., & Swaab, T. Y. (2007). Coreference and lexical repetition: Mechanisms of discourse integration. Memory and Cognition, 4, 801–815.Find this resource:

                                                                                                                                                            Li, X., Hagoort, P., & Yang, Y. (2008). Event-related potential evidence on the influence of accentuation in spoken discourse comprehension in Chinese. Journal of Cognitive Neuroscience, 20(5), 906–915.Find this resource:

                                                                                                                                                              Li, X., Yang, Y., & Hagoort, P. (2008). Pitch accent and lexical tone processing in Chinese discourse comprehension: An ERP study. Brain Research, 1222, 192–200.Find this resource:

                                                                                                                                                                Lotze, N., Tune, S., Schlesewsky, M., & Bornkessel-Schlesewsky, I. (2011). Meaningful physical changes mediate lexical-semantic integration: Top-down and form-based (p. 385) bottom-up information sources interact in the N400. Neuropsychologia, 49, 3573–3582.Find this resource:

                                                                                                                                                                  Luck, S. J. (2005). An introduction to the event-related potential technique. Cambridge, MA: MIT Press.Find this resource:

                                                                                                                                                                    Magne, C., Astésano, C. Lacheret-Dujour, A., Morel, M., Alter, K., & Besson, M. (2005). On-line processing of “pop-out” words in spoken French dialogues. Journal of Cognitive Neuroscience, 17, 740–756.Find this resource:

                                                                                                                                                                      Mesulam, M. M. (1990). Large-scale neurocognitive networks and distributed processing for attention, language, and memory. Annals of Neurology, 28, 597–613.Find this resource:

                                                                                                                                                                        Mesulam, M. M. (1998). From sensation to cognition. Brain, 121, 1013–1052.Find this resource:

                                                                                                                                                                          Molinaro, N., Carreiras, M., & Duñabeitia, J. A. (2012). Semantic combinatorial processing of non-anomalous expressions. NeuroImage, 59, 3488–3501.Find this resource:

                                                                                                                                                                            Näätänen, R., & Picton, T. (1987). The N1 wave of the human electric and magnetic response to sound: A review and an analysis of the component structure. Psychophysiology, 24, 375–425.Find this resource:

                                                                                                                                                                              Nieuwland, M. S., Ditman, T., & Kuperberg, G. R. (2010). On the incrementality of pragmatic processing: An ERP investigation of informativeness and pragmatic abilities. Journal of Memory and Language, 63(3), 324–346.Find this resource:

                                                                                                                                                                                Nieuwland, M. S., & Otten, M., & Van Berkum, J. J. A. (2007). Who are you talking about? Tracking discourse-level referential processing with ERPs. Journal of Cognitive Neuroscience, 19, 228–236.Find this resource:

                                                                                                                                                                                  Nieuwland, M. S., Petersson, K. M., & Van Berkum, J. J. A. (2007). On sense and reference: Examining the functional neuroanatomy of referential processing. NeuroImage, 37, 993–1004.Find this resource:

                                                                                                                                                                                    Nieuwland, M. S., & Van Berkum, J. J. A. (2005). Testing the limits of the semantic illusion phenomenon: ERPs reveal temporary semantic change deafness in discourse comprehension. Cognitive Brain Research, 24, 691–701.Find this resource:

                                                                                                                                                                                      Noveck, I. A., & Posada, A. (2003). Characterizing the time course of an implicature: An evoked potentials study. Brain and Language, 85(2), 203–210.Find this resource:

                                                                                                                                                                                        Nunberg, G. (1979). The non-uniqueness of semantic solutions: Polysemy. Linguistics and Philosophy, 3(2), 143–184.Find this resource:

                                                                                                                                                                                          Osterhout, L., & Holcomb, P. J. (1992). Event-related brain potentials elicited by syntactic anomaly. Journal of Memory and Language, 31(6), 785–806.Find this resource:

                                                                                                                                                                                            Otten, M., & Van Berkum, J. J. A. (2007). What makes a discourse constraining? Comparing the effects of discourse message and scenario fit on the discourse-dependent N400-effect, Brain Research, 1153, 166–177.Find this resource:

                                                                                                                                                                                              Pulvermüller, F. (1999). Words in the brain’s language. Behavioral and Brain Sciences, 22, 252–279.Find this resource:

                                                                                                                                                                                                Pulvermüller, F. (2001). Brain reflections of words and their meaning. Trends in Cognitive Sciences, 5, 517–524.Find this resource:

                                                                                                                                                                                                  Regel, S., Gunter, T., & Friederici, A. D. (2011). Isn’t it ironic? An electrophysiological exploration of figurative language processing. Journal of Cognitive Neuroscience, 23(2), 277–293.Find this resource:

                                                                                                                                                                                                    Rogalsky, C., & Hickok, G. (2011). The role of Broca’s area in sentence comprehension. Journal of Cognitive Neuroscience, 23, 1664–1680.Find this resource:

                                                                                                                                                                                                      Rogers, T., & McClelland, J. (2004). Semantic cognition: A parallel distributed processing approach. Cambridge, MA: The MIT Press.Find this resource:

                                                                                                                                                                                                        Russell, B. (1905). On denoting. Mind, 14(56), 479–493.Find this resource:

                                                                                                                                                                                                          Schilbach, L., Timmermans, B., Reddy, V., Costall, A., Bente, G., Schlicht, T., & Vogeley, K. (2013). Toward a second-person neuroscience. Behavioral and Brain Sciences, 36(4), 393–414.Find this resource:

                                                                                                                                                                                                            Schlenker, B. R. (1980). Impression management: The self-concept, social identity, and interpersonal relations. Monterey, CA: Brooks/Cole.Find this resource:

                                                                                                                                                                                                              Schober, M. F., & Clark, H. H. (1989). Understanding the addressees and overhearers. Cognitive Psychology, 21(2), 211–232.Find this resource:

                                                                                                                                                                                                                Schumacher, P. (2011). The hepatitis called…. In J. Meibauer & M. Steinbach (Eds.), Experimental pragmatics/semantics (pp. 199–219). Amsterdam/Philadelphia: John Benjamins.Find this resource:

                                                                                                                                                                                                                  Shalom, D. B., & Poeppel, D. (2007). Functional anatomic models of language: Assembling the pieces. The Neuroscientist, 30, 119–127.Find this resource:

                                                                                                                                                                                                                    Sperber, D., & Wilson, D. (1986). Relevance. Oxford: Blackwell Publishers.Find this resource:

                                                                                                                                                                                                                      Sperber, D., & Wilson, D. (1995). Relevance (2nd ed.). Oxford: Blackwell Publishers.Find this resource:

                                                                                                                                                                                                                        Spotorno, N., Cheylus, A., Van der Henst, J-B., & Noveck, I. A. (2013). What’s behind a P600? Integration operations during irony processing. PLoS ONE, 8(6), 1–10.Find this resource:

                                                                                                                                                                                                                          Stephens, G. J., Silbert, L. J., & Hasson, U. (2010). Speaker—Listener neural coupling underlies successful communication. Proceedings of the National Academy of Sciences, 107(32), 14425–14430.Find this resource:

                                                                                                                                                                                                                            St. George, M., Mannes, S., & Hoffmann, J. E. (1994). Global expectancy and language comprehension. Journal of Cognitive Neuroscience, 6(1), 70–83.Find this resource:

                                                                                                                                                                                                                              Stroud, C. M. A., & Phillips, C. (2012). Examining the evidence for an independent semantic analyzer: An ERP study in Spanish. Brain and Language, 120, 108–126.Find this resource:

                                                                                                                                                                                                                                Suda, M., Takei, Y., Aoyama, Y., Narita, K., Sato, T., Fukuda, M., & Mikuni, M. (2010). Frontopolar activation during face-to-face conversation: An in situ study using near-infrared spectroscopy. Neuropsychologia, 48(2), 441–447.Find this resource:

                                                                                                                                                                                                                                  Swaab, T. Y., Camblin, C. C., & Gordon, P. C. (2004). Electrophysiological evidence for reversed lexical repetition effects in language processing. Journal of Cognitive Neuroscience, 16(5), 715–726.Find this resource:

                                                                                                                                                                                                                                    Taylor, R. C. (2013). Tracking referents: Markedness, world knowledge and pronoun resolution. PhD thesis, University of Groningen.Find this resource:

                                                                                                                                                                                                                                      Taylor, R. C., Stowe, L. A., Redeker, G., & Hoeks, J. C. J. (in preparation). Accented pronouns and unexpected referents. An ERP study. Manuscript in preparation.Find this resource:

                                                                                                                                                                                                                                        Toepel, U., & Alter, K. (2004). On the independence of information structure processing from prosody. In A. Steube (Ed.), Information structure: Theoretical and empirical aspects (pp. 227–240). Berlin: Walter de Gruyter.Find this resource:

                                                                                                                                                                                                                                          Toepel, U., Pannenkamp, A., & Alter, K. (2007). Catching the news: Processing strategies in listening to dialogs as measured by ERPs. Behavioral and Brain Functions, 3(53), 1–13.Find this resource:

                                                                                                                                                                                                                                            Turken, U., & Dronkers, N. F. (2011). The neural architecture of the language comprehension network: Converging evidence from lesion and connectivity analyses. Frontiers in Systems Neuroscience, 5, 1–20.Find this resource:

                                                                                                                                                                                                                                              Tyler, L., Marlsen-Wilson, W., Randal, B., Wright, P., Devereux, B., Zhuang, J., Papoutsi, M., & Stamatakis, E. (2011). Left inferior frontal cortex and syntax: Function, structure, and behaviour in patients with left hemisphere damage. Brain, 134, 415–431.Find this resource:

                                                                                                                                                                                                                                                (p. 386) Van Berkum, J. J. A. (2009). The “neuropragmatics” of simple utterance comprehension: An ERP review. In U. Sauerland & Yatsushiro (Eds.), Semantics and pragmatics: From experiment to theory (pp. 276–316). New York: Palgrave MacMillan.Find this resource:

                                                                                                                                                                                                                                                  Van Berkum, J. J. A. (2012). The electrophysiology of discourse and conversation. In M. Spivey, M. Joanisse, & K. McRae (Eds.), The Cambridge handbook of psycholinguistics (pp. 589–614). Cambridge, UK: Cambridge University Press.Find this resource:

                                                                                                                                                                                                                                                    Van Berkum, J. J. A., Brown, C. M., & Hagoort, P. (1999). Early referential context effects in sentence processing: Evidence from event-related brain potentials. Journal of Memory and Language, 41, 147–182.Find this resource:

                                                                                                                                                                                                                                                      Van Berkum, J. J. A., Hagoort, P., & Brown, C. M. (1999). Semantic integration in sentences and discourse: Evidence from the N400. Journal of Cognitive Neuroscience, 11(6), 657–671.Find this resource:

                                                                                                                                                                                                                                                        Van Berkum, J. J. A., Koornneef, A. W., Otten, M., & Nieuwland, M. S. (2007). Establishing reference in language comprehension: An electrophysiological perspective. Brain Research, 1146, 158–171.Find this resource:

                                                                                                                                                                                                                                                          Van Berkum, J. J. A., Van den Brink, D., Tesink, C, M, J, Y., Kos, M., & Hagoort, P. (2008). The neural integration of speaker and message, Journal of Cognitive Neuroscience, 20 (4), 580–591.Find this resource:

                                                                                                                                                                                                                                                            Van Berkum, J. J. A., Zwitserlood, P., Brown, C. M., & Hagoort, P. (2003). When and how do listeners relate a sentence to a wider discourse? Evidence from the N400 effect. Cognitive Brain Research, 17, 701–718.Find this resource:

                                                                                                                                                                                                                                                              Van de Meerendonk, N., Kolk, H. H. J., Chwilla, D. J., & Vissers, C. T. W. M. (2009). Monitoring in language perception. Language and Linguistics Compass, 3, 1211–1224.Find this resource:

                                                                                                                                                                                                                                                                Van de Meerendonk, N, Kolk, H. H. J., Vissers, C. T. W. M., & Chwilla, D. J. (2010). Monitoring in language perception: Mild and strong conflicts elicit different ERP patterns. Journal of Cognitive Neuroscience, 22, 67–82.Find this resource:

                                                                                                                                                                                                                                                                  Van Herten, M., Kolk, H. H. J., & Chwilla, D. J. (2005). An ERP study of P600 effects elicited by semantic anomalies. Cognitive Brain Research, 22, 241–255.Find this resource:

                                                                                                                                                                                                                                                                    Van Herten, M., Chwilla, D. J., & Kolk. H. H. J. (2006). When heuristics clash with parsing routines: ERP evidence from conflict monitoring in sentence perception. Journal of Cognitive Neuroscience, 18, 1181–1197.Find this resource:

                                                                                                                                                                                                                                                                      Van Petten, C., & Luka, B. J. (2012). Prediction during language comprehension: Benefits, costs, and ERP components. International Journal of Psychophysiology, 83(2), 176–190.Find this resource:

                                                                                                                                                                                                                                                                        Vigneau, M., Beaucousin, V., Herv, P., Duffau, H, Crivello, F., Houd, O., Mazoyer, B., Tzourio-Mazoyer, N. (2006). Meta-analyzing left hemisphere language areas: Phonology, semantics, and sentence processing. NeuroImage, 30, 1440–1422.Find this resource:

                                                                                                                                                                                                                                                                          Vissers, C. T. W. M., Chwilla, D. J., & Kolk, H. H. J. (2006). Monitoring in language perception: The effect of misspellings of words in highly constrained sentences. Brain Research, 1106, 150–163.Find this resource:

                                                                                                                                                                                                                                                                            Vissers, C. T. W. M., Chwilla, D. J., & Kolk, H. H. J. (2007). The interplay of heuristics and parsing routines in sentence comprehension: Evidence from ERPs and reaction times. Biological Psychology, 75, 8–18.Find this resource:

                                                                                                                                                                                                                                                                              Vissers, C. T. W. M., Chwilla, U. G., Egger, J. I. M., & Chwilla, D. J. (2013). The interplay between mood and language comprehension: Evidence from P600 to semantic reversal anomalies. Neuropsychologia, 51(6), 1027–1039.Find this resource:

                                                                                                                                                                                                                                                                                Wang, L., Bastiaansen, M., Yang, Y., & Hagoort, P. (2011). The influence of information structure on the depth of semantic processing: How focus and pitch accent determine the size of the N400 effect. Neuropsychologia, 49, 813–820.Find this resource:

                                                                                                                                                                                                                                                                                  Wang, L., Hagoort, P., & Yang, Y. (2009). Semantic illusion depends on information structure: ERP evidence. Brain Research, 1282, 50–56.Find this resource:

                                                                                                                                                                                                                                                                                    Weiller, C., Musso, M., Rijntjes, M., & Saur, D. (2009). Please don’t underestimate the ventral pathway in language. Trends in Cognitive Sciences, 13, 369–370.Find this resource:

                                                                                                                                                                                                                                                                                      Ye, Z., & Zhou, X. (2008). Involvement of cognitive control in sentence comprehension: Evidence from ERPs. Brain Research, 1203, 103–115.Find this resource:

                                                                                                                                                                                                                                                                                        Zwaan, R. A., & Radvansky, G. A. (1998). Situation models in language comprehension and memory. Psychological Bulletin, 123, 162–185.Find this resource: