From everyday conversations, we know that gestures can enhance, disambiguate, or even change the intended meaning expressed by a speaker. Especially iconic co-speech gestures can provide nuanced information to, e.g., a talked about action, such as speed or orientation; and yet, gestures seem to not possess fixed meanings on their own. This begs the question whether and how humans use information present in gestures to process speech. To investigate whether iconic co-speech gestures modulate semantic predictions for upcoming linguistic content, we designed an EEG (electroencephalography) experiment. We focused on measuring the modulation of the N400 component on words that are expected or not, based on the gesture used in prior context, since the N400 is strongly linked to the predictability of the stimulus, e.g., it tends to be larger for words that are semantically less appropriate or less expected in a given context (Kutas & Federmeier, 2011).

We used naturalistic videos of a person saying a sentence uttering a general verb, featuring no gesture or an iconic co-speech gesture representing a more specific action. The target sentence was presented on screen and contained an instrument noun followed by its required action verb, both either matching or mismatching the previously seen gesture, giving us three conditions: Neutral (no gesture previously seen), Match, and Mismatch. After reading the target sentence, participants were prompted to rate whether the target sentence was a sensible continuation of the content of the context video. The linguistic material did not vary in modulating expectations about upcoming words, so any change during processing would be due to a change in the gesture. To avoid effects caused by unfitting gestures we ensured that all iconic co-speech gestures were congruent with the simultaneously uttered context sentence.

This approach is novel in several regards: First, experiments measuring ERPs elicited by linguistic targets following a gesture used paradigms of isolated, silent gesture videos followed by single probe words. Second, experiments using full sentences focused on the (mis)match between co-speech gestures and simultaneously uttered phrases or words (Hintz et al., 2023; Özyürek et al., 2007). Some also varied the linguistic context sentences, adding another factor potentially influencing the processing of utterances and co-speech gestures (Hintz et al., 2023). Nevertheless, these and other studies (see review by Arachchige et al., 2021) showed that iconic co-speech gestures can interact with the semantic processing of linguistic input.

Since we consider iconic co-speech gestures as useful tools when it comes to disambiguating spoken utterances, we expected the Mismatches to be more surprising and thus elicit an N400 effect. For the Neutrals, our expectation was that we get an intermediate N400 effect since there is no gesture present that could lower or raise the expectation about upcoming input. Since we presented the target sentences word by word and the tool nouns preceded the action verbs, we additionally expect these effects to be stronger for the nouns as these were the first words either matching or mismatching the gesture.

For noun targets, we observed a significant negativity for Mismatch vs. Match that was in line with an N400 effect. For the verb targets, a significant negativity effect was found for the comparison Mismatch vs. Match, starting around 300ms and extending towards the end of the epoch, most prominent over the central-medial and posterior regions, suggesting a N400 effect with a sustained negativity. Another significant negativity effect is present in the comparison Match vs. Neutral, spanning over a similar time window.

For both targets, nouns and verbs, we found sustained positivity for Neutral-Match and Neutral-Mismatch which could be interpreted as a modulation of the P300, a positivity elicited between 250 and 500ms post-stimulus onset, known to be modulated by the probability of the target and by its relation to the task. Especially the P300b subcomponent tends to be larger for items that are task-relevant, and thus awaited in the experiment (Polich, 2003, 2007).

To conclude, listeners cannot only integrate the information presented in an iconic co-speech gesture, but the gestures also make a difference for a listener’s probabilistic prediction regarding an upcoming instrument noun and a following action verb, and thus has a semantic effect on linguistic comprehension. The found N400 suggests that matching gestures facilitate linguistic understanding even for the upcoming linguistic content and not only for co-occurring linguistic content.

Alday, P. M., & Kretzschmar, F. (2019). Speed-Accuracy Tradeoffs in Brain and Behavior: Testing the Independence of P300 and N400 Related Processes in Behavioral Responses to Sentence Categorization. Frontiers in Human Neuroscience, 13, 285. https://doi.org/10.3389/fnhum.2019.00285

Habets, B., Kita, S., Shao, Z., Özyurek, A., & Hagoort, P. (2011). The role of synchrony and ambiguity in speech-gesture integration during comprehension. Journal of Cognitive Neuroscience, 23(8), 1845–1854. https://doi.org/10.1162/jocn.2010.21462

Hintz, F., Khoe, Y. H., Strauß, A., Psomakas, A. J. A., & Holler, J. (2023). Electrophysiological evidence for the enhancement of gesture-speech integration by linguistic predictability during multimodal discourse comprehension. Cognitive, Affective, & Behavioral Neuroscience, 23(2), 340–353. https://doi.org/10.3758/s13415-023-01074-8

Kandana Arachchige, K. G., Simoes Loureiro, I., Blekic, W., Rossignol, M., & Lefebvre, L. (2021). The Role of Iconic Gestures in Speech Comprehension: An Overview of Various Methodologies. Frontiers in Psychology, 12. https://doi.org/10.3389/fpsyg.2021.634074

Kutas, M., & Federmeier, K. D. (2011). Thirty Years and Counting: Finding Meaning in the N400 Component of the Event-Related Brain Potential (ERP). Annual Review of Psychology, 62(1), 621–647. https://doi.org/10.1146/annurev.psych.093008.131123

Özyürek, A., Willems, R. M., Kita, S., & Hagoort, P. (2007). On-line integration of semantic information from speech and gesture: Insights from event-related brain potentials. Journal of Cognitive Neuroscience, 19(4), 605–616. https://doi.org/10.1162/jocn.2007.19.4.605

Polich, J. (2003). Theoretical overview of P3a and P3b. In Polich (Hrsg.), Detection of Change: Event-Related Potential and fMRI Findings (S. 83–98). Kluwer Academic Press: Boston.

Polich, J. (2007). Updating P300: An Integrative Theory of P3a and P3b. Clinical Neurophysiology: Official Journal of the International Federation of Clinical Neurophysiology, 118(10), 2128–2148. https://doi.org/10.1016/j.clinph.2007.04.019

Roehm, D., Bornkessel-Schlesewsky, I., Rösler, F., & Schlesewsky, M. (2007). To predict or not to predict: Influences of task and strategy on the processing of semantic relations. Journal of Cognitive Neuroscience, 19(8), 1259–1274.

Wu, Y. C., & Coulson, S. (2007). How iconic gestures enhance communication: An ERP study. Brain and Language, 101(3), 234–245. https://doi.org/10.1016/j.bandl.2006.12.003

Co-predications are a class of sentences thought to be a serious puzzle for truth-conditional semantics. Suppose that I use (1):
(1) The book, which was checked out of the library, was written by Kripke.
Whilst talking to you about my copy of Naming and Necessity. A standard truth-conditional semantics will say that the core of (1)'s meaning is:

(1) is true iff there exists exactly one x such that x is a book, x was checked out of the library and x was written by Kripke.
This is oversimplified, but it isn't entirely wrong. The point is that we ascribe two predicates, checked out of the library and written by Kripke to the referent of the book. The problem: naively, Kripke didn't write my copy of Naming and Necessity. It was printed by a machine. Naively, I didn't check Naming and Necessity} out of the library. It's a piece of informational content with no spatiotemporal properties. Therefore, the right-hand side of the bi-Conditional doesn't hold, and (1) is either false or a category mistake, contrary to intuition. In a series of recent papers and a book manuscript, Liebesman & Magidor (hereafter "L&M") address this puzzle by rejecting the naive metaphysics, instead defending:
Property Versatility View (PVV):
Co-predications are possible because properties aren't sortally restricted. (This is oversimplified. L&M merely think the sortal restrictions are much more relaxed than the standard view (L&M 2017, p149)

For example, book-types can be checked out of libraries if their tokens are. Resultantly, there's nothing incoherent about (1)'s truth-conditions.

PVV elegantly resolves the puzzle of co-predication. This only increases the burden to scrutinise it. That's the aim of this paper. I'm going to produce novel counting ambiguities (novel, at least, for the literature on co-predication), and use it to undermine PVV.

One of PVV's purported advantages is its handling of ambiguity in counting sentences. Consider:

One-Collection:
The only book written by Jane Austen checked out of the library is a copy of her collected works, call it b. That is, b is Sense and Sensibility, Pride and Prejudice, Mansfield Park, Emma, Northanger Abbey, Persuasion and Lady Susan all bound into a single volume (adapted from Asher 2008, p161)

How many books were checked out of the library in One-Collection? Intuitively there are two possible answers:

(2) One book written by Jane Austen was checked out of the library.
(3) Seven books written by Jane Austen were checked out of the library.

PVV predicts (2) and (3) have both true and false readings. Due to property-versatility, we get predicate extensions like:

[[book]] ={b, Sense & Sensibility, Naming & Necessity,...}
[[checked out of the library]] ={b, Sense & Sensibility, Pride & Prejudice,...}

Then in a context, we use book restrictedly, so we either count only book-tokens or only book-types (L&M 2017, p136)

In this paper, I consider another counting ambiguity and argue that it cannot be accommodated within PVV. The ambiguity in question are textit{event-related} readings of counting sentences. Consider:

Toll-Booth:
You work at a toll booth on a lock. One day, the ships Vesroth and Vargas I pass through the lock together. They both pay the toll. Later that day, Vargas I comes back through again, and she pays a second toll.

Krifka (1990) observed you can use sentences like:

(4) Three ships passed through the lock.

To report how many times a ship passed through the lock (hence the term "event-related"). We've counted Vargas I as two ships, despite being a single entity.

There are two extant approaches to event-related readings, and I consider the consequences of adding either to PVV. One one approach, the Determiner View, there is a silent determiner which measures event-participants (e.g. Krifka (1990), Doetjes & Honcoop (1997)). On another approach, the Stage View, we count stages (time-slices of individuals) instead of individuals (e.g. Barker (1990), Gotham (2021)). I consider:

Library:
The library has one copy of Jane Austen's collected works, $b$. Last week, $b$ was checked out, returned, and then checked out again. No other books by Jane Austen were checked out.

Intuitively, the following:

(5) Fourteen books written by Jane Austen were checked out of the library.

Is not true in Library. I argue that on both the Determiner View and Stage View, PVV incorrectly predicts that (5) has a true reading. PVV tells us that book-types are books, written by Jane Austen and can be checked out of libraries. So on the Determiner View, we get the reading because there are two distinct events where 7 books written by Austen are checked out of the library (Sense and Sensibility, Mansfield Park, and so on). This yields a count of 14 books. On the Stage View, we get the reading because if Mansfield Park is checked out of the library twice, then there are at least two stages: MP-during-first-checkout, and MP-during-second-checkout. Extending this reasoning to Austen's other novels yields a count of 14 books.

After considering and rejecting some responses that could be made on behalf of PVV, I propose that we strive for a unified theory for counting sentences upon which token, type and event-related readings are derived. The upshot: counting ambiguities are largely an independent project. They should be part of the semantic and pragmatic toolkit for any theory of co-predication. PVV thus doesn't have the advantage with counting ambiguities it might be thought to have.

References

Asher, N. (2008). A Type Driven Theory of Predication with Complex Types.
Fundamenta Informaticae, 84(2), 151–183. Retrieved June 22, 2023, from
https://search.ebscohost.com/login.aspx?direct=true&db=a9h&AN=
33410207&site=ehost-live&scope=site&authtype=shib&custid=s8978330
Barker, C. (1999). Individuation and Quantification. Linguistic Inquiry, 30(4),
683–691. JSTOR: 4179087. Retrieved November 10, 2023, from https://
www.jstor.org/stable/4179087
Doetjes, J., & Honcoop, M. (1997). The Semantics of Event-Related Readings:
A Case for Pair-Quantification. In A. Szabolcsi (Ed.), Ways of Scope Taking
(pp. 263–310). doi:10.1007/978-94-011-5814-5 8
Gotham, M. (2021). Event-related readings and degrees of difference. Proceed-
ings of Sinn und Bedeutung, 25, 325–339. doi:10.18148/sub/2021.v25i0.940
Krifka, M. (1990). Four thousand ships passed through the lock: Object-induced
measure functions on events. Linguistics and Philosophy, 13(5), 487–520.
doi:10.1007/BF00627291
Liebesman, D., & Magidor, O. (2017). Copredication and Property Inheritance.
Philosophical Issues, 27, 131–166. JSTOR: 26611258. Retrieved June 14, 2023,
from https://www.jstor.org/stable/26611258