Canada

In describing images, visual and linguistic processes coordinate with each other and they both proceed in a linear fashion. Three main theories about the nature of the relation between these processes and how they unfold over time have been proposed in the literature of &#39;linearization&#39;. In this work, we investigate the hypotheses put forward by these theories utilizing a corpus of spoken image descriptions with speakers&#39; eye-movement data. We explore the time alignment between the fixations on objects and the utterance of the corresponding nouns in the data. In contrast to previous studies, this dataset allows us to inspect unrestricted language production in the context of real scenes on a larger scale. We find both confirming and conflicting evidence for each of the theories in question, suggesting that the intricate relation between eye movements and language production may involve mechanisms proposed by all three theories.


CogSci 2022

Time Alignment between Gaze and Speech in Image Descriptions: Exploring Theories of Linearization

EMNLP 2021 is planned to be a hybrid event in Punta Cana, Dominican Republic, with both on-site and fully virtual participation possible. The experience for on-site participants would closely approximate a normal pre-COVID *ACL conference, with 5-6 thematically organized parallel sessions and live Q/A and interactive discussion immediately after the talks. Presentations by virtual participants will be equitably interleaved with those of on-site participants, projected on the auditorium screens as if on-site, and also followed immediately by live Q/A and interactive discussion at a time during reasonable waking hours for the virtual presenter. For all participants, on-site and virtual, who are unable to attend a session due to either time-zone issues or because they are participating in another session live, talk recordings and slides will be available online at a minimum after the live presentation (and in many cases before as well), and questions may be submitted in advance on session-specific discussion boards and answered live in session with the usual visual aids if desired.

<iframe style="width:700px;height:400px" src="https://online.fliphtml5.com/ebtyf/ceby/"  seamless="seamless" scrolling="no" frameborder="0" allowtransparency="true" allowfullscreen="true" ></iframe>

Please Note: The EMNLP registration system is not currently connected to the underline site as we are still in the process of building out EMNLP 2021. You will receive access instructions from underline the week of November 1st. 

Access is given only to EMNLP upon registration, if you have not registered please do so [here](https://2021.emnlp.org/registration).

Registered attendees will receive access the week of November 1st.

EMNLP 2021

EMNLP 2021 is planned to be a hybrid event in Punta Cana, Dominican Republic, with both on-site and fully virtual participation possible.

In describing images, visual and linguistic processes coordinate with each other and they both proceed in a linear fashion. Three main theories about the nature of the relation between these processes and how they unfold over time have been proposed in the literature of 'linearization'. In this work, we investigate the hypotheses put forward by these theories utilizing a corpus of spoken image descriptions with speakers' eye-movement data. We explore the time alignment between the fixations on objects and the utterance of the corresponding nouns in the data. In contrast to previous studies, this dataset allows us to inspect unrestricted language production in the context of real scenes on a larger scale. We find both confirming and conflicting evidence for each of the theories in question, suggesting that the intricate relation between eye movements and language production may involve mechanisms proposed by all three theories.


poster

Join us for CogSci2022! Our 44th Annual Conference will be centered around the theme of diversity, with an accompanying range of keynotes and invited speakers who themselves highlight the diversity of our field. Our three symposia are Cognition across Diverse Populations, Cognition within Diverse Societies, and Global Challenges for Cognitive Science. The conference is being offered in a fully hybrid environment for onsite and online attendees.
<br>
<br>[**The Program at a Glance can be found here**](https://cognitivesciencesociety.org/wp-content/uploads/2022/07/Cogsci-2022-PAAG-Detailed-July-12final-1.pdf)
<br>[**The Detailed Program can be found here**](https://cognitivesciencesociety.org/wp-content/uploads/2022/08/DeArmond-CogSci2022-program-v113-final.pdf)
<br>
<br>

This conference requires a registration to attend. Please use the link below to secure your ticket. The website will be available to all registered attendees on July 25th. 

[Register to Attend](https://cognitivesciencesociety.org/registration/)

Registration Is Required

The 44th Annual Meeting of the Cognitive Science Society highlighted research on the theme Cognitive Diversity in addition to the full breadth and diversity of research topics offered by the society’s membership. 





How do children within the autism spectrum (ASD) integrate linguistic and visual information during real-time processing of sentences and dynamic scenes? We employed a visual word paradigm with sentences containing different verb types (causatives such as 'crack'; perception/psychological verbs such as 'examine') and with different scene contexts (agent in the scene moving towards a target object or remaining neutral). Participants were a group of 10-year old ASD and typically developing children (TD). We reasoned that the two dominant theories about ASD cognitive processing style—the weak central coherence theory (WCC) and the enhanced perceptual function (EPF)—make different predictions on how and where information about linguistic and visual information may be integrated. WCC predicts that ASD children would have greater difficulty with sentence and scene semantics while EPF predicts an early, post-perceptual integration via verb-specific and object (but not full scene) representations. The results suggest that ASD children differ from TD children by computing the link between verb and object faster, suggesting an enhanced post-perceptual integration.


Enhanced Visual-Linguistic Interaction in Children within the Autism Spectrum: Evidence from the Visual-World Paradigm with Dynamic Scenes

During speech perception, humans constantly sense multiple sources of information. As so, the McGurk effect has been typically conceived as a prototypical example of language multi-modal integration, specifically, it has been used to study audio-visual
integration during speech. The McGurk effect arises when in-congruent audio-visual stimuli are paired and perceived as a different syllable ( auditory /ba/ +
visual /ga/ = percept “da” ).

We developed a hierarchical computer model, based on self-organizing maps and Hebbian learning to study speech multi-modal integration. Our architecture allows studying the McGurk effect purely on bottom-up processing of audio-visual information.
We trained several versions of the model and measured the activation similarity between
McGurk alike stimuli and congruent ones by means of mutual information. Our results suggest that the illusory percept arises from the best congruent representation that reduces uncertainty. Furthermore, the reliability of each sensory modality determines the best congruent representation.


Multimodal integration as predictions. An explanation of the McGurk effect

Correlation judgments are at the core of belief formation. In previous studies of correlation judgment from 2D scatterplots, observers underestimate correlations, and display stronger underestimation biases when the scatterplot is shown in a landscape view than in a portrait view. Yet, it is unclear how these biases arise. Here, we propose that observers are Bayesian learners who perform “mental regression” using the observed data points in graph. Accordingly, judgment errors can arise from biased visual information sampling. We test our model’s predictions with two eye-tracking experiments and find that the Bayesian learning model, applied to information obtained from visual fixation data, replicates classic behavioral findings. The model also predicts trial-level estimation biases at a high accuracy level. Our study shows how computational models trained on process-level data can shed light on the cognitive mechanisms underlying belief formation, and yield theory-driven practical implications for data visualization and statistical communication.

Information sampling explains Bayesian learners’ biases in correlation judgment

The traveling salesperson problem (TSP) is an NP-Hard problem that computers find difficult to solve. Humans are surprisingly good at solving the TSP, with solutions within 10% of optimal for problems with up to 100 points, constructed in time linear with the number of points. We propose that humans solve the TSP by initially clustering the points and then connecting them first within and then between clusters. In this study, 67 participants first clustered 40 stimuli and then solved them as TSPs. Strikingly, participants' TSP solutions perfectly followed their clusters for 52% of the stimuli. Further, participants' TSP solutions' were more congruent with their clusters for stimuli with statistically higher levels of clustered structure. This provides strong evidence for the clustering proposal. Random TSP solutions, however, showed no such congruence to cluster structure. These findings suggest that clustering might be a fundamental ability for reasoning about graph-theoretic algorithmic problems.

Use of clustering in human solutions of the traveling salesperson problem

Humans are uniquely adept at extracting structure from the world around them. It is well known that people often form hierarchical task representations during learning, even when a task does not explicitly necessitate a hierarchical representation. Still, how individuals capitalize on this structure to facilitate behavior is an open question. In the present study, we address this question by carefully examining patterns of response time switch costs in a hierarchically structured visuomotor association learning task, to adjudicate between multiple models of behavior. We find evidence that participants do appear to navigate through a hierarchical representation of stimulus-response associations as they prepare responses, rather than maintaining a non-hierarchical, flat model or being primarily affected by changes in stimulus features. These results establish the existence of hierarchical mental representations even for static visuomotor mappings, and imply that such representations are internally navigated in an orderly manner during action selection.

Climbing the Tree: Structured Hierarchical Representations in Visuomotor Maps

Three-dimensional objects pose a challenge for our visual system, since we can only view objects from a single limited perspective at a given moment. Previous work found that given a limited perspective, infants represent 3D objects as complete volumes. Our study replicated this finding in 4- to 7-year-olds and adults, using an explicit prediction measure rather than looking times. We also explored whether humans have a bias to represent visually limited 3D objects as symmetrical rather than asymmetrical across shape, size, texture, and color. Overall, there was an above-chance preference for full volumetric and symmetrical object completion that increased with age. Low-level perceptual similarity of choices did not predict participants’ choices. Moreover, we evaluated ResNet-50 neural networks on the same tasks: they represented objects as complete volumes, but did not show substantial preference for symmetrical 3D representations. This raises the possibility that incorporating human symmetry biases could improve computer vision.

Three-Dimensional Object Completion in Humans and Computational Models

Because there are many situations in our daily life in which the option space is not discrete but continuous, recently developed decision models have been able to examine the cognitive processes underlying choice in laboratory tasks with a continuous outcome space. One of the most important of these continuous models is the circular diffusion model (CDM) by Smith, which has been shown to account for continuous space data from a wide range of paradigms, including color identification, orientation, brightness, pricing. However, in addition to the inherent complexity of this model, it has become more complex in order to predict reliable data patterns, making it a tool only for experts. Here we propose a more easy version of the CDM, the EZ version, to fit the model on continuous scale data. The EZ-CDM for continuous choice space tasks can estimate the parameter values for the cognitive processes underlying without considering the response time distribution but only using traditionally favored summary statistics (i.e. the mean and variance of response time, and angular variance of accuracy.) by simple formulas that can be computed easily and needs neither theoretical knowledge of model fitting nor much programming skills. Here, we formulate the EZ method and show that, despite being easy and fast to calculate, it’s performance in recovering true parameters is acceptable.

An EZ-circular diffusion model of continuous decision processes

Affective factors such as anxiety, confidence, and motivation can impair and enhance task performance. Here, we used drift diffusion modeling (DDM) to examine how these variables affect visualization, manipulation, and decision making on a mental rotation task (MRT). The effects of affective factors on visuospatial reasoning are largely unknown, perhaps in part because analyses are generally concerned with overall accuracy and reaction time (RT), without decomposing the stages of processing. With DDM, we decompose performance on a MRT into separate processing components, particularly the speed of information update (drift rate) and the amount of evidence accumulation (decision threshold). 106 adult participants performed a two-alternative forced-choice (2-AFC) MRT, and throughout, they rated their levels of anxiety, confidence, and motivation. We found that although anxiety, confidence, and motivation all impacted drift rate, only confidence affected the decision threshold. Moreover, we observed a unique role for confidence in mediating the links between gender and model parameters, as well as a unique moderating role of motivation in this mediation. Altogether, these findings shed light on the interrelations between affective factors in accounting for mental rotation performance in men and women, including the unique combination of confidence and motivation in explaining the gender difference in mental rotation performance.

Affective Factors Affect Visuospatial Decision-making: A Dri Diffusion Modeling Approach

To achieve goals, people leverage cognitive control to adjust how they process information. Here we show that frequent adjustments in information processing strategies (e.g., response threshold) within a single task give rise to reconfiguration costs. In two experiments we induced different performance goals in a Stroop task via explicit instruction or incentives, and these goals either varied or were fixed across different blocks. Across both experiments, we find smaller adjustments in control intensity when people frequently adjust the amount of control they exert, relative to blocks in which they don’t. We show that these results can be accounted for with a model that maximizes reward rate while minimizing reconfiguration costs (proportional to the Euclidean distances between the previous and current control signals). These findings suggest that cognitive control adjustments are regularized to constrain larger adjustments in control, which has important implications for computational modeling and measurement of motivated cognitive control.

Empirical and Computational Evidence for Reconfiguration Costs During Within-Task Adjustments in Cognitive Control

People are often reluctant to reconsider their choices, sticking with their goals even when it is clear that they would be better off abandoning them. Explanations for this abound, including loss aversion, sunk costs, social and reputational pressures, and resource rational consideration of the costs of replanning. Here we propose another hypothesis: In adopting a goal, you immediately reap the rewards of gaining information about what to do and how to act. Insofar as goals are rewarding in themselves, we predict that unless a goal is specifically devalued or the costs associated with it are very high, the default is not to engage in any reconsideration at all. We test this hypothesis by creating a stripped down scenario involving choices between two goals with transparently obvious cost differentials. The task is designed to minimize other factors that might contribute to goal persistence and indeed, we test both adults and very young children on virtually the same task to ensure that the cognitive load for adults is negligible. Both adults (Experiments 1-2) and 4-6-year-old children (Experiments 3-4) choose the less costly of two goals when shown the costs and goals together. However, when participants are shown the goals first, and only then shown that their chosen goal is more costly than the alternative, participants stick with higher cost goals, unless the goals are explicitly devalued.

Premium content

Downloads

Next from CogSci 2022

Enhanced Visual-Linguistic Interaction in Children within the Autism Spectrum: Evidence from the Visual-World Paradigm with Dynamic Scenes

Similar lecture

Comparing Grammatical Theories of Code-Mixing