A Computational Framework to Study Hierarchical Processing in Visual Narratives
Saved in:
| Title: | A Computational Framework to Study Hierarchical Processing in Visual Narratives |
|---|---|
| Language: | English |
| Authors: | Aditya Upadhyayula, Neil Cohn |
| Source: | Cognitive Science. 2025 49(5). |
| Availability: | Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us |
| Peer Reviewed: | Y |
| Page Count: | 29 |
| Publication Date: | 2025 |
| Document Type: | Journal Articles Reports - Research |
| Education Level: | Higher Education Postsecondary Education |
| Descriptors: | Visual Perception, Comprehension, Vertical Organization, Story Grammar, Preferences, Models, Markov Processes, Cartoons, Undergraduate Students |
| Geographic Terms: | Massachusetts |
| DOI: | 10.1111/cogs.70050 |
| ISSN: | 0364-0213 1551-6709 |
| Abstract: | Theories of visual narrative comprehension have advocated for a hierarchical grammar-based comprehension mechanism, but only limited work has investigated this hierarchy. Here, we provide a computational framework inspired by computational psycholinguistics to address hierarchy in visual narratives. The predictions generated by this framework were compared against behavior data to draw inferences about the hierarchical properties of visual narratives. A segmentation task--where participants ranked all possible segmental boundaries--demonstrated that participants' preferences were predicted by visual narrative grammar. Three kinds of models using surprisal theory--an Earley parser, a hidden Markov model (HMM), and an n-gram model--were then used to generate segmentation preferences for the same task. Earley parser's preferences were based on a hierarchical grammar with recursion properties, while the HMM and the n-grams used a flattened grammar for visual narrative comprehension. Given the differences in the mechanics of these models, contrasting their predictions against behavior data could provide crucial insights into understanding the underlying mechanisms of visual narrative comprehension. By investigating grammatical systems outside of language, this research provides new directions to explore the generic makeup of the cognitive structure of mental representations. |
| Abstractor: | As Provided |
| Notes: | https://osf.io/s2h5x |
| Entry Date: | 2025 |
| Accession Number: | EJ1472084 |
| Database: | ERIC |
|
Full text is not displayed to guests.
Login for full access.
|
|
| Abstract: | Theories of visual narrative comprehension have advocated for a hierarchical grammar-based comprehension mechanism, but only limited work has investigated this hierarchy. Here, we provide a computational framework inspired by computational psycholinguistics to address hierarchy in visual narratives. The predictions generated by this framework were compared against behavior data to draw inferences about the hierarchical properties of visual narratives. A segmentation task--where participants ranked all possible segmental boundaries--demonstrated that participants' preferences were predicted by visual narrative grammar. Three kinds of models using surprisal theory--an Earley parser, a hidden Markov model (HMM), and an n-gram model--were then used to generate segmentation preferences for the same task. Earley parser's preferences were based on a hierarchical grammar with recursion properties, while the HMM and the n-grams used a flattened grammar for visual narrative comprehension. Given the differences in the mechanics of these models, contrasting their predictions against behavior data could provide crucial insights into understanding the underlying mechanisms of visual narrative comprehension. By investigating grammatical systems outside of language, this research provides new directions to explore the generic makeup of the cognitive structure of mental representations. |
|---|---|
| ISSN: | 0364-0213 1551-6709 |
| DOI: | 10.1111/cogs.70050 |
