View in EDS PDF Full Text

Imperfect Captions Paired to Audiovisual Stimuli Improve Speech Recognition but Do Not Reduce Listening-Related Fatigue.

Saved in:
Bibliographic Details
Title: Imperfect Captions Paired to Audiovisual Stimuli Improve Speech Recognition but Do Not Reduce Listening-Related Fatigue.
Authors: Esser, Katie1, Picou, Erin M.2, Hornsby, Benjamin W. Y.2 ben.hornsby@vumc.org
Source: American Journal of Audiology. Sep2025, Vol. 34 Issue 3, p661-675. 15p.
Subject Terms: *Audiovisual materials, *Cognitive testing, *Data analysis, *Fatigue (Physiology), *Listening, *Audiometry, *Pre-tests & post-tests, *Memory, *Speech perception, *Automation, *Hearing disorders, *Comparative studies, *Adults, Automatic speech recognition, Sensory stimulation, Task performance, Ecology, Research funding, Speech, Statistical sampling, Empirical research, Treatment effectiveness, Randomized controlled trials, Descriptive statistics, Analysis of variance, Statistics, Data analysis software
Abstract: Purpose: Understanding speech in challenging environments can be fatiguing for individuals with and without hearing loss. Empirical research examining interventions to reduce such fatigue, however, is limited. Our study investigated the effects of imperfect captions, such as those created using automated speech recognition systems, on listening-related fatigue during a challenging speech task. Method: Twenty-two adults (aged 18--63 years) with essentially normal hearing completed a sustained dual task designed to induce listening-related fatigue. The primary task was audiovisual sentence recognition in quiet, with speech levels individually adjusted for ~70% correct performance (without captions). The secondary task was response time (RT) to topic words presented during the speech task. Participants completed the dual task with or without captions. Subjective fatigue ratings were obtained before, during, and after the dual task. Vigilant attention was measured via the secondary task and before and after the dual task using visual stimuli (RTs to a visual marker presented at random intervals). Results: Subjective fatigue ratings increased significantly over the course of the dual task. However, this increase was larger for the group who had captions, even though they had better sentence recognition overall. Evidence of behavioral fatigue (slowed RTs over time) was also present but only for those in the Caption group. Anecdotal reports from study participants suggest that the increased fatigue for the Caption group was related to the cognitive challenge of combining information from the time-locked audiovisual cues and the time-delayed, and imperfect, text captions. Conclusions: Even though they were time-delayed and contained inaccuracies, captions improved speech recognition. However, this benefit was accompanied by greater increases in subjective and behavioral fatigue. Therefore, in some conditions imperfect captions can have negative consequences for listening-related fatigue. Further research is needed to determine whether this pattern holds in different circumstances, such as with audio-only stimuli. Supplemental Material: https://doi.org/10.23641/asha.29646482 [ABSTRACT FROM AUTHOR]
Copyright of American Journal of Audiology is the property of American Speech-Language-Hearing Association and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)
Database: Education Research Complete
FullText Links:
  – Type: pdflink
Text:
  Availability: 0
Header DbId: ehh
DbLabel: Education Research Complete
An: 187706690
AccessLevel: 6
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Imperfect Captions Paired to Audiovisual Stimuli Improve Speech Recognition but Do Not Reduce Listening-Related Fatigue.
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Esser%2C+Katie%22">Esser, Katie</searchLink><relatesTo>1</relatesTo><br /><searchLink fieldCode="AR" term="%22Picou%2C+Erin+M%2E%22">Picou, Erin M.</searchLink><relatesTo>2</relatesTo><br /><searchLink fieldCode="AR" term="%22Hornsby%2C+Benjamin+W%2E+Y%2E%22">Hornsby, Benjamin W. Y.</searchLink><relatesTo>2</relatesTo><i> ben.hornsby@vumc.org</i>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="JN" term="%22American+Journal+of+Audiology%22">American Journal of Audiology</searchLink>. Sep2025, Vol. 34 Issue 3, p661-675. 15p.
– Name: Subject
  Label: Subject Terms
  Group: Su
  Data: *<searchLink fieldCode="DE" term="%22Audiovisual+materials%22">Audiovisual materials</searchLink><br />*<searchLink fieldCode="DE" term="%22Cognitive+testing%22">Cognitive testing</searchLink><br />*<searchLink fieldCode="DE" term="%22Data+analysis%22">Data analysis</searchLink><br />*<searchLink fieldCode="DE" term="%22Fatigue+%28Physiology%29%22">Fatigue (Physiology)</searchLink><br />*<searchLink fieldCode="DE" term="%22Listening%22">Listening</searchLink><br />*<searchLink fieldCode="DE" term="%22Audiometry%22">Audiometry</searchLink><br />*<searchLink fieldCode="DE" term="%22Pre-tests+%26+post-tests%22">Pre-tests & post-tests</searchLink><br />*<searchLink fieldCode="DE" term="%22Memory%22">Memory</searchLink><br />*<searchLink fieldCode="DE" term="%22Speech+perception%22">Speech perception</searchLink><br />*<searchLink fieldCode="DE" term="%22Automation%22">Automation</searchLink><br />*<searchLink fieldCode="DE" term="%22Hearing+disorders%22">Hearing disorders</searchLink><br />*<searchLink fieldCode="DE" term="%22Comparative+studies%22">Comparative studies</searchLink><br />*<searchLink fieldCode="DE" term="%22Adults%22">Adults</searchLink><br /><searchLink fieldCode="DE" term="%22Automatic+speech+recognition%22">Automatic speech recognition</searchLink><br /><searchLink fieldCode="DE" term="%22Sensory+stimulation%22">Sensory stimulation</searchLink><br /><searchLink fieldCode="DE" term="%22Task+performance%22">Task performance</searchLink><br /><searchLink fieldCode="DE" term="%22Ecology%22">Ecology</searchLink><br /><searchLink fieldCode="DE" term="%22Research+funding%22">Research funding</searchLink><br /><searchLink fieldCode="DE" term="%22Speech%22">Speech</searchLink><br /><searchLink fieldCode="DE" term="%22Statistical+sampling%22">Statistical sampling</searchLink><br /><searchLink fieldCode="DE" term="%22Empirical+research%22">Empirical research</searchLink><br /><searchLink fieldCode="DE" term="%22Treatment+effectiveness%22">Treatment effectiveness</searchLink><br /><searchLink fieldCode="DE" term="%22Randomized+controlled+trials%22">Randomized controlled trials</searchLink><br /><searchLink fieldCode="DE" term="%22Descriptive+statistics%22">Descriptive statistics</searchLink><br /><searchLink fieldCode="DE" term="%22Analysis+of+variance%22">Analysis of variance</searchLink><br /><searchLink fieldCode="DE" term="%22Statistics%22">Statistics</searchLink><br /><searchLink fieldCode="DE" term="%22Data+analysis+software%22">Data analysis software</searchLink>
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Purpose: Understanding speech in challenging environments can be fatiguing for individuals with and without hearing loss. Empirical research examining interventions to reduce such fatigue, however, is limited. Our study investigated the effects of imperfect captions, such as those created using automated speech recognition systems, on listening-related fatigue during a challenging speech task. Method: Twenty-two adults (aged 18--63 years) with essentially normal hearing completed a sustained dual task designed to induce listening-related fatigue. The primary task was audiovisual sentence recognition in quiet, with speech levels individually adjusted for ~70% correct performance (without captions). The secondary task was response time (RT) to topic words presented during the speech task. Participants completed the dual task with or without captions. Subjective fatigue ratings were obtained before, during, and after the dual task. Vigilant attention was measured via the secondary task and before and after the dual task using visual stimuli (RTs to a visual marker presented at random intervals). Results: Subjective fatigue ratings increased significantly over the course of the dual task. However, this increase was larger for the group who had captions, even though they had better sentence recognition overall. Evidence of behavioral fatigue (slowed RTs over time) was also present but only for those in the Caption group. Anecdotal reports from study participants suggest that the increased fatigue for the Caption group was related to the cognitive challenge of combining information from the time-locked audiovisual cues and the time-delayed, and imperfect, text captions. Conclusions: Even though they were time-delayed and contained inaccuracies, captions improved speech recognition. However, this benefit was accompanied by greater increases in subjective and behavioral fatigue. Therefore, in some conditions imperfect captions can have negative consequences for listening-related fatigue. Further research is needed to determine whether this pattern holds in different circumstances, such as with audio-only stimuli. Supplemental Material: https://doi.org/10.23641/asha.29646482 [ABSTRACT FROM AUTHOR]
– Name: AbstractSuppliedCopyright
  Label:
  Group: Ab
  Data: <i>Copyright of American Journal of Audiology is the property of American Speech-Language-Hearing Association and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract.</i> (Copyright applies to all Abstracts.)
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=ehh&AN=187706690
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1044/2025_AJA-25-00018
    Languages:
      – Code: eng
        Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 15
        StartPage: 661
    Subjects:
      – SubjectFull: Audiovisual materials
        Type: general
      – SubjectFull: Cognitive testing
        Type: general
      – SubjectFull: Data analysis
        Type: general
      – SubjectFull: Fatigue (Physiology)
        Type: general
      – SubjectFull: Listening
        Type: general
      – SubjectFull: Audiometry
        Type: general
      – SubjectFull: Pre-tests & post-tests
        Type: general
      – SubjectFull: Memory
        Type: general
      – SubjectFull: Speech perception
        Type: general
      – SubjectFull: Automation
        Type: general
      – SubjectFull: Hearing disorders
        Type: general
      – SubjectFull: Comparative studies
        Type: general
      – SubjectFull: Adults
        Type: general
      – SubjectFull: Automatic speech recognition
        Type: general
      – SubjectFull: Sensory stimulation
        Type: general
      – SubjectFull: Task performance
        Type: general
      – SubjectFull: Ecology
        Type: general
      – SubjectFull: Research funding
        Type: general
      – SubjectFull: Speech
        Type: general
      – SubjectFull: Statistical sampling
        Type: general
      – SubjectFull: Empirical research
        Type: general
      – SubjectFull: Treatment effectiveness
        Type: general
      – SubjectFull: Randomized controlled trials
        Type: general
      – SubjectFull: Descriptive statistics
        Type: general
      – SubjectFull: Analysis of variance
        Type: general
      – SubjectFull: Statistics
        Type: general
      – SubjectFull: Data analysis software
        Type: general
    Titles:
      – TitleFull: Imperfect Captions Paired to Audiovisual Stimuli Improve Speech Recognition but Do Not Reduce Listening-Related Fatigue.
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Esser, Katie
      – PersonEntity:
          Name:
            NameFull: Picou, Erin M.
      – PersonEntity:
          Name:
            NameFull: Hornsby, Benjamin W. Y.
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 09
              Text: Sep2025
              Type: published
              Y: 2025
          Identifiers:
            – Type: issn-print
              Value: 10590889
          Numbering:
            – Type: volume
              Value: 34
            – Type: issue
              Value: 3
          Titles:
            – TitleFull: American Journal of Audiology
              Type: main
ResultId 1