View in EDS PDF Full Text

Automated Scoring of the Speech Intelligibility Test Using Autoscore.

Saved in:
Bibliographic Details
Title: Automated Scoring of the Speech Intelligibility Test Using Autoscore.
Authors: Stipancic, Kaila L.1 klstip@buffalo.edu, Barrett, Tyson S.2, Tjaden, Kris1, Borrie, Stephanie A.2
Source: American Journal of Speech-Language Pathology. 2025 Supplement, Vol. 34, p2397-2408. 12p.
Subject Terms: *Dysarthria, *Computer software, *Intelligibility of speech, *Speech evaluation, *Automation, *Computer assisted instruction, *Speech perception, Research funding, Questionnaires, Descriptive statistics, Confidence intervals
Abstract: Purpose: The purpose of the current study was to develop and test extensions to Autoscore, an automated approach for scoring listener transcriptions against target stimuli, for scoring the Speech Intelligibility Test (SIT), a widely used test for quantifying intelligibility in individuals with dysarthria. Method: Three main extensions to Autoscore were created including a compound rule, a contractions rule, and a numbers rule. We used two sets of previously collected listener SIT transcripts (N = 4,642) from databases of dysarthric speakers to evaluate the accuracy of the Autoscore SIT extensions. A human scorer and SIT-extended Autoscore were used to score sentence transcripts in both data sets. Scoring performance was determined by (a) comparing Autoscore and human scores using intraclass correlations (ICCs) at individual sentence and speaker levels and (b) comparing SIT-extended Autoscore performance to the original Autoscore with ICCs. Results: At both the individual sentence and speaker levels, Autoscore and the human scorer were nearly identical for both Data Set 1 (ICC = .9922 and ICC = .9767, respectively) and Data Set 2 (ICC = .9934 and ICC = .9946, respectively). Where disagreements between Autoscore and a human scorer occurred, the differences were often small (i.e., within 1 or 2 points). Across the two data sets (N = 4,642 sentences), SIT-extended Autoscore rendered 510 disagreements with the human scorer (vs. 571 disagreements for the original Autoscore). Discussion: Overall, SIT-extended Autoscore performed as well as human scorers and substantially improved scoring accuracy relative to the original version of Autoscore. Coupled with the substantial time and effort saving provided by Autoscore, its utility has been strengthened by the extensions developed and tested here. [ABSTRACT FROM AUTHOR]
Copyright of American Journal of Speech-Language Pathology is the property of American Speech-Language-Hearing Association and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)
Database: Education Research Complete
FullText Links:
  – Type: pdflink
Text:
  Availability: 0
Header DbId: ehh
DbLabel: Education Research Complete
An: 187102566
AccessLevel: 6
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Automated Scoring of the Speech Intelligibility Test Using Autoscore.
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Stipancic%2C+Kaila+L%2E%22">Stipancic, Kaila L.</searchLink><relatesTo>1</relatesTo><i> klstip@buffalo.edu</i><br /><searchLink fieldCode="AR" term="%22Barrett%2C+Tyson+S%2E%22">Barrett, Tyson S.</searchLink><relatesTo>2</relatesTo><br /><searchLink fieldCode="AR" term="%22Tjaden%2C+Kris%22">Tjaden, Kris</searchLink><relatesTo>1</relatesTo><br /><searchLink fieldCode="AR" term="%22Borrie%2C+Stephanie+A%2E%22">Borrie, Stephanie A.</searchLink><relatesTo>2</relatesTo>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="JN" term="%22American+Journal+of+Speech-Language+Pathology%22">American Journal of Speech-Language Pathology</searchLink>. 2025 Supplement, Vol. 34, p2397-2408. 12p.
– Name: Subject
  Label: Subject Terms
  Group: Su
  Data: *<searchLink fieldCode="DE" term="%22Dysarthria%22">Dysarthria</searchLink><br />*<searchLink fieldCode="DE" term="%22Computer+software%22">Computer software</searchLink><br />*<searchLink fieldCode="DE" term="%22Intelligibility+of+speech%22">Intelligibility of speech</searchLink><br />*<searchLink fieldCode="DE" term="%22Speech+evaluation%22">Speech evaluation</searchLink><br />*<searchLink fieldCode="DE" term="%22Automation%22">Automation</searchLink><br />*<searchLink fieldCode="DE" term="%22Computer+assisted+instruction%22">Computer assisted instruction</searchLink><br />*<searchLink fieldCode="DE" term="%22Speech+perception%22">Speech perception</searchLink><br /><searchLink fieldCode="DE" term="%22Research+funding%22">Research funding</searchLink><br /><searchLink fieldCode="DE" term="%22Questionnaires%22">Questionnaires</searchLink><br /><searchLink fieldCode="DE" term="%22Descriptive+statistics%22">Descriptive statistics</searchLink><br /><searchLink fieldCode="DE" term="%22Confidence+intervals%22">Confidence intervals</searchLink>
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Purpose: The purpose of the current study was to develop and test extensions to Autoscore, an automated approach for scoring listener transcriptions against target stimuli, for scoring the Speech Intelligibility Test (SIT), a widely used test for quantifying intelligibility in individuals with dysarthria. Method: Three main extensions to Autoscore were created including a compound rule, a contractions rule, and a numbers rule. We used two sets of previously collected listener SIT transcripts (N = 4,642) from databases of dysarthric speakers to evaluate the accuracy of the Autoscore SIT extensions. A human scorer and SIT-extended Autoscore were used to score sentence transcripts in both data sets. Scoring performance was determined by (a) comparing Autoscore and human scores using intraclass correlations (ICCs) at individual sentence and speaker levels and (b) comparing SIT-extended Autoscore performance to the original Autoscore with ICCs. Results: At both the individual sentence and speaker levels, Autoscore and the human scorer were nearly identical for both Data Set 1 (ICC = .9922 and ICC = .9767, respectively) and Data Set 2 (ICC = .9934 and ICC = .9946, respectively). Where disagreements between Autoscore and a human scorer occurred, the differences were often small (i.e., within 1 or 2 points). Across the two data sets (N = 4,642 sentences), SIT-extended Autoscore rendered 510 disagreements with the human scorer (vs. 571 disagreements for the original Autoscore). Discussion: Overall, SIT-extended Autoscore performed as well as human scorers and substantially improved scoring accuracy relative to the original version of Autoscore. Coupled with the substantial time and effort saving provided by Autoscore, its utility has been strengthened by the extensions developed and tested here. [ABSTRACT FROM AUTHOR]
– Name: AbstractSuppliedCopyright
  Label:
  Group: Ab
  Data: <i>Copyright of American Journal of Speech-Language Pathology is the property of American Speech-Language-Hearing Association and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract.</i> (Copyright applies to all Abstracts.)
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=ehh&AN=187102566
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1044/2024_AJSLP-24-00276
    Languages:
      – Code: eng
        Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 12
        StartPage: 2397
    Subjects:
      – SubjectFull: Dysarthria
        Type: general
      – SubjectFull: Computer software
        Type: general
      – SubjectFull: Intelligibility of speech
        Type: general
      – SubjectFull: Speech evaluation
        Type: general
      – SubjectFull: Automation
        Type: general
      – SubjectFull: Computer assisted instruction
        Type: general
      – SubjectFull: Speech perception
        Type: general
      – SubjectFull: Research funding
        Type: general
      – SubjectFull: Questionnaires
        Type: general
      – SubjectFull: Descriptive statistics
        Type: general
      – SubjectFull: Confidence intervals
        Type: general
    Titles:
      – TitleFull: Automated Scoring of the Speech Intelligibility Test Using Autoscore.
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Stipancic, Kaila L.
      – PersonEntity:
          Name:
            NameFull: Barrett, Tyson S.
      – PersonEntity:
          Name:
            NameFull: Tjaden, Kris
      – PersonEntity:
          Name:
            NameFull: Borrie, Stephanie A.
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 02
              M: 07
              Text: 2025 Supplement
              Type: published
              Y: 2025
          Identifiers:
            – Type: issn-print
              Value: 10580360
          Numbering:
            – Type: volume
              Value: 34
          Titles:
            – TitleFull: American Journal of Speech-Language Pathology
              Type: main
ResultId 1