View in EDS

Evaluating Popular MOOC Platforms by Generative Artificial Intelligence (AI) Robots: How Consistent Are the Robots?

Saved in:
Bibliographic Details
Title: Evaluating Popular MOOC Platforms by Generative Artificial Intelligence (AI) Robots: How Consistent Are the Robots?
Language: English
Authors: Victor K. Y. Chan
Source: International Association for Development of the Information Society. 2023.
Availability: International Association for the Development of the Information Society. e-mail: secretariat@iadis.org; Web site: http://www.iadisportal.org
Peer Reviewed: Y
Page Count: 8
Publication Date: 2023
Document Type: Speeches/Meeting Papers
Reports - Research
Descriptors: MOOCs, Robotics, Scores, Learning Management Systems, Learner Engagement, Cost Effectiveness, Interpersonal Relationship, Artificial Intelligence, Computer Software Evaluation, Correlation, Instructional Design, Evaluation Methods
Abstract: This article intends to investigate the consistency between a few popular generative AI robots in the evaluation of massive open online course (MOOC) platforms. The four robots experimented with in the study were Claude+, GPT-4, Sage, and Dragonfly, which were tasked with awarding rating scores to the eight major dimensions, namely (1) content/course quality, (2) pedagogical design, (3) learner support, (4) technology infrastructure, (5) social interaction, (6) learner engagement, (7) instructor support, and (8) cost-effectiveness, of the 31 currently very popular MOOC platforms. Only Claude+'s and Dragonfly's rating scores turned out to be amenable to statistical analysis. For each of the two robots, the minimum, the maximum, the range, and the standard deviation of the rating scores for each of the eight dimensions were computed across all the 31 MOOC platforms. The rating score difference for each of the eight dimensions between the two robots was calculated for each platform. The mean of the absolute value, the minimum, the maximum, the range, and the standard deviation of the differences for each dimension between the two robots were calculated across all platforms. A paired sample t-test was then applied to each dimension for the rating score difference between the two robots over all the platforms. Finally, a correlation coefficient of the rating scores was computed for each of the eight dimensions between the two robots across all the MOOC platforms. The computational results were to reveal whether the two robots awarded discrimination in evaluating each dimension across the platforms, whether any of the two robots systematically underrated or overrated any dimension with respect to the other robot, and whether there was consistency between the two robots in evaluating each dimension across the platforms. It was found that discrimination was prominent in the evaluation of all dimensions save Dragonfly's rating of the dimensions learner support, technology infrastructure, and instructor support, Claude+ systematically underrated all dimensions (p < 0.000 < 0.05) compared with Dragonfly except for the dimension cost-effectiveness, which Claude+ systematically overrated (p = 0.003 < 0.05), and the evaluation by the two robots was consistent only for the dimensions content/course quality, pedagogical design, and learner engagement with the correlation coefficients ranging from 0.445 to 0.632 (p from 0.000 to 0.012 < 0.05). Consistency implies at least the partial trustworthiness of the evaluation of these MOOC platforms by either of these two popular generative AI robots based on the analogous concept of convergent validity for an operationalized instrument to measure an abstract construct. [For the full proceedings, see ED636095.]
Abstractor: As Provided
Entry Date: 2024
Accession Number: ED636613
Database: ERIC
FullText Text:
  Availability: 0
CustomLinks:
  – Url: https://eric.ed.gov/contentdelivery/servlet/ERICServlet?accno=ED636613
    Name: ERIC Full Text
    Category: fullText
    Text: Full Text from ERIC
Header DbId: eric
DbLabel: ERIC
An: ED636613
AccessLevel: 3
PubType: Conference
PubTypeId: conference
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Evaluating Popular MOOC Platforms by Generative Artificial Intelligence (AI) Robots: How Consistent Are the Robots?
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: &lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Victor+K%2E+Y%2E+Chan%22&quot;&gt;Victor K. Y. Chan&lt;/searchLink&gt;
– Name: TitleSource
  Label: Source
  Group: Src
  Data: &lt;searchLink fieldCode=&quot;SO&quot; term=&quot;%22International+Association+for+Development+of+the+Information+Society%22&quot;&gt;&lt;i&gt;International Association for Development of the Information Society&lt;/i&gt;&lt;/searchLink&gt;. 2023.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: International Association for the Development of the Information Society. e-mail: secretariat@iadis.org; Web site: http://www.iadisportal.org
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 8
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2023
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Speeches/Meeting Papers&lt;br /&gt;Reports - Research
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: &lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22MOOCs%22&quot;&gt;MOOCs&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22Robotics%22&quot;&gt;Robotics&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22Scores%22&quot;&gt;Scores&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22Learning+Management+Systems%22&quot;&gt;Learning Management Systems&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22Learner+Engagement%22&quot;&gt;Learner Engagement&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22Cost+Effectiveness%22&quot;&gt;Cost Effectiveness&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22Interpersonal+Relationship%22&quot;&gt;Interpersonal Relationship&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22Artificial+Intelligence%22&quot;&gt;Artificial Intelligence&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22Computer+Software+Evaluation%22&quot;&gt;Computer Software Evaluation&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22Correlation%22&quot;&gt;Correlation&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22Instructional+Design%22&quot;&gt;Instructional Design&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22Evaluation+Methods%22&quot;&gt;Evaluation Methods&lt;/searchLink&gt;
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: This article intends to investigate the consistency between a few popular generative AI robots in the evaluation of massive open online course (MOOC) platforms. The four robots experimented with in the study were Claude+, GPT-4, Sage, and Dragonfly, which were tasked with awarding rating scores to the eight major dimensions, namely (1) content/course quality, (2) pedagogical design, (3) learner support, (4) technology infrastructure, (5) social interaction, (6) learner engagement, (7) instructor support, and (8) cost-effectiveness, of the 31 currently very popular MOOC platforms. Only Claude+&#39;s and Dragonfly&#39;s rating scores turned out to be amenable to statistical analysis. For each of the two robots, the minimum, the maximum, the range, and the standard deviation of the rating scores for each of the eight dimensions were computed across all the 31 MOOC platforms. The rating score difference for each of the eight dimensions between the two robots was calculated for each platform. The mean of the absolute value, the minimum, the maximum, the range, and the standard deviation of the differences for each dimension between the two robots were calculated across all platforms. A paired sample t-test was then applied to each dimension for the rating score difference between the two robots over all the platforms. Finally, a correlation coefficient of the rating scores was computed for each of the eight dimensions between the two robots across all the MOOC platforms. The computational results were to reveal whether the two robots awarded discrimination in evaluating each dimension across the platforms, whether any of the two robots systematically underrated or overrated any dimension with respect to the other robot, and whether there was consistency between the two robots in evaluating each dimension across the platforms. It was found that discrimination was prominent in the evaluation of all dimensions save Dragonfly&#39;s rating of the dimensions learner support, technology infrastructure, and instructor support, Claude+ systematically underrated all dimensions (p &lt; 0.000 &lt; 0.05) compared with Dragonfly except for the dimension cost-effectiveness, which Claude+ systematically overrated (p = 0.003 &lt; 0.05), and the evaluation by the two robots was consistent only for the dimensions content/course quality, pedagogical design, and learner engagement with the correlation coefficients ranging from 0.445 to 0.632 (p from 0.000 to 0.012 &lt; 0.05). Consistency implies at least the partial trustworthiness of the evaluation of these MOOC platforms by either of these two popular generative AI robots based on the analogous concept of convergent validity for an operationalized instrument to measure an abstract construct. [For the full proceedings, see ED636095.]
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2024
– Name: AN
  Label: Accession Number
  Group: ID
  Data: ED636613
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=ED636613
RecordInfo BibRecord:
  BibEntity:
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 8
    Subjects:
      – SubjectFull: MOOCs
        Type: general
      – SubjectFull: Robotics
        Type: general
      – SubjectFull: Scores
        Type: general
      – SubjectFull: Learning Management Systems
        Type: general
      – SubjectFull: Learner Engagement
        Type: general
      – SubjectFull: Cost Effectiveness
        Type: general
      – SubjectFull: Interpersonal Relationship
        Type: general
      – SubjectFull: Artificial Intelligence
        Type: general
      – SubjectFull: Computer Software Evaluation
        Type: general
      – SubjectFull: Correlation
        Type: general
      – SubjectFull: Instructional Design
        Type: general
      – SubjectFull: Evaluation Methods
        Type: general
    Titles:
      – TitleFull: Evaluating Popular MOOC Platforms by Generative Artificial Intelligence (AI) Robots: How Consistent Are the Robots?
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Victor K. Y. Chan
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 01
              Type: published
              Y: 2023
          Titles:
            – TitleFull: International Association for Development of the Information Society
              Type: main
ResultId 1