View in EDS

User-Centric Evaluation of Recommender Systems in Social Learning Platforms: Accuracy is Just the Tip of the Iceberg

Saved in:
Bibliographic Details
Title: User-Centric Evaluation of Recommender Systems in Social Learning Platforms: Accuracy is Just the Tip of the Iceberg
Language: English
Authors: Fazeli, Soude (ORCID 0000-0003-1250-994X), Drachsler, Hendrik, Bitter-Rijpkema, Marlies, Brouns, Francis (ORCID 0000-0002-6240-2684), Brouns, Wim van der Vegt, Sloep, Peter B.
Source: IEEE Transactions on Learning Technologies. Jul-Sep 2018 11(3):294-306.
Availability: Institute of Electrical and Electronics Engineers, Inc. 445 Hoes Lane, Piscataway, NJ 08854. Tel: 732-981-0060; Web site: http://ieeexplore.ieee.org/xpl/RecentIssue.jsp?punumber=4620076
Peer Reviewed: Y
Page Count: 13
Publication Date: 2018
Document Type: Journal Articles
Reports - Research
Tests/Questionnaires
Descriptors: Evaluation, Socialization, Accuracy, Prediction, Neighborhoods, Memory, Mathematics, Foreign Countries, User Satisfaction (Information), Information Systems
Geographic Terms: Greece, Netherlands, Romania, United Kingdom, Cyprus, Germany, Serbia, Bulgaria, Croatia, Estonia, Ireland, Lithuania, Poland, Portugal, Spain
DOI: 10.1109/TLT.2017.2732349
ISSN: 1939-1382
Abstract: Recommender systems provide users with content they might be interested in. Conventionally, recommender systems are evaluated mostly by using prediction accuracy metrics only. But, the ultimate goal of a recommender system is to increase user satisfaction. Therefore, evaluations that measure user satisfaction should also be performed before deploying a recommender system in a real target environment. Such evaluations are laborious and complicated compared to the traditional, data-centric evaluations, though. In this study, we carried out a user-centric evaluation of state-of-the-art recommender systems as well as a graph-based approach in the ecologically valid setting of an authentic social learning platform. We also conducted a data-centric evaluation on the same data to investigate the added value of user-centric evaluations and how user satisfaction of a recommender system is related to its performance in terms of accuracy metrics. Our findings suggest that user-centric evaluation results are not necessarily in line with data-centric evaluation results. We conclude that the traditional evaluation of recommender systems in terms of prediction accuracy only does not suffice to judge performance of recommender systems on the user side. Moreover, the user-centric evaluation provides valuable insights in how candidate algorithms perform on each of the five quality metrics for recommendations: usefulness, accuracy, novelty, diversity, and serendipity.
Abstractor: As Provided
Number of References: 40
Entry Date: 2018
Accession Number: EJ1192608
Database: ERIC
FullText Text:
  Availability: 0
Header DbId: eric
DbLabel: ERIC
An: EJ1192608
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: User-Centric Evaluation of Recommender Systems in Social Learning Platforms: Accuracy is Just the Tip of the Iceberg
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Fazeli%2C+Soude%22">Fazeli, Soude</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0003-1250-994X">0000-0003-1250-994X</externalLink>)<br /><searchLink fieldCode="AR" term="%22Drachsler%2C+Hendrik%22">Drachsler, Hendrik</searchLink><br /><searchLink fieldCode="AR" term="%22Bitter-Rijpkema%2C+Marlies%22">Bitter-Rijpkema, Marlies</searchLink><br /><searchLink fieldCode="AR" term="%22Brouns%2C+Francis%22">Brouns, Francis</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0002-6240-2684">0000-0002-6240-2684</externalLink>)<br /><searchLink fieldCode="AR" term="%22Brouns%2C+Wim+van+der+Vegt%22">Brouns, Wim van der Vegt</searchLink><br /><searchLink fieldCode="AR" term="%22Sloep%2C+Peter+B%2E%22">Sloep, Peter B.</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22IEEE+Transactions+on+Learning+Technologies%22"><i>IEEE Transactions on Learning Technologies</i></searchLink>. Jul-Sep 2018 11(3):294-306.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Institute of Electrical and Electronics Engineers, Inc. 445 Hoes Lane, Piscataway, NJ 08854. Tel: 732-981-0060; Web site: http://ieeexplore.ieee.org/xpl/RecentIssue.jsp?punumber=4620076
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 13
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2018
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research<br />Tests/Questionnaires
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Evaluation%22">Evaluation</searchLink><br /><searchLink fieldCode="DE" term="%22Socialization%22">Socialization</searchLink><br /><searchLink fieldCode="DE" term="%22Accuracy%22">Accuracy</searchLink><br /><searchLink fieldCode="DE" term="%22Prediction%22">Prediction</searchLink><br /><searchLink fieldCode="DE" term="%22Neighborhoods%22">Neighborhoods</searchLink><br /><searchLink fieldCode="DE" term="%22Memory%22">Memory</searchLink><br /><searchLink fieldCode="DE" term="%22Mathematics%22">Mathematics</searchLink><br /><searchLink fieldCode="DE" term="%22Foreign+Countries%22">Foreign Countries</searchLink><br /><searchLink fieldCode="DE" term="%22User+Satisfaction+%28Information%29%22">User Satisfaction (Information)</searchLink><br /><searchLink fieldCode="DE" term="%22Information+Systems%22">Information Systems</searchLink>
– Name: Subject
  Label: Geographic Terms
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Greece%22">Greece</searchLink><br /><searchLink fieldCode="DE" term="%22Netherlands%22">Netherlands</searchLink><br /><searchLink fieldCode="DE" term="%22Romania%22">Romania</searchLink><br /><searchLink fieldCode="DE" term="%22United+Kingdom%22">United Kingdom</searchLink><br /><searchLink fieldCode="DE" term="%22Cyprus%22">Cyprus</searchLink><br /><searchLink fieldCode="DE" term="%22Germany%22">Germany</searchLink><br /><searchLink fieldCode="DE" term="%22Serbia%22">Serbia</searchLink><br /><searchLink fieldCode="DE" term="%22Bulgaria%22">Bulgaria</searchLink><br /><searchLink fieldCode="DE" term="%22Croatia%22">Croatia</searchLink><br /><searchLink fieldCode="DE" term="%22Estonia%22">Estonia</searchLink><br /><searchLink fieldCode="DE" term="%22Ireland%22">Ireland</searchLink><br /><searchLink fieldCode="DE" term="%22Lithuania%22">Lithuania</searchLink><br /><searchLink fieldCode="DE" term="%22Poland%22">Poland</searchLink><br /><searchLink fieldCode="DE" term="%22Portugal%22">Portugal</searchLink><br /><searchLink fieldCode="DE" term="%22Spain%22">Spain</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1109/TLT.2017.2732349
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 1939-1382
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Recommender systems provide users with content they might be interested in. Conventionally, recommender systems are evaluated mostly by using prediction accuracy metrics only. But, the ultimate goal of a recommender system is to increase user satisfaction. Therefore, evaluations that measure user satisfaction should also be performed before deploying a recommender system in a real target environment. Such evaluations are laborious and complicated compared to the traditional, data-centric evaluations, though. In this study, we carried out a user-centric evaluation of state-of-the-art recommender systems as well as a graph-based approach in the ecologically valid setting of an authentic social learning platform. We also conducted a data-centric evaluation on the same data to investigate the added value of user-centric evaluations and how user satisfaction of a recommender system is related to its performance in terms of accuracy metrics. Our findings suggest that user-centric evaluation results are not necessarily in line with data-centric evaluation results. We conclude that the traditional evaluation of recommender systems in terms of prediction accuracy only does not suffice to judge performance of recommender systems on the user side. Moreover, the user-centric evaluation provides valuable insights in how candidate algorithms perform on each of the five quality metrics for recommendations: usefulness, accuracy, novelty, diversity, and serendipity.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: Ref
  Label: Number of References
  Group: RefInfo
  Data: 40
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2018
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1192608
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1192608
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1109/TLT.2017.2732349
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 13
        StartPage: 294
    Subjects:
      – SubjectFull: Evaluation
        Type: general
      – SubjectFull: Socialization
        Type: general
      – SubjectFull: Accuracy
        Type: general
      – SubjectFull: Prediction
        Type: general
      – SubjectFull: Neighborhoods
        Type: general
      – SubjectFull: Memory
        Type: general
      – SubjectFull: Mathematics
        Type: general
      – SubjectFull: Foreign Countries
        Type: general
      – SubjectFull: User Satisfaction (Information)
        Type: general
      – SubjectFull: Information Systems
        Type: general
      – SubjectFull: Greece
        Type: general
      – SubjectFull: Netherlands
        Type: general
      – SubjectFull: Romania
        Type: general
      – SubjectFull: United Kingdom
        Type: general
      – SubjectFull: Cyprus
        Type: general
      – SubjectFull: Germany
        Type: general
      – SubjectFull: Serbia
        Type: general
      – SubjectFull: Bulgaria
        Type: general
      – SubjectFull: Croatia
        Type: general
      – SubjectFull: Estonia
        Type: general
      – SubjectFull: Ireland
        Type: general
      – SubjectFull: Lithuania
        Type: general
      – SubjectFull: Poland
        Type: general
      – SubjectFull: Portugal
        Type: general
      – SubjectFull: Spain
        Type: general
    Titles:
      – TitleFull: User-Centric Evaluation of Recommender Systems in Social Learning Platforms: Accuracy is Just the Tip of the Iceberg
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Fazeli, Soude
      – PersonEntity:
          Name:
            NameFull: Drachsler, Hendrik
      – PersonEntity:
          Name:
            NameFull: Bitter-Rijpkema, Marlies
      – PersonEntity:
          Name:
            NameFull: Brouns, Francis
      – PersonEntity:
          Name:
            NameFull: Brouns, Wim van der Vegt
      – PersonEntity:
          Name:
            NameFull: Sloep, Peter B.
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 01
              Type: published
              Y: 2018
          Identifiers:
            – Type: issn-print
              Value: 1939-1382
          Numbering:
            – Type: volume
              Value: 11
            – Type: issue
              Value: 3
          Titles:
            – TitleFull: IEEE Transactions on Learning Technologies
              Type: main
ResultId 1