User-Centric Evaluation of Recommender Systems in Social Learning Platforms: Accuracy is Just the Tip of the Iceberg
Saved in:
| Title: | User-Centric Evaluation of Recommender Systems in Social Learning Platforms: Accuracy is Just the Tip of the Iceberg |
|---|---|
| Language: | English |
| Authors: | Fazeli, Soude (ORCID |
| Source: | IEEE Transactions on Learning Technologies. Jul-Sep 2018 11(3):294-306. |
| Availability: | Institute of Electrical and Electronics Engineers, Inc. 445 Hoes Lane, Piscataway, NJ 08854. Tel: 732-981-0060; Web site: http://ieeexplore.ieee.org/xpl/RecentIssue.jsp?punumber=4620076 |
| Peer Reviewed: | Y |
| Page Count: | 13 |
| Publication Date: | 2018 |
| Document Type: | Journal Articles Reports - Research Tests/Questionnaires |
| Descriptors: | Evaluation, Socialization, Accuracy, Prediction, Neighborhoods, Memory, Mathematics, Foreign Countries, User Satisfaction (Information), Information Systems |
| Geographic Terms: | Greece, Netherlands, Romania, United Kingdom, Cyprus, Germany, Serbia, Bulgaria, Croatia, Estonia, Ireland, Lithuania, Poland, Portugal, Spain |
| DOI: | 10.1109/TLT.2017.2732349 |
| ISSN: | 1939-1382 |
| Abstract: | Recommender systems provide users with content they might be interested in. Conventionally, recommender systems are evaluated mostly by using prediction accuracy metrics only. But, the ultimate goal of a recommender system is to increase user satisfaction. Therefore, evaluations that measure user satisfaction should also be performed before deploying a recommender system in a real target environment. Such evaluations are laborious and complicated compared to the traditional, data-centric evaluations, though. In this study, we carried out a user-centric evaluation of state-of-the-art recommender systems as well as a graph-based approach in the ecologically valid setting of an authentic social learning platform. We also conducted a data-centric evaluation on the same data to investigate the added value of user-centric evaluations and how user satisfaction of a recommender system is related to its performance in terms of accuracy metrics. Our findings suggest that user-centric evaluation results are not necessarily in line with data-centric evaluation results. We conclude that the traditional evaluation of recommender systems in terms of prediction accuracy only does not suffice to judge performance of recommender systems on the user side. Moreover, the user-centric evaluation provides valuable insights in how candidate algorithms perform on each of the five quality metrics for recommendations: usefulness, accuracy, novelty, diversity, and serendipity. |
| Abstractor: | As Provided |
| Number of References: | 40 |
| Entry Date: | 2018 |
| Accession Number: | EJ1192608 |
| Database: | ERIC |
| Abstract: | Recommender systems provide users with content they might be interested in. Conventionally, recommender systems are evaluated mostly by using prediction accuracy metrics only. But, the ultimate goal of a recommender system is to increase user satisfaction. Therefore, evaluations that measure user satisfaction should also be performed before deploying a recommender system in a real target environment. Such evaluations are laborious and complicated compared to the traditional, data-centric evaluations, though. In this study, we carried out a user-centric evaluation of state-of-the-art recommender systems as well as a graph-based approach in the ecologically valid setting of an authentic social learning platform. We also conducted a data-centric evaluation on the same data to investigate the added value of user-centric evaluations and how user satisfaction of a recommender system is related to its performance in terms of accuracy metrics. Our findings suggest that user-centric evaluation results are not necessarily in line with data-centric evaluation results. We conclude that the traditional evaluation of recommender systems in terms of prediction accuracy only does not suffice to judge performance of recommender systems on the user side. Moreover, the user-centric evaluation provides valuable insights in how candidate algorithms perform on each of the five quality metrics for recommendations: usefulness, accuracy, novelty, diversity, and serendipity. |
|---|---|
| ISSN: | 1939-1382 |
| DOI: | 10.1109/TLT.2017.2732349 |
