Automated Assessment of the Quality of Peer Reviews Using Natural Language Processing Techniques
Saved in:
| Title: | Automated Assessment of the Quality of Peer Reviews Using Natural Language Processing Techniques |
|---|---|
| Language: | English |
| Authors: | Ramachandran, Lakshmi, Gehringer, Edward F., Yadav, Ravi K. |
| Source: | International Journal of Artificial Intelligence in Education. Sep 2017 27(3):534-581. |
| Availability: | Springer. 233 Spring Street, New York, NY 10013. Tel: 800-777-4643; Tel: 212-460-1500; Fax: 212-348-4505; e-mail: service-ny@springer.com; Web site: http://www.springerlink.com |
| Peer Reviewed: | Y |
| Page Count: | 48 |
| Publication Date: | 2017 |
| Sponsoring Agency: | National Science Foundation (NSF) |
| Contract Number: | 0942279 |
| Document Type: | Journal Articles Reports - Research |
| Descriptors: | Natural Language Processing, Peer Evaluation, Educational Quality, Meta Analysis, Automation, Computer Software, Feedback (Response), Authors, Writing for Publication, Plagiarism, Evaluation Methods |
| DOI: | 10.1007/s40593-016-0132-x |
| ISSN: | 1560-4292 |
| Abstract: | A "review" is textual feedback provided by a reviewer to the author of a submitted version. Peer reviews are used in academic publishing and in education to assess student work. While reviews are important to e-commerce sites like Amazon and e-bay, which use them to assess the quality of products and services, our work focuses on academic reviewing. We seek to help reviewers improve the quality of their reviews. One way to measure review quality is through "metareview" or review of reviews. We develop an automated metareview software that provides rapid feedback to reviewers on their assessment of authors' submissions. To measure review quality, we employ metrics such as: review content type, review relevance, review's coverage of a submission, review tone, review volume and review plagiarism (from the submission or from other reviews). We use natural language processing and machine-learning techniques to calculate these metrics. We summarize results from experiments to evaluate our review quality metrics: review content, relevance and coverage, and a study to analyze user perceptions of importance and usefulness of these metrics. Our approaches were evaluated on data from Expertiza and the Scaffolded Writing and Rewriting in the Discipline (SWoRD) project, which are two collaborative web-based learning applications. |
| Abstractor: | As Provided |
| Number of References: | 67 |
| Entry Date: | 2017 |
| Accession Number: | EJ1148519 |
| Database: | ERIC |
| Abstract: | A "review" is textual feedback provided by a reviewer to the author of a submitted version. Peer reviews are used in academic publishing and in education to assess student work. While reviews are important to e-commerce sites like Amazon and e-bay, which use them to assess the quality of products and services, our work focuses on academic reviewing. We seek to help reviewers improve the quality of their reviews. One way to measure review quality is through "metareview" or review of reviews. We develop an automated metareview software that provides rapid feedback to reviewers on their assessment of authors' submissions. To measure review quality, we employ metrics such as: review content type, review relevance, review's coverage of a submission, review tone, review volume and review plagiarism (from the submission or from other reviews). We use natural language processing and machine-learning techniques to calculate these metrics. We summarize results from experiments to evaluate our review quality metrics: review content, relevance and coverage, and a study to analyze user perceptions of importance and usefulness of these metrics. Our approaches were evaluated on data from Expertiza and the Scaffolded Writing and Rewriting in the Discipline (SWoRD) project, which are two collaborative web-based learning applications. |
|---|---|
| ISSN: | 1560-4292 |
| DOI: | 10.1007/s40593-016-0132-x |
