Comparing Student and Generative Artificial Intelligence Chatbot Responses to Organic Chemistry Writing-to-Learn Assignments
Saved in:
| Title: | Comparing Student and Generative Artificial Intelligence Chatbot Responses to Organic Chemistry Writing-to-Learn Assignments |
|---|---|
| Language: | English |
| Authors: | Field M. Watts (ORCID |
| Source: | Journal of Chemical Education. 2023 100(10):3806-3817. |
| Availability: | Division of Chemical Education, Inc. and ACS Publications Division of the American Chemical Society. 1155 Sixteenth Street NW, Washington, DC 20036. Tel: 800-227-5558; Tel: 202-872-4600; e-mail: eic@jce.acs.org; Web site: http://pubs.acs.org/jchemeduc |
| Peer Reviewed: | Y |
| Page Count: | 12 |
| Publication Date: | 2023 |
| Document Type: | Journal Articles Reports - Research |
| Education Level: | Higher Education Postsecondary Education |
| Descriptors: | College Students, Science Instruction, Organic Chemistry, Thinking Skills, Artificial Intelligence, Writing Assignments, Expository Writing, Essays, Comparative Testing, Science Process Skills, Integrity, Student Behavior |
| DOI: | 10.1021/acs.jchemed.3c00664 |
| ISSN: | 0021-9584 1938-1328 |
| Abstract: | Chemistry education research demonstrates the value of open-ended writing tasks, such as writing-to-learn (WTL) assignments, for supporting students' learning with topics including reasoning about reaction mechanisms. The emergence of generative artificial intelligence (AI)technology, such as chatbots ChatGPT and Bard, raises concerns regarding the value of open-ended writing tasks in the classroom; one concern involves academic integrity and whether students will use these chatbots to produce sufficient responses to open-ended writing tasks. The present study investigates the degree to which generative AI chatbots exhibit mechanistic reasoning in response to organic chemistry WTL assignments. We produced responses from three generative AI chatbots (ChatGPT-3.5, ChatGPT-4, and Bard) to two WTL assignments developed to elicit students' mechanistic reasoning. Using previously reported machine learning models for analyzing student writing in response to the WTL assignments, we analyzed the chatbot responses for the inclusion of features pertinent to mechanistic reasoning. Herein, we report quantitative analyses of (1) the differences between chatbot responses on the two assignments and (2) the differences between chatbot and authentic student responses. Findings indicate that chatbots respond differently to different WTL assignments. Additionally, the chatbots rarely incorporated the discussion of electron movement, a key feature of mechanistic reasoning. Furthermore, the chatbots, in general, do not engage in mechanistic reasoning at the same level as students. We contextualize the results by considering academic integrity with the assumption that students' intentions are to engage in academically honest behavior, and we focus on understanding the ethical uses of generative AI for classroom assignments. |
| Abstractor: | As Provided |
| Entry Date: | 2024 |
| Accession Number: | EJ1452324 |
| Database: | ERIC |
| Abstract: | Chemistry education research demonstrates the value of open-ended writing tasks, such as writing-to-learn (WTL) assignments, for supporting students' learning with topics including reasoning about reaction mechanisms. The emergence of generative artificial intelligence (AI)technology, such as chatbots ChatGPT and Bard, raises concerns regarding the value of open-ended writing tasks in the classroom; one concern involves academic integrity and whether students will use these chatbots to produce sufficient responses to open-ended writing tasks. The present study investigates the degree to which generative AI chatbots exhibit mechanistic reasoning in response to organic chemistry WTL assignments. We produced responses from three generative AI chatbots (ChatGPT-3.5, ChatGPT-4, and Bard) to two WTL assignments developed to elicit students' mechanistic reasoning. Using previously reported machine learning models for analyzing student writing in response to the WTL assignments, we analyzed the chatbot responses for the inclusion of features pertinent to mechanistic reasoning. Herein, we report quantitative analyses of (1) the differences between chatbot responses on the two assignments and (2) the differences between chatbot and authentic student responses. Findings indicate that chatbots respond differently to different WTL assignments. Additionally, the chatbots rarely incorporated the discussion of electron movement, a key feature of mechanistic reasoning. Furthermore, the chatbots, in general, do not engage in mechanistic reasoning at the same level as students. We contextualize the results by considering academic integrity with the assumption that students' intentions are to engage in academically honest behavior, and we focus on understanding the ethical uses of generative AI for classroom assignments. |
|---|---|
| ISSN: | 0021-9584 1938-1328 |
| DOI: | 10.1021/acs.jchemed.3c00664 |
