Assessing the Promise and Pitfalls of ChatGPT for Automated CS1-Driven Code Generation
Saved in:
| Title: | Assessing the Promise and Pitfalls of ChatGPT for Automated CS1-Driven Code Generation |
|---|---|
| Language: | English |
| Authors: | Muhammad Fawad Akbar Khan, Max Ramsdell, Erik Falor, Hamid Karimi |
| Source: | International Educational Data Mining Society. 2024. |
| Availability: | International Educational Data Mining Society. e-mail: admin@educationaldatamining.org; Web site: https://educationaldatamining.org/conferences/ |
| Peer Reviewed: | Y |
| Page Count: | 13 |
| Publication Date: | 2024 |
| Sponsoring Agency: | National Science Foundation (NSF), Division of Equity for Excellence in STEM (EES) |
| Contract Number: | 2321304 |
| Document Type: | Speeches/Meeting Papers Reports - Research |
| Descriptors: | Artificial Intelligence, Automation, Computer Science Education, Programming, Computer Software, Engineering, Cues |
| Abstract: | This paper undertakes a thorough evaluation of ChatGPT's code generation capabilities, contrasting them with those of human programmers from both educational and software engineering standpoints. The emphasis is placed on elucidating its importance in these intertwined domains. To facilitate a robust analysis, we curated a novel dataset comprising 131 code-generation prompts spanning five categories. The study encompasses 262 code samples generated by ChatGPT and humans, with a meticulous manual assessment methodology prioritizing correctness, comprehensibility, and security using 14 established code quality metrics. Noteworthy strengths include ChatGPT's proficiency in crafting concise, efficient code, particularly excelling in data analysis tasks (93.1% accuracy). However, limitations are observed in handling visual-graphical challenges. Comparative analysis with human-generated code highlights ChatGPT's inclination towards modular design and superior error handling. Machine learning models effectively distinguish ChatGPT from human code with up to 88% accuracy, indicating detectable coding style disparities. By offering profound insights into ChatGPT's code generation capabilities and limitations through quantitative metrics and qualitative analysis, this study contributes significantly to the advancement of AI-based programming assistants. The curated dataset and methodology establish a robust foundation for future research in this evolving domain, reinforcing its importance in shaping the future landscape of computer science education and software engineering. [For the complete proceedings, see ED675485.] |
| Abstractor: | As Provided |
| Notes: | https://github.com/DSAatUSU/ChatGPT-promises-and-pitfalls |
| Entry Date: | 2025 |
| Accession Number: | ED675593 |
| Database: | ERIC |
Be the first to leave a comment!
