Parameter-efficient fine-tuning of small language models for code generation: a comparative study of Gemma, Qwen 2.5 and Llama 3.2.
Saved in:
| Title: | Parameter-efficient fine-tuning of small language models for code generation: a comparative study of Gemma, Qwen 2.5 and Llama 3.2. |
|---|---|
| Authors: | Nguyen, Van-Viet1 nvviet@ictu.edu.vn, Nguyen, The-Vinh1 vinhnt@ictu.edu.vn, Nguyen, Huu-Khanh2 khanhnh@tnu.edu.vn, Vu, Duc-Quang1 vdquang@ictu.edu.vn |
| Source: | International Journal of Electrical & Computer Engineering (2088-8708). Feb2026, Vol. 16 Issue 1, p278-287. 10p. |
| Subjects: | Language models, Code generators, Mathematical optimization, Computer software |
| Abstract: | Large language models (LLMs) have demonstrated impressive capabilities in code generation; however, their high computational demands, privacy limitations, and challenges in edge deployment restrict their practical use in domain-specific applications. This study explores the effectiveness of parameter efficient fine-tuning for small language models (SLMs) with fewer than 3 billion parameters. We adopt a hybrid approach that combines low-rank adaptation (LoRA) and 4-bit quantization (QLoRA) to reduce finetuning costs while preserving semantic consistency. Experiments on the CodeAlpaca-20k dataset reveal that SLMs fine-tuned with this method outperform larger baseline models, including Phi-3 Mini 4K base, in ROUGE-L. Notably, applying our approach to the LLaMA 3 3B and Qwen2.5 3B models yielded performance improvements of 54% and 55%, respectively, over untuned counterparts. We evaluate models developed by major artificial intelligence (AI) providers Google (Gemma 2B), Meta (LLaMA 3 1B/3B), and Alibaba (Qwen2.5 1.5B/3B) and show that parameter-efficient fine-tuning enables them to serve as cost-effective, highperforming alternatives to larger LLMs. These findings highlight the potential of SLMs as scalable solutions for domain-specific software engineering tasks, supporting broader adoption and democratization of neural code synthesis. [ABSTRACT FROM AUTHOR] |
| Copyright of International Journal of Electrical & Computer Engineering (2088-8708) is the property of Institute of Advanced Engineering & Science and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.) | |
| Database: | Engineering Source |
| FullText | Links: – Type: pdflink Text: Availability: 0 |
|---|---|
| Header | DbId: egs DbLabel: Engineering Source An: 191308299 AccessLevel: 6 PubType: Academic Journal PubTypeId: academicJournal PreciseRelevancyScore: 0 |
| IllustrationInfo | |
| Items | – Name: Title Label: Title Group: Ti Data: Parameter-efficient fine-tuning of small language models for code generation: a comparative study of Gemma, Qwen 2.5 and Llama 3.2. – Name: Author Label: Authors Group: Au Data: <searchLink fieldCode="AR" term="%22Nguyen%2C+Van-Viet%22">Nguyen, Van-Viet</searchLink><relatesTo>1</relatesTo><i> nvviet@ictu.edu.vn</i><br /><searchLink fieldCode="AR" term="%22Nguyen%2C+The-Vinh%22">Nguyen, The-Vinh</searchLink><relatesTo>1</relatesTo><i> vinhnt@ictu.edu.vn</i><br /><searchLink fieldCode="AR" term="%22Nguyen%2C+Huu-Khanh%22">Nguyen, Huu-Khanh</searchLink><relatesTo>2</relatesTo><i> khanhnh@tnu.edu.vn</i><br /><searchLink fieldCode="AR" term="%22Vu%2C+Duc-Quang%22">Vu, Duc-Quang</searchLink><relatesTo>1</relatesTo><i> vdquang@ictu.edu.vn</i> – Name: TitleSource Label: Source Group: Src Data: <searchLink fieldCode="JN" term="%22International+Journal+of+Electrical+%26+Computer+Engineering+%282088-8708%29%22">International Journal of Electrical & Computer Engineering (2088-8708)</searchLink>. Feb2026, Vol. 16 Issue 1, p278-287. 10p. – Name: Subject Label: Subjects Group: Su Data: <searchLink fieldCode="DE" term="%22Language+models%22">Language models</searchLink><br /><searchLink fieldCode="DE" term="%22Code+generators%22">Code generators</searchLink><br /><searchLink fieldCode="DE" term="%22Mathematical+optimization%22">Mathematical optimization</searchLink><br /><searchLink fieldCode="DE" term="%22Computer+software%22">Computer software</searchLink> – Name: Abstract Label: Abstract Group: Ab Data: Large language models (LLMs) have demonstrated impressive capabilities in code generation; however, their high computational demands, privacy limitations, and challenges in edge deployment restrict their practical use in domain-specific applications. This study explores the effectiveness of parameter efficient fine-tuning for small language models (SLMs) with fewer than 3 billion parameters. We adopt a hybrid approach that combines low-rank adaptation (LoRA) and 4-bit quantization (QLoRA) to reduce finetuning costs while preserving semantic consistency. Experiments on the CodeAlpaca-20k dataset reveal that SLMs fine-tuned with this method outperform larger baseline models, including Phi-3 Mini 4K base, in ROUGE-L. Notably, applying our approach to the LLaMA 3 3B and Qwen2.5 3B models yielded performance improvements of 54% and 55%, respectively, over untuned counterparts. We evaluate models developed by major artificial intelligence (AI) providers Google (Gemma 2B), Meta (LLaMA 3 1B/3B), and Alibaba (Qwen2.5 1.5B/3B) and show that parameter-efficient fine-tuning enables them to serve as cost-effective, highperforming alternatives to larger LLMs. These findings highlight the potential of SLMs as scalable solutions for domain-specific software engineering tasks, supporting broader adoption and democratization of neural code synthesis. [ABSTRACT FROM AUTHOR] – Name: AbstractSuppliedCopyright Label: Group: Ab Data: <i>Copyright of International Journal of Electrical & Computer Engineering (2088-8708) is the property of Institute of Advanced Engineering & Science and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract.</i> (Copyright applies to all Abstracts.) |
| PLink | https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=egs&AN=191308299 |
| RecordInfo | BibRecord: BibEntity: Identifiers: – Type: doi Value: 10.11591/ijece.v16i1.pp278-287 Languages: – Code: eng Text: English PhysicalDescription: Pagination: PageCount: 10 StartPage: 278 Subjects: – SubjectFull: Language models Type: general – SubjectFull: Code generators Type: general – SubjectFull: Mathematical optimization Type: general – SubjectFull: Computer software Type: general Titles: – TitleFull: Parameter-efficient fine-tuning of small language models for code generation: a comparative study of Gemma, Qwen 2.5 and Llama 3.2. Type: main BibRelationships: HasContributorRelationships: – PersonEntity: Name: NameFull: Nguyen, Van-Viet – PersonEntity: Name: NameFull: Nguyen, The-Vinh – PersonEntity: Name: NameFull: Nguyen, Huu-Khanh – PersonEntity: Name: NameFull: Vu, Duc-Quang IsPartOfRelationships: – BibEntity: Dates: – D: 01 M: 02 Text: Feb2026 Type: published Y: 2026 Identifiers: – Type: issn-print Value: 20888708 Numbering: – Type: volume Value: 16 – Type: issue Value: 1 Titles: – TitleFull: International Journal of Electrical & Computer Engineering (2088-8708) Type: main |
| ResultId | 1 |
