Development of image-based decision support systems utilizing information extracted from radiological free-text report databases with text-based transformers.
Saved in:
| Title: | Development of image-based decision support systems utilizing information extracted from radiological free-text report databases with text-based transformers. |
|---|---|
| Authors: | Nowak, Sebastian1 (AUTHOR) sebastian.nowak@ukbonn.de, Schneider, Helen2 (AUTHOR), Layer, Yannik C.1 (AUTHOR), Theis, Maike1 (AUTHOR), Biesner, David2 (AUTHOR), Block, Wolfgang1 (AUTHOR), Wulff, Benjamin2 (AUTHOR), Attenberger, Ulrike I.1 (AUTHOR), Sifa, Rafet2 (AUTHOR), Sprinkart, Alois M.1 (AUTHOR) |
| Source: | European Radiology. May2024, Vol. 34 Issue 5, p2895-2904. 10p. |
| Subjects: | Decision support systems, Receiver operating characteristic curves, Artificial intelligence, Database design, Information storage & retrieval systems |
| Abstract: | Objectives: To investigate the potential and limitations of utilizing transformer-based report annotation for on-site development of image-based diagnostic decision support systems (DDSS). Methods: The study included 88,353 chest X-rays from 19,581 intensive care unit (ICU) patients. To label the presence of six typical findings in 17,041 images, the corresponding free-text reports of the attending radiologists were assessed by medical research assistants ("gold labels"). Automatically generated "silver" labels were extracted for all reports by transformer models trained on gold labels. To investigate the benefit of such silver labels, the image-based models were trained using three approaches: with gold labels only (MG), with silver labels first, then with gold labels (MS/G), and with silver and gold labels together (MS+G). To investigate the influence of invested annotation effort, the experiments were repeated with different numbers (N) of gold-annotated reports for training the transformer and image-based models and tested on 2099 gold-annotated images. Significant differences in macro-averaged area under the receiver operating characteristic curve (AUC) were assessed by non-overlapping 95% confidence intervals. Results: Utilizing transformer-based silver labels showed significantly higher macro-averaged AUC than training solely with gold labels (N = 1000: MG 67.8 [66.0–69.6], MS/G 77.9 [76.2–79.6]; N = 14,580: MG 74.5 [72.8–76.2], MS/G 80.9 [79.4–82.4]). Training with silver and gold labels together was beneficial using only 500 gold labels (MS+G 76.4 [74.7–78.0], MS/G 75.3 [73.5–77.0]). Conclusions: Transformer-based annotation has potential for unlocking free-text report databases for the development of image-based DDSS. However, on-site development of image-based DDSS could benefit from more sophisticated annotation pipelines including further information than a single radiological report. Clinical relevance statement: Leveraging clinical databases for on-site development of artificial intelligence (AI)–based diagnostic decision support systems by text-based transformers could promote the application of AI in clinical practice by circumventing highly regulated data exchanges with third parties. Key Points: • The amount of data from a database that can be used to develop AI-assisted diagnostic decision systems is often limited by the need for time-consuming identification of pathologies by radiologists. • The transformer-based structuring of free-text radiological reports shows potential to unlock corresponding image databases for on-site development of image-based diagnostic decision support systems. • However, the quality of image annotations generated solely on the content of a single radiology report may be limited by potential inaccuracies and incompleteness of this report. [ABSTRACT FROM AUTHOR] |
| Copyright of European Radiology is the property of Springer Nature and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.) | |
| Database: | Engineering Source |
|
Full text is not displayed to guests.
Login for full access.
|
|
| FullText | Links: – Type: pdflink Text: Availability: 1 |
|---|---|
| Header | DbId: egs DbLabel: Engineering Source An: 177463589 AccessLevel: 6 PubType: Academic Journal PubTypeId: academicJournal PreciseRelevancyScore: 0 |
| IllustrationInfo | |
| Items | – Name: Title Label: Title Group: Ti Data: Development of image-based decision support systems utilizing information extracted from radiological free-text report databases with text-based transformers. – Name: Author Label: Authors Group: Au Data: <searchLink fieldCode="AR" term="%22Nowak%2C+Sebastian%22">Nowak, Sebastian</searchLink><relatesTo>1</relatesTo> (AUTHOR)<i> sebastian.nowak@ukbonn.de</i><br /><searchLink fieldCode="AR" term="%22Schneider%2C+Helen%22">Schneider, Helen</searchLink><relatesTo>2</relatesTo> (AUTHOR)<br /><searchLink fieldCode="AR" term="%22Layer%2C+Yannik+C%2E%22">Layer, Yannik C.</searchLink><relatesTo>1</relatesTo> (AUTHOR)<br /><searchLink fieldCode="AR" term="%22Theis%2C+Maike%22">Theis, Maike</searchLink><relatesTo>1</relatesTo> (AUTHOR)<br /><searchLink fieldCode="AR" term="%22Biesner%2C+David%22">Biesner, David</searchLink><relatesTo>2</relatesTo> (AUTHOR)<br /><searchLink fieldCode="AR" term="%22Block%2C+Wolfgang%22">Block, Wolfgang</searchLink><relatesTo>1</relatesTo> (AUTHOR)<br /><searchLink fieldCode="AR" term="%22Wulff%2C+Benjamin%22">Wulff, Benjamin</searchLink><relatesTo>2</relatesTo> (AUTHOR)<br /><searchLink fieldCode="AR" term="%22Attenberger%2C+Ulrike+I%2E%22">Attenberger, Ulrike I.</searchLink><relatesTo>1</relatesTo> (AUTHOR)<br /><searchLink fieldCode="AR" term="%22Sifa%2C+Rafet%22">Sifa, Rafet</searchLink><relatesTo>2</relatesTo> (AUTHOR)<br /><searchLink fieldCode="AR" term="%22Sprinkart%2C+Alois+M%2E%22">Sprinkart, Alois M.</searchLink><relatesTo>1</relatesTo> (AUTHOR) – Name: TitleSource Label: Source Group: Src Data: <searchLink fieldCode="JN" term="%22European+Radiology%22">European Radiology</searchLink>. May2024, Vol. 34 Issue 5, p2895-2904. 10p. – Name: Subject Label: Subjects Group: Su Data: <searchLink fieldCode="DE" term="%22Decision+support+systems%22">Decision support systems</searchLink><br /><searchLink fieldCode="DE" term="%22Receiver+operating+characteristic+curves%22">Receiver operating characteristic curves</searchLink><br /><searchLink fieldCode="DE" term="%22Artificial+intelligence%22">Artificial intelligence</searchLink><br /><searchLink fieldCode="DE" term="%22Database+design%22">Database design</searchLink><br /><searchLink fieldCode="DE" term="%22Information+storage+%26+retrieval+systems%22">Information storage & retrieval systems</searchLink> – Name: Abstract Label: Abstract Group: Ab Data: Objectives: To investigate the potential and limitations of utilizing transformer-based report annotation for on-site development of image-based diagnostic decision support systems (DDSS). Methods: The study included 88,353 chest X-rays from 19,581 intensive care unit (ICU) patients. To label the presence of six typical findings in 17,041 images, the corresponding free-text reports of the attending radiologists were assessed by medical research assistants ("gold labels"). Automatically generated "silver" labels were extracted for all reports by transformer models trained on gold labels. To investigate the benefit of such silver labels, the image-based models were trained using three approaches: with gold labels only (MG), with silver labels first, then with gold labels (MS/G), and with silver and gold labels together (MS+G). To investigate the influence of invested annotation effort, the experiments were repeated with different numbers (N) of gold-annotated reports for training the transformer and image-based models and tested on 2099 gold-annotated images. Significant differences in macro-averaged area under the receiver operating characteristic curve (AUC) were assessed by non-overlapping 95% confidence intervals. Results: Utilizing transformer-based silver labels showed significantly higher macro-averaged AUC than training solely with gold labels (N = 1000: MG 67.8 [66.0–69.6], MS/G 77.9 [76.2–79.6]; N = 14,580: MG 74.5 [72.8–76.2], MS/G 80.9 [79.4–82.4]). Training with silver and gold labels together was beneficial using only 500 gold labels (MS+G 76.4 [74.7–78.0], MS/G 75.3 [73.5–77.0]). Conclusions: Transformer-based annotation has potential for unlocking free-text report databases for the development of image-based DDSS. However, on-site development of image-based DDSS could benefit from more sophisticated annotation pipelines including further information than a single radiological report. Clinical relevance statement: Leveraging clinical databases for on-site development of artificial intelligence (AI)–based diagnostic decision support systems by text-based transformers could promote the application of AI in clinical practice by circumventing highly regulated data exchanges with third parties. Key Points: • The amount of data from a database that can be used to develop AI-assisted diagnostic decision systems is often limited by the need for time-consuming identification of pathologies by radiologists. • The transformer-based structuring of free-text radiological reports shows potential to unlock corresponding image databases for on-site development of image-based diagnostic decision support systems. • However, the quality of image annotations generated solely on the content of a single radiology report may be limited by potential inaccuracies and incompleteness of this report. [ABSTRACT FROM AUTHOR] – Name: AbstractSuppliedCopyright Label: Group: Ab Data: <i>Copyright of European Radiology is the property of Springer Nature and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract.</i> (Copyright applies to all Abstracts.) |
| PLink | https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=egs&AN=177463589 |
| RecordInfo | BibRecord: BibEntity: Identifiers: – Type: doi Value: 10.1007/s00330-023-10373-0 Languages: – Code: eng Text: English PhysicalDescription: Pagination: PageCount: 10 StartPage: 2895 Subjects: – SubjectFull: Decision support systems Type: general – SubjectFull: Receiver operating characteristic curves Type: general – SubjectFull: Artificial intelligence Type: general – SubjectFull: Database design Type: general – SubjectFull: Information storage & retrieval systems Type: general Titles: – TitleFull: Development of image-based decision support systems utilizing information extracted from radiological free-text report databases with text-based transformers. Type: main BibRelationships: HasContributorRelationships: – PersonEntity: Name: NameFull: Nowak, Sebastian – PersonEntity: Name: NameFull: Schneider, Helen – PersonEntity: Name: NameFull: Layer, Yannik C. – PersonEntity: Name: NameFull: Theis, Maike – PersonEntity: Name: NameFull: Biesner, David – PersonEntity: Name: NameFull: Block, Wolfgang – PersonEntity: Name: NameFull: Wulff, Benjamin – PersonEntity: Name: NameFull: Attenberger, Ulrike I. – PersonEntity: Name: NameFull: Sifa, Rafet – PersonEntity: Name: NameFull: Sprinkart, Alois M. IsPartOfRelationships: – BibEntity: Dates: – D: 01 M: 05 Text: May2024 Type: published Y: 2024 Identifiers: – Type: issn-print Value: 09387994 Numbering: – Type: volume Value: 34 – Type: issue Value: 5 Titles: – TitleFull: European Radiology Type: main |
| ResultId | 1 |
