View in EDS

The Current State of Methods for Establishing Reliability in Qualitative Chemistry Education Research Articles

Saved in:
Bibliographic Details
Title: The Current State of Methods for Establishing Reliability in Qualitative Chemistry Education Research Articles
Language: English
Authors: Watts, Field M. (ORCID 0000-0002-1800-1816), Finkenstaedt-Quinn, Solaire A. (ORCID 0000-0003-2098-074X)
Source: Chemistry Education Research and Practice. Jul 2021 22(3):565-578.
Availability: Royal Society of Chemistry. Thomas Graham House, Science Park, Milton Road, Cambridge, CB4 0WF, UK. Tel: +44-1223 420066; Fax: +44-1223 423623; e-mail: cerp@rsc.org; Web site: http://www.rsc.org/cerp
Peer Reviewed: Y
Page Count: 14
Publication Date: 2021
Document Type: Journal Articles
Information Analyses
Reports - Research
Descriptors: Qualitative Research, Educational Research, Research Methodology, Chemistry, Reliability, Journal Articles, Evidence
DOI: 10.1039/d1rp00007a
ISSN: 1756-1108
Abstract: The tradition of qualitative research drives much of chemistry education research activity. When performing qualitative studies, researchers must demonstrate the trustworthiness of their analysis so researchers and practitioners consuming their work can understand if and how the presented research claims and conclusions might be transferable to their unique educational settings. There are a number of steps researchers can take to demonstrate the trustworthiness of their work, one of which is demonstrating and reporting evidence of reliability. The purpose of this methodological review is to investigate the methods researchers use to establish and report reliability for chemistry education research articles including a qualitative research component. Drawing from the literature on qualitative research methodology and content analysis, we describe the approaches for establishing the reliability of qualitative data analysis using various measures of inter-rater reliability and processes including negotiated agreement. We used this background literature to guide our review of research articles containing a qualitative component and published in "Chemistry Education Research and Practice" and the "Journal of Chemical Education" from the years 2010 through 2019 for whether they report evidence of reliability. We followed this by a more in-depth analysis of how articles from the years 2017 through 2019 discuss reliability. Our analysis indicates that, overall, researchers are presenting evidence of reliability in chemistry education research (CER) articles by reporting reliability measures, describing a process of negotiated agreement, or mentioning reliability and the steps taken to demonstrate it. However, there is a reliance on reporting only percent agreement, which is not considered an acceptable measure of reliability when used on its own. In addition, the descriptions of how reliability was established were not always clear, which may make it difficult for readers to evaluate the veracity of research findings. Our findings indicate that, as a field, CER researchers should be more cognizant of the appropriateness of how we establish reliability for qualitative analysis and should more clearly present the processes by which reliability was established in CER manuscripts.
Abstractor: As Provided
Entry Date: 2021
Accession Number: EJ1300848
Database: ERIC
FullText Text:
  Availability: 0
Header DbId: eric
DbLabel: ERIC
An: EJ1300848
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: The Current State of Methods for Establishing Reliability in Qualitative Chemistry Education Research Articles
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Watts%2C+Field+M%2E%22">Watts, Field M.</searchLink> (ORCID <externalLink term="http://orcid.org/0000-0002-1800-1816">0000-0002-1800-1816</externalLink>)<br /><searchLink fieldCode="AR" term="%22Finkenstaedt-Quinn%2C+Solaire+A%2E%22">Finkenstaedt-Quinn, Solaire A.</searchLink> (ORCID <externalLink term="http://orcid.org/0000-0003-2098-074X">0000-0003-2098-074X</externalLink>)
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Chemistry+Education+Research+and+Practice%22"><i>Chemistry Education Research and Practice</i></searchLink>. Jul 2021 22(3):565-578.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Royal Society of Chemistry. Thomas Graham House, Science Park, Milton Road, Cambridge, CB4 0WF, UK. Tel: +44-1223 420066; Fax: +44-1223 423623; e-mail: cerp@rsc.org; Web site: http://www.rsc.org/cerp
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 14
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2021
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Information Analyses<br />Reports - Research
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Qualitative+Research%22">Qualitative Research</searchLink><br /><searchLink fieldCode="DE" term="%22Educational+Research%22">Educational Research</searchLink><br /><searchLink fieldCode="DE" term="%22Research+Methodology%22">Research Methodology</searchLink><br /><searchLink fieldCode="DE" term="%22Chemistry%22">Chemistry</searchLink><br /><searchLink fieldCode="DE" term="%22Reliability%22">Reliability</searchLink><br /><searchLink fieldCode="DE" term="%22Journal+Articles%22">Journal Articles</searchLink><br /><searchLink fieldCode="DE" term="%22Evidence%22">Evidence</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1039/d1rp00007a
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 1756-1108
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: The tradition of qualitative research drives much of chemistry education research activity. When performing qualitative studies, researchers must demonstrate the trustworthiness of their analysis so researchers and practitioners consuming their work can understand if and how the presented research claims and conclusions might be transferable to their unique educational settings. There are a number of steps researchers can take to demonstrate the trustworthiness of their work, one of which is demonstrating and reporting evidence of reliability. The purpose of this methodological review is to investigate the methods researchers use to establish and report reliability for chemistry education research articles including a qualitative research component. Drawing from the literature on qualitative research methodology and content analysis, we describe the approaches for establishing the reliability of qualitative data analysis using various measures of inter-rater reliability and processes including negotiated agreement. We used this background literature to guide our review of research articles containing a qualitative component and published in "Chemistry Education Research and Practice" and the "Journal of Chemical Education" from the years 2010 through 2019 for whether they report evidence of reliability. We followed this by a more in-depth analysis of how articles from the years 2017 through 2019 discuss reliability. Our analysis indicates that, overall, researchers are presenting evidence of reliability in chemistry education research (CER) articles by reporting reliability measures, describing a process of negotiated agreement, or mentioning reliability and the steps taken to demonstrate it. However, there is a reliance on reporting only percent agreement, which is not considered an acceptable measure of reliability when used on its own. In addition, the descriptions of how reliability was established were not always clear, which may make it difficult for readers to evaluate the veracity of research findings. Our findings indicate that, as a field, CER researchers should be more cognizant of the appropriateness of how we establish reliability for qualitative analysis and should more clearly present the processes by which reliability was established in CER manuscripts.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2021
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1300848
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1300848
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1039/d1rp00007a
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 14
        StartPage: 565
    Subjects:
      – SubjectFull: Qualitative Research
        Type: general
      – SubjectFull: Educational Research
        Type: general
      – SubjectFull: Research Methodology
        Type: general
      – SubjectFull: Chemistry
        Type: general
      – SubjectFull: Reliability
        Type: general
      – SubjectFull: Journal Articles
        Type: general
      – SubjectFull: Evidence
        Type: general
    Titles:
      – TitleFull: The Current State of Methods for Establishing Reliability in Qualitative Chemistry Education Research Articles
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Watts, Field M.
      – PersonEntity:
          Name:
            NameFull: Finkenstaedt-Quinn, Solaire A.
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 07
              Type: published
              Y: 2021
          Identifiers:
            – Type: issn-electronic
              Value: 1756-1108
          Numbering:
            – Type: volume
              Value: 22
            – Type: issue
              Value: 3
          Titles:
            – TitleFull: Chemistry Education Research and Practice
              Type: main
ResultId 1