View in EDS HTML Full Text PDF Full Text

SeroTacker-RoB: A Decision Rule-Based Algorithm for Reproducible Risk of Bias Assessment of Seroprevalence Studies

Saved in:
Bibliographic Details
Title: SeroTacker-RoB: A Decision Rule-Based Algorithm for Reproducible Risk of Bias Assessment of Seroprevalence Studies
Language: English
Authors: Bobrovitz, Niklas (ORCID 0000-0001-7883-4484), Noël, Kim, Li, Zihan, Cao, Christian, Deveaux, Gabriel, Selemon, Anabel, Clifton, David A., Yanes-Lane, Mercedes, Yan, Tingting, Arora, Rahul K.
Source: Research Synthesis Methods. May 2023 14(3):414-426.
Availability: Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
Peer Reviewed: Y
Page Count: 13
Publication Date: 2023
Document Type: Journal Articles
Information Analyses
Reports - Research
Descriptors: Decision Making, Algorithms, Risk, Bias, Check Lists, Efficiency, Incidence, Evidence, Synthesis, Diseases
DOI: 10.1002/jrsm.1620
ISSN: 1759-2879
1759-2887
Abstract: Risk of bias (RoB) assessments are a core element of evidence synthesis but can be time consuming and subjective. We aimed to develop a decision rule-based algorithm for RoB assessment of seroprevalence studies. We developed the SeroTracker-RoB algorithm. The algorithm derives seven objective and two subjective critical appraisal items from the Joanna Briggs Institute Critical Appraisal Checklist for Prevalence studies and implements decision rules that determine study risk of bias based on the items. Decision rules were validated using the SeroTracker seroprevalence study database, which included non-algorithmic RoB judgments from two reviewers. We quantified efficiency as the mean difference in time for the algorithmic and non-algorithmic assessments of 80 randomly selected articles, coverage as the proportion of studies where the decision rules yielded an assessment, and reliability using intraclass correlations comparing algorithmic and non-algorithmic assessments for 2070 articles. A set of decision rules with 61 branches was developed using responses to the nine critical appraisal items. The algorithmic approach was faster than non-algorithmic assessment (mean reduction 2.32 min [SD 1.09] per article), classified 100% (n = 2070) of studies, and had good reliability compared to non-algorithmic assessment (ICC 0.77, 95% CI 0.74-0.80). We built the SeroTracker-RoB Excel Tool, which embeds this algorithm for use by other researchers. The SeroTracker-RoB decision-rule based algorithm was faster than non-algorithmic assessment with complete coverage and good reliability. This algorithm enabled rapid, transparent, and reproducible RoB evaluations of seroprevalence studies and may support evidence synthesis efforts during future disease outbreaks. This decision rule-based approach could be applied to other types of prevalence studies.
Abstractor: As Provided
Entry Date: 2023
Accession Number: EJ1376524
Database: ERIC
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwHfFADKtBO_zUGwVksctmu-AAAA4jCB3wYJKoZIhvcNAQcGoIHRMIHOAgEAMIHIBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDHmx0ewh6nhEVxce2gIBEICBmi0FSBVgN1wXlilEXqdt8iTRTbmKEt-hBleb8szkuoB4ZtfYp3-nGhWz8YKISKTxebbqqNvO1frY-H1AKd5WMlTgaP9Q3f8VeF4JFmQFj8OnqIg-QxhAZjsZKehpEfijdgX0jipBuMJL6vLXqEAggglMEQCO73aCDKg9He2iNM2v4ZOr1yTd0oV4dCBXYpuS7fzMbpx9tzPFgwU=
Text:
  Availability: 1
  Value: <anid>AN0163567925;[bdct]01may.23;2023May09.08:05;v2.2.500</anid> <title id="AN0163567925-1">SeroTracker‐RoB: A decision rule‐based algorithm for reproducible risk of bias assessment of seroprevalence studies </title> <p>Risk of bias (RoB) assessments are a core element of evidence synthesis but can be time consuming and subjective. We aimed to develop a decision rule‐based algorithm for RoB assessment of seroprevalence studies. We developed the SeroTracker‐RoB algorithm. The algorithm derives seven objective and two subjective critical appraisal items from the Joanna Briggs Institute Critical Appraisal Checklist for Prevalence studies and implements decision rules that determine study risk of bias based on the items. Decision rules were validated using the SeroTracker seroprevalence study database, which included non‐algorithmic RoB judgments from two reviewers. We quantified efficiency as the mean difference in time for the algorithmic and non‐algorithmic assessments of 80 randomly selected articles, coverage as the proportion of studies where the decision rules yielded an assessment, and reliability using intraclass correlations comparing algorithmic and non‐algorithmic assessments for 2070 articles. A set of decision rules with 61 branches was developed using responses to the nine critical appraisal items. The algorithmic approach was faster than non‐algorithmic assessment (mean reduction 2.32 min [SD 1.09] per article), classified 100% (n = 2070) of studies, and had good reliability compared to non‐algorithmic assessment (ICC 0.77, 95% CI 0.74–0.80). We built the SeroTracker‐RoB Excel Tool, which embeds this algorithm for use by other researchers. The SeroTracker‐RoB decision‐rule based algorithm was faster than non‐algorithmic assessment with complete coverage and good reliability. This algorithm enabled rapid, transparent, and reproducible RoB evaluations of seroprevalence studies and may support evidence synthesis efforts during future disease outbreaks. This decision rule‐based approach could be applied to other types of prevalence studies.</p> <p>Keywords: algorithm; critical appraisal; decision rule; evidence synthesis; infectious disease; prevalence; risk of bias</p> <hd id="AN0163567925-2">Highlights</hd> <p></p> <hd id="AN0163567925-3">What is already known</hd> <p>Risk of bias assessments are a core element of evidence synthesis but can be time consuming and subjective. As such, there is a need to expedite objective and reproducible assessments using validated tools. Rapid evaluation of infection prevalence studies is of particular importance given that these studies are conducted during disease outbreaks and pandemics to inform public health decision making. However, there are currently no tools for expedited risk of bias assessment of prevalence studies.</p> <hd id="AN0163567925-4">What is new</hd> <p>We developed a reproducible algorithmic approach to risk of bias assessment for SARS‐CoV‐2 seroprevalence studies. This algorithm includes seven objectively determined critical appraisal items and two subjectively determined critical appraisal items derived from the Joanna Briggs Institute Critical Appraisal Checklist for Prevalence studies, and a decision tree, which determines risk of bias based on checklist items. The algorithmic approach was 4.6 times faster than traditional non‐algorithmic assessment, successfully categorized all 2070 studies that it was tested on, and had good agreement with non‐algorithmic assessments. We built a simple Excel tool so that other researchers can use this algorithmic approach.</p> <hd id="AN0163567925-5">Potential impact for Research Synthesis Methods readers outside the authors' field</hd> <p>The SeroTracker‐RoB decision rule‐based algorithm and Excel Tool enable rapid, transparent, and reproducible risk of bias assessments for SARS‐CoV‐2 seroprevalence studies and could be readily adapted for other types of prevalence studies. Moreover, we show how to operationalize a critical appraisal checklist and develop a decision tree to algorithmically generate risk of bias assessments. These processes may be applicable to risk of bias assessment for other types of studies and in other scientific disciplines.</p> <hd id="AN0163567925-6">INTRODUCTION</hd> <p>Seroprevalence studies use antibody tests to estimate the prevalence of infection or vaccination.[<reflink idref="bib1" id="ref1">1</reflink>] These studies have been used for decades to measure the true extent of infection,[<reflink idref="bib2" id="ref2">2</reflink>] quantify protection resulting from previous infection or vaccination, and inform public health measures.</p> <p>During the COVID‐19 pandemic, there has been an unprecedented increase in the utilization of seroprevalence studies, with results reported from over 3500 such studies as of June 1, 2022.[<reflink idref="bib3" id="ref3">3</reflink>] These studies have made important contributions to the pandemic response, but their methods have varied widely.[[<reflink idref="bib2" id="ref4">2</reflink>], [<reflink idref="bib4" id="ref5">4</reflink>]] Accordingly, robust risk of bias (RoB) assessments have been crucial for synthesizing and utilizing trustworthy seroprevalence data for public health decision‐making.</p> <p>Risk of bias refers to systematic error in study results that can arise because of flaws in study design, conduct, analysis, interpretation, or reporting. As such, risk of bias assessment includes identification of study factors and safeguards that protect against systematic error and an empirical construct for making a judgment about the potential bias resulting from the absence of these factors and safeguards (e.g., low, moderate, and high risk of bias).[<reflink idref="bib5" id="ref6">5</reflink>] These constructs vary but can include counting the number of missing study safeguards or identifying important patterns of missing safeguards, depending on the assessors' judgments regarding the possible influence on study estimates.</p> <p>There are several validated tools for RoB assessment of prevalence studies but little consistency in the use of these tools across evidence synthesis efforts.[[<reflink idref="bib6" id="ref7">6</reflink>]] This demonstrates uncertainty about the most appropriate tool for use in evidence synthesis. Furthermore, the validated tools available include heterogeneous evaluation criteria, are time‐consuming to use, and yield RoB assessments with inherent variation given a reliance on evaluators' subjective judgments.[[<reflink idref="bib8" id="ref8">8</reflink>], [<reflink idref="bib10" id="ref9">10</reflink>]] These limitations present a major challenge for rapid and reproducible synthesis of seroprevalence studies. Indeed, meta‐epidemiological reviews of prevalence studies suggest that a unified approach to RoB assessment using a validated tool is needed.[<reflink idref="bib6" id="ref10">6</reflink>]</p> <p>The Joanna Briggs Institute (JBI) Critical Appraisal Checklist for Prevalence Studies is an appropriate candidate to form the basis for such a unified approach.[[<reflink idref="bib6" id="ref11">6</reflink>]] The tool was developed for the explicit purpose of appraising disease prevalence studies by the Joanna Briggs Institute, an organization that develops methodologies and guidance on the process of conducting systematic reviews.[<reflink idref="bib12" id="ref12">12</reflink>] Items were developed based on systematic reviews of prevalence assessment and expert opinion to address key issues of internal and external validity for prevalence studies (i.e., presence or absence of key study safeguards against systematic error).[<reflink idref="bib13" id="ref13">13</reflink>] The tool has been validated by multiple investigator groups and compares favorably to other tools in terms of content validity, construct validity, and usability.[[<reflink idref="bib6" id="ref14">6</reflink>], [<reflink idref="bib14" id="ref15">14</reflink>]] Furthermore, the JBI tool is the most commonly utilized tool for appraisal of prevalence studies in evidence synthesis projects, indicating a greater level of support for this tool relative to the many others available. However, the nine‐item JBI checklist requires subjective judgment for completion. This tool also has no provision for overall RoB assessment, which is often sought by investigators as it can serve as a valuable summary metric for analysis, reporting, and interpretation of a collection of evidence (e.g., sub‐group analysis that exclude studies at higher risk of bias). As a result, this checklist has been operationalized differently by groups conducting evidence synthesis, with each effort defining their own approach to derive overall RoB assessments.[[<reflink idref="bib16" id="ref16">16</reflink>], [<reflink idref="bib18" id="ref17">18</reflink>]] In some instances, this has resulted in markedly different item ratings and RoB assessments for the same body of underlying literature.[[<reflink idref="bib16" id="ref18">16</reflink>], [<reflink idref="bib19" id="ref19">19</reflink>]]</p> <p>We aimed to develop an objective, rapid, and reproducible approach to RoB assessment for seroprevalence studies. In this manuscript, we describe the SeroTracker‐RoB decision rule‐based algorithmic, which involves use of a critical appraisal tool derived from the JBI checklist and application of decision rules to the critical appraisal results to yield an overall RoB assessment. We evaluated the efficiency, coverage, and reliability of this algorithmic approach, and developed an Excel tool that implements this algorithm for use by other researchers.</p> <hd id="AN0163567925-7">METHODS</hd> <p></p> <hd id="AN0163567925-8">The SeroTracker‐RoB decision rule‐based algorithm</hd> <p>We developed an algorithm for RoB assessments of seroprevalence that involves two components: (<reflink idref="bib1" id="ref20">1</reflink>) a critical appraisal checklist that was derived from the Joanna Briggs Institute Critical Appraisal Checklist for Prevalence Studies[<reflink idref="bib12" id="ref21">12</reflink>]; (<reflink idref="bib2" id="ref22">2</reflink>) decision rules that are applied to the responses of the critical appraisal checklist.</p> <hd id="AN0163567925-9">Component 1: Critical appraisal checklist derived from the JBI checklist</hd> <p>The critical appraisal checklist we use is an operationalized version of the JBI Critical Appraisal Checklist for Prevalence Studies. Our operationalization of the items aimed to reduce the number of subjective judgments required and instead allow objective completion based on routinely extracted data from a given seroprevalence study.[<reflink idref="bib20" id="ref23">20</reflink>]</p> <p>The original JBI checklist requires that reviewers subjectively judging nine categorical items based on data reported in a prevalence study: (<reflink idref="bib1" id="ref24">1</reflink>) sample frame appropriateness; (<reflink idref="bib2" id="ref25">2</reflink>) sampling method; (<reflink idref="bib3" id="ref26">3</reflink>) sample size/calculation; (<reflink idref="bib4" id="ref27">4</reflink>) subject and setting described in detail; (<reflink idref="bib5" id="ref28">5</reflink>) representativeness of sample within analysis; (<reflink idref="bib6" id="ref29">6</reflink>) test sensitivity and specificity; (<reflink idref="bib7" id="ref30">7</reflink>) consistent test use; (<reflink idref="bib8" id="ref31">8</reflink>) appropriate statistical adjustment; and (<reflink idref="bib9" id="ref32">9</reflink>) response rate.[<reflink idref="bib12" id="ref33">12</reflink>]</p> <p>In our operationalization, six of the nine JBI checklist items (item 2, 3, 5, 6, 7, and 8) were modified to add conditions that enabled them to be judged in a binary fashion based on objective data reported in a prevalence study (Table 1, Supplementary File 1). For example, Item 2 (were participants recruited in an appropriate way?) was marked as "Yes" if probability sampling methods or entire population sampling was used and "No" if any other method was used.</p> <p>1 TABLE Summary of operationalized critical appraisal checklist items derived from the Joanna Briggs Institute Checklist for Prevalence Studies</p> <p> <ephtml> <table><thead valign="bottom"><tr><th align="left">Operationalized critical appraisal checklist items informing the decision‐rule based algorithm for seroprevalence study risk of bias</th><th align="left">Response options and criteria</th></tr></thead><tbody valign="top"><tr><td>Item A (JBI Item 1): Was the sample frame appropriate to address the target population?</td><td>Yes: Sample frame was described and was appropriate to address the target population.No: Sample frame was not appropriate to address the target population (e.g., blood donors do not represent the general population, doctors do not represent all health care providers).Unclear: Sample frame was not described.</td></tr><tr><td>Item B (JBI Item 2): Were study participants recruited in an appropriate way?</td><td>Yes: Probability sampling method (simple or stratified random) or entire sample (e.g., an entire town) was used.No: Non‐probability sampling method was used.Unclear: Sampling method was described but it was unclear if it was a probability‐based technique.</td></tr><tr><td>Item C (JBI Item 3): Was the sample size adequate?</td><td>Yes: A sample size was calculated based on an assumed prevalence rate and a sample precision that was less than or equal to half the assumed prevalence. Studies meeting the required sample size are judged as yes. The sample size calculation is as follows: n = (Z<sup>2</sup> × P × (1 − P))/d<sup>2</sup>where, n = sample size; Z = Z statistic for level of confidence (95%); P = expected prevalence; d = precision (half the assumed prevalence). In cases where the authors of the included study calculate a required sample size using locally assumed prevalence levels and a level of precision that is less than or equal to half the assumed prevalence, this item is marked as "Yes" if that sample size was successfully acquired.No: Study does not meet the required sample size.Unclear: Sample size not reported.</td></tr><tr><td>Item D (JBI Item 5): Was data analysis conducted with sufficient coverage of the identified sample?</td><td>Yes: The demographic characteristics of the sample were representative of the population in both the main and sub‐group analyses. Our assessment of seroprevalence studies evaluated reporting of age and sex. However, investigators may wish to consider an expanded list of potentially relevant characteristics.No: The demographic characteristics of the sample were not representative of the population in both the main and sub‐group analyses.Unclear: Information is not provided about demographic characteristics of the sample.</td></tr><tr><td>Item E (JBI item 6): Were valid methods used for the identification of the condition?</td><td>Yes: The measurement test or a combination of measurement tests exceeded the threshold for adequate sensitivity and specificity according to leading health authorities and regulators or used one of the following multi‐test algorithms:<list list-type="Bullet"><list-item><p>Two test algorithms using a commercial or in‐house binding assay with confirmatory testing using virus neutralization assay irrespective of combined sensitivity/specificity.</p></list-item><list-item><p>Two test algorithms using a commercial or in‐house binding assay with confirmatory testing using virus neutralization assay irrespective of combined sensitivity/specificity.</p></list-item><list-item><p>Three test confirmatory approach during which two tests are used in parallel followed by a third confirmatory test for discordant results.</p></list-item><list-item><p>Three test series approach whereby positive is defined by at least two positives out of three tests.</p></list-item><list-item><p>Three test parallel approach whereby positive is defined by at least one positive test and no negative tests.</p></list-item></list>No: The detection test does not exceed the threshold for adequate sensitivity and specificity according to leading health authorities and regulators or did not use a valid multi‐test algorithm (see above).Unclear: Test sensitivity and specificity not reported.</td></tr><tr><td>Item F (JBI Item 7): Was the condition measured in a standard, reliable way for all participants?</td><td>Yes: The same measurement test and procedures were used for all participants.No: Different measurement tests and procedures were used for participants.Unclear: No details were provided about which participants received which measurement tests.</td></tr><tr><td>Item G (JBI Item 8): Was there appropriate adjustment for test characteristics?</td><td>Yes: Provided statistical adjustment for test characteristics and the information necessary to determine the numerator, denominator, prevalence estimate, and confidence interval.No: Did not statistically adjust for test characteristics or did not provide the information necessary to determine the numerator, denominator, prevalence estimate, and confidence interval. If adjustment was not reported, then it was considered as having not been conducted.</td></tr><tr><td>Item H (JBI item 8): Was there appropriate adjustment for population characteristics?</td><td>Yes: Provided statistical adjustment for population characteristics or the sample was representative of the population (probability sampling) and provided the information necessary to determine the numerator, denominator, prevalence estimate, and confidence interval.No: Did not statistically adjust for population characteristics and the sample was not likely representative of the population (non‐probability sampling) or did not provide the information necessary to determine the numerator, denominator, prevalence estimate, and confidence interval. If adjustment was not reported, then it was considered as having not been conducted.</td></tr><tr><td>Item I (JBI Item 9): Was the response rate adequate, and if not, was the low response rate unlikely to introduce bias?</td><td>Yes: Response rate was greater than or equal to 60% or the sample was representative of the target population.No: Response rate was less than 60% and the sample was not representative of the target population.Unclear: The response rate was not provided and it was unclear if the sample was representative of the target population.</td></tr><tr><td>Overall risk of bias</td><td>Low: The estimates are very likely correct for the target population. To obtain a low risk of bias classification, all criteria must be met or departures from the criteria must be minimal and unlikely to impact on the validity and reliability of the prevalence estimate. These include sample sizes that are just below the threshold when all other criteria are met,  test characteristics below the threshold but corrections for the test performance, and response rates that are just below the threshold in the context of probability‐based sampling of an appropriate sampling frame with population weighted seroprevalence estimates.Moderate: The estimates are likely correct for the target population. To obtain a moderate risk of bias classification, most criteria must be met and departures from the criteria are likely to have only a small impact on the validity and reliability of the prevalence estimates.High: The estimates are not likely correct for the target population. To obtain a high risk of bias, many criteria must not be met or departures from criteria are likely to have a major impact on the validity and reliability of the prevalence estimates.Unclear: There was insufficient information to assess the risk of bias.</td></tr></tbody></table> </ephtml> </p> <p>Item 8 (appropriate statistical analysis) was split in two parts to differentiate studies that adjusted prevalence estimates for test sensitivity and specificity but not population demographics and vice versa. We separated these items as failing to adjust for test characteristics and failing to adjust for population characteristics have different risk of bias implications from one another. Item 4 was eliminated as it is a reporting item that would not impact on risk of bias. Full details of the operationalization for each item are reported in Supplementary File 1.</p> <p>To clarify when we are referring to the operationalized version and original version, we relabelled the operationalized items using alphabetic characters (Item A–J) and will hereafter refer to the operationalized checklist as the critical appraisal checklist (Supplementary File 1).</p> <p>The critical appraisal checklist items were then categorized as relating to selection biases or measurement biases. Selection biases related to the extent to which the seroprevalence estimated in the study sample was representative of the seroprevalence in the target population (items A, B, C, D, H, and I). Measurement biases were those related to measurement error (items E, F, and G).</p> <p>As a first step towards a tool to implement the algorithmic RoB evaluations, we created coding logic for each critical appraisal item based on key information that would be extracted as part of a critical appraisal or systematic review of prevalence studies (Supplementary File 1).[<reflink idref="bib21" id="ref34">21</reflink>] Seven of the nine items can be instantaneously completed using this coding logic applied to data extracted from studies. Coding logic could not be created for two items, as they require study‐, sample‐, and target population‐specific contextual judgments: item A (whether the sample frame was representative of the target population) and item D (whether the characteristics of the sample were representative of the target population, in both the main and sub‐group analyses). We added to pre‐existing JBI guidance to offer detailed instructions and illustrative examples on how to complete these two subjective items.</p> <hd id="AN0163567925-10">Component 2: Developing decision rules for risk of bias assessment</hd> <p>We developed decision rules that could be applied to the critical appraisal items responses to generate an overall RoB assessment (low, moderate, or high) for each study (Figure 1). The use of decision rules was selected as it allows for all checklist items to be considered together in evaluating overall study RoB. The decision rules were developed based on published guidance on estimating disease prevalence,[<reflink idref="bib21" id="ref35">21</reflink>] reports on the evaluation of prevalence studies,[[<reflink idref="bib6" id="ref36">6</reflink>], [<reflink idref="bib22" id="ref37">22</reflink>]] evidence of the impact of bias on research results,[[<reflink idref="bib23" id="ref38">23</reflink>], [<reflink idref="bib25" id="ref39">25</reflink>], [<reflink idref="bib27" id="ref40">27</reflink>]] opinions of experts in evidence synthesis and infectious disease epidemiology, and guidance created by SeroTracker researchers after evaluation of thousands of seroprevalence studies (Supplementary File 1).[[<reflink idref="bib2" id="ref41">2</reflink>], [<reflink idref="bib4" id="ref42">4</reflink>]]</p> <p> <img src="https://imageserver.ebscohost.com/img/embimages/rdk/BDCT/01may23/jrsm1620-fig-0001.jpg?ephost1=dGJyMNXb4kSepq84yOvqOLCmsE6epq5Srqa4SK6WxWXS" alt="jrsm1620-fig-0001.jpg" title="1 SeroTracker decision rules for risk of bias assessment [Colour figure can be viewed at wileyonlinelibrary.com]" /> </p> <p></p> <p>The decision rules were based on the extent of selection and measurement bias present in a study. Meta‐epidemiological studies have shown that selection and measurement biases have been associated with overestimates and underestimates.[[<reflink idref="bib23" id="ref43">23</reflink>], [<reflink idref="bib25" id="ref44">25</reflink>], [<reflink idref="bib27" id="ref45">27</reflink>]] As such, the extent of these biases was mapped to the different levels of risk of bias.</p> <p>We defined selection bias to occur when the sample from which estimates are derived differs systematically from the target population. Selection bias may arise from factors affecting sample recruitment, size, and retention including using a sample frame that is not representative of the target population, a low response rate which may increase the chance that participants differ from non‐participants and drop‐outs, and a low sample size which may increase the chance of known and unknown imbalances in participant characteristics compared to the target population.[[<reflink idref="bib22" id="ref46">22</reflink>], [<reflink idref="bib25" id="ref47">25</reflink>]] We defined measurement bias to occur when there is systematic error in the measurement of the outcome. Measurement bias may arise from low detection test sensitivity and specificity or use of different measurement methods for different participants.[[<reflink idref="bib22" id="ref48">22</reflink>], [<reflink idref="bib27" id="ref49">27</reflink>]]</p> <p>The decision rules considered studies to be low RoB if there was no selection bias (i.e., appropriate sample frame, probability sampling, adequate sample size, and statistical adjustment for population characteristics) and no measurement bias (i.e., adequate sensitivity/specificity and/or adjustment for test performance). These rules generally considered studies at moderate RoB if they had some selection bias (e.g., inappropriate sample frame but probability sampling, adequate sample size, and statistical adjustment for population characteristics) or no measurement biases. In contrast, studies were considered high RoB if the study had considerable selection bias (e.g., non‐probability sampling and inadequate sample size) and considerable measurement bias (i.e., poor sensitivity/specificity and lack of adjustment for test performance).</p> <hd id="AN0163567925-12">Validation dataset</hd> <p>The SeroTracker‐RoB decision‐rule based algorithm was evaluated using data from SeroTracker's living systematic review database of SARS‐CoV‐2 seroprevalence studies.[<reflink idref="bib3" id="ref50">3</reflink>] The protocol for the review was registered and published (PROSPERO: CRD42020183634, version July 21, 2021).</p> <p>The dataset used for validation included all seroprevalence studies included in the SeroTracker database with publication dates between January 1, 2020 to November 17, 2021. The 2070 studies in this evaluation dataset included peer‐reviewed literature, preprints, government and non‐governmental organization reports, and media articles.</p> <p>Non‐algorithmic RoB and algorithmic RoB assessments were completed for each study. Non‐algorithmic RoB assessments were completed by two independent reviewers, each of whom examined data routinely extracted from each article to complete the critical appraisal checklist (Supplementary File 1) and performed a RoB assessment using general guidance that we developed and used as a team prior to the development of the algorithm (Supplementary File 1). This guidance described key considerations for assessing risk of bias but did not function as a formalized algorithm. There were five directives regarding conditions wherein studies should not be rated as low risk of bias; however, we did not measure adherence to these directives and reviewers could override the guidance if they provided reasonable justification and a second reviewer agreed with the justification.</p> <p>Disagreements were resolved by consensus. Algorithmic RoB assessments were conducted using the SeroTracker‐RoB decision rule‐based algorithm as outlined above, involving application of the coding logic for the critical appraisal checklist items and decision rules to the same data that was used to generate the non‐algorithmic assessments. The coding logic and decision rules were implemented in an AirTable programmable spreadsheet.</p> <hd id="AN0163567925-13">Evaluating efficiency, coverage, and reliability</hd> <p></p> <hd id="AN0163567925-14">Sample size</hd> <p>We calculated the required sample size for analysis to precisely estimate inter‐rater reliability between the algorithmic and non‐algorithmic approaches.[[<reflink idref="bib29" id="ref51">29</reflink>], [<reflink idref="bib31" id="ref52">31</reflink>]] To inform the sample size calculation an estimate of the expected reliability was required. We used estimates of reliability for the non‐algorithmic approach, the baseline approach to RoB assessment for articles included in the SeroTracker database, to serve as the expected reliability in the calculation. The database included ratings for 2070 articles conducted by 12 reviewers in pairs. The reliability for the non‐algorithmic approach was determined by calculating absolute agreement using a two‐way random‐effects single‐measures ICC and was found to be 0.74 (95% CI 0.71–0.76). With <emph>k</emph> = 2 raters (non‐algorithmic vs. algorithm), alpha = 0.05, and an anticipated ICC of 0.74 the required sample size to estimate ICC within a precision of 0.2 was 80.</p> <hd id="AN0163567925-15">Efficiency</hd> <p>We assessed efficiency by comparing time to complete non‐algorithmic and algorithmic RoB assessments for 80 seroprevalence articles that were randomly selected from the SeroTracker dataset. Five SeroTracker systematic reviewers were timed as they each completed between five and 50 non‐algorithmic assessments. The time required for the non‐algorithmic approach included assessment of each checklist item and evaluation of overall RoB. The time required for the algorithmic approach included only the time taken to evaluate two critical appraisal checklist items requiring subjective judgment (Item A, Item D). There was instantaneous application of the coding logic for the other seven objectively determined critical appraisal items and the decision rules for RoB. We quantify the mean difference in time between the non‐algorithmic and algorithmic approaches. Formal statistical testing was not conducted as all algorithmic assessments were, by definition, shorter in duration given their instantaneous completion based on coding logic.</p> <hd id="AN0163567925-16">Coverage</hd> <p>To determine coverage, we calculated the proportion of studies in the dataset for which the decision rule‐based algorithm yielded a RoB assessment.</p> <hd id="AN0163567925-17">Reliability</hd> <p>To evaluate reliability of the RoB assessments across consensus non‐algorithmic and algorithmic approaches, we calculated the absolute agreement between overall RoB assessments using a two‐way random‐effects average‐measures intraclass correlation (ICC).[[<reflink idref="bib29" id="ref53">29</reflink>], [<reflink idref="bib31" id="ref54">31</reflink>]] We used the entire SeroTracker database of 2070 articles to assess the reliability of the algorithmic compared to non‐algorithmic approach. All data are available at SeroTracker.com. Analyses were conducted using STATA 14 (College Station, TX: StataCorp LP).</p> <hd id="AN0163567925-18">RESULTS</hd> <p></p> <hd id="AN0163567925-19">Decision rules for RoB</hd> <p>The decision tree to classify overall study RoB is shown in Figure 1. The decision tree classified overall RoB as low, moderate, or high based on the categorical ratings for each critical appraisal checklist item, considering each item in turn to arrive at an overall judgment (i.e., item A, item B, etc.). The tree included 61 decision rules, which could lead to low (<emph>n</emph> = 6), moderate (<emph>n</emph> = 29), or high (<emph>n</emph> = 26) RoB.</p> <hd id="AN0163567925-20">Efficiency</hd> <p>Use of the SeroTracker‐RoB algorithmic approach resulted in faster RoB assessments compared to the non‐algorithmic approach with a mean reduction of 2.32 min (SD 1.09) per article (algorithm mean 0.64 [SD 0.24] vs. non‐algorithm mean 2.97 [SD 1.16]). The algorithmic approach took less time than the non‐algorithmic approach for all 80 articles evaluated.</p> <hd id="AN0163567925-21">Coverage</hd> <p>The algorithmic approach yielded a RoB assessment for 100% (<emph>n</emph> = 2070) of the SARS‐CoV‐2 seroprevalence studies in the SeroTracker database.</p> <hd id="AN0163567925-22">Reliability</hd> <p>ICC for the reliability of the RoB assessment was 0.77 (95% confidence interval [CI] 0.74–0.80) between the SeroTracker‐RoB algorithmic approach and consensus non‐algorithmic review. A summary of the risk of bias assessments for the algorithmic and non‐algorithmic approaches are shown in Figure 2.</p> <p> <img src="https://imageserver.ebscohost.com/img/embimages/rdk/BDCT/01may23/jrsm1620-fig-0002.jpg?ephost1=dGJyMNXb4kSepq84yOvqOLCmsE6epq5Srqa4SK6WxWXS" alt="jrsm1620-fig-0002.jpg" title="2 Comparison of non‐algorithmic and algorithmic risk of bias assessments for 2070 seroprevalence studies [Colour figure can be viewed at wileyonlinelibrary.com]" /> </p> <p></p> <hd id="AN0163567925-24">The SeroTracker‐RoB Excel Tool</hd> <p>A Microsoft Excel tool (the "SeroTracker‐RoB Excel Tool") that implements the algorithmic approach to prevalence study RoB assessment can be found in Supplementary File 2. This tool includes the operationalized critical appraisal checklist items with coding logic and decision rules. The decision tree logic was implemented in Excel using Visual Basic for Applications.</p> <p>The SeroTracker‐RoB Excel Tool includes four sheets: (<reflink idref="bib1" id="ref55">1</reflink>) a legend sheet to describe the tool and orient the user; (<reflink idref="bib2" id="ref56">2</reflink>) a data extraction sheet with 25 core data fields for reviews of prevalence studies, including the two critical appraisal checklist items requiring subjective assessment; seven user‐defined thresholds for objectively determined critical appraisal items (i.e., minimum sample size, test validity, and response rate); seven objective critical appraisal item variables; and a variable with the algorithmically determined RoB rating; (<reflink idref="bib3" id="ref57">3</reflink>) a sheet summarizing the risk of bias ratings from the data extraction sheet; a data dictionary sheet with descriptions of the data extraction fields, instructions for extraction, and the critical appraisal item criteria and coding logic for each item; and (<reflink idref="bib4" id="ref58">4</reflink>) a data validation sheet where options for dropdown menus can be edited by the user. Using built in formulas and these extracted fields, the tool completes the checklist and produces an overall RoB rating. Users who wish to manually enter responses to the critical appraisal items, as opposed to rely on coding logic applied to extracted study data, can enter their ratings into the nine critical appraisal item variables, and still take advantage of the SeroTracker‐RoB decision rules for an algorithmically determined overall RoB assessment.</p> <hd id="AN0163567925-25">DISCUSSION</hd> <p>We developed the SeroTracker‐RoB decision rule‐based algorithm for RoB assessments of seroprevalence studies. This tool has the potential to lay the foundation for rapid, robust, and reproducible RoB assessment of prevalence studies.</p> <p>There is no consensus on the most valid approach to expediting RoB assessments.[[<reflink idref="bib6" id="ref59">6</reflink>]] Some studies have used summary scores, adding the number of criteria met for a given critical appraisal checklist and creating a threshold on that score to assess RoB.[[<reflink idref="bib16" id="ref60">16</reflink>], [<reflink idref="bib18" id="ref61">18</reflink>]] However, this approach weights each checklist item equally, which does not reflect the different implications of critical appraisal concepts for RoB. For example, when considering different item ratings for the critical appraisal checklist, marking six out of nine items "Yes" could be achieved by 84 different combinations of item responses, which have different implications for RoB.[[<reflink idref="bib22" id="ref62">22</reflink>], [<reflink idref="bib25" id="ref63">25</reflink>], [<reflink idref="bib27" id="ref64">27</reflink>], [<reflink idref="bib33" id="ref65">33</reflink>], [<reflink idref="bib35" id="ref66">35</reflink>]]</p> <p>A comparison of two reviews of seroprevalence studies in Africa highlights how different approaches to risk of bias based on the JBI checklist can result in different assessments for the same underlying body of literature.[[<reflink idref="bib16" id="ref67">16</reflink>], [<reflink idref="bib19" id="ref68">19</reflink>]] The reviews used similar search strategies and study inclusion criteria. One review used a summary scoring method whereby more JBI items scored as "Yes" indicated higher quality and lower risk of bias.[<reflink idref="bib16" id="ref69">16</reflink>] The items rated as "Yes" were summated and 9, 5–7 and ≤4 indicated high quality, moderate quality and low quality studies. They found that the majority were generally at lower risk of bias (74% high quality, 22% moderate quality, and 4% low quality). In contrast, a similar review using the SeroTracker algorithmic approach found that most studies were at moderate or high risk of bias (24% low risk of bias, 40% moderate risk of bias, and 37% high risk of bias).[<reflink idref="bib19" id="ref70">19</reflink>]</p> <p>Weighted averages better reflect the relevance of each item, overcoming some limitations of simple summary scores. However, checklist items cannot always be considered independently in assessing RoB, thereby introducing complexity in the derivation of a weighted score. For example, it is more important to correct for antibody test performance (item I = yes) when assay sensitivity and specificity are low (item E = no).</p> <p>Considering combinations of items together may enable better expedited RoB assessment. Some studies have trained deep learning algorithms to identify relevant text in publications of randomized controlled trials and predict RoB assessments, with reasonable accuracy compared to human reviewers.[[<reflink idref="bib33" id="ref71">33</reflink>]] However, these algorithms have largely been trained on small datasets and have limited interpretability and transparency due to the "black‐box" decision making of the models.</p> <p>A decision rule‐based approach, on the other hand, provides a transparent and interpretable model to expedite RoB assessment.[<reflink idref="bib35" id="ref72">35</reflink>] The tree structure captures interactions between critical appraisal checklist features, is easy to implement, and enables clear visualization of which combinations of features are most important for RoB. The decision tree in the SeroTracker‐RoB algorithmic approach reveals two key axes of bias in seroprevalence studies: selection biases, related to the correspondence between the sample frame and target population, sampling representativeness, response rate, and population weighting; and measurement biases, related to detection test performance, correction for that performance, and consistency of test use across participants. Clusters of rules in this decision tree also make clear how features in each axis interact to determine study RoB.</p> <p>RoB assessment is a time‐consuming component of conducting systematic reviews.[[<reflink idref="bib5" id="ref73">5</reflink>], [<reflink idref="bib8" id="ref74">8</reflink>]] As such, improving efficiency of the assessment process may help to reduce the burden of evidence synthesis. This is particularly important for living reviews and reviews during health emergencies, which may need to process a high volume of information quickly to inform public health decision making.</p> <p>The SeroTracker project exemplifies how time‐savings can be of value. There is an average of 35 new studies added each week to the SeroTracker living systematic review project.[<reflink idref="bib3" id="ref75">3</reflink>] The SeroTracker‐RoB algorithmic approach described in this manuscript is 4.6 times faster than the non‐algorithmic approach for assessing study RoB, which took approximately 3 min. Thus, using the algorithmic approach would save nearly 3 h of reviewer time each week.</p> <p>In traditional non‐algorithmic RoB assessment, there is imperfect agreement between even trained human reviewers.[[<reflink idref="bib5" id="ref76">5</reflink>], [<reflink idref="bib8" id="ref77">8</reflink>], [<reflink idref="bib10" id="ref78">10</reflink>]] Inter‐rater reliability for RoB between two independent reviewers using the non‐algorithmic approach was moderate (ICC 0.74). As such, non‐algorithmic review cannot be considered a perfect standard, and the 0.77 ICC between the SeroTracker‐RoB algorithmic and non‐algorithmic approach in part reflects the heterogeneity and inconsistency of non‐algorithmic assessment. This highlights the benefit of an algorithmic approach that yields reproducible assessments.</p> <p>There have been many systematic reviews of severe acute respiratory syndrome coronavirus 2 (SARS‐CoV‐2) seroprevalence studies to date.[[<reflink idref="bib2" id="ref79">2</reflink>], [<reflink idref="bib4" id="ref80">4</reflink>], [<reflink idref="bib16" id="ref81">16</reflink>], [<reflink idref="bib18" id="ref82">18</reflink>], [<reflink idref="bib36" id="ref83">36</reflink>]] Several of these reviews use the JBI checklist as a foundation for critical appraisal, but they implement the checklist and judge overall RoB in different ways.[[<reflink idref="bib2" id="ref84">2</reflink>], [<reflink idref="bib4" id="ref85">4</reflink>], [<reflink idref="bib16" id="ref86">16</reflink>], [<reflink idref="bib18" id="ref87">18</reflink>]] For this reason, we developed the SeroTracker‐RoB Excel Tool (Supplementary File 2), which embeds the coding logic for instantaneous completion of seven of the nine critical appraisal items and application of the decision rules in a user‐friendly data extraction sheet. This tool may be valuable to other investigators seeking to conduct rapid and reliable RoB assessments for prevalence studies—an important endeavor, given the ongoing value of SARS‐CoV‐2 seroprevalence studies for surveillance as the virus becomes endemic,[<reflink idref="bib37" id="ref88">37</reflink>] and given the many prevalence studies conducted for other diseases and conditions.</p> <p>The decision rule‐based algorithm may be suitable for use in evaluating a broad array of prevalence studies. Users of the algorithm tool can adapt the thresholds for the objectively determined critical appraisal items to meet their unique needs. For example, we judged valid methods of detecting SARS‐CoV‐2 antibodies using 90% sensitivity and 97% specificity thresholds established by the World Health Organization,[<reflink idref="bib38" id="ref89">38</reflink>] but this criterion will vary for other conditions. To accommodate these needs, the SeroTracker‐RoB Excel Tool includes embedded user‐set thresholds that can easily be altered as required. Our description of each item in Supplementary File 1 also includes clear instructions on which criteria users may wish to alter for their own purposes.</p> <p>This study had several strengths. First, to our knowledge, this is the first validation of decision rules for RoB in seroprevalence studies, with the validation dataset including thousands of heterogeneous SARS‐CoV‐2 seroprevalence studies across study designs, regions, and target populations. Second, the use of a decision rule approach enables transparent RoB assessment, while also ensuring reproducibility and speed. Finally, the decision rules have a robust foundation in the validated and widely used JBI critical appraisal checklist, published guidance on estimating disease prevalence, evidence on the impact of bias on empirical estimates, opinions of methodological experts, and the experiences of researchers that have conducted thousands of RoB assessments for seroprevalence studies.</p> <p>This study had several limitations. First, the SeroTracker‐RoB algorithmic approach requires reviewers to make subjective judgments on two critical appraisal items (Item A, Item D), which may be associated with response variation and reviewer burden. However, we provide clear guidance for making those assessments. It is theoretically possible to develop objective standards for the completion of Item A and Item D however, there are practical challenges to this. For example, Item A (Was the sample frame appropriate to address the target population?) could be operationalized by determining whether evidence had been cited to show that the proposed sample frame had previously been shown to be representative of a target population. However, this would require substantial pre‐existing literature, which may not exist for every topic. Item D (whether the characteristics of the sample were representative of the target population, in both the main and sub‐group analyses) could be operationalized by assessing whether statistical testing demonstrated that the sample demographics were not different from the population demographics for every analysis. However, this would still require judgment about which demographic characteristics should be matching. The relevant set of characteristics may be specific to the diseases and populations being studied. Thus, subjectivity and content expertise may always be required for these items. Second, the decision rule was derived in part using expert judgment. However, the transparency of this algorithm allows for scientific debate and further refinement of the decision rules, if needed. Third, the generalizability of the SeroTracker‐RoB decision rules is unclear. Although the validation database used was large and robust, the data came exclusively from SARS‐CoV‐2 seroprevalence studies. Applying this approach to prevalence studies for other pathogens or conditions may require additional validation or adaptation.</p> <hd id="AN0163567925-26">CONCLUSIONS</hd> <p>We developed and validated the SeroTracker‐RoB decision rule‐based algorithm, which enables rapid, transparent, and reproducible risk of bias assessment for seroprevalence studies. This algorithm includes seven objectively determined critical appraisal items and two subjectively determined critical appraisal items derived from the Joanna Briggs Institute Critical Appraisal Checklist for Prevalence studies, and interpretable decision rules to assess overall risk of bias. The SeroTracker‐RoB Excel Tool embeds the algorithm in a data extraction sheet enabling its use by other researchers conducting evidence synthesis of seroprevalence studies and adaptation by researchers conducting evidence synthesis of other types of prevalence studies.</p> <hd id="AN0163567925-27">AUTHOR CONTRIBUTIONS</hd> <p> <bold>Niklas Bobrovitz:</bold> Conceptualization (equal); Formal analysis (equal); Funding acquisition (equal); Investigation (equal); Methodology (equal); Project administration (supporting); Supervision (equal); Validation (equal); Writing – original draft (equal); Writing – reviewing and editing (equal). <bold>Kim Noël:</bold> Formal analysis (equal); Investigation (equal); Methodology (equal); Visualization (supporting); Writing – original draft (equal); Writing – reviewing and editing (supporting). <bold>Zihan Li:</bold> Data curation (lead); Formal analysis (equal); Investigation (equal); Methodology (equal); Software (equal); Validation (equal); Visualization (lead); Writing – reviewing and editing (supporting). <bold>Christian Cao:</bold> Investigation (equal); Writing – reviewing and editing (supporting). <bold>Gabriel Deveaux:</bold> Formal analysis (equal); Methodology (equal); Software (equal); Writing – reviewing and editing (supporting). <bold>Anabel Selemon:</bold> Investigation (equal); Writing – reviewing and editing (supporting). <bold>David A. Clifton:</bold> Methodology (equal); Supervision (equal); Writing – reviewing and editing (supporting). <bold>Mercedes Yanes‐Lane:</bold> Investigation (equal); Writing – reviewing and editing (supporting). <bold>Tingting Yan:</bold> Funding acquisition (equal); Project administration (equal); Writing – reviewing and editing (supporting). <bold>Rahul K. Arora:</bold> Conceptualization (equal); Formal analysis (equal); Funding acquisition (equal); Methodology (equal); Project administration (equal); Supervision (equal); Writing – original draft (equal); Writing – reviewing and editing (equal).</p> <hd id="AN0163567925-28">FUNDING INFORMATION</hd> <p>SeroTracker receives funding for SARS‐CoV‐2 seroprevalence study evidence synthesis from the Public Health Agency of Canada through Canada's COVID‐19 Immunity Task Force, the World Health Organization Health Emergencies Programme, the Robert Koch Institute, and the Canadian Medical Association Joule Innovation Fund. No funding source had any role in the design of this study, its execution, analyses, interpretation of the data, or decision to submit results. This manuscript does not necessarily reflect the views of the World Health Organization or any other funder.</p> <hd id="AN0163567925-29">CONFLICT OF INTEREST</hd> <p>Rahul K. Arora was previously a Technical Consultant for the Bill and Melinda Gates Foundation Strategic Investment Fund, is a minority shareholder of Alethea Medical, and was a former Senior Policy Advisor at Health Canada. Each of these relationships is entirely unrelated to the present work. David A. Clifton reports consulting fees from Sensyne Health, Oxford University Innovation, and BioBeats, each outside the submitted work. Mercedes Yanes‐Lane has received grants from the Bill and Melinda Gates Foundation, not pertaining to the present study. No other authors have conflicts of interest to report.</p> <hd id="AN0163567925-30">DATA AVAILABILITY STATEMENT</hd> <p>All data are available at SeroTracker.com.</p> <p>GRAPH: Supplementary file 1 Full details of the critical appraisal checklist items derived from the Joanna Briggs Institute Checklist for Prevalence Studies and principles guiding the non‐algorithmic and algorithmic risk of bias assessment.</p> <p>GRAPH: Supplementary file 2 SeroTracker decision rule‐based algorithm tool for overall risk of bias assessment.</p> <ref id="AN0163567925-31"> <title> Footnotes </title> <blist> <bibl id="bib1" idref="ref1" type="bt">1</bibl> <bibtext> Niklas Bobrovitz and Kim Noël contributed equally as co‐first author</bibtext> </blist> </ref> <ref id="AN0163567925-32"> <title> REFERENCES </title> <blist> <bibtext> Cheng MP, Yansouni CP, Basta NE, et al. Serodiagnostics for severe acute respiratory syndrome‐related coronavirus 2: a narrative review. Ann Intern Med. 2020 ; 173 (6): 450 ‐ 460. doi: 10.7326/M20‐2854</bibtext> </blist> <blist> <bibl id="bib2" idref="ref2" type="bt">2</bibl> <bibtext> Bobrovitz N, Arora RK, Cao C, et al. Global seroprevalence of SARS‐CoV‐2 antibodies: a systematic review and meta‐analysis. PLOS One. 2021 ; 16 (6): e0252617. doi: 10.1371/journal.pone.0252617</bibtext> </blist> <blist> <bibl id="bib3" idref="ref3" type="bt">3</bibl> <bibtext> Arora RK, Joseph A, Van Wyk J, et al. SeroTracker: a global SARS‐CoV‐2 seroprevalence dashboard. Lancet Infect Dis. 2021 ; 21 (4): e75 ‐ e76. doi: 10.1016/S1473‐3099(20)30631‐9</bibtext> </blist> <blist> <bibl id="bib4" idref="ref5" type="bt">4</bibl> <bibtext> Bergeri I, Whelan MG, Ware H, et al. Global SARS‐CoV‐2 seroprevalence from January 2020 to April 2022: a systematic review and meta‐analysis of standardized population‐based studies. PLoS Med. 2022 ; 19 (11): e1004107. doi: 10.1371/journal.pmed.1004107</bibtext> </blist> <blist> <bibl id="bib5" idref="ref6" type="bt">5</bibl> <bibtext> Hartling L, Ospina M, Liang Y, et al. Risk of bias versus quality assessment of randomised controlled trials: cross sectional study. BMJ. 2009 ; 339 : b4012. doi: 10.1136/bmj.b4012</bibtext> </blist> <blist> <bibl id="bib6" idref="ref7" type="bt">6</bibl> <bibtext> Migliavaca CB, Stein C, Colpani V, Munn Z, Falavigna M. Quality assessment of prevalence studies: a systematic review. J Clin Epidemiol. 2020 ; 127 : 59 ‐ 68. doi: 10.1016/j.jclinepi.2020.06.039</bibtext> </blist> <blist> <bibl id="bib7" idref="ref30" type="bt">7</bibl> <bibtext> Borges Migliavaca C, Stein C, Colpani V, et al. How are systematic reviews of prevalence conducted? A methodological study. BMC Med Res Methodol. 2020 ; 20 (1): 96. doi: 10.1186/s12874‐020‐00975‐3</bibtext> </blist> <blist> <bibl id="bib8" idref="ref8" type="bt">8</bibl> <bibtext> Hartling L, Bond K, Vandermeer B, Seida J, Dryden DM, Rowe BH. Applying the risk of bias tool in a systematic review of combination long‐acting beta‐agonists and inhaled corticosteroids for persistent asthma. PLOS One. 2011 ; 6 (2): e17242. doi: 10.1371/journal.pone.0017242</bibtext> </blist> <blist> <bibl id="bib9" idref="ref32" type="bt">9</bibl> <bibtext> Hartling L, Hamm MP, Milne A, et al. Testing the risk of bias tool showed low reliability between individual reviewers and across consensus assessments of reviewer pairs. J Clin Epidemiol. 2013 ; 66 (9): 973 ‐ 981. doi: 10.1016/j.jclinepi.2012.07.005</bibtext> </blist> <blist> <bibtext> Hartling L, Milne A, Hamm MP, et al. Testing the Newcastle Ottawa scale showed low reliability between individual reviewers. J Clin Epidemiol. 2013 ; 66 (9): 982 ‐ 993. doi: 10.1016/j.jclinepi.2013.03.003</bibtext> </blist> <blist> <bibtext> Hoy D, Brooks P, Woolf A, et al. Assessing risk of bias in prevalence studies: modification of an existing tool and evidence of interrater agreement. J Clin Epidemiol. 2012 ; 65 (9): 934 ‐ 939. doi: 10.1016/j.jclinepi.2011.11.014</bibtext> </blist> <blist> <bibtext> Joanna Briggs Institute. The Joanna Briggs Institute. https://jbi.global/</bibtext> </blist> <blist> <bibtext> Munn Z, Moola S, Riitano D, Lisy K. The development of a critical appraisal tool for use in systematic reviews addressing questions of prevalence. Int J Health Policy Manag. 2014 ; 3 (3): 123 ‐ 128. doi: 10.15171/ijhpm.2014.71</bibtext> </blist> <blist> <bibtext> Glasgow MJ, Edlin R, Harding JE. Comparison of risk‐of‐bias assessment approaches for selection of studies reporting prevalence for economic analyses. BMJ Open. 2020 ; 10 (9): e037324. doi: 10.1136/bmjopen‐2020‐037324</bibtext> </blist> <blist> <bibtext> Ma LL, Wang YY, Yang ZH, Huang D, Weng H, Zeng XT. Methodological quality (risk of bias) assessment tools for primary and secondary medical studies: what are they and which is better? Mil Med Res. 2020 ; 7 (1): 7. doi: 10.1186/s40779‐020‐00238‐8</bibtext> </blist> <blist> <bibtext> Chisale MRO, Ramazanu S, Mwale SE, et al. Seroprevalence of anti‐SARS‐CoV‐2 antibodies in Africa: a systematic review and meta‐analysis. Rev Med Virol. 2021 ; 32 : e2271. doi: 10.1002/rmv.2271</bibtext> </blist> <blist> <bibtext> Galanis P, Vraka I, Fragkou D, Bilali A, Kaitelidou D. Seroprevalence of SARS‐CoV‐2 antibodies and associated factors in healthcare workers: a systematic review and meta‐analysis. J Hosp Infect. 2021 ; 108 : 120 ‐ 134. doi: 10.1016/j.jhin.2020.11.008</bibtext> </blist> <blist> <bibtext> Rostami A, Sepidarkish M, Leeflang MMG, et al. SARS‐CoV‐2 seroprevalence worldwide: a systematic review and meta‐analysis. Clin Microbiol Infect off Publ Eur Soc Clin Microbiol Infect Dis. 2021 ; 27 (3): 331 ‐ 340. doi: 10.1016/j.cmi.2020.10.020</bibtext> </blist> <blist> <bibtext> Lewis H, Ware H, Whelan M, et al. SARS‐CoV‐2 infection in Africa: a systematic review and meta‐analysis of standardised seroprevalence studies, from January 2020 to December 2021. medRxiv ; 2022.</bibtext> </blist> <blist> <bibtext> World Health Organization Seroepidemiology Technical Working Group. ROSES‐S: statement from the World Health Organization on the reporting of seroepidemiologic studies for SARS‐CoV‐2. Influenza Other Respi Viruses. 2021 ; 15 (5): 561 ‐ 568. doi: 10.1111/irv.12870</bibtext> </blist> <blist> <bibtext> Munn Z, Moola S, Lisy K, Riitano D, Tufanaru C. Methodological guidance for systematic reviews of observational epidemiological studies reporting prevalence and cumulative incidence data. JBI Evid Implement. 2015 ; 13 (3): 147 ‐ 153. https://journals.lww.com/ijebh/Fulltext/2015/09000/Methodological_guidance_for_systematic_reviews_of.6.aspx</bibtext> </blist> <blist> <bibtext> Sterne J, Hernan M, McAleenan A, Reeves B, Higgins J. Chapter 25: Assessing risk of bias in a non‐randomized study. Cochrane Handbook for Systematic Reviews of Interventions Version 6.3 (Updated February 2022). Cochrane ; 2022. <ulink href="http://www.training.cochrane.org/handbook">www.training.cochrane.org/handbook</ulink></bibtext> </blist> <blist> <bibtext> Berkman N, Santaguida P, Viswanathan M, Morton S. The Empirical Evidence of Bias in Trials Measuring Treatment Differences. Agency for Healthcare Research and Quality ; 2014. https://<ulink href="http://www.ncbi.nlm.nih.gov/books/NBK253181/">www.ncbi.nlm.nih.gov/books/NBK253181/</ulink></bibtext> </blist> <blist> <bibtext> Savović J, Jones H, Altman D, et al. Influence of reported study design characteristics on intervention effect estimates from randomised controlled trials: combined analysis of meta‐epidemiological studies. Health Technol Assess. 2012 ; 16 (35): 1 ‐ 82. doi: 10.3310/hta16350</bibtext> </blist> <blist> <bibtext> Catalogue of Bias Collaboration, Nunan D, Bankhead C, Aronson J. Selection Bias. Centre for Evidence Based Medicine ; 2017. https://catalogofbias.org/biases/selection-bias/</bibtext> </blist> <blist> <bibtext> Catalogue of Bias Collaboration, Spencer E, Brassey J, Mahtani K, Heneghan C. Wrong Sample Size Bias. Centre for Evidence Based Medicine ; 2017. https://catalogofbias.org/biases/wrong-sample-size-bias/</bibtext> </blist> <blist> <bibtext> Catalogue of Bias Collaboration, O'Sullivan J, Banerjee A, Pluddemann A. Detection Bias. Centre for Evidence Based Medicine ; 2017. https://catalogofbias.org/biases/detection-bias/</bibtext> </blist> <blist> <bibtext> Catalogue of Bias Collaboration, Spencer E, Mahtani K, Brassey J, Heneghan C. Misclassification Bias. Centre for Evidence Based Medicine ; 2018. https://catalogofbias.org/biases/misclassification-bias/</bibtext> </blist> <blist> <bibtext> Koo TK, Li MY. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J Chiropr Med. 2016 ; 15 (2): 155 ‐ 163. doi: 10.1016/j.jcm.2016.02.012</bibtext> </blist> <blist> <bibtext> McGraw K, Wong S. Forming inferences about some intraclass correlation coefficients. Psychol Methods. 1996 ; 1 (1): 30 ‐ 46. doi: 10.1037/1082‐989X.1.1.30</bibtext> </blist> <blist> <bibtext> Shrout P, Fleiss J. Intraclass correlations: uses in assessing rater reliability. Psychol Bull. 1979 ; 86 (2): 420 ‐ 428. doi: 10.1037/0033‐2909.86.2.420</bibtext> </blist> <blist> <bibtext> Bonett DG. Sample size requirements for estimating intraclass correlations with desired precision. Stat Med. 2002 ; 21 (9): 1331 ‐ 1335. doi: 10.1002/sim.1108</bibtext> </blist> <blist> <bibtext> Sanderson S, Tatt ID, Higgins JP. Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography. Int J Epidemiol. 2007 ; 36 (3): 666 ‐ 676. doi: 10.1093/ije/dym018</bibtext> </blist> <blist> <bibtext> Jüni P, Witschi A, Bloch R, Egger M. The hazards of scoring the quality of clinical trials for meta‐analysis. Jama. 1999 ; 282 (11): 1054 ‐ 1060. doi: 10.1001/jama.282.11.1054</bibtext> </blist> <blist> <bibtext> Greenland S, O'rourke K. On the bias produced by quality scores in meta‐analysis, and a hierarchical view of proposed solutions. Biostatistics. 2001 ; 2 (4): 463 ‐ 471. doi: 10.1093/biostatistics/2.4.463</bibtext> </blist> <blist> <bibtext> Chen X, Chen Z, Azman AS, et al. Serological evidence of human infection with SARS‐CoV‐2: a systematic review and meta‐analysis. Lancet Glob Health. 2021 ; 9 (5): e598 ‐ e609. doi: 10.1016/S2214‐109X(21)00026‐7</bibtext> </blist> <blist> <bibtext> European Centre for Disease Prevention and Control. COVID‐19 Surveillance Guidance. European Centre for Disease Prevention and Control ; 2021. https://<ulink href="http://www.ecdc.europa.eu/en/publications-data/covid-19-surveillance-guidance">www.ecdc.europa.eu/en/publications-data/covid-19-surveillance-guidance</ulink></bibtext> </blist> <blist> <bibtext> World Health Organization. WHO COVID‐19 Case Definition – Updated in Public Health Surveillance for COVID‐19. World Health Organization. Accessed September 2, 2022; 2022. https://<ulink href="http://www.who.int/publications-detail-redirect/WHO-2019-nCoVSurveillance%5fCase%5fDefinition-2022.1">www.who.int/publications-detail-redirect/WHO-2019-nCoVSurveillance%5fCase%5fDefinition-2022.1</ulink></bibtext> </blist> </ref> <aug> <p>By Niklas Bobrovitz; Kim Noël; Zihan Li; Christian Cao; Gabriel Deveaux; Anabel Selemon; David A. Clifton; Mercedes Yanes‐Lane; Tingting Yan and Rahul K. Arora</p> <p>Reported by Author; Author; Author; Author; Author; Author; Author; Author; Author; Author</p> </aug> <nolink nlid="nl1" bibid="bib10" firstref="ref9"></nolink> <nolink nlid="nl2" bibid="bib12" firstref="ref12"></nolink> <nolink nlid="nl3" bibid="bib13" firstref="ref13"></nolink> <nolink nlid="nl4" bibid="bib14" firstref="ref15"></nolink> <nolink nlid="nl5" bibid="bib16" firstref="ref16"></nolink> <nolink nlid="nl6" bibid="bib18" firstref="ref17"></nolink> <nolink nlid="nl7" bibid="bib19" firstref="ref19"></nolink> <nolink nlid="nl8" bibid="bib20" firstref="ref23"></nolink> <nolink nlid="nl9" bibid="bib21" firstref="ref34"></nolink> <nolink nlid="nl10" bibid="bib22" firstref="ref37"></nolink> <nolink nlid="nl11" bibid="bib23" firstref="ref38"></nolink> <nolink nlid="nl12" bibid="bib25" firstref="ref39"></nolink> <nolink nlid="nl13" bibid="bib27" firstref="ref40"></nolink> <nolink nlid="nl14" bibid="bib29" firstref="ref51"></nolink> <nolink nlid="nl15" bibid="bib31" firstref="ref52"></nolink> <nolink nlid="nl16" bibid="bib33" firstref="ref65"></nolink> <nolink nlid="nl17" bibid="bib35" firstref="ref66"></nolink> <nolink nlid="nl18" bibid="bib36" firstref="ref83"></nolink> <nolink nlid="nl19" bibid="bib37" firstref="ref88"></nolink> <nolink nlid="nl20" bibid="bib38" firstref="ref89"></nolink>
Header DbId: eric
DbLabel: ERIC
An: EJ1376524
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: SeroTacker-RoB: A Decision Rule-Based Algorithm for Reproducible Risk of Bias Assessment of Seroprevalence Studies
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Bobrovitz%2C+Niklas%22">Bobrovitz, Niklas</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0001-7883-4484">0000-0001-7883-4484</externalLink>)<br /><searchLink fieldCode="AR" term="%22Noël%2C+Kim%22">Noël, Kim</searchLink><br /><searchLink fieldCode="AR" term="%22Li%2C+Zihan%22">Li, Zihan</searchLink><br /><searchLink fieldCode="AR" term="%22Cao%2C+Christian%22">Cao, Christian</searchLink><br /><searchLink fieldCode="AR" term="%22Deveaux%2C+Gabriel%22">Deveaux, Gabriel</searchLink><br /><searchLink fieldCode="AR" term="%22Selemon%2C+Anabel%22">Selemon, Anabel</searchLink><br /><searchLink fieldCode="AR" term="%22Clifton%2C+David+A%2E%22">Clifton, David A.</searchLink><br /><searchLink fieldCode="AR" term="%22Yanes-Lane%2C+Mercedes%22">Yanes-Lane, Mercedes</searchLink><br /><searchLink fieldCode="AR" term="%22Yan%2C+Tingting%22">Yan, Tingting</searchLink><br /><searchLink fieldCode="AR" term="%22Arora%2C+Rahul+K%2E%22">Arora, Rahul K.</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Research+Synthesis+Methods%22"><i>Research Synthesis Methods</i></searchLink>. May 2023 14(3):414-426.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 13
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2023
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Information Analyses<br />Reports - Research
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Decision+Making%22">Decision Making</searchLink><br /><searchLink fieldCode="DE" term="%22Algorithms%22">Algorithms</searchLink><br /><searchLink fieldCode="DE" term="%22Risk%22">Risk</searchLink><br /><searchLink fieldCode="DE" term="%22Bias%22">Bias</searchLink><br /><searchLink fieldCode="DE" term="%22Check+Lists%22">Check Lists</searchLink><br /><searchLink fieldCode="DE" term="%22Efficiency%22">Efficiency</searchLink><br /><searchLink fieldCode="DE" term="%22Incidence%22">Incidence</searchLink><br /><searchLink fieldCode="DE" term="%22Evidence%22">Evidence</searchLink><br /><searchLink fieldCode="DE" term="%22Synthesis%22">Synthesis</searchLink><br /><searchLink fieldCode="DE" term="%22Diseases%22">Diseases</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1002/jrsm.1620
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 1759-2879<br />1759-2887
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Risk of bias (RoB) assessments are a core element of evidence synthesis but can be time consuming and subjective. We aimed to develop a decision rule-based algorithm for RoB assessment of seroprevalence studies. We developed the SeroTracker-RoB algorithm. The algorithm derives seven objective and two subjective critical appraisal items from the Joanna Briggs Institute Critical Appraisal Checklist for Prevalence studies and implements decision rules that determine study risk of bias based on the items. Decision rules were validated using the SeroTracker seroprevalence study database, which included non-algorithmic RoB judgments from two reviewers. We quantified efficiency as the mean difference in time for the algorithmic and non-algorithmic assessments of 80 randomly selected articles, coverage as the proportion of studies where the decision rules yielded an assessment, and reliability using intraclass correlations comparing algorithmic and non-algorithmic assessments for 2070 articles. A set of decision rules with 61 branches was developed using responses to the nine critical appraisal items. The algorithmic approach was faster than non-algorithmic assessment (mean reduction 2.32 min [SD 1.09] per article), classified 100% (n = 2070) of studies, and had good reliability compared to non-algorithmic assessment (ICC 0.77, 95% CI 0.74-0.80). We built the SeroTracker-RoB Excel Tool, which embeds this algorithm for use by other researchers. The SeroTracker-RoB decision-rule based algorithm was faster than non-algorithmic assessment with complete coverage and good reliability. This algorithm enabled rapid, transparent, and reproducible RoB evaluations of seroprevalence studies and may support evidence synthesis efforts during future disease outbreaks. This decision rule-based approach could be applied to other types of prevalence studies.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2023
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1376524
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1376524
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1002/jrsm.1620
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 13
        StartPage: 414
    Subjects:
      – SubjectFull: Decision Making
        Type: general
      – SubjectFull: Algorithms
        Type: general
      – SubjectFull: Risk
        Type: general
      – SubjectFull: Bias
        Type: general
      – SubjectFull: Check Lists
        Type: general
      – SubjectFull: Efficiency
        Type: general
      – SubjectFull: Incidence
        Type: general
      – SubjectFull: Evidence
        Type: general
      – SubjectFull: Synthesis
        Type: general
      – SubjectFull: Diseases
        Type: general
    Titles:
      – TitleFull: SeroTacker-RoB: A Decision Rule-Based Algorithm for Reproducible Risk of Bias Assessment of Seroprevalence Studies
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Bobrovitz, Niklas
      – PersonEntity:
          Name:
            NameFull: Noël, Kim
      – PersonEntity:
          Name:
            NameFull: Li, Zihan
      – PersonEntity:
          Name:
            NameFull: Cao, Christian
      – PersonEntity:
          Name:
            NameFull: Deveaux, Gabriel
      – PersonEntity:
          Name:
            NameFull: Selemon, Anabel
      – PersonEntity:
          Name:
            NameFull: Clifton, David A.
      – PersonEntity:
          Name:
            NameFull: Yanes-Lane, Mercedes
      – PersonEntity:
          Name:
            NameFull: Yan, Tingting
      – PersonEntity:
          Name:
            NameFull: Arora, Rahul K.
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 05
              Type: published
              Y: 2023
          Identifiers:
            – Type: issn-print
              Value: 1759-2879
            – Type: issn-electronic
              Value: 1759-2887
          Numbering:
            – Type: volume
              Value: 14
            – Type: issue
              Value: 3
          Titles:
            – TitleFull: Research Synthesis Methods
              Type: main
ResultId 1