View in EDS HTML Full Text PDF Full Text

Beyond Google Scholar, Scopus, and Web of Science: An Evaluation of the Backward and Forward Citation Coverage of 59 Databases' Citation Indices

Saved in:
Bibliographic Details
Title: Beyond Google Scholar, Scopus, and Web of Science: An Evaluation of the Backward and Forward Citation Coverage of 59 Databases' Citation Indices
Language: English
Authors: Michael Gusenbauer (ORCID 0000-0001-7768-2351)
Source: Research Synthesis Methods. 2024 15(5):802-817.
Availability: Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
Peer Reviewed: Y
Page Count: 16
Publication Date: 2024
Document Type: Journal Articles
Information Analyses
Reports - Research
Descriptors: Citation Indexes, Information Retrieval, Databases, Web Sites, Online Searching, Computer Software Evaluation, Users (Information), Access to Information, User Satisfaction (Information)
DOI: 10.1002/jrsm.1729
ISSN: 1759-2879
1759-2887
Abstract: Citation indices providing information on backward citation (BWC) and forward citation (FWC) links are essential for literature discovery, bibliographic analysis, and knowledge synthesis, especially when language barriers impede document identification. However, the suitability of citation indices varies. While some have been analyzed, the majority, whether new or established, lack comprehensive evaluation. Therefore, this study evaluates the citation coverage of the citation indices of 59 databases, encompassing the widely used Google Scholar, Scopus, and Web of Science alongside many others never previously analyzed, such as the emerging Lens, Scite, Dimensions, and OpenAlex or the subject-specific PubMed and JSTOR. Through a comprehensive analysis using 259 journal articles from across disciplines, this research aims to guide scholars in selecting indices with broader document coverage and more accurate and comprehensive backward and forward citation links. Key findings highlight Google Scholar, ResearchGate, Semantic Scholar, and Lens as leading options for FWC searching, with Lens providing superior download capabilities. For BWC searching, the Web of Science Core Collection can be recommended over Scopus for accuracy. BWC information from publisher databases such as IEEE Xplore or ScienceDirect was generally found to be the most accurate, yet only available for a limited number of articles. The findings will help scholars conducting systematic reviews, meta-analyses, and bibliometric analyses to select the most suitable databases for citation searching.
Abstractor: As Provided
Notes: https://osf.io/74brd
Entry Date: 2024
Accession Number: EJ1448648
Database: ERIC
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwEoKERQZwzPDlNw0DF2QSJiAAAA4zCB4AYJKoZIhvcNAQcGoIHSMIHPAgEAMIHJBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDFgvjYRwyhrCc-6w9AIBEICBmy6C6bHCqJqlsZD8fKNhE9XSP7-pBCQw16t326D2vh93c1NMnu6uMwUXePX84knf_lZxXOdw8Qp-rFdKF4YtywvhCTOUbeoSyBSpn7CLoz0STPixqCRZN2FjsmAqbZ-W05ZPil--sLYlNsnEikfJxbuS9K_bMyzJUiCg9sCUGbTdlepFx5Dk0evQasA0o4Xh9XRAAbQ6fU0nJe30
Text:
  Availability: 1
  Value: <anid>AN0180900140;[bdct]01sep.24;2024Nov18.04:21;v2.2.500</anid> <title id="AN0180900140-1">Beyond Google Scholar, Scopus, and Web of Science: An evaluation of the backward and forward citation coverage of 59 databases' citation indices </title> <p>Citation indices providing information on backward citation (BWC) and forward citation (FWC) links are essential for literature discovery, bibliographic analysis, and knowledge synthesis, especially when language barriers impede document identification. However, the suitability of citation indices varies. While some have been analyzed, the majority, whether new or established, lack comprehensive evaluation. Therefore, this study evaluates the citation coverage of the citation indices of 59 databases, encompassing the widely used Google Scholar, Scopus, and Web of Science alongside many others never previously analyzed, such as the emerging Lens, Scite, Dimensions, and OpenAlex or the subject‐specific PubMed and JSTOR. Through a comprehensive analysis using 259 journal articles from across disciplines, this research aims to guide scholars in selecting indices with broader document coverage and more accurate and comprehensive backward and forward citation links. Key findings highlight Google Scholar, ResearchGate, Semantic Scholar, and Lens as leading options for FWC searching, with Lens providing superior download capabilities. For BWC searching, the Web of Science Core Collection can be recommended over Scopus for accuracy. BWC information from publisher databases such as IEEE Xplore or ScienceDirect was generally found to be the most accurate, yet only available for a limited number of articles. The findings will help scholars conducting systematic reviews, meta‐analyses, and bibliometric analyses to select the most suitable databases for citation searching.</p> <p>Keywords: backward citation searching; citation coverage; citation index; forward citation searching; reference searching; snowballing</p> <hd id="AN0180900140-2">Highlights</hd> <p></p> <hd id="AN0180900140-3">What is already known?</hd> <p></p> <ulist> <item> The coverage of databases' citation indices determines the effectiveness of forward and backward citation searching—essential for knowledge discovery, knowledge synthesis, and bibliometric analyses.</item> </ulist> <hd id="AN0180900140-4">What is new?</hd> <p></p> <ulist> <item> This paper analyzes the coverage of the most relevant databases' citation indices in science, showing which indices are more complete or error‐free than others.</item> </ulist> <hd id="AN0180900140-5">Potential impact for Research Synthesis Methods readers</hd> <p></p> <ulist> <item> The findings can improve readers' search strategies by informing them where to search most effectively, whether at smaller specialized citation indices or larger multidisciplinary ones.</item> </ulist> <hd id="AN0180900140-6">INTRODUCTION</hd> <p>Searching citations is one of the essential methods to discover academic literature. Alongside keyword searches, backward and forward citation searches constitute an important and mandatory source for finding literature for knowledge synthesis.[[<reflink idref="bib1" id="ref1">1</reflink>], [<reflink idref="bib3" id="ref2">3</reflink>]] While keyword search identifies documents based on commonalities of language, citation‐based search identifies documents based on commonalities of citations. Such commonalities reflect authors' judgments that another study is relevant to their own. As documents identified through citation searches use diverse language, citation searches produce unique results that would not be identified via alternative methods. Citation searches are particularly likely to yield unique results in the case of hard‐to‐identify documents or in fields with non‐standardized language.[[<reflink idref="bib4" id="ref3">4</reflink>]] In some cases, they may even be better suited than keyword searches for identifying the bulk of relevant literature.[[<reflink idref="bib6" id="ref4">6</reflink>], [<reflink idref="bib8" id="ref5">8</reflink>]] Irrespective of whether authors use citation searches as a principal or supplementary search method, such searches have been found to improve search comprehensiveness[<reflink idref="bib9" id="ref6">9</reflink>] and increase retrieval potential[<reflink idref="bib10" id="ref7">10</reflink>] and may be regarded as keyword searches' complementary "safety net."[<reflink idref="bib4" id="ref8">4</reflink>]<sups>(p.8)</sups></p> <p>Due to citation searches' power to identify literature, they are applied to researchers' day‐to‐day searches of academic literature, in bibliometric citation network analyses, and in systematic searches for systematic literature reviews and meta‐analyses. The Cochrane Handbook—"widely regarded as a benchmark"[<reflink idref="bib11" id="ref9">11</reflink>]<sups>(p.25)</sups> in systematic review methodology across disciplines—deems backward citation (BWC) searching a mandatory requirement for systematic reviews[<reflink idref="bib3" id="ref10">3</reflink>] and forward citation (FWC) searching a suggested method.[<reflink idref="bib2" id="ref11">2</reflink>] Indeed, 81% of Cochrane Systematic Reviews already use BWC searching, while 12% of those in medicine use FWC searching.[<reflink idref="bib12" id="ref12">12</reflink>]</p> <p>Citation searching is particularly relevant in fields that use diverse, non‐standardized language that might be difficult to capture with language‐based keyword searching alone. In other words, citation searching is important when "core concepts are difficult to capture using keywords, e.g., where core concepts are described inconsistently due to systemic reporting deficiencies, or due to historical development of terminology."[[<reflink idref="bib6" id="ref13">6</reflink>], [<reflink idref="bib13" id="ref14">13</reflink>]]<sups>(p.170),</sups>[[<reflink idref="bib14" id="ref15">14</reflink>], [<reflink idref="bib16" id="ref16">16</reflink>], [<reflink idref="bib18" id="ref17">18</reflink>]] Thus, sciences involving abstract conceptualizations of phenomena, such as psychology, management and economics, social sciences, and humanities, particularly benefit from discovering relevant articles beyond familiar terminology. It is mainly these fields in which knowledge synthesis has also undergone a considerable uplift[<reflink idref="bib19" id="ref18">19</reflink>] and thus may benefit from citation searching.</p> <p>In order to effectively employ forward or backward citation search techniques, particularly in systematic reviews, it is essential to know what coverage different citation indices offer.[<reflink idref="bib14" id="ref19">14</reflink>] To inform (re)searchers, previous studies have focused on analyzing only a handful of citation indices, such as those of Google Scholar, Web of Science Core Collection, and Scopus.[<reflink idref="bib14" id="ref20">14</reflink>] Some studies have compared two of these (i.e., Web of Science Core Collection and Google Scholar,[<reflink idref="bib20" id="ref21">20</reflink>] Scopus and Google Scholar,[<reflink idref="bib21" id="ref22">21</reflink>] Web of Science Core Collection and Scopus[<reflink idref="bib22" id="ref23">22</reflink>])—or all three.[[<reflink idref="bib23" id="ref24">23</reflink>]] More recent comparisons have found that Scopus seems somewhat superior to the Web of Science Core Collection[<reflink idref="bib22" id="ref25">22</reflink>] regarding FWC coverage. Others have found Web of Science Core Collection to be preferable to Google Scholar in terms of administrative and screening costs,[[<reflink idref="bib14" id="ref26">14</reflink>], [<reflink idref="bib25" id="ref27">25</reflink>]] yet Google Scholar superior in terms of recall.[[<reflink idref="bib20" id="ref28">20</reflink>]]</p> <p>As many bibliographic studies and literature reviews search just one citation index,[<reflink idref="bib22" id="ref29">22</reflink>] the choice of citation index matters greatly for the quality of the subsequent analysis—particularly in view of such indices' variable coverage (see, e.g., Whear et al.,[<reflink idref="bib26" id="ref30">26</reflink>]) and quality. For example, Web of Science Core Collection was recently found to be the most popular index for FWC searching among a sample of Cochrane reviews[<reflink idref="bib13" id="ref31">13</reflink>] and medical systematic reviews.[<reflink idref="bib25" id="ref32">25</reflink>] Others rely on Google Scholar, the most comprehensive citation index available.[<reflink idref="bib27" id="ref33">27</reflink>] However, as Google Scholar's searches are regularly found to be inefficient and difficult to retrieve,[[<reflink idref="bib7" id="ref34">7</reflink>], [<reflink idref="bib20" id="ref35">20</reflink>]] it is worth evaluating alternatives.</p> <p>While analyses of the three main citation indices are valuable, we have little information on the dozens of other indices that have never been analyzed. We currently know little about how the dominant three compare against newly launched citation indices (e.g., OpenAlex, Semantic Scholar, Dimensions, Lens), how they compare against indices suspected of being less comprehensive (i.e., subject‐specific databases such as PubMed and JSTOR), and how well they cover the reference lists of full texts that are relied upon in BWC searching. Particularly, newer citation indices such as Lens, which offers free access to citation information, have become valuable resources for collecting citation information alongside citation tools.[<reflink idref="bib28" id="ref36">28</reflink>] Furthermore, some citation indices, such as OpenAlex—also listed by the Cochrane Handbook's technical supplement to search[<reflink idref="bib2" id="ref37">2</reflink>]—have not been quantitatively analyzed before. A recent Delphi study[<reflink idref="bib29" id="ref38">29</reflink>] has highlighted the need for further research in understanding citation indices. Addressing this research gap, the present study seeks to answer: <emph>What is the comparative coverage, comprehensiveness, and accuracy of backward and forward citation indices at major scholarly literature databases?</emph></p> <p>The term "database" is used liberally in this study to describe any academic information service that makes scholarly documents available via a search interface. This may include platform providers such as Web of Science, search engines such as Google Scholar, publisher databases such as SpringerLink, or bibliographic databases such as PubMed. This study focuses on databases that provide citation information through a "citation index": a part of the database or an external database that supplies information on the citation linkages between documents covered on the database.</p> <p>This study focuses on both forward and backward citation searching and uses "citation searching" as the umbrella term for many different labels.[<reflink idref="bib29" id="ref39">29</reflink>] Researchers have used many names to describe these citation‐searching methods: pearl growing, snowballing, reference harvesting, ancestry approach, citation chaining, citation or footnote chasing, citation mining, and backward or forward chaining. All these terms describe the same methods, which use associations via citations to establish some sort of forward‐ or backward‐oriented relevance.</p> <hd id="AN0180900140-7">METHODS</hd> <p>This study followed three steps to derive concrete recommendations on which citation indexes are most powerful. First, I selected many relevant databases' citation indices, including the most popular and relevant indices currently available in academia. Second, I selected a list of documents that was representative across all fields of science and that was covered across the citation indices. Third, I analyzed these documents' forward and backward citation coverage across citation indices. The considerations of these steps are detailed below.</p> <hd id="AN0180900140-8">Database selection</hd> <p>The selection of databases that support forward or backward citation searching was based on www.searchsmart.org, a website comparing more than 100 of the most relevant academic databases. I selected all its databases that (<reflink idref="bib1" id="ref40">1</reflink>) support either BWC or FWC information and (<reflink idref="bib2" id="ref41">2</reflink>) covered journal articles. The focus on journal articles was ideal as the subsequent analysis needed to be based on a common document type that was most universally available across databases and that most researchers find relevant. A consequence of this focus was that dedicated theses, pre‐print, clinical trials, or conference proceedings databases and their citation indices could not be analyzed.</p> <p>A database was considered to support BWC or FWC information when it provided at least a list of references that linked to the individual forward or backward citation records. It was not enough for databases to provide the citations in unlinked HTML or text format; in order to be included, they had to link to the actual reference. Databases such as that of Naver Academic, which provide only the <emph>count</emph> of citations with no references to the individual citations, were excluded. Further, the search interfaces of BASE and Ingenta Connect had frequent issues with displaying BWC/FWC results; I included their references if they would load and show individual references, even if they were displayed as simple DOIs. For both BWC and FWC capabilities to be included for analysis, it did not matter whether the citation data was provided natively or the database used some external provider's index (e.g., Semantic Scholar or Dimensions) and presented it through its own interface. In total, I identified 59 databases' citation indices that were selected for subsequent analysis (see Table 1). While the resulting list covers most major indices, it is noteworthy that it does not include all indices (in particular, no non‐English focused databases).</p> <p>1 TABLE Databases' forward citation coverage (ranked by highest FWC score).</p> <p> <ephtml> <table><thead valign="bottom"><tr><th align="left">Database name</th><th align="left">Open/closed</th><th align="left">Subject coverage (via ASJC)</th><th align="left">Database size</th><th align="left">(a): % of articles covered</th><th align="left">(b): % of articles with FWC info</th><th align="left">(c): median deviation of max FWC</th><th align="left">FWC score: b × (1−c)</th></tr></thead><tbody valign="top"><tr><td align="left">Google Scholar</td><td align="left">Open</td><td align="left">Multidisciplinary</td><td align="left">Large</td><td align="char" char=".">100.0%</td><td align="char" char=".">100.0%</td><td align="char" char=".">0.0%</td><td align="left">100.0%</td></tr><tr><td align="left">ResearchGate</td><td align="left">Open</td><td align="left">Multidisciplinary</td><td align="left">Large</td><td align="char" char=".">95.8%</td><td align="char" char=".">100.0%</td><td align="char" char=".">−17.2%</td><td align="left">82.8%</td></tr><tr><td align="left">ScienceDirect</td><td align="left">Open</td><td align="left">Multidisciplinary</td><td align="left">Medium</td><td align="char" char=".">26.6%</td><td align="char" char=".">100.0%</td><td align="char" char=".">−22.5%</td><td align="left">77.5%</td></tr><tr><td align="left">Semantic Scholar</td><td align="left">Open</td><td align="left">Multidisciplinary</td><td align="left">Large</td><td align="char" char=".">98.5%</td><td align="char" char=".">100.0%</td><td align="char" char=".">−27.3%</td><td align="left">72.7%</td></tr><tr><td align="left">Lens</td><td align="left">Open</td><td align="left">Multidisciplinary</td><td align="left">Large</td><td align="char" char=".">96.5%</td><td align="char" char=".">100.0%</td><td align="char" char=".">−28.9%</td><td align="left">71.1%</td></tr><tr><td align="left">IEEE Xplore</td><td align="left">Open</td><td align="left">Engineering</td><td align="left">Small</td><td align="char" char=".">3.9%</td><td align="char" char=".">100.0%</td><td align="char" char=".">−29.7%</td><td align="left">70.3%</td></tr><tr><td align="left">Scite</td><td align="left">Open</td><td align="left">Multidisciplinary</td><td align="left">Large</td><td align="char" char=".">97.7%</td><td align="char" char=".">97.6%</td><td align="char" char=".">−28.1%</td><td align="left">70.2%</td></tr><tr><td align="left">Dimensions</td><td align="left">Open</td><td align="left">Multidisciplinary</td><td align="left">Large</td><td align="char" char=".">97.7%</td><td align="char" char=".">100.0%</td><td align="char" char=".">−30.4%</td><td align="left">69.6%</td></tr><tr><td align="left">OpenAlex</td><td align="left">Open</td><td align="left">Multidisciplinary</td><td align="left">Large</td><td align="char" char=".">100.0%</td><td align="char" char=".">99.6%</td><td align="char" char=".">−31.4%</td><td align="left">68.3%</td></tr><tr><td align="left">Scinapse</td><td align="left">Open</td><td align="left">Multidisciplinary</td><td align="left">Large</td><td align="char" char=".">60.2%</td><td align="char" char=".">99.4%</td><td align="char" char=".">−32.3%</td><td align="left">67.2%</td></tr><tr><td align="left">Scopus</td><td align="left">Closed</td><td align="left">Multidisciplinary</td><td align="left">Medium</td><td align="char" char=".">98.1%</td><td align="char" char=".">100.0%</td><td align="char" char=".">−33.7%</td><td align="left">66.3%</td></tr><tr><td align="left">Science Citation Index Expanded (via Web of Science)</td><td align="left">Closed</td><td align="left">Multidisciplinary</td><td align="left">Medium</td><td align="char" char=".">64.9%</td><td align="char" char=".">100.0%</td><td align="char" char=".">−37.1%</td><td align="left">62.9%</td></tr><tr><td align="left">Scilit</td><td align="left">Open</td><td align="left">Multidisciplinary</td><td align="left">Large</td><td align="char" char=".">99.6%</td><td align="char" char=".">99.6%</td><td align="char" char=".">−36.9%</td><td align="left">62.9%</td></tr><tr><td align="left">Medline (via Web of Science)</td><td align="left">Closed</td><td align="left">Medicine</td><td align="left">Medium</td><td align="char" char=".">43.2%</td><td align="char" char=".">100.0%</td><td align="char" char=".">−40.5%</td><td align="left">59.5%</td></tr><tr><td align="left">SAGE Journals Online</td><td align="left">Open</td><td align="left">Medicine</td><td align="left">Small</td><td align="char" char=".">5.4%</td><td align="char" char=".">100.0%</td><td align="char" char=".">−40.5%</td><td align="left">59.5%</td></tr><tr><td align="left">Web of Science Core Collection</td><td align="left">Closed</td><td align="left">Multidisciplinary</td><td align="left">Medium</td><td align="char" char=".">86.5%</td><td align="char" char=".">100.0%</td><td align="char" char=".">−42.3%</td><td align="left">57.7%</td></tr><tr><td align="left">Wiley Online Library</td><td align="left">Open</td><td align="left">Multidisciplinary</td><td align="left">Medium</td><td align="char" char=".">11.2%</td><td align="char" char=".">100.0%</td><td align="char" char=".">−42.6%</td><td align="left">57.4%</td></tr><tr><td align="left">SpringerLink</td><td align="left">Open</td><td align="left">Multidisciplinary</td><td align="left">Medium</td><td align="char" char=".">10.4%</td><td align="char" char=".">96.3%</td><td align="char" char=".">−44.8%</td><td align="left">53.1%</td></tr><tr><td align="left">CAS SciFinder‐n</td><td align="left">Closed</td><td align="left">Multidisciplinary</td><td align="left">Medium</td><td align="char" char=".">49.4%</td><td align="char" char=".">99.2%</td><td align="char" char=".">−54.2%</td><td align="left">45.4%</td></tr><tr><td align="left">Social Sciences Citation Index (via Web of Science)</td><td align="left">Closed</td><td align="left">Social Sciences</td><td align="left">Medium</td><td align="char" char=".">32.0%</td><td align="char" char=".">100.0%</td><td align="char" char=".">−54.7%</td><td align="left">45.3%</td></tr><tr><td align="left">Arts & Humanities Citation Index (via Web of Science)</td><td align="left">Closed</td><td align="left">Arts and Humanities</td><td align="left">Small</td><td align="char" char=".">3.9%</td><td align="char" char=".">100.0%</td><td align="char" char=".">−60.0%</td><td align="left">40.0%</td></tr><tr><td align="left">Bielefeld Academic Search Engine</td><td align="left">Open</td><td align="left">Multidisciplinary</td><td align="left">Large</td><td align="char" char=".">93.1%</td><td align="char" char=".">63.5%</td><td align="char" char=".">−43.2%</td><td align="left">36.1%</td></tr><tr><td align="left">Conference Proceedings Citation Index—Science (via Web of Science)</td><td align="left">Closed</td><td align="left">Multidisciplinary</td><td align="left">Medium</td><td align="char" char=".">4.6%</td><td align="char" char=".">91.7%</td><td align="char" char=".">−61.1%</td><td align="left">35.6%</td></tr><tr><td align="left">Conference Proceedings Citation Index—Social Science & Humanities (via Web of Science)</td><td align="left">Closed</td><td align="left">Social Sciences</td><td align="left">Small</td><td align="char" char=".">4.2%</td><td align="char" char=".">100.0%</td><td align="char" char=".">−65.4%</td><td align="left">34.6%</td></tr><tr><td align="left">BIOSIS Citation Index (via Web of Science)</td><td align="left">Closed</td><td align="left">Medicine</td><td align="left">Medium</td><td align="char" char=".">21.2%</td><td align="char" char=".">96.4%</td><td align="char" char=".">−65.5%</td><td align="left">33.2%</td></tr><tr><td align="left">Emerging Sources Citation Index (via Web of Science)</td><td align="left">Closed</td><td align="left">Multidisciplinary</td><td align="left">Small</td><td align="char" char=".">8.5%</td><td align="char" char=".">100.0%</td><td align="char" char=".">−69.8%</td><td align="left">30.2%</td></tr><tr><td align="left">Europe PMC</td><td align="left">Open</td><td align="left">Medicine</td><td align="left">Medium</td><td align="char" char=".">44.4%</td><td align="char" char=".">98.3%</td><td align="char" char=".">−71.1%</td><td align="left">28.4%</td></tr><tr><td align="left">Taylor and Francis Online</td><td align="left">Open</td><td align="left">Multidisciplinary</td><td align="left">SLmall</td><td align="char" char=".">7.7%</td><td align="char" char=".">80.0%</td><td align="char" char=".">−65.2%</td><td align="left">27.9%</td></tr><tr><td align="left">PubMed</td><td align="left">Open</td><td align="left">Medicine</td><td align="left">Medium</td><td align="char" char=".">42.9%</td><td align="char" char=".">99.1%</td><td align="char" char=".">−73.7%</td><td align="left">26.1%</td></tr><tr><td align="left">ACM Guide to Computing Literature</td><td align="left">Open</td><td align="left">Computer Science</td><td align="left">Small</td><td align="char" char=".">3.9%</td><td align="char" char=".">100.0%</td><td align="char" char=".">−74.6%</td><td align="left">25.4%</td></tr><tr><td align="left">Sociological Abstracts (via ProQuest)</td><td align="left">Closed</td><td align="left">Social Sciences</td><td align="left">Small</td><td align="char" char=".">6.9%</td><td align="char" char=".">88.9%</td><td align="char" char=".">−76.3%</td><td align="left">21.1%</td></tr><tr><td align="left">ScienceOpen</td><td align="left">Open</td><td align="left">Multidisciplinary</td><td align="left">Medium</td><td align="char" char=".">96.1%</td><td align="char" char=".">97.6%</td><td align="char" char=".">−84.3%</td><td align="left">15.3%</td></tr><tr><td align="left">zbMATH Open</td><td align="left">Open</td><td align="left">Mathematics</td><td align="left">Small</td><td align="char" char=".">3.9%</td><td align="char" char=".">90.0%</td><td align="char" char=".">−84.3%</td><td align="left">14.1%</td></tr><tr><td align="left">Social Science Premium Collection (via ProQuest)</td><td align="left">Closed</td><td align="left">Social Sciences</td><td align="left">Medium</td><td align="char" char=".">20.5%</td><td align="char" char=".">92.5%</td><td align="char" char=".">−85.7%</td><td align="left">13.2%</td></tr><tr><td align="left">Public Health Database (via ProQuest)</td><td align="left">Closed</td><td align="left">Medicine</td><td align="left">Small</td><td align="char" char=".">9.7%</td><td align="char" char=".">72.0%</td><td align="char" char=".">−83.4%</td><td align="left">11.9%</td></tr><tr><td align="left">APA PsycInfo (via EBSCOhost)</td><td align="left">Closed</td><td align="left">Psychology</td><td align="left">Small</td><td align="char" char=".">20.1%</td><td align="char" char=".">84.6%</td><td align="char" char=".">−86.0%</td><td align="left">11.8%</td></tr><tr><td align="left">International Bibliography of the Social Sciences (via ProQuest)</td><td align="left">Closed</td><td align="left">Social Sciences</td><td align="left">Small</td><td align="char" char=".">9.7%</td><td align="char" char=".">92.0%</td><td align="char" char=".">−88.9%</td><td align="left">10.2%</td></tr><tr><td align="left">RePEc (via IDEAS)</td><td align="left">Open</td><td align="left">Economics, Econometrics, and Finance</td><td align="left">Small</td><td align="char" char=".">12.0%</td><td align="char" char=".">90.3%</td><td align="char" char=".">−89.7%</td><td align="left">9.3%</td></tr><tr><td align="left">GeoRef (via ProQuest)</td><td align="left">Closed</td><td align="left">Earth and Planetary Sciences</td><td align="left">Small</td><td align="char" char=".">3.9%</td><td align="char" char=".">60.0%</td><td align="char" char=".">−84.7%</td><td align="left">9.2%</td></tr><tr><td align="left">RePEc (via EconPapers)</td><td align="left">Open</td><td align="left">Economics, Econometrics, and Finance</td><td align="left">Small</td><td align="char" char=".">12.7%</td><td align="char" char=".">90.9%</td><td align="char" char=".">−89.9%</td><td align="left">9.2%</td></tr><tr><td align="left">SocINDEX (via EBSCOhost)</td><td align="left">Closed</td><td align="left">Social Sciences</td><td align="left">Small</td><td align="char" char=".">5.8%</td><td align="char" char=".">100.0%</td><td align="char" char=".">−91.2%</td><td align="left">8.8%</td></tr><tr><td align="left">Nursing & Allied Health Database (via ProQuest)</td><td align="left">Closed</td><td align="left">Medicine</td><td align="left">Small</td><td align="char" char=".">11.6%</td><td align="char" char=".">76.7%</td><td align="char" char=".">−89.2%</td><td align="left">8.3%</td></tr><tr><td align="left">Environmental Science Database (via ProQuest)</td><td align="left">Closed</td><td align="left">Multidisciplinary</td><td align="left">Small</td><td align="char" char=".">6.6%</td><td align="char" char=".">76.5%</td><td align="char" char=".">−89.2%</td><td align="left">8.2%</td></tr><tr><td align="left">Business Source Premier (via EBSCOhost)</td><td align="left">Closed</td><td align="left">Multidisciplinary</td><td align="left">Medium</td><td align="char" char=".">12.0%</td><td align="char" char=".">90.3%</td><td align="char" char=".">−92.6%</td><td align="left">6.7%</td></tr><tr><td align="left">GreenFILE (via EBSCOhost)</td><td align="left">Closed</td><td align="left">Environmental Science</td><td align="left">Small</td><td align="char" char=".">4.6%</td><td align="char" char=".">83.3%</td><td align="char" char=".">−93.3%</td><td align="left">5.6%</td></tr><tr><td align="left">SciTech Premium Collection (via ProQuest)</td><td align="left">Closed</td><td align="left">Multidisciplinary</td><td align="left">Large</td><td align="char" char=".">68.3%</td><td align="char" char=".">85.3%</td><td align="char" char=".">−94.8%</td><td align="left">4.4%</td></tr><tr><td align="left">Biological Science Database (via ProQuest)</td><td align="left">Closed</td><td align="left">Biochemistry, Genetics, and Molecular Biology</td><td align="left">Small</td><td align="char" char=".">11.2%</td><td align="char" char=".">79.3%</td><td align="char" char=".">−94.9%</td><td align="left">4.0%</td></tr><tr><td align="left">ABI/Inform Global (via ProQuest)</td><td align="left">Closed</td><td align="left">Business, Management, and Accounting</td><td align="left">Medium</td><td align="char" char=".">12.7%</td><td align="char" char=".">69.7%</td><td align="char" char=".">−94.8%</td><td align="left">3.7%</td></tr><tr><td align="left">Medline (via Ovid)</td><td align="left">Closed</td><td align="left">Medicine</td><td align="left">Medium</td><td align="char" char=".">40.9%</td><td align="char" char=".">80.2%</td><td align="char" char=".">−95.4%</td><td align="left">3.7%</td></tr><tr><td align="left">JSTOR</td><td align="left">Open</td><td align="left">Social Sciences</td><td align="left">Medium</td><td align="char" char=".">6.9%</td><td align="char" char=".">83.3%</td><td align="char" char=".">−97.0%</td><td align="left">2.5%</td></tr><tr><td align="left">Earth, Atmospheric & Aquatic Science Database (via ProQuest)</td><td align="left">Closed</td><td align="left">Multidisciplinary</td><td align="left">Small</td><td align="char" char=".">4.6%</td><td align="char" char=".">58.3%</td><td align="char" char=".">−96.1%</td><td align="left">2.3%</td></tr><tr><td align="left">Ingenta Connect</td><td align="left">Open</td><td align="left">Multidisciplinary</td><td align="left">Small</td><td align="char" char=".">6.9%</td><td align="char" char=".">5.6%</td><td align="char" char=".">−82.9%</td><td align="left">1.0%</td></tr><tr><td align="left">APA PsycInfo (via Ovid)</td><td align="left">Closed</td><td align="left">Psychology</td><td align="left">Small</td><td align="char" char=".">19.7%</td><td align="char" char=".">51.0%</td><td align="char" char=".">−98.7%</td><td align="left">0.7%</td></tr><tr><td align="left">Academic Search Premier (via EBSCOhost)</td><td align="left">Closed</td><td align="left">Multidisciplinary</td><td align="left">Medium</td><td align="char" char=".">47.9%</td><td align="char" char=".">43.5%</td><td align="char" char=".">−98.7%</td><td align="left">0.6%</td></tr><tr><td align="left">EconStor</td><td align="left">Open</td><td align="left">Economics, Econometrics, and Finance</td><td align="left">Small</td><td align="char" char=".">3.5%</td><td align="char" char=".">22.2%</td><td align="char" char=".">−97.6%</td><td align="left">0.5%</td></tr><tr><td align="left">Epistemonikos</td><td align="left">Open</td><td align="left">Medicine</td><td align="left">Small</td><td align="char" char=".">4.6%</td><td align="char" char=".">41.7%</td><td align="char" char=".">−99.6%</td><td align="left">0.2%</td></tr><tr><td align="left">Public Library of Science (PLOS)</td><td align="left">Open</td><td align="left">Biochemistry, Genetics, and Molecular Biology</td><td align="left">Small</td><td align="char" char=".">3.5%</td><td align="char" char=".">0.0%</td><td align="char" char=".">‐</td><td align="left">N/a</td></tr><tr><td align="left">Academic Search Elite (via EBSCOhost)</td><td align="left">Closed</td><td align="left">Multidisciplinary</td><td align="left">Medium</td><td align="char" char=".">47.9%</td><td align="char" char=".">0.0%</td><td align="char" char=".">‐</td><td align="left">N/a</td></tr><tr><td align="left">Naver Academic</td><td align="left">Open</td><td align="left">Multidisciplinary</td><td align="left">Large</td><td align="char" char=".">84.9%</td><td align="char" char=".">0.0%</td><td align="char" char=".">‐</td><td align="left">N/a</td></tr></tbody></table> </ephtml> </p> <p>1 <emph>Note</emph>: Bold indicates the most important information.</p> <p>2 Abbreviation: FWC, forward citation.</p> <hd id="AN0180900140-9">Article selection</hd> <p>I used four criteria to select relevant documents that I could use to compare the coverage and comprehensiveness of BWC and FWC indices. Relevant documents had to (<reflink idref="bib1" id="ref42">1</reflink>) be journal articles; (<reflink idref="bib2" id="ref43">2</reflink>) have a DOI for identification; (<reflink idref="bib3" id="ref44">3</reflink>) have a full text available, allowing manual reference‐list counts; and (<reflink idref="bib4" id="ref45">4</reflink>) have a title with no symbols or special characters. This last criterion was necessary as articles with special characters are frequently missed by keyword searches[<reflink idref="bib30" id="ref46">30</reflink>]—a method this analysis needed to rely on in case a database did not support DOI identification. I validated the identifiability of these articles through manual keyword searches. Further, I selected articles from all 26 All Science Journal Classification (ASJC) disciplines published in the same year (2018) to establish a common denominator. I selected articles with many FWCs in order to obtain greater variance in citation rates for the subsequent analysis (i.e., zero citations provide little information on citation coverage). I took great care to select only single articles from the same journal to prevent journals being over‐represented. I applied these criteria to Scopus searches (for search strings, see Supplementary Material 1) and ranked articles according to their citation counts. Starting with the most cited, I screened each article for title‐fit and included six references from different journals. Thus, rather than including the same journal twice, I instead selected a less‐cited article from a different journal. To obtain a stratified sample across disciplines, I collected six articles per ASJC category, totaling 156 (see data in online repository).</p> <p>After I had analyzed the citation coverage of all databases' indices, I found that some smaller databases, often from specific publishers, covered only a few or even no articles out of the set of 156. Thus, to make sure that at least nine articles were covered by each database, I searched these specific databases for articles to include in the overall analysis. In this second, database‐specific sampling round, not all initial sampling criteria could be fulfilled. While all articles had names without symbols and special characters, included DOIs, and had an accessible full text, stratification across ASJC categories was not feasible. Few of the databases would have this classification system in place, and including one article per each of the 26 categories would have excessively expanded the number of articles included. Thus, I used "absurd" queries[<reflink idref="bib27" id="ref47">27</reflink>] to obtain a list of articles irrespective of discipline yet ranked according to their FWC count. As the citation indices of these databases were often considerably more limited, I included articles from earlier years to obtain articles with more citations. In cases where citation rankings were not available, I clicked through the results set page by page and selected articles for which citation data was available and that had relatively higher citation rates than other articles in the results set. Through this process, 103 additional articles were added, increasing the selected articles to 259. All of these are from different journals except for multiple results from four journals added in the second round: <emph>PLOS One</emph>, <emph>BioScience</emph>, <emph>Info Technology & People</emph>, and <emph>PLOS Medicine</emph>.</p> <hd id="AN0180900140-10">Analysis of databases' forward and backward citation coverage</hd> <p>To collect all the BWC and FWC information, 15,281 queries identified the 259 articles across 59 databases. Most of the queries had to be performed manually and one by one, as only a few databases allowed bulk downloading of citation information and most had different data structures that made automation unfeasible. To minimize biases introduced by analyzing citation information at different time points, I narrowed the time span of data collection: all data was collected between October 9 and November 30, 2023. The exception was OpenAlex, whose new search interface was only launched on January 12, 2024, so it was analyzed later; as a consequence, OpenAlex's FWC score will be slightly distorted upwards. In the analysis, I first checked whether an article was available in the database by searching for either the articles' DOIs or their titles. In analyzing citation information, I took great care to only compare the exact published journal version of the articles, excluding versions such as preprints, comments, errata, or rejoinders to the articles that might bear the same name.</p> <p>When an article was available on the database, I extracted the number of FWCs and BWCs that were available. In some cases, it was unclear whether zero FWCs for a specific article indicated that the article genuinely had zero citations or simply that no information was available. In order to be consistent and conservative in my calculations, I interpreted zero FWCs as a missing value. As a result, I only compared articles with at least one citation available, likely distorting the citation scores of inferior indices upward somewhat. For BWCs, this procedure was different. As there was only one article among the 259 with zero BWCs, I interpreted zero BWCs as an actual value. The analyses in this study are replicable and may be expanded or updated (see online repository for the list of 259 articles and detailed results).</p> <hd id="AN0180900140-11">Determination of the quality of backward and forward citation indices</hd> <p>I used scores to determine the quality of the BWC and FWC indices. First, I collected data on the availability of citation information in the documents covered by a database. Some databases did not have citation information for all documents, so this was crucial information. Second, I determined the coverage of the citation index compared to some optimal values. For FWCs, the optimal value was the maximum FWC count available at one of the 59 databases that served as the benchmark. The FWC score was calculated with <emph>b ×</emph> (1−<emph>c</emph>), where <emph>b</emph> was the share of articles with FWC info and <emph>c</emph> was the indices' median deviation of the maximum FWC benchmark value. I chose the median here to ensure the score was not dependent on outliers of extraordinarily high or low FWCs that occurred for some articles.</p> <p>For BWCs, the optimal value was the actual number of references from the full text. I collected the reference quantity by manually checking the full text of the 259 documents. To validate the accuracy of the scores, all counts were rechecked and compared to databases that were found to have (close to) 100% reliable BWC coverage. For BWCs, I denoted the mean percentage error in addition to the mean <emph>absolute</emph> percentage error to also capture the direction of the error, that is, whether indices systematically over‐ or underreported the number of BWCs. If BWC indices covered exactly the number of full‐text references, their coverage was considered accurate, that is, with zero errors. Formally, the calculation of the BWC score was <emph>d ×</emph> (<emph>1</emph>−<emph>f</emph>), where <emph>d</emph> was the share of articles with BWC information and <emph>f</emph> was the mean absolute percentage error. I chose the mean here as there were few outliers and most BWC values were correct, often producing median deviations of zero. Thus, the mean is more sensitive to the magnitude of existing deviations. The FWC and BWC scores account for the completeness, coverage, and accuracy of citation information, yet they do not account for the number of articles covered by a database. This way, smaller databases that cover just a few articles were not disadvantaged compared to larger ones that cover many.</p> <hd id="AN0180900140-12">RESULTS</hd> <p>This section details the results derived from examining 59 databases' citation indices. Alongside these findings, we include further information on database characteristics for further context in Table 1. For each database or index, this table indicates accessibility (open access or closed), subject coverage (multidisciplinary or focused on a primary subject as defined by www.searchsmart.org), and size, categorized as small (fewer than 10 million documents), medium (up to 100 million documents), or large (over 100 million documents).</p> <p>Results show that the 259 articles used to compute the BWC and FWC scores were unevenly accessible across databases. In general, larger databases also covered more of the articles—a relation visible in a Pearson correlation of <emph>r</emph> = 0.84. Google Scholar and OpenAlex were the only databases to cover all the articles, while PLOS and Econstor had the least coverage, identifying just nine.</p> <p>Half of the databases provided FWC information for all or almost all (96%) of the articles they covered, of which two thirds provided it for all articles. The other half provided only fragmented information, with Ingenta Connect providing the least—just one article (6%) had FWC information. I found a notable correlation between database size and FWC coverage (<emph>r</emph> = 0.63), meaning larger databases have more complete FWC indices. There are multiple reasons why this correlation is lower than for article coverage (<emph>r</emph> = 0.84): first, issues in data provision and linking[<reflink idref="bib22" id="ref48">22</reflink>] mean that some covered articles do not appear in BWCs or FWCs. For example, ScienceOpen, SciTech Premium Collection (via ProQuest), and Academic Search Premier (via EBSCOhost) have comparatively large databases, yet the citation information they provide is weak. Another reason is that certain providers artificially limit coverage; for example, Ebsco offers different citation coverage across products. Moreover, smaller databases have broad FWC or BWC coverage due to the use of external data providers that extend their citation indices far beyond their databases' article coverage (e.g., IEEEXplore, PLOS, ScienceDirect, Sage). All these findings mean that database size alone is not a perfect indicator for citation coverage.</p> <p>The number of BWCs across all science fields in 2018 was a median of 54.5 and a mean of 93.9. No database provided BWC information for all articles; the closest to full coverage were Scopus, OpenAlex, Lens, and ScienceOpen. Among the articles that were covered, about 36% of all databases provided complete BWC information, while another 17% provided BWC information for more than 90% of articles. The remaining databases provided more fragmented information, with Academic Search Premier (via EBSCOhost) providing the least—just 7% of its articles had BWC information. The analysis shows that both large, multidisciplinary databases and smaller publisher databases can provide comprehensive and accurate BWC information. Along these lines, database size was found to be weakly correlated with the accuracy of the BWC information (<emph>r</emph> = 0.16). A likely reason may be that smaller publisher databases, in particular, had comparatively good BWC information. In comparison, larger databases were not always accurate, often overstating the number of BWCs an article had.</p> <hd id="AN0180900140-13">Verdict: The best options for forward citation searching</hd> <p>The analysis confirms that Google Scholar has the broadest FWC coverage, both in terms of how many articles are equipped with FWC information and the number of FWCs linked. However, the analysis also revealed that Google Scholar's FWC information was not the most comprehensive for all articles: while it did provide superior FWC coverage for 243 articles (94%), it was surpassed in the other 16—most often by ResearchGate (<reflink idref="bib11" id="ref49">11</reflink>), OpenAlex (<reflink idref="bib2" id="ref50">2</reflink>), Semantic Scholar (<reflink idref="bib1" id="ref51">1</reflink>), Dimensions (<reflink idref="bib1" id="ref52">1</reflink>), and Econpapers (<reflink idref="bib1" id="ref53">1</reflink>).</p> <p>Regarding FWC score, Google Scholar was the leader and set the standard for the runners‐up: ResearchGate, ScienceDirect, Semantic Scholar, and Lens. While Google Scholar is the preferable option in most cases, there may be reasons to turn to less complete citation indices. Specifically, Google Scholar's limited download options and its quality‐agnostic indexing of all documents that its automated crawlers assume to be scholarly mean that it is not always the optimal option. Procedures for downloading Google Scholar's FWCs have been somewhat improved with the introduction of Publish or Perish,[<reflink idref="bib31" id="ref54">31</reflink>] a software application that allows the search and retrieval of up to 1000 Google Scholar citations—although citations above this threshold remain inaccessible. If researchers want to download all FWCs of documents, their best choice will be Lens, which currently allows 50,000 citation references to be downloaded—a far more powerful option than those of the similarly comprehensive indices ResearchGate, ScienceDirect, or Semantic Scholar.</p> <p>The choice of databases that cover a high proportion of relevant documents will be important for systematic reviewers, who need to minimize the cost of reviewing hundreds or thousands of potentially relevant articles. If researchers want to retrieve the FWCs of articles published in higher‐quality journals, their preferred option will be Scopus, with its curated content and plentiful download options. Here, researchers may currently download 2000 or 20,000 citation references (depending on download settings). Web of Science Core Collection, Scopus' often‐compared alternative, was found to be inferior, with narrower FWC coverage along with more restrictive download options of 1000 or 5000 citation references (depending on download settings). Moreover, Lens, Scopus, and Web of Science prove even more useful, as all three offer simultaneous downloading of the FWCs (and BWCs) of multiple search results simultaneously, which few other indices allow.</p> <p>Other databases with FWC information will likely only be relevant for researchers who have reason to use more limited citation information—for example, if they wish to download publisher‐ or database‐specific citations. When the goal is comprehensiveness, whether overall or of higher‐quality articles, these smaller‐coverage alternatives will have a lesser role to play. Overall, however, the analyses show that researchers should generally avoid using specialized databases for FWC searches since their coverage is often severely limited compared to the more capable multidisciplinary options.</p> <hd id="AN0180900140-14">Verdict: The best options for BWC searching</hd> <p>The best databases for BWC searching will cover the most articles with BWC information and exhibit the smallest deviation from the actual number of references in full texts. The BWC score, a product of these factors, revealed specialized databases like ACM Guide to Computing Literature, IEEE Xplore, SAGE Journals Online, PLOS, and ScienceDirect as leaders. These top‐rated databases are predominantly publisher‐based and provide high‐quality data through direct access to the articles they host. Noteworthy is the difference between ScienceDirect, which provides better‐quality BWC data due to its exclusive coverage of in‐house articles, and the somewhat inferior Scopus, which includes citation information from external data sources. However, data quality also varies among publishers. For instance, while Taylor and Francis Online only provided BWC information for about 70% of its articles, SpringerLink covers 96%.</p> <p>While publisher databases offer strong BWC scores, researchers will want to access more comprehensive databases that cover a larger share of the articles for which they seek citations. Among these types of databases, the Web of Science Core Collection has the broadest BWC coverage (100%) and the fewest BWC errors (5.8%). In case of deviations, the database overstates references rather than omitting them, meaning researchers will miss fewer BWCs. Scopus, as the next best alternative to Web of Science Core Collection, has a slightly higher rate of BWC errors (10.1%) while having the same BWC coverage (100%). As with the FWC results, Lens will also be the best option to identify BWCs among large databases: its BWC accuracy (17.7%) and BWC coverage (100%) surpass that of larger counterparts such as Semantic Scholar, ResearchGate, Dimensions, Scite, or OpenAlex.</p> <hd id="AN0180900140-15">Differences in citation coverage at platform providers</hd> <p>In addition to revealing where best to search for FWC and BWC references, the analysis also afforded insights into some noteworthy database particularities. A key observation is the variation in citation information available at databases offered by platform providers such as ProQuest, EBSCOhost, Web of Science, and Ovid. Specifically, there are noticeable discrepancies in the FWC information provided for the same article obtained from different databases from the same provider. For instance, substantial differences in FWCs were observed between Medline and PsycInfo (both under EBSCOhost) and between the Social Science Citation Index and Biosis (both under Web of Science). It appears that these providers restrict citation indices to the specific corpus of references contained within each database, despite having access to additional references in other databases they host. Additionally, the analysis highlighted a unique aspect of EBSCOhost: this provider offers various versions of its databases, each offering different citation coverage. For example, Academic Search <emph>Premier</emph> lists both BWC and FWC information, while Academic Search <emph>Elite</emph> does not.</p> <p>The analysis of BWC information showed consistent data across Web of Science's various databases. The observed differences in BWC scores were attributable to the varying availability of articles across these databases rather than disparities in the completeness of BWC data per article. In contrast, at EBSCOhost I also found minor and, at Ovid, even major variations in BWC information, depending on the specific database used to access the article. A notable exception was ProQuest: across all 11 databases from this provider that I tested, both FWC and BWC information remained uniform, demonstrating a consistency not observed in the other database providers.</p> <p>This study also examined the influence of database providers on the quality of citation information, focusing on PsycInfo, RePEc, and Medline across different platforms. The comparison revealed significant disparities in citation comprehensiveness depending on the provider. For PsycInfo, EBSCOhost offered more complete FWC information (11.8%) compared to Ovid (0.7%), whereas Ovid provided more comprehensive BWC information (93.2%) than EBSCOhost (86.9%). Regarding RePEc, BWC information was slightly better at IDEAS (26.0%) than EconPapers (23.9%), and both providers showed comparably patchy FWC completeness (about 9%). These findings highlight the critical role of database selection in research, underscoring the limitations of relying on a single provider for complete citation information.</p> <p>Medical researchers must choose which system they will use to access the Medline database. Options include dedicated database providers or systems such as PubMed and Europe PMC that include Medline as part of their larger databases. The analysis revealed notable differences in citation information based on the systems with which they are accessed. Medline accessed through Web of Science demonstrated superior FWC information (59.5%) compared to Ovid (3.7%). EBSCOhost was excluded from this analysis as it lacks either FWC or BWC information for Medline. Web of Science's Medline also led in BWC completeness (87.0%), outperforming Ovid's Medline (40.3%). Interestingly, Europe PMC showed better BWC (81.0%) and FWC (28.4%) information than PubMed (60.2% BWC, 26.1% FWC). Although PubMed's BWC information is accurate (4.6% mean absolute percentage error), it is only provided for 63.1% of its articles. In contrast, PubMed's FWC information, while very complete (99.1%), still only covers 26.1% of the citations identified by the benchmark index. Europe PMC exhibits slightly better performance, aligning with Gusenbauer's[<reflink idref="bib32" id="ref55">32</reflink>] findings of superior medical data coverage. This underscores the need for greater awareness within the medical research community about the advantages of Europe PMC, especially given that PubMed seems to be about 60 times more popular than Europe PMC.[<reflink idref="bib1" id="ref56">1</reflink>]</p> <hd id="AN0180900140-16">DISCUSSION</hd> <p></p> <hd id="AN0180900140-17">How do the results compare to prior research?</hd> <p>This study's analysis of citation indices' BWC and FWC quality yields insights crucial for identifying optimal resources for document discovery, addressing a significant research gap also highlighted by a recent Delphi panel of knowledge synthesis experts.[<reflink idref="bib29" id="ref57">29</reflink>]</p> <p>First, the analysis updates previous studies that compared single indices' FWC and BWC coverage—mainly those of Google Scholar, Scopus, and Web of Science, which are the resources that have dominated systematic review practice in the past.[[<reflink idref="bib2" id="ref58">2</reflink>], [<reflink idref="bib14" id="ref59">14</reflink>]] The results show that databases' coverage has changed over the last decade, magnifying differences in coverage, particularly in Google Scholar's FWC coverage advantage over Scopus and Web of Science.[[<reflink idref="bib7" id="ref60">7</reflink>], [<reflink idref="bib23" id="ref61">23</reflink>]] Google Scholar was found to provide the most FWC references for almost all articles and was thus confirmed as providing the broadest FWC coverage.[<reflink idref="bib7" id="ref62">7</reflink>] Further, the results confirm earlier results on Scopus being somewhat preferable to Web of Science Core Collection regarding FWC coverage.[<reflink idref="bib22" id="ref63">22</reflink>]</p> <p>Second, the study underscores the emergence of new resources challenging established databases. Compared to Google Scholar's benchmark position, researchers should be aware of emerging alternatives: for FWCs, ResearchGate, ScienceDirect, and particularly Lens are the most viable options. Other newcomers, such as Scite, Dimensions, OpenAlex, and Scinapse, are good options, too, yet with slightly smaller coverage. For BWCs, the analysis showed that these emerging options have accuracy disadvantages compared to publisher databases or the curated databases of Web of Science and Scopus, which are still preferable to the emerging resources.</p> <hd id="AN0180900140-18">How many citation indices should be searched for knowledge synthesis?</hd> <p>These findings also help to answer the question of whether one[<reflink idref="bib22" id="ref64">22</reflink>] or multiple[<reflink idref="bib25" id="ref65">25</reflink>] citation indices should be searched in systematic reviews. For systematic reviewers, the Cochrane Handbook's MECIR item C30 clearly states expectations of comprehensiveness regarding mandatory BWC searching: "Searches for studies should be as extensive as possible in order to reduce the risk of publication bias and to identify as much relevant evidence as possible."[<reflink idref="bib3" id="ref66">3</reflink>]<sups>(p.19)</sups> On the other hand, the technical supplement to searching further clarifies that "it is not a requirement for Cochrane Reviews [...] to conduct exhaustive citation searching using multiple citation indexes."[<reflink idref="bib2" id="ref67">2</reflink>]<sups>(p.8)</sups> Turning to FWC searching, the Cochrane Handbook's technical supplement states that it "is good practice to carry out forwards citation searching,"[<reflink idref="bib2" id="ref68">2</reflink>]<sups>(p.5)</sups> yet "review authors and information specialists should consider the time and resources available and the likelihood of identifying unique studies for the review question, when planning whether and how to conduct forwards citation searching."[<reflink idref="bib2" id="ref69">2</reflink>]<sups>(p.8)</sups></p> <p>Thus, in principle, the BWC information quality of the Web of Science Core Collection and Scopus databases should suffice for Cochrane's current requirements to identify almost all BWCs, if authors only search only one of the two. In case of errors, both databases overstate the number of references rather than understating them. Using these databases over manual reference‐list checking makes screening more effective, as authors may deduplicate BWC references and screen based on the abstract information provided by databases. However, with effective deduplication tools such as that provided by SR Accelerator,[<reflink idref="bib33" id="ref70">33</reflink>] BWC information from multiple databases can be merged most resourcefully. Selecting multiple capable citation indices will likely bridge coverage gaps in individual indices and thus better satisfy the exhaustiveness criterion of knowledge synthesis. Thus, with new efficiencies in effective deduplication and dataset organization, the additionally identified BWC references may improve BWC search comprehensiveness enough to warrant the extra effort required to search multiple indices—a procedure that is increasingly advocated among knowledge synthesis specialists.[<reflink idref="bib29" id="ref71">29</reflink>]</p> <p>For FWC searching, review authors need to weigh the additional workload of including multiple databases with the value of identifying more relevant hits due to larger search scope.[<reflink idref="bib13" id="ref72">13</reflink>] This decision on which database(s) to use will primarily depend on the scope of the study and the type of documents authors seek. When authors seek FWCs for articles from higher‐quality journals, it might be advisable to search databases with more focused coverage, such as Scopus, Web of Science, or some subject‐focused database (see also bin Ali & Tanveer[<reflink idref="bib21" id="ref73">21</reflink>]). If the goal is to include gray literature too, Google Scholar's superior comprehensiveness will yield more relevant results—however, some studies deem FWC searching at Google Scholar unfeasible due to a lack of functionality,[<reflink idref="bib21" id="ref74">21</reflink>] excessive workload,[<reflink idref="bib20" id="ref75">20</reflink>] and inferior quality of covered studies.[<reflink idref="bib7" id="ref76">7</reflink>] Further, authors will not be able to retrieve more than 1000 FWCs at Google Scholar. In these cases, while Google Scholar theoretically provides the most FWCs, it actually offers fewer retrievable citations than the alternatives. Thus, to evaluate the best citation databases, it is vital to consider options to manipulate, access, and retrieve search results, as well as coverage (for more information, see <ulink href="http://www.searchsmart.org">www.searchsmart.org</ulink>).</p> <p>The runners‐up to Google Scholar are ResearchGate, ScienceDirect, and Semantic Scholar. Yet, all offer only limited data‐handling functionality—or, in the case of ScienceDirect, are limited to the contents of Elsevier. Thus, Lens, with 71.1% of Google Scholar's coverage, is the next best alternative due to its superior download options. Lens is also an interesting choice for citation searching because it may be accessed through applications such as Citationchaser[<reflink idref="bib28" id="ref77">28</reflink>] and SpiderCite[<reflink idref="bib33" id="ref78">33</reflink>] that focus on handling large FWC and BWC datasets. An advantage of searching for FWCs (or BWCs) in Lens' native search interface is that results may be further filtered by document types, data ranges, or journals, or via additional searches within the results set.</p> <hd id="AN0180900140-19">Suggested resources for citation visualization</hd> <p>The findings of this study will also be relevant for bibliometric analyses of citation networks, as the comprehensiveness of the citation indices will determine the coverage and shape of the resulting network graphs. Like knowledge synthesists, bibliometric researchers must decide which databases they will rely on. While there are many bibliometric tools, only a few are popular and explicitly mention the support of specific databases' citation indices. Among them are VOSviewer,[<reflink idref="bib34" id="ref79">34</reflink>] CiteSpace,[<reflink idref="bib35" id="ref80">35</reflink>] and Bibliometrix.[<reflink idref="bib36" id="ref81">36</reflink>] Table 2 highlights all databases that are supported by one or more of the bibliometric tools. The ones supported by all three tools are Dimensions, Lens, PubMed, Scopus, and Web of Science (Core Collection and others).[[<reflink idref="bib34" id="ref82">34</reflink>], [<reflink idref="bib37" id="ref83">37</reflink>]]</p> <p>2 TABLE Databases' backward citation coverage (ranked by highest BWC score).</p> <p> <ephtml> <table><thead valign="bottom"><tr><th align="left">Database name</th><th align="left">Supported by popular bibliometric tools</th><th align="left">(a): % of articles covered</th><th align="left">(d): % of articles with BWC info</th><th align="left">(e): mean percentage error</th><th align="left">(f): mean absolute percentage error</th><th align="left">BWC score: d × (1−f)</th></tr></thead><tbody valign="top"><tr><td align="left">ACM Guide to Computing Literature</td><td align="left">‐</td><td align="char" char=".">3.9%</td><td align="char" char=".">100.0%</td><td align="char" char=".">± 0.0%</td><td align="char" char=".">0.0%</td><td align="left">100.0%</td></tr><tr><td align="left">IEEE Xplore</td><td align="left">‐</td><td align="char" char=".">3.9%</td><td align="char" char=".">100.0%</td><td align="char" char=".">± 0.0%</td><td align="char" char=".">0.0%</td><td align="left">100.0%</td></tr><tr><td align="left">SAGE Journals Online</td><td align="left">‐</td><td align="char" char=".">5.4%</td><td align="char" char=".">100.0%</td><td align="char" char=".">−0.9%</td><td align="char" char=".">0.9%</td><td align="left">99.1%</td></tr><tr><td align="left">Public Library of Science (PLOS)</td><td align="left">‐</td><td align="char" char=".">3.5%</td><td align="char" char=".">100.0%</td><td align="char" char=".">−1.0%</td><td align="char" char=".">1.0%</td><td align="left">99.0%</td></tr><tr><td align="left">ScienceDirect</td><td align="left">‐</td><td align="char" char=".">26.6%</td><td align="char" char=".">100.0%</td><td align="char" char=".">+1.3%</td><td align="char" char=".">1.5%</td><td align="left">98.5%</td></tr><tr><td align="left">Arts & Humanities Citation Index (via Web of Science)</td><td align="left">‐</td><td align="char" char=".">3.9%</td><td align="char" char=".">100.0%</td><td align="char" char=".">−0.8%</td><td align="char" char=".">1.7%</td><td align="left">98.3%</td></tr><tr><td align="left">Emerging Sources Citation Index (via Web of Science)</td><td align="left">Bibliometrix; CiteSpace; VOSviewer</td><td align="char" char=".">8.5%</td><td align="char" char=".">100.0%</td><td align="char" char=".">−0.1%</td><td align="char" char=".">2.7%</td><td align="left">97.3%</td></tr><tr><td align="left">Conference Proceedings Citation Index—Social Science & Humanities (via Web of Science)</td><td align="left">Bibliometrix; CiteSpace; VOSviewer</td><td align="char" char=".">4.2%</td><td align="char" char=".">100.0%</td><td align="char" char=".">+1.9%</td><td align="char" char=".">3.0%</td><td align="left">97.0%</td></tr><tr><td align="left">Social Sciences Citation Index (via Web of Science)</td><td align="left">Bibliometrix; CiteSpace; VOSviewer</td><td align="char" char=".">32.0%</td><td align="char" char=".">100.0%</td><td align="char" char=".">+1.2%</td><td align="char" char=".">4.9%</td><td align="left">95.1%</td></tr><tr><td align="left">JSTOR</td><td align="left">‐</td><td align="char" char=".">6.9%</td><td align="char" char=".">100.0%</td><td align="char" char=".">−0.8%</td><td align="char" char=".">5.0%</td><td align="left">95.0%</td></tr><tr><td align="left">Web of Science Core Collection</td><td align="left">Bibliometrix; CiteSpace; VOSviewer</td><td align="char" char=".">86.5%</td><td align="char" char=".">100.0%</td><td align="char" char=".">+2.9%</td><td align="char" char=".">5.8%</td><td align="left">94.2%</td></tr><tr><td align="left">Science Citation Index Expanded (via Web of Science)</td><td align="left">Bibliometrix; CiteSpace; VOSviewer</td><td align="char" char=".">64.9%</td><td align="char" char=".">100.0%</td><td align="char" char=".">+2.6%</td><td align="char" char=".">6.1%</td><td align="left">93.9%</td></tr><tr><td align="left">SpringerLink</td><td align="left">‐</td><td align="char" char=".">10.4%</td><td align="char" char=".">96.3%</td><td align="char" char=".">+1.5%</td><td align="char" char=".">2.9%</td><td align="left">93.5%</td></tr><tr><td align="left">APA PsycInfo (via Ovid)</td><td align="left">‐</td><td align="char" char=".">19.7%</td><td align="char" char=".">100.0%</td><td align="char" char=".">+3.5%</td><td align="char" char=".">6.8%</td><td align="left">93.2%</td></tr><tr><td align="left">Wiley Online Library</td><td align="left">‐</td><td align="char" char=".">11.2%</td><td align="char" char=".">100.0%</td><td align="char" char=".">−7.0%</td><td align="char" char=".">7.7%</td><td align="left">92.3%</td></tr><tr><td align="left">Scopus</td><td align="left">Bibliometrix; CiteSpace; VOSviewer</td><td align="char" char=".">98.1%</td><td align="char" char=".">100.0%</td><td align="char" char=".">+2.8%</td><td align="char" char=".">10.1%</td><td align="left">89.9%</td></tr><tr><td align="left">Medline (via Web of Science)</td><td align="left">Bibliometrix; CiteSpace; VOSviewer</td><td align="char" char=".">43.2%</td><td align="char" char=".">97.3%</td><td align="char" char=".">+4.7%</td><td align="char" char=".">10.6%</td><td align="left">87.0%</td></tr><tr><td align="left">APA PsycInfo (via EBSCOhost)</td><td align="left">‐</td><td align="char" char=".">20.1%</td><td align="char" char=".">92.3%</td><td align="char" char=".">+5.2%</td><td align="char" char=".">5.9%</td><td align="left">86.9%</td></tr><tr><td align="left">BIOSIS Citation Index (via Web of Science)</td><td align="left">‐</td><td align="char" char=".">21.2%</td><td align="char" char=".">100.0%</td><td align="char" char=".">+11.5%</td><td align="char" char=".">13.6%</td><td align="left">86.4%</td></tr><tr><td align="left">Lens</td><td align="left">Bibliometrix; CiteSpace; VOSviewer</td><td align="char" char=".">96.5%</td><td align="char" char=".">100.0%</td><td align="char" char=".">−10.1%</td><td align="char" char=".">17.7%</td><td align="left">82.3%</td></tr><tr><td align="left">Europe PMC</td><td align="left">VOSviewer</td><td align="char" char=".">44.4%</td><td align="char" char=".">84.3%</td><td align="char" char=".">−0.8%</td><td align="char" char=".">4.0%</td><td align="left">81.0%</td></tr><tr><td align="left">Semantic Scholar</td><td align="left">CiteSpace; VOSviewer</td><td align="char" char=".">98.5%</td><td align="char" char=".">96.9%</td><td align="char" char=".">+4.2%</td><td align="char" char=".">17.0%</td><td align="left">80.4%</td></tr><tr><td align="left">SocINDEX (via EBSCOhost)</td><td align="left">‐</td><td align="char" char=".">5.8%</td><td align="char" char=".">86.7%</td><td align="char" char=".">−1.9%</td><td align="char" char=".">9.1%</td><td align="left">78.7%</td></tr><tr><td align="left">Conference Proceedings Citation Index—Science (via Web of Science)</td><td align="left">Bibliometrix; CiteSpace; VOSviewer</td><td align="char" char=".">4.6%</td><td align="char" char=".">100.0%</td><td align="char" char=".">−20.7%</td><td align="char" char=".">21.5%</td><td align="left">78.5%</td></tr><tr><td align="left">ResearchGate</td><td align="left">‐</td><td align="char" char=".">95.8%</td><td align="char" char=".">99.2%</td><td align="char" char=".">+14.2%</td><td align="char" char=".">23.1%</td><td align="left">76.3%</td></tr><tr><td align="left">Sociological Abstracts (via ProQuest)</td><td align="left">‐</td><td align="char" char=".">6.9%</td><td align="char" char=".">94.4%</td><td align="char" char=".">+15.8%</td><td align="char" char=".">19.3%</td><td align="left">76.2%</td></tr><tr><td align="left">Scinapse</td><td align="left">‐</td><td align="char" char=".">60.2%</td><td align="char" char=".">100.0%</td><td align="char" char=".">−23.0%</td><td align="char" char=".">23.9%</td><td align="left">76.1%</td></tr><tr><td align="left">Dimensions</td><td align="left">CiteSpace; Bibliometrix; VOSviewer</td><td align="char" char=".">97.7%</td><td align="char" char=".">96.0%</td><td align="char" char=".">−19.3%</td><td align="char" char=".">21.1%</td><td align="left">75.8%</td></tr><tr><td align="left">Scite</td><td align="left">‐</td><td align="char" char=".">97.7%</td><td align="char" char=".">97.6%</td><td align="char" char=".">−21.4%</td><td align="char" char=".">24.8%</td><td align="left">73.4%</td></tr><tr><td align="left">GreenFILE (via EBSCOhost)</td><td align="left">‐</td><td align="char" char=".">4.6%</td><td align="char" char=".">75.0%</td><td align="char" char=".">+1.0%</td><td align="char" char=".">4.3%</td><td align="left">71.8%</td></tr><tr><td align="left">OpenAlex</td><td align="left">VOSviewer</td><td align="char" char=".">100.0%</td><td align="char" char=".">96.9%</td><td align="char" char=".">−23.9%</td><td align="char" char=".">28.9%</td><td align="left">68.9%</td></tr><tr><td align="left">Taylor and Francis Online</td><td align="left">‐</td><td align="char" char=".">7.7%</td><td align="char" char=".">70.0%</td><td align="char" char=".">+4.1%</td><td align="char" char=".">4.1%</td><td align="left">67.2%</td></tr><tr><td align="left">CAS SciFinder‐n</td><td align="left">‐</td><td align="char" char=".">49.4%</td><td align="char" char=".">78.1%</td><td align="char" char=".">−5.2%</td><td align="char" char=".">14.1%</td><td align="left">67.1%</td></tr><tr><td align="left">Scilit</td><td align="left">‐</td><td align="char" char=".">99.6%</td><td align="char" char=".">93.8%</td><td align="char" char=".">−25.3%</td><td align="char" char=".">29.3%</td><td align="left">66.3%</td></tr><tr><td align="left">Social Science Premium Collection (via ProQuest)</td><td align="left">‐</td><td align="char" char=".">20.5%</td><td align="char" char=".">75.5%</td><td align="char" char=".">+9.6%</td><td align="char" char=".">12.6%</td><td align="left">66.0%</td></tr><tr><td align="left">PubMed</td><td align="left">Bibliometrix; CiteSpace; VOSviewer</td><td align="char" char=".">42.9%</td><td align="char" char=".">63.1%</td><td align="char" char=".">−0.9%</td><td align="char" char=".">4.6%</td><td align="left">60.2%</td></tr><tr><td align="left">zbMATH Open</td><td align="left">‐</td><td align="char" char=".">3.9%</td><td align="char" char=".">60.0%</td><td align="char" char=".">±0.0%</td><td align="char" char=".">0.0%</td><td align="left">60.0%</td></tr><tr><td align="left">Public Health Database (via ProQuest)</td><td align="left">‐</td><td align="char" char=".">9.7%</td><td align="char" char=".">64.0%</td><td align="char" char=".">−3.8%</td><td align="char" char=".">8.7%</td><td align="left">58.5%</td></tr><tr><td align="left">Naver Academic</td><td align="left">‐</td><td align="char" char=".">84.9%</td><td align="char" char=".">100.0%</td><td align="char" char=".">−42.1%</td><td align="char" char=".">43.5%</td><td align="left">56.5%</td></tr><tr><td align="left">Nursing & Allied Health Database (via ProQuest)</td><td align="left">‐</td><td align="char" char=".">11.6%</td><td align="char" char=".">63.3%</td><td align="char" char=".">−6.1%</td><td align="char" char=".">12.7%</td><td align="left">55.3%</td></tr><tr><td align="left">International Bibliography of the Social Sciences (via ProQuest)</td><td align="left">‐</td><td align="char" char=".">9.7%</td><td align="char" char=".">68.0%</td><td align="char" char=".">+16.1%</td><td align="char" char=".">18.8%</td><td align="left">55.2%</td></tr><tr><td align="left">Bielefeld Academic Search Engine</td><td align="left">‐</td><td align="char" char=".">93.1%</td><td align="char" char=".">66.0%</td><td align="char" char=".">−20.4%</td><td align="char" char=".">26.1%</td><td align="left">48.8%</td></tr><tr><td align="left">Business Source Premier (via EBSCOhost)</td><td align="left">‐</td><td align="char" char=".">12.0%</td><td align="char" char=".">54.8%</td><td align="char" char=".">−5.5%</td><td align="char" char=".">11.1%</td><td align="left">48.8%</td></tr><tr><td align="left">Environmental Science Database (via ProQuest)</td><td align="left">‐</td><td align="char" char=".">6.6%</td><td align="char" char=".">58.8%</td><td align="char" char=".">−17.8%</td><td align="char" char=".">17.8%</td><td align="left">48.4%</td></tr><tr><td align="left">Earth, Atmospheric & Aquatic Science Database (via ProQuest)</td><td align="left">‐</td><td align="char" char=".">4.6%</td><td align="char" char=".">50.0%</td><td align="char" char=".">+5.3%</td><td align="char" char=".">5.6%</td><td align="left">47.2%</td></tr><tr><td align="left">ABI/Inform Global (via ProQuest)</td><td align="left">‐</td><td align="char" char=".">12.7%</td><td align="char" char=".">51.5%</td><td align="char" char=".">+11.2%</td><td align="char" char=".">13.2%</td><td align="left">44.7%</td></tr><tr><td align="left">Biological Science Database (via ProQuest)</td><td align="left">‐</td><td align="char" char=".">11.2%</td><td align="char" char=".">55.2%</td><td align="char" char=".">−1.4%</td><td align="char" char=".">19.8%</td><td align="left">44.3%</td></tr><tr><td align="left">Medline (via Ovid)</td><td align="left">‐</td><td align="char" char=".">40.9%</td><td align="char" char=".">59.4%</td><td align="char" char=".">−30.0%</td><td align="char" char=".">32.1%</td><td align="left">40.3%</td></tr><tr><td align="left">SciTech Premium Collection (via ProQuest)</td><td align="left">‐</td><td align="char" char=".">68.3%</td><td align="char" char=".">39.0%</td><td align="char" char=".">+3.7%</td><td align="char" char=".">10.4%</td><td align="left">34.9%</td></tr><tr><td align="left">ScienceOpen</td><td align="left">‐</td><td align="char" char=".">96.1%</td><td align="char" char=".">100.0%</td><td align="char" char=".">−68.4%</td><td align="char" char=".">68.8%</td><td align="left">31.2%</td></tr><tr><td align="left">RePEc (via IDEAS)</td><td align="left">‐</td><td align="char" char=".">12.0%</td><td align="char" char=".">74.2%</td><td align="char" char=".">−64.9%</td><td align="char" char=".">64.9%</td><td align="left">26.0%</td></tr><tr><td align="left">RePEc (via EconPapers)</td><td align="left">‐</td><td align="char" char=".">12.7%</td><td align="char" char=".">72.7%</td><td align="char" char=".">−67.2%</td><td align="char" char=".">67.2%</td><td align="left">23.9%</td></tr><tr><td align="left">EconStor</td><td align="left">‐</td><td align="char" char=".">3.5%</td><td align="char" char=".">33.3%</td><td align="char" char=".">−33.9%</td><td align="char" char=".">33.9%</td><td align="left">22.0%</td></tr><tr><td align="left">Epistemonikos</td><td align="left">‐</td><td align="char" char=".">4.6%</td><td align="char" char=".">50.0%</td><td align="char" char=".">−65.0%</td><td align="char" char=".">65.0%</td><td align="left">17.5%</td></tr><tr><td align="left">Ingenta Connect</td><td align="left">‐</td><td align="char" char=".">6.9%</td><td align="char" char=".">27.8%</td><td align="char" char=".">−60.9%</td><td align="char" char=".">60.9%</td><td align="left">10.9%</td></tr><tr><td align="left">GeoRef (via ProQuest)</td><td align="left">‐</td><td align="char" char=".">3.9%</td><td align="char" char=".">10.0%</td><td align="char" char=".">±0.0%</td><td align="char" char=".">0.0%</td><td align="left">10.0%</td></tr><tr><td align="left">Academic Search Premier (via EBSCOhost)</td><td align="left">‐</td><td align="char" char=".">47.9%</td><td align="char" char=".">7.3%</td><td align="char" char=".">−4.6%</td><td align="char" char=".">4.6%</td><td align="left">6.9%</td></tr><tr><td align="left">Academic Search Elite (via EBSCOhost)</td><td align="left">‐</td><td align="char" char=".">47.9%</td><td align="char" char=".">0.0%</td><td align="char" char=".">‐</td><td align="char" char=".">‐</td><td align="left">N/a</td></tr><tr><td align="left">Google Scholar</td><td align="left">‐</td><td align="char" char=".">100.0%</td><td align="char" char=".">0.0%</td><td align="char" char=".">‐</td><td align="char" char=".">‐</td><td align="left">N/a</td></tr></tbody></table> </ephtml> </p> <ulist> <item>3 <emph>Note</emph>: Bold indicates the most important information.</item> <item>4 Abbreviation: BWC, backward citation.</item> </ulist> <p>In most instances, bibliometric tools primarily rely on BWC information to build network graphs. For example, VOSviewer builds its network based on BWC information, with the option to include FWCs to determine the size of the connected circles. Users must select one specific citation database when performing analyses at VOSviewer (and others), making the choice of a suitable option even more important. Thus, researchers will want to use the citation index with the broadest citation coverage of the data types they are interested in—information provided in Tables 1 and 2.</p> <hd id="AN0180900140-20">Suggested freely available resources</hd> <p>Due to restrictions on database access, not every researcher has the ability to access all the citation indices evaluated in this study. Fortunately, the top‐rated indices for FWC searches are openly accessible (as shown in Table 1), offering valuable resources for citation searches to all researchers, especially those from resource‐limited settings. However, notable disadvantages arise when accessing the restricted databases of Scopus and Web of Science, particularly for BWC searches where citation accuracy is critical. In this case, Lens and Semantic Scholar, as the next best options, are less accurate, as they under‐report and over‐report citation references respectively. To mitigate this, researchers may consider using multiple citation indices or manually screening reference lists for BWC searches, even though these approaches will result in a heavier workload.</p> <hd id="AN0180900140-21">Emerging citation qualifiers</hd> <p>This study analyzed "conventional" FWCs and BWCs—that is, information on one study citing another. In recent years, some databases have started classifying these citation linkages. Semantic Scholar began providing "citation types," classifying highly influential citations, background citations, methods citations, and results citations.[<reflink idref="bib39" id="ref84">39</reflink>] Later, Scite introduced "smart citations," which "reveal how a scientific paper has been cited by providing the context of the citation and a classification system describing whether it provides supporting or contrasting evidence for the cited claim, or if it just mentions it."[<reflink idref="bib40" id="ref85">40</reflink>]<sups>(p883)</sups> Scite's smart citations have been adopted by ACS Publications, arXiv, Wiley, The Royal Society Publishing, and PNAS.[<reflink idref="bib41" id="ref86">41</reflink>] Given this growing interest, Web of Science also introduced "citation classes" classifying how citations are used—that is, background, basis, support, differ, and discuss. The usefulness of these citation types is an area yet to be explored. As authors have manifold reasons to cite other studies, none of which are reflected in the "conventional" FWC and BWC information, classifying or weighing citations based on importance or context promises rich insights into the scientific discourse. Thus, the emerging citation qualifiers are a promising area for future research and a tool that may also improve the effectiveness of knowledge discovery.</p> <hd id="AN0180900140-22">Limitations</hd> <p>This study is subject to certain limitations in both scope and methodology. Firstly, while the analysis utilizes a set of multidisciplinary articles to estimate the overall BWC and FWC coverage of various databases, it does not account for the variability in citation rates across different academic fields.[<reflink idref="bib42" id="ref87">42</reflink>] Despite this, the findings align with previous discipline‐specific studies, suggesting that the results are likely consistent and applicable across disciplines. Second, the analysis was based on journal articles—arguably the most common and sought‐after type of scholarly document. This study did not include other document types like conference papers, dissertations, or gray literature. Therefore, the results may differ somewhat for citation indices that have systematic variations in citation coverage across different document types.</p> <p>Third, this study did not delve into the detailed composition of the BWC or FWC information, nor did it explore the overlap between different citation indices. The expansive scope of analyzing a large number of databases necessitated a quantitative approach, leaving in‐depth analyses beyond the study's scope. BWC accuracy was determined based on a match of the number of references manually counted in full texts and the number reported in the BWC index. While this quantitative counting method will reflect accuracy in most cases, there will be some cases where counts match, but the underlying references differ, creating some bias toward overestimating BWC quality. Further, FWC scores were determined based on the database with the highest FWC count—most often Google Scholar. Google Scholar's citation information is automatically generated by bots that search far and wide, making its citation information the most comprehensive. As a downside, however, the quality of its citation information is not validated. Recently, it was found that Google Scholar's citation counts include a small yet non‐negligible portion of citations to nonexistent works—citations that were bought from dubious services.[<reflink idref="bib43" id="ref88">43</reflink>] As a consequence, the FWC score of Google Scholar (and others?) is likely biased upwards, potentially somewhat overstating the differences in actual FWC coverage compared to other databases.</p> <p>The extent to which other databases' citation indices are also affected by these issues leaves scope for future research. Researchers aiming to limit searches to higher‐quality document types may use the more limited indices of Scopus or Web of Science, which have some quality controls in place. By comparing the FWC scores with that of Google Scholar, researchers can estimate pro rata what proportion of citations they are likely to miss out on. Overall, as no service will be entirely error‐free, every research project should include a step of manually validating individual references before they are analyzed or synthesized.</p> <hd id="AN0180900140-23">CONCLUSION</hd> <p>This study advances our understanding of the utility of 59 databases' forward and backward citation indices relevant for scholarly knowledge discovery. It evaluates the distinct strengths and limitations of key citation indices such as Google Scholar, Web of Science, Scopus, Lens, OpenAlex, and PubMed in terms of coverage and accuracy. Google Scholar was reaffirmed as having the highest FWC counts, which also explains why researchers seem to like using it to showcase their academic impact. The analysis highlights the relative advantages and drawbacks of dominant proprietary databases such as Scopus and Web of Science, contrasting them with open‐access alternatives. For BWC searches, subject‐specific publisher databases like ACM Guide to Computing Literature or IEEE Xplore emerged as the most accurate, yet with restricted document scopes and retrieval options.</p> <p>These findings are vital for researchers engaged in systematic reviews, meta‐analyses, and bibliometric analyses, as the outcomes of these studies are heavily influenced by the coverage and quality of the citation indices utilized. For systematic searchers, integrating citation searches alongside principal (keyword‐based) search strategies is critical to augment results and avoid pitfalls such as "citation bias"[<reflink idref="bib44" id="ref89">44</reflink>] or "citation islands."[<reflink idref="bib45" id="ref90">45</reflink>] Scholars conducting bibliometric analyses should consider how variations in citation indices' coverage impact their findings. To compare the qualities of citation indices, <ulink href="http://www.searchsmart.org">www.searchsmart.org</ulink> provides access to FWC/BWC scores. There, researchers can assess the performance of different databases' citation indices, along with other relevant functional characteristics, to identify the most suitable databases for their systematic reviews, meta‐analyses, or bibliometric analyses.</p> <hd id="AN0180900140-24">AUTHOR CONTRIBUTIONS</hd> <p> <bold>Michael Gusenbauer:</bold> Conceptualization; investigation; writing – original draft; methodology; validation; visualization; writing – review and editing; software; formal analysis; project administration; data curation; resources.</p> <hd id="AN0180900140-25">ACKNOWLEDGMENT</hd> <p>I would also like to express my gratitude to the editors and reviewers for their guidance.</p> <hd id="AN0180900140-26">FUNDING INFORMATION</hd> <p>The author received no financial support for the research, authorship, and/or publication of this article.</p> <hd id="AN0180900140-27">CONFLICT OF INTEREST STATEMENT</hd> <p>The author is the creator of Search Smart (https://<ulink href="http://www.searchsmart.org/),">www.searchsmart.org/),</ulink> a non‐proprietary website to compare academic databases and search systems.</p> <hd id="AN0180900140-28">DATA AVAILABILITY STATEMENT</hd> <p>Supplementary Material 1 for this article are available online and the data can be accessed under https://osf.io/74brd/.</p> <p>GRAPH: DATA S1. Supporting Information.</p> <ref id="AN0180900140-29"> <title> Footnotes </title> <blist> <bibl id="bib1" idref="ref1" type="bt">1</bibl> <bibtext> Comparison based on Similarweb page‐visit data from December 2023.</bibtext> </blist> <blist> <bibl id="bib2" idref="ref11" type="bt">2</bibl> <bibtext> [Correction added on 15 July 2024, after first online publication: 'Acknowledgment' and 'Conflict of Interest Statement' have been added in this version.]</bibtext> </blist> </ref> <ref id="AN0180900140-30"> <title> REFERENCES </title> <blist> <bibtext> Xiao Y, Watson M. Guidance on conducting a systematic literature review. J Plan Educ Res. 2019 ; 39 (1): 93 ‐ 112. doi: 10.1177/0739456X17723971</bibtext> </blist> <blist> <bibtext> Lefebvre C, Glanville J, Briscoe S, et al. Technical supplement to chapter 4: searching for and selecting studies. In: Higgins J, Thomas J, Chandler J, et al., eds. Cochrane Handbook for Systematic Reviews of Interventions. Version 6.4 (updated August 2023). Cochrane, 2023. Available from <ulink href="http://www.training.cochrane.org/handbook">www.training.cochrane.org/handbook</ulink></bibtext> </blist> <blist> <bibl id="bib3" idref="ref2" type="bt">3</bibl> <bibtext> Higgins J, Lasserson T, Thomas J, Flemyng E, Churchill R. Methodological Expectations of Cochrane Intervention Reviews (MECIR). Version August 2023. Cochrane ; 2023 Accessed January 26, 2023.</bibtext> </blist> <blist> <bibl id="bib4" idref="ref3" type="bt">4</bibl> <bibtext> Cantrell A, Booth A, Chambers D. A systematic review case study of urgent and emergency care configuration found citation searching of Web of Science and Google Scholar of similar value. Health Inf Libr J. 2022 ; 41 : 166 ‐ 181. doi: 10.1111/hir.12428</bibtext> </blist> <blist> <bibl id="bib5" type="bt">5</bibl> <bibtext> Bethel AC, Rogers M, Abbott R. Use of a search summary table to improve systematic review search methods, results, and efficiency. J Med Libr Assoc. 2021 ; 109 (1): 97 ‐ 106. doi: 10.5195/jmla.2021.809</bibtext> </blist> <blist> <bibl id="bib6" idref="ref4" type="bt">6</bibl> <bibtext> Cooper C, Lovell R, Husk K, Booth A, Garside R. Supplementary search methods were more effective and offered better value than bibliographic database searching: a case study from public health and environmental enhancement. Res Synth Methods. 2018 ; 9 (2): 195 ‐ 223. doi: 10.1002/jrsm.1286</bibtext> </blist> <blist> <bibl id="bib7" idref="ref34" type="bt">7</bibl> <bibtext> Wright K, Golder S, Rodriguez‐Lopez R. Citation searching: a systematic review case study of multiple risk behaviour interventions. BMC Med Res Methodol. 2014 ; 14 : 1 ‐ 8.</bibtext> </blist> <blist> <bibl id="bib8" idref="ref5" type="bt">8</bibl> <bibtext> Booth A, Wright J, Briscoe S. Scoping and searching to support realist approaches. In: Emmel N, Greenhalgh J, Manzano A, Monaghan M, Dalkin S, eds. Doing Realist Research. SAGE ; 2018 : 147 ‐ 166.</bibtext> </blist> <blist> <bibl id="bib9" idref="ref6" type="bt">9</bibl> <bibtext> Goossen K, Tenckhoff S, Probst P, et al. Optimal literature search for systematic reviews in surgery. Langenbeck's Arch Surg. 2018 ; 403 (1): 119 ‐ 129. doi: 10.1007/s00423‐017‐1646‐x</bibtext> </blist> <blist> <bibtext> Frandsen TF, Eriksen MB. Supplementary strategies identified additional eligible studies in qualitative systematic reviews. J Clin Epidemiol. 2023 ; 159 : 85 ‐ 91. doi: 10.1016/j.jclinepi.2023.04.017</bibtext> </blist> <blist> <bibtext> Briner RB, Denyer D, Rousseau DM. Evidence‐based management: concept cleanup time? Acad Manag Perspect. 2009 ; 23 (4): 19 ‐ 32.</bibtext> </blist> <blist> <bibtext> Page MJ, Shamseer L, Altman DG, et al. Epidemiology and reporting characteristics of systematic reviews of biomedical research: a cross‐sectional study. PLoS Med. 2016 ; 13 (5): e1002028. doi: 10.1371/journal.pmed.1002028</bibtext> </blist> <blist> <bibtext> Briscoe S, Bethel A, Rogers M. Conduct and reporting of citation searching in Cochrane systematic reviews: a cross‐sectional study. Res Synth Methods. 2020 ; 11 (2): 169 ‐ 180. doi: 10.1002/jrsm.1355</bibtext> </blist> <blist> <bibtext> Cooper C, Booth A, Britten N, Garside R. A comparison of results of empirical studies of supplementary search techniques and recommendations in review methodology handbooks: a methodological review. Syst Rev. 2017 ; 6 (1): 234. doi: 10.1186/s13643‐017‐0625‐1</bibtext> </blist> <blist> <bibtext> Greenhalgh T, Peacock R. Effectiveness and efficiency of search methods in systematic reviews of complex evidence: audit of primary sources. BMJ. 2005 ; 331 (7524): 1064 ‐ 1065. doi: 10.1136/bmj.38636.593461.68</bibtext> </blist> <blist> <bibtext> Hinde S, Spackman E. Bidirectional citation searching to completion: an exploration of literature searching methods. PharmacoEconomics. 2015 ; 33 (1): 5 ‐ 11. doi: 10.1007/s40273‐014‐0205‐3</bibtext> </blist> <blist> <bibtext> Papaioannou D, Sutton A, Carroll C, Booth A, Wong R. Literature searching for social science systematic reviews: consideration of a range of search techniques. Health Inf Libr J. 2010 ; 27 (2): 114 ‐ 122. doi: 10.1111/j.1471‐1842.2009.00863.x</bibtext> </blist> <blist> <bibtext> Linder SK, Kamath GR, Pratt GF, Saraykar SS, Volk RJ. Citation searches are more sensitive than keyword searches to identify studies using specific measurement instruments. J Clin Epidemiol. 2015 ; 68 (4): 412 ‐ 417. doi: 10.1016/j.jclinepi.2014.10.008</bibtext> </blist> <blist> <bibtext> Gusenbauer M. The age of abundant scholarly information and its synthesis—a time when 'just google it' is no longer enough. Res Synth Methods. 2021 ; 12 (6): 684 ‐ 691. doi: 10.1002/jrsm.1520</bibtext> </blist> <blist> <bibtext> Levay P, Ainsworth N, Kettle R, Morgan A. Identifying evidence for public health guidance: a comparison of citation searching with Web of Science and Google Scholar. Res Synth Methods. 2016 ; 7 (1): 34 ‐ 45. doi: 10.1002/jrsm.1158</bibtext> </blist> <blist> <bibtext> Bin Ali N, Tanveer B. A comparison of citation sources for reference and citation‐based search in systematic literature reviews. EISEJ. 2022 ; 16 (1): 220106. doi: 10.37190/e‐Inf220106</bibtext> </blist> <blist> <bibtext> Rogers M, Bethel A, Briscoe S. Resources for forwards citation searching for implementation studies in dementia care: a case study comparing Web of Science and Scopus. Res Synth Methods. 2020 ; 11 (3): 379 ‐ 386. doi: 10.1002/jrsm.1400</bibtext> </blist> <blist> <bibtext> Bakkalbasi N, Bauer K, Glover J, Wang L. Three options for citation tracking: Google scholar, Scopus and Web of Science. Biomed Digit Libr. 2006 ; 3 : 7. doi: 10.1186/1742‐5581‐3‐7</bibtext> </blist> <blist> <bibtext> Kulkarni AV, Aziz B, Shams I, Busse JW. Comparisons of citations in Web of Science, Scopus, and Google Scholar for articles published in general medical journals. JAMA. 2009 ; 302 (10): 1092 ‐ 1096. doi: 10.1001/jama.2009.1307</bibtext> </blist> <blist> <bibtext> Hirt J, Nordhausen T, Appenzeller‐Herzog C, Ewald H. Citation tracking for systematic literature searching: a scoping review. Res Synth Methods. 2023 ; 14 (3): 563 ‐ 579. doi: 10.1002/jrsm.1635</bibtext> </blist> <blist> <bibtext> Whear R, Coon JT, Bethel A, Abbott R, Stein K, Garside R. What is the impact of using outdoor spaces such as gardens on the physical and mental well‐being of those with dementia? A systematic review of quantitative and qualitative evidence. J Am Med Dir Assoc. 2014 ; 15 (10): 697 ‐ 705. doi: 10.1016/j.jamda.2014.05.013</bibtext> </blist> <blist> <bibtext> Gusenbauer M. Google Scholar to overshadow them all? Comparing the sizes of 12 academic search engines and bibliographic databases. Scientometrics. 2019 ; 118 (1): 177 ‐ 214. doi: 10.1007/s11192‐018‐2958‐5</bibtext> </blist> <blist> <bibtext> Haddaway NR, Grainger MJ, Gray CT. Citationchaser: a tool for transparent and efficient forward and backward citation chasing in systematic searching. Res Synth Methods. 2022 ; 13 (4): 533 ‐ 545. doi: 10.1002/jrsm.1563</bibtext> </blist> <blist> <bibtext> Hirt J, Nordhausen T, Fuerst T, Ewald H, Appenzeller‐Herzog C. Guidance on terminology, application, and reporting of citation searching: the TARCiS statement. Br Med J. 2024 ; 385 : e078384. doi: 10.1136/bmj‐2023‐078384</bibtext> </blist> <blist> <bibtext> Zhou ZQ, Tse TH, Witheridge M. Metamorphic robustness testing: exposing hidden defects in citation statistics and journal impact factors. IIEEE Trans Software Eng. 2019 ; 47 (6): 1164 ‐ 1183. doi: 10.1109/tse.2019.2915065</bibtext> </blist> <blist> <bibtext> Harzing AW. Publish or Perish. 2007 https://harzing.com/resources/publish-or-perish</bibtext> </blist> <blist> <bibtext> Gusenbauer M. Search where you will find most: comparing the disciplinary coverage of 56 bibliographic databases. Scientometrics. 2022 ; 127 : 2683 ‐ 2745. doi: 10.1007/s11192‐022‐04289‐7</bibtext> </blist> <blist> <bibtext> Clark J, Glasziou P, Del Mar C, Bannach‐Brown A, Stehlik P, Scott AM. A full systematic review was completed in 2 weeks using automation tools: a case study. J Clin Epidemiol. 2020 ; 121 : 81 ‐ 90. doi: 10.1016/j.jclinepi.2020.01.008</bibtext> </blist> <blist> <bibtext> van Eck NJ, Waltman L. Software survey: VOSviewer, a computer program for bibliometric mapping. Scientometrics. 2010 ; 84 (2): 523 ‐ 538. doi: 10.1007/s11192‐009‐0146‐3</bibtext> </blist> <blist> <bibtext> Chen C. CiteSpace II: detecting and visualizing emerging trends and transient patterns in scientific literature. J Am Soc Inf Sci. 2006 ; 57 (3): 359 ‐ 377. doi: 10.1002/asi.20317</bibtext> </blist> <blist> <bibtext> Aria M, Cuccurullo C. Bibliometrix: an R‐tool for comprehensive science mapping analysis. J Informet. 2017 ; 11 (4): 959 ‐ 975.</bibtext> </blist> <blist> <bibtext> Bibliometrix. Frequently asked questions. https://<ulink href="http://www.bibliometrix.org/home/index.php/faq">www.bibliometrix.org/home/index.php/faq</ulink> Accessed February 21, 2024.</bibtext> </blist> <blist> <bibtext> CiteSpace. FAQ. https://citespace.podia.com/faq Updated February 22, 2024.</bibtext> </blist> <blist> <bibtext> Valenzuela M, Ha V, Etzioni O. Identifying Meaningful Citations. AAAI workshop: Scholarly big data. 2015 <ulink href="http://ai2-website.s3.amazonaws.com/publications/valenzuelahameaningfulcitations.pdf">http://ai2-website.s3.amazonaws.com/publications/valenzuelahameaningfulcitations.pdf</ulink></bibtext> </blist> <blist> <bibtext> Nicholson JM, Mordaunt M, Lopez P, et al. Scite: a smart citation index that displays the context of citations and classifies their intent using deep learning. Quant Sci Stud. 2021 ; 2 (3): 882 ‐ 898. doi: 10.1162/qss_a_00146</bibtext> </blist> <blist> <bibtext> scite. The future of citations: displaying citation statements natively on the version of record. https://scite.ai/blog/2022-09-13_native-smart-cites</bibtext> </blist> <blist> <bibtext> Nicolaisen J, Frandsen TF. Number of references: a large‐scale study of interval ratios. Scientometrics. 2021 ; 126 (1): 259 ‐ 285. doi: 10.1007/s11192‐020‐03764‐3</bibtext> </blist> <blist> <bibtext> Ibrahim H, Liu F, Zaki Y, Rahwan T. Google Scholar is manipulatable. 2024 <ulink href="http://arxiv.org/pdf/2402.04607v1">http://arxiv.org/pdf/2402.04607v1</ulink></bibtext> </blist> <blist> <bibtext> Booth A. Cochrane or Cock‐Eyed? How Should We Conduct Systematic Reviews of Qualitative Research? Taking a Critical Stance. Paper presented at the Qualitative Evidence‐based Practice Conference, Coventry University ; 2001.</bibtext> </blist> <blist> <bibtext> Robinson KA, Dunn AG, Tsafnat G, Glasziou P. Citation networks of related trials are often disconnected: implications for bidirectional citation searches. J Clin Epidemiol. 2014 ; 67 (7): 793 ‐ 799.</bibtext> </blist> </ref> <aug> <p>By Michael Gusenbauer</p> <p>Reported by Author</p> </aug> <nolink nlid="nl1" bibid="bib10" firstref="ref7"></nolink> <nolink nlid="nl2" bibid="bib11" firstref="ref9"></nolink> <nolink nlid="nl3" bibid="bib12" firstref="ref12"></nolink> <nolink nlid="nl4" bibid="bib13" firstref="ref14"></nolink> <nolink nlid="nl5" bibid="bib14" firstref="ref15"></nolink> <nolink nlid="nl6" bibid="bib16" firstref="ref16"></nolink> <nolink nlid="nl7" bibid="bib18" firstref="ref17"></nolink> <nolink nlid="nl8" bibid="bib19" firstref="ref18"></nolink> <nolink nlid="nl9" bibid="bib20" firstref="ref21"></nolink> <nolink nlid="nl10" bibid="bib21" firstref="ref22"></nolink> <nolink nlid="nl11" bibid="bib22" firstref="ref23"></nolink> <nolink nlid="nl12" bibid="bib23" firstref="ref24"></nolink> <nolink nlid="nl13" bibid="bib25" firstref="ref27"></nolink> <nolink nlid="nl14" bibid="bib26" firstref="ref30"></nolink> <nolink nlid="nl15" bibid="bib27" firstref="ref33"></nolink> <nolink nlid="nl16" bibid="bib28" firstref="ref36"></nolink> <nolink nlid="nl17" bibid="bib29" firstref="ref38"></nolink> <nolink nlid="nl18" bibid="bib30" firstref="ref46"></nolink> <nolink nlid="nl19" bibid="bib31" firstref="ref54"></nolink> <nolink nlid="nl20" bibid="bib32" firstref="ref55"></nolink> <nolink nlid="nl21" bibid="bib33" firstref="ref70"></nolink> <nolink nlid="nl22" bibid="bib34" firstref="ref79"></nolink> <nolink nlid="nl23" bibid="bib35" firstref="ref80"></nolink> <nolink nlid="nl24" bibid="bib36" firstref="ref81"></nolink> <nolink nlid="nl25" bibid="bib37" firstref="ref83"></nolink> <nolink nlid="nl26" bibid="bib39" firstref="ref84"></nolink> <nolink nlid="nl27" bibid="bib40" firstref="ref85"></nolink> <nolink nlid="nl28" bibid="bib41" firstref="ref86"></nolink> <nolink nlid="nl29" bibid="bib42" firstref="ref87"></nolink> <nolink nlid="nl30" bibid="bib43" firstref="ref88"></nolink> <nolink nlid="nl31" bibid="bib44" firstref="ref89"></nolink> <nolink nlid="nl32" bibid="bib45" firstref="ref90"></nolink>
Header DbId: eric
DbLabel: ERIC
An: EJ1448648
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Beyond Google Scholar, Scopus, and Web of Science: An Evaluation of the Backward and Forward Citation Coverage of 59 Databases' Citation Indices
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Michael+Gusenbauer%22">Michael Gusenbauer</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0001-7768-2351">0000-0001-7768-2351</externalLink>)
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Research+Synthesis+Methods%22"><i>Research Synthesis Methods</i></searchLink>. 2024 15(5):802-817.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 16
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2024
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Information Analyses<br />Reports - Research
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Citation+Indexes%22">Citation Indexes</searchLink><br /><searchLink fieldCode="DE" term="%22Information+Retrieval%22">Information Retrieval</searchLink><br /><searchLink fieldCode="DE" term="%22Databases%22">Databases</searchLink><br /><searchLink fieldCode="DE" term="%22Web+Sites%22">Web Sites</searchLink><br /><searchLink fieldCode="DE" term="%22Online+Searching%22">Online Searching</searchLink><br /><searchLink fieldCode="DE" term="%22Computer+Software+Evaluation%22">Computer Software Evaluation</searchLink><br /><searchLink fieldCode="DE" term="%22Users+%28Information%29%22">Users (Information)</searchLink><br /><searchLink fieldCode="DE" term="%22Access+to+Information%22">Access to Information</searchLink><br /><searchLink fieldCode="DE" term="%22User+Satisfaction+%28Information%29%22">User Satisfaction (Information)</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1002/jrsm.1729
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 1759-2879<br />1759-2887
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Citation indices providing information on backward citation (BWC) and forward citation (FWC) links are essential for literature discovery, bibliographic analysis, and knowledge synthesis, especially when language barriers impede document identification. However, the suitability of citation indices varies. While some have been analyzed, the majority, whether new or established, lack comprehensive evaluation. Therefore, this study evaluates the citation coverage of the citation indices of 59 databases, encompassing the widely used Google Scholar, Scopus, and Web of Science alongside many others never previously analyzed, such as the emerging Lens, Scite, Dimensions, and OpenAlex or the subject-specific PubMed and JSTOR. Through a comprehensive analysis using 259 journal articles from across disciplines, this research aims to guide scholars in selecting indices with broader document coverage and more accurate and comprehensive backward and forward citation links. Key findings highlight Google Scholar, ResearchGate, Semantic Scholar, and Lens as leading options for FWC searching, with Lens providing superior download capabilities. For BWC searching, the Web of Science Core Collection can be recommended over Scopus for accuracy. BWC information from publisher databases such as IEEE Xplore or ScienceDirect was generally found to be the most accurate, yet only available for a limited number of articles. The findings will help scholars conducting systematic reviews, meta-analyses, and bibliometric analyses to select the most suitable databases for citation searching.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: Note
  Label: Notes
  Group: Note
  Data: https://osf.io/74brd
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2024
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1448648
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1448648
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1002/jrsm.1729
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 16
        StartPage: 802
    Subjects:
      – SubjectFull: Citation Indexes
        Type: general
      – SubjectFull: Information Retrieval
        Type: general
      – SubjectFull: Databases
        Type: general
      – SubjectFull: Web Sites
        Type: general
      – SubjectFull: Online Searching
        Type: general
      – SubjectFull: Computer Software Evaluation
        Type: general
      – SubjectFull: Users (Information)
        Type: general
      – SubjectFull: Access to Information
        Type: general
      – SubjectFull: User Satisfaction (Information)
        Type: general
    Titles:
      – TitleFull: Beyond Google Scholar, Scopus, and Web of Science: An Evaluation of the Backward and Forward Citation Coverage of 59 Databases' Citation Indices
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Michael Gusenbauer
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 09
              Type: published
              Y: 2024
          Identifiers:
            – Type: issn-print
              Value: 1759-2879
            – Type: issn-electronic
              Value: 1759-2887
          Numbering:
            – Type: volume
              Value: 15
            – Type: issue
              Value: 5
          Titles:
            – TitleFull: Research Synthesis Methods
              Type: main
ResultId 1