View in EDS HTML Full Text PDF Full Text

Measuring Scientific Impact of Fisheries and Aquaculture Research-for-Development Projects in South East Asia and the Pacific

Saved in:
Bibliographic Details
Title: Measuring Scientific Impact of Fisheries and Aquaculture Research-for-Development Projects in South East Asia and the Pacific
Language: English
Authors: Hiruy, Kiros, Murphy, Ray, Lewis, Tom, White, William T., Purcell, Steven W. (ORCID 0000-0002-6227-6454)
Source: Research Evaluation. Oct 2019 28(4):313-325.
Availability: Oxford University Press. Great Clarendon Street, Oxford, OX2 6DP, UK. Tel: +44-1865-353907; Fax: +44-1865-353485; e-mail: jnls.cust.serv@oxfordjournals.org; Web site: http://applij.oxfordjournals.org/
Peer Reviewed: Y
Page Count: 13
Publication Date: 2019
Document Type: Journal Articles
Reports - Research
Descriptors: Animal Husbandry, Agriculture, Developing Nations, Research Projects, Research Utilization, Influences, Foreign Countries, Scientific Research, Publications, Bibliometrics
Geographic Terms: Asia, Australia
DOI: 10.1093/reseval/rvz019
ISSN: 0958-2029
Abstract: Research-for-development (R4D), as a mode of foreign aid, is a practical way to support developing countries. However, few studies have assessed the scientific impacts of R4D projects. Here, we applied an integrated assessment approach to evaluate the scientific impact of research projects commissioned by the Australian Centre for International Agricultural Research's Fisheries Program in the Asia-Pacific region between 2000 and 2012. We use publication metrics and expert panel reviews to examine trends in scientific outputs among 73 fisheries and aquaculture research projects. Among projects, there was considerable variation in publication types and outputs (including academic papers, magazine articles, policy reports, books, and book chapters), and projects produced, on average, 10 publications at a funded-dollar cost of AU$41,000 per publication. Bilateral and multilateral projects tended to produce more refereed journal articles and citations per funded dollar than single-country projects, and publication outputs were poor for certain countries. With the exception of fisheries projects, larger (more highly funded) projects did not produce more journal articles per funded dollar than smaller projects. Project duration had no significant effect on the number of refereed publications, citation rates, or total citations per funded dollar across projects. Aquaculture projects had greater publication impact, per funded dollar than fisheries projects. Beyond the bibliometric measures, qualitative findings indicated that scientific impact was strongly influenced by motivations of project leaders and their institutions. We identified disparate impact performance among industries and countries. These findings could inform future overseas aid investments, policies and strategies. This study offers useful benchmarks for gauging scientific impacts of R4D programs and shows the value of using approaches that go beyond bibliometric measures.
Abstractor: As Provided
Entry Date: 2019
Accession Number: EJ1232171
Database: ERIC
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwGqEfEXgfoBtf9Mdv1SPuEuAAAA4zCB4AYJKoZIhvcNAQcGoIHSMIHPAgEAMIHJBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDCDiqcejja33UxlDOwIBEICBm2OaPgXDwrYBsw0JGT1p63IzbhWh9TA2Etl4oXMkIQw3sqkYdr7fKqlR3PkcBNAu6N_4Bw3ja6t8x6CFNUiX7mcQ-5PX2FZxoxve3UDQpA9fO5Tdqn01HymoludqaKgvNKAgn7nnz49NSgawLdsgXZ5KwXZDCbk3j_uWJ-HKskecgfSn7BMA1L9ICelhSFqd98RLc9QXsWHeSGKT
Text:
  Availability: 1
  Value: <anid>AN0139164413;p9101oct.19;2019Oct18.05:51;v2.2.500</anid> <title id="AN0139164413-1">Measuring scientific impact of fisheries and aquaculture research-for-development projects in South East Asia and the Pacific </title> <p>Research-for-development (R4D), as a mode of foreign aid, is a practical way to support developing countries. However, few studies have assessed the scientific impacts of R4D projects. Here, we applied an integrated assessment approach to evaluate the scientific impact of research projects commissioned by the Australian Centre for International Agricultural Research's Fisheries Program in the Asia-Pacific region between 2000 and 2012. We use publication metrics and expert panel reviews to examine trends in scientific outputs among 73 fisheries and aquaculture research projects. 'Among projects, there was considerable variation in publication types and outputs (including academic papers, magazine articles, policy reports, books, and book chapters), and projects produced, on average, 10 publications at a funded-dollar cost of AU$41,000 per publication. Bilateral and multilateral projects tended to produce more refereed journal articles and citations per funded dollar than single-country projects, and publication outputs were poor for certain countries. With the exception of fisheries projects, larger (more highly funded) projects did not produce more journal articles per funded dollar than smaller projects. Project duration had no significant effect on the number of refereed publications, citation rates, or total citations per funded dollar across projects. Aquaculture projects had greater publication impact, per funded dollar than fisheries projects. Beyond the bibliometric measures, qualitative findings indicated that scientific impact was strongly influenced by motivations of project leaders and their institutions. We identified disparate impact performance among industries and countries. These findings could inform future overseas aid investments, policies and strategies. This study offers useful benchmarks for gauging scientific impacts of R4D programs and shows the value of using approaches that go beyond bibliometric measures.</p> <p>Keywords: research for development; research evaluation; scientific impact; bibliometrics; Southeast Asia; Pacific Islands</p> <hd id="AN0139164413-2">1. Introduction</hd> <p>Research-for-Development (R4D) in developing countries constitutes a significant portion of foreign aid budgets of developed nations and philanthropic organizations ([<reflink idref="bib43" id="ref1">43</reflink>]). This is mainly due to the assumption that scientific research can contribute to economic growth, human capital development, the production of new products and technologies, and provide evidence for policy and practice in developing countries ([<reflink idref="bib34" id="ref2">34</reflink>]; [<reflink idref="bib18" id="ref3">18</reflink>]; [<reflink idref="bib14" id="ref4">14</reflink>]). However, the impact of foreign aid can be highly contextual ([<reflink idref="bib29" id="ref5">29</reflink>]). Thus, attention has turned to critically evaluating the outcomes from R4D organizations and identifying strategies that better direct foreign aid investment in research ([<reflink idref="bib24" id="ref6">24</reflink>]; [<reflink idref="bib14" id="ref7">14</reflink>]). Strategies by funding organizations on the best duration and size (i.e. funding quantum) of projects would benefit from evaluations across projects within industry types, yet few such comparative analyses exist.</p> <p>R4D impacts can take the form of (<reflink idref="bib1" id="ref8">1</reflink>) academic or scientific impact: knowledge and capacity building; and (<reflink idref="bib2" id="ref9">2</reflink>) development impact: policy and product development, social, economic, and livelihood impacts ([<reflink idref="bib18" id="ref10">18</reflink>]). Gauging the impact of government-funded research is integral to justifying foreign-aid funding and adapting funding strategies to maximize impact. While most studies are concerned with evaluating the economic impact of projects, scientific impact is more loosely defined ([<reflink idref="bib22" id="ref11">22</reflink>]; [<reflink idref="bib4" id="ref12">4</reflink>]). Nonetheless, there is increasing global interest in assessing the scientific impact of publicly funded research, i.e. the outcome of the knowledge produced as a result of research in technological innovation, and its impact on society, the economy and the environment ([<reflink idref="bib22" id="ref13">22</reflink>]; [<reflink idref="bib25" id="ref14">25</reflink>]; [<reflink idref="bib11" id="ref15">11</reflink>]; [<reflink idref="bib34" id="ref16">34</reflink>]; [<reflink idref="bib3" id="ref17">3</reflink>]; [<reflink idref="bib44" id="ref18">44</reflink>]; [<reflink idref="bib2" id="ref19">2</reflink>]; [<reflink idref="bib7" id="ref20">7</reflink>]).</p> <p>Hence, an impetus for this study was to understand trends in the scientific impact of publicly funded research conducted in developing countries. This is important for two reasons. On one hand funding countries seek to account for public funds and understand the academic and development impacts of foreign aid. On the other, as the implications for development can be profound, developing countries are becoming increasingly interested in understanding the contribution of R4D in developing their scientific capabilities ([<reflink idref="bib28" id="ref21">28</reflink>]). Here, we report the scientific impact of the Australian Centre for International Agricultural Research (ACIAR) Fisheries Program in the Asia-Pacific region between 2000 and 2012 through bibliometrics measures and assess capacity building impacts of the program through qualitative research. Our analysis does not include broader societal development impacts.</p> <hd id="AN0139164413-3">2. Publication evaluation</hd> <p>A principal issue in evaluating 'scientific impact' has been the lack of a widely accepted definition of the term itself. [<reflink idref="bib4" id="ref22">4</reflink>] concluded that scientific impact is a multi-dimensional construct that cannot be adequately understood or measured by any single indicator, and called for further analysis and discussion on the utility of various metrics of scientific impact. Most scientific research is intended to support, advance, or achieve goals that are extrinsic to science itself ([<reflink idref="bib35" id="ref23">35</reflink>]). Hence, when evaluating the scientific impact of large research programs integrated assessment models that incorporate both quantitative (e.g. bibliometrics) and qualitative (e.g. peer review) assessment methods are more useful than assessments based either on bibliometrics or peer review alone ([<reflink idref="bib11" id="ref24">11</reflink>]; [<reflink idref="bib44" id="ref25">44</reflink>]; [<reflink idref="bib7" id="ref26">7</reflink>]).</p> <p>Bibliometric indicators are traditionally considered as objective, reliable, and cost-effective measures of published research outputs, and are complementary to informed peer review (Braun, Glänzel Schubert 2006; [<reflink idref="bib4" id="ref27">4</reflink>]; [<reflink idref="bib11" id="ref28">11</reflink>]; [<reflink idref="bib3" id="ref29">3</reflink>]; [<reflink idref="bib42" id="ref30">42</reflink>]). Citation rate of published works is one of the most commonly used proxy measures of scientific impact (Braun, Glänzel Schubert 2006; [<reflink idref="bib46" id="ref31">46</reflink>]). While a large number of citation impact indicators exist, most are derivations of a few basic indicators, namely the total and average number of citations of the publications of a research unit ([<reflink idref="bib42" id="ref32">42</reflink>]). However, indicators based on counting highly cited publications are regarded as more robust than those based on total or average citation counts ([<reflink idref="bib46" id="ref33">46</reflink>]). This is because citation distributions tend to be highly skewed, and therefore the total or the average number of citations of a set of publications may be strongly influenced by one or a few highly cited publications ('outliers') ([<reflink idref="bib42" id="ref34">42</reflink>]).</p> <p>The three most important multidisciplinary databases to derive bibliometric data are Web of Science (WoS), Scopus, and Google Scholar (GS) ([<reflink idref="bib42" id="ref35">42</reflink>]). Both Scopus and WoS cover the mainstream English international scientific journals and are comparable in terms of citation data. Coverage of the natural sciences fields in GS was previously considered poor compared to WoS and Scopus, but that appears no longer to be the case ([<reflink idref="bib46" id="ref36">46</reflink>]; [<reflink idref="bib42" id="ref37">42</reflink>]). GS has the advantage of counting citations beyond ISI indexed journals, such as from international non-English journals and published reports and theses, and so can be considered to represent scientific impact both within and beyond academia ([<reflink idref="bib23" id="ref38">23</reflink>]). However, GS has less quality control over citation coverage ([<reflink idref="bib46" id="ref39">46</reflink>]; [<reflink idref="bib42" id="ref40">42</reflink>]) and requires more effort to collate bibliometric data than WoS or Scopus ([<reflink idref="bib30" id="ref41">30</reflink>]).</p> <p>Altmetrics, or 'webometrics', are alternative metrics to measure scientific impact from website usage data. Altmetrics have arisen from the online platforms on which scientific literature is now published, accessed and communicated through social media. Website usage rates provide a prediction of future citation rates ([<reflink idref="bib9" id="ref42">9</reflink>]). However, [<reflink idref="bib15" id="ref43">15</reflink>] found the correlations weak and surmised that altmetrics do not reflect the same kind of impact as citations. The consensus is that altmetrics cannot presently be used for formal research assessment ([<reflink idref="bib45" id="ref44">45</reflink>]; [<reflink idref="bib39" id="ref45">39</reflink>]).</p> <p>Assessment of the scientific impact of research or academic institutions using bibliometric indicators would usually occur by retrieving papers in which the name of the institution is found in the authors' addresses. However, R4D organizations that fund research would not usually be found in the authors' affiliations on published papers. Therefore, a publication database has to be manually constructed for each project or researcher who received funding from that R4D organization, as practised by [<reflink idref="bib11" id="ref46">11</reflink>] for a Canadian research funding organization.</p> <p>While bibliometric indicators are accepted measures for evaluating the output and outcome of a research program, they can be complemented by experts' opinions or peer review ([<reflink idref="bib37" id="ref47">37</reflink>]; [<reflink idref="bib3" id="ref48">3</reflink>]). [<reflink idref="bib37" id="ref49">37</reflink>], for example, prefers scientific impact to be evaluated by a panel of experts with an in-depth understanding of the particular field of research, arguing that bibliometric indicators are temporally biased in the way citation data is collected and calculated. On the other hand, [<reflink idref="bib3" id="ref50">3</reflink>] note that the assessments of scientific impact should not be based on peer review alone, citing potential 'conflicts of interest' and the challenge that peer review poses in establishing a uniform yardstick across disciplines or research areas. The assessment of the scientific impact of an organization by an appointed panel of experts involves making qualitative and quantitative assessments according to established rules ([<reflink idref="bib37" id="ref51">37</reflink>]: 1). However, this necessitates appointing experts with sufficient level of expertise and experience.</p> <hd id="AN0139164413-4">3. Measuring the scientific impact of R4D: a case study</hd> <p>R4D organizations strive to inform solutions for socio-economic problems and industry development through scientific research that could influence practice and policy. A formal assessment of the impact of projects funded by R4D organizations is necessary for accountability of the use of public funds, for benchmarking performance, to improve the design of future projects and to set performance improvement targets. The term R4D is used here to describe organizations that catalyze development in developing countries through brokering and funding research partnerships between scientists and their institutions.</p> <p>The ACIAR operates as an R4D organization under Australia's Foreign Aid Program. ACIAR's Fisheries Research Program, which covers aquaculture and capture fisheries, hereafter called 'the Program', aims to improve the productivity and sustainability of fisheries and aquatic farming systems in partner countries and Australia through international research partnerships. Key goals are to improve food security, improve livelihoods, and better manage natural resources. The geographic focus of the Program is Southeast Asia (Indonesia, Timor Leste, the Philippines, and Mekong countries), Pacific Island countries and Papua New Guinea (PNG). Projects within the Program are facilitated and funded by ACIAR. Research studies of varying duration and funding amount are coordinated through a variety of commissioned Australian research institutions (including universities, state departments, and other research providers) or International Agricultural Research Centres (IARCs).</p> <p>In the current study, we undertook a retrospective evaluation of the scientific impact of the Program from 2000 to 2012. This assessment complements an economics-focused evaluation of ACIAR projects published previously ([<reflink idref="bib19" id="ref52">19</reflink>]). Our primary objectives were to establish scientific impact benchmarks and to identify trends for improving the design of future projects, and in doing so, identify lessons learned to guide future project/program design, as distinct from auditing past performance. One of our guiding research questions was whether the duration or funding quantum to projects improved the efficiency of project outputs and measures of scientific impact—in essence—do longer, or larger projects give better 'bang for the buck' than shorter or smaller projects?</p> <p>While the debate continues to underlie the term 'scientific impact' ([<reflink idref="bib4" id="ref53">4</reflink>]), for the purposes of this article, we have adopted it to mean 'impact on the scientific community'. It comprises both the 'publication output and impact' (i.e. uptake of the Program's research by the scientific community) and 'academic impact' (i.e. formal scientific capacity development arising from the Program). We used an integrated assessment approach for impact assessment ([<reflink idref="bib10" id="ref54">10</reflink>]; [<reflink idref="bib33" id="ref55">33</reflink>]). This approach integrates results from publication metrics, expert judgement, document analysis, surveys, interviews, and case studies. This study describes the evaluation of scientific impact and presents approaches to measuring the scientific impact of R4D programs using the Program's projects in Southeast Asia and the Pacific countries as a case study.</p> <hd id="AN0139164413-5">4. Data and methods</hd> <p></p> <hd id="AN0139164413-6">4.1 Data collection methods</hd> <p></p> <hd id="AN0139164413-7">4.1.1 Publication database</hd> <p>Our analysis focused on the Program projects that commenced and concluded between the beginning of 2000 and the end of 2012. The census date for the inclusion of publication and citation data was 17 March 2016. We considered this as a reasonable citation window to allow sufficient time for most of the relevant data and findings to have been published and made available for our inclusion in the publication and citation analysis (see [<reflink idref="bib42" id="ref56">42</reflink>]). In total, 90 research projects across southeast Asia and the Pacific countries fell within this timeframe. However, our publication and citation analysis accounted for only 73 of the 90 projects[<reflink idref="bib1" id="ref57">1</reflink>] for which we were able to obtain data (see Supplementary Table 1). Collectively these 73 projects had received AU$30.4 million in ACIAR funding (See Supplementary Table 1). The number of projects per funding range for these projects is presented in Table 1. The remaining 17 projects with no data were similar to the 73 projects that produced publications, in terms of industry sector and country of research. However, the majority (13 out of 17 projects), were small, short-term research and development activities having an average project duration of slightly above 1 year and average funding of $67,000. These projects were excluded from the analysis as the majority were scoping studies with no forecasted publication outputs.</p> <p>The Program provided the initial publication database. To verify all types of publications and identify any additional publications for each project, we contacted respective research project leaders. The publication database was further populated using appropriate online search engines for the publications provided. Projects were categorized by funding amount, project leaders, industry sector, commissioned agency, and country/region. Project publications were identified as either a book, chapter in a scholarly book, an article in a scholarly peer-reviewed journal, conference publication (peer-reviewed or non-peer-reviewed), final report, technical report, internal report, magazine article, policy paper, or database.</p> <p>Table 1. Number of projects per project funding range</p> <p> <ephtml> <table><thead><tr><th>Project funding range (AU$) </th><th>Number of projects </th></tr></thead><tbody><tr><td>1,000–49,999 </td><td>9 </td></tr><tr><td>50,000–99,999 </td><td>10 </td></tr><tr><td>100,000–199,999 </td><td>13 </td></tr><tr><td>200,000–499,999 </td><td>20 </td></tr><tr><td>500,000–999,999 </td><td>14 </td></tr><tr><td>1,000,000–1,999,999 </td><td>7 </td></tr></tbody></table> </ephtml> </p> <p>Both the Elsevier's Scopus database and Thomson Reuters' WoS database were then searched for all scholarly peer-reviewed journal articles listed in the publications database. For articles indexed in the WoS, we recorded the 2014 5-year impact factor of the journal in which the article was published.</p> <p>To account for the capacity building outputs of the Program, we extracted information on project capacity building activities from project reports. These included educational tours and visits, mentoring activities, workshops and seminars, masterclasses, and fellowships and scholarships. The data were confirmed through interviews and survey with project leaders.</p> <hd id="AN0139164413-8">4.1.2 Surveys, interviews and case studies</hd> <p>In addition to creating a publications database, we conducted surveys, interviews and case studies to confirm the publication record of projects and to collect further information about scientific impact. Thirty-nine research project leaders, representing 60% of the project leaders between 2000 and the end of 2012, completed a questionnaire survey.</p> <p>We further contacted project leaders[<reflink idref="bib2" id="ref58">2</reflink>] of the 73 projects for interviews. Of these, 15 project leaders representing 35 of the 73 projects[<reflink idref="bib3" id="ref59">3</reflink>] who were willing and available for interviews were interviewed. Project leaders were asked about their view on the scientific and policy impact of their projects, the publication decision-making process and motivation to publish, and barriers within the publication process. We also interviewed 15 partner-country collaborating-institution leaders and researchers from five case study countries, namely: Indonesia, Lao PDR, PNG, Solomon Islands, and Vietnam. These interviews sought to understand the academic/scientific impact: capacity building and knowledge production impacts of the projects from the perspective of the partner country scientists and leaders. Also, five case studies of the Program projects across the five selected countries were carried out. These case studies were selected based on project diversity and representativeness. The case studies provided insights into the academic impact of the Program projects.</p> <hd id="AN0139164413-9">4.2 Peer review—the Expert Panel</hd> <p>An Expert Panel was appointed to review the Program's scientific impact. The Expert Panel was made up of three well-published and regarded experts (from Australia, the UK, and Canada) with extensive experience in managing R4D organizations with relevant national and international aquaculture and fisheries research programs. The Expert Panel was involved in assessing both the methodology—whether the publication metrics chosen to measure the publication output and impact of the Program were appropriate for this purpose—and the results obtained.</p> <hd id="AN0139164413-10">4.3 Data analysis</hd> <p>To assess the publication output and academic impact of the Program, we used standard publication metrics. These include publication output, citations per publication output, and a field-weighted citation impact (FWCI). Quantitative data analyzes were complemented by, and triangulated with, a qualitative analysis of the survey data and semi-structured interviews and findings of the Expert Panel ([<reflink idref="bib33" id="ref60">33</reflink>]). All qualitative data were manually coded by one researcher for consistency. Narrative and deliberative analyzes were used to extract emerging themes related to knowledge production and capacity building ([<reflink idref="bib10" id="ref61">10</reflink>]).</p> <p>We sought to compare the results of this study with other comparable research and development programs such as the Canadian International Development Agency, and the UK Department for International Development but could not find comparable data.</p> <p>For each project, we calculated the average number of refereed journal articles per AU$100,000 funded. We also measured the 'normalized citation rate' (NCR)—citations attracted by projects per year per funded dollar ([<reflink idref="bib42" id="ref62">42</reflink>]). The NCR was calculated as the number of citations across each project's total refereed publications, normalized to the recency of projects relative to the census date of data collection. The citation data included only the citations of Scopus-listed journal articles. Reported start date of each project was usually when contracts were signed, and the research activities usually commenced one year afterwards, so 1 year was subtracted from the number of years from project start dates for all projects for consistency. The NCR per AU$100,000 funded to each project was calculated as:</p> <p> <ephtml> <math xmlns="http://www.w3.org/1998/Math/MathML"><mi mathvariant="normal">NCR</mi><mo>=</mo><mi mathvariant="normal" /><mfrac><mrow><mrow><mo stretchy="false">∑</mo><mrow><msub><mrow><mi>C</mi></mrow><mrow><mi>n</mi></mrow></msub></mrow></mrow></mrow><mrow><mi>Y</mi><mo>-</mo><mn>1</mn></mrow></mfrac><mi mathvariant="normal" /><mo>÷</mo><mi mathvariant="normal" /><mfrac><mrow><mi>F</mi></mrow><mrow><mn>100</mn><mo>,</mo><mn>000</mn></mrow></mfrac></math> </ephtml> </p> <p>where,</p> <p> <emph>C<subs>n</subs></emph> = Total citations of all (n) refereed publications for a project at the census date</p> <p> <emph>Y</emph> = Years from the reported project start date to the census date</p> <p> <emph>F</emph> = Total project funding</p> <p>Projects were considered Bi/Multilateral if the research was conducted in more than one country. The number of Scopus citations across all refereed journal articles published from a project was counted only for projects producing refereed journal articles listed in Scopus.</p> <p>Average citation rates of refereed journal articles produced by projects reflect the 'quality' or 'impact' of the research within the scientific community. Average citation rates were calculated across refereed publications within each project by dividing the total citations at the census date by the number of years since each article was published.</p> <p>Relationships between the duration of R4D projects and impact metrics and the total funding for R4D projects and impact metrics were tested using non-linear regression modelling. The modelling was performed on data from Aquaculture, Fisheries, and Shared (Aquaculture/Fisheries) projects separately to avoid potential bias and error due to potential pre-existing relationships among the various types of projects. To avoid overfitting and spurious relationships, we tested the best fitting curves against one-parameter and two-parameter equation forms using DataFit software. The best curve by <emph>r</emph><sups>2</sups> value was chosen. Only statistically significant relationships were displayed in graphs. Confidence intervals to statistically significant relationships were fitted using SigmaPlot software.</p> <hd id="AN0139164413-11">5. Results</hd> <p></p> <hd id="AN0139164413-12">5.1 Publication metrics</hd> <p></p> <hd id="AN0139164413-13">5.1.1 Overall publication outputs of the ACIAR Fisheries Program</hd> <p>The 73 projects included in this study produced 734 publications during the reporting period, equating to an average output of ten publications per project. The most frequent type of publication (28%, n<emph></emph>=<emph></emph>208) was scholarly peer-reviewed journal articles (Figure 1). The next most common type was final reports (n<emph></emph>=<emph></emph>110; 15%) and non-peer-reviewed conference papers (n<emph></emph>=<emph></emph>86; 12%). The number of final reports (<reflink idref="bib110" id="ref63">110</reflink>) is more than the number of projects as some projects had several 'mini-projects' for which separate 'final reports' were submitted. 'Other' publications are outputs such as newsletters, brochures, bulletins, technical notes, or maps. With overall funding for the 73 projects amounting to AU$30.4 million, the 734 publications were produced at an average investment of about AU$41,000 per publication. Some of the publication types could be classed as non-scientific, such as policy papers and magazine articles. Many others, such as conference publications, theses, final reports and book chapters, could present scientific data and results such that we could not logically make a simple distinction between publications that were scientific or not.</p> <p>Graph: Figure 1. Publication output of the Program by publication type.</p> <p>Graph: Figure 2. Publication output distribution by project.</p> <p>The number of publications produced per project was highly variable (Figure 2). More than 90% of the publications came from half of the projects (n<emph></emph>=<emph></emph>37), with half of all publications coming from eight projects. One project, 'Improved hatchery and growout technology for marine finfish in the Asia-Pacific region' (FIS/2002/077), had the greatest output with 127 total publications (17% of all publications). These were: magazine articles (n<emph></emph>=<emph></emph>33), chapters in a scholarly book (n<emph></emph>=<emph></emph>29), conference publications—non-peer-reviewed (n<emph></emph>=<emph></emph>24), journal articles (n<emph></emph>=<emph></emph>17), and other publications <emph>(</emph>n<emph></emph>=<emph></emph>24).</p> <p>Graph: Figure 3. Number of scholarly peer-reviewed journal articles by average citation by year.</p> <hd id="AN0139164413-14">5.1.2 Peer-reviewed journal article outputs</hd> <p>The 73 projects for which publication data were available produced 208 articles in scholarly peer-reviewed journals during the reporting period. This equates to an average output of 2.8 journal articles per project, representing an average project funding of $146,000 per peer-reviewed scientific paper. In comparing this cost against other projects, one must bear in mind that many of the ACIAR projects employ researchers with the project funds, so this value does not simply reflect the direct costs of research. As in the case of total publication output for the Program, there was an uneven distribution of journal articles per project. Half of the projects (n<emph></emph>=<emph></emph>37) produced 98% of the journal articles (204 out of 208) during the census period. Nearly 45% of the projects (n<emph></emph>=<emph></emph>33) had not published any journal articles by the census date. Eight projects (10%) produced 56% of the total journal articles (116 of 208 journal articles) for the Program across the review period.</p> <hd id="AN0139164413-15">5.1.3 Citations per publication (journal articles)</hd> <p>The number of journal articles with citation information in Elsevier's Scopus database (<reflink idref="bib155" id="ref64">155</reflink>) was marginally higher than Thomson Reuters' WoS database (<reflink idref="bib152" id="ref65">152</reflink>). Around two-thirds of the journal articles were reported to have been cited less than twice per year by both Scopus (64%) and the WoS (68%) (Figure 3). Scopus reported 31 journal articles (from 14 projects) as being cited on average between three and 12 times per year. Out of those 31 articles, 12 were co-authored by one or more partner-country researcher, and partner-country researchers were the lead authors on seven articles.</p> <p>Graph: Figure 4. Mean number of Scopus citations per AU$100,000 funded to projects per year since projects started (blue bars, left-hand axis scale) and mean number of refereed journal articles per AU$100,000 funded to projects (yellow or grey bars, right-hand axis scale), with project data grouped by industry type. Error bars are standard errors of the mean. Sample sizes (n): Aquaculture = 41, Aquaculture/Fisheries Shared = 7, Fisheries = 25.</p> <hd id="AN0139164413-16">5.1.4 Field-weighted citation impact (journal articles)</hd> <p>FWCI takes into account the differences in research across disciplines and compares the number of citations of publications with the average of other similar publications indexed in the Scopus database ([<reflink idref="bib13" id="ref66">13</reflink>]). A score of 1.0 indicates that the publications have been cited at world average for similar publications, while greater than 1.0 indicates that the publications have been cited more than would be expected based on the world average for similar publications ([<reflink idref="bib13" id="ref67">13</reflink>]). FWCI metrics is useful to benchmark and provide a useful way to evaluate researchers' citation performance. Over three-quarters of the journal articles had FWCI of less than 1.0, or the article was not listed in the Scopus database at all. The journal article with the highest FWCI was a paper titled 'Vulnerability and resilience of remote rural communities to shocks and global changes: Empirical analysis from Solomon Islands' published in the journal <emph>Global Environmental Change</emph> ([<reflink idref="bib36" id="ref68">36</reflink>]). This article was an output of the project 'Improving resilience and adaptive capacity of fisheries-dependent communities in the Solomon Islands' (FIS/2007/116). The paper had FWCI of 5.3 and the most citations per year (12 citations per year) of any journal article in the publications database.</p> <hd id="AN0139164413-17">5.1.5 Scopus citations and refereed publications by industry type</hd> <p>Overall, the average number of journal articles published per AU$100,000 of ACIAR investment was much higher for Aquaculture projects than for either capture fisheries, termed 'Fisheries' hereafter, or Shared projects. Likewise, the average number of all Scopus citations per $100,000 of funding per year since project start was higher for Aquaculture (2.7) compared to Fisheries (0.6) and Shared projects (0.5) (Figure 4). However, as noted previously, project publication and citation data in this review are skewed by a number of relatively small projects. In particular, two relatively small Aquaculture projects (Projects FIS/2002/083 and FIS/2006/009, both in the AU$50,000–$99,000 funding range) dominated the publications and citations per AU$100,000 per year since project start for the Program overall. This is reflected in the large standard errors for Aquaculture projects.</p> <p>Graph: Figure 5. Mean number of Scopus citations per AU$100,000 funded to projects per year since projects started (blue or black bars, left-hand axis scale) and mean number of refereed journal articles per AU$100,000 funded to projects (yellow bars, right-hand axis scale), with project data grouped by country/region. Error bars are standard errors of the mean. Number of projects (n): Bi/Multilateral = 17, Indonesia = 15, Philippines = 6, Vietnam = 8, PNG = 11, Pacific Islands = 12, and Other (single) = 4. Countries in the "Other" category were Sri Lanka, India, and Cambodia.</p> <hd id="AN0139164413-18">5.1.6 Scopus citations and refereed publications by country</hd> <p>The number of refereed articles and corresponding citation rates per funded dollar were higher for bilateral and multilateral projects than those conducted in a single country (Figure 5). Moreover, three of the top five projects with a high number of citations were bi/multilateral. A few outlying projects had a strong influence on this significant variation. In particular, the small bilateral project FIS/2002/083 was a notable outlier to the rest of the Program projects, producing 16.4 journal articles and 31.5 citations per $100,000 per year since the project started compared to a mean of 1.6 journal articles and 3.4 citations per $100,000 per year since projects started across all bi/multilateral projects. Projects conducted solely in Vietnam also tended to yield relatively high numbers of refereed articles and citations per funded dollar. However, the data for both metrics were particularly variable for both bi/multilateral projects and projects in Vietnam, as indicated by large error bars (Figure 5).</p> <p>Graph: Figure 6. Scatter plots of total project funding versus scientific impact metrics: (a) number of refereed journal publications per AU$100,000 funded, (b) average citation rate of publications produced by projects (average number of Scopus citations per year), and (c) Scopus citations per AU$100,000 funded per year since projects started until the census date. Only the statistically significant relationships for any project type are shown. Solid lines are the fitted non-linear relationships; dashed lines are 95% confidence intervals. Line colours or the different shades correspond to the colour coding by project type given in the legend (green or grey = Aquaculture; red or black = Fisheries; light grey= Shared); refer to the electronic version of the article for colours.</p> <p>There was little difference between the numbers of refereed journal article citations per funded dollar per year since project start for projects conducted solely in Indonesia, PNG, Pacific Islands, and Other single-country projects (blue or black bars, Figure 5). Projects conducted solely in PNG or the Philippines tended to produce fewer refereed journal article publications per funded dollar, compared to the other regions/countries (yellow or grey bars, Figure 5). Despite the lower average productivity, it is worth noting that some multi-country (bi/multilateral) projects involving these other countries did produce many refereed articles and had relatively high citation rates.</p> <hd id="AN0139164413-19">5.1.7 Publication and citation numbers as a function of funding</hd> <p>Project publication and citation data were skewed by the relatively large number of journal articles produced from a few outstanding projects. The number of refereed journal articles produced by projects per funded dollar increased significantly for Fisheries projects funded by more than $600,000 (<emph>F</emph><subs>1, 23</subs> = 7.4, P<emph></emph>=<emph></emph>0.012) (Figure 6a). However, the level of funding to projects did not significantly influence the number of refereed articles produced by either Aquaculture projects or Shared projects (P<emph></emph>><emph></emph>0.05).</p> <p>Graph: Figure 7. Scatter plots of project duration vs scientific impact metrics: (a) number of refereed journal publications per AU$100,000 funded, (b) average citation rate of publications produced by projects (average number of Scopus citations per year), and (c) Scopus citations per AU$100,000 funded per year since projects started until the census date. Fitted relationships are not shown, as none were statistically significant for any project type.</p> <p>Our analysis indicated that average citation rates of refereed journal articles were not related to the level of project funding for neither Aquaculture, Fisheries, nor Shared projects (P<emph></emph>><emph></emph>0.05 for all three industry types) (Figure 6b). That is, projects with greater amounts of funding did not tend to produce refereed journal articles with greater or lesser impact than projects with modest funding.</p> <p>The number of citations of all refereed journal articles produced by projects per funded dollar, normalized by years since projects started, was significantly higher for projects with slighter funding for both Aquaculture (<emph>F</emph><subs>1, 22</subs>=4.9, P<emph></emph>=<emph></emph>0.04) and Shared projects (<emph>F</emph><subs>1, 4</subs>=13.9, P<emph></emph>=<emph></emph>0.02), but not for Fisheries projects (P<emph></emph>=<emph></emph>0.20) (Figure 6c). That is, Aquaculture and Shared projects with greater funding tended to produce articles that, on average, had lower 'impact' than projects with less funding. Notably, the (best) fitted curves were closely overlapping for these two industry types, showing high consistency in the relationship across the fitted values.</p> <hd id="AN0139164413-20">5.1.8 Publication and citation numbers as a function of project duration</hd> <p>Analysis of publication and Scopus citation numbers for Fisheries, Aquaculture, and Shared projects as a function of project duration (Figure 7a–c) revealed no significant relationships (P > 0.05 for all industry types on all three panels). Simply put, the duration of a project did not tend to significantly affect either the number of refereed publications per funded dollar, the average citation rate of refereed publications produced by projects, or the number of citations of all refereed articles produced by projects per funded dollar per year after project activities started.</p> <hd id="AN0139164413-21">5.2 Scientific impact—knowledge production and capacity building in partner countries</hd> <p>Inconsistent reporting of research capacity building activities and impacts in partner countries among projects limited our capacity to fully account for the scientific (research) impact of the Program ([<reflink idref="bib32" id="ref69">32</reflink>]). However, a synthesis of reported outputs from Program projects (Table 2) shows that, beyond publishing scientific outputs, projects invested considerable resources in capacity building activities. Many projects organized educational tours and visits to Australia or other countries, institutionalized mentoring processes, short training, and postgraduate programs.</p> <p>Table 2. Reported formal capacity development activities</p> <p> <ephtml> <table><thead><tr><th>Main activity </th><th>No. of people involved </th><th>Specific activities and processes </th><th>Who was involved </th><th>Notes </th></tr></thead><tbody><tr><td>Visits (educational tours and visits to Australia and other countries) </td><td>120 </td><td>Project initiation and planning, knowledge sharing, postgrad studies, staff exchange, placement, on-farm training, manuscript preparation, conferences, and workshops </td><td>Partner country researchers, officers, farmers; managers, staff, trainees, and government officials </td><td>Country visit's duration can be in the order of weeks or months </td></tr><tr><td>Mentoring </td><td>Not recorded </td><td>Information exchange through collaborations, scientific support, joint field trips, structured learning opportunities, organized task forces, institutional 'twinning', planning, and websites (platforms) </td><td>Mainly partner country researchers, officers, farmers, managers, staff, and trainees </td><td>Although extensive formal and informal mentoring was noted in projects documents, numbers were not reported </td></tr><tr><td>Workshops and seminars </td><td>2,500 </td><td>Various—too many to list </td><td>Partner country researchers, officers, farmers, managers, staff, trainees, and government officials. </td><td>Of the 215 workshops/seminars cited, only a quarter recorded numbers of participants. </td></tr><tr><td>Masterclass (training) </td><td>29 </td><td>Aquaculture Nutrition Master Class, Aquafeed Extrusion Masterclass </td><td>Partner country researchers, technical officers </td><td /></tr><tr><td>John Dillon Fellowship </td><td>10 </td><td>Various </td><td>Researchers and research managers </td><td /></tr><tr><td>John Allwright Fellowship (MSc and PhD) </td><td>28 MScs, 6 PhDs, 6 others </td><td>Various </td><td>Partner country researchers and research managers </td><td /></tr></tbody></table> </ephtml> </p> <p>The majority of the workshops were noted as having been structured as 'train-the-trainer' activities, to enable participants to provide subsequent training to others. At least 215 different workshops were mentioned in the project documents, with over 2,500 participants recorded. However, only about a quarter of reports noted the actual number of people who attend the events, suggesting that the actual number of participants may well be several thousand more. The most common outcomes of the training and workshops were considered to be (<reflink idref="bib1" id="ref70">1</reflink>) increased capability of project members and their institutions in research planning and execution; and (<reflink idref="bib2" id="ref71">2</reflink>) increased publication output, particularly the number of papers authored by local staff.</p> <p>With regards to short courses and postgraduate studies, at least 29 people attended master classes, with the majority in the field of aquaculture. Twenty-eight people undertook project-supported Masters-level courses, and six undertook PhD degrees. All these students attended Australian Universities, inferring a corresponding increase in the pool of highly trained scientists with whom Australian scientists could collaborate. For every Australian researcher involved in the Program research projects, there were at least three partner country researchers involved.</p> <p>All project leaders and partner-country collaborators interviewed during this study agreed that the Program contributed significantly to the scientific capacity development of partner-country researchers. Project leaders reported that the training and capacity development activities contributed to the improvement in experimental design and data analysis skills of researchers in partner countries, thereby positively influencing research output. Several project leaders also mentioned their time investments in mentoring partner country scientists and government officers through collaborative writing of scientific publications and reports. These forms of capacity building resulted in the increased international exposure of the work of partner country scientists. As one scientist from Indonesia put it, '... the Australians have helped us to become good scientists. That is very important, we work on the research and learn how to become a good scientist'.</p> <p>The case studies also indicated significant capacity-building impact. For example, the capacity building of staff at the Research Institute for Aquaculture No. 1 (RIA 1) supported the establishment of a state-of-the-art oyster hatchery research facility in Cat Ba, Vietnam in 2006. The capacity building of RIA 1 Vietnamese scientists included the training of seven aquaculture officers in Australian hatcheries, 25 undergraduate students and three MSc students were involved directly in the Program. In addition, three staff members received John Allwright Fellowships[<reflink idref="bib4" id="ref72">4</reflink>] to undertake PhD studies in Australia. The increased capabilities at RIA 1 through these capacity building activities has significantly enhanced the capacity of RIA 1 staff to pursue independent research and contribute to industry development.</p> <p>In Indonesia, the ACIAR Program has been working with the Institute for Mariculture Research and Development (IMRAD) through several projects and has contributed significantly to the development of 'grassroots' hatchery technology for milkfish and grouper. ACIAR projects contributed to the development of the capability of IMRAD as a research institute and in developing the capacity of its scientists. More than 100 hatchery operators and researchers from 22 countries received training. Indonesian project staff attest that the projects improved the capacity of partner country scientists to plan and conduct research, and improved scientists' confidence and capability to present at international conferences, which in turn, improved staff publication rates.</p> <p>The Program also provided capacity building outcomes for Australian researchers and Australian research institutions (Commissioned Agencies) by providing significant, varied and ongoing staff and student training, development, and publication opportunities. Nearly all the Australian researchers interviewed considered the Program to have played a major role in their career development and teaching practices. Australian researchers also reported that participation in these projects provided opportunities to learn about and be trained in areas that are of international significance, improving the broader research capacity of Australian science. From the perspectives of researchers involved, the Program clearly contributed to building and retaining high calibre researchers and scientific capability in Australia.</p> <hd id="AN0139164413-22">5.3 Factors influencing the publication agenda and project outputs</hd> <p>Qualitative analysis of the interview and survey data highlighted that three main factors influenced the publication agenda and outputs and involvement in the publication process: (<reflink idref="bib1" id="ref73">1</reflink>) project leaders' leadership and motivational ability; (<reflink idref="bib2" id="ref74">2</reflink>) institutional culture and drive of participating institutions (both Australian and partner agencies); and (<reflink idref="bib3" id="ref75">3</reflink>) individual drive of project leaders and partner agency researchers.</p> <hd id="AN0139164413-23">5.3.1 Project leaders' leadership and motivational ability</hd> <p>Individual project leaders were the primary drivers influencing both the publication agenda and outputs and the motivation to publish within project teams. In particular, the ability of a Project Leader to harness team members' aspirations and motivate the team was considered as an important factor in determining publication output. Project leaders who were keen to engage their team members in publishing had higher publication outputs. Project leaders considered that encouraging partner agency researchers to publish was an essential element of their role.</p> <hd id="AN0139164413-24">5.3.2 Institutional culture</hd> <p>A secondary driver of publication agenda and output was the publication culture of the Project Leader's employers. Institutional culture reportedly influenced the team members' drive to publish. Project leaders reported that there was significant variance between partner agencies regarding the perceived importance of publications and that this was largely influenced by institutional culture and related factors. However, it was noted that this variance was more pronounced at the start of projects and diminished as projects progressed (i.e. both the perceived importance of publications and the drive to publish increased over time). Substantial in-kind support by partner agencies and project partners also was reported to bolster publication success.</p> <hd id="AN0139164413-25">5.3.3 Personal motivation of project leaders and partner agency researchers</hd> <p>Although the data demonstrated that the above two factors were the main factors that played a significant role in publication agenda and output, the individual drive of project leaders and partner agency researchers was also important. Project leaders reported that publication records were highly regarded as evidence of capability and capacity of partner agency researchers.</p> <p>In addition to the above three factors, other, secondary, factors that were also reported to influence publication decision making and outputs included:</p> <p></p> <ulist> <item> The Program's flexibility in project planning, review and adaptive project management;</item> <p></p> <item> Individual researchers' and organizational relationships and credibility with a specific journal;</item> <p></p> <item> Previous publication records for partner agency researchers; and</item> <p></p> <item> Evolution within the global publication environment, including the growing reach of open access journals.</item> </ulist> <hd id="AN0139164413-26">6. Discussion</hd> <p></p> <hd id="AN0139164413-27">6.1 Measuring and benchmarking the scientific impact of an R4D organization</hd> <p>Our bibliometric analysis provides a point of departure for benchmarking R4D organizations in the absence of comparable data for other similar R4D organizations. The Expert Panel engaged for this project considered that, overall, the reported publication output and citation rate were relatively low, with a corresponding relatively high average investment per publication. However, caution is needed when comparing the number of 'all publications' across projects because publication value/impact will vary among publication types and the same research can be used in different types of publications. The Panel noted that it is important to consider the operating environment from which publications arise when comparing publication and citation data. The publications in this case study arise from projects in developing countries with facilities and field conditions incomparable to those of developmental research in developed countries. The Panel noted that ACIAR is both a development agency and a research organization and that measuring scientific impact of the Program based on its publication outputs alone was insufficient.</p> <p>Our findings showed that data on publication outputs and citation rates were skewed because of a few outstanding projects or publications. [<reflink idref="bib42" id="ref76">42</reflink>] noted that this effect, driven by a few relatively highly cited publications, is commonplace. Sub-field differences also influence citation rates, for example, taxonomic papers, of which 11 from this study were published in the journal <emph>Zootaxa</emph>, generally have low citations because once a species is described, it is rarely necessary subsequently to cite the original journal article. This is also reflected in the relatively low impact factor of that journal: 0.945. The use of citation metrics for the field of taxonomy has been considered as an impediment to biodiversity research (e.g. [<reflink idref="bib40" id="ref77">40</reflink>]). Thus, citation metrics should be used cautiously as a measure of scientific impact for R4D organizations.</p> <p>Another noteworthy point is the apparent value of collaborative research. The literature suggests that international collaboration and interdisciplinarity are positively related to increased publication and citation ([<reflink idref="bib5" id="ref78">5</reflink>]; [<reflink idref="bib41" id="ref79">41</reflink>]). This generality seems to hold true in the current study, which showed bi-lateral and multi-lateral projects on average tended to produce a larger number of journal articles and attracted more citations per funded dollar compared to projects conducted in a single partner country (Figure 5). The scope of this article did not allow us to analyze why this is the case. Nonetheless, we can say that multi-lateral projects are likely to have access to a larger pool of resources, and so the resulting publications could be more consequential. Further, publications arising from larger data sets across multiple countries tend to offer more generalizable lessons, which attract greater citations. As [<reflink idref="bib27" id="ref80">27</reflink>] suggested, R4D organizations should assess the costs of research partnerships and collaboration with partner countries and associated benefits to ensure that the desired outcomes of research are achieved and use of resources optimized.</p> <p>Analyzes of the correlations between the funding quantum to projects and overall publication output yielded mixed results. We found that well-funded Fisheries projects tended to produce more refereed publications per funded dollar than modestly funded projects, but funding had little effect on this measure of productivity for Aquaculture or Shared projects. This disparity could be due, in part, to the fact that fisheries are often geographically broad and projects require greater funding to gather publishable data than aquaculture projects that can require fewer resources to complete a publishable study. Projects with smaller amount of funds actually produced refereed articles with greater total number of citations per funded dollar in the Aquaculture and Shared sectors, but not so for Fisheries projects. In this case, smaller (i.e. less funded) aquaculture projects might already have physical resources (hatchery, tanks, and ponds) and a greater proportion of funding goes to research and data collection, whereas projects requiring greater infrastructure expenditure have larger budgets and so produce fewer publications per funded dollar. Importantly, the amount of funding to projects did not significantly affect the scientific 'impact' of project publications per funded dollar.</p> <p>Many World Bank project leaders believe that longer and greater-funded projects are more successful ([<reflink idref="bib26" id="ref81">26</reflink>]). However, empirical analysis contradicts this claim, although caution in interpretation is needed since, for example, 'difficult' projects might take longer to complete and are more likely to have poor impact ([<reflink idref="bib17" id="ref82">17</reflink>]). In our study, project duration did not significantly affect any of the three bibliometric measures used. While longer-term projects might build greater capital in data and technological advances compared to shorter projects, the publication quality and scientific impacts per funded dollar are clearly not enhanced per funded dollar. Thus, while there might be administrative benefits to funding agencies in commissioning longer projects, there is little evidence of improved scientific-output 'bang for the buck' over shorter projects.</p> <p>As we found, bibliometrics results explain only one part of the scientific impacts of R4D organizations (see [<reflink idref="bib31" id="ref83">31</reflink>]; [<reflink idref="bib14" id="ref84">14</reflink>]). Scientific impact also encompasses the creation of new knowledge, technologies, products, and services; and capacity building ([<reflink idref="bib16" id="ref85">16</reflink>]; SFC, HEFCE and HEFCW 2012). Therefore, measuring the scientific impact of R4D organizations should include an understanding of (<reflink idref="bib1" id="ref86">1</reflink>) the socio-cultural, political, and economic contexts under which the various research projects were carried out; (<reflink idref="bib2" id="ref87">2</reflink>) activities, including capacity building, carried out by the Program in the partner countries; and (<reflink idref="bib3" id="ref88">3</reflink>) the different actors involved in the research projects.</p> <hd id="AN0139164413-28">6.2 The context in partner countries, capacity building, project leaders' motivation and scie...</hd> <p>As discussed earlier, through the Program, ACIAR commissioned various Australian research institutions and IARCs to conduct research in Southeast Asia, Pacific Islands countries, and PNG. Hence, the Program operated in an environment that involved diverse people and projects within complex social, institutional, and political milieus. These complexities are likely to have influenced the overall scientific output and outcomes of projects in different countries ([<reflink idref="bib14" id="ref89">14</reflink>]).</p> <p>For example, our analyzes showed a marked variation in the publication output from projects based within different regions/countries. We found relatively large differences between the number of journal articles and citations per funded dollar among countries. Notably, projects based in PNG and the Philippines had a much lower journal article output and citations per funded dollar. This might be due to a complex set of factors including socio-cultural, political, and economic contexts and the institutional and personal capabilities in those countries. Nonetheless, the findings suggest that if publication metrics are considered an important yardstick for scientific impact, assessments of this nature could help to shape policy on target countries to maximize scientific impact and identify countries in which scientific publication training could be most influential.</p> <p>The evaluation revealed that a significant amount of financial and human resources were invested by the Program to build the capacity of researchers and their institutions in partner countries through educational tours and visits, mentoring, workshops and seminars, training, Masters, and PhD programs. The high frequency of joint-authorship with developing-country scientists and students accounts for further impact in the long term. In other R4D research fields, research is often led by developed-country researchers, and publication often follows a 'mentor-apprentice model', whereby developing-country researchers tend to be co-authors ([<reflink idref="bib6" id="ref90">6</reflink>]). In our study, the prevalence of peer-reviewed publications led by project leaders and scientists from developed countries (especially highly cited papers) reflects the fact that research organizations commissioned by ACIAR to lead projects were from Australia or other international agricultural research centres. Developing country researchers often lacked skills and/or found it difficult to allocate time to write and prepare manuscripts to a level required by high-tier journals. Nonetheless, co-authorship and mentoring constitute important capacity development for research institutions in partner countries. These capacity building activities are expected to have had a broader scientific impact by way of facilitating the creation of new knowledge, changes in the communities and their livelihoods, and changes in policy and practices, which are unlikely to be reflected in publication metrics of the Program's projects ([<reflink idref="bib14" id="ref91">14</reflink>]).</p> <p>Qualitative data indicated that the primary drivers of publication outputs were the motivation of the project leaders and their institutional culture. We anticipate that project leaders' interest in reinforcing their global recognition in the scientific community, combined with their desire to gain significant standing and recognition within their institutions, together influence their publishing behaviour. This suggests that a funding institution's policies and procedures might not exert much influence over the publication outputs of its projects. Improving the scientific impact of funded projects might need transparent and enforceable policies in this regard, including the development of policies that incentivize collaborative work and co-authorship. If the capacity development benefits for partner countries are to be sustained, there is a need for R4D agencies to encourage the development of policies and programs that facilitate co-authorship and collaboration between developed and developing country researchers. The argument posed in Section 5.1 that the socio-cultural, political and economic contexts of research need to be considered in measuring the scientific impact of R4D organizations is of relevance here as well. Therefore, we suggest that the scientific impact of R4D agencies, in contrast to purely academic institutions, must be measured against a broad suite of metrics.</p> <hd id="AN0139164413-29">6.3 Bibliometrics and the scientific impact of R4D organizations: some reflections</hd> <p>Earlier, we discussed the utility of formal publication metrics as a departure point in benchmarking R4D organizations. We also found that other factors (e.g. new knowledge, capacity development, and context) are important considerations, which traditional metrics may not necessarily capture. One argument posed in the literature (e.g. [<reflink idref="bib21" id="ref92">21</reflink>]), and by the Expert Panel, was that publication outputs, citations and impact factors are discipline-biased and context-dependent. Our earlier discussion about the field of taxonomy is a case in point. Bibliometrics, therefore, is not an all-encompassing indication of scientific impact. Unless projects are explicitly designed to focus on scientific research, and not policy or technology development outcomes, publication metrics can be inadequate.</p> <p>Publishing in high impact factor journals and attracting citations does not always guarantee high readership by the public, practitioners, and policy-makers, and thus, does not necessarily translate to high-impact policy and practice. Indeed, [<reflink idref="bib20" id="ref93">20</reflink>] argue that there is no significant relation between practical relevance and citation rates. [<reflink idref="bib12" id="ref94">12</reflink>] further concluded that, in some cases, non-mainstream journals with low impact factors might offer better opportunities for scientists in developing countries to disseminate knowledge to local communities. Others have also argued that the ranking of a journal might not necessarily be a good indicator of the practical relevance of scholarship ([<reflink idref="bib1" id="ref95">1</reflink>]). Papers published in highly ranked journals might be popular among, and frequently cited by, academics and researchers who speak with each other circuitously. However, such publications might be inaccessible to the general public, practitioners, and policy-makers—especially in developing countries where English is not the first language.</p> <p>With these arguments in mind, we contend that bibliometrics should be considered as 'a means', rather than 'an end', to the assessment of the scientific impact of R4D programs. Bibliometrics can indicate where scientific impacts fit and are progressing against defined benchmarks of relevance to the Program. However, such metrics are only one part of 'the aggregate effect of the changes in practice, products and policies by the final user groups' ([<reflink idref="bib16" id="ref96">16</reflink>]: 25).</p> <p>Bibliometrics offers a normative and objective indication of the scientific impact of R4D organizations; however, caution must be exercised when comparing across disciplines. For example, a FWCI of 0.75 might reflect a high impact R4D-funded publication from a small-island state but could be considered low for research within a developed country. Differentiation between research conducted in and for developing countries and research conducted in and for the benefit of a developed country is therefore essential.</p> <hd id="AN0139164413-30">7. Conclusion</hd> <p>This study had two principal objectives: to provide an assessment of the scientific impact of ACIAR's Fisheries Program between 2000 and 2012 as a departure point for benchmarking similar R4D organizations and to stimulate discussion among the broader R4D community regarding appropriate measures of scientific impact.</p> <p>With regard to the first objective, our methodology focussed on bibliometrics, including the number of peer-reviewed journal articles produced through Program projects and the number of citations these articles received. To critically analyze these data, we sought the opinions of an Expert Panel and gathered qualitative data on key drivers behind publication outputs. Our analysis found that bibliometrics offers useful initial measures to understand the scientific impact of an R4D Program and are often the metrics valued by funding agencies and employers of the researchers involved.</p> <p>Our data also suggested that, without specific policies, R4D Programs might have a little direct bearing on the publication output of funded projects. Key publication drivers were the capacity and motivation of the project leaders and the institutional culture and requirements of their employers. We also found that, in some cases, there were significant relationships between publication/citation metrics and the sector funded, the country in which the project was based and the total project funding. Projects involving multiple partner countries tended to have a higher scientific impact, but the outcomes were variable. Generally speaking, projects with greater funding or longer time frames did not produce greater 'bang for the buck'—inferring that small and modestly-funded projects were (roughly) as productive in scientific impacts per funded dollar. We can, therefore, conclude that R4D institutions require clear and enforceable policies and procedures about publication outputs if their funding programs are to be assessed and benchmarked upon bibliometric measures.</p> <p>Our analysis also suggests that the metrics used to evaluate the scientific impacts of R4D organizations need to be broader and more nuanced than those of the researchers and their respective commissioned academic institutions. Such measures should include aspects of capability development and recognition of the benefits of research to the R4D agency's home country. R4D organizations are also advised to engage with their key stakeholders to agree on the measures of importance, and how relevant data are to be collected and analyzed.</p> <hd id="AN0139164413-31">Acknowledgements</hd> <p>The authors wish to thank the ACIAR Fisheries Program and RDS Partners Pty Ltd for providing the funding and time required to conduct the evaluation and write this paper. The authors would also like to thank the Project Leaders and members of the Expert Panel who generously gave their time and knowledge during the review process. The authors would also like to thank the editor and the reviewers for providing such a valuable and constructive comments that have improved the quality of the paper.</p> <hd id="AN0139164413-32">Funding</hd> <p>This work was supported by the Australian Centre for International Agricultural Research [Project No. C2016/108 and FIS/2010/096 to S.W.P.].</p> <p> <emph>Conflict of interest statement.</emph> None declared.</p> <ref id="AN0139164413-33"> <title> Footnotes </title> <blist> <bibl id="bib1" idref="ref8" type="bt">1</bibl> <bibtext> As at census date of 17 March 2016, publication data had not been provided for 17 of the 90 projects.</bibtext> </blist> <blist> <bibl id="bib2" idref="ref9" type="bt">2</bibl> <bibtext> With the exception of a small number of project leaders who had retired, were non-contactable or had passed away.</bibtext> </blist> <blist> <bibl id="bib3" idref="ref17" type="bt">3</bibl> <bibtext> The project leaders interviewed were also project leaders on a further 15 ACIAR Fisheries projects that were not included in our publications analysis because they either (1) ended after the end of 2012 or (2) were ongoing at the time of the interview.</bibtext> </blist> <blist> <bibl id="bib4" idref="ref12" type="bt">4</bibl> <bibtext> The John Allwright Fellowship is an ACIAR Fellowships Scheme that provides the opportunity for partner-country scientists involved in ACIAR-supported collaborative research projects to obtain postgraduate qualifications at Australian tertiary institutions. The Scheme aimed at enhancing research capacity in ACIAR's partner country institutions.</bibtext> </blist> </ref> <ref id="AN0139164413-34"> <title> References </title> <blist> <bibtext> Adler N. J., Harzing A.-W. (2009) ' When Knowledge Wins: Transcending the Sense and Nonsense of Academic Rankings ', Academy of Management Learning & Education , 8 / 1 : 72 – 95. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Bartlett A. G. (2016) ' Evaluating Relative Success of Donor-Funded Collaborative Research Projects ', Research Evaluation , 25/ 4 : 405 – 15. # WorldCat</bibtext> </blist> <blist> <bibtext> Bertocchi G. et al. (2015) ' Bibliometric Evaluation vs Informed Peer Review: Evidence from Italy ', Research Policy , 44 / 2 : 451 – 66. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Bollen J. Van de Sompel, H., Hagberg, A., and Chute, R. (2009) ' A Principal Component Analysis of 39 Scientific Impact Measures ', PloS one , 4 / 6 : e6022. doi: 10.1371/journal.pone.0006022. Google Scholar Crossref Search ADS PubMed WorldCat</bibtext> </blist> <blist> <bibl id="bib5" idref="ref78" type="bt">5</bibl> <bibtext> Bornmann L., Daniel H.-D. (2005) ' Does the h-Index for Ranking of Scientists Really Work? ', Scientometrics , 65 / 3 : 391 – 2. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibl id="bib6" idref="ref90" type="bt">6</bibl> <bibtext> Boshoff N. (2009) ' Neo-Colonialism and Research Collaboration in Central Africa ', Scientometrics , 81 / 2 : 413 – 34. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibl id="bib7" idref="ref20" type="bt">7</bibl> <bibtext> Bozeman B., Youtie J. (2017) ' Socio-Economic Impacts and Public Value of Government-Funded Research: Lessons from Four US National Science Foundation Initiatives ', Research Policy , 46 / 8 : 1387 – 98. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibl id="bib8" type="bt">8</bibl> <bibtext> Braun T., Glänzel W., Schubert A. (2006) ' A Hirsch-Type Index for Journals ', Scientometrics , 69/ 1 : 169 – 73. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibl id="bib9" idref="ref42" type="bt">9</bibl> <bibtext> Brody T., Harnad S., Carr L. (2006) ' Earlier Web Usage Statistics as Predictors of Later Citation Impact ', Journal of the Association for Information Science and Technology , 57 / 8 : 1060 – 72. # WorldCat</bibtext> </blist> <blist> <bibtext> Bryman A. (2008) Social Research Methods , 3rd edn. New York : Oxford University Press. Google Preview # WorldCat COPAC</bibtext> </blist> <blist> <bibtext> Campbell D. et al. (2010) ' Bibliometrics as a Performance Measurement Tool for Research Evaluation: The Case of Research Funded by the National Cancer Institute of Canada ', American Journal of Evaluation , 31 / 1 : 66 – 83. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Chavarro D., Tang P., Ràfols I. (2017) ' Why Researchers Publish in Non-Mainstream Journals: Training, Knowledge Bridging, and Gap Filling ', Research Policy , 46 / 9 : 1666 – 80. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Colledge L. (2017) Snowball Metrics Recipe Book, 3rd edn. Amsterdam, the Netherlands: Snowball Metrics program partners < https://<ulink href="http://www.snowballmetrics.com/reports/">www.snowballmetrics.com/reports/</ulink> accessed 4 April 2019.</bibtext> </blist> <blist> <bibtext> Confraria H., Mira G. M., Wang L. (2017) ' Determinants of Citation Impact: A Comparative Analysis of the Global South versus the Global North ', Research Policy , 46 / 1 : 265 – 79. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Costas R., Zahedi Z., Wouters P. (2015) ' Do "Altmetrics" Correlate with Citations? Extensive Comparison of Altmetric Indicators with Citations from a Multidisciplinary Perspective ', Journal of the Association for Information Science and Technology , 66 / 10 : 2003 – 19. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Davis J. Gordon, J., Pearce, D., and Templeton, D. (2008) ' Guidelines for Assessing the Impacts of ACIAR's Research Activities ', ACIAR Impact Assessment Series No. 58 , 120pp. # WorldCat</bibtext> </blist> <blist> <bibtext> Denizer C., Kaufmann D., Kraay A. (2013) ' Good Countries or Good Projects? Macro and Micro Correlates of World Bank Project Performance ', Journal of Development Economics , 105 : 288 – 302. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> DFID (2014) What is the Evidence on the Impact of Research on International Development? DFID Literature Review. UK: Department for International Development.</bibtext> </blist> <blist> <bibtext> Farmer B. Duncan, R., Enright, T, and Jarvie, W. (2013) Independent Review of the Australian Centre for International Agricultural Research (ACIAR) , Canberra : Australian Centre for International Agricultural Research.</bibtext> </blist> <blist> <bibtext> Flickinger M. Tuschke, A., Gruber-Muecke, T., and Fiedler, M. (2014) ' In Search of Rigor, Relevance, and Legitimacy: What Drives the Impact of Publications? ', Journal of Business Economics , 84 / 1 : 99 – 128. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Garfield E. (1999) ' Journal Impact Factor: A Brief Review ', Canadian Medical Association Journal , 161 / 8 : 979 – 80. Google Scholar PubMed # WorldCat</bibtext> </blist> <blist> <bibtext> Godin B., Doré C. (2004) Measuring the Impacts of Science: Beyond the Economic Dimension, CSIIC Working Paper <<ulink href="http://sta.uwi.edu/rdifund/documents/GodinDoreImpacts.pdf">http://sta.uwi.edu/rdifund/documents/GodinDoreImpacts.pdf</ulink> accessed 3 July 2018.</bibtext> </blist> <blist> <bibtext> Harzing A.-W. K., Van der Wal R. (2008) ' Google Scholar as a New Source for Citation Analysis ', Ethics in Science and Environmental Politics , 8 / 1 : 61 – 73. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Herdt R. W. (2012) ' People, Institutions, and Technology: A Personal View of the Role of Foundations in International Agricultural Research and Development 1960–2010 ', Food Policy , 37 / 2 : 179 – 90. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Hicks D. Tomizawa, H., Saitoh, Y., and Kobayashi, S. (2004) ' Bibliometric Techniques in the Evaluation of Federally Funded Research in the United States ', Research Evaluation , 13 / 2 : 76 – 86. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Ika L. A., Diallo A., Thuillier D. (2012) ' Critical Success Factors for World Bank Projects: An Empirical Investigation ', International Journal of Project Management , 30 : 105 – 16. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Katz J. S., Martin B. R. (1997) ' What Is Research Collaboration? ', Research Policy , 26 / 1 : 1 – 18. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Mazzoleni R., Nelson R. R. (2005) The Roles of Research at Universities and Public Labs in Economic Catch-Up, LEM Working Paper Series, No. 2006/01, Scuola Superiore Sant'Anna, Laboratory of Economics and Management (LEM), Pisa.</bibtext> </blist> <blist> <bibtext> McGillivray M. Feeny, S., Hermes, N., and Lensink, R. (2006) ' Controversies over the Impact of Development Aid: It Works; It Doesn't; It Can, But That Depends ... ', Journal of International Development , 18 / 7 : 1031 – 50. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Meho L. I., Yang K. (2007) ' Impact of Data Sources on Citation Counts and Rankings of LIS Faculty: Web of Science versus Scopus and Google Scholar ', Journal of the Association for Information Science and Technology , 58 / 13 : 2105 – 25. # WorldCat</bibtext> </blist> <blist> <bibtext> Moed H. F. (2005) Citation Analysis in Research Evaluation. Information Science and Knowledge Management. Dordrecht, The Netherlands : Springer. Google Preview # WorldCat COPAC</bibtext> </blist> <blist> <bibtext> Mullen J., Gray D., Meyer J. (2015) 'Evaluating the impact of capacity building by ACIAR', The 59th AARES Annual Conference, February 10–13, 2015, Rotorua, New Zealand.</bibtext> </blist> <blist> <bibtext> Patton M. Q. (2015) Qualitative Research & Evaluation Methods: Integrating Theory and Practice , 4th edn. Thousand Oaks, CA : SAGE Publications. Google Preview # WorldCat COPAC</bibtext> </blist> <blist> <bibtext> Pearson S. Chudleigh, P., Simpson, S., and Schofield, N. (2012) ' Learning to Invest Better: Using Ex-Post Investment Analysis on Agri-Environmental Research and Development ', Research Evaluation , 21 / 2 : 136 – 51. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Sarewitz D., Pielke R. A. (2007) ' The Neglected Heart of Science Policy: Reconciling Supply of and Demand for Science ', Environmental Science & Policy , 10 / 1 : 5 – 16. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Schwarz A.-M. et al. (2011) ' Vulnerability and Resilience of Remote Rural Communities to Shocks and Global Changes: Empirical Analysis from Solomon Islands ', Global Environmental Change , 21 / 3 : 1128 – 40. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Seglen P. O. (1997) ' Why the Impact Factor of Journals Should Not Be Used for Evaluating Research ', British Medical Journal , 314 / 7079 : 498 – 513. Google Scholar Crossref Search ADS PubMed WorldCat</bibtext> </blist> <blist> <bibtext> SFC, HEFCE, and HEFCW (2012) Assessment Framework and Guidance on Submissions (REF 02.2011) <<ulink href="http://www.ref.ac.uk/2014/pubs/2011-02/">http://www.ref.ac.uk/2014/pubs/2011-02/</ulink>> accessed 20 July 2016.</bibtext> </blist> <blist> <bibtext> Thelwall M. Haustein, S., Larivière, V., and Sugimoto, C. R. (2013) ' Do Altmetrics Work? Twitter and Ten Other Social Web Services ', PloS One , 8 / 5 : e64841. doi: 10.1371/journal.pone.0064841. Google Scholar Crossref Search ADS PubMed WorldCat</bibtext> </blist> <blist> <bibtext> Valdecasas A. G., Castroviejo S., Marcus L. F. (2000) ' Reliance on the Citation Index Undermines the Study of Biodiversity ', Nature , 403 / 6771 : 698. Google Scholar Crossref Search ADS PubMed WorldCat</bibtext> </blist> <blist> <bibtext> Vogel R., Hattke F., Petersen J. (2017) ' Journal Rankings in Management and Business Studies: What Rules Do we Play by? ', Research Policy , 46 / 10 : 1707 – 22. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Waltman L. (2016) ' A Review of the Literature on Citation Impact Indicators ', Journal of Informetrics , 10 /2 : 365 – 91. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> White H. (1992) ' The Macroeconomic Impact of Development Aid: A Critical Survey ', The Journal of Development Studies , 28 / 2 : 163 – 240. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Wilsdon J. et al. (2015) The Metric Tide: Report of the Independent Review of the Role of Metrics in Research Assessment and Management. doi: 10.13140/RG.2.1.4929.1363. Google Preview # WorldCat COPAC</bibtext> </blist> <blist> <bibtext> Wouters P., Costas R. (2012) Users, Narcissism and Control: Tracking the Impact of Scholarly Publications in the 21st Century. Utrecht: SURFfoundation. Google Preview # WorldCat COPAC</bibtext> </blist> <blist> <bibtext> Wouters P. et al. (2015) The Metric Tide: Literature Review (Supplementary Report I to the Independent Review of the Role of Metrics in Research Assessment and Management. HEFCE. doi: 10.13140/RG.2.1.5066.3520. Google Preview # WorldCat COPAC</bibtext> </blist> </ref> <aug> <p>By Kiros Hiruy; Ray Murphy; Tom Lewis; William T White and Steven W Purcell</p> <p>Reported by Author; Author; Author; Author; Author</p> </aug> <nolink nlid="nl1" bibid="bib43" firstref="ref1"></nolink> <nolink nlid="nl2" bibid="bib34" firstref="ref2"></nolink> <nolink nlid="nl3" bibid="bib18" firstref="ref3"></nolink> <nolink nlid="nl4" bibid="bib14" firstref="ref4"></nolink> <nolink nlid="nl5" bibid="bib29" firstref="ref5"></nolink> <nolink nlid="nl6" bibid="bib24" firstref="ref6"></nolink> <nolink nlid="nl7" bibid="bib22" firstref="ref11"></nolink> <nolink nlid="nl8" bibid="bib25" firstref="ref14"></nolink> <nolink nlid="nl9" bibid="bib11" firstref="ref15"></nolink> <nolink nlid="nl10" bibid="bib44" firstref="ref18"></nolink> <nolink nlid="nl11" bibid="bib28" firstref="ref21"></nolink> <nolink nlid="nl12" bibid="bib35" firstref="ref23"></nolink> <nolink nlid="nl13" bibid="bib42" firstref="ref30"></nolink> <nolink nlid="nl14" bibid="bib46" firstref="ref31"></nolink> <nolink nlid="nl15" bibid="bib23" firstref="ref38"></nolink> <nolink nlid="nl16" bibid="bib30" firstref="ref41"></nolink> <nolink nlid="nl17" bibid="bib15" firstref="ref43"></nolink> <nolink nlid="nl18" bibid="bib45" firstref="ref44"></nolink> <nolink nlid="nl19" bibid="bib39" firstref="ref45"></nolink> <nolink nlid="nl20" bibid="bib37" firstref="ref47"></nolink> <nolink nlid="nl21" bibid="bib19" firstref="ref52"></nolink> <nolink nlid="nl22" bibid="bib10" firstref="ref54"></nolink> <nolink nlid="nl23" bibid="bib33" firstref="ref55"></nolink> <nolink nlid="nl24" bibid="bib110" firstref="ref63"></nolink> <nolink nlid="nl25" bibid="bib155" firstref="ref64"></nolink> <nolink nlid="nl26" bibid="bib152" firstref="ref65"></nolink> <nolink nlid="nl27" bibid="bib13" firstref="ref66"></nolink> <nolink nlid="nl28" bibid="bib36" firstref="ref68"></nolink> <nolink nlid="nl29" bibid="bib32" firstref="ref69"></nolink> <nolink nlid="nl30" bibid="bib40" firstref="ref77"></nolink> <nolink nlid="nl31" bibid="bib41" firstref="ref79"></nolink> <nolink nlid="nl32" bibid="bib27" firstref="ref80"></nolink> <nolink nlid="nl33" bibid="bib26" firstref="ref81"></nolink> <nolink nlid="nl34" bibid="bib17" firstref="ref82"></nolink> <nolink nlid="nl35" bibid="bib31" firstref="ref83"></nolink> <nolink nlid="nl36" bibid="bib16" firstref="ref85"></nolink> <nolink nlid="nl37" bibid="bib21" firstref="ref92"></nolink> <nolink nlid="nl38" bibid="bib20" firstref="ref93"></nolink> <nolink nlid="nl39" bibid="bib12" firstref="ref94"></nolink>
Header DbId: eric
DbLabel: ERIC
An: EJ1232171
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Measuring Scientific Impact of Fisheries and Aquaculture Research-for-Development Projects in South East Asia and the Pacific
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Hiruy%2C+Kiros%22">Hiruy, Kiros</searchLink><br /><searchLink fieldCode="AR" term="%22Murphy%2C+Ray%22">Murphy, Ray</searchLink><br /><searchLink fieldCode="AR" term="%22Lewis%2C+Tom%22">Lewis, Tom</searchLink><br /><searchLink fieldCode="AR" term="%22White%2C+William+T%2E%22">White, William T.</searchLink><br /><searchLink fieldCode="AR" term="%22Purcell%2C+Steven+W%2E%22">Purcell, Steven W.</searchLink> (ORCID <externalLink term="http://orcid.org/0000-0002-6227-6454">0000-0002-6227-6454</externalLink>)
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Research+Evaluation%22"><i>Research Evaluation</i></searchLink>. Oct 2019 28(4):313-325.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Oxford University Press. Great Clarendon Street, Oxford, OX2 6DP, UK. Tel: +44-1865-353907; Fax: +44-1865-353485; e-mail: jnls.cust.serv@oxfordjournals.org; Web site: http://applij.oxfordjournals.org/
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 13
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2019
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Animal+Husbandry%22">Animal Husbandry</searchLink><br /><searchLink fieldCode="DE" term="%22Agriculture%22">Agriculture</searchLink><br /><searchLink fieldCode="DE" term="%22Developing+Nations%22">Developing Nations</searchLink><br /><searchLink fieldCode="DE" term="%22Research+Projects%22">Research Projects</searchLink><br /><searchLink fieldCode="DE" term="%22Research+Utilization%22">Research Utilization</searchLink><br /><searchLink fieldCode="DE" term="%22Influences%22">Influences</searchLink><br /><searchLink fieldCode="DE" term="%22Foreign+Countries%22">Foreign Countries</searchLink><br /><searchLink fieldCode="DE" term="%22Scientific+Research%22">Scientific Research</searchLink><br /><searchLink fieldCode="DE" term="%22Publications%22">Publications</searchLink><br /><searchLink fieldCode="DE" term="%22Bibliometrics%22">Bibliometrics</searchLink>
– Name: Subject
  Label: Geographic Terms
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Asia%22">Asia</searchLink><br /><searchLink fieldCode="DE" term="%22Australia%22">Australia</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1093/reseval/rvz019
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 0958-2029
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Research-for-development (R4D), as a mode of foreign aid, is a practical way to support developing countries. However, few studies have assessed the scientific impacts of R4D projects. Here, we applied an integrated assessment approach to evaluate the scientific impact of research projects commissioned by the Australian Centre for International Agricultural Research's Fisheries Program in the Asia-Pacific region between 2000 and 2012. We use publication metrics and expert panel reviews to examine trends in scientific outputs among 73 fisheries and aquaculture research projects. Among projects, there was considerable variation in publication types and outputs (including academic papers, magazine articles, policy reports, books, and book chapters), and projects produced, on average, 10 publications at a funded-dollar cost of AU$41,000 per publication. Bilateral and multilateral projects tended to produce more refereed journal articles and citations per funded dollar than single-country projects, and publication outputs were poor for certain countries. With the exception of fisheries projects, larger (more highly funded) projects did not produce more journal articles per funded dollar than smaller projects. Project duration had no significant effect on the number of refereed publications, citation rates, or total citations per funded dollar across projects. Aquaculture projects had greater publication impact, per funded dollar than fisheries projects. Beyond the bibliometric measures, qualitative findings indicated that scientific impact was strongly influenced by motivations of project leaders and their institutions. We identified disparate impact performance among industries and countries. These findings could inform future overseas aid investments, policies and strategies. This study offers useful benchmarks for gauging scientific impacts of R4D programs and shows the value of using approaches that go beyond bibliometric measures.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2019
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1232171
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1232171
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1093/reseval/rvz019
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 13
        StartPage: 313
    Subjects:
      – SubjectFull: Animal Husbandry
        Type: general
      – SubjectFull: Agriculture
        Type: general
      – SubjectFull: Developing Nations
        Type: general
      – SubjectFull: Research Projects
        Type: general
      – SubjectFull: Research Utilization
        Type: general
      – SubjectFull: Influences
        Type: general
      – SubjectFull: Foreign Countries
        Type: general
      – SubjectFull: Scientific Research
        Type: general
      – SubjectFull: Publications
        Type: general
      – SubjectFull: Bibliometrics
        Type: general
      – SubjectFull: Asia
        Type: general
      – SubjectFull: Australia
        Type: general
    Titles:
      – TitleFull: Measuring Scientific Impact of Fisheries and Aquaculture Research-for-Development Projects in South East Asia and the Pacific
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Hiruy, Kiros
      – PersonEntity:
          Name:
            NameFull: Murphy, Ray
      – PersonEntity:
          Name:
            NameFull: Lewis, Tom
      – PersonEntity:
          Name:
            NameFull: White, William T.
      – PersonEntity:
          Name:
            NameFull: Purcell, Steven W.
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 10
              Type: published
              Y: 2019
          Identifiers:
            – Type: issn-print
              Value: 0958-2029
          Numbering:
            – Type: volume
              Value: 28
            – Type: issue
              Value: 4
          Titles:
            – TitleFull: Research Evaluation
              Type: main
ResultId 1