View in EDS HTML Full Text PDF Full Text

A Review of Literature on Evaluating the Scientific, Social and Political Impact of Social Sciences and Humanities Research

Saved in:
Bibliographic Details
Title: A Review of Literature on Evaluating the Scientific, Social and Political Impact of Social Sciences and Humanities Research
Language: English
Authors: Reale, Emanuela, Avramov, Dragana, Canhial, Kubra, Donovan, Claire, Flecha, Ramon, Holm, Poul, Larkin, Charles, Lepori, Benedetto, Mosoni-Fried, Judith, Oliver, Esther, Primeri, Emilia, Puigvert, Lidia, Scharnhorst, Andrea, Schubert, Andràs, Soler, Marta, Soòs, Sàndor, Sordé, Teresa, Travis, Charles, Van Horik, René
Source: Research Evaluation. Oct 2018 27(4):298-308.
Availability: Oxford University Press. Great Clarendon Street, Oxford, OX2 6DP, UK. Tel: +44-1865-353907; Fax: +44-1865-353485; e-mail: jnls.cust.serv@oxfordjournals.org; Web site: http://applij.oxfordjournals.org/
Peer Reviewed: Y
Page Count: 11
Publication Date: 2018
Document Type: Journal Articles
Reports - Research
Descriptors: Literature Reviews, Social Influences, Political Influences, Social Sciences, Humanities, Research, Foreign Countries
Geographic Terms: European Union
DOI: 10.1093/reseval/rvx025
ISSN: 0958-2029
Abstract: Recently, the need to contribute to the evaluation of the scientific, social, and political impact of Social Sciences and Humanities (SSH) research has become a demand of policy makers and society. The international scientific community has made significant advances that have transformed the impact of evaluation landscape. This article reviews the existing scientific knowledge on evaluation tools and techniques that are applied to assess the scientific impact of SSH research; the changing structure of social and political impacts of SSH research is investigated based on an overarching research question: to what extent do scholars attempt to apply methods, instruments, and approaches that take into account the distinctive features of SSH? The review also includes examples of European Union (EU) projects that demonstrate these impacts. This article culminates in a discussion of the development of the assessment of different impacts and identifies limitations, and areas and topics to explore in the future.
Abstractor: As Provided
Entry Date: 2019
Accession Number: EJ1208332
Database: ERIC
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwH1jKjpDFF1XkI0GHU3Gxf6AAAA4jCB3wYJKoZIhvcNAQcGoIHRMIHOAgEAMIHIBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDH5BoqnkcEnyzP5cWgIBEICBmixpgx8sp2ERAwz9-myS8kDh7QS48NHbE_BRX5dudVP_vb057s4nxmvQ-X79Y3VjvtzDiULLA-bFfRWjB0YxdqoTvyR_wyEeF0wLculbRwlK-s4TcpruYjoTEsb7FTm5GeMwif6bES8ohrEpqfF3JutfXD_rYo2y-71AkGMVq-EoXdu-_J47jeCGdRZdn-UuHKGmuiPB5-V4Lpo=
Text:
  Availability: 1
  Value: <anid>AN0133426128;p9101oct.18;2018Dec08.04:26;v2.2.500</anid> <title id="AN0133426128-1">A review of literature on evaluating the scientific, social and political impact of social sciences and humanities research </title> <p>Recently, the need to contribute to the evaluation of the scientific, social, and political impact of Social Sciences and Humanities (SSH) research has become a demand of policy makers and society. The international scientific community has made significant advances that have transformed the impact of evaluation landscape. This article reviews the existing scientific knowledge on evaluation tools and techniques that are applied to assess the scientific impact of SSH research; the changing structure of social and political impacts of SSH research is investigated based on an overarching research question: to what extent do scholars attempt to apply methods, instruments, and approaches that take into account the distinctive features of SSH? The review also includes examples of European Union (EU) projects that demonstrate these impacts. This article culminates in a discussion of the development of the assessment of different impacts and identifies limitations, and areas and topics to explore in the future.</p> <p>Keywords: social sciences; humanities; impact assessment; evaluation</p> <hd id="AN0133426128-2">1. Introduction</hd> <p>A key concern of contemporary research policies is to demonstrate the 'impact' of research, or the value that public investment in research generates for increasing scientific competitiveness and excellence of the country, wealth creation, productivity, and social well-being. Impact is often understood as a change that research outcomes produce upon academic activities, the economy, and society at large. However, speaking of 'attributable change' poses some problems, such as finding adequate tools and methods to measure impact, the time lag between the effect produced and the research activities that are supposed to have generated it, as well as the problem of disentangling the extent to which the research results were the sole or most significant causes of the effect produced.</p> <p>In addition to these well-known shortcomings, another problem is the different modes by which disciplinary fields are likely to impact academic communities, the economy, and society. This is especially true for Social Sciences and Humanities (SSH) because of their organizational and epistemic characteristics, and the type of outcomes that differentiate them from the Science, Technology, Engineering, and Mathematics (STEM) disciplines ([<reflink idref="bib120" id="ref1">120</reflink>];[<reflink idref="bib7" id="ref2">7</reflink>]).</p> <p>The research questions the article addresses are: How far does impact assessment in SSH attempt to apply methods, instruments and approaches that take into account the distinctive features of SSH? and: Are the dominant (STEM) perspectives of impact evaluation applicable to SSH research? We investigate these questions through the review of the literature, which sheds light on the state of the art of knowledge production in the field, and allows us to understand what is still missing in the analysis and therefore the relevant gaps that need to be addressed.</p> <p>In the review, we articulate the concept of impact by distinguishing between the different structures of scientific, social, and political impacts of SSH research. The assumption is that unpacking the concept of impact should allow us to improve knowledge of the different types of changes that are likely to be produced, and hence to assess them in a more robust manner.</p> <p>The article is organized as follows: Section 2 explains how we frame the systematic literature review to identify the key themes and dimensions emerging in SSH evaluation; Section 3 presents the method and data used for the review; Sections 4–6 present the review's findings and focus on three types of research impact: scientific, social, and political; Section 7 discusses advances in knowledge of each type of impact, connections between these impacts, and their limitations. Section 8 presents the article's conclusions, which centre on the need to advance knowledge of how to understand, measure, and assess SSH research impacts.</p> <hd id="AN0133426128-3">2. Framework and method of the literature review</hd> <p>This article aims to deepen our knowledge of specific features of assessing the impact of SSH research in three respects. First, it highlights the<emph>relationship between science and society</emph>and its bearing on research conduct and evaluation. In recent years, top-down political demand has led publicly funded research agencies to increasingly account for the scientific and broader societal impacts of the research that they support, which has in turn required researchers and research institutions to provide evidence of these impacts. This has been accompanied by bottom-up demands from academic researchers and research users (for example, within civil society) to articulate the value of research for society ([<reflink idref="bib8" id="ref3">8</reflink>]).</p> <p>Secondly, the review emphasizes the relevance within the literature of<emph>discipline-specific research evaluation</emph>. At national and cross-national levels, research evaluation and research policy tends to be designed from the perspective of the life sciences and the natural sciences, with SSH research commonly an afterthought ([<reflink idref="bib30" id="ref4">30</reflink>]). To bridge the gap between science and society, traditional research evaluation schemes are being reconsidered and reformed ([<reflink idref="bib4" id="ref5">4</reflink>]) in ways that make the scientific and public value of SSH research potentially more visible, and to this end non-standard research assessment tools have been developed and applied to evaluating the impact of SSH research.</p> <p>Thirdly, although a dedicated literature exists on research evaluation, other relevant pockets of literature are focused within an extensive range of SSH journals, as well as within books, chapters, and grey literature. This article reviews these<emph>diverse contributions</emph>to the scholarly literature on the evaluation of the scientific, social, and political impact of SSH research.</p> <hd id="AN0133426128-4">2.1 Motivations driving the investigation</hd> <p>As discussed above, policy makers justify the investment of public resources in R&D in terms of scientific advances that are likely to stimulate knowledge production (scientific impact). Wider impacts include the possibility of providing solutions to perennial policy problems (policy impact), and creating interventions to improve societal challenges (societal impact). Policy makers therefore want to understand (define, measure, and capture) these effects to be sure that they are using public funding to sustain 'good science'. However, the policy makers' perspective of 'good science' is not aligned with what scholars in all fields consider to be good science, since their main preoccupations are often with the robustness of the methods used, the reliability of tests and analysis, and the integrity of the research effort ([<reflink idref="bib55" id="ref6">55</reflink>]).</p> <p>This divide is even more important for SSH due to the distinctive features that differentiate it from natural sciences. For example, several authors characterize SSH according to organizational and institutional perspectives ([<reflink idref="bib120" id="ref7">120</reflink>]), the likelihood of SSH to be local or internationally oriented ([<reflink idref="bib46" id="ref8">46</reflink>]), and reflexivity and the appreciation of novel research efforts ([<reflink idref="bib55" id="ref9">55</reflink>];[<reflink idref="bib119" id="ref10">119</reflink>];[<reflink idref="bib89" id="ref11">89</reflink>]). Other recent pieces of work ([<reflink idref="bib13" id="ref12">13</reflink>];[<reflink idref="bib81" id="ref13">81</reflink>]) have pointed out that SSH is more interested in new approaches—which are the essence of originality, rather than in new theories or findings; SSH are reflexive and non-cumulative sciences, contrary to the normative and cumulative structure of natural sciences, so judgements on the value of research can vary depending on the existing different schools of thought ([<reflink idref="bib119" id="ref14">119</reflink>];[<reflink idref="bib90" id="ref15">90</reflink>]). These characteristics have distinct effects on impact assessment. Despite the fact that a distinctive feature of SSH scholarship is a commitment to developing research for the good of society, the interest of scholars is often not oriented towards producing 'usable' results ([<reflink idref="bib10" id="ref16">10</reflink>];[<reflink idref="bib9" id="ref17">9</reflink>]), but to influence and orient society, to maintain cultural heritage, and to create capabilities of self-understanding in different contexts ([<reflink idref="bib88" id="ref18">88</reflink>];[<reflink idref="bib89" id="ref19">89</reflink>];[<reflink idref="bib109" id="ref20">109</reflink>]). Thus, scholars highlight that SSH impact cannot be assessed as 'return on investment' (Weingart 2007). It therefore follows that looking for similarities and normative solutions to assess SSH research impact would not produce reliable results because it clashes with the internal diversity of the disciplines ([<reflink idref="bib71" id="ref21">71</reflink>]).</p> <hd id="AN0133426128-5">2.2 Distinguishing between impacts</hd> <p>The article addresses three different types of impact (scientific, social, and political) assuming that this distinction will allow a better understanding of how SSH research is likely to generate change in science and in society.</p> <p>Scientific impact is commonly defined as a change in research, which breaks the dominant paradigm and influences future research investigations. In fact, 'there is a distinction between "academic impact" that is understood as the intellectual contribution to a person's field of study within academia and the "external socioeconomic impact" beyond academia' ([<reflink idref="bib92" id="ref22">92</reflink>]: 21). However, in SSH the identification of 'dominant paradigms' is difficult due to the co-existence of competing paradigms and the difficulty in finding a common definition of what research quality means ([<reflink idref="bib90" id="ref23">90</reflink>]), and finding common criteria to assess it ([<reflink idref="bib55" id="ref24">55</reflink>]). Thus, scientific impact in SSH research is related to the capacity of founding new schools of thought and influencing future research in the field. It is not important for new schools to become dominant within the field: in SSH, different schools can co-exist; what is important is the fact that the new schools are producing research that follows a different approach.</p> <p>The conceptualization of the social impact of research remains an ongoing effort.[<reflink idref="bib42" id="ref25">42</reflink>]makes the distinction between scientific impact, dissemination, political impact and social impact, and argues that social impact can be understood as the culmination of the prior three stages of research. Therefore, the social impact of research occurs when published and disseminated results, which have been transferred into a policy or an NGO-led initiative, produce improvements in relation to the stated goals of society. However, a major problem in the literature is the lack of consensus on the meaning of the word 'social'. In some publications, a broad spectrum of social impact areas is listed: human rights, social cohesion, economic cohesion, employment, human capital formation, public health and safety, social protection and social services, liveable communities, culture, consumer interests, security, governance, international cooperation, role of SMEs, lessons learnt and success stories ([<reflink idref="bib39" id="ref26">39</reflink>]). At the other end of the social impact spectrum, the social impact domain is limited to a few items that pertain to the living conditions of people: welfare, well-being, and quality of life, customs and habits of life, i.e. consumption, work, sexuality, sports, and food ([<reflink idref="bib52" id="ref27">52</reflink>]). Sometimes the terms 'social' and 'societal' are interchangeable ([<reflink idref="bib16" id="ref28">16</reflink>]: 218); in other cases, a distinction is made.</p> <p>The assessment of the political impact of research as a separate from social impact has gained momentum in Europe, especially when investigating the relationships between science and policy and how to enhance the impact of the results of research on the policy process. Political impact of research occurs when knowledge is transferred, that is, when decision makers and/or social actors employ the published and disseminated results as the basis for their policies and/or actions ([<reflink idref="bib42" id="ref29">42</reflink>]). Although it significantly overlaps with the concept of social impact, its specific features relate to the fact that it addresses transformations that are produced in policy development and in the policy process (motivations and rationales, policy design, policy implementation, and policy assessment). For this reason, the article addresses political impact separately from social impact, taking into account both the macro-politics of dealing with large-scale decisions affecting the solution of complex problems, and policy related to micro-scale implementation of specific intervention techniques.</p> <hd id="AN0133426128-6">3. Method and data</hd> <p>The three types of impact the article addresses produce different orders of change within science and society. SSH research generates scientific impact when it influences the production of further research outputs following new approaches for analysis or based on new results. Changes related to social impact affect the cultural, economic, and social life of individuals, organizations, and institutions. Political impact incorporates the contents of research into political decisions, and motivations and rationales for political action and priority setting.</p> <p>In this review, we aim to understand the way in which the aforementioned changes occur, and how far new avenues, such as the importance of researchers engaging and interacting with society, were explored. To enable this, we use the analytical framework of critical communicative methodology (CCM) proposed by Gómez and colleagues (2011), which considers people as transformative social agents who are able to produce changes in the existing order. CCM considers that change from research is likely to occur when 'lifeworld is incorporated into the research process from the beginning to end' (p. 238). Two analytical dimensions characterize the CCM methodology, namely, the exclusionary one and the transformative one. The former refers to the barriers impeding non-academic individuals and groups from participating or from enjoining benefits; the latter dimension refers to the actions that help to overcome the barriers and produce a change in a given social situation ([<reflink idref="bib53" id="ref30">53</reflink>]). Thus the analysis presents evidence of what the literature on the impact of SSH research found to be factors that promoted or inhibited the successful scientific, social, and political impact of research.</p> <p>Because the fields investigated have very different types of research outputs, the article used the following sources for data collection:</p> <p></p> <ulist> <item> Journal articles: the literature search was conducted using the Web of Science (WoS) and SCOPUS databases;</item> <p></p> <item> Books, reports, and working papers;</item> <p></p> <item> CORDIS database: exploration of EU FP6 and EU FP7 projects (2006–12);</item> <p></p> <item> The EU FP7 Flash-it project[<reflink idref="bib1" id="ref31">1</reflink>]as a source for relevant research reports;</item> <p></p> <item> Web searches, e.g. Science Europe Association and other research centres and institutes in Europe and throughout the world;</item> <p></p> <item> Guidelines for applicants and evaluators, including searches of European Commission databases of funded projects;</item> <p></p> <item> Grey literature from relevant evaluation bodies and institutions.</item> </ulist> <p>A snowballing strategy was employed to identify additional sources. For instance, in cases in which the selected sources referred to specific projects, the search for data was extended to include project reports and other available online data.</p> <p>The review focused on eight disciplinary fields: Economics and Business; Educational Sciences, Media, and Communication; Humanities; Law; Life Sciences; Political Science; Psychology; and Sociology and Socio-economic Geography. Numerous combinations of keywords were employed as search terms to detect the impact of research; these terms were also applied to search key research evaluation journals. The analysis covered the period 2006–12 to coincide with the European Union's Sixth and Seventh Framework programmes. Some key pieces of literature published before 2006 and after 2012 were also included to describe the evolution of the concepts of and approaches to evaluating the impact of SSH research. To handle the extensive range of literature, different teams scanned the different disciplinary areas and independently identified important literature in this area. For these studies, the findings of the review were shared using standardized<emph>grids</emph>, which were presented as templates to annotate the studies.</p> <hd id="AN0133426128-7">3.1 Data analysis</hd> <p>A total of 288 grids were completed for the literature reviewed; two types of information were recorded: one dealing with approaches to the evaluation of the impact of SSH research, and one documenting examples of SSH research that had achieved impact (scientific, social, political). In this way we could, on the one hand, understand the new approaches and tools for evaluation methods and instruments the literature produced; and on the other hand, understand the capability of the observed approaches to properly asses the achievement of an impact.</p> <p>Several limitations should be acknowledged. First, using the WoS and Scopus database and the snowball strategy, journal papers account for 95% of the reviewed literature (233 of the 275 publications that were surveyed include articles that were published between 2009 and 2012). Secondly, scientific production in the social sciences tends to be more concentrated in journal articles, and journal articles comprise a significant part of the humanities and law; thus, the latter fields are less well-represented than other fields in this review. Thirdly, the capacity to trace research outputs which are neither publicly available nor have been cited by other publications is also limited. Last, the English-language output is overrepresented, and literature written in national languages (other than English) only appears in the case of studies that have received citations.</p> <p>The selected journal papers were located in 172 scholarly journals, which encompassed the entire range of SSH. These papers were not concentrated in particular core journals, with the exception of<emph>Research Evaluation</emph>, which provided 40 articles for review, followed by<emph>American Psychologist</emph>(<reflink idref="bib10" id="ref32">10</reflink>) and<emph>Scientometrics</emph>(<reflink idref="bib7" id="ref33">7</reflink>). The remaining literature included a small number of working and discussion papers (<reflink idref="bib5" id="ref34">5</reflink>), book chapters (<reflink idref="bib3" id="ref35">3</reflink>), a book (<reflink idref="bib1" id="ref36">1</reflink>), a doctoral dissertation (<reflink idref="bib1" id="ref37">1</reflink>), and 'hidden' pieces of literature in the form of EU FP6 or FP7 project documents (<reflink idref="bib3" id="ref38">3</reflink>).</p> <hd id="AN0133426128-8">4. Scientific impact of SSH research</hd> <p>The debate on scientific impact has a large place in the literature on SSH research; in the past 10 years, it was mainly pushed by the advent of national ex post research assessments, and by the importance that the use of metrics gained in impact evaluation. In this respect, the value of bibliometric indicators was highlighted for STEM disciplines as more objective and less costly than other methods, but the possibility of applying these types of indicators to SSH research was highly questioned.</p> <hd id="AN0133426128-9">4.1 Bibliometrics, scientometrics, informetrics and other metrics in STEM research</hd> <p>The scientific impact of STEM research on the policy agenda has existed since the emergence of 'Big Science' after the Second World War ([<reflink idref="bib96" id="ref39">96</reflink>]). The growth of science, the need to monitor (public) spending, and the recent shift to a knowledge-based economy caused the growth of the specialized scientific discipline of bibliometrics, scientometrics, and informetrics ([<reflink idref="bib25" id="ref40">25</reflink>]). The literature is dominated by quantitative, bibliometric approaches to assess the scientific impact of research, including the use of the journal impact factor (a measure that is based on the average number of citations to articles in a specific journal) and other citation-based methods ([<reflink idref="bib49" id="ref41">49</reflink>]). Traditionally the measurement of scientific impact focused on publications ([<reflink idref="bib105" id="ref42">105</reflink>]). Increasingly, we observe the tendency to focus on individual authors as the unit of analysis ([<reflink idref="bib122" id="ref43">122</reflink>]). A new indicator that has gained significant attention is the h-index, which was proposed by[<reflink idref="bib65" id="ref44">65</reflink>]. Tibor Braun and colleagues proposed to employ Hirsch-type indexes as useful complements to journal impact factors and to evaluate the scientific impact of research ([<reflink idref="bib18" id="ref45">18</reflink>]).</p> <p>Despite these efforts, scholars in STEM disciplines have always questioned the extent to which scientific impact, measured by numbers of citations, reflects the 'quality' or 'importance' of a single journal paper or a body of scientific work. New ideas in technological innovation and instruments tend to emerge at the boundaries of scientific fields led by atypical researchers, who may never achieve recognition from their academic peers ([<reflink idref="bib68" id="ref46">68</reflink>]). However, the fascination with numbers and simple data is still alive. Simple measures, such as the journal impact factor, which produce one number, are easy to apply and are attractive for many organizations that address evaluations of scientific impact. This simplicity belies the contested nature of the application of these measures ([<reflink idref="bib51" id="ref47">51</reflink>];[<reflink idref="bib77" id="ref48">77</reflink>]). Therefore, in scientometrics, a debate about the validity of different indicators and the continuous development of new indicators is underway. One example is the Scimago Journal Rank, which is a citation impact index that considers the relative prestige of journals that cite a particular journal paper ([<reflink idref="bib54" id="ref49">54</reflink>]). Thus, current metric-based evaluation practices are not without dispute, even for the natural sciences, among those that are subject to evaluation ([<reflink idref="bib11" id="ref50">11</reflink>]), and among those that develop metrics ([<reflink idref="bib64" id="ref51">64</reflink>]). Criticisms deal with computation of indicators and with the reliability of citations as a proxy of scientific impact.</p> <p>In the mid-1980s, changes in scholarly communication and practices, which are informed by open-access principles, gained attention from epistemic communities and research institutions, and new indicators were developed based on the Web: web indicators, webometrics, cybermetrics, and altmetrics ([<reflink idref="bib14" id="ref52">14</reflink>],[<reflink idref="bib15" id="ref53">15</reflink>]), which were supposed to complement citations to understand scientific impact of research outputs, attempting to measure the circulation and use of the research outputs within the scholars' community. One method for ordering webometrics is a timeline of their appearance that start from calculations about the use of the Web in scholarly communication: web indicators, webometrics, cybermetrics ([<reflink idref="bib104" id="ref54">104</reflink>]); Web 2.0—user-generated content and the emergence of altmetrics ([<reflink idref="bib97" id="ref55">97</reflink>];[<reflink idref="bib17" id="ref56">17</reflink>]); and the Semantic web—automatically generated impact stories (seehttps://impactstory.org/).[<reflink idref="bib2" id="ref57">2</reflink>]</p> <hd id="AN0133426128-10">4.2 The use of metrics for assessing scientific impact of SSH research</hd> <p>Do metrics for measuring scientific impact works in the case of SSH research? There are different answers to this question.</p> <p>Perspectives range from the idea that SSH disciplines are less scientifically developed and that existing metrics will become a better fit as they mature, to the viewpoint that the scientific impact of SSH research cannot be captured by blunt metrics and can only be assessed by peer review ([<reflink idref="bib31" id="ref58">31</reflink>];[<reflink idref="bib7" id="ref59">7</reflink>]).[<reflink idref="bib62" id="ref60">62</reflink>]noted the existence of four literature in social sciences: international journal articles, books, national journals, and non-scholarly publications, and despite the movement towards academics privileging the first type of publication, the importance of the other types of literature remained high.</p> <p>A recent review on evaluation practices indicated that several authors outline bibliometrics in SS as one resource among many for scientific impact assessment, which can provide better results when used in combination with other metrics and information sources than when it is used as a separate tool ([<reflink idref="bib29" id="ref61">29</reflink>]). This evidence raises the issue of selecting the most suitable mixed method for impact assessment. Conversely, bibliometrics do not emerge as an advantage for the assessment of the impact of humanities research. However, specific cases, such as psychology and linguistics, have a consolidated arena of relevant indexed journals in the fields.</p> <p>Notwithstanding the aforementioned evidence, the majority of discussions occupy a middle ground and are dependent on bibliometrics but seek alternative techniques that are more suited to the production and consumption of SSH research, overcoming the most important limitations, such as the different citation behaviour of SSH when compared with natural sciences ([<reflink idref="bib57" id="ref62">57</reflink>]), the different types of outputs ([<reflink idref="bib63" id="ref63">63</reflink>];[<reflink idref="bib114" id="ref64">114</reflink>]), the heterogeneity of the audience (scholars from international or national arena, non-scholarly audience) to which many research outputs are oriented ([<reflink idref="bib56" id="ref65">56</reflink>]) and the inadequate coverage of the existing data sets ([<reflink idref="bib2" id="ref66">2</reflink>];[<reflink idref="bib93" id="ref67">93</reflink>]). The issue of building a comprehensive SSH database that encompasses domestic publications has also been explored by scholars' literature ([<reflink idref="bib63" id="ref68">63</reflink>];[<reflink idref="bib108" id="ref69">108</reflink>]), and within the ESF Scoping Project, pointing out needs and recommendations to implement it ([<reflink idref="bib90" id="ref70">90</reflink>]). Other results show that the 'selectivity of journals in their choice of papers for publication' and 'journal diffusion' are sensitive and useful indicators to measure the impact of scholarly journals in the humanities ([<reflink idref="bib28" id="ref71">28</reflink>]).[<reflink idref="bib75" id="ref72">75</reflink>]employed a novel data gathering and analysis technique to map a heterogeneous social science field (communication studies) in a culturally, socio-political, and linguistically diverse country (Switzerland), which combined the use of data that were captured from CVs and WoS indicators.</p> <p>Some scholars from SSH have also recommended the use of alternative statistics, such as using Google Scholar, to capture citations that appear in both articles and books ([<reflink idref="bib67" id="ref73">67</reflink>];[<reflink idref="bib98" id="ref74">98</reflink>]). Others suggest that understanding the extent of the scientific impact of research, especially in SSH, may take many years due to long time-lags in expected citation patterns ([<reflink idref="bib97" id="ref75">97</reflink>]), thus the use of altmetrics and similar tools to create real-time inputs about how an article or a research report is being utilized could be useful (cut/paste activities, citations in media reports, online newspapers, peer-review discussions, and blogs). These webometric indicators, such as article usage data (HTML views and PDF downloads), should also be considered in the research evaluation process. However, other scholars have pointed out that the use of altmetrics in SSH produces the same advantages for other fields of science, having the same drawbacks and shortcomings as bibliometric data ([<reflink idref="bib82" id="ref76">82</reflink>]).</p> <p>In sum, notwithstanding the efforts to use WoS data and scientometric techniques to assess the scientific impact of SSH research and the transformation of some disciplinary fields, the analysis of the literature identified many shortcomings in the proposed methods and solutions, which indicate that impact evaluations based on bibliometric resources generally underestimate the value of the SSH research outputs ([<reflink idref="bib7" id="ref77">7</reflink>]). Alternative metrics, methods, and data sources are being increasingly explored to understand their potential as an alternative to bibliometrics for the scientific impact assessment of SSH research. Scholars therefore have expressed interest in the new developments to understand how far they can contribute to the long-term assessment of the impact in SSH ([<reflink idref="bib90" id="ref78">90</reflink>]), pointing out the need to engage with scholars in the fields examined to construct appropriate indicators ([<reflink idref="bib85" id="ref79">85</reflink>]; KNAW 2011).</p> <hd id="AN0133426128-11">5. Social impact of social sciences and humanities research</hd> <p>Despite the uncertainties related to properly defining social impact, there is general agreement in the literature that social impact is the change or the influence that research can have on society. The most debated problem is how this change takes place.</p> <p>The literature on STEM highlights a variety of processes that research outputs follow before they can produce an impact on society. For instance, in Australia's Research Quality Framework, definitions of research impact were co-produced with the research community and were defined as 'adding to the social, economic, natural, and cultural capital of the nation' ([<reflink idref="bib32" id="ref80">32</reflink>]: 54). Recent evidence coming from the 2014 Research Excellence Framework assessment of research impact in the UK show that social impact is defined as a change or a difference—positive or negative, produced by research ([<reflink idref="bib102" id="ref81">102</reflink>]). Weinberg and colleagues (2014) identified the effect of science funding on short-term economic activity. The authors concluded that scientific activity has an economic impact on society by identifying the number of people who are directly employed in the research and the products and goods purchased by scientific institutions. In this respect, a large consensus suggests that social/societal effects of (social) research may not only exhibit a positive nature ('benefits') but also exhibit a negative nature and have disadvantageous consequences.</p> <p>Theoretical advances in the conceptualization of social impact have affected evaluation methodologies and indicators. In our review, we find both ex ante evaluations of research projects concerning possible social impacts and ex post evaluations that monitor the impact of research ([<reflink idref="bib66" id="ref82">66</reflink>];[<reflink idref="bib16" id="ref83">16</reflink>]; Social Sciences and Humanities Scientific Committees 2013). Two major groups of methods are distinguished: qualitative methods (including peer review, case studies, and surveys) and quantitative methods (development and use of statistical indicators and, in certain fields, advanced mathematical models such as econometric models).[<reflink idref="bib3" id="ref84">3</reflink>]</p> <p>In SSH research, because shortcomings affect the use of indicators, successful practices for assessing impact are generally considered to be those that combine or integrate narratives with relevant qualitative and complementary quantitative indicators to grasp the multidimensional and contextual nature of complex societal phenomena ([<reflink idref="bib113" id="ref85">113</reflink>];[<reflink idref="bib47" id="ref86">47</reflink>];[<reflink idref="bib40" id="ref87">40</reflink>];[<reflink idref="bib106" id="ref88">106</reflink>];[<reflink idref="bib26" id="ref89">26</reflink>];[<reflink idref="bib33" id="ref90">33</reflink>];[<reflink idref="bib92" id="ref91">92</reflink>]). Assessment methods should focus on process rather than on results, which allows us 'to identify how relevant research is conducted and the processes by which it is applied, or not' ([<reflink idref="bib80" id="ref92">80</reflink>]).</p> <p>Bastow and colleagues (2014) discuss a range of types of impacts of social scientists via engagement with business, government, the third sector, and the public via the media. Where types of engagement can be identified, the authors conclude that an assessment of the impacts of these activities is difficult.[<reflink idref="bib72" id="ref93">72</reflink>]has sought to identify the types of impact of research based on the attitudes of academics towards impact, the nature of the interactions between researchers and users, and the processes of using research to inform policymaking. Literature on research utilization discusses a number of models of researcher–user interactions in SSH, which focus on the extent to which the research is led by the researcher, by the user, or based on an interactive process ([<reflink idref="bib1" id="ref94">1</reflink>]).</p> <p>Several specific, combined, or integrated social impact tools have been developed encompassing both social impact and political impact, whose purpose is to observe how the actors involved in the knowledge production (scholars, policy makers, beneficiaries, stakeholders) interact and communicate, and how far these interactions are likely to produce a change.Table 1summarizes few examples.</p> <p>Main social impact tools emerging in the literature review</p> <p> <ephtml> <table><thead><tr><th>Impact tools</th><th>Aim</th><th>Method</th><th>Literature</th></tr></thead><tbody><tr><td>Payback Framework</td><td>To represent the research process and paybacks at different stages</td><td>Using mixed methods case studies to gather the policy benefits from undertaking research between researchers and different actors, such as policy makers, stakeholders, and social movements</td><td><xref ref-type="bibr" rid="bibr20">Buxton and Hanney 1994</xref>,<xref ref-type="bibr" rid="bibr21">1996, 1998</xref>;<xref ref-type="bibr" rid="bibr121">Wooding et al. 2007</xref>;<xref ref-type="bibr" rid="bibr34">Donovan and Hanney 2011</xref>;<xref ref-type="bibr" rid="bibr60">Henshall 2011</xref>;<xref ref-type="bibr" rid="bibr92">Penfield et al. 2014</xref>;<xref ref-type="bibr" rid="bibr69">Klautzer et al. 2011</xref></td></tr><tr><td>SIAMPI Social Impact Assessment Methods for research funding instruments</td><td>To shed light on how social impact occurs and to develop methods to assess the impact</td><td>Using case studies to assess the productive interactions between researchers and stakeholders generating socially relevant applications</td><td>SIAMPI 2011;<xref ref-type="bibr" rid="bibr80">Molas-Gallart and Tang 2011</xref>;<xref ref-type="bibr" rid="bibr112">Spaapen and van Drooge 2011</xref>;<xref ref-type="bibr" rid="bibr92">Penfield et al. 2014</xref>;<xref ref-type="bibr" rid="bibr27">De Jong et al. 2014</xref></td></tr><tr><td>Successful actions</td><td>Identifying actions that have been scientifically proven to be successful in addressing social problems in any context in which they have been implemented</td><td>Checking actions based on results coming from research efforts that were successfully implemented thus generating efficiency and equity through participatory methods and techniques</td><td><xref ref-type="bibr" rid="bibr116">Valls and Padrós 2011</xref>;<xref ref-type="bibr" rid="bibr36">Elboj 2014</xref>;<xref ref-type="bibr" rid="bibr45">Fletcha, 2015</xref></td></tr><tr><td>Social Impact Open Repository SIOIR</td><td>Providing an open access repository to display, share, and store the social impact of research results</td><td>Calculating a social impact score using the evidence provided by scholars about the changes their works concretely produce in society</td><td><xref ref-type="bibr" rid="bibr44">Flecha, Soler-Gallart and Sordé 2015</xref></td></tr><tr><td>Agora Model</td><td>Making indicators as living documents to support science and society interactions</td><td>Multi-actor interaction to improve science and society relationships engaging scholars and stakeholders in open debates</td><td><xref ref-type="bibr" rid="bibr6">Barré 2001</xref></td></tr><tr><td>Opportunity approach</td><td>To assess the consistency between policy design, policy implementation, and actors' choices</td><td>Analysing opportunities that are intended, provided, perceived, and mobilized by policy actors and beneficiaries of research programmes using case studies</td><td><xref ref-type="bibr" rid="bibr100">Reale et al. 2014</xref>;<xref ref-type="bibr" rid="bibr86">Nedeva et al. 2012</xref></td></tr></tbody></table> </ephtml> </p> <p>The tools inTable 1clearly demonstrate efforts to make central the consideration of differences existing between disciplinary fields and the specificities of SSH research. In this respect, they represent a step forward in understanding hidden transformations produced within science and society—either positive or negative—during the research process and after research outputs are produced, in delivering impact, and disentangling the effects of the process of knowledge production, rather than only concentrating on impact in relation to the final outputs. Moreover, both the exclusionary and transformative dimensions of CCM (see Section 3) are likely to be included in the impact assessment.</p> <hd id="AN0133426128-12">6. Political impact of social sciences and humanities research</hd> <p>The debate about the political impact of SSH research is held not only in social and academic fora but also as part of the political research agenda ([<reflink idref="bib79" id="ref95">79</reflink>];[<reflink idref="bib74" id="ref96">74</reflink>]). In this review, we discuss the identified venues by which political impact is enhanced, which involve how researchers perform and communicate their research findings and how policy makers use scientific knowledge to inform their decisions. Among these mechanisms, ways in which these two worlds interact, how research agendas are policy-oriented, and the processes of co-creation are included.</p> <hd id="AN0133426128-13">6.1 Relationships between science and policy</hd> <p>A body of literature has been dedicated to the study of the relationship between research and politics.[<reflink idref="bib12" id="ref97">12</reflink>]have highlighted the need for changes in traditional research assessments by creating mechanisms that can be applied to identify how research generates findings that can be usefully reported to inform politics and practice. We also need to learn how policy makers use evidence from the social sciences in their practices to address social problems. According to[<reflink idref="bib103" id="ref98">103</reflink>], better contexts can be constructed and are needed to enhance an appropriate process for policymaking. The literature also highlights that political impact is difficult to attribute to a specific research project if both researchers and players outside the research system do not participate in impact assessment (Rymer 2011). They can achieve this by producing evidence briefings that are based on systematic reviews ([<reflink idref="bib23" id="ref99">23</reflink>]) or by participating in advisory committees on legal practice and policy ([<reflink idref="bib26" id="ref100">26</reflink>]).</p> <p>To understand the processes and actors behind successful policymaking that applies evidence from scientific research, some authors have emphasized exploring 'productive interactions' (see Section 5). Similarly, the role of different stakeholders in research has been the focus of many studies that aim to describe the most effective processes in translating evidence into political impact while considering occasional or structured partnerships among stakeholders in the scientific research process ([<reflink idref="bib26" id="ref101">26</reflink>];[<reflink idref="bib117" id="ref102">117</reflink>]).</p> <p>In recent years, the number of problem-oriented or policy-oriented research calls has increased in Europe. Scholars often remark that traditional indicators do not sufficiently measure policy-relevant effects when assessing the political impact of this type of research as they relate to traditional modes of knowledge production, known as Mode 1 ([<reflink idref="bib50" id="ref103">50</reflink>]). They do not consider how research contributes to ameliorating the problems that societies face or how policymakers use evidence.[<reflink idref="bib37" id="ref104">37</reflink>]conceptualized this type of policy-oriented research as Mode 2 research, i.e. transdisciplinary collaborative modes of knowledge production that is oriented towards policy and society, and highlighted the need to construct new<emph>indicators</emph>—Mode 2 indicators—to better monitor the research impact.[<reflink idref="bib4" id="ref105">4</reflink>]</p> <hd id="AN0133426128-14">6.2 Co-production of research and research impacts</hd> <p>Although the main focus of the literature is on barriers to achieve political impact, some authors also point out how the co-production of research between academics and policymakers can facilitate research impact.[<reflink idref="bib35" id="ref106">35</reflink>]investigated the co-production of research between academics and policymakers, particularly in terms of negotiating complex governance processes. They believed that 'If public managers and policy-makers become more reflective and researchers more action-oriented, they can meet in joint enquiry' ([<reflink idref="bib35" id="ref107">35</reflink>]: 228); thus, they championed the idea of a 'community of inquiry' located 'in the middle between science and practice' and where social scientists and practitioners can 'co-produce knowledge to cope with practical challenges' ([<reflink idref="bib35" id="ref108">35</reflink>]: 230–32), a condition that is supposed to overcome the potential negative effect of policy makers that seek to reject research that does not fit with their preconceived needs. O'Hare, Coaffee and Hawkesworth (2010) highlighted the negotiated context of co-produced research and introduced the idea of academics and practitioners working together as 'critical friends' to 'negotiate clear independence' (O'Hare, Coaffee and Hawkesworth 2010: 246).</p> <p>[<reflink idref="bib24" id="ref109">24</reflink>]assessed the benefits of co-produced research with local government as a research partner, using randomized control trials. They reported a series of local-level experiments that are co-produced with policymakers and public service providers, including (<reflink idref="bib1" id="ref110">1</reflink>) to evaluate the promotion of household recycling participation in 6,580 households; and (<reflink idref="bib2" id="ref111">2</reflink>) to evaluate the impact of school-based education on the environmental attitudes and behaviours of 715 primary school students and their families in 27 primary schools. In terms of co-production, the authors concluded that 'partners want to have equal say over the research methodology, and negotiations cover both the research and the intervention' and as the 'collective nature of civic interventions can militate against individual randomization' researchers can encounter 'ethical and moral objections from principled public service practitioners' ([<reflink idref="bib24" id="ref112">24</reflink>]: 161).</p> <p>From the perspective of research management from within the UK's Economic and Social Research Council (ESRC),[<reflink idref="bib3" id="ref113">3</reflink>]suggest that both<emph>ex ante</emph>evaluation criteria and<emph>ex post</emph>evaluation criteria should rely on a conceptual model that includes 'the crucial role of co-production in achieving impact' as the ESRC's impact evaluation work has demonstrated that 'sustained involvement of [non-academic research] users is one of the most important determinants of policy impact' ([<reflink idref="bib3" id="ref114">3</reflink>]: 209–10). They argue that the effective co-production of research entails that research end users should be 'involved throughout the research process, from agenda-setting, through design, fieldwork and communication of outcomes' ([<reflink idref="bib3" id="ref115">3</reflink>]: 209).</p> <p>However, less attention has been placed on studying how these partnerships include the voices of the most vulnerable end users throughout the entire research process. Some articles provide evidence of these successful actions, which are designed via dialogue among scientists, civil society, and policy makers ([<reflink idref="bib43" id="ref116">43</reflink>]). When these successful actions address targets in the political agenda and provide evidence of overcoming inequality (for example, by reducing school dropout or creating sustainable employment), transfer into policy tends to be smooth and attributable to a specific research project. In some examples, such as the case of the inclusion of ethnic minorities, these partnerships can shape the practice of policymaking in discussing and evaluating action plans for social inclusion ([<reflink idref="bib84" id="ref117">84</reflink>]).</p> <p>However, a clear gap emerged in the analysis, which refers to the lack of investigation of the possible negative impacts of engagement on research agendas, and the lack of willingness of policy makers to use evidence of impact to become more reflexive.</p> <hd id="AN0133426128-15">7. Discussion</hd> <p>We now summarize the main findings of the literature review presented in the article, especially the exclusionary and transformative dimensions of CCM (see Section 3), and what inhibits or allows for various types of research impact to occur.</p> <p>Regarding scientific impact in SSH, scholars focused on bibliometric analyses, indicators, and tools that are related to publications, to understand the extent to which these can be applied to SSH research; the limitations that affect metrics for SSH are not yet solved and a clear knowledge gap still exists. The gap also refers to the strong orientation of the scholars' efforts towards considering scientific impact as a change produced by a single (or a combination of) piece(s) of research, with a limited interest in deepening conditions of the research processes contributing to generating an impact in the interested fields ([<reflink idref="bib83" id="ref118">83</reflink>]).</p> <p>Generally speaking, it is recognized that changes in scholarly communication are likely to transform and improve our capability to understand the scientific impact of research outputs and surpass the simple paper publication. Although new digital traces will inspire the search for new automatic metrics, understanding of the limits and possible drawbacks of metrics-only approaches in SSH is increasing. In the scientific discourse, we strive for a subtler use of indicators and its combination with qualitative methods of evaluation. The latter extends from traditional peer review to tracing narratives and success stories that are both supported by automatic means. If such an effect exists, then the role of time and timescales in deploying aspects of the impact should be considered.</p> <p>In both political and social impact of SSH research and, to some extent, scientific impact, we observed an increasing trend towards responding to the demand to create new opportunities for participation and public engagement of researchers and stakeholders. Creating shared dialogical spaces and promoting processes of co-production of research between academics and policy makers are assessed as promising practices that are likely to create greater political impact, and in some cases (not all), social impact.[<reflink idref="bib5" id="ref119">5</reflink>]</p> <p>Despite significant divergence, some common elements are recognized in the reviewed literature. In terms of conceptualization, political impact refers to the transfer of research findings to the political sphere to inform decision-making or policy design, and social impact refers to the extent to which an action from a policy or a civil society-led action has actually contributed to improve identified social challenges.</p> <p>Another important aspect is whether researchers generate interventions based on research findings and provide evidence on resulting social improvements, or whether researchers identify actions that have a positive impact on society and analyse their features to create possibilities for transferability. Two different perspectives emerged in the review regarding impact assessment. The first is building indicators and metrics to 'measure' impact, and assumes impact as a magnitude of forward progress. The second assesses the extent to which conditions for an impact to occur have been created and mobilized, and assumes that impact is an emerging property, which depends on later choices and events beyond the scholars' immediate control. The literature shows that both avenues are explored as far as SSH research is concerned, but the latter gained more interest and consensus that the former.</p> <p>Regarding problems of attributing and identifying the political and social impacts of particular pieces of research, the literature review has demonstrated the need for additional improvements in methods, techniques, metrics, and methodologies to better grasp the impact of research in SSH fields. Despite the technical limitations, we have identified a transformation of the scientific community in being increasingly aware of not only the crucial importance of achieving these types of impacts but also their responsibility to gather evidence and information to support impact claims. For instance, evidence emerged in the literature on the contribution and impact of SSH research on policies related to social problems, such as social exclusion, gender discrimination, and other relevant social challenges ([<reflink idref="bib73" id="ref120">73</reflink>];[<reflink idref="bib99" id="ref121">99</reflink>]; Sordé-Martì et al. 2014), demonstrating possibilities for such data collection.</p> <p>Last but not least, the analysis found that SS research was well represented in the specialized literature on political and social impacts. This finding applies to the humanities in more recent time, a delay that may be attributed to the characteristics of the epistemic communities that are included within the humanities, and are traditionally less focused on demonstrating an 'impact' to external stakeholders ([<reflink idref="bib89" id="ref122">89</reflink>]). There are also signals that the interest in SSH research on impact assessment is likely to improve in the future. For instance, the recent HERAVALUE project under the FP7 noted that 'there is evidence of a genuine commitment to A&HR, and that policymakers and the academy are inching towards a common language' ([<reflink idref="bib59" id="ref123">59</reflink>]: 27), and the more recent literature demonstrates important efforts of scholars to critically engage with the problem of impact in SSH research ([<reflink idref="bib90" id="ref124">90</reflink>];[<reflink idref="bib76" id="ref125">76</reflink>]).</p> <hd id="AN0133426128-16">8. Conclusion</hd> <p>The main purpose of this article was to shed light on how scholars have dealt with the problem of assessing the impact of SSH research, and whether impact assessment approaches and techniques developed for STEM disciplines can legitimately be used for SSH.</p> <p>The analysis shows that scientific impact has gained a space in the literature of SSH; however, the predominant methods tend to underestimate the value of SSH research outputs because efforts fail to properly take into account the distinctive features of SSH research that differ from the natural sciences. In addition, the presence of adverse feelings of SSH scholars about quantifications and indicators ([<reflink idref="bib90" id="ref126">90</reflink>]) still emerge in the analysis.</p> <p>Some recent pieces of literature provide evidence that the most important and unavoidable social contributions of SSH are in providing an understanding of shared values, improving social awareness—also in an historical perspective—of our common cultural heritage, and the maintenance over the generations of the constitutive elements of our society ([<reflink idref="bib109" id="ref127">109</reflink>]). In this respect the separation between SSH and other fields is a category mistake, since all contribute to the human well-being ([<reflink idref="bib13" id="ref128">13</reflink>]).</p> <p>We have identified areas that require additional exploration in future studies. First, as social impact and political assessments are already performed in several national contexts and other initiatives with substantial differences, the need to produce systematic and comparable assessments of these processes is evident ([<reflink idref="bib95" id="ref129">95</reflink>]).</p> <p>Secondly, although each process for assessing the social impact of SSH research has different levels of development in different countries, considerable improvements in terms of identifying and analysing convergences affecting these processes are needed.</p> <p>Lastly, a third area involves the investigation of why research does not achieve an envisioned impact (scientific, political, or social). The surveyed literature addressed either methodological issues that are related to the assessment of impact of SSH, or impact assessments using different methods. In the latter case, almost all studies emphasize the transformative dimension by demonstrating how impact has been achieved and how metrics and methodological approaches are likely to determine the change produced by research. Failure is unacceptable and rendered invisible, and so the exclusionary dimension does not emerge as a central item to be assessed. A possible explanation may be that scholars generally consider outputs to be publishable to represent successful positive achievements. Another explanation may be that the collective literature aims to show unintended and negative consequences of existing assessment approaches rather than searching for explanations of the lack of impact. A further possibility is the long-term perspective of impact in SSH research, which considers impact a normal effect of research ([<reflink idref="bib89" id="ref130">89</reflink>]) but occurring at a time that can be very distant from the research activities and is therefore difficult to investigate. Thus, the exclusionary dimension is a topic that deserves special attention and may inform future investigations.</p> <hd id="AN0133426128-17">Funding</hd> <p>This study was supported by the European Union's Seventh Framework Programme for research, technological development and demonstration (FP7/2014-2017) under grant agreement number 613202 (IMPACT-EV Evaluating the Impact and Outcomes of European SSH Research).</p> <ref id="AN0133426128-18"> <title> Footnotes </title> <blist> <bibl id="bib1" idref="ref31" type="bt">1</bibl> <bibtext> Flash-it is a project funded under the European Union's Seventh Framework Programme that seeks, among other things, to standardize, analyse, synthesizes, and disseminate research results by creating a SSH network and the technological tools necessary to facilitate this.</bibtext> </blist> <blist> <bibl id="bib2" idref="ref57" type="bt">2</bibl> <bibtext> These developments also affect the analysis of SSH research impact. A pioneer in the area of open access, which also explores altmetrics, is the Public Library of Science PLOS Medicine Editors, 2006. The PLOS explores tools to track the post-publication reception of any research (Fenner 2014). This process has also been promoted by SSH scholars to generate Open Access initiatives, such as the Public Knowledge Project (MacGregor, Stranack and Willinsky 2014).</bibtext> </blist> <blist> <bibl id="bib3" idref="ref35" type="bt">3</bibl> <bibtext> See, for instance, a previous literature review of the societal impact of research (Gibbons et al. 1994 ; Newby 1994; Hanney, Packwood and Buxton 2000 ; Hessels and Lente 2010 ; de Jong et al. 2011 ; Holbrook and Frodeman 2011 ; United States Government Accountability Office 2012 ; Bornmann 2013).</bibtext> </blist> <blist> <bibl id="bib4" idref="ref5" type="bt">4</bibl> <bibtext> The Mode 1 and Mode 2 concepts originally comes from Gibbons et al. (1994).</bibtext> </blist> <blist> <bibl id="bib5" idref="ref34" type="bt">5</bibl> <bibtext> One recent example from the impact assessment of the Arts and Humanities Research Council in UK, in which the impact of Arts and Humanities Research was assessed regarding contributions to the development of creative industries, cultural engagement, new skills enhancement, and collaborative opportunities to sustain the emergence of new ideas and perspectives (AHRC 2015).</bibtext> </blist> </ref> <ref id="AN0133426128-19"> <title> References </title> <blist> <bibtext> Amara N., Ouimet M., Landry R. (2004) ' New Evidence on Instrumental, Conceptual and Symbolic Utilization of University Research in Government Agencies ', Science Communication, 26 : 75.</bibtext> </blist> <blist> <bibtext> Archambault E. et al. (2006) ' Benchmarking Scientific Output in the Social Sciences and Humanities: The Limits of Existing Databases ', Scientometrics, 68 / 3 : 329 – 42.</bibtext> </blist> <blist> <bibtext> Armstrong F., Alsop A. (2010) ' Debate: Co-Production can Contribute to Research Impact in the Social Sciences ', Public Money and Management, 30 / 4 : 208 – 10.</bibtext> </blist> <blist> <bibtext> Arnold E. (2004) ' Evaluating Research and Innovation Policy: A Systems World Needs Systems Evaluations ', Research Evaluation, 13 / 1 : 3 – 17. doi:10.3152/147154404781776509.</bibtext> </blist> <blist> <bibtext> AHRC-Arts and Humanities Research Council. (2015), The Impact of AHRC Research 2013-2014. Swindon: AHRC.</bibtext> </blist> <blist> <bibl id="bib6" type="bt">6</bibl> <bibtext> Barré R. (2001) ' The Agora Model of Innovation Systems, S&T Indicators for a Democratic Knowledge Society ', Research Evaluation, 10 / 1 : 13 – 18.</bibtext> </blist> <blist> <bibl id="bib7" idref="ref2" type="bt">7</bibl> <bibtext> Bastow S., Dunleavy P., Tinkler J. (2014) The Impact of the Social Sciences. Los Angeles; London; New Dehli; Singapore; Washington, DC : SAGE.</bibtext> </blist> <blist> <bibl id="bib8" idref="ref3" type="bt">8</bibl> <bibtext> Beck U. (1992) Risk Society. Towards a New Modernity. London : SAGE.</bibtext> </blist> <blist> <bibl id="bib9" idref="ref17" type="bt">9</bibl> <bibtext> Benneworth P. (2015) ' Tracing How Arts and Humanities Research Translates, Circulates and Consolidates in Society. How Have Scholars Been Reacting to Diverse Impact and Public Value Agendas? ', Arts and Humanities in Higher Education, 14 / 1 : 45 – 60.</bibtext> </blist> <blist> <bibtext> Berubé M. (2002) ' The Utility of the Arts and Humanities ', Arts and Humanities in Higher Education, 2 / 1 : 23 – 40.</bibtext> </blist> <blist> <bibtext> Blockmans W., Engwall L., Weaire D. (2014). Bibliometrics: Use and Abuse in the Review of Research Performance: Proceedings from a symposium held in Stockholm, 23-25 May 2013. London: Portland Press.</bibtext> </blist> <blist> <bibtext> Boaz A., Ashby D. (2003) 'Fit for Purpose? Assessing Research Quality for Evidence Based Policy and Practice', Working Paper 11. UK: ESRC Centre for Evidence Based Policy and Practice.</bibtext> </blist> <blist> <bibtext> Bod R. (2013) A New History of the Humanities. Oxford : Oxford University Press.</bibtext> </blist> <blist> <bibtext> Borgman C. et al. (1990) Scholarly Communication and Bibliometrics. Newbury Park : Sage.</bibtext> </blist> <blist> <bibtext> Borgman C. et al. (2007) Scholarship in the Digital Age: Information, Infrastructure, and the Internet. Cambridge, MA : MIT Press.</bibtext> </blist> <blist> <bibtext> Bornmann L. (2013) ' What is Societal Impact of Research and How can it be Assessed? A Literature Survey ', Journal of the American Society for Information Science and Technology, 64 / 2 : 217 – 33.</bibtext> </blist> <blist> <bibtext> Bornmann L. (2014) Measuring the Broader Impact of Research: The Potential of Altmetrics. Digital Libraries; Physics and Society; Applications. < <ulink href="http://arxiv.org/abs/1406.7091">http://arxiv.org/abs/1406.7091</ulink> >.</bibtext> </blist> <blist> <bibtext> Braun T., Glänzel W., Schubert A. (2006) ' A Hirsch-Type Index for Journals ', Scientometrics, 69 / 1 : 169 – 73.</bibtext> </blist> <blist> <bibtext> Brewer J. D. (2011) ' The Impact of Impact ', Research Evaluation, 20 / 3 : 255 – 6.</bibtext> </blist> <blist> <bibtext> Buxton M., Hanney S. (1994) Assessing Payback from Department of Health Research and Development: Preliminary Report. Volume 1: The Main Report. Uxbridge : HERG, Brunel University.</bibtext> </blist> <blist> <bibtext> Buxton M., Hanney S. (1996) ' How Can Payback from Health Services Research Be Assessed? ', Journal of Health Services Research and Policy, 1 / 1 : 35 – 43.</bibtext> </blist> <blist> <bibtext> Buxton M., Hanney S. (1998) ' Evaluating the NHS Research and Development Programme: Will the Programme Give Value for Money? ', Journal of the Royal Society of Medicine, 91 Suppl. 3 / 35 : 2 – 6.</bibtext> </blist> <blist> <bibtext> Chambers D. et al. (2012) ' Use of Evidence from Systematic Reviews to Inform Commissioning Decisions: A Case Study ', Policy and Evidence, 8 / 2 : 141 – 8.</bibtext> </blist> <blist> <bibtext> Cotterill S., Richardson L. (2010) ' Expanding the Use of Experiments on Civic Behaviour: Experiments with Local Government as a Research Partner ', The Annals of the American Academy of Political and Social Science, 628 : 148 – 64.</bibtext> </blist> <blist> <bibtext> De Bellis N. (2009) Bibliometrics and Citation Analysis: From the Science Citation Index to Cybermetrics. Lanham, MD : Scarecrow Press.</bibtext> </blist> <blist> <bibtext> De Jong S. et al. (2011) ' Evaluation of Research in Context: An Approach and Two Cases ', Research Evaluation, 20 / 1 : 61 – 72.</bibtext> </blist> <blist> <bibtext> De Jong S. et al. (2014) ' Understanding Societal Impact Through Productive Interactions: ICT Research as a Case ', Research Evaluation, 23 / 2 : 89 – 102.</bibtext> </blist> <blist> <bibtext> De Marchi M., Lorenzetti E. (2016) ' Measuring the Impact of Scholarly Journals in the Humanities Field ', Scientometrics, 106 : 253 – 61.</bibtext> </blist> <blist> <bibtext> de Rijcke S. et al. (2016) ' Evaluation Practices and Effects of Indicator Use—A Literature Review ', Research Evaluation, 25 : 161 – 9. doi:10.1093/reseval/rvv038.</bibtext> </blist> <blist> <bibtext> Donovan C. (2005) ' The Governance of Social Science and Everyday Epistemology ', Public Administration, 83 / 3 : 597 – 615.</bibtext> </blist> <blist> <bibtext> Donovan C. (2007) ' Guest Editor's Introduction: Future Pathways for Science Policy and Research Assessment: Metrics vs Peer Review, Quality vs Impact ', Science and Public Policy, 34 / 8 : 538 – 42.</bibtext> </blist> <blist> <bibtext> Donovan C. (2008) ' The Australian Research Quality Framework: A Live Experiment in Capturing the Social, Economic, Environmental, and Cultural Returns of Publicly Funded Research ', New Directions for Evaluation, 118 : 47 – 60.</bibtext> </blist> <blist> <bibtext> Donovan C. (2011) ' State of the Art in Assessing Research Impact: Introduction to a Special Issue ', Research Evaluation, 20 / 3 : 175 – 9.</bibtext> </blist> <blist> <bibtext> Donovan C., Hanney S. (2011) ' The ′Payback Framework′ Explained ', Research Evaluation, 20 / 3 : 181 – 3.</bibtext> </blist> <blist> <bibtext> Duijn M., Rijnveld M., van Hulst M. (2010) ' Meeting in the Middle: Joining Reflection and Action in Complex Public Sector Projects ', Public Money and Management, 30 / 4 : 227 – 33.</bibtext> </blist> <blist> <bibtext> Elboj C. (2014) ' In the Path to Regaining Social Sciences Legitimacy Through Public Sociology ', International and Multidisciplinary Journal of Social Sciences, 3 / 2 : 158 – 81.</bibtext> </blist> <blist> <bibtext> Ernø-Kjølhede E., Hansson F. (2011) ' Measuring Research Performance During a Changing Relationship Between Science and Society ', Research Evaluation, 20 / 2 : 130 – 42. doi: 10.3152/095820211X12941371876544.</bibtext> </blist> <blist> <bibtext> ESRC. (2010). Research Performance and Economic Impact Report 2010/II. < <ulink href="http://www.esrc.ac.uk/files/research/research-and-impact-evaluation/research-performance-and-economic-impact-report-2010-11/">http://www.esrc.ac.uk/files/research/research-and-impact-evaluation/research-performance-and-economic-impact-report-2010-11/</ulink> >.</bibtext> </blist> <blist> <bibtext> European Commission. (2005) Assessing the Social and Environmental Impacts of European Research. Brussels : Directorate-General for Research, European Communities.</bibtext> </blist> <blist> <bibtext> Evaluating Research in Context (ERiC). (2010) Evaluating the Societal Relevance of Academic Research: A Guide. Delft, The Netherlands : Delft University of Technology.</bibtext> </blist> <blist> <bibtext> Fenner M. (2014) 'Altmetrics and Other Novel Measures for Scientific Impact', in: Bartling S., Friesike S. (eds) Opening Science, pp. 179 – 89. Dordrecht : Springer.</bibtext> </blist> <blist> <bibtext> Flecha R. (2014) '¿Cómo conseguir o aumentar el Impacto de los proyectos y cómo liderar un proyecto de referencia?/How to improve the social impact of the projects?', Horizonte 2020 y Patrimonio Cultural: Investigación e Innovación, 03/31/2014, Madrid, Spain.</bibtext> </blist> <blist> <bibtext> Flecha R., Soler M. (2014) ' Communicative Methodology: Successful Actions and Dialogic Politics ', Current Sociology, 62 / 2 : 232 – 42.</bibtext> </blist> <blist> <bibtext> Flecha R., Soler-Gallart M., Sordé T. (2015) ' Social Impact: Europe Must Fund Social Science ', Nature, 528 : 193. doi:10.1038/528193d.</bibtext> </blist> <blist> <bibtext> Fletcha R. (2015). Successful Educational Actions for Inclusion and Social Cohesion in Europe. Dordrect : Springer.</bibtext> </blist> <blist> <bibtext> Forbes I., Abrams D. (2004) ' International Social Science Research: Craft Industry or Baby Behemoth? ', International Social Science Journal, 56 / 180 : 227 – 44.</bibtext> </blist> <blist> <bibtext> Gabolde J. (1998) ' New Challenges for Indicators in Science and Technology Policy-Making: A European View ', Research Evaluation, 7 / 2 : 99 – 104.</bibtext> </blist> <blist> <bibtext> Garfield E. (1980) ' Is Information Retrieval in the Arts and Humanities Inherently Different from That in Science? ', The Library Quarterly, 50 : 40 – 57.</bibtext> </blist> <blist> <bibtext> Garfield E. (2006) ' The History and Meaning of the Journal Impact Factor ', JAMA: The Journal of the American Medical Association, 295 / 1 : 90 – 3. doi:10.1001/jama.295.1.90.</bibtext> </blist> <blist> <bibtext> Gibbons M. et al. (1994) The New Production of Knowldege. The Dynamics of Science and Research in Contemporary Societies. London; Thousand Oaks; New Delhi : Sage.</bibtext> </blist> <blist> <bibtext> Glänzel W., Moed H. F. (2002) ' Journal Impact Measures in Bibliometric Research ', Scientometrics, 53 / 2 : 171 – 93. doi:10.1023/A:1014848323806.</bibtext> </blist> <blist> <bibtext> Godin B., Doré C. (2005) 'Measuring the Impacts of Science; Beyond the Economic Dimension, INRS Urbanisation, Culture et Sociult', HIST Lecture. Helsinki, Finland : Helsinki Institute for Science and Technology Studies.</bibtext> </blist> <blist> <bibtext> Gómez A., Puigvert L., Flecha R. (2011) ' Critical Communicative Methodology: Informing Real Social Transformation Through Research ', Qualitative Inquiry, 17 / 3 : 235 – 45. doi: 10.1177/1077800410397802.</bibtext> </blist> <blist> <bibtext> González-Pereira B., Guerrero-Bote V. P., Moya-Arnegón F. (2010) ' A New Approach to the Metric of Jounals' Scientific Prestige: The SJR Indicator ', Journal of Informetrics, 4 / 3 : 379 – 91. doi: 10.1016/j.joi.2010.03.002.</bibtext> </blist> <blist> <bibtext> Guetzkow J., Lamont M., Mallard G. (2004) ' What is Originality in the Humanities and the Social Sciences? ', American Sociological Review, 69 : 190 – 2012.</bibtext> </blist> <blist> <bibtext> Hammarfelt B. (2012). 'Following the Footnotes: A Bibliometric Analysis of Citation Patterns in Literary Studies', Doctoral dissertation. Skrifter utgivna vid institutionen för ABM vid Uppsala Universitet, Vol. 5. Uppsala: Uppsala Universitet. < <ulink href="http://www.diva-portal.org/smash/get/diva2:511996/FULLTEXT01.pdf">http://www.diva-portal.org/smash/get/diva2:511996/FULLTEXT01.pdf</ulink> >.</bibtext> </blist> <blist> <bibtext> Hammarfelt B. (2014) ' Using Altmetrics for Assessing Research Impact in the Humanities ', Scientometrics, 101 / 2 : 1419 – 30.</bibtext> </blist> <blist> <bibtext> Hanney S., Packwood T., Buxton M. (2000) ' Evaluating the Benefits from Health Research and Development Centres: A Categorization, a Model, and Examples of Application ', Evaluation: The International Journal of Theory, Research and Practice, 6 : 137 – 60.</bibtext> </blist> <blist> <bibtext> Hazelkorn E. (2014). ' Making an Impact: New Directions for Arts and Humanities Research ', Articles. Paper 51. <ulink href="http://arrow.dit.ie/cserart/51">http://arrow.dit.ie/cserart/51</ulink>.</bibtext> </blist> <blist> <bibtext> Henshall C. (2011) ' The Impact of Payback Research: Developing and Using Evidence in Policy ', Research Evaluation, 20 / 3 : 257 – 8.</bibtext> </blist> <blist> <bibtext> Hessels L. K., Lente H. (2010) ' The Mixed Blessing of Mode 2 Knowledge Production ', Science, Technology and Innovation Studies, 6 / 1 : 65 – 9.</bibtext> </blist> <blist> <bibtext> Hicks D. (2004). 'The Four Literatures of Social Science', in Moed H. F., Glänzel W., Schmoch U. (eds) Handbook of Quantitative Science and Technology Research, pp. 473 – 96. The Netherlands : Kluwer Academic Publisher.</bibtext> </blist> <blist> <bibtext> Hicks D., Wang J. (2009). Towards a Bibliometric Database for the Social Sciences and Humanities. School of Public Policy : Georgia University of Technology. < https://works.bepress.com/diana%5fhicks/18/ >.</bibtext> </blist> <blist> <bibtext> Hicks D. et al. (2015) ' Bibliometrics: The Leiden Manifesto for Research Metrics ', Nature, 520 / 7548 : 429 – 31.</bibtext> </blist> <blist> <bibtext> Hirsch J. E. (2005) ' An Index to Quantify and Individual's Scientific Research Output ', Proceedings of the National Academy of Sciences of the United States of America, 102 / 46 : 16569 – 72. doi: 10.1073/pnas.0507655102.</bibtext> </blist> <blist> <bibtext> Holbrook J. B., Frodeman R. (2011) ' Peer Review and the ex ante Assessment of Societal Impacts ', Research Evaluation, 20 / 3 : 239 – 46.</bibtext> </blist> <blist> <bibtext> Jacobs J. A. (2011) 'Journal Rankings in Sociology: Using the H Index with Google Scholar', PSC Working Paper Series, PSC 11-05. Philadelphia : Population Studies Center, University of Pennsylvania.</bibtext> </blist> <blist> <bibtext> Joerges B., Shinn T. (2001) Instrumentation: Between Science, State, and Industry. Dordrecht : Kluwer Academic Publishers.</bibtext> </blist> <blist> <bibtext> Klautzer L. et al. (2011) ' Assessing Policy and Practice Impacts of Social Science Research: The Application of the Payback Framework to Assess the Future of Work programme ', Research Evaluation, 20 / 3 : 201 – 9.</bibtext> </blist> <blist> <bibtext> KNAW—Royal Netherlands Academy of Arts and Sciences. (2011). Quality Indicators for Research in the Humanities. Amsterdam : KNAW.</bibtext> </blist> <blist> <bibtext> Kuhlmann S. (1998) ' Moderation of Policy Making? Science and Technology Policy Evaluation Beyond Impact Measurement—The Case of Germany ', Evaluation, 4 / 2 : 192 – 205.</bibtext> </blist> <blist> <bibtext> Lam A. (2011) ' What Motivates Academic Scientists to Engage in Research Commercialization: 'Gold' 'Ribbon' or 'Puzzle'? ', Research Policy, 40 : 1354 – 68.</bibtext> </blist> <blist> <bibtext> Larivière V. et al. (2013) ' Bibliometrics: Global Gender Disparities in Science ', Nature, 504 / 7479 : 211 – 13. doi:10.1038/504211a.</bibtext> </blist> <blist> <bibtext> Lemay M. A., Sá C. (2012) ' Complexity Sciences: Towards an Alternative Approach to Understanding the Use of Academic Research ', Evidence and Policy, 8 / 4 : 473 – 94.</bibtext> </blist> <blist> <bibtext> Lepori B., Probst C. (2009) ' Using Curricula Vitae for Mapping Scientific Fields: A small-Scale Experience for Swiss Communication Sciences ', Research Evaluation, 18 / 2 : 125 – 34.</bibtext> </blist> <blist> <bibtext> Levitt R. et al. (2010) Assessing the Impact of Arts and Humanities Research at the University of Cambridge. Cambridge : RAND Europe.</bibtext> </blist> <blist> <bibtext> Leydesdorff L., Hammarfelt B., Salah A. (2011) ' The Structure of the Arts and Humanities Citation Index: A Mapping on the Basis of Aggregated Citations Among 1,157 Journals ', Journal of the American Society for Information Science and Technology, 62 : 2414 – 26. doi:10.1002/asi.21636.</bibtext> </blist> <blist> <bibtext> MacGregor J., Stranack K., Willinsky J. (2014) 'The Public Knowledge Project: Open Source Tools for Open Acces to Scholarly Communication', in Bartling S., Friesike S. (eds) Opening Science, pp. 165 – 75. Dordrecht : Springer.</bibtext> </blist> <blist> <bibtext> Meagher L., Lyall C., Nutley S. (2008) ' Flows of Knowledge, Experitise and Influence: A Method for Assessing Policy and Practice Impacts from Social Science Research ', Research Evaluation, 17 / 3 : 163 – 73.</bibtext> </blist> <blist> <bibtext> Molas-Gallart J., Tang P. (2011) ' Tracing "Productive Interactions" to Identify Social Impacts: An Example from the Social Sciences ', Research Evaluation, 20 / 3 : 219 – 26.</bibtext> </blist> <blist> <bibtext> Molas-Gallart J. (2015) ' Research Evaluation and the Assessment of Public Value ', Arts and Humanities in Higher Education, 14 / 1 : 111 – 26.</bibtext> </blist> <blist> <bibtext> Mohammadi E., Thelwall M. (2014) ' Mendeley Readership Altmetrics for the Social Sciences and Humanities: Research Evaluation and Knowledge Flows ', Journal of the Association for Information Science and Technology, 65 / 8 : 1627 – 38.</bibtext> </blist> <blist> <bibtext> Morton S. (2015) ' Progressing Research Impact Assessment: A 'Contributions' Approach ', Research Evaluation, 24 : 405 – 19.</bibtext> </blist> <blist> <bibtext> Munté A., Serradell O., Sordé T. (2011) ' From Research to Policy: Roma Participation Through Communicative Organization ', Qualitative Inquiry, 17 / 3 : 256 – 66.</bibtext> </blist> <blist> <bibtext> Nederhof A. J. (2006) ' Bibliometric Monitoring of Research Performance in the Social Sciences and the Humanities: A Review ', Scientometrics, 66 / 1 : 81 – 100.</bibtext> </blist> <blist> <bibtext> Nedeva M., Braun D., Edler J., Glaser J., Laredo P., Laredo G., Luukkonen T., Stampfer M., Thomas D., Whitley R. (2012) ' Understanding and Assessing the Impact and Outcomes of the European Research Council and its Funding Schemes. European Research Council ', <ulink href="http://erc.europa.eu/documents/understanding-and-assessing-impact-and-outcomes-erc-and-its-funding-schemes-eurecia-final-">http://erc.europa.eu/documents/understanding-and-assessing-impact-and-outcomes-erc-and-its-funding-schemes-eurecia-final-</ulink>.</bibtext> </blist> <blist> <bibtext> Newby H. (1994) ' The challenge for social science: a new role in public policy-making ', Research Evaluation, 4 / 1 : 6 – 11.</bibtext> </blist> <blist> <bibtext> Nussbaum M. (2010) Not for Profit. Why Democracy Needs the Humanities. Princeton : Princeton University Press.</bibtext> </blist> <blist> <bibtext> Ochsner M. et al. (2013) ' Four Types of Research in the Humanities: Setting the Stage for Research Quality Criteria ', Research Evaluation, 22 : 369 – 83.</bibtext> </blist> <blist> <bibtext> Ochsner M., Hug S. E., Daniel H. D. (eds) (2016). Research and Assessment in the Humanities. Toward criteria and Procedures. Switzerland : Springer.</bibtext> </blist> <blist> <bibtext> O'Hare P., Coaffee J., Hawkesworth M. (2010) ' Managing Sensitive Relations in Co-produced Panning Research ', Public Money and Management, 30 / 4 : 243 – 50.</bibtext> </blist> <blist> <bibtext> Penfield T. et al. (2014) ' Assessment, Evaluations, and Definitions of Research Impact: A Review ', Research Evaluation, 23 / 1 : 21 – 32.</bibtext> </blist> <blist> <bibtext> Peric B. et al. (2013). Arts and Humanities Research Assessment Bibliography (AHRABi). Zurich : ETH Zurich.</bibtext> </blist> <blist> <bibtext> PLoS Medicine Editors. (2006) ' The Impact Factor Game ', PLoS Medicine, 3 / 6 : e291. doi:10.1371/journal.pmed.0030291.</bibtext> </blist> <blist> <bibtext> Price A. M., Peterson L. P. (2016) ' Scientific Progress, Risk, and Development: Explaining Attitudes Toward Science Cross-Nationally ', International Sociology, 31 / 1 : 57 – 80.</bibtext> </blist> <blist> <bibtext> Price D. J. S. (1963) Little science, big science. New York : Columbia Univ. Press.</bibtext> </blist> <blist> <bibtext> Priem J. T. et al. (2010) Altmetrics: A manifesto. < <ulink href="http://altmetrics.org/manifesto/">http://altmetrics.org/manifesto/</ulink> >.</bibtext> </blist> <blist> <bibtext> Prins A. A. M. et al. (2016) ' Using Google Scholar in Research Evaluation of Humanities and Social Science Programs: A Comparison with Web of Science Data ', Research Evaluation, 25 / 3 : 264 – 70.</bibtext> </blist> <blist> <bibtext> Rawlings C. M., McFarland D. A. (2011) ' Influence Flows in the Academy: Using Affiliation Networks to Assess Peer Effects Among Researchers ', Social Science Research, 40 / 3 : 1001 – 17. doi:10.1016/j.ssresearch.2010.10.002.</bibtext> </blist> <blist> <bibtext> Reale E. et al. (2014) ' Evaluation Through Impact: A Different Viewpoint ', Fteval Journal for Research and Technology Policy Evaluation, 39 : 36 – 41.</bibtext> </blist> <blist> <bibtext> Rymer L. (2011) 'Measuring the Impact of Research–the Context for Metric Development', in The Group of Eight (ed.), Go8 Backgounder, Vol. 23. Turner : ACT.</bibtext> </blist> <blist> <bibtext> Samuel G. N., Derrick G. (2015) ' Societal Impact Evaluation: Exploring Evaluator Perceptions of the Characterization of Impact Under the REF2014 ', Research Evaluation, 24 : 229 – 41.</bibtext> </blist> <blist> <bibtext> Sanderson I. (2009) ' Intelligent Policy Making for a Complex World: Pragmatism, Evidence and Learning ', Political Studies, 57 / 4 : 699 – 719.</bibtext> </blist> <blist> <bibtext> Scharnhorst A., Wouters P., van den Besselaar P. (2006) ' What Does the Web Represent? From Virtual Ethnography to Web Indicators ', Special Issue of Cybermetrics, 10 / 1 : 1 – 3.</bibtext> </blist> <blist> <bibtext> Scharnhorst A., Garfield E. (2010) ' Tracing Scientific Influence ', Journal of Dynamics of Socio-Economic Systems, 2 / 1 : 1 – 33.</bibtext> </blist> <blist> <bibtext> Schmoch U. et al. (2010) ' How to Use Indicators to Measure Scientific Performance: A Balanced Approach ', Research Evaluation, 19 / 1 : 2 – 18.</bibtext> </blist> <blist> <bibtext> SIAMPI. (2011) Social Impact Assessment Methods for Research and Funding Instruments Through the Study of Productive Interactions. Final report. < http://www. siampi.edu/ >.</bibtext> </blist> <blist> <bibtext> Sivertsen G., Larsen B. (2012) ' Comprehensive Bibliographic Coverage of the Social Sciences and Humanities in a Citation Index: An Empirical Analysis of the Potential ', Scientometrics, 91 : 567 – 75.</bibtext> </blist> <blist> <bibtext> Small H. (2013) The Value of the Humanities. Oxford : Oxford University Press.</bibtext> </blist> <blist> <bibtext> Social Sciences and Humanities Scientific Committees. (2013) Humanities and Social Sciences in Horizon 2020 Societal Challenges: Implementation and Monitoring. < <ulink href="http://www.scienceeurope.org/uploads/PublicDocumentsAndSpeeches/SSH%5fHorizon2020%5fWEB%5ffin.pdf">http://www.scienceeurope.org/uploads/PublicDocumentsAndSpeeches/SSH%5fHorizon2020%5fWEB%5ffin.pdf</ulink> >.</bibtext> </blist> <blist> <bibtext> Sordé-Martí T. et al. (2014) ' Solidarity Networks that Challenge Racialized Discourses: The Case of Romani Immigrant Women in Spain ', European Journal of Women's Studies, 21 / 1 : 87 – 102.</bibtext> </blist> <blist> <bibtext> Spaapen J., van Drooge L. (2011) ' Introducing "Productive Interactions" in Social Impact Assessment ', Research Evaluation, 20 / 3 : 211 – 18.</bibtext> </blist> <blist> <bibtext> Spaapen J., van Drooge L., Sylvain C. (1993) ' Assessing the Value of Research for Society ', Research Evaluation, 3 / 2 : 117 – 26.</bibtext> </blist> <blist> <bibtext> Torres-Salinas D. et al. (2013) ' Mapping Citation Patterns of Book Chapters in the Book Citation Index ', Journal of Informetrics, 7 / 2 : 412 – 24. doi:10.1016/j.joi.2013.01.004.</bibtext> </blist> <blist> <bibtext> United States Government Accountability Office. (2012) Designing Evaluations. Washington, DC : United States Government Accountability Office.</bibtext> </blist> <blist> <bibtext> Valls R., Padrós M. (2011) ' Using Dialogic Research to Overcome Poverty: From Principles to Action ', European Journal of Education, 46 / 2 : 173 – 83.</bibtext> </blist> <blist> <bibtext> Wehrens R., Bekker M., Bal R. (2012) ' Dutch Academic Collaborative Centres for Public Health: Development Through Time—Issues, Dilemmas and Coping Strategies ', Evidence and Policy, 8 / 2 : 149 – 70.</bibtext> </blist> <blist> <bibtext> Weinberg B. A. et al. (2014) ' Science Funding and Short-Term Economic Activity ', Science, 344 / 6179 : 41 – 3.</bibtext> </blist> <blist> <bibtext> Weingart P., Schwechheimer H. (2007) Conceptualizing and Measuring Excellence in the Social Sciences and Humanities. Global SSH. < <ulink href="http://www.globalsocialscience.org">www.globalsocialscience.org</ulink> >.</bibtext> </blist> <blist> <bibtext> Whitley R. (2000) The Intellectual and Social Organization of the Sciences. Oxford : Oxford University Press.</bibtext> </blist> <blist> <bibtext> Wooding S. E. et al. (2007) Policy and practice impacts of research funded by the Economic and Social Research Council. A case study of the Future of Work programme, approach and analysis. < <ulink href="http://www.rand.org/content/dam/rand/pubs/technical%5freports/2007/RAND%5fTR435.pdf">http://www.rand.org/content/dam/rand/pubs/technical%5freports/2007/RAND%5fTR435.pdf</ulink> >.</bibtext> </blist> <blist> <bibtext> Wouters P., Costas R. (2012) Users, Narcissism and Control—Tracking the Impact of Scholarly Publications in the 21 st century. < <ulink href="http://www.surffoundation.nl/en/publicaties/Pages/Users%5fnarcissism%5fcontrol.aspx">http://www.surffoundation.nl/en/publicaties/Pages/Users%5fnarcissism%5fcontrol.aspx</ulink> >.</bibtext> </blist> </ref> <aug> <p>By Emanuela Reale; Dragana Avramov; Kubra Canhial; Claire Donovan; Ramon Flecha; Poul Holm; Charles Larkin; Benedetto Lepori; Judith Mosoni-Fried and Esther Oliver</p> <p>Reported by Author; Author; Author; Author; Author; Author; Author; Author; Author; Author</p> </aug> <nolink nlid="nl1" bibid="bib120" firstref="ref1"></nolink> <nolink nlid="nl2" bibid="bib30" firstref="ref4"></nolink> <nolink nlid="nl3" bibid="bib55" firstref="ref6"></nolink> <nolink nlid="nl4" bibid="bib46" firstref="ref8"></nolink> <nolink nlid="nl5" bibid="bib119" firstref="ref10"></nolink> <nolink nlid="nl6" bibid="bib89" firstref="ref11"></nolink> <nolink nlid="nl7" bibid="bib13" firstref="ref12"></nolink> <nolink nlid="nl8" bibid="bib81" firstref="ref13"></nolink> <nolink nlid="nl9" bibid="bib90" firstref="ref15"></nolink> <nolink nlid="nl10" bibid="bib10" firstref="ref16"></nolink> <nolink nlid="nl11" bibid="bib88" firstref="ref18"></nolink> <nolink nlid="nl12" bibid="bib109" firstref="ref20"></nolink> <nolink nlid="nl13" bibid="bib71" firstref="ref21"></nolink> <nolink nlid="nl14" bibid="bib92" firstref="ref22"></nolink> <nolink nlid="nl15" bibid="bib42" firstref="ref25"></nolink> <nolink nlid="nl16" bibid="bib39" firstref="ref26"></nolink> <nolink nlid="nl17" bibid="bib52" firstref="ref27"></nolink> <nolink nlid="nl18" bibid="bib16" firstref="ref28"></nolink> <nolink nlid="nl19" bibid="bib53" firstref="ref30"></nolink> <nolink nlid="nl20" bibid="bib96" firstref="ref39"></nolink> <nolink nlid="nl21" bibid="bib25" firstref="ref40"></nolink> <nolink nlid="nl22" bibid="bib49" firstref="ref41"></nolink> <nolink nlid="nl23" bibid="bib105" firstref="ref42"></nolink> <nolink nlid="nl24" bibid="bib122" firstref="ref43"></nolink> <nolink nlid="nl25" bibid="bib65" firstref="ref44"></nolink> <nolink nlid="nl26" bibid="bib18" firstref="ref45"></nolink> <nolink nlid="nl27" bibid="bib68" firstref="ref46"></nolink> <nolink nlid="nl28" bibid="bib51" firstref="ref47"></nolink> <nolink nlid="nl29" bibid="bib77" firstref="ref48"></nolink> <nolink nlid="nl30" bibid="bib54" firstref="ref49"></nolink> <nolink nlid="nl31" bibid="bib11" firstref="ref50"></nolink> <nolink nlid="nl32" bibid="bib64" firstref="ref51"></nolink> <nolink nlid="nl33" bibid="bib14" firstref="ref52"></nolink> <nolink nlid="nl34" bibid="bib15" firstref="ref53"></nolink> <nolink nlid="nl35" bibid="bib104" firstref="ref54"></nolink> <nolink nlid="nl36" bibid="bib97" firstref="ref55"></nolink> <nolink nlid="nl37" bibid="bib17" firstref="ref56"></nolink> <nolink nlid="nl38" bibid="bib31" firstref="ref58"></nolink> <nolink nlid="nl39" bibid="bib62" firstref="ref60"></nolink> <nolink nlid="nl40" bibid="bib29" firstref="ref61"></nolink> <nolink nlid="nl41" bibid="bib57" firstref="ref62"></nolink> <nolink nlid="nl42" bibid="bib63" firstref="ref63"></nolink> <nolink nlid="nl43" bibid="bib114" firstref="ref64"></nolink> <nolink nlid="nl44" bibid="bib56" firstref="ref65"></nolink> <nolink nlid="nl45" bibid="bib93" firstref="ref67"></nolink> <nolink nlid="nl46" bibid="bib108" firstref="ref69"></nolink> <nolink nlid="nl47" bibid="bib28" firstref="ref71"></nolink> <nolink nlid="nl48" bibid="bib75" firstref="ref72"></nolink> <nolink nlid="nl49" bibid="bib67" firstref="ref73"></nolink> <nolink nlid="nl50" bibid="bib98" firstref="ref74"></nolink> <nolink nlid="nl51" bibid="bib82" firstref="ref76"></nolink> <nolink nlid="nl52" bibid="bib85" firstref="ref79"></nolink> <nolink nlid="nl53" bibid="bib32" firstref="ref80"></nolink> <nolink nlid="nl54" bibid="bib102" firstref="ref81"></nolink> <nolink nlid="nl55" bibid="bib66" firstref="ref82"></nolink> <nolink nlid="nl56" bibid="bib113" firstref="ref85"></nolink> <nolink nlid="nl57" bibid="bib47" firstref="ref86"></nolink> <nolink nlid="nl58" bibid="bib40" firstref="ref87"></nolink> <nolink nlid="nl59" bibid="bib106" firstref="ref88"></nolink> <nolink nlid="nl60" bibid="bib26" firstref="ref89"></nolink> <nolink nlid="nl61" bibid="bib33" firstref="ref90"></nolink> <nolink nlid="nl62" bibid="bib80" firstref="ref92"></nolink> <nolink nlid="nl63" bibid="bib72" firstref="ref93"></nolink> <nolink nlid="nl64" bibid="bib79" firstref="ref95"></nolink> <nolink nlid="nl65" bibid="bib74" firstref="ref96"></nolink> <nolink nlid="nl66" bibid="bib12" firstref="ref97"></nolink> <nolink nlid="nl67" bibid="bib103" firstref="ref98"></nolink> <nolink nlid="nl68" bibid="bib23" firstref="ref99"></nolink> <nolink nlid="nl69" bibid="bib117" firstref="ref102"></nolink> <nolink nlid="nl70" bibid="bib50" firstref="ref103"></nolink> <nolink nlid="nl71" bibid="bib37" firstref="ref104"></nolink> <nolink nlid="nl72" bibid="bib35" firstref="ref106"></nolink> <nolink nlid="nl73" bibid="bib24" firstref="ref109"></nolink> <nolink nlid="nl74" bibid="bib43" firstref="ref116"></nolink> <nolink nlid="nl75" bibid="bib84" firstref="ref117"></nolink> <nolink nlid="nl76" bibid="bib83" firstref="ref118"></nolink> <nolink nlid="nl77" bibid="bib73" firstref="ref120"></nolink> <nolink nlid="nl78" bibid="bib99" firstref="ref121"></nolink> <nolink nlid="nl79" bibid="bib59" firstref="ref123"></nolink> <nolink nlid="nl80" bibid="bib76" firstref="ref125"></nolink> <nolink nlid="nl81" bibid="bib95" firstref="ref129"></nolink>
Header DbId: eric
DbLabel: ERIC
An: EJ1208332
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: A Review of Literature on Evaluating the Scientific, Social and Political Impact of Social Sciences and Humanities Research
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Reale%2C+Emanuela%22">Reale, Emanuela</searchLink><br /><searchLink fieldCode="AR" term="%22Avramov%2C+Dragana%22">Avramov, Dragana</searchLink><br /><searchLink fieldCode="AR" term="%22Canhial%2C+Kubra%22">Canhial, Kubra</searchLink><br /><searchLink fieldCode="AR" term="%22Donovan%2C+Claire%22">Donovan, Claire</searchLink><br /><searchLink fieldCode="AR" term="%22Flecha%2C+Ramon%22">Flecha, Ramon</searchLink><br /><searchLink fieldCode="AR" term="%22Holm%2C+Poul%22">Holm, Poul</searchLink><br /><searchLink fieldCode="AR" term="%22Larkin%2C+Charles%22">Larkin, Charles</searchLink><br /><searchLink fieldCode="AR" term="%22Lepori%2C+Benedetto%22">Lepori, Benedetto</searchLink><br /><searchLink fieldCode="AR" term="%22Mosoni-Fried%2C+Judith%22">Mosoni-Fried, Judith</searchLink><br /><searchLink fieldCode="AR" term="%22Oliver%2C+Esther%22">Oliver, Esther</searchLink><br /><searchLink fieldCode="AR" term="%22Primeri%2C+Emilia%22">Primeri, Emilia</searchLink><br /><searchLink fieldCode="AR" term="%22Puigvert%2C+Lidia%22">Puigvert, Lidia</searchLink><br /><searchLink fieldCode="AR" term="%22Scharnhorst%2C+Andrea%22">Scharnhorst, Andrea</searchLink><br /><searchLink fieldCode="AR" term="%22Schubert%2C+Andràs%22">Schubert, Andràs</searchLink><br /><searchLink fieldCode="AR" term="%22Soler%2C+Marta%22">Soler, Marta</searchLink><br /><searchLink fieldCode="AR" term="%22Soòs%2C+Sàndor%22">Soòs, Sàndor</searchLink><br /><searchLink fieldCode="AR" term="%22Sordé%2C+Teresa%22">Sordé, Teresa</searchLink><br /><searchLink fieldCode="AR" term="%22Travis%2C+Charles%22">Travis, Charles</searchLink><br /><searchLink fieldCode="AR" term="%22Van+Horik%2C+René%22">Van Horik, René</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Research+Evaluation%22"><i>Research Evaluation</i></searchLink>. Oct 2018 27(4):298-308.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Oxford University Press. Great Clarendon Street, Oxford, OX2 6DP, UK. Tel: +44-1865-353907; Fax: +44-1865-353485; e-mail: jnls.cust.serv@oxfordjournals.org; Web site: http://applij.oxfordjournals.org/
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 11
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2018
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Literature+Reviews%22">Literature Reviews</searchLink><br /><searchLink fieldCode="DE" term="%22Social+Influences%22">Social Influences</searchLink><br /><searchLink fieldCode="DE" term="%22Political+Influences%22">Political Influences</searchLink><br /><searchLink fieldCode="DE" term="%22Social+Sciences%22">Social Sciences</searchLink><br /><searchLink fieldCode="DE" term="%22Humanities%22">Humanities</searchLink><br /><searchLink fieldCode="DE" term="%22Research%22">Research</searchLink><br /><searchLink fieldCode="DE" term="%22Foreign+Countries%22">Foreign Countries</searchLink>
– Name: Subject
  Label: Geographic Terms
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22European+Union%22">European Union</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1093/reseval/rvx025
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 0958-2029
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Recently, the need to contribute to the evaluation of the scientific, social, and political impact of Social Sciences and Humanities (SSH) research has become a demand of policy makers and society. The international scientific community has made significant advances that have transformed the impact of evaluation landscape. This article reviews the existing scientific knowledge on evaluation tools and techniques that are applied to assess the scientific impact of SSH research; the changing structure of social and political impacts of SSH research is investigated based on an overarching research question: to what extent do scholars attempt to apply methods, instruments, and approaches that take into account the distinctive features of SSH? The review also includes examples of European Union (EU) projects that demonstrate these impacts. This article culminates in a discussion of the development of the assessment of different impacts and identifies limitations, and areas and topics to explore in the future.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2019
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1208332
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1208332
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1093/reseval/rvx025
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 11
        StartPage: 298
    Subjects:
      – SubjectFull: Literature Reviews
        Type: general
      – SubjectFull: Social Influences
        Type: general
      – SubjectFull: Political Influences
        Type: general
      – SubjectFull: Social Sciences
        Type: general
      – SubjectFull: Humanities
        Type: general
      – SubjectFull: Research
        Type: general
      – SubjectFull: Foreign Countries
        Type: general
      – SubjectFull: European Union
        Type: general
    Titles:
      – TitleFull: A Review of Literature on Evaluating the Scientific, Social and Political Impact of Social Sciences and Humanities Research
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Reale, Emanuela
      – PersonEntity:
          Name:
            NameFull: Avramov, Dragana
      – PersonEntity:
          Name:
            NameFull: Canhial, Kubra
      – PersonEntity:
          Name:
            NameFull: Donovan, Claire
      – PersonEntity:
          Name:
            NameFull: Flecha, Ramon
      – PersonEntity:
          Name:
            NameFull: Holm, Poul
      – PersonEntity:
          Name:
            NameFull: Larkin, Charles
      – PersonEntity:
          Name:
            NameFull: Lepori, Benedetto
      – PersonEntity:
          Name:
            NameFull: Mosoni-Fried, Judith
      – PersonEntity:
          Name:
            NameFull: Oliver, Esther
      – PersonEntity:
          Name:
            NameFull: Primeri, Emilia
      – PersonEntity:
          Name:
            NameFull: Puigvert, Lidia
      – PersonEntity:
          Name:
            NameFull: Scharnhorst, Andrea
      – PersonEntity:
          Name:
            NameFull: Schubert, Andràs
      – PersonEntity:
          Name:
            NameFull: Soler, Marta
      – PersonEntity:
          Name:
            NameFull: Soòs, Sàndor
      – PersonEntity:
          Name:
            NameFull: Sordé, Teresa
      – PersonEntity:
          Name:
            NameFull: Travis, Charles
      – PersonEntity:
          Name:
            NameFull: Van Horik, René
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 10
              Type: published
              Y: 2018
          Identifiers:
            – Type: issn-print
              Value: 0958-2029
          Numbering:
            – Type: volume
              Value: 27
            – Type: issue
              Value: 4
          Titles:
            – TitleFull: Research Evaluation
              Type: main
ResultId 1