n.d.). Understanding what impact looks like across the various strands of research and the variety of indicators and proxies used to evidence impact will be important to developing a meaningful assessment. The range and diversity of frameworks developed reflect the variation in purpose of evaluation including the stakeholders for whom the assessment takes place, along with the type of impact and evidence anticipated. What are the reasons behind trying to understand and evaluate research impact? Decker et al. RAND Europe, Capturing Research Impacts. 0000002318 00000 n We suggest that developing systems that focus on recording impact information alone will not provide all that is required to link research to ensuing events and impacts, systems require the capacity to capture any interactions between researchers, the institution, and external stakeholders and link these with research findings and outputs or interim impacts to provide a network of data. 0000007223 00000 n Incorporating assessment of the wider socio-economic impact began using metrics-based indicators such as Intellectual Property registered and commercial income generated (Australian Research Council 2008). Using the above definition of evaluation, program evaluation approaches were classified into four categories. Impact is not static, it will develop and change over time, and this development may be an increase or decrease in the current degree of impact. Measurement assessment and evaluation also enables educators to measure the skills, knowledge, beliefs, and attitude of the learners. n.d.). What are the methodologies and frameworks that have been employed globally to evaluate research impact and how do these compare? Case studies are ideal for showcasing impact, but should they be used to critically evaluate impact? Evaluative research has many benefits, including identifying whether a product works as intended, and uncovering areas for improvement within your solution. The Economic and Social Benefits of HRB-funded Research, Measuring the Economic and Social Impact of the Arts: A Review, Research Excellence Framework Impact Pilot Exercise: Findings of the Expert Panels, Assessment Framework and Guidance on Submissions, Research Impact Evaluation, a Wider Context. Collating the evidence and indicators of impact is a significant task that is being undertaken within universities and institutions globally. They risk being monetized or converted into a lowest common denominator in an attempt to compare the cost of a new theatre against that of a hospital. Metrics have commonly been used as a measure of impact, for example, in terms of profit made, number of jobs provided, number of trained personnel recruited, number of visitors to an exhibition, number of items purchased, and so on. If impact is short-lived and has come and gone within an assessment period, how will it be viewed and considered? Overview of the types of information that systems need to capture and link. Frameworks for assessing impact have been designed and are employed at an organizational level addressing the specific requirements of the organization and stakeholders. (2007:11-12), describes and explains the different types of value claim. The authors propose a new definition for measurement process based on the identification of the type of measurand and other metrological elements at each measurement process identified. working paper). Evaluation of impact in terms of reach and significance allows all disciplines of research and types of impact to be assessed side-by-side (Scoble et al. Definition of evaluation. To allow comparisons between institutions, identifying a comprehensive taxonomy of impact, and the evidence for it, that can be used universally is seen to be very valuable. It has been acknowledged that outstanding leaps forward in knowledge and understanding come from immersing in a background of intellectual thinking that one is able to see further by standing on the shoulders of giants. 1.3. Husbands-Fealing suggests that to assist identification of causality for impact assessment, it is useful to develop a theoretical framework to map the actors, activities, linkages, outputs, and impacts within the system under evaluation, which shows how later phases result from earlier ones. The Payback Framework systematically links research with the associated benefits (Scoble et al. 0000007307 00000 n While looking forward, we will be able to reduce this problem in the future, identifying, capturing, and storing the evidence in such a way that it can be used in the decades to come is a difficulty that we will need to tackle. The transition to routine capture of impact data not only requires the development of tools and systems to help with implementation but also a cultural change to develop practices, currently undertaken by a few to be incorporated as standard behaviour among researchers and universities. Although some might find the distinction somewhat marginal or even confusing, this differentiation between outputs, outcomes, and impacts is important, and has been highlighted, not only for the impacts derived from university research (Kelly and McNicol 2011) but also for work done in the charitable sector (Ebrahim and Rangan, 2010; Berg and Mnsson 2011; Kelly and McNicoll 2011). There are a couple of types of authorship to be aware of. This petition was signed by 17,570 academics (52,409 academics were returned to the 2008 Research Assessment Exercise), including Nobel laureates and Fellows of the Royal Society (University and College Union 2011). Baselines and controls need to be captured alongside change to demonstrate the degree of impact. (2011) Maximising the Impacts of Your Research: A Handbook for Social Scientists (Pubd online) <, Lets Make Science Metrics More Scientific, Measuring Impact Under CERIF (MICE) Project Blog, Information systems of research funding agencies in the era of the Big Data. The point at which assessment takes place will therefore influence the degree and significance of that impact. Explain. New Directions for Evaluation, Impact is a Strong Weapon for Making an Evidence-Based Case Study for Enhanced Research Support but a State-of-the-Art Approach to Measurement is Needed, The Limits of Nonprofit Impact: A Contingency Framework for Measuring Social Performance, Evaluation in National Research Funding Agencies: Approaches, Experiences and Case Studies, Methodologies for Assessing and Evidencing Research Impact. Cb)5. Times Higher Education, Assessing the Impact of Social Science Research: Conceptual, Methodological and Practical Issues, A Profile of Federal-Grant Administrative Burden Among Federal Demonstration Partnership Faculty, Department for Business, Innovation and Skills, The Australian Research Quality Framework: A live experiment in capturing the social, economic, environmental and cultural returns of publicly funded research, Reforming the Evaluation of Research. Prague, Czech Republic, Health ResearchMaking an Impact. Impact assessments raise concerns over the steer of research towards disciplines and topics in which impact is more easily evidenced and that provide economic impacts that could subsequently lead to a devaluation of blue skies research. In designing systems and tools for collating data related to impact, it is important to consider who will populate the database and ensure that the time and capability required for capture of information is considered. Impact has become the term of choice in the UK for research influence beyond academia. A university which fails in this respect has no reason for existence. If knowledge exchange events could be captured, for example, electronically as they occur or automatically if flagged from an electronic calendar or a diary, then far more of these events could be recorded with relative ease. Every piece of research results in a unique tapestry of impact and despite the MICE taxonomy having more than 100 indicators, it was found that these did not suffice. , . It can be seen from the panel guidance produced by HEFCE to illustrate impacts and evidence that it is expected that impact and evidence will vary according to discipline (REF2014 2012). 0000007777 00000 n The ability to write a persuasive well-evidenced case study may influence the assessment of impact. Downloadable! The current definition of health, formulated by the WHO, is no longer adequate for dealing with the new challenges in health care systems. This database of evidence needs to establish both where impact can be directly attributed to a piece of research as well as various contributions to impact made during the pathway. Muffat says - "Evaluation is a continuous process and is concerned with than the formal academic achievement of pupils. What are the challenges associated with understanding and evaluating research impact? The REF will therefore assess three aspects of research: Research impact is assessed in two formats, first, through an impact template that describes the approach to enabling impact within a unit of assessment, and second, using impact case studies that describe the impact taking place following excellent research within a unit of assessment (REF2014 2011a). Evaluate means to assess the value of something. Dennis Atsu Dake. It is a process that involves careful gathering and evaluating of data on the actions, features, and consequences of a program. As Donovan (2011) comments, Impact is a strong weapon for making an evidence based case to governments for enhanced research support. It is worth considering the degree to which indicators are defined and provide broader definitions with greater flexibility. In putting together evidence for the REF, impact can be attributed to a specific piece of research if it made a distinctive contribution (REF2014 2011a). These sometimes dissim- ilar views are due to the varied training and background of the writers in terms of their profession, concerned with different aspects of the education process. trailer << /Size 97 /Info 56 0 R /Root 61 0 R /Prev 396309 /ID[<8e25eff8b2a14de14f726c982689692f><7a12c7ae849dc37acf9c7481d18bb8c5>] >> startxref 0 %%EOF 61 0 obj << /Type /Catalog /Pages 55 0 R /Metadata 57 0 R /AcroForm 62 0 R >> endobj 62 0 obj << /Fields [ ] /DR << /Font << /ZaDb 38 0 R /Helv 39 0 R >> /Encoding << /PDFDocEncoding 40 0 R >> >> /DA (/Helv 0 Tf 0 g ) >> endobj 95 0 obj << /S 414 /T 529 /V 585 /Filter /FlateDecode /Length 96 0 R >> stream 0000001325 00000 n This is a metric that has been used within the charitable sector (Berg and Mnsson 2011) and also features as evidence in the REF guidance for panel D (REF2014 2012). A discussion on the benefits and drawbacks of a range of evaluation tools (bibliometrics, economic rate of return, peer review, case study, logic modelling, and benchmarking) can be found in the article by Grant (2006). The time lag between research and impact varies enormously. RAND selected four frameworks to represent the international arena (Grant et al. 0000011585 00000 n It is acknowledged in the article by Mugabushaka and Papazoglou (2012) that it will take years to fully incorporate the impacts of ERC funding. What is the Difference between Formative and Summative Evaluation through Example? This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0/), which permits unrestricted reuse, distribution, and reproduction in any medium, provided the original work is properly cited. Also called evaluative writing, evaluative essay or report, and critical evaluation essay . This raises the questions of whether UK business and industry should not invest in the research that will deliver them impacts and who will fund basic research if not the government? In the UK, evaluation of academic and broader socio-economic impact takes place separately. The verb evaluate means to form an idea of something or to give a judgment about something. They are often written with a reader from a particular stakeholder group in mind and will present a view of impact from a particular perspective. However, the Achilles heel of any such attempt, as critics suggest, is the creation of a system that rewards what it can measure and codify, with the knock-on effect of directing research projects to deliver within the measures and categories that reward. 2008), developed during the mid-1990s by Buxton and Hanney, working at Brunel University. In the UK, UK Department for Business, Innovation, and Skills provided funding of 150 million for knowledge exchange in 201112 to help universities and colleges support the economic recovery and growth, and contribute to wider society (Department for Business, Innovation and Skills 2012). Evaluative research is a type of research used to evaluate a product or concept, and collect data to help improve your solution. The process of evaluation involves figuring out how well the goals have been accomplished. HEIs overview. Figure 2 demonstrates the information that systems will need to capture and link. The university imparts information, but it imparts it imaginatively. In line with its mandate to support better evaluation, EvalNet is committed to working with partners in the global evaluation community to address these concerns, and is currently exploring options for additional work. The understanding of the term impact varies considerably and as such the objectives of an impact assessment need to be thoroughly understood before evidence is collated. Although it can be envisaged that the range of impacts derived from research of different disciplines are likely to vary, one might question whether it makes sense to compare impacts within disciplines when the range of impact can vary enormously, for example, from business development to cultural changes or saving lives? Why should this be the case? Any information on the context of the data will be valuable to understanding the degree to which impact has taken place. It is important to emphasize that Not everyone within the higher education sector itself is convinced that evaluation of higher education activity is a worthwhile task (Kelly and McNicoll 2011). 0000004019 00000 n The case study approach, recommended by the RQF, was combined with significance and reach as criteria for assessment. If this research is to be assessed alongside more applied research, it is important that we are able to at least determine the contribution of basic research. Wigley (1988, p 21) defines it as "a data reduction process that involves the . Evaluation is a procedure that reviews a program critically. Findings from a Research Impact Pilot, Institutional Strategies for Capturing Socio-Economic Impact of Research, Journal of Higher Education Policy and Management, Introducing Productive Interactions in Social Impact Assessment, Measuring the Impact of Publicly Funded Research, Department of Education, Science and Training, Statement on the Research Excellence Framework Proposals, Handbook on the Theory and Practice of Program Evaluation, Policy and Practice Impacts of Research Funded by the Economic Social Research Council. An evaluation essay or report is a type of argument that provides evidence to justify a writer's opinions about a subject. In the UK, more sophisticated assessments of impact incorporating wider socio-economic benefits were first investigated within the fields of Biomedical and Health Sciences (Grant 2006), an area of research that wanted to be able to justify the significant investment it received. One of the advantages of this method is that less input is required compared with capturing the full route from research to impact. There is . 2006; Nason et al. 0000342937 00000 n Although based on the RQF, the REF did not adopt all of the suggestions held within, for example, the option of allowing research groups to opt out of impact assessment should the nature or stage of research deem it unsuitable (Donovan 2008). However, it must be remembered that in the case of the UK REF, impact is only considered that is based on research that has taken place within the institution submitting the case study. 2. 0000012122 00000 n It is possible to incorporate both metrics and narratives within systems, for example, within the Research Outcomes System and Researchfish, currently used by several of the UK research councils to allow impacts to be recorded; although recording narratives has the advantage of allowing some context to be documented, it may make the evidence less flexible for use by different stakeholder groups (which include government, funding bodies, research assessment agencies, research providers, and user communities) for whom the purpose of analysis may vary (Davies et al. These case studies were reviewed by expert panels and, as with the RQF, they found that it was possible to assess impact and develop impact profiles using the case study approach (REF2014 2010). Time, attribution, impact. 2009). Clearly there is the possibility that the potential new drug will fail at any one of these phases but each phase can be classed as an interim impact of the original discovery work on route to the delivery of health benefits, but the time at which an impact assessment takes place will influence the degree of impact that has taken place. Aspects of impact, such as value of Intellectual Property, are currently recorded by universities in the UK through their Higher Education Business and Community Interaction Survey return to Higher Education Statistics Agency; however, as with other public and charitable sector organizations, showcasing impact is an important part of attracting and retaining donors and support (Kelly and McNicoll 2011). (2007), Nason et al. Inform funding. The RQF was developed to demonstrate and justify public expenditure on research, and as part of this framework, a pilot assessment was undertaken by the Australian Technology Network. Research findings will be taken up in other branches of research and developed further before socio-economic impact occurs, by which point, attribution becomes a huge challenge. (2007) surveyed researchers in the US top research institutions during 2005; the survey of more than 6000 researchers found that, on average, more than 40% of their time was spent doing administrative tasks. A taxonomy of impact categories was then produced onto which impact could be mapped. The quality and reliability of impact indicators will vary according to the impact we are trying to describe and link to research. It is very important to make sure people who have contributed to a paper, are given credit as authors. In the majority of cases, a number of types of evidence will be required to provide an overview of impact. These traditional bibliometric techniques can be regarded as giving only a partial picture of full impact (Bornmann and Marx 2013) with no link to causality.
Interesting Facts About Leo Constellation,
Camden Aquarium Birthday Party,
Act Of Contrition In Vietnamese,
Articles D