Scientific method Essay Example for Free

Scientific method Essay Policy and practice impacts of research funded by the Economic and Social Research Council A case study of the Future of Work programme, approach and analysis Steven Wooding, Edward Nason, Lisa Klautzer, Jennifer Rubin, Stephen Hanney, Jonathan Grant Policy and practice impacts of research funded by the Economic and Social Research Council A case study of the Future of Work programme, approach and analysis Steven Wooding, Edward Nason, Lisa Klautzer, Jennifer Rubin, Stephen Hanney, Jonathan Grant Prepared for the Economic and Social Research Council. The research described in this report was prepared for the Economic and Social Research Council. The RAND Corporation is a nonprofit research organization providing objective analysis and effective solutions that address the challenges facing the public and private sectors around the world. RAND’s publications do not necessarily reflect the opinions of its research clients and sponsors. R ® is a registered trademark.  © Copyright 2007 RAND Corporation All rights reserved. No part of this book may be reproduced in any form by any electronic or mechanical means (including photocopying, recording, or information storage and retrieval) without permission in writing from RAND. Published 2007 by the RAND Corporation 1776 Main Street, P. O. Box 2138, Santa Monica, CA 90407-2138 1200 South Hayes Street, Arlington, VA 22202-5050 4570 Fifth Avenue, Suite 600, Pittsburgh, PA 15213-2665 Westbrook Centre, Milton Road, Cambridge CB4 1YG, United Kingdom RAND URL: http://www.rand. org/ RAND Europe URL: http://www. rand. org/randeurope. To order RAND documents or to obtain additional information, contact Distribution Services: Telephone: (310) 451-7002; Fax: (310) 451-6915; Email: [emailprotected] org Preface This report, prepared for the Economic and Social Research Council (ESRC), examines the impact of the ESRC’s ‘Future of Work’ programme on policy makers, professional practitioners and other groups outside academia. It also explores the applicability of the ‘Payback Framework’, a conceptual model for research evaluation, to social science. The Future of Work programme was an initiative that aimed to bring together leading researchers in the United Kingdom in an investigation of the future prospects for paid and unpaid work. The first phase of the programme started in October 1998, followed by a second phase in January 2001. The report is presented in two volumes. This volume presents the conclusions of the research and summarises the methods and results. The second volume includes a brief literature review of the evaluation of social science and the influence of research on policy; an overview of the Future of Work programme; detailed analysis of a survey of Future of Work PIs (Principal Investigators); and four complete case study narratives of projects from the programme. The report will be of interest to the ESRC and policy makers in the wider social science and policy community who are interested in how social science informs policy and practice. It will also be of interest to those developing methods to evaluate research. The research was led by RAND Europe in collaboration with the Health Economics Research Group (HERG). RAND Europe is an independent not-for-profit think tank and research organisation that serves the public interest by providing evidence for policy making and public debate. HERG, a Specialist Research Institute of Brunel University, has as one of its main research themes, methodological and empirical studies of the impact of research. This report has been peer reviewed in accordance with RAND’s quality assurance standards (see http://www. rand. org/about/standards/) and therefore may be represented as a RAND Europe product. For more information about RAND Europe or this document, please contact Steven Wooding Senior Policy Analyst Tel: +44 1223 273897 Email: [emailprotected] org RAND Europe Westbrook Centre, Milton Road Cambridge. CB4 1YG, United Kingdom Jonathan Grant Deputy to the President Tel: +44 1223 293 893 Email: [emailprotected] org [emailprotected] org iii Contents Preface iii Overview of impact vii Executive summary ix Acknowledgments.. xiii CHAPTER 1 Introduction 1 CHAPTER 2 Methodology and project structure .. 3 2. 1 The analytical framework . 3 2. 2 Initial tasks .. 5 2. 2. 1 Brief review of social science impacts literature . 5 2. 2. 2 Review of FoW documentation 5 2. 2. 3 Key informant interviews. 5 2. 2. 4 Output of initial tasks 5 2. 3 Payback survey .. 6 2. 4 Interim report 7 2. 5 Case studies. 7 2. 6 Analysis workshop 9 CHAPTER 3 Results . 11 3. 1 Key findings from the literature review and key informant interviews.. 11 3. 2 Summary of results from survey.. 13 3. 3 User interviews 26 3. 4 Case study summaries.. 27 3. 5 Case study A. 27 3. 6 Case study B. 30 3. 7 Case study C 33 3. 8 Case study D 35 3. 9 Concluding comments 37 CHAPTER 4 Discussion.. 39 4. 1 Impact of the FoW programme .. 39 4. 1. 1 The FoW programme has had significant wider impacts on policy and practice 39 v Policy and practice impacts of ESRC funded research RAND Europe 4. 2 4. 3 4. 4 4. 1. 2 What impact has the FoW programme had? 39 4. 1. 3 Why has the FoW programme had an impact? 40 4. 1. 4 What affects the impact of projects .. 41 Applying the Payback Model to wider impacts of social science .. 42 4. 2. 1 The Payback Framework can be applied to social science .. 42 4. 2. 2 Generalisation of categories.. 42 4. 2. 3 General points about assessing impacts of social science . 43 Further research.. 45 Concluding comments. 46 REFERENCES. 47 Reference list.. 49 vi Overview of impact The Future of Work (FoW) programme succeeded in bringing together an interdisciplinary group of academics, stimulating constructive discussions and providing access to policy makers to ensure the wider impact of the research. The FoW programme had significant academic and wider impacts. There was substantial output from the programme in the area of knowledge production. To date this has included 11 books and 69 book chapters; four journal special issues and over 100 peer reviewed articles, alongside over 200 conference presentations. Six of the researchers felt they had changed the direction of their research field. In terms of capacity development, involvement in the FoW programme was seen as a moderate or considerable contributor to 20 academic promotions, and the most common benefit of the programme cited by researchers were the opportunities to meet other researchers and in providing fora for discussion. In our survey of Principal Investigators (PIs) they reported 50 policy impacts, across a range of organisations including national government, political parties, employers and unions. Contributions to the policy debate included more than 60 working papers and official reports; seminars for the DTI, Low Pay Commission and Cabinet Office. More directly there were nine secondments, which placed the researchers in a policy environment, including a senior role in the DTI Women and Equality unit, where the researcher was able to influence strategy and policy decisions relating to equality. Further specific examples of policy impact were: the chairmanship of the TUC Partnership Institute by a researcher, allowing him to impact on employer/union relationships; the drafting of guidance notes on complying with employment legislation for the DTI by a research group; direct input into the Work and Families Bill (2003), which introduced new legislation on maternity and paternity leave; and citation in a House of Lords judgement on pay and conditions, specifically looking at unfair dismissal. Outside the government sectors our case studies identified a number of impacts on employers, including changes in workload policies and career structure; effects on maternity and family friendly working practices in a large consulting organisation and the negotiation of union-employer partnership deals. vii Policy and practice impacts of ESRC funded research RAND Europe Dissemination Almost half of the PIs felt the extensive networks of the Programme Director, and steering committee, had provided them with direct access to policy makers. These policy makers included those in the Work Foundation and a government agency. In addition, the FoW Media Fellow enhanced the impact of the research on more distant policy makers. He achieved this by producing research summaries set in the context of current policy and other research findings. Crucially, he worked to timescales suitable for policy makers rather than those of researchers. Evaluation methodology This evaluation used the Payback Framework as a conceptual structure and showed that the framework is useful for evaluating the wider impacts of social science. We used a number of techniques to collect data for the study: document review; key informant interviews; an on-line survey; and case studies. However, we found that some impacts are inaccessible to evaluation, because of political sensitivity or anonymity guarantees. viii Executive summary We examined how the ESRC Future of Work (FoW) programme influenced policy and professional practice. While doing so we reflected on the methods used to assess and identify impacts. Specifically, we considered whether the Payback Framework, a conceptual model for research evaluation, was appropriate for social science. Here we summarise the key findings. The FoW brought together an interdisciplinary group of academics, stimulated constructive discussion and provided access to policy makers. This may be especially significant given the view expressed by key informants that employment policy and management practices may be especially hard to reach with evidence as they are heavily contextual and apt to be influenced by fashion and ideology. Impacts The FoW programme had significant impacts on knowledge and research. This was evident in the numerous publications and conference presentations attributed to the programme. Most Principal Investigators (PIs) attributed incremental changes in their field of research to their projects, and some attributed a clear change of direction in their field of research to their projects. Most of the projects also influenced other researchers. The FoW programme had significant impacts on public policy. Although some PIs could identify specific impacts of their research, many found it difficult to identify actual policies they had influenced. PIs generally thought they had influenced policy in an incremental way and informed the policy debate. PIs also gave many presentations of FoW research to policy audiences. The FoW programme had significant impacts on career development. More than 75% of PIs thought the FoW programme had helped them to form networks with researchers, policy makers and practitioners; nearly half of PIs attributed career development for researchers to their FoW projects, including nine secondments to government. The FoW programme impacted on the policies and practice of organisations. There were many presentations given in organisations; PIs thought organisational practices were influenced by the research, but only some were easily identifiable. The policy environment determines policy impact. In one case the heightened awareness among policy makers of issues around maternity leave and women returning to work ix Policy and practice impacts of ESRC funded research RAND Europe provided fertile ground for research on how women make these decisions. In a second case the waning interest in union-employer partnerships was thought to have reduced the impact of a TUC institute chaired by a former FoW researcher. In general, the FoW research seldom caused major changes in policy but often resulted in impacts such as stimulating debate, fine-tuning policy, dispelling myths and providing confirmatory support. Dissemination The FoW programme provided access to policy makers. It effectively combined the networks of the Director and steering committee, and provided the researchers access to these networks which included key policy makers in the DTI, Low Pay Commission (LPC) and Cabinet Office. The FoW Media Fellow enhanced the impact on policy makers. This was achieved largely for two reasons. First, because his summaries of the FoW research were produced to a timescale suitable for policy makers, rather than researchers. And second because they were accessible to policy makers: setting the FoW research in the context of other research and current policy discussions. Researchers and policy makers differed in their views on how best to disseminate to policy makers. The two groups consider different channels to be important: researchers favouring academic publications, policy makers favouring the Media Fellow’s publications. The Payback Framework is a useful model for evaluating social science research. The Payback Framework provides a structure for research evaluation. It comprises a logic model of the research and dissemination process and a classification scheme for the immediate and wider impacts of research. This consists of five categories: Knowledge; Impacts on future research; Impacts on policy; Impacts on practice and Wider social and economic impacts. Both the literature review and fieldwork showed that the Framework could be effectively applied to social science research. Impacts and attribution Some impacts may be inaccessible to evaluation, for example some impacts were politically sensitive, so participants requested that they were not discussed. Also, subjects of the original research may have been influenced by their participation in that research, but their identity could not be revealed to the researchers in this evaluation. A confluence of inputs and incremental ‘knowledge creep’ make it difficult to attribute policy change to a given input. The Payback Framework provides a structure in which to explore the context within which projects are developed. However, the incremental nature of policy remains a difficulty in assessing impact at the project level. There are few mechanisms in social science to codify and synthesise research. In contrast to biomedical science, in the fields covered by the FoW programme there are fewer formal mechanisms to systematically review research; these mechanisms can offer tracers of policy influence. x RAND Europe Executive summary Timing Research on impacts may happen too early or too late. If research on impacts occurs too early, some impacts may not yet have occurred. If it occurs too late, certain impacts may have already come and gone. This possibility of transience makes it harder to investigate the impacts, as they may not be captured by a current snapshot of policies and policy debates. In order to provide a comprehensive view of the wider impacts of research this project suggests it would be important to warn researchers at the start of the project about likely evaluations; provide researchers with a mechanism to capture early impacts; and then evaluate research after further impacts have had time to develop, probably 5-10 years after completion of the research. The literature suggests that for research relating to ‘hot topics’ in policy, initial impact is likely to occur earlier and that 2 years post completion may provide the best time frame for evaluation. Implementation of evaluation There was widespread cooperation in the evaluation. The majority of PIs (including all case study PIs), 80% of nominated research users, and others nominated by PIs, agreed to participate in the research when approached. Researchers and users may prefer structured interviews to written surveys. Our experience also suggests that such interviews would provide more useful information for evaluation. Our on-line survey required significantly more of most participants’ time than predicted. xi Acknowledgments This study would have been impossible without the generous support of those involved with the Future of Work programme – we thank them for their constructive criticism and for sparing their time for interviews and to fill in our survey. We would particularly like to thank those researchers whose grants were selected as case studies. We would also like to thank Veronica Littlewood of the Economic and Social Research Council for her help and advice, including at the analysis workshop. Finally, we would like to thank Professor Martin Buxton and Stijn Hoorens who acted as the quality assurance reviewers. xiii CHAPTER 1 Introduction This report explores the wider impacts of social science research: how research affects policy, practitioner behaviour and public opinion. It does this by examining the Economic and Social Research Council’s (ESRC) Future of Work (FoW) research programme. The ESRC is the UK’s largest research funder and training agency addressing economic and social concerns. As such it aims to provide high quality research on issues of importance to business, the public sector and government. The FoW programme set out to bring together leading UK researchers, across a wide range of disciplines, in order to investigate the future prospects for paid and unpaid work. The programme was shaped by a consultation exercise involving 140 policy makers, academics and practitioners, carried out by Professor Peter Nolan in 1997. The first phase of the programme started in October 1998, followed by a second phase in January 2001. The total funding of the programme amounted to ? 4 million and attracted 221 applications for the first phase, of which 19 were supported. A further eight projects were supported in the second phase. The aims of the programme are shown in Box 1. †¢ †¢ †¢ †¢ †¢ †¢ To create the evidence base that would then ground theories of work To enhance public understanding of the critical developments most likely to impact on people’s working lives To deepen accounts of the future of work by systematic mapping of past and present shifts and continuities To foster interdisciplinary and comparative perspectives To use innovative methods to engage with research users To act as a focus for debate within and between the academic, practitioner and policy-making communities Box 1. Aims of the FoW programme. Peter Nolan, Montague Burton Chair of Industrial Relations at Leeds, went on to direct the programme. He encouraged interaction between the research groups, promoted dissemination of programme findings and raised the programme’s profile. The Director was assisted by a programme advisory committee of senior representatives from government, the Trades Union Congress (TUC), business and academia. The advisory group attended meetings, participated in site visits to meet researchers, and provided access to policy networks. The written dissemination activities were led by Robert Taylor, a 1. Policy and practice impacts of ESRC funded research RAND Europe former journalist at the Financial Times, who was appointed as the programme’s Media Fellow. He wrote a series of seven booklets, aimed at policy makers, that described FoW research and set it in context. The output of the programme has been substantial: its outputs so far include 11 books, 69 book chapters, over 100 refereed articles and over 400 media mentions. This study explores the wider impacts of the programme in more detail. Over the past decade there has been an increasing culture of accountability affecting government spending. This climate has led ESRC to investigate the most effective ways to evaluate social science research, and to demonstrate the wider impact of its research on society. This report builds on experience of evaluating research in the health services and biomedical settings and seeks to apply it to social science. In this work we take the Payback Framework, originally developed by the Health Economics Research Group (HERG) at Brunel University, and test its applicability to social science. The Payback Framework was initially developed to examine the payback of health services research (Buxton et al., 1994; Buxton and Hanney, 1994; Buxton and Hanney, 1996). It was further developed in an earlier ESRC analysis of non-academic impact from research (Cave and Hanney, 1996) and subsequently extended to examine basic and clinical biomedical research (Wooding et al. , 2005; Wooding et al. , 2004). This study tested whether the Framework could be applied to examine the payback of social science research. To do this we briefly reviewed the literature on social science evaluation and the common models for examining the impact of evidence on policy, and concluded that they could be aligned with the Payback Framework. We then used the Payback Framework to examine the research projects in the FoW programme. We used the Payback Framework to structure a programme-wide questionnaire and a series of four case studies. Finally, we used these three streams of evidence to summarise the wider impacts of the FoW study and to see what can be learnt from the programme. We also used the evidence to develop a refined Payback Framework and consider its applicability for evaluating the wider impacts of social science research. 2 CHAPTER 2 Methodology and project structure. This project set out to examine the wider impacts of the FoW programme and to test the applicability of the Payback Framework to social science. It used a number of data collection methods (shown schematically in Figure 1). First we conducted a brief review of the literature concerning the ways in which social science affects policy and how the impacts of social science can be assessed. To develop our understanding of the FoW programme we reviewed documents from the ESRC and interviewed key individuals. We then surveyed all the Principal Investigators (PIs) who held grants from the FoW programme  to investigate the wider impacts of their grants, and asked them to nominate a user of their research for follow up via a telephone interview. To examine the pathways to impact in more detail we carried out four case studies of FoW PIs. The data collected throughout the study were then analysed in a one-day workshop. More detail on each of these stages is provided in the sections that follow. Figure 1. Project schematic 2. 1 The analytical framework The analytical framework for the study was based on the Payback Framework. The Payback Framework consists of two elements: a logic model representation of the complete research process (for the purposes of research evaluation), and a series of categories to classify the individual paybacks from research. The logic model and categories of the Payback Framework that served as the starting point for this study are presented below in Figure 2 and Box 2 respectively. The logic model provides a framework for analysing the ‘story’ of a research idea from initial inception 3 Policy and practice impacts of ESRC funded research RAND Europe. (Stage 0) through the research process (Stage 2) into dissemination (Interface B) and on towards its impact on people and society (Stage 6). The model is meant as a research tool to facilitate cross-case analysis. It does this by providing a common structure for each case study thereby ensuring cognate information for each study is recorded in the same place. The model is not meant to imply that the research process itself is linear. If necessary, individual pieces of information can be recorded in more than one place in the Framework to ensure they are picked up in the relevant cross-case comparisons. Figure 2. The version of the Payback Framework used as a starting point for the study1 Knowledge production Journal articles; conference presentations; books; book chapters; research reports Research targeting and capacity building Better targeting of future research; development of research skills, personnel and overall research capacity; staff development and educational benefits Informing policy and product development Improved information bases for political and executive decisions; development of pharmaceutical products and therapeutic techniques Health and health sector benefits. Improved health; cost reduction in delivery of existing services; qualitative improvements in the process of delivery; improved equity in service delivery Broader economic benefits Wider economic benefits from commercial exploitation of innovations arising from RD; economic benefits from a healthy workforce and reduction in working days lost Box 2. The payback categories of the Payback Framework used as a starting point for the study (Source: Hanney et al. , 2004) The categories of the Payback Framework are considered in the Discussion section of this report where the applicability of the model to social science research is discussed. 1 Source: Hanney et al. , 2004 4 RAND Europe Methodology and project structure 2. 2 2. 2. 1 Initial tasks Brief review of social science impacts literature A brief review of the literature, presented in Volume II, examines frameworks of evaluation previously used to examine the impact of social science research as well as models of research impact. The review was intended to identify lessons from the literature that would inform the current study, but was not intended to be comprehensive. 2. 2. 2 Review of FoW documentation At the outset of the project we also reviewed the ESRC records covering the FoW programme. This informed our interview protocol for the key informant interviews and provided the basis of our overview of the FoW programme, which was used as background information throughout the study. This overview is presented in Volume II. 2. 2. 3 Key informant interviews To gain a deeper understanding of the overall context and impact of the FoW programme we carried out interviews with six key informants. Suitable key informants were identified by ESRC and by the Director of the FoW programme: o o o o o o Professor Peter Nolan (FoW Programme Director) John Hougham (Chair of Advisory Board) Professor Toby Wall (Member of panel that appointed the Programme Director and Member of Advisory Panel) Bill Callaghan (Member of Research Priorities Board and Chair of Commissioning Panel) Professor William Brown (PI on two grants, one in each phase of the programme) Robert Taylor (Programme Media Fellow). These interviews were written up and examined for themes relating to the wider impact of social science and the FoW programme. These themes were then clustered and used to inform the refinement of the Payback Framework. We also carried out a second follow-up interview with Peter Nolan late in the project to examine various issues that had been raised in the survey and case study phases of the project. 2. 2. 4 Output of initial tasks The findings of the initial tasks suggested that the logic model aspect of the Payback Framework was generally appropriate for the social sciences. However, the categories needed some generalisation; our initial revision of these categories was presented in our interim report to the ESRC and are shown in Box 3. 5 Policy and practice impacts of ESRC funded research RAND Europe Knowledge production Journal articles; conference presentations; books + chapters; research reports Research targeting and capacity building Sparking new research proposals; providing research training; supporting career advancement Informing policy and product development Raising the profile/awareness of existing research among policy/practitioners makers; dispelling/resisting myths; providing policy options; prioritising areas; designing management assessment tools; developing benchmarking protocols Employment sector benefits. Improved working conditions; higher participation in workforce; more effective regulation Societal and broader economic benefits Lower stress among workers; improved public health; improved mental health through decreased unemployment; greater productivity; improved equity Box 3: Draft Payback Categories for the Social Sciences afer initial tasks. 2. 3 Payback survey To examine the range and types of payback produced across the FoW programme we invited all the PIs to complete an online survey. The survey concentrated on the wider impacts of the projects, but also asked some questions about the initiation of the research. The survey questions were based on those used in previous payback studies and modified in light of the key informant interviews and literature review2. PIs were invited to participate in the survey using personalised emails which contained a direct hyperlink to their questionnaire. PIs who had grants in both phases of the FoW programme received two emails linking to two separate surveys. The survey was implemented using MMIC web questionnaire software. 3 Data were downloaded from MMIC and analysed using SPSS version 14 and Microsoft Excel version 2000. 4 The questionnaire was originally drafted on paper. The paper draft was reviewed by the ESRC and by both of the project’s quality assurance reviewers. After incorporating their comments it was converted into a web questionnaire and again reviewed by the ESRC. We also asked a RAND researcher from outside the project team to test the questionnaire by talking us through their thoughts as they filled it in. This helped us to identify misunderstandings and confusing questions. 2 Payback questionnaires first used in Buxton et al. , 2000 and subsequently refined for payback analysis of the NHS Research Implementation Methods Programme and the Dutch and UK Health.