RAE2001 logo

RAE 3/98

RAE 2001 and health-related research: consultation

To Heads of HEFCE-funded institutions
Heads of DENI-funded universities
Heads of HEFCW-funded institutions
Heads of SHEFC-funded institutions
Of interest to those responsible for Health-related research, Funding
Reference98/3
ResponseBy 16 October 1998
Publication dateAugust 1998
Enquiries toDavid Pilsbury tel 0117 931 7123 e-mail d.pilsbury@hefce.ac.uk


Executive summary

Purpose

1. This document seeks views on proposals for assessing health-related research in the next Research Assessment Exercise (RAE) in 2001.

Key points

2. Proposals were drawn up by a joint task group of the Higher Education Funding Council for England and the Department of Health, in association with the funding bodies for Scotland, Wales and Northern Ireland.

3. The funding bodies announced generic changes to the process for the next assessment exercise in RAE 1/98 'RAE 2001: key decisions and issues for further consultation'. This paper should be read in conjunction with that document.

4. Issues for consultation are highlighted throughout the text and summarised in Annex A. Responses will form the basis of the task group's advice to the four funding bodies.

Action required

5. The task group wishes to seek views from all organisations with an interest in health-related research and the RAE. This document is being sent to all higher education institutions in the UK, and to other relevant bodies (a list is available from the HEFCE). In particular, responses are sought from institutions that submitted to UOAs 1,2,3,4,5, 9, 10 or 11 in the 1996 RAE, or are planning to do so in 2001. Comments from other organisations are also welcomed; additional copies of the document are available free from the HEFCE, or on its web site: http://www.hefce.ac.uk.

6. Responses should be returned by 16 October 1998 to:

David Pilsbury
HEFCE
Northavon House
Coldharbour Lane
Bristol BS16 1QD
Introduction

7. After the 1996 Research Assessment Exercise (RAE), the four funding bodies for UK higher education published a consultation paper on the future of research assessment, 'Research Assessment: Consultation' (RAE 2/97).

8. Following that consultation, the funding bodies published a joint circular (RAE 1/98) which announced key decisions for the framework and operation of the next RAE in 2001, and consults further on issues of detail.

9. Many of the generic changes to the processes for the RAE will benefit the assessment of health-related research (see paragraphs 21-31).

10. However, there remain some issues that are specific to the assessment of health-related research, and this paper seeks views on how they might be addressed. This paper should be read in conjunction with RAE 1/98.


1 The Higher Education Funding Council for England, the Higher Education Funding Council for Wales, the Scottish Higher Education Funding Council and the Department of Education for Northern Ireland.


Issues

11. A number of higher education institutions (HEIs) expressed concerns about the relationship between assessment and funding in the medical units of assessment (UOAs 1-3). The broad coverage of these UOAs means that diverse research groups are often submitted together, which may impact adversely on the rating achieved. In addition, the very large numbers of staff submitted means that a small change in the quality rating translates into large changes in funding. Also, there was concern that the volume of research submitted for assessment, particularly in UOA 3 (over 4,000 submitted researchers), placed undue time and workload pressure on those making the assessments.

12. In addition, there was a general concern expressed to the Department of Health (DH) and the National Health Service (NHS) that the RAE had not always enabled panels to take fully into account the nature of the submissible research carried out in universities which was of direct relevance to the health service. This applies to a number of health-related areas (UOAs 1,2,3,4,5,9,10,11 - see Annex D).

Task groups

13. In response to these and other concerns, the DH and the HEFCE agreed to establish two task groups. The Task Group on Health-related Research was charged with reviewing how the RAE might deal with health-related research in relation to units of assessment, criteria for assessment and membership of assessment panels. It was also asked to consider whether special measures were required to appropriately assess research of direct relevance to the health service, particularly in rapidly developing and strategically important areas such as primary care and health services research.

14. The membership of the Task Group on Health-Related Research is given in Annex B; the terms of reference are in Annex C.

15. The other task group was established in part in response to the recommendations of the Richards Report on Clinical Academic Careers (CVCP, July 1997). It is chaired by Professor Alasdair Breckenridge, who also chairs the funding bodies' Joint Medical Advisory Committee (JMAC). This group is considering how best to take account of the interdependency between research, teaching and patient care in the funding of university medical schools. In particular, it will investigate what steps universities and NHS employers could agree to take in order to reconcile competing pressures on clinical academics. It will also consider the impact of HEFCE and DH funding decisions on university medical schools and NHS service providers. Further information on the task group is available from David Noyce at the HEFCE on 0117 931 7349, e-mail d.noyce@hefce.ac.uk.

16. The Task Group on Health-related Research met twice between autumn 1997 and spring 1998 to agree its terms of reference, consider papers and develop proposals that could be consulted upon. These discussions included reviewing the generic changes and developments proposed for the next RAE, and considering to what extent they might meet the concerns noted above. The DH and NHS interests arise from their strategic aims, and the need for appropriate research capacity to facilitate high quality research with utility for the health services.

17. The task group proposed a definition of the field to indicate the range and complexity of the issues it raises. This is given below in paragraph 32.

18. Paragraphs 21-31 of this paper set out the generic changes planned and under consideration for the operation of the next RAE. Clearly they have positive implications for health-related research as well as other research areas.

19. Paragraphs 32-47 of the paper describe the specific issues arising in respect of the assessment of health-related research. In all cases comment and proposals are sought, though in some instances preferred options are indicated.

20. Annex A of this paper sets out the key issues on which views are sought, and the timetable for responses. Responses will form the basis of advice from the task group to the funding bodies, which may in turn inform their planning of the detailed aspects of the RAE and the operation of panels.

Generic changes planned by the funding bodies for the RAE in 2001

Appointment of panels and panel chairs

21. There have been concerns that 'rolling forward' two-thirds of panel members from one RAE cycle to the next has emphasised continuity rather than dynamism. In response, the funding bodies have announced that for the next RAE panel members will only be allowed to serve in two successive exercises, and panel chairs in a maximum of three (but only twice as chair). This includes service on panels for any new or reconfigured UOAs. Further, panel chairs will be elected by the membership of outgoing panels rather than nominated by the outgoing chair alone. This will reduce any perceived bias in the appointment of panel chairs, and any tendency for panel membership to reinforce research orthodoxies.

22. For the 1996 RAE, we invited almost 1,000 bodies to make nominations to panels. They included subject associations, learned societies, professional bodies and those representing industrial, business and other users of research and other interested parties. We intend again to invite these organisations to nominate panel members. We have also issued a consultation document (RAE 2/98) that seeks comments on whether and how this list should be updated. Respondents are asked to identify organisations which should be consulted but are not on the list, and the reasons for their inclusion.

Arrangements for inter-panel consistency

23. To address concerns about the consistency of marking, umbrella groups will be created comprising the chairs of panels in related areas. These groups will confer at both the criteria setting stage and during the assessment process. In addition they will provide a forum in which panel chairs will be required to defend the top ratings they intend to award.

Membership of panels

24. The funding bodies will consult widely to secure nominations for panel members and will publish the names of those appointed. There will also be measures to ensure that the user perspective is appropriately represented in the assessment process, and panel membership will include the users of research.

25. The user community will be broadly defined and particular consideration will be given to the most effective way of including users. This may include a specific role in setting assessment criteria and the development of special user sub-panels.

Sub-panels

26. As in the 1996 RAE, sub-panels will be used for the assessment of specialist research areas within a UOA that do not themselves justify the creation of a separate UOA and panel. However, the funding bodies have already agreed and announced that sub-panels will not themselves carry executive power, but will act in an advisory role to the main panel. Sub-panel membership will be decided with the same degree of openness as the membership of main panels. Their relationship to main panels, including the weight that their advice will carry, will be made explicit.

27. Where panels consult specialist advisers during the assessment process they will be encouraged to seek them first from the pool of individuals nominated to sit on panels. However, it is expected that an increase in the use of sub-panels would reduce the need to consult ad hoc specialist advisers.

Assessment criteria

28. In response to concerns that the core criteria in the 1996 RAE were not always applied in a way that was transparent, reasonable and appropriate for different UOAs, the funding bodies will facilitate panels building subject-specific criteria around a common framework, and through consultation with their subject communities. Moreover, they will promote and enable interaction between panels in both the planning and assessment stages, to prevent unwarranted variations in the practices of panels assessing similar work. The process of criteria-setting will begin much sooner than for the 1996 RAE, with a view to publishing draft criteria two years before the submission date. Consultation will be facilitated between panel chairs in related groups, in setting criteria and during the assessment phase, to ensure that panels in similar subjects apply their criteria and the rating scale consistently.

Interdisciplinary research

29. The funding bodies are jointly conducting a study of interdisciplinary research in relation to the RAE. Its conclusions will inform further debate from the autumn. In the meantime, the following approaches are being taken forward:

  1. An additional submission form on which interdisciplinary submissions are flagged. This will identify all panels to whom the submission should be referred and, by including a brief description of the activity, provide a better contextual basis for assessment.
  2. A secretarial mechanism to facilitate and catalogue the cross-referrals requested in submissions, as well as those made at panels' discretion.
  3. Ensuring that panels are broadly based; include people who themselves research in interdisciplinary fields or teams; and articulate explicit criteria and working methods for the treatment of interdisciplinary research.

Collaborative research

30. For the 1996 RAE, HEIs were able to make joint submissions where collaborative research groups existed between one or more HEIs. Moreover, HEIs were able to refer to collaborations within and between institutions in the textual parts of submissions. This facility will be retained but the funding bodies are considering what further mechanisms can be developed to facilitate the submission of collaborative research in the RAE.

Summary

31. A number of generic changes are planned for the 2001 RAE which respond to the increasing complexity of the research landscape. These process changes will be at least as beneficial to the assessment of health-related research as to any other subject area. However, there remain some problems that are specific to health-related research. These relate primarily to the volume of work being assessed (with the consequent implications for funding), and the need to ensure appropriate assessment of research of direct relevance to the health service, particularly in rapidly developing and strategically important areas such as primary care and health services research. It should be noted that the NHS and DH spend over £450 million per year on health-related research, primarily funded through a levy on the patient care budget.

Specific issues in health-related research

Health-related research - the scope

32. The task group interpreted its terms of reference as to develop proposals to ensure that all research of importance to the NHS is appropriately valued by the relevant assessment panels. Its area of concern therefore encompasses all clinical research across all relevant professional disciplines, including research which seeks knowledge of the optimal way to provide care to patients. It includes all applied clinical research, research involving new health technologies, and research into service delivery and organisational issues, across all relevant professional disciplines. It includes research using a range of methodologies, both qualitative and quantitative, and which seeks to explain as well as measure the effect of health services on health processes and outcomes. It excludes audit, operational and market research, routine testing, material prepared primarily for teaching, and simple collections of existing research with no original component.

RAE Units of Assessment

33. After full discussion - including HEFCE receiving the views of representative bodies, 1996 RAE panel members and other interested parties - the task group concluded that there was no case for a major restructuring of the existing UOAs or for developing additional UOAs, for example for health services research. The overwhelming view was that improving the processes within the existing structure would be more beneficial. In particular, it concluded that although dividing some UOAs into smaller units, particularly UOAs 1-3, could address some of the concerns about size and range, it would militate against assessing the interdisciplinary nature of much of the research submitted in these UOAs; therefore it was not recommended.

34. However, the use of sub-panels was considered worthy of further debate. Such a development would address many of the concerns raised, without affecting the existing unit structure, and would therefore provide comparability with previous RAEs.

35. The task group seeks responses to the following questions:

  1. Are sub-panels an appropriate way to address many of the concerns previously expressed about the assessment of health-related research?
  2. If so, should any of the main panels still be reconfigured?
  3. If sub-panels are not appropriate, what other specific changes should be made, and what is the rationale for the proposed (re)configuration and/or (dis)aggregation or other change?

Sub-panels: configuration and membership

36. Noting that sub-panels provide a relatively straightforward solution to the volume problems in the three clinical units of assessment, the task group discussed the specific use of sub-panels to cover health-related research. These sub-panels would need to include assessors with relevant expertise and experience. The task group seeks views on the proposals for the number and range of sub-panels. A balance needs to be struck between the risks of disaggregation if there are too many panels, and failure to address the issues raised following the 1996 RAE if there are too few. As an aid to debate, Annex E shows one approach to defining sub-panels for UOA 3 (hospital-based clinical subjects).

37. The task group seeks views on:

  1. How many sub-panels are appropriate for each panel, given that too many might fragment the assessment process, while too few would fail to address the issues raised following the 1996 RAE?
  2. What is the most appropriate way of constructing sub-panels to ensure appropriate expertise is represented?

38. In keeping with the funding bodies' general proposals for the next RAE, where sub-panels are formed they should be chaired by a member of the main panel and should include those competent to assess in the relevant sub-panel area. The task group recommends that sub-panel members should include a range of interests, including the DH and the NHS. In the past the DH has had observer status on some panels, as have a number of other organisations. The group recommends that when such nominees are selected to join panels or sub-panels through the general nominations process, they should play an active and useful part in the criteria setting and assessment process, where previously they had no active role.

39. Paragraph 22 above details the consultation process (RAE 2/98) whereby we are updating the list of organisations from whom panel nominations are sought.

40. The task group seeks views on:

  1. Which bodies representing the users of research submitted in UOAs 1,2,3,4,5,9,10,11 should be invited to make nominations to panel membership.
  2. How to ensure that, in keeping with the approach to sub-panels generally in the RAE, user representatives play a full part in the operation of the health-related panels.
Sub panels: role

41. The group considered how sub-panels in health-related UOAs could be used to reach an overall view about the quality of submissions. Clearly, any submission could include research falling in one or several of the sub-panels for the UOA. However, the funding bodies have already announced that sub-panels will not carry executive power, but will act in an advisory role to the main panel.

42. The group's recommendation on the use of sub-panels' advice in agreeing ratings for each submission is influenced by the condition that there should be no special treatment for health-related UOAs within the agreed RAE framework. The proposal develops the de facto practice of many panels in previous RAEs, whereby they built up the final rating for a submission from a judgement about the quality of the sub-areas within it. Thus, each sub-panel reporting to a main panel could advise the panel on an indicative rating for the sub-area it had assessed. The main panel, which would comprise at least the chair of each sub-panel, would decide upon a rating for entire submissions, informed by the sub-panel recommendations.

43. However, the additional possibilities created by this more structured approach are for sub-panel ratings, as agreed by the main panels, to be made publicly available; and, as a separate issue, to be made available to the funding bodies to consider in allocating funding. The latter approach would have the advantage of reducing the funding turbulence associated with rating changes in the large UOAs, but would require a robust mechanism for attributing volume measures to one of the sub-panels.

44. The task group seeks views on:

  1. Whether sub-panels should be created to advise the panel on ratings for sub-areas within a submission.
  2. Whether sub-panel scores agreed by the main panel should be made publicly available.
  3. Whether the sub-panel scores agreed by the main panel should be considered by the funding bodies in allocating funding.
  4. What might be a robust approach to attributing volume measures to sub-panels.

Assessment of research of direct relevance to the health service

45. The task group considered the operation of the RAE in assessing research relevant to the health service, particularly in areas like health services research where there was increasing investment in good quality research, and the area itself was improving. Since the RAE gives considerable weight to track record, including evidence of past publications, it has been argued that it might unreasonably disadvantage 'relevant', new or developing subject areas.

46. The funding bodies have announced that panels will consult subject communities in setting criteria relevant to the subject area, within a broad common framework. Membership of panels will be broader, with a more clearly defined role for user representatives. Panels will also be encouraged to specify a structure to the textual parts of submissions about research culture and plans. Further, there will be a number of mechanisms to facilitate the assessment of interdisciplinary research. Finally, panels will be required to provide some feedback on each submission against the published criteria, as well as a brief generic report on the unit of assessment. The task group concluded that these process amendments were probably sufficient to improve the assessment in particular of 'relevant' research areas, particularly those that are developing rapidly and/or interdisciplinary in nature.

47. The task group seeks responses to the following:

  1. Are the proposed changes sufficient to ensure appropriate assessment of research that is directly relevant to the health service?
  2. Will the changes take account of the rapid development of many of these research areas?
  3. If not, what other approaches could be adopted that would be applicable to all UOAs?
  4. How should the panels provide feedback to institutions so that it is useful to them, but does not seek to (re)define research strategies?


Annexes

Annex A - Timetable for the consultation and summary of issues

Annex B - Membership of the Joint Task Group on Health-related Research

Annex C - Terms of reference for the joint task group

Annex D - Units of Assessment in the RAE 1996

Annex E - Possible sub-panels of the hospital-based clinical panel

[ Home | About the RAE2001 | Panels | Guidance for panel members | Guidance for institutions | Data collection | Publications | Contacts ]