RAE logo

Report of the work of the Joint Funding Bodies/CBI Task Group on User/Industrial Participation in the RAE 2001

Introduction

1. This group was convened as one of several to advise on particular aspects of the 2001 RAE. The Research Assessment Exercise (RAE) is conducted jointly by the Higher Education Funding Council for England (HEFCE), the Higher Education Funding Council for Wales (HEFCEW) the Scottish Higher Education Funding Council (SHEFC) and the Department of Education for Northern Ireland (DENI). A number of groups have been established by the funding bodies to look at issues of panel membership, criteria and assessment processes, and the inclusion of industrial and user perspectives in the Research Assessment Exercise (RAE). This group was established with the Confederation of British Industry (CBI) to look specifically at these questions in relation to research relevant to industry or other users of research.

Terms of reference

2. The terms of reference of the group were to consider how the RAE might best involve user or industrial representatives, and in particular to consider:

  • How to ensure that the quality in relevant research is properly assessed.
  • The nature of the panel membership and process for appointment of user or industrial representatives.
  • How to ensure that panel working methods capture user and industrial perspectives, including exploring the appropriateness of developing sub-panels for user or industrial representatives.

3. The membership of the group is at Annex A.

Panel structure

4. The group gave careful consideration to the most effective panel structure to facilitate the involvement of users. The group considered that user representatives should have full and equal membership of panels. However, it was accepted that serving on sub-panels would probably involve less work for individual users and this might facilitate the participation of more of those working for private-sector organisations. It was acknowledged that the Built Environment panel successfully employed a user sub-panel for the 1996 RAE.

5. The group noted that full panel members participated in setting criteria and working methods for the panel, and in the final decision making process, whereas all sub-panel members did not. It was felt that without participating in setting criteria and working methods, the role of users would remain peripheral to the RAE process. It was therefore recommended that wherever possible user involvement should take the form of full membership of the main panel such that users would participate in all processes and decisions of the panel. In order to undertake this role these 'user representatives' should be competent to contribute to the assessment process. Where panels felt the need for further advice they might still establish sub-panels or consult appropriate specialist advisers, but these should not be seen as substitutes for having users as full and equal members of the panel. The group noted, however, that the funding councils had already decided that any sub-panel set up by a main panel in RAE 2001 should usually include at least two members of the main panel. The group also recommended that in science, engineering and certain social sciences panels, user representatives should normally make up approximately 25 per cent of the membership. It was recognised that this proportion might be exceeded in particular units of assessment.

Panel membership

6. The group sought to identify and overcome the potential barriers to effective user participation in the exercise. Among those noted were the heavy time commitment required to be a panel member; a possible lack of understanding of the benefits of user participation in the exercise; and likely difficulty in accessing the cited research outputs.

7. The group concluded that the commitment of time required could not be substantially reduced without undermining the principle of equality between user and academic panel members. It regarded this as essential to the credibility of user involvement in the RAE, and hence to the credibility of the proper assessment of relevant research.

8. User representatives might not have easy access to the resources of academic libraries in order to read and assess work submitted in the form of journal articles or other academic publications. The group considered requiring all HEIs to send copies of the work submitted, to be held centrally and made available to panel members. However it noted that there was no facility to receive, organise and make available such a central library of submitted items. To provide such a facility was deemed not to be cost-effective. The group recommended that the RAE team should consider how it could reduce the burden on all panel members, particularly the burden of administrative tasks, with special regard to the needs of user representatives. User representatives should be provided by the RAE team with copies of all the material they required in order to undertake the assessment process.

9. The group also recommended that a guide to the RAE should be drawn up for the benefit of potential industrial or other user panel members. This could be circulated widely, including by the CBI among its membership, to provide information about the RAE process and encourage the nomination of suitable panel members. This guide is at Annex B. The group was also pleased to report that a substantial number of suitable industrial and other user nominees had been identified so that the number of panels with user participation in the 2001 exercise is three times as great as in 1996, and the overall proportion of user membership across the panels as a whole is doubled. Further details of user membership in the 2001 RAE are at Annex C.

Assessment process and criteria

10. The group noted that the outputs of research relevant to industrial or other users may often take the form of a patent, a new product or other novel output, rather than a publication. While research outputs such as patents and documented products have always been eligible for consideration in the RAE, there has not always been confidence among industrialists, users and researchers that they have been appropriately assessed.

11. Equally, while information on research income from industrial sources has been collected alongside income from Research Council sources, there has not always been confidence that panels have treated it appropriately when forming their judgements.

12. The group noted that where the outputs of research undertaken for industry or government took the form of reports or analyses, these were sometimes confidential to the body which had commissioned them.

13. The group recommended that while the RAE should remain concerned solely with assessing the quality of research, the assessment process should be flexible enough to identify that quality, regardless of the form of the outputs. The means of assessing quality vary considerably between disciplines, and was therefore largely a matter for panels. It was noted that a number of panels had dealt successfully with novel outputs in RAE 1996 and there was therefore every reason to be confident that all panels would be able to accommodate such outputs for RAE 2001. The group noted that the success of a product could not necessarily be taken as evidence that it embodied the outcomes of high-quality research, nor could the converse apply. For instance, a patent which had never been used for the manufacture of an item might nevertheless embody research of the highest quality.

14. The group strongly endorsed the principles of parity and neutrality stated in the Guidance on Submissions (RAE 2/99), particularly the principle of parity. Application of these principles would ensure that all information about research income was appropriately used by the panel, regardless of the source of the income.

15. The access of panels to confidential materials presented generic problems. The group observed that the panel would be interested only in the quality of the research evident in a confidential report, not the confidential information itself. This would reassure providers of this type of material that they could release material to the panel without compromising the confidential nature of the work. Responsibility for making the material available, and for agreeing with the research partner that the material might be submitted, should rest with the submitting HEI. The panel would not accept such material without the written agreement of the industrial or other research partner.

16. The group recommended that where an HEI wished to include confidential material in its submission, it should provide the panel with a letter from the body to whom the work was confidential, confirming that it would be made available and setting the work in the context in which it had been commissioned and used. The group also recommended that further consideration should be given to the treatment of confidential material by panels within the framework set out by the group. This would ensure that users could have confidence in the arrangements made for handling and assessing of confidential material.

17. As a follow-on to the work of the task group, a seminar on Industry, Commerce and the RAE is to be held. This will be co-hosted by the DTI, CBI and the funding bodies. The seminar will give representatives of RAE panels the opportunity to discuss the application of their criteria and working methods to the assessment of industrial and commercial work. It will be based around case studies on collaborative research and the research funding processes used in industry and commerce. Each RAE panel will be invited to send up to two representatives.

Summary of recommendations

18. Wherever possible user involvement should take the form of full membership of the main panel such that users shall participate in all processes and decisions of the panel. In order to undertake this role these 'user representatives' should be competent to contribute to the assessment process. Where panels feel the need for further advice they might still establish sub-panels or consult appropriate specialist advisers, but these should not be seen as substitutes for having users as full and equal members of the panel.

19. In science, engineering and certain social sciences panels, user representatives should normally make up approximately 25 per cent of the membership.

20. The RAE team should consider how it could reduce the burden on all panel members, particularly the burden of administrative tasks, with special regard to the needs of user representatives. User representatives should be provided by the RAE team with copies of all the material they required in order to undertake the assessment process.

21. A guide to the RAE should be drawn up for the benefit of potential industrial or other user panel members. This could be circulated widely, including by the CBI among its membership, to provide information about the RAE process and encourage the nomination of suitable panel members.

22. While the RAE should remain concerned solely with assessing the quality of research, the assessment process should be flexible enough to identify that quality, regardless of the form of the outputs.

23. Where an HEI wishes to include confidential material in its submission, it should provide the panel with a letter from the body to whom the work is confidential, confirming that it will be made available and setting the work in the context in which it has been commissioned and used.

24. Further consideration should be given to the treatment of confidential material by panels within the framework set out by the group. This will ensure that users can have confidence in the arrangements made for handling and assessing of confidential material.


  • Annex A - Task Group Members
  • Annex B - The RAE Briefing Paper
  • Annex C - User representation on RAE panels

Last updated 19 June 2000

[ Home | About the RAE2001 | Panels | Guidance for panel members | Guidance for institutions | Data collection | Publications | Contacts ]