Section 2

Section 1: Overview of the Research Assessment Exercise

Purpose and Conduct of the RAE

1.1. The primary purpose of the 2001 RAE is to produce ratings of research quality which will be used by the higher education funding bodies in determining the main grant for research to the institutions they fund with effect from 2002-03. The funding bodies may also use the exercise to inform policy development. The RAE is conducted jointly by the Higher Education Funding Council for England (HEFCE), the Scottish Higher Education Funding Council (SHEFC), the Higher Education Funding Council for Wales (HEFCW) and the Department of Education for Northern Ireland (DENI). The exercise is managed by the RAE Team, based at the HEFCE offices, on behalf of the four funding bodies. All UK HEIs receiving funding from these bodies are eligible to participate.

1.2. The assessment process is based on peer review and is not mechanistic. Panels will use their professional judgement to form a view about the overall quality of the research activity described in each submission in the round, taking into account all the evidence presented. They will form their judgements in the context of statements of criteria for assessment which they will prepare and publish in advance. The statements of criteria will be developed from a generic template provided by the funding bodies and in consultation with subject communities.

1.3.The other principles by which the RAE is governed are:

  1. Clarity.
    A large and complex body of regulations and procedures governs the RAE. In addition, anecdote, partial information and speculation surround the exercise. All written documents and statements about the RAE, therefore, will be clear and consistent and misinformation about the RAE will be challenged wherever possible by unambiguous statements of the factual position. This principle will guide both the preparation of formal written documents and the advice and interpretation provided by the RAE Team. Similarly it will be a key element in preparing statements of criteria and working methods.
  2. Consistency.
    Assessments made through the RAE should be consistent especially across cognate areas and in the calibration of quality ratings against international standards of excellence.
  3. Continuity.
    The RAE develops through an evolutionary process, building on and learning from previous exercises. With each successive exercise a balance has to be struck between continuity and development. In general, changes are only made where they can bring demonstrable improvements which outweigh the cost of implementing them.
  4. Credibility.
    As was demonstrated through responses to the funding bodiesí consultation on the conduct of the RAE, the methodology, format and processes employed in the exercise are credible to those being assessed. It is essential that this credibility be maintained.
  5. Efficiency.
    Previous RAEs have been extremely cost-effective given the value of public funds distributed according to their ratings. The cost of the exercise, including the cost to HEIs, should continue to be the minimum consistent with a robust and defensible process.
  6. Neutrality.
    The RAE exists to assess the quality of research in HEIs. It should carry out that function without distorting what it is measuring. In other words, the RAE should not encourage or discourage any particular type of activity or behaviour, other than providing a general stimulus to the improvement of research quality overall.
  7. Parity.
    The RAE is concerned only with assessing the quality of research of participating HEIs, regardless of its type, form or place of output.
  8. Transparency.
    The credibility of the RAE is reinforced by transparency about the process for making decisions. This principle has already been used throughout the consultation process and associated work. Except where there is a need to preserve confidentiality (for example in panelsí discussions or when dealing with the names of nominees for panel membership or with the strategic research plans of institutions) all decisions and decision-making processes will be explained openly.

1.4. The funding bodies have consulted widely on the format and conduct of the RAE in preparation for the 2001 exercise. An initial consultation round was announced in November 1997 in "Research Assessment: Consultation" (RAE 2/97). The outcomes of that round, with follow-up questions, appeared in July 1998 in "Research Assessment Exercise 2001: key decisions and issues for further consultation" (RAE 1/98). Consultation has also taken place on the bodies to be invited to nominate panel members (RAE 2/98) and on arrangements for assessing health-related research (RAE 3/98)1. The detailed arrangements for the 2001 RAE have been developed taking account of the outcomes of this consultation.

1.5. The 2001 RAE will follow broadly the same approach as previous exercises. Eligible HEIs will be invited to submit their research activity for assessment. Submissions will be made to a number of subject-based Units of Assessment (UoAs). The information supplied by HEIs will provide the basis for peer review assessment of research quality by specialist panels. Submissions will be in a standard form which includes both quantitative and descriptive elements. As well as providing information specifically requested by the funding bodies, institutions will have the opportunity to draw panelsí attention to any matters which they consider particularly significant to judgements of research quality. All information submitted will be subject to audit by the funding bodies.

1.6. The census date for the exercise is 31 March 2001. Institutions are invited to provide information on staff in post on that date and on publications and other forms of assessable output which they have produced during the assessment period. The assessment period is 1 January 1994 to 31 December 2000 for arts and humanities subjects (UoAs 45 to 67 inclusive), and 1 January 1996 to 31 December 2000 for other subjects (UoAs 1 to 44, 68 and 69). All information must be submitted by 30 April 2001. The full timetable for the exercise is set out in Section 6.

1.7. In order to ensure that institutions that have invested in developing high quality research activity, and will continue to do so, are not disadvantaged by staff leaving immediately before the RAE, a new category of staff, A*, will be introduced. This will apply to staff who transfer between HEIs which are eligible to participate in the RAE within the twelve months preceeding the census date. It will apply to both the old and the new employing institution. Category A* staff will be taken into account in the judgement of quality for both institutions but will only be counted for funding volume purposes in the employing institution at the census date.

1.8. Institutions need put forward for assessment only the work of academic staff who are actively engaged in research. It is for institutions to decide which individuals they choose to submit. They are asked to provide summary information about any academic staff whose work is not submitted for assessment to inform panels about the context in which submitted research has taken place. The number and proportion of staff submitted will be included in the published RAE results.

1.9. Institutions are invited to list up to four items of research output for each member of staff whose research is to be taken into account (up to two items for Category A* staff). These may be any form of available assessable output. All forms of output will be treated on an equitable basis: panels will be concerned only with assessing the quality of research presented.

1.10. Institutions must be able to make available, or arrange access to, any item of output cited. The RAE Team will request from institutions all work which is not reasonably readily available to panel members. Work requested must be made available to the RAE Team within three working days. If an HEI cannot make available or arrange access to an item of output that item will be discounted in the panelís assessment.

1.11. Proper account will be taken of the complete range of research. Panels will give full recognition to work of direct relevance to the needs of commerce and industry, as well as to the public and voluntary sectors. All research, whether applied, basic or strategic, will be given equal weight: panels will be concerned only with the quality of the work submitted for review.

Definition of Research

1.12. The definition of research which applies in the exercise is:

ĎResearchí for the purpose of the RAE is to be understood as original investigation undertaken in order to gain knowledge and understanding. It includes work of direct relevance to the needs of commerce and industry, as well as to the public and voluntary sectors; scholarship*; the invention and generation of ideas, images, performances and artefacts including design, where these lead to new or substantially improved insights; and the use of existing knowledge in experimental development to produce new or substantially improved materials, devices, products and processes, including design and construction. It excludes routine testing and analysis of materials, components and processes, e.g. for the maintenance of national standards, as distinct from the development of new analytical techniques. It also excludes the development of teaching materials that do not embody original research.
* Scholarship for the RAE is defined as the creation, development and maintenance of the intellectual infrastructure of subjects and disciplines, in forms such as dictionaries, scholarly editions, catalogues and contributions to major research databases.

Units of Assessment

1.13. The Units of Assessment for the 2001 RAE are the same as those used in 1996 with the exception that, following consultation, the separate Biochemistry UoA has been discontinued. All UoAs will be described in a standard format which indicates the key subject areas they encompass. These descriptions will be published alongside the statements of panel criteria and working methods before the end of 1999. A list of UoAs is given in Section 5.

Assessment Panels

1.14. The membership of the assessment panels, the method of their selection, and an account of their role and operation, will be set out in RAE 3/99. Where appropriate, the membership of panels has been designed to enable the full participation of users of research including those from industry, the voluntary sector, the professions and Government. This broadening of the expertise available to panels will assist in demonstrating that the full range of research submitted is properly assessed.

1.15. The criteria and detailed working methods that each panel will apply to the assessment process will be composed during 1999. These will include the arrangements for using sub-panels and specialist advisers (if any). Criteria and working methods will be developed within a common framework, while recognising the particular characteristics of each discipline. The establishment of criteria will involve further consultation with subject communities including HEIs. The funding bodies will publish the panelsí criteria and working methods before the end of 1999.

Rating Scale

1.16. While the method of assessment used by panels will reflect the nature of the subjects they are assessing, their ratings will indicate quality of a broadly similar level across all subjects. As in previous exercises, therefore, the panelsí ratings of research quality will be expressed in terms of a standard scale with common definitions of the points. The rating scale and definitions of points are given in Section 2.

1.17. As already announced by the funding bodies in RAE 1/98, the same rating scale will be used as in 1996. The definitions of the points have been amended, following consultation with a group of former panel chairs, in order to give greater clarity. In particular, references to sub-areas have been removed from the definitions. Panels will reach a judgement on the balance of quality across all activity submitted. Within their submissions, institutions will be able to describe the configuration of their research groups and structures. As in previous exercises, panels will be able to indicate research groups which are identified as having a quality of research which is significantly above the overall rating for a submission.

1.18. Submissions which contain no evidence of research will not be awarded a rating.

Interdisciplinary Research

1.19. In preparing for the 2001 RAE the funding bodies commissioned a study of the treatment of interdisciplinary research in the 1996 exercise. The conclusions of that study were published in "Interdisciplinary Research and the Research Assessment Exercise" (RAE 1/99). Taking the findings of the study into account, the funding bodies will ensure that all panels have appropriate mechanisms in place to assess interdisciplinary research. Details of the arrangements are set out in Section 4.

Multiple Submissions

1.20. Normally there should be only one submission per UoA per institution. In exceptional circumstances, however, an institution may be allowed to make more than one submission to the same UoA. The prior permission of the funding bodies will be required by any institution wishing to make a multiple submission. Separate guidance on the circumstances in which multiple submissions may be allowed, and the arrangements for applying for permission, will be issued at a later date.

1.21. Institutions which have merged during the assessment period should normally make a single submission. If they wish to make a multiple submission the prior permission of the funding bodies will be required.

Joint Submissions

1.22. Where they are engaged in collaborative work, two or more institutions may make a joint submission to a UoA if they choose to do so. Joint submissions are encouraged where they offer the most appropriate means of presenting collaborative work for assessment. Where institutions are engaged in collaborative research with institutions outside the UK they may not make a joint submission but may refer to the collaboration in the textual parts of their submission.

1.23. The conditions governing joint submissions are set out in Section 4.

Content of Submissions

1.24. The content of submissions is summarised below. The number of the relevant RAE form in which the information will be presented is given in brackets for ease of reference. Detailed notes on and definitions of the information required are given in Section 3 of this circular. Any additional information requirements will be agreed by panels and published in their statements of criteria and working methods.

  1. Overall staff summary (RA0)
    Summary information on all academic staff and academic support staff in each submitting institution.
  2. Research active individuals details (RA1)
    Detailed information on individuals selected by the institution for inclusion as research active.
  3. Research output (RA2)
    For each individual named as research active and in post on 31 March 2001, up to four items of research output (up to two for Category A*) produced during the period 1 January 1994 to 31 December 2000 for arts and humanities subjects (UoAs 45 to 67 inclusive); and 1 January 1996 to 31 December 2000 for other subjects (UoAs 1 to 44, 68 and 69).
  4. Research students (RA3a)
    Numbers of full-time and part-time postgraduate research students and degrees awarded.
  5. Research studentships (RA3b)
    Numbers of postgraduate research studentships and source of funding.
  6. External research income (RA4)
    Amounts and sources of external funding.
  7. Textual description (RA5)
    Including information about the environment, structure, policies and strategies within which research is undertaken and developed.
  8. General observations and additional information (RA6)

Including information about indicators of research excellence and peer esteem which cannot be given elsewhere in the return.

1.25. The consultation process indicated support for a degree of self-assessment within RAE submissions. Accordingly, panels will specify a structure to the textual parts of the submissions (RA5 and RA6) which facilitates self-assessment under a number of headings. Panels will make these specifications within a basic common framework. Self-assessment will not include suggested ratings.

1.26. In addition to the standard information listed above, panels may request specific, additional information (such as quantitative indicators of total research output and other contextual information) where they can make a case for doing so. Any such additional information requirements will be specified within the panelsí statements of criteria and working methods.

1.27. When submissions are circulated to the assessment panels certain standard analyses of the quantitative data will be made available as a matter of course (for example, relating research income and numbers of research students to numbers of research active staff). Further specific analyses of the data may be made available if requested by individual panels. Details of the analyses available to panels will be published with the panelsí statements of criteria.

1.28. In exceptional cases, HEIs may be asked to provide further information where a submission is unclear or ambiguous.

Checking and Verification

1.29. A proportion of submissions from each institution will be audited as a matter of course. In addition, panel members will be asked to draw attention to any areas of concern. Institutions should therefore be prepared to provide detailed justification for the information submitted. Where discrepancies are identified a more detailed audit will be conducted. The funding bodies will consider what action to take in cases where significant discrepancies are found. A full statement of the arrangements for audit will be issued by the funding bodies at a later stage.

Publication of the Results

1.30. It is expected that the ratings from the assessment will be published in December 2001. These will show the rating awarded to each submission and the number and proportion of staff submitted for assessment in each case. In addition, to improve the arrangements for feedback from the exercise, the funding bodies will produce:

  1. A published report by each panel confirming their working methods and giving a brief account of their observations about the strengths, weaknesses and intensity of activity of the research areas falling within the UoA.
  2. Feedback on each submission summarising the reason for the rating awarded with reference to the panelís published criteria. This feedback will be sent only to the head of the institution concerned.

The reports and feedback will be available in early 2002.

1.31. The funding bodies will also publish on the Internet those parts of submissions that contain factual data and textual information about the research activity. This will include the names of selected staff and the listings of their research output. Personal and contractual details, and details of the future research plans of institutions, will be removed. Data on research students and research assistants supervised will be presented in aggregate. Institutions will be given an opportunity to indicate any staff who should be omitted from the published list for reasons of security.

Data Protection

1.32. All information required for the RAE will be collected, stored and processed in accordance with the requirements of the Data Protection Act.

Last updated 15 September 2000

[ Home | About the RAE2001 | Panels | Guidance for panel members | Guidance for institutions | Data collection | Publications | Contacts ]