RAE2001 logo

Contents
Section II
Section III

RAE Circular 5/99

Assessment panels’ criteria and working methods

Section I: Overview of the Research Assessment Exercise

1.1 The arrangements and regulations for the 2001 Research Assessment Exercise (RAE), and detailed information on the data requirements, are set out in full in ‘Research Assessment Exercise 2001: Guidance on Submissions’ (RAE 2/99). This document should be read in conjunction with RAE 2/99. The criteria and working methods for assessment panels contained in Section III below should be read in conjunction with RAE 2/99 and with the information in Sections I and II of this circular.

Purpose and Conduct of the RAE

1.2 The primary purpose of the 2001 RAE is to produce ratings of research quality which will be used by the higher education funding bodies in determining the main grant for research to the institutions they fund, with effect from 2002-03. The RAE is conducted jointly by the Higher Education Funding Council for England (HEFCE), the Scottish Higher Education Funding Council (SHEFC), the Higher Education Funding Council for Wales (HEFCW), and the Department of Higher and Further Education, Training and Employment. The exercise is managed by the RAE Team, based at the HEFCE offices, on behalf of the four funding bodies.

1.3 The assessment process is based on peer review and is not mechanistic. Assessment panels will use their professional judgement to form a view about the overall quality of the research activity described in each submission, taking into account all the evidence presented. They will form their judgements in the context of the statements of criteria and working methods published in this document.

1.4 The other principles by which the RAE is governed are:

  1. Clarity.
    A large and complex body of regulations and procedures governs the RAE. All written documents and statements about the RAE should therefore be clear and consistent, and misinformation about the RAE will be challenged wherever possible.

  2. Consistency.
    Assessments made through the RAE should be consistent, especially across cognate areas and in the calibration of quality ratings against international standards of excellence.

  3. Continuity.
    The RAE develops through an evolutionary process, building on and learning from previous exercises. With each successive exercise a balance has to be struck between continuity and development. In general, changes are only made where they can bring demonstrable improvements which outweigh the cost of implementing them.

  4. Credibility.
    As was demonstrated through responses to the funding bodies’ consultation on the conduct of the RAE, the methodology, format and processes employed in the exercise are credible to those being assessed. It is essential that this credibility be maintained.

  5. Efficiency.
    Previous RAEs have been extremely cost-effective given the value of public funds distributed according to their ratings. The cost of the exercise, including the cost to Higher Education Institutions (HEIs), should continue to be the minimum consistent with a robust and defensible process.

  6. Neutrality.
    The RAE exists to assess the quality of research in HEIs. It should carry out that function without distorting what it is measuring. In other words, the RAE should not encourage or discourage any particular type of activity or behaviour, other than providing a general stimulus to the improvement of research quality overall.

  7. Parity.
    The RAE is concerned only with assessing the quality of research in participating HEIs, regardless of the type, form or place of output of that research.

  8. Transparency.
    The credibility of the RAE is reinforced by transparency about the process for making decisions. All decisions and decision-making processes will be explained openly, except where there is a need to preserve confidentiality (for example in panels’ discussions, or when dealing with the names of nominees for panel membership or with the strategic research plans of institutions).

1.5 The 2001 RAE will follow broadly the same approach as previous exercises. Eligible HEIs will be invited to submit their research activity for assessment. Submissions will be made to a number of subject-based Units of Assessment (UoAs). The information supplied by HEIs will provide the basis for peer review assessment of research quality by specialist panels. Submissions will be in a standard form which includes both quantitative and descriptive elements. As well as providing information specifically requested by the funding bodies, institutions will have the opportunity to draw panels’ attention to any matters which they consider particularly significant to judgements of research quality. All information submitted will be subject to audit by the funding bodies. A statement of the audit arrangements for the RAE will be issued during 2000.

1.6 The census date for the exercise is 31 March 2001. Institutions are invited to provide information on staff in post on that date, and on publications and other forms of assessable output which they have produced during the assessment period. The assessment period is 1 January 1994 to 31 December 2000 for arts and humanities subjects (UoAs 45 to 67 inclusive), and 1 January 1996 to 31 December 2000 for other subjects (UoAs 1 to 44, 68 and 69). All information must be submitted by 30 April 2001.

1.7 There is an overlap between the assessment period for the 1996 and 2001 RAEs. Research outputs and other information relating to the overlap period may be returned to the 2001 RAE and will be given equal weight with all other evidence.

1.8 Institutions are invited to list up to four items of research output for each individual (Category A or C) whose research is to be taken into account (up to two items for Category A*). These may be any form of available assessable output. All forms of output will be treated on an equitable basis: panels will be concerned only with assessing the quality of research presented.

1.9 Proper account will be taken of the complete range of research. Panels will give full recognition to work of direct relevance to commerce and industry, as well as to the public and voluntary sectors. All research, whether applied, basic or strategic, will be given equal weight: panels will be concerned only with the quality of the work submitted for review.

1.10 Research into the teaching and learning process within higher education (pedagogic research) is regarded by the funding bodies as a valid and valued form of research activity. It will be assessed by all subject panels on an equitable basis with other forms of research.

Definition of Research and Eligible Outputs

1.11 The definition of research which applies in the exercise is as follows.

‘Research’ for the purpose of the RAE is to be understood as original investigation undertaken in order to gain knowledge and understanding. It includes work of direct relevance to the needs of commerce and industry, as well as to the public and voluntary sectors; scholarship*; the invention and generation of ideas, images, performances and artefacts including design, where these lead to new or substantially improved insights; and the use of existing knowledge in experimental development to produce new or substantially improved materials, devices, products and processes, including design and construction. It excludes routine testing and analysis of materials, components and processes, e.g, for the maintenance of national standards, as distinct from the development of new analytical techniques. It also excludes the development of teaching materials that do not embody original research.

* Scholarship is defined for the RAE as the creation, development and maintenance of the intellectual infrastructure of subjects and disciplines, in forms such as dictionaries, scholarly editions, catalogues and contributions to major research databases.

1.12 In order to be eligible for submission, all items of research output must embody the outcomes of research as defined for the purposes of the RAE. Teaching materials are admissible if they can be shown to embody research outputs within the RAE definition, but the preparation of teaching material in itself is not accepted as a research activity for the purposes of the RAE.

Assessment Panels

1.13 The Units of Assessment for the 2001 RAE are the same as those used in 1996 with the exception that, following consultation, the separate Biochemistry UoA has been discontinued. All UoAs are described at the beginning of the appropriate criteria statement, in a format which indicates the key subject areas they encompass.

1.14 The membership of the assessment panels, the method of their selection, and an account of their role and operation, are set out in ‘Research Assessment Exercise 2001: Membership of Assessment Panels’ (RAE 3/99.) Umbrella Groups

1.15 Throughout the RAE the Chairs of assessment panels in cognate areas will meet in Umbrella Groups. The Umbrella Groups are:


    I Medical and Biological Sciences (UoA 1 to 17)
    II Physical Sciences and Engineering (UoA 18 to 32)
    III Social Sciences (UoA 33 to 44, 68, 69)
    IV Area Studies and Languages (UoA 45 to 56)
    V Arts and Humanities (UoA 57 to 67)

1.16 The role of the Umbrella Groups is to facilitate cross-panel consistency in the assessment process and application of standards. Particular attention will be given to areas of work which span the boundaries of units of assessment, including interdisciplinary research.

Rating Scale

1.17 Ratings of research quality will be expressed in terms of a standard scale with common definitions of the points. The rating scale and definitions which will be used in the 2001 RAE are shown below. Full notes on the rating scale are given in RAE 2/99.

5* (five star) Quality that equates to attainable levels of international excellence in more than half of the research activity submitted and attainable levels of national excellence in the remainder.
5 Quality that equates to attainable levels of international excellence in up to half of the research activity submitted and to attainable levels of national excellence in virtually all of the remainder.
4 Quality that equates to attainable levels of national excellence in virtually all of the research activity submitted, showing some evidence of international excellence.
3a Quality that equates to attainable levels of national excellence in over two-thirds of the research activity submitted, possibly showing evidence of international excellence.
3b Quality that equates to attainable levels of national excellence in more than half of the research activity submitted.
2 Quality that equates to attainable levels of national excellence in up to half of the research activity submitted.
1 Quality that equates to attainable levels of national excellence in none, or virtually none, of the research activity submitted.

1.18 Panels will reach a judgement on the balance of quality across all activity submitted. Within their submissions, institutions will be able to describe the configuration of their research groups and structures.

1.19 Panels will be encouraged to indicate research areas which are identified as having a quality of research which is at least two points above the overall rating for a submission. These "flags" will be published alongside the ratings.

Content of Submissions

1.20 The content of submissions is summarised below. The number of the relevant RAE form in which the information will be presented is given in brackets for ease of reference. Detailed notes on and definitions of the information required are given in RAE 2/99.

  1. Overall staff summary (RA0)
    Summary information on all academic staff and academic support staff in each submitting institution.

  2. Research active individuals' details (RA1)
    Detailed information on individuals selected by the institution for inclusion as research active.

  3. Research output (RA2)
    For each individual named as research active and in post on 31 March 2001, up to four items of research output (up to two for Category A*) produced during the period 1 January 1994 to 31 December 2000 for arts and humanities subjects (UoAs 45 to 67 inclusive); and between 1 January 1996 and 31 December 2000 for other subjects (UoAs 1 to 44, 68 and 69).

  4. Research students (RA3a)
    Numbers of full-time and part-time postgraduate research students and degrees awarded.

  5. Research studentships (RA3b)
    Numbers of postgraduate research studentships and sources of funding.

  6. External research income (RA4)
    Amounts and sources of external funding.

  7. Textual description (RA5)
    Including information about the environment, structure, policies and strategies within which research is undertaken and developed.

  8. General observations and additional information (RA6)
    Including information about indicators of research excellence, peer esteem and other relevant factors which cannot be given elsewhere in the return. This will include a confidential section on individual staff circumstances, and an opportunity for institutions to comment on the contribution to research activity made by staff not returned as research active.

1.21 In addition to the standard information listed above, panels may request specific, additional information (such as quantitative indicators of total research output or other contextual information) where they have made a case for doing so. Any such additional information requirements are specified in the panels’ statements of criteria and working methods.

1.22 When submissions are circulated to the assessment panels, certain standard analyses of the quantitative data will be made available as a matter of course. Further specific analyses of the data may be made available if requested by individual panels. The standard analyses available to panels will be:

For Each Submission

  • percentage of academic staff selected for assessment
  • total number of staff by category (headcount)
  • total number of staff selected for assessment (research active staff)
  • total number of research fellows
  • total number of NHS-funded staff
  • total number of research assistants who are supervised by research active staff
  • research assistants supervised per research active staff
  • total number of research students who are supervised by research active staff
  • research students supervised per research active staff
  • total number of publications listed for assessment
  • numbers of staff with 0, 1, 2, 3 or 4 research outputs (five separate totals)
  • total number of studentships (listed by sponsor)
  • studentships (listed by sponsor) per research active staff
  • studentships per research student
  • total research income (listed by source of income)
  • research income (listed by source of income) per research active staff

For Each Unit of Assessment

  • total number of research active staff
  • total number of staff by category (headcount)
  • average number of research assistants supervised per research active staff
  • average number of research students supervised per research active staff
  • total number of publications listed
  • average number of research degrees awarded per annum (by category)
  • average number of studentships (listed by sponsor) per research active staff
  • average number of studentships (listed by sponsor) per research student
  • average research income (listed by source of income) per research active staff

1.23 Numbers of staff, assistants and students will be expressed as full-time equivalents in the analyses, unless otherwise specified. In all cases, analysis per research active staff will be presented by Category A/A* and by Category A/A* plus Category C. These analyses are in addition to the standard listing of data and information presented to panels in forms RA1 to RA6. Additional analyses specific to individual panels may also be prepared.

1.24 In exceptional cases, HEIs may be asked to provide further information where a submission is unclear or ambiguous.

Publication of the Results

1.25 It is expected that the ratings from the assessment will be published in December 2001. These will show the rating awarded to each submission and the number and proportion of staff submitted for assessment in each case. In addition, to improve the feedback from the exercise, the funding bodies will produce in early 2002:

  1. A published report by each panel confirming their working methods and giving a brief account of their observations about the strengths, weaknesses and intensity of activity of the research areas falling within the UoA.

  2. Feedback on each submission, summarising the reason for the rating awarded, with reference to the panel’s published criteria. This feedback will be sent only to the head of the institution concerned.

1.26 The funding bodies will also publish on the Internet those parts of submissions that contain factual data and textual information about the research activity. This will include the names of selected staff (RA1) and the listings of their research output (RA2). Personal and contractual details, and details of the future research plans of institutions, will be removed. Institutions should be aware, therefore, that all information in the textual parts of their submissions except that included in Research Strategy (RA5c) and Individual Staff Circumstances (RA6b) will be published. Data on research students and research assistants supervised on the census date (RA1) will be presented in aggregate. Data on Research Students and Research Studentships (RA3) and Research Income (RA4) will be published as submitted. Institutions will be given an opportunity to indicate any staff who should be omitted from the published list for reasons of security.

 


[ Contents ]

 


Last updated 22 December 1999

[ Home | About the RAE2001 | Panels | Guidance for panel members | Guidance for institutions | Data collection | Publications | Contacts ]