RAE Circular 5/99
Assessment panels criteria and working methods
Section II: Introduction to Criteria and working methods
2.1 For the purposes of the RAE:
Criteria and Working Methods
2.2 The funding bodies recognise that HEIs should be given the opportunity to prepare their submissions for assessment with the fullest possible knowledge of the assessment framework and processes. Accordingly, assessment panels for the 2001 RAE have been established far in advance of the census date, so that they can determine how they intend to operate and decide the criteria they will adopt in making judgements.
2.3 The criteria statements in this document are not subject to change. Panels will be permitted to depart from their published statements of criteria and working methods only in exceptional circumstances.
2.4 Panel members will exercise their knowledge, judgement and expertise to reach as a group a clear and soundly based view on the quality of research. Panels must be able to show in all cases how their quality judgements relate to the evidence before them and to the criteria which they have stated they will apply. Written feedback to institutions will take the form of a statement of the reasons for the rating awarded, with reference to the panels criteria.
2.5 In all cases, the rating for a submissions will be awarded by the panel for the UoA to which it was originally submitted, against the criteria for that UoA. This will apply regardless of any cross-referral or consultation with specialist advisers or sub-panels which may be involved.
2.6 The criteria and working methods of panels have been prepared against a common template provided by the funding bodies. The template is available on the RAE web site.
2.7 Each panel's criteria include a descriptor for the UoA. These are intended to indicate the key subject areas covered by the UoA, based on the funding bodies' consultation on UoA coverage and panel membership. They are not intended to be exhaustive descriptions of the discipline(s) concerned. The UoA descriptors should be referred to by HEIs in deciding where to submit their research activities.
2.8 The assessment criteria and working methods judged appropriate to a particular discipline vary significantly in some aspects across the different UoAs. However, certain key elements are covered in all cases and panels working in cognate disciplines will adopt a broadly similar approach.
2.9 Each statement of criteria describes the panels approach to all the evidence put before it in HEIs submissions.
2.10 HEIs making submissions may ask for work to be cross-referred to other relevant panels when they believe the work concerned spans the boundary between UoAs and/or is interdisciplinary in character. Such requests for cross-referral will be mandatory and will be acted upon automatically by the RAE Team. Panels may also cross-refer work when they believe this will enhance the assessment process, even where this has not been requested by the submitting HEI. In all cases of cross-referral the whole submission will be made available to all panels concerned. Assessment of cross-referred work must normally include discussion at a meeting including at least one member of each panel involved. Responsibility for the rating awarded will remain with the panel for the UoA to which the work was originally submitted.
Sub-Panels and Specialist Advisers
2.11 Where a panel envisages establishing one or more sub-panels to advise on particular areas or aspects, or anticipates a need to take outside specialist advice, this is indicated in its statement. The membership of any sub-panels will be published before the receipt of submissions. The need for specialist advisers will often only become apparent after receipt of submissions, and panels will have an opportunity to appoint advisers at that point. The names of all specialist advisers consulted will be published after the assessment has been concluded. Where there are sub-panels or specialist advisers the main panel must consider their advice before ratings are awarded but responsibility for the rating awarded will remain with the panel for the UoA to which the work was originally submitted.
2.12 Each panel is required to consult by correspondence a number of non-UK based experts in its subject area before awarding the highest rating to any submission. These experts will provide advice on how the panel has set and applied the standard of international excellence. They will have the same status and be subject to the same conditions as specialist advisers. Their names will be published after the assessment has been concluded and will be kept confidential until then.
2.13 Research active individuals may be returned in one of the following five categories:
2.14 The research to be assessed is primarily that undertaken within the assessment period by the staff shown as research active on the census date in each submitting HEI (although the exercise is not concerned with rating individuals). The evidence for the quality of research undertaken by all listed researchers will be considered. Staff not listed as research active by HEIs in any of the five categories A to D will be disregarded, except insofar as their contribution to research activity is described in form RA6.
2.15 The work of Category C staff will not be given less weight purely because they have a different contractual relationship to the institution. However, panels may reasonably form a view as to the extent and value of the contribution made by individuals listed in Category C in the light of evidence available.
2.16 Staff who have transferred between UK HEIs in the twelve months before the census date (Category A*) may be returned as research active by both their former and current employing HEIs. They will be taken into account in the assessment of quality for both institutions.
2.17 The situation of staff new to a research career (for example, junior staff or more senior staff who have recently taken up an academic post after working in industry or after a professional career) will be taken into account in reaching overall judgements of quality where it is indicated in submissions.
2.18 The situation of staff who have taken maternity leave or other career breaks, who hold part-time contracts, who are disabled, or who have been absent for long periods through illness (where this is indicated by HEIs) will be taken into account in reaching overall judgements of quality where it is indicated in submissions.
2.19 Institutions are asked to supply details of up to four items of research output produced during the assessment period by each listed member of Category A or C staff, and up to two items for category A* staff. In general, panels will pay particular attention to this part of the evidence in assessing the quality of research. It is a requirement, however, that all parts of submissions are considered fully and properly by panels and that the research output cited is viewed in this context.
2.20 The funding bodies have committed to ensuring that assessment panels give full recognition to work of direct relevance to the needs of industry and commerce, and that all research, whether applied or basic/strategic, should be given equal weight.
2.21 The definition of research output is deliberately broad. In principle any form of publicly available assessable output embodying the outcome of research, as defined for the RAE, may be cited. HEIs must have confidence that any output cited will be fully and properly assessed: panels may not regard any particular form of output as of greater or lesser quality than another per se. In addition to printed academic work, research outputs may include new materials, devices, images, products and buildings; intellectual property, whether in patents or other forms; performances, exhibits or events; and work published in non-print media. The only exception to the requirement that outputs must be publicly available is where they are confidential. Examples would include research reports for companies that are commercially sensitive, or reports for government departments or agencies that have not been released into the public domain. In such instances, institutions will have to make appropriate arrangements for panels to have access to the outputs. Responsibility will rest with the submitting institution to ensure that all necessary permissions for access to confidential work have been obtained.
2.22 The funding bodies will issue further guidance on the arrangements which will apply to the submissions of web-based materials and certain other forms of output during 2000.
2.23 Evidence that research outputs have already been reviewed or refereed by peers may be used by panels as one measure of quality. However, the absence of such review may not, in itself, be taken to imply lower quality. Panels will also have regard to all reviewing processes, as appropriate, including those operated by users of research in commissioning or funding research work.
2.24 For research outputs produced in a language other than English or Welsh, HEIs are required to submit a short abstract describing the content and nature of the work in the other relevant details field of form RA2. This is intended to enable panels to identify appropriate special advisers to whom the work may be referred. The abstracts themselves will not form the basis for assessment. This requirement is waived for outputs submitted to any of the language panels where the work is produced in any of the main languages in the panels UoA.
2.25 It is not expected that panels will examine in detail all the research outputs cited. All panels must, however, examine a sufficient proportion which, in their opinion, will enable them to make an informed judgement on the quality of the work presented. Panels indicate in their criteria statements what arrangements they will use for sampling work submitted, and their approach to assessing work which is not examined in detail.
2.26 Each panel indicates the minimum proportion of research outputs which it will examine in detail. This is a collective responsibility, not a requirement for each individual panel member. The phrase 'examine in detail' is used to indicate reading, reading substantially from or, in the case of outputs which by their nature are not capable of being read, an equivalent level of scrutiny. Panel members are not required to re-examine work which they have already examined in detail outside the RAE process, and may include such work as part of the minimum overall requirement. Where "virtually all" is the phrase used to describe the proportion to be examined in detail this should be understood as in excess of 90%.
2.27 All research output not examined in detail will be reviewed by panels on the basis of its citation in RA2.
2.28 There is no automatic penalty for failure to cite four items of research output: each case must be looked at on its own merits. HEIs are invited to comment on the circumstances of individual researchers for whom fewer than four outputs are listed. These may include (see also 2.17 and 2.18):
2.29 Regular academic and academic-related duties, including teaching and administration, are not generally regarded as offering explanation in themselves for listing fewer than four items of research output against an individual.
2.30 Where a panel concludes that a department has shown no good reason why certain staff returned as research active have failed to produce the normally expected quantity of work during the assessment period, that will be reflected in the rating awarded.
Research students, studentships and external income
2.31 In their statements, panels comment on how they will use the data provided on research students, studentships, and external income in assessing quality.
2.32 It is not be acceptable for panels simply to state that they regard certain sources of income as more reliable than others as indicators of research quality, without giving further guidance on their approach. The point relating to review processes and research outputs in paragraph 2.21 above also applies to sources of studentships and income.
2.33 Panels have suggested a structure and content for RA5 and RA6 within their criteria statements. HEIs are encouraged to follow the structure and content suggested wherever appropriate. The structure facilitates a degree of self-assessment. Self-assessment may not include suggested ratings.
2.34 RA5 is concerned with the research strategies, policies, structure and environment. RA6 relates to marks of esteem attached to individual researchers, and additional observations. It includes an opportunity for HEIs to comment, in confidence, on circumstances which have reduced the research activity of specific individuals during the assessment period. RA6 also enables HEIs to describe the contribution to research of Category A/A* staff not returned as research active.
2.35 RA5 and RA6 will be subject to a combined length limit, as follows:
for submissions over 20 FTE Category A/A* staff
2.36 The RAE software will automatically control the length, and the font and page dimensions.
Declarations of Interest and Working Methods
2.37 All panel members, panel secretaries, specialist advisers, non-UK based experts, and sub-panel members will be required to declare those HEIs eligible to participate in the RAE in which they have a material interest. A material interest arises in the case of the HEI by which the individual is currently employed or has been employed during the period 1996 to 2001 inclusive, or where an immediate family member (parent, partner, sibling or child) is currently employed. They will not participate in the assessment of any HEI in which they have declared an interest and will be required to withdraw from any panel meeting during discussion of the submission from such an HEI. A register of interests will be maintained by the RAE Manager.
[ Contents ]
Last updated 22 December 1999