Criteria and Working Methods
SECTION 2: CRITERIA AND WORKING METHODS
2.1 The funding bodies recognise that HEIs should be given the opportunity to prepare their submissions for assessment with the fullest possible knowledge of the assessment framework and processes. Accordingly, assessment panels for the 2001 RAE have been established far in advance of the census date so that they can determine how they intend to operate and decide the criteria they will adopt in making judgements. The criteria will be published by the end of 1999, 16 months before the census date.
2.2 Criteria statements once made publicly available should not be subject to change. Panels will be permitted to depart from their published statements of criteria and working methods only in exceptional circumstances.
2.3 Panel members will exercise their knowledge, judgement and expertise to reach as a group a clear and soundly based view on the quality of research. It will be important that panels are able to show in all cases how their quality judgements relate to the evidence before them and to the criteria which they have stated they will apply. Written feedback to institutions will take the form of a statement of the reasons for the rating awarded with reference to the panel’s criteria.
2.4 The Criteria and Working Methods statements will include;
2.5 The panel’s role is to assess the quality of research demonstrated in each submission in the round, looking at all the evidence provided. In order to achieve this panels will need to establish criteria against which research quality can be measured irrespective of form, place of output or type.
2.6 It is anticipated that the assessment criteria and working methods judged appropriate to a particular discipline may vary significantly in some aspects across the different UoAs. However, certain key elements will need to be covered in all cases; and it is to be expected that panels working in cognate disciplines will adopt a broadly similar approach. It will be important to ensure that the extent of variation in the approaches described within the statements is not out of keeping with the extent to which the research approaches and methods vary between disciplines.
2.7 Panels and their Chairs will be encouraged to work collaboratively to achieve an appropriate degree of uniformity. A small number of umbrella groups of panel chairs will be established which will meet during the assessment stage to discuss consistency of approach and marking standards, to consider the assessment of areas of work which sit close to, or cross, the boundaries of UoAs, and to address other areas of common concern.
2.8 Panels covering more than one UoA should consider how far each requires its own criteria.
2.9 Each statement of criteria will describe the panel’s approach to all the evidence put before it in HEI submissions. It will be necessary for panels to indicate the methods they intend to use to form and refine their collective judgement of quality and in particular:
2.10 It will be necessary for each panel to give an indication of how it proposes to interpret the rating points, especially in terms of its attitude to the definition of “international excellence” in its subject.
2.11 Panels will also have to consider the arrangements which they will adopt for involving the required group of non-UK based experts before awarding the highest ratings.
2.12 Each panel should consider how it will approach the assessment of interdisciplinary research. In this, it should be guided by the conclusions of the report “Interdisciplinary Research and the Research Assessment Exercise”, published by the funding bodies as RAE 1/99.
2.13 HEIs making submissions may ask for work to be cross-referred to other relevant panels. Such requests for cross-referral shall be mandatory and shall be acted upon automatically by the RAE Team. Panels may also cross-refer work when they believe this will enhance the assessment process even where this has not been requested by the submitting HEI. In all cases of cross-referral the whole submission will be made available to all panels concerned. Assessment of cross-referred work must normally include discussion at a meeting including at least one member of each panel involved. Responsibility for the rating awarded will remain with the panel for the UoA to which the work was originally submitted.
2.14 Where a panel envisages establishing one or more sub-panels to advise on particular areas or aspects, or anticipates a need to take outside specialist advice, this should be indicated. The membership of any sub-panels will be published before the receipt of submissions so departments can be advised on the assessment of specialist areas. Advice from sub-panels or specialist advisers must be considered by the main panel before ratings are awarded.
2.15 In exceptional circumstances where a panel receives submissions containing significant bodies of work that neither it nor its sub-panel(s) feel competent to assess it will be possible to set up ad hoc advisory arrangements. However, the proposed use of sub-panels and specialist advisers should be set out in advance, as far as possible, in the statements of working methods.
2.16 The components of the information collected in submissions are listed below and some of the issues around their interpretation which panels will have to consider are highlighted. (A full content list for submissions is given at Annex B).
Research active staff
2.17 The research to be assessed is primarily that undertaken within the assessment period by the staff shown as research active on the census date in each submitting HEI (although the exercise is not concerned with rating the work of individuals.) The evidence for the quality of research undertaken by all listed staff should be considered. Staff not listed by HEIs in any of the five categories A to D should be disregarded.
2.18 The following points should be borne in mind:
2.19 Institutions will be asked to supply details of up to four items of research output produced by each listed member of Category A or C staff and up to two items for category A* staff during the assessment period. It is generally recognised that panels will pay particular attention to this part of the evidence in assessing the quality of research. It is essential, however that all parts of submissions are considered fully and properly and that the research output cited is viewed in this context.
2.20 The funding bodies sponsoring the 2001 RAE have committed to ensuring that assessment panels give full recognition to work of direct relevance to the needs of industry and commerce and that all research, whether applied or basic/strategic, should be given equal weight.
2.21 The definition of "research output" is deliberately broad. In principle any form of publicly available assessable output embodying the outcome of research, as defined for the RAE (Annex A), may be cited. HEIs must have confidence that any output cited will be fully and properly assessed and panels may not regard any particular form of output as of greater or lesser quality than another per se. In addition to printed academic work, research output may include new materials, devices, images, products and buildings; intellectual property, whether in patents or other forms; performances, exhibits or events; work published in non-print media. The only exception to the requirement that outputs must be publicly available is where they are confidential. Examples would include research reports for companies which are commercially sensitive or reports for government departments or agencies which have not been released into the public domain. In such instances institutions will have to make appropriate arrangements for panels to access the outputs. Responsibility will rest with the submitting institution to ensure that all necessary permissions for access to confidential work have been obtained.
2.22 Panels will need to consider the range of research outputs they may receive and establish how the quality of different outputs can be assessed. Evidence that research outputs have already been reviewed or refereed by peers may be used as one measure of quality. However the absence of such review may not, in itself, be taken to imply lower quality. Panels should also have regard to all reviewing processes, including those operated by users of research in commissioning or funding research work.
2.23 It is not expected that all panels will examine all the research outputs cited. All panels must, however, indicate the arrangements which they will use for sampling work submitted and the approach which they will take to assessing work which is not examined in detail.
2.24 Panels should consider their treatment of the following activities or outputs:
2.25 The outcome of the RAE exercise is likely to be used to determine funding in a “quality times volume” formula so the quality ratings must reflect the work of all of the staff listed by an HEI in a unit of assessment. A department listing some clearly excellent researchers alongside others whose work is assessed to be of significantly lower quality, or for whom there is no evidence of worthwhile output, must therefore recognise that it may receive a lower quality rating than if it had submitted only its demonstrably excellent staff. This will inevitably focus attention upon staff listed as research active whose output appears to be low or non-existent. It is important that the panel uses its judgement, taking into account all of the evidence presented, to distinguish cases where low productivity reflects low quality from those where there are genuine and acceptable reasons for low productivity.
2.26 There should be no automatic penalty for failure to cite four items of research output: each case must be looked at on its own merits. The Guidance on Submissions states that “up to four” research outputs may be cited and that these should “best reflect the quality of work of the member of staff”. The question of how far volume of output may reliably be used as an indicator of quality – in respect either of individual staff or of a department - needs to be approached with some caution. Different panels may take differing views on this reflecting their knowledge of normal practice and expectations in their own subject area. Even where volume of output is considered to be relevant panels should consider the reasons why output was low in a particular case. For example the panel may consider whether:
2.27 Where a panel concludes that a department has shown no good reason why certain staff had failed to produce work of high quality during the assessment period that should be reflected in the rating awarded.
Research students, studentships and external income
2.28 Panels should consider how they will approach the data provided on research students, studentships, and external income in assessing quality.
2.29 It will not be acceptable for panels simply to indicate that they regard income from certain sources as more reliably indicative of research quality than from others. The point relating to review processes and research outputs in paragraph 2.22 above applies equally to sources of studentships and income.
2.30 Panels should specify a structure which allows departments to explain the context of their submission in terms of their research environment and organisation and strategies for promoting and developing research. The structure should facilitate a degree of self-assessment. Self -assessment will not include suggested ratings.
2.31 The structure specified by panels should allow HEIs the opportunity to explain, where applicable, their support for interdisciplinary research, any other UoAs to which related work has been submitted, and instances where there is a bad fit between departmental structure and the UoA framework which has led to departments being split between UoAs.
2.32 Wherever relevant panels should offer institutions the opportunity to describe their relationship in research with industry and commerce. Panels should give an indication of the use they will make of this information in their assessments
2.33 Where appropriate, panels should encourage institutions to comment on the account they have taken of Government policy initiatives such as the Foresight programme in developing their research strategies and initiatives. Panels should give an indication of the use they will make of this information in their assessments.
2.34 Panels should offer guidance to institutions on the arrangements for describing the research groupings. It will be possible for institutions to identify staff with particular research groups through the use of a numerical indicator.
2.35 Panels will need to structure this section into headings so that information, such as evidence of esteem of excellence, and explanations about staff productivity, can be easily presented.
2.36 In addition to the standard information listed, panels will be permitted to request specific, additional information such as quantitative indicators of total research output and other contextual information where they are reasonable, justifiable and explicit. Any such additional information requirements will be specified within the panels’ statements of criteria and working methods. Additional information requirements will be subject to the approval of the funding bodies.
Last updated 4 June 1999