Section III: Panelsí Criteria and Working Methods
3.3 Hospital-based Clinical Subjects, UoA 3
3.3.1 The UoA includes hospital based clinical subjects and their cognate sub-disciplines excluding those specified in UoA 1 & 2.
3.3.2 The Panel expects to receive submissions from all areas of hospital-based clinical studies. It expects that submitted work will deal with basic and clinical aspects of hospital-based research. Submissions which are primarily concerned with laboratory based subjects (as defined in UoA 1) or community based subjects (as defined in UoA 2) should be submitted to those UoAs. However, the UoA 3 Panel is well aware of the breadth of its field and the probability of overlap. It will work closely with other panels where their work may overlap.
3.3.3 Submissions which span the boundary between two or more UoAs will be cross-referred, using the Panel's judgement or where requested by the submitting HEI, to the most appropriate UoA for advice; this could include cross-referral to non-medical panels. The final grade decision will remain with the Panel for the UoA to which the submission was originally directed by the submitting HEI. It is the responsibility of the submitting HEI to determine which would be the most appropriate UoA to assess its profile of research activity.
3.3.4 There will be four sub-panels within the panel structure of UoA 3. These will cover Neurosciences, Cardiovascular Medicine, Cancer Studies, and Infection and Immunology. The sub-panels will operate in common with UoA 1 and will also be available to provide advice, if required, to the Panel for UoA 2. There will be cross-representation in the membership of the Neurosciences sub-panel and the Psychiatry sub-panel of UoA 2. Each sub-panel will be chaired by a main panel member and will have an approximately equal number of panel and outside expert members. The membership of sub-panels will be published in advance of the submission date.
3.3.5 The role of the sub-panels is to provide a wider range of expertise and viewpoints, particularly in relation to the research output cited in submissions. Sub-panels will also further the accurate and consistent assessment of the large quantity of medical research which is of an interdisciplinary nature by allowing panel members from other clinical UoAs to offer their perspectives in a common forum. Sub-panels will examine in detail the research output referred to them. In reading submissions, examining items of output and reaching decisions sub-panels will work within the procedures established by main panels.
3.3.6 As part of their advice to the UoA 3 Panel, sub-panels will suggest a grade, using the RAE rating scale, for work which they have examined. The grade, when ratified by the UoA 3 Panel, will be published, together with the FTE number of staff referred to the sub-panel. This will apply only when HEIs request referral of part of their submission to a sub-panel. The Panel may additionally refer items of output submitted in order to utilise the expertise of a sub-panel . In this instance, the advice given will not include a proposed grade.
3.3.7 It is the responsibility of the submitting HEI to determine whether specifically to request sub-panel assessment. There is no advantage or disadvantage to HEIs requesting the involvement of sub-panels, nor is it expected that submissions should mirror sub-panel areas. The final grade decision remains with the main panel at all times and the Panel will ensure that all activity submitted is taken fully into account in arriving at an overall grade. Sub-panels will receive the relevant part of form RA2 and all of forms RA5 and RA6 from the submission. Main panels will consider the advice of the sub-panels on these sections in the context of the entire submission and decide on the final grade. HEIs, therefore, should indicate whether, and if so, which part(s) of their submission they wish to be considered by which sub-panel(s). No part of a submission may be directed to more than one sub panel. When requesting referral to a sub-panel the normal expectation is that an institution will request referral of all of its work in the appropriate area (e.g. neurosciences, oncology) contained in the submission.
3.3.8 The Panel recognises that it may have to seek specialist advice in areas where despite the use of Sub Panels, some fields may be outside panel members' expertise. The use of specialist advisers will be decided once submissions have been received and any gaps in expertise identified.
3.3.9 The Panel recognises the interdisciplinary nature of its field, which is reflected in the experience its members have as inter-disciplinary researchers. Cross-UoA consistency and equity will be maintained by regular meetings between panel chairs in cognate fields to maintain common standards. There will be meetings between panel chairs in UoA 1, 2 and 3 to address the particular issues of the medical disciplines, in addition to the umbrella meetings involving a wider range of disciplines. HEIs should be reassured that interdisciplinary research will not be disadvantaged and should seek to submit their research in groupings which most accurately reflect activity in the institution.
3.3.10 Joint submissions shared between two HEIs will be considered in the normal way, as if they had originated as a single submission from one HEI.
Treatment of Evidence
3.1.11 Quality of submissions will be judged by the Panel on the basis of the following measures:
Of these measures greatest weight will be afforded to quality of research output.
Research Output (RA2)
3.3.12 The Panel expects the research output it has to review to consist very largely of papers in scientific journals, monographs and books, but other forms of output, including patents and other applied research, will also be accepted and judged on their individual merits. The evaluation of research papers will not use quantitative bibliometric indices. Panel members will be guided by their professional judgement on the quality of the research output, bearing in mind the widespread differences in the specialist areas included within the UoA.
3.3.13 It is recognised that original research may be embedded in systematic reviews (e.g., the Cochrane type) and in the annotation and interpretation of curated data and sample collections, and these will also be considered as indicators of quality. Refereed conference papers will be assessed by applying principles similar to those for the outputs listed above. The Panel will give full recognition to work of direct relevance to the needs of commerce and industry, the NHS and other parts of the public and voluntary sectors, including patents and other applied research. Equal weight will be given to such research in the assessment of its scientific excellence.
3.3.14 Order of authorship can be an indicator of relative contribution to a multi-authored item of output. However, the Panel recognises that this is not always the case. HEIs should comment, in the 'Other relevant details' field of RA2, on the contribution of a submitted member of staff to a multi-authored item of output where this is not apparent from either the order of authorship or the content of the item. HEIs citing a single item of multi-authored output for more than one member of staff will not be disadvantaged, provided that each member of staff's contribution to the piece can be regarded as displaying sufficient depth and quality. HEIs may wish to comment on this, where relevant, in RA5.
3.3.15 A submission of less than four research outputs per active researcher may be indicative of lack of depth within the subject area or of low productivity, but there will be no automatic penalty for citing fewer than four items of research.
3.3.16 The main Panel and, where so required, sub-panels and other sources of expertise will use their combined expertise in undertaking review and establishing the merit of the work submitted. Those contributing to the assessment will be asked to use their expertise and judgement in assessing the work, using criteria including the originality, scientific rigour, potential and actual implementation and impact in health care, contribution to knowledge and conceptual framework of the field.
Research Students and Research Studentships (RA3)
3.3.17 The number of research-based higher degrees awarded and the number of studentships will be regarded as indicators of quality, with higher degrees, and doctorates in particular, rated more highly than studentships.
3.3.18 Greater credit will be given to studentships subject to rigorous competition.
External Research Income (RA4)
3.3.19 External research income will be used as a measure of peer judgement of standing or promise. This specifically includes funding from NHS R&D sources where these can be directly attributed to the HEI, but excludes infrastructure support provided through Culyer funding.
3.3.20 Amongst academic funding sources, greater weighting will be given to those forms of support known to be the result of a rigorous peer-reviewed assessment process and open to broad competition. Due account will be taken of long term support from major funding agencies known to require the very highest application standards, such as support provided by charities in, or governed by the rules of, the AMRC. Institutions are requested to identify separately charities income derived from AMRC sources. Industrial income and awards will also be considered as indicators of users' esteem, and in cases of sizeable and sustained support, may be regarded as equivalent to peer reviewed funding.
Research Structure and Environment
3.3.21 HEIs should define research groups, including who belongs to them (referring to RA1) , their prime activities, how they operate and their main achievements. To facilitate analysis of submissions HEIs will be expected to identify research groups in both the staff record (RA1) and research outputs form (RA2). HEIs may also wish to comment on the operation of major research groups under RA5. The submission's research groups need not be related to sub-panels' structures. Research groups selected and identified by the HEI should be consistent between RA1, RA2, RA5 and RA6.
3.3.22 List other UoAs to which related work has been submitted and detail any difficulties of fit between departmental structure and the UoA framework.
3.3.23 Explain the mechanisms and practices for promoting research and sustaining and developing an active and vital research culture. Assessors will be asked in particular to consider data on long term support, and on senior and other fellowships. This information will be referred to during the discussion of each submission.
3.3.24 Describe the nature and quality of the research infrastructure, including facilities for research students. The research organisation stated in RA5 and RA6 will be tested against the information provided in the rest of the submission.
3.3.25 Describe any arrangements which are in place for supporting interdisciplinary or collaborative research.
3.3.26 Provide information on relationships with industry and commerce or other research users and where appropriate on the account taken of Government policy initiatives and objectives.
3.3.27 Describe the arrangements for the development and support of the research work of staff.
3.3.28 Describe any arrangements which are in place for developing the research of younger and /or new researchers and for integrating them into a wider, supportive research culture.
3.3.29 HEIs should where appropriate, explain the role and contribution of Category A* staff who have joined the department.
3.3.30 Where appropriate, comment on how the departure of staff in categories A* B and D has affected the strength, coherence and research culture of the department at the census date.
3.3.31 Provide a statement about the main objectives and activities in research over the next five years. The Panel's attention should be drawn to ongoing research work that is not producing immediate visible outcomes. Future research plans stated in RA5 and RA6 will be tested against the information provided in the rest of the submission. Well defined and potentially realisable objectives will be viewed as more impressive than general descriptions of intent.
3.3.32 Where relevant evaluate achievements against the research plans put forward in the 1996 RAE. HEIs should be aware that the Panel will have access to RA5 and RA6 statements from the 1996 RAE.
3.3.33 Provide a self-assessment of performance in relation to the issues detailed above in the textual commentary. HEIs should be aware that the Panel will have access to RA5 and RA6 statements from the 1996 RAE. Any changes since 1996 which affect the research capacity of the HEI will be of interest to the Panel. The self-assessment should be drafted in light of this fact, and tied into research groups as defined within the submission for 2001.
Evidence of Esteem
3.3.34 List indicators of peer esteem which relate to the staff submitted.
Individual Staff Circumstances
3.3.35 Note any individual staff circumstances which have significantly affected their contribution to the submission (e.g. periods of sick leave, career breaks, engagement on long term projects etc).
Contributions by Non Research-active Staff
3.3.36 Institutions may also wish to discuss the relative contribution made to research by staff who have not been returned as research active.
3.3.37 The major element of the Panel's judgements are based on qualitative measures; these will be compared with quantitative measures provided in the RAE submission by HEIs that are indicative of quality such as value of external research income, number of research assistants and studentships and numbers of higher degrees awarded. In general, research excellence is facilitated by research groupings of adequate size, however there may be exceptions to this, for example where there is external collaborative research, and the Panel will take account of such factors. The ensuing figures will be used to assess submissions in terms of their relative standing with regard to these factors and will be compared against qualitative assessment.
3.3.38 The Panel will assess all submissions and each submission will initially receive the same number of assessors. The arrangements for interdisciplinary research, seeking outside expert advice and use of sub-panels have already been outlined above.
3.3.39 All panel members will read all submissions. Each submission will then be assigned to two main panel members who will produce a general assessment of the complete submission. In preparing such assessment, advice from other main panel members, other assessment panels, sub-panels or external specialist advisers will be taken into account.
3.3.40 A minimum of 25% of all research outputs (i.e. items of research output) will be examined in detail by either main or sub-panel members or specialist advisers. The remaining submitted items will be reviewed. Therefore, all submitted items within each overall submission will have been taken into account in the main panel's final assessment.
3.3.41 At further meetings one of the two main panel members assigned to each submission will act as the lead speaker to introduce submission to the other panel members. Questions and discussion will then follow to determine the rating of sub-groups of research within each submission and to explore any other point relevant to the final assessment. During this discussion reference will be made to the contribution of individual items of output to the overall submission.
3.3.42 The initial UoA 3 Panel ratings will be based on a ballot of panel members. The results of the ballots for individual HEIs, particularly their spread over two or more grades, will be reviewed at the end of the meeting. Where required, after further discussion a second ballot will then be held focusing on the two grades which received the highest support in the first ballot. The result of the second ballot will be final, and in the event of a tie the higher grade will apply.
International Ratings and Experts
3.3.43 The Panel recognises that much medical research in the UK is of international standing. It will define international excellence as on a par with the best work in its area anywhere in the world. The Panel normally expects output of this standard to be known by non-UK based experts, and acknowledged as a contribution to the field of knowledge. The Panel's perception of this standard will be tested against the view of non-UK based experts who will offer advice to the main Panel, in the cases of submissions provisionally rated 5 or 5*.
Last updated 30 March 2000