Section III: Panelsí Criteria and Working Methods
3.18 Computer Science, UoA 25
3.18.1 The coverage of the Computer Science Unit of Assessment includes the theoretical and practical study of the following: adaptive systems; algorithms; artificial intelligence; computer architecture and engineering; computer graphics; computer vision; databases; dependable systems; distributed systems; formal methods; high performance computing; human computer interaction; information retrieval; information systems; machine learning; multimedia; networks and communications; operating systems; pattern recognition; programming languages; software engineering; and speech and language technology.
3.18.2 The Panel recognises that Computer Science is all pervasive and that submissions may contribute both to the science and technology of computing and to its applications. It expects that there will be boundary issues and degrees of overlap with other panels in many areas, and it will work with other panels to ensure the proper assessment of the applications of computer science and of interdisciplinary and multidisciplinary research.
3.18.3 The Panel will cross-refer submissions across boundaries to other panels when requested, and in other cases will cross-refer submissions to other panels where their expertise is considered necessary for the proper assessment of the submission. It will also make use of meetings of an umbrella group of cognate panel chairs during the assessment phase.
3.18.4 The Panel will consider outputs that result from research into the teaching process within Computer Science. The Panel will draw on the expert advice of the Education Panel and will appoint specialist advisers whenever necessary to assist with assessing pedagogy.
Sub-Panels and Specialist Advice
3.18.5 The Panel will have an Information Systems sub-panel, which will be joint with the Library and Information Management Panel. Institutions are asked to indicate (on form RA2) which parts of their submission should be referred to this sub-panel.
3.18.6 The Panel will seek the specialist advice of other panels or sub-panels when it cannot adequately assess areas submitted itself; in exceptional circumstances it will seek external advice from a specialist adviser.
3.18.7 The Panel will be informed by Circular RAE 1/99 in assessing interdisciplinary research. Interdisciplinary and multidisciplinary research will be treated on an equal basis to other forms of research. The Panel will cross-refer such work when the Panel considers its assessment requires advice from another panel or sub-panel. Where necessary, the Panel will seek external advice from a specialist adviser.
3.18.8 The Panel will give equal consideration to joint submissions from more than one higher education institution and single submissions. With joint submissions the Panel will pay particular attention to the relationships detailed in the RA5 forms submitted in 1996 and 2001, where these exist, and pay attention to the coherence of the joint submission.
Treatment of Evidence
3.18.9 The quality of submissions will be judged mainly on the basis of the following four criteria:
Of these, a) and b) assume greater importance than c) and d).
3.18.10 The Panel requests that submissions are structured according to the primary research groupings. These groupings should reflect as closely as possible the natural working structure of the staff submitted, and one-person groupings are permitted. Individuals belonging to more than one grouping should be indicated.
3.18.11 Information on research income, the number of research students and the number of research degrees awarded will be normalised by FTE.
Research Output (RA2)
3.18.12 The forms of output the Panel expects to see cited include books; chapters in books; articles in journals; conference contributions; and other outputs including creative media and multimedia, standards documents, patents, products and processes, items of software, and technical reports, including consultancy reports. Publications in all forms will be given equal consideration. Where possible a URL should be indicated in RA2. In cases where non-printed works or teaching materials embodying research outcomes have been listed in RA2, institutions are asked to describe, in the 'Other relevant details' field of RA2, the nature of the research involved so that it is clear to the Panel what the research content is.
3.18.13 The panel collectively will examine in detail at least 10% of the total output cited in submissions and will pay particular attention to outputs that are critical to the final ratings. It will not develop a ranked list of publication outlets.
3.18.14 The Panel expects that the inclusion of registered patents, software and less orthodox outputs will be accompanied by an indication of their scope, importance and impact in the 'Other relevant details' field of RA2 so that the Panel can give them full consideration. Any evidence of success in a rigorous third party assessment of research quality using criteria appropriate to the form of output should be cited.
3.18.15 Outputs will normally be understood to attain a level of international excellence if, for example, they compare well with work published by a leading journal of international excellence, or presented at a conference of recognised international standing. Similarly, they will normally be understood to attain a level of national excellence if, for example, they compare well with work published in a journal of recognised national standing or presented at a conference of recognised national standing.
Research Students and Research Studentships (RA3)
3.18.16 The Panel expects to place higher emphasis on research outputs, such as the number of degrees awarded, than on research studentships. The number of higher degrees awarded will be regarded as an indicator of quality, with doctorates being rated more highly than research masters.
Research Income (RA4)
3.18.17 External research income which is attracted from sources with peer review processes will be regarded as an indicator of academic esteem. External research income which is attracted from industry, and other funders of research, will be regarded as an indicator of esteem and of the breadth of impact of a department's research activity.
3.18.18 The submission should describe in RA5 the overall research strategy of the submitting department. This should include reference to its approach to interdisciplinary and multidisciplinary work and, where appropriate, to the research plans set out in the submission to the 1996 Exercise. This form should also include a section to bring to the attention of the Panel particular areas of work in RA2 where it considers particular attention or specialist advice may be necessary, especially when these are disseminated in non-standard outlets. It should also flag (in RA2) cited outputs which it is appropriate that the Information Systems Sub-Panel consider.
3.18.19 Departments may, if they wish, also include in this section a self-assessment of their research performance over the period of assessment, though without reference to a suggested rating for the submission.
3.18.20 For each research grouping, RA5 should describe its key research-related achievements during the assessment period and provide a statement of its current research activities and future research strategy for 2001 and beyond, making reference to research outputs cited in RA2.
3.18.21 Departments should also comment, if appropriate, on the account they have taken of Government policy initiatives in developing their research strategies and initiatives. Where appropriate, the Panel will take this into account in considering the department's research strategy in the round.
3.18.22 Departments should note that the Panel does not consider a critical mass of researchers to be a primary factor in considering research quality in Computer Science.
Evidence of Esteem
3.18.23 In general, the Panel will take account of evidence of peer esteem. Examples include: keynote and invited addresses, editorships of journals, chairmanships and programme committee memberships of well-recognised conferences, professional awards, memberships of grant awarding bodies, distinguished dissertations and best paper awards.
3.18.24 Submissions should also describe the impact of their research including, if appropriate, the impact on wealth creation and the quality of life. They should also describe the main external and international collaborations with which they have been involved since the last RAE.
Individual Staff Circumstances
3.18.25 Institutions are invited to note any individual staff circumstances which have significantly affected their contribution to the submission (e.g. periods of sick leave, career breaks, engagement on long term projects etc).
Contributions by Non Research-active Staff
3.18.26 Institutions may also wish to discuss the relative contribution made to research by staff who have not been returned as research active.
3.18.27 In general, the Panel will not adopt a quantitative approach to assessing the evidence presented, and it will aim to reach decisions by consensus. However if a consensus cannot be reached, then voting will be used, with the Chair holding a further casting vote in the case of a tie.
3.18.28 Panel members will review every submission. The detailed work will be divided between panel members by matching their backgrounds to the main research groupings within submissions, while attempting to maintain a balance representative of the sector in the allocation to each member. Each submission will be initially assessed in detail by three panel members, each of whom will speak to his or her assessment.
3.18.29 These assessments, together with the spoken presentations, will inform the Panel's discussion. Where there is unresolved disagreement among the initial assessments, or between them and the views of other panel members, the submission will be revisited.
3.18.30 Where the Panel cannot assess part of the evidence sufficiently to determine the rating of a submission (whether single disciplinary, interdisciplinary or multidisciplinary work is under consideration) it will seek external advice: this may be from another panel or specialist advisers.
3.18.31 On the basis of this process, the Panel will reach provisional ratings for each research grouping. A preliminary rating for the whole submission will then be made, taking into account the evidence presented in the submission as a whole, and weighted according to the number of research active staff in each of the research groupings. Account will be taken of activities which span two or more research groups where this leads to enhanced quality.
3.18.32 These ratings, moderated by further discussion, will lead by an iterative procedure to the allocation of final ratings in accord with the funding bodies' definition of the rating scale. The discussion will pay attention to the less quantifiable parts of a submission. The Panel as a whole will consider every submission at least twice to ensure equitable and adequate consideration of the evidence. It will also seek the advice of a group of non-UK based experts to confirm its highest ratings.
3.18.33 Broad comparisons and calibration of interdisciplinary research across different units of assessment will be achieved through umbrella groups of panel chairs of cognate subject areas. This activity will lead to a further refinement of the preliminary grades. At subsequent meetings of the Panel, the decisions will be further refined and finalised.
Last updated 17 April 2000