Section III: Panelsí Criteria and Working Methods
3.9 Psychology, UoA 13
3.9.1 For the purposes of the RAE, psychology includes all areas of human experimental psychology (including perception, cognition, psycholinguistics); mathematical and statistical psychology; neuropsychology; psychometrics; all areas of biological psychology (psychopharmacology, psychophysiology, behavioural neuroscience, animal learning, comparative and evolutionary psychology); social psychology; developmental psychology; all branches of applied psychology including clinical, health, counselling, educational, forensic and occupational psychology, cognitive ergonomics and human factors. It includes psychologically relevant areas of neuroscience and cognitive science.
3.9.2 It is anticipated that submissions will vary in terms of the range of areas covered, but all submissions will be assessed using the same criteria irrespective of breadth and focus. The Panel will assess research into the teaching of psychology using the same criteria as applied to other areas of research.
3.9.3 Panel members have between them a broad range of experience in conducting and assessing research across the different areas contained within the definition above. Nevertheless there may be cases where the Panel identifies work that it considers outside its areas of expertise. Where a submission is in an interdisciplinary area which goes beyond the expertise of the Psychology Panel alone, the Panel will work with another panel or panels, and/or specialist advisers. The Panel will take this approach to judge submissions in Ergonomics for which the Psychology Panel is willing to act as host panel for assessment.
3.9.4 Where a specialised area of research requires expertise beyond that of the Panel, specialist advisers may be consulted. The Panel expects to appoint specialist advisers to help in the assessment of Counselling Psychology, Critical Psychology and Ergonomics, and any other types of activity where this appears necessary on the basis of reported research activity within the submissions received.
3.9.5 A sub-panel of users of research will be convened. Members of the sub-panel will be chosen to have appropriate experience of commissioning, using and/or evaluating research, and the membership will be chosen to reflect the broad range of sectors which make use of psychological research, such as: Health; education; defence; prison and police services; transport; voluntary sector; specific charities (or umbrella organisations); industrial sectors aligned with Technology Foresight areas and/or of relevance to Psychology e.g. communications; pharmaceuticals; food and drink etc; test publishers; market research.
3.9.6 This sub-panel will provide user perspectives on the strengths and weaknesses of the sector as a whole, and on particular submissions. The sub-panel may also be used to test claims about relevance made within specific submissions.
3.9.7 The sub-panel will meet separately at least once and will additionally attend one meeting with the full Panel. Two members of the main Panel will attend meetings of the sub-panel (one of these taking the chair) to facilitate liaison and communication between the sub-panel and the main Panel.
3.9.8 The Panel has expertise in conducting and assessing interdisciplinary research, and members with experience of such work will lead discussion of each interdisciplinary submission. Where a request is made for the submission to be seen by another panel then this consultation will occur. Where no such request is made, other panels and/or specialist advisers may be consulted, where judged appropriate by the Panel.
3.9.9 Where two or more institutions submit jointly the Psychology Panel will assess the submission as a whole.
3.9.10 The quality of submissions to the Psychology Panel will be judged on the basis of the following indicators.
3.9.11 The quality of publications and other forms of research output cited will carry most weight in the assessment process, but all indicators will be scrutinised and all will contribute to the overall assessment of a submission.
3.9.12 Where institutions submit low numbers of staff for assessment, the Panel will consider whether there is sufficient "critical mass" to sustain a research culture, and to achieve the stated research plans and objectives. Small submissions (where fewer than 10 staff are submitted as research-active) are encouraged to describe fully any aspects of their research environment within or beyond the submission which help sustain and support the research activities submitted for assessment.
Research Output (RA2)
3.9.13 The Panel expects to have the following types of output nominated for assessment: Journal articles, books, chapters in books, published conference papers (including abstracts, which should be clearly flagged as such in the 'Other relevant details' field of RA2), government reports, technical or other reports, test materials and patents. Institutions may also wish to submit other forms of output where these meet the definition of research set out in the funding bodies' Guidance on Submissions.
3.9.14 All forms of output will be judged according to the same criteria. In assessing cited output, the Panel will seek to establish whether the research reported is competent, appropriately innovative, and makes a contribution to psychological knowledge, theory or practice. The Panel will collectively examine in detail at least 25% of cited outputs.
3.9.15 In selecting this sample, the Panel will focus primarily on those items which are in publication outlets not recognised as being of international standards of excellence, or which are unfamiliar or less familiar to the Panel.
3.9.16 The Panel will recognise full, original outputs published in outlets with internationally-recognised, rigorous refereeing and editorial procedures as meeting the standards required for international excellence. The Panel will not collectively rank outlets but will be guided by its individual and collective judgements of reviewing and refereeing standards. The Panel may accept guidance from specialist advisers or by other panels where appropriate. Publication outlets recognised as being of international level standards include some published from within as well as beyond the UK. Outputs in refereed or non-refereed outlets which are not recognised as meeting these criteria will not be assumed to be of lesser quality, but these publications will be sampled and their quality assessed by the Panel. The Panel recognises that it is appropriate for some types of research to be published in less prominent or less mainstream outlets, and that high quality psychological research may also be published in outlets associated with other disciplines, and will sample and/or seek specialist advice on publications of this kind.
3.9.17 Research judged by the Panel to be of international standards of excellence will be at least of a quality which is comparable with work at the forefront of UK national work on the topic, competitive with the best work internationally, and making a significant or substantial contribution to psychological knowledge, theory or practice. Research judged to be of attainable levels of national excellence will fall short of this level but will still be judged to be of good, nationally competitive quality making a valuable contribution to psychological knowledge, theory or practice.
3.9.18 Published conference abstracts will not normally contain enough detail to be judged as meeting attainable levels of national or international excellence. However, conference abstracts may be a useful indication of additional scientific activity by junior members of staff (as defined in paragraph 3.9.21) who would not be expected to have four full publications, or where established researchers are engaged in long-term projects which have not yet delivered full publications.
3.9.19 One of the ways in which the Panel will examine the research activity within a submission will be to examine the proportions of submitted individual researchers whose activities meet attainable levels of international, and national excellence.
3.9.20 For an individual established researcher to be judged as achieving attainable levels of international excellence, the Panel would expect to observe substantial evidence of international excellence in their set of nominated research outputs. An individual showing less evidence of international excellence but with a set of four research outputs (or other forms of output) of good quality would be judged as reaching attainable levels of national excellence on outputs alone, though other outstanding evidence of international peer esteem for research provided in the additional information in RA6 or elsewhere may lift the assessment of an individual to international level where the research output strength alone is marginal.
3.9.21 For researchers at a more junior stage (those whose doctorate has been awarded since the 1996 RAE, or who have commenced a research career in this period), assessment will be based upon the quality of those research outputs nominated, evidence of other scientific activities (e.g. from abstracts, information in RA6) plus judgements of the prospects for their future development given the described research environment and stated plans for future research and career development.
3.9.22 So that the Panel has the best possible evidence upon which to base its judgements, different members of the same submission should wherever possible avoid duplicating their nominated research outputs. Institutions may wish to comment on the involvement of staff in particular collaborative work or co-authored outputs in form RA6.
3.9.23 To be judged as meeting international or national levels of excellence in research outputs, a researcher does not need to be first-named or senior author on all their cited outputs. However, the Panel will look for evidence that the researcher has made an independent contribution to the research reported. Submissions should use the 'Other relevant details' field of RA2 to signal this contribution where this can be done briefly (e.g. "First author"; or "Senior author" or, for example, "neuropsychologist" in a multi-disciplinary output) but where the contribution cannot be stated briefly they should elaborate on the roles played by individual researchers to collaborative projects within RA6.
3.9.24 Where appropriate, the 'Other relevant details' field on RA2 may also be used to provide additional, factual information, including, for example, that a conference paper reports an invited keynote address or that a multi-authored paper arises from an international collaboration. However, this field should not be used to express opinions about the quality of the output. Any information given in comments must be potentially verifiable through audit.
Research Students and Studentships (RA3)
3.9.25 Numerical indicators of numbers of research students, research degrees awarded, and studentships will be normalised by number of research active staff cited in form RA1. Research student activity will be regarded as an indicator of vitality and research culture, and their presence and the success of their research activities (as reflected in degrees awarded, or otherwise described on RA6) will be regarded positively as an indication of a vibrant research environment. Institutions should note that professional doctorates such as the DClinPsych may not meet the definition of research students for RAE purposes and, accordingly, may not be returnable in RA3. Where they are returned, institutions should disaggregate the number of students on different types of research degree in RA5. Numbers and sources of funded studentships are only of marginal importance in judgements of quality. While success in attracting competitive studentships is a further indicator of peer esteem, the funding of studentships internally may also be viewed positively as a sign of investment in the research culture. Details of any students or studentships which contribute to the overall research culture and activity but are excluded from the definition of research students in RA3 should be given in RA5. Details of students supervised externally may also be mentioned in RA5.
Research Income (RA4)
3.9.26 External research income which is attracted from sources with peer review processes will be regarded as an indicator of esteem. External research income which is attracted from industry and other funders of research will be regarded as an indicator of esteem and of the breadth of impact of submitted research activity. Information on external research income will be normalised by the number of research active staff cited in form RA1. It is expected that in form RA5 submissions will comment on the relationship between the generation of research income and their research strategy. RA5 should also be used to mention any grants held at other institutions (hence not reflected in RA4 expenditure), or research-related awards from funding council initiatives which are not recorded in RA4. Diversity of funding sources will be regarded as a strength.
3.9.27 Form RA5 should detail the current structure, organisation and environment of the submission. Submissions should address each of the following areas.
Research Structure and Environment
3.9.28 Submissions should describe research groupings or areas, listing which submitted researchers contribute to each, and how research activities are organised within each area. This section should also indicate how each grouping or area is supported by research income, and give details of any particularly significant sources of income which have enhanced or helped to develop the area. A full list of grants awarded is not required. Submissions should describe the main achievements of each area since the last RAE, drawing attention to recent achievements or activities not yet reflected in publications in RA2 (e.g. significant papers 'in press') or research spend in RA4 (significant new awards which have not yet commenced).
3.9.29 Submissions should list other UoAs to which related work has been submitted and detail any difficulties of fit between departmental structure and the UoA framework.
3.9.30 Submissions should explain the mechanisms and practices for promoting research and sustaining and developing an active and vital research culture. Submissions should describe the nature and quality of the research infrastructure, including facilities for research students.
3.9.31 Submissions may indicate how they encourage and support long term research projects here, and describe any arrangements which are in place for supporting interdisciplinary or collaborative research. They may provide information on relationships with industry and commerce or other research users, and where appropriate on the account taken of Government policy initiatives and objectives.
3.9.32 Submissions should describe the arrangements for the development and support of research staff, including contract research staff. They should describe any arrangements which are in place for developing the research of members of staff who are new to research, or who have previously been research inactive, and for integrating them into a wider, supportive research culture.
3.9.33 The submission, where appropriate, should comment on how the departure of staff in categories A* B and D has affected the strength, coherence and research culture of the submission at the census date, and may also explain the role and contribution of staff who have been recruited more recently.
3.9.34 Submissions should provide a statement about the main objectives and activities in research over the next five years. Well-defined and potentially realisable objectives will be viewed as more impressive than general declarations of intent.
3.9.35 Submissions should evaluate progress and success in the period. The Panel will have available the future research plans submitted in the 1996 RAE. Each submission should evaluate its progress against the plans submitted in 1996, and describe how and why these plans may have been modified. Particular strengths in the submission should be highlighted and elaboration given on apparent weaknesses.
Evidence of Esteem
3.9.36 This section should detail additional indicators of research activity, quality and/or peer esteem. Information provided in this section must be sufficiently clear to be potentially verifiable through audit.
3.9.37 The submission should give examples (since January 1996) of (no more than two of) each of the following for each individual researcher listed in RA1 (indicating 'who, what, when and/or where'):
3.9.38 The indicators listed in 3.9.37 will only be used to enhance judgements of quality based upon research outputs and other indicators of research quality.
3.9.39 For established researchers the Panel will examine the indicators for clear evidence of international-level esteem. For less experienced staff, activities such as reviewing grants or journal articles will be viewed as positive signs of research activity and promise.
3.9.40 In addition to this individual information, aggregate information for the submission may be used to help inform assessment of the research environment and culture, together with the vitality and breadth of a submission's research activities. It could include any of the following:
Individual Staff Circumstances
3.9.41 This section should appear under RA6b and should include any individual staff circumstances which have significantly affected their contribution to the submission eg., periods of sick leave, career breaks, engagement on long term projects, also details of junior staff.Contributions by Non Research-active Staff
3.9.42 Institutions may also wish to discuss the relative contribution made to research by staff who have not been returned as research active.
3.9.43 The Panel will be taking a structured, but non-quantitative, approach to its assessments.
3.9.44 The Panel's assessments of quality will be reached by an examination of the whole submission at a number of levels. The proportions of individual researchers reaching levels of international and national excellence will form one component of the assessment, but the Panel will also examine research outputs in aggregate across the submission as a whole. Although the quality of research outputs will be regarded as the principal indicator, the Panel will assess the quality of the research of individual researchers and their collective outputs alongside the evidence from other parts of the submission describing research students, research grants and the research infrastructure and vitality. The Panel will look for evidence that other indicators add value to the quality of individual researchers particularly when judging the activities of less established researchers. The overall assessment will be based upon the submission as a whole.
3.9.45 Before the first grading meeting the Panel will meet to agree who will lead discussion on particular submissions, which outputs will be sampled and by whom, which submissions need to be sent to specialist advisers and/or to other panels, and the timescales over which this advice will be sought.
3.9.46 All members of the Panel will read all submissions, with two or three (for larger submissions) individuals nominated to take the lead in the discussion of individual submissions.
3.9.47 Prior to the first grading meeting each individual panel member will independently grade all submissions (excluding any in which they have a declared interest), based upon the criteria above. Gradings will be submitted and collated independently by the secretary prior to the first grading meeting of the Panel. These individual gradings will be used to establish preliminary gradings, to identify submissions about which there is disagreement and to identify those which are on the borderline between gradings. Preliminary assessments will identify submissions or areas within them where further work needs to be conducted. The preliminary assessments will also identify a set of submissions which will be referred to the non-UK based experts for comment.
3.9.48 The sub-panel of users will receive all submissions, and will meet to define and discuss its working procedures. The sub-panel will be invited to comment upon the field as a whole and individual submissions where appropriate. There will be one meeting of the user sub-panel with members of the Psychology Panel.
3.9.49 The Panel will consult with a group of non-UK based experts drawn from a range of areas of the discipline and from different parts of the world, who will be asked to comment upon the Panel's identification of international excellence. The Panel will make a preliminary identification of such submissions relatively early in its cycle of meetings, indicating these to the group of international experts at this stage and asking for their assessments in writing before its penultimate grading meeting. Such assessments will be advisory.
3.9.50 The Panel expects to reach all its decisions by consensus. In the unlikely event that consensus cannot be achieved, disputes will be resolved by a vote of all panel members (excluding those with declared interests in the issue to be resolved), with the majority view determining the decision. The Panel will ensure consistent and equitable treatment of all submissions by applying the criteria equally to all submissions, using specialist advisers when necessary, using the advice of non-UK based experts, and by deciding its gradings in a cycle of meetings, returning to reconsider grades reached early in the process in the light of decisions taken at later meetings.
Last updated 30 March 2000