Section III: Panelsí Criteria and Working Methods
3.59 Education, UoA 68
3.59.1 The criteria and working methods for assessment in 2001 resemble those from the Research Assessment Exercise (RAE) 1996, but with some additional clarification and changes to reflect new general requirements for RAE 2001. The Report of the work of the RAE 2001 Task Group on Education Research has been taken into account in effecting some of the expansion and amendments. In particular the Education Panel has been expanded to include members from the broad spectrum of the "user" community for educational research, who will make up approximately one quarter of the membership. As in 1996, the Panel has tried to avoid outlining a single model for submissions in setting its criteria.
Research: diversity and quality
3.59.2 The diversity of research in education, in content and in methodology, requires the Panel to be flexible in marking the boundaries of work relevant to the RAE. For example, curriculum, teaching and assessment materials may be included, but only where these are based on, or developed through, research and have been published. The inclusion of such items should be justified explicitly in submissions in relation to the underlying research. It is the quality of the research alone that will be assessed. The characteristics of quality that the Panel will use in making its judgements about research will include such features as originality, the contribution to the advancement of knowledge, methodological strength, scholarly rigour and relevance for other researchers, policy makers or practitioners.
3.59.3 Basic, strategic and applied research will all be given equal consideration. In addition to work that develops theoretical understanding, a great deal of research in education is intended to shape policy and practice. As recommended by the Task Group on Education Research, the Panel will ensure that practical or practice-oriented research is considered on its merits alongside more theoretical research. The quality of research will often be demonstrated through its influence on other researchers working in the same field, or on policy makers and practitioners. Evidence of such influence will be taken into account, though the Panel recognises that such impact is not always direct and can take some time to become observable. The Panel is also aware that research can fulfil an important function by offering independent criticism of policy or practice. Submissions should also identify in RA6 research carried out in the department that is regarded by others as "cutting edge" theoretically, substantively, or in methods of investigation.
3.59.4 The Education Unit of Assessment and its associated communities of users may be broadly described as concerned with research into such areas as:
Pre-school, primary, secondary, further, higher, teacher or other professional, adult, continuing, vocational and community education or training, work-based learning and lifelong learning.
Assessment, curriculum, teaching, pedagogy, learning and ICT in education.
Such as special educational needs; counselling; comparative, international and development education; education and industry; education policy; organisation, governance and management; social exclusion/inclusion and equity issues in education.
History, psychology, philosophy, sociology and other disciplines of education.
Qualitative, quantitative, ethnographic, evaluation, action research and other methodological procedures used in educational research.
3.59.5 The Education Panel will not be competent to assess work, whether in Continuing Education or elsewhere, which has no significant "education" dimension (for example, which is exclusively in archaeology or literature). Such research should therefore be submitted whenever possible to other relevant subject panels.
3.59.6 The Education Panel also expects to be asked by some other subject panels to advise on research-based work that has particularly strong pedagogic dimensions.
3.59.7 In judging a department's work, the Panel will show no general preference either for breadth of research activity or for concentration on a few research areas. The nature of that choice and the reasons for it should, however, be made clear in the submission.
3.59.8 The Panel will examine submissions that span the boundary between two or more UoAs on a case by case basis and will either request that another panel or will itself take the lead for the submission, depending on the balance of work submitted.
3.59.9 The Education Panel will create a sub-panel to advise it on the Continuing Education components of submissions where these are substantial. Two members of the Panel with research experience in continuing education will be joined for this purpose by up to five others of whom one will be a "user" representative. Continuing Education is not closely defined, but it may include elements such as adult learning, training of adults, work-based learning, access and wider participation, lifelong learning and informal or community-based learning. These elements will normally be in the post-compulsory period of education or training. Research in further, higher and teacher education will, however, be considered by the main panel with advice from the sub-panel where, on a case by case basis, this is deemed appropriate. Departments are invited to indicate in their submissions which parts, if any, they consider appropriate for consideration as research in continuing education. The Panel will also refer to the sub-panel those submissions, or parts thereof, it judges apposite.
3.59.10 As recommended by the Task Group on Education Research, the Panel will review this position once the submissions have been received. If it is clear that there are distinct areas of work that would benefit from assessment in other dedicated sub-panels, such will be created to advise the main Panel.
3.59.11 The Panel expects to require specialist advice in the assessment of submissions in the Welsh Language and will seek additional advice where necessary in other areas.
3.59.12 Education is essentially an interdisciplinary area and the Panel has therefore been convened with representations from a broad spread of relevant expertise and disciplines to ensure coverage of the great majority of submissions. In cases where, however, the Panel feels it is not fully competent to assess a piece of interdisciplinary work, that part of the submission will be assessed on a case by case basis for specialist advice or consideration by another Panel.
3.59.13 The Panel will consider joint submissions on a case by case basis. The validity and appropriateness of presenting the submission in a joint form will be scrutinised.
Treatment of Evidence
3.59.14 The Education Panel's collective judgements will be based firstly on:
and secondly on:
3.59.15 Amongst the characteristics of both the quality of research outputs and the vitality of research culture may be included the educational significance of the research and its relevance for the academic community, policy makers and practitioners.
3.59.16 In relation to (b) above, careful attention will be paid to accounts in RA5 of the department's research strategies, priorities and structures, and how its research achievements are perceived in that context and relate to the strategy outlined in the 1996 RAE submission. It should therefore be made clear where in the submission evidence supporting these statements is to be found. There is no minimum number of active researchers that the Panel has identified as a necessary critical mass, and it takes no account of the proportion of staff submitted. However, where small numbers (ten or less) are returned, it suggests that departments should address this in their accounts of their research culture.
3.59.17 Submissions should make clear how the department construes its research and assesses its research strengths, thereby providing a context for evaluating the extent and quality of its research activity. In 1996 departments were asked to frame their submissions in terms of "sub-areas" and the Panel recognises that this concept has influenced the ways in which many departments now organise their research. In 2001, however, institutions are invited to identify the "activities" which characterise the research submitted and research groups where such groups exist. It is important that submissions indicate clearly in RA5 how the department is presenting its activities and which individuals relate to each activity. The Panel recognises that some individuals will have contributed to more than one activity and that their research output will be distributed accordingly and that some activities may relate to the work of only one individual.
3.59.18 In arriving at a single grade from the ratings of the various research activities, the Education Panel will reach its judgements in ways that recognise the department's overall research performance, strategy and culture. It will take account of the relative size of research groups and the coherence of the elements within their individual activities, in deciding how each will influence the department's overall grade.
3.59.19 In assessing international and national excellence, the Panel will keep in mind the differences between the English, Northern Irish, Scottish and Welsh contexts, the distinctiveness of some systemic features and developments, and the importance of some country-specific journals and other publications.
3.59.20 The identification of international excellence will be based on the quality of the research and not on its nature or scope. Such excellence may be defined as research that is as good as the leading research in those countries where there is a significant body of work in the field. The research would make a significant contribution to the field and might be expected to display at least some of the following: substantial knowledge of developments in theory and practice internationally, significant empirical findings, conceptual contributions, innovative methodologies or techniques, theoretical developments or contributions to innovative developments in policy and practice. Indicators of international recognition of such quality (eg invited key note addresses to major international events, active involvement in eminent research networks, or significant consultancies) may be included in RA6, but will be treated with caution by the Panel.
Research Output (RA2)
3.59.21 The Panel will give its greatest attention to research output from those submissions judged to be near the borderline between grades at all points on the scale. Output which has not been peer-refeeed and work with which the Panel is not already familiar are the likely focus for its reading.
3.59.22 Consideration will be given to the dissemination of research findings in professional journals and other related media where, for example, the emphasis is on communicating them to practitioners and/or contributing to policy discussion. However, where outputs of this kind are cited it will be important for submissions to indicate clearly their relationship to the underlying research and to how the quality of that research can be assessed. To facilitate this and following the recommendations of the Task Group Report, the Panel has requested the addition of three new fields to Form RA2, to be recorded against each piece of research output. These will cover the field of the enquiry (e.g. "teacher education", or "sociological theory developed in HE setting"), the prime audience and educational significance (e.g. "policy makers: homework policies in primary schools") and a succinct description of the theoretical and methodological approach. Departments should use RA5 to provide further information.
3.59.23 No ranking will be applied to different kinds of written research output such as books, book chapters, papers in refereed journals, research reports to funding bodies, papers in refereed conference proceedings and articles or books reviewing research. Each will be assessed on its own merits although rigorous refereeing of publications and editorial procedures may be used as one indicator of quality. Departments may wish to argue in RA5 that the relative significance of different kinds of publication varies among different areas of research in education. The same is true for "publication" in other media such as electronic journals, CD-ROMs, broadcasts, etc. The significance of the editorship of books will depend on the involvement of a substantial research role (for example, in having co-ordinated and contributed to the research programme being reported). Editorship of academic journals may be included in RA6 as an indicator of esteem. Teaching or training material and development work may also be included but must have an explicit research content. Reviews evaluating, synthesising and disseminating the research of others may be seen as an important part of research.
3.59.24 It is anticipated that the majority of cited works will take a traditional printed form. However, the submission of other outputs from research such as videos, software and software related products such as manuals will be acceptable. In all kinds of research output, where it is seen as necessary, the Panel will seek clarification to assure itself that the product describes research as distinct from educational practice. For all kinds of outputs, including such items as broadcasts or software, copies must be available for the Panel to view if required.
3.59.25 The Panel recognises that co-authored publications may provide valuable evidence of research teams working collaboratively within or across institutions and of inter-disciplinary research. Departments are invited to highlight in RA6 the contribution of particular research-active staff to such publications where the contribution has been especially substantial. The Panel will draw no inferences from the order of the authorship on joint publications. However, if there is multiple citing of the same publication, i.e. by two or more members in a department, then it is expected that the submission will justify the inclusion of these multiple citations.
3.59.26 It will not be assumed that fewer than four publications is in itself an indicator of lower academic output, but submissions will be scrutinised for explanations of apparent shortfalls. The Panel recognises that less established members of staff, especially those new to academic careers, may well have an output that does not equate to what would reasonably be expected of an experienced researcher. It will take this into account in making its assessments, and will view favourably the inclusion of newer staff who are active but are maturing in terms of research performance, together with evidence of a research environment which supports their development.
3.59.27 All the cited output will be reviewed. The Panel collectively will examine in detail 10% of the research output cited in the submissions. This will include at least a minimum of 5% from each submission, but the Panel will examine as much as necessary to make an informed judgement. The additional reading will concentrate on submissions identified as borderline from the early reading, and on those elements of output for which the quality is least well known and has been least subjected to other well understood and competent refereeing processes. The Panel has, however, identified two broad criteria (see paragraphs 3.59.14 and 3.59.2) by which all outputs will be judged, regardless of the medium in which they are published.
Research Students and Research Studentships (RA3)
3.59.28 The Panel will consider the numbers per active researcher of research students, research studentships and higher degrees completed within the assessment period as features of the vitality and quality of the research environment. However, it will treat these numbers with caution taking into account, for example, regional and locational differences which may influence recruitment of research students. Departments may wish to identify such factors in their submissions.
3.59.29 Research carried out by research students can represent a significant component of a department's overall research activity. Evidence of the quality of this aspect of research, and the support given to it through training and other resourcing, will be considered. ESRC recognition for research training will be one of the indicators used to assess this part of an institution's profile. Studentships won in open competition will be important; those provided by institutions themselves may be seen as indicators of support for the research culture.
External Research Income (RA4)
3.59.30 The Panel will also consider information on research income per active researcher as supporting data for other parts of the submission. Particular note will be taken of grants gained through competitive and peer-reviewed procedures. It will be helpful if individual grants are explicitly linked in RA5 to the relevant activity/sub-group within the department. Institutions are also encouraged to explain how their research income may be used as an indicator of research productivity and quality.
3.59.31 The Panel will pay careful attention to accounts in RA5 and RA6 of the department's research strategies, priorities and structures, and how its research achievements are perceived in that context. It should therefore be made clear where in the submission evidence supporting these statements is to be found. Submissions should also refer explicitly to significant divergences from the research plans submitted in 1996, new research directions and future plans. There may be good reasons why earlier plans have not been fully implemented, but such reasons should be made explicit and changes should be explained in terms of the department's current research structure and strategy. The Panel should not be expected to infer what should be made explicit.
3.59.32 Departments may wish to incorporate an analytic self-assessment of performance in their submission. They are invited where appropriate to draw the Panel's attention to the research of any group they wish to emphasise, to any ongoing research which is not producing immediate visible outcomes despite being seen as important and to specific points which are relevant to judgements about quality.
3.59.33 The Panel requests information be provided in Form RA5 on the following. In addition, please see also paragraphs 3.59.16, 3.59.17 and 3.59.22.
Research Strategy and Structure, which could include any or all of the following:
Staffing Policy, which could include any or all of the following:
Evidence of Esteem
3.59.34 The Panel requests that RA6 be used to provide additional evidence of peer esteem, which relate to the staff submitted. For example, editorship of academic journals may be included in RA6 as an indicator of academic esteem together with any indicators deemed to relate to international excellence or work which is externally regarded as cutting edge.
Individual Staff Circumstances
3.59.35 In arriving at its overall grades, the Panel will place individual achievements in the context of all the work submitted. It may be appropriate in some submissions for the apparently low productivity of a researcher identified in RA1 to be explained (for example, where the research has a long lead-time or is subject to constraints against early publication) or to indicate where promising but as yet inexperienced researchers have been recruited.
Contributions by Non Research-active Staff
3.59.36 Departments may wish to identify legitimate and genuine contributions made to research activity by staff who, for various reasons have not been included in the submissions as research active.
3.59.37 Departments are encouraged to include other information not specifically mentioned above which is pertinent to and assists a full assessment of the research work in which they are involved.
3.59.38 Despite the anticipated high number of submissions to the Education Unit of Assessment, the Panel has agreed that each member should look at all submissions in the first stage of the assessment. Amongst other advantages, this will provide the opportunity to identify submissions, or parts of submissions, on which specialist advice will be needed or which may be sent to another subject panel. It will also identify submissions that indicate a need for, or are deemed to require, advice from the Continuing Education Sub-Panel. The full Panel will in all cases discuss the recommendations of that sub-panel. All judgements of parts of submissions will be incorporated within the overall judgement of research performance, strategy and culture of the department being assessed.
3.59.39 The Panel will then undertake a "pilot exercise" assessing a representative cross section of approximately ten returns in order to establish common methods of working. Following the pilot phase, the Panel will be divided into three or four groups, each with at least one "user" representative. One third (or one quarter) of the submissions (including the ten pilot returns) will be assigned to each group. Individual panel members will independently assess each submission and the group will then aim to come to a consensus view on the ratings. The full Panel will then convene to discuss any issues of principle or practice arising out of the assessments and to discuss any submission about which there is major disagreement or uncertainty. This process will be repeated two or three times over a series of meetings, by which time all submissions will have been read and assessed by all members of the Panel. The full Panel will then meet to discuss all the assessments and to arrive at a provisional grading. At this point, the Panel will reconsider all borderline cases.
3.59.40 The Panel will nominate a small number of internationally known non-UK based academics to moderate its grading process in the range of 4, 5 and 5*. They will be given full submissions sampled from the three top rating categories and the commentaries produced by the Panel in making their provisional decisions, together with information on the criteria and assessment procedures. The Panel will consider the advice that is offered to it but will retain the responsibility for the final decisions.
Last updated 19 April 2000