Report of the work of the RAE 2001 Task Group on Education Research
- This group was convened as one of several to advise on particular aspects of the 2001 RAE. The HEFCE has established a number of groups to look at issues of panel membership, criteria and assessment processes and the inclusion of industrial and user perspectives in the Research Assessment Exercise. This group looked specifically at these questions in relation to education research.
Terms of reference
- The terms of reference of the group were:
To consider how the RAE might best deal with education research, in particular to consider:
- the appropriateness of developing sub-panels for the education unit of assessment;
- the nature of the panel membership and process for appointment;
- how to ensure that cross-disciplinary working, methodological innovation, impact on practice and methods of dissemination are taken into account within the assessment process.
The membership of the group is Annex A.
A wide range of education research topics and themes are returned to the education panel, and concern has been expressed about the ability of a single panel to assess such a wide range of work. Consequently, the group considered whether there was a case for establishing a number of sub-panels focused on particular aspects of education research. In 1996 there was a single sub-panel for continuing education. This worked well: continuing education was a sufficiently distinct topic of a sufficient size. The group considered whether other sub-panels would be appropriate, and considered in particular research themes identified by the National Foundation for Education Research (NFER) as the basis for such sub-panels. The group concluded that the concerns expressed about the assessment of education research would be better addressed by a modest increase in the size of the main panel rather than by breaking up the education panel into sub-panels – except for a sub-panel for continuing education research as had existed previously.
The group looked particularly closely at the case for sub-panels in education policy, education practice and education management, but concluded that the relationship between these was close, and that to divide the assessment of research in these topics would be artificial and potentially damaging. It also looked closely at the case for a separate sub-panel for higher education pedagogic research – that is to say research relevant to the teaching of discipline subjects in higher education. There is concern that there is unclarity about where such research should be returned and that, particularly when it is returned to the subject discipline panel – for example research into the teaching of physics in higher education to the physics panel – it is not assessed properly.
The group agrees that it is most important that the academic community should have confidence that such research will be properly assessed, and recommends that it should in general be returned to the education panel. An analysis conducted by the Centre for policy Studies in education (CPSE) at the University of Leeds for the HEFCE confirmed that in 1996 this was the almost universal practice. However, there will be occasions when it may be inappropriate to return such research to the education panel (for example where no other education research exists in an institution), and in such cases the group advises that the panel to which such research is returned should seek the advice of the education panel about the quality of the work. However, there does not seem to the group to be any more reason to create a specific sub-panel for the assessment of higher education pedagogic research than there is to create panels for the assessment of school level research. In addition, the CPSE study concluded that there was little evidence to suggest that a sub-panel for pedagogic research was required, but that the panel ought to review submissions to the 2001 exercise to see if submitting behaviour had changed.
We noted that the 1996 panel split into teams to manage the workload (with an active process of cross-validation). On balance the group felt that this approach should be repeated and that a panel drawn broad and large enough to handle the workload and requirement for different perspectives was preferable to the creation of formal sub-panels. A modest enlargement of the panel could address many of the same concerns as the creation of sub-panels, but sub-panels would bring extra complexity and yield few additional benefits.
The group therefore recommends that the education panel should adopt the same initial structure as for the 1996 exercise – a single continuing educational sub-panel beneath the main education panel, but that the panel should be somewhat enlarged to increase the perspectives represented and in order to manage the workload. The group also recommends that this approach should be reviewed following receipt of submissions, and if it is clear that there are distinct areas of work that would benefit from assessment in dedicated sub-panels, such should be created to advise the main panel. In particular, it may prove that submissions may indicate a need for a sub-panel to advise on the assessment of education research in relation to information and communication technology.
Panel membership and assessment process
A major issue concerning education research is that practical or practice-orientated research should not be accorded lower status, and should be considered on its merits alongside more theoretical work. It is essential that the RAE should identify and recognise quality no matter what its nature. There is a further general concern about educational research, that more should be of direct relevance to teaching practice, but that is beyond the terms of reference of the group. The group notes, however, that the RAE is not the only lever for change; and indeed the HEFCE PedR (Teaching and Learning) Initiative, under which significant sums are reserved specifically for pedagogic research of relevance to practitioners, is explicitly aimed at addressing this issue.
The group believes that in order to ensure that practice-orientated research is properly assessed, users of research should be represented on the panel. This concern with user representation, and the need to ensure that applied or practice orientated research is properly assessed, goes wider than education research. More generally the funding bodies are taking steps to ensure that panels have appropriate representation of users to enable such research to be properly assessed.
In concluding that users of research should be represented on the education panel, the group nevertheless believes that the definition of ‘users’ should not be drawn too tightly and should not focus on a single form of user (e.g. school teachers) as this would not allow the full range of appropriate perspectives to be represented. The TTA "cockshy" drawn from the person specification for their user panel (Annex B) is recommended as a good starting point for identifying the requirements of a teacher practitioner, but a broader person specification is required for panel user representatives more generally. Further, it is recognised that the chair will need to exercise considerable discretion in choosing user representatives in order to obtain the right balance on the panel. Moreover, and most importantly, in order to maintain the credibility of the assessment process it is essential that users should have direct experience of education research and should evidently be competent to assess its quality.
The group recommends that around 25 per cent of the members of the education panel in 2001 should be broadly defined as users. For instance, in addition to the chair, there might be twelve academic member and three or four users. The group further recommends that if the panel works in a similar way to 1996, by splitting into teams, each team should contain at least one user representative. In addition the definition of user should be drawn widely, they should be experienced in education research and some should have current and direct involvement in education at classroom, school or LEA levels.
Assessment process and criteria
Many of the generic changes in the RAE process will help to improve the assessment of education research. In particular, the changes in the rules for selection of panel members and chairs, arrangements for increasing consistency and coherence between panels, provision for agreeing and consulting upon criteria, and the more structured approach to the textual parts of submissions (RA5 and RA6) will serve to improve the assessment of education research.
In 1996 the main determinants of the rating were research outputs (usually published papers, conference proceedings etc) and research income (determined by the outcome of peer review processes in funding organisations). This was enhanced by a more subjective analysis of the narrative parts of submissions contained in RA5 and RA6, which helped form a view of the research culture, and these together formed the basis for the judgements of quality.
The group considered whether to recommend that impact and/or relevance should be included as a criterion. It must be remembered that the purpose of the RAE is to identify quality and to inform the allocation of research funds based on quality. However, there is an argument that relevance to users, the effectiveness with which research is disseminated and the extent to which it is taken up are quality indicators. The group felt that it would be desirable for the assessment process to enable impact and relevance to be recorded and taken into account by the panel. A paper discussing how this might be achieved, prepared by members of the task group, is presented at Annex C.
The published assessment criteria for the 1996 exercise included the statement that ‘positive consideration will be given to the dissemination of research findings in professional journals and related media, where the emphasis is on communicating them to practitioners and/or contributing to policy discussion. Where cited publications are of this kind, it will be important for the submissions to indicate clearly their relationship to the underlying research and to how the quality of that research can be assessed’. The group believes that the RAE process should allow for impact, relevance and dissemination to be recorded more systematically than was the case in 1996, and for the panels to take this into account as appropriate in their judgements about the quality of the work submitted.
The group therefore recommends that in 2001 the educational panel should, while taking as its primary criteria the quality of published research outputs and the research culture, develop a mechanism to record the influence of the research undertaken – in terms of the relevance and/or dissemination and/or impact – such that this type of information could be used as a secondary assessment criterion. This could be facilitated through adding three additional fields to the RA2 returns, which will the field of the enquiry, the prime audience and educational significance, and a description of the theoretical and methodological approach to be recorded against each publication.
Summary of recommendations
That the education panel should adopt the same initial structure as for the 1996 exercise – a single continuing educational sub-panel beneath the main education panel, but that the panel should be somewhat enlarged to increase the perspectives represented and in order to manage the workload. The group also recommends that this approach should be reviewed following receipt of submissions, and if it is clear that there are distinct areas of work that would benefit from assessment in dedicated sub-panels, such should be created to advise the main panel. In particular, it may prove that submissions may indicate a need for a sub-panel to advise on the assessment of education research in relation to information and communication technology.
That around 25 per cent of the members of the education panel in 2001 should be broadly defined as users. For instance, in addition to the chair, there might be twelve academic member and three or four users. The group further recommends that if the panel works in a similar way to 1996, by splitting into teams, each team should contain at least one user representative. In addition the definition of user should be drawn widely, they should be experienced in education research and some should have current and direct involvement in education at classroom, school or LEA levels.
That the educational panel should, while taking as its primary criteria the quality of published research outputs and the research culture, develop a mechanism to record the influence of the research undertaken – in terms of the relevance and/or dissemination and/or impact – such that this type of information could be used as a secondary assessment criterion. This could be facilitated through adding three additional fields to the RA2 returns, which will allow the field of the enquiry, the prime audience and educational significance, and a description of the theoretical and methodological approach to be recorded against each publication.
Education Task Group Members
Mr B Bekhradnia (Chair)
Director of Policy, HEFCE
Dr D Pilsbury
Head of Research Policy, HEFCE
Professor M Bassey
Executive Secretary, British Education Research Association
Professor S Brown
Professor of Education, University of Stirling
Professor D Hargreaves
Professor of Education, University of Cambridge
Mr A R Martin
Principal Research Officer, DfEE
Professor P Mortimore
Director, Institute of Education, London
Professor A Pollard
Professor of Education, University of Bristol
Ms J Sebba
Senior Educational Advisor, DfEE
Ms F Sulke
Head of Teacher Training, Teacher Training Agency
TTA "cockshy" person specification for teachers involved in commenting on research
Teachers involved in commenting on research need to have the ability to:
- evaluate the relevance of research questions for improving teaching and learning;
- evaluate the quality of research and of evidence and findings and their interpretation by researchers;
- evaluate the effectiveness of dissemination and impact strategies.
Evidence of such abilities may be found in experience and/or knowledge of:
- experience of using research and evidence carried out by others to challenge and support colleagues to improve their practice;
- experience of conducting research and/or collecting the interpreting evidence and presenting it for testing and interpretation by peers in the public domain;
- experience of participating in research and development projects grounded in systematic enquiry beyond their own school i.e. at local, regional or national level;
- experience of supporting and advising academic researchers through the processes of conducting and disseminating research;
- experience of using research and evidence to improve their own practice.
Extract from a paper submitted to the task group by Professors Pollard and Bassey
- It is suggested that the RA2 returns in Education have three additional fields as descriptions for publications that could help the panel in making its judgements. These are: field(s) of enquiry, prime audience and educational significance, theoretical and methodological description.
Thus the information provided for each publication would be:
Name of author (surname followed by initials or forenames) (as in RAE 1996)
Sub-area (as decided by the institution) (as in RAE 1996)
Year of publication/output (as in RAE 1996)
Type of output (A to E) (as in RAE 1996)
(a) Names of co-authors (as in RAE 1996)
(b) Name of journal/publisher/conference or venue (as in RAE 1996)
(c) Volume number and pagination if applicable (as in RAE 1996)
(d) Field(s) of enquiry (addition for RAE 2001)
(e) Prime audience and educational significance (addition for RAE 2001)
(f) Theoretical and methodological description (addition for RAE 2001)
- Field(s) of enquiry (say up to three in order of priority), based on the BERA analysis reported in Research Intelligence 61, using key numbers. Where this is found to be impractical the miscellaneous category of X could be used. Word descriptions should also be given. (Not more than 64 alphanumeric characters).
art education in nursery education
sociological theory developed in HE setting
music education for SEN children in primary schools
astrological views of headteachers
It is suggested that both the key number and the word description be given so that the former could be used for the submitting department to give an overview of the whole submission and the latter for the convenience of the panel members in their reading.
We suggest that this would considerably facilitate the work of the panel in making clear in what educational field each publication is located and in showing where the main work of a department lies. It might lead to some focusing of the initial reading of experts within the panel.
- Prime audience and educational significance might be seen as one, two or three of researchers, policy-makers and practitioners plus an elaboration giving educational significance. (One line only – not more than 64 alphanumeric characters: perhaps abbreviations would help, e.g. res or p-m or pract).
Researchers: development of understanding of children's reading
Practitioners: critique of mental arithmetic procedures
Policy-makers: homework policies in primary schools
Researchers and policy-makers: challenge to concept of LEAs
Researchers and historians: education in the reign of George III
Researchers: birth-date and cognitive development
We suggest that this could help underpin the making of judgements by the panel on educational significance.
- Theoretical and methodological description. Here 'theoretical' refers to any conceptual framework within which the research may be set. 'Methodological' is taken to mean research approach and arena for data collection, analysis and interpretation. (One line only – not more than 64 alphanumeric characters).
Some illustrations show how this could illuminate the title of an article with value:
Ethnographic case studies in 3 primary schools in Kent
Random survey of 10% of secondary schools in Lancashire
Action research on changing management in an FE college
Slavin-type review of 33 studies of calculators in primary ed
Post-modernist critique of UK policy on teacher education
Master's level text on qualitative research in education
Geographical analysis of LMS funding documents
We suggest that this would alleviate the problem met by the previous panel of knowing what a paper is about when it is not possible to actually read it.
Last updated 7 June 2001
| About the RAE2001
| Guidance for panel members
| Guidance for institutions
| Contacts ]