RAE logo Note: This version of the document contains the Contents and Summary only. The full report is available as a Word (2 Mbytes) or Word zipped (500K) file.

Interdisciplinary Research and the
Research Assessment Exercise

A Report by Evaluation Associates Ltd
March 1999
For the UK Higher Education Funding Bodies

Contents

  • Summary
  • The Interdisciplinarity Project
    • Context
    • The Project
  • Objective 1: Volume
  • Objective 2: Impacts
    • Perceptions
    • The RAE and Researcher Behaviour
    • Other Factors
  • Objective 3: Practices
    • Institutional and Researcher Practices
    • Panel Practices
  • Objectives 4 & 5: Strategies and Objectives
    • Strengthening the RAE: Consistency
    • Strengthening the RAE: Boundary Critical Submissions
    • RAE Recommendations
  • Objective 6: The Longer Term
    • The Evolving RAE
    • Agenda 2006
  • Annex 1: Funding Bodies Consultation
  • Annex 2: Tables
    • The Respondents
    • Factors that Affect Research
    • Taxonomy
  • Notes

Summary

Messages from the Interdisciplinarity Survey

Our report has six main messages:
  1. Interdisciplinary research is pervasive through higher education research. Around four-fifths of researchers report that they are engaged in at least some interdisciplinary work.
  2. Departments and researchers widely believe the RAE inhibits interdisciplinary research. Many institutions and individuals are changing the way they do research as a result of the RAE.
  3. Despite these beliefs, there is no evidence that the RAE systematically discriminated against interdisciplinary research in 1996.
  4. In 1996, there was however:
    • a lack of consistency in the treatment of interdisciplinary research by individual panels, and
    • a problem with departments that needed to split researchers between panels or seek cross referral between panels: these boundary critical submissions gained lower ratings.
  5. The procedural changes we propose for RAE 2001 should give researchers the confidence that interdisciplinary research will receive parity of treatment by all panels in RAE 2001.
  6. There remains a need for a longer-term debate ¨ Agenda 2006 ¨about:
    • the assessment of interdisciplinary research in the RAE and elsewhere
    • the impact of the RAE on institution and researcher practices.

The Interdisciplinary Research Project

Our project was commissioned by the Funding Bodies and had six objectives. We set out to meet these through a large-scale survey of practices and attitudes in higher education research. Below, we briefly outline our conclusions based on the 5,832 responses to our questionnaires, and on discussions with institution managers and researchers.

Objective 1: Volume

To estimate the current proportion of UK research which is interdisciplinary. We have found that interdisciplinary and multidisciplinary research is pervasive throughout the higher education research system. The majority of researchers practise a ŠdualityĂ of research, switching from single disciplinary to interdisciplinary research according to the nature of the research problem or project they are engaged with. Key findings are:
  • researchers spend an average of 46% of their research time on interdisciplinary research
  • 20% of researchers are engaged only in single disciplinary research
  • 10% of researchers are engaged only in interdisciplinary research
  • 70% of researchers are engaged in both single and interdisciplinary research
  • researchers in the medical and biological sciences spend the highest proportion of their research time on interdisciplinary research (53%)
  • the arts and humanities are least involved (43%)
  • female researchers are more involved with interdisciplinary research (50%) than their male colleagues (45%).

Objective 2: Impacts

To assess whether and to what extent the RAE acts as a disincentive to the pursuit of interdisciplinary and multidisciplinary research, and if it does, whether certain areas are more affected than others. There is strong criticism of the impact of the RAE on interdisciplinary research:
  • one quarter of researchers (24%), and nearly one fifth of RAE 1996 panel members (17%), believe that the RAE strongly inhibits interdisciplinary research
  • researchers that are more heavily involved in interdisciplinary research are the most critical, as are those that achieved lower ratings in RAE 1996
  • criticism is strongest in the social sciences, and in the arts and humanities.
A smaller number of researchers believe that the RAE is having a positive influence:
  • 6% of researchers and 6% of RAE 1996 panel members believe that the RAE strongly promotes interdisciplinary research.
Heads of departments report that about one third of departments have adapted their strategies as a result of the perceived treatment of interdisciplinary research in the RAE:
  • 17% report the RAE has discouraged their departments from pursuing interdisciplinary research
  • 14% state it has encouraged interdisciplinary research in their departments
  • 69% report that the RAE has had no effect on their departmentsĂ interdisciplinary research.
The RAE has also encouraged researchers to concentrate on:
  • the quality of their research
  • publishing more papers, particularly in peer reviewed journals
  • research that produces results in the shorter rather than longer term.

Objective 3: Practices

To identify institutional, researcher and panel practices and tactics in handling interdisciplinary submissions to the 1996 RAE, including multiple submission and cross-referral. Overall, RAE 1996 panels did not strongly discriminate for or against interdisciplinary research, although the most interdisciplinary researchers received slightly lower ratings. The average ratings awarded to researchersĂ departments, based on a seven point scale, were:
  • 4.77: high single disciplinary researchers (those committing <=33% time to interdisciplinary research)
  • 4.77: interdisciplinary/single disciplinary researchers (33-66% interdisciplinary)
  • 4.72: high interdisciplinary researchers (67%+ interdisciplinary).
There was, however, variation between panels, with some awarding higher ratings to single disciplinary research, and others rewarding interdisciplinary research. A particular problem was the fit between departmental structures and the RAE. Nearly one quarter of departments (23%) split their researchers between panels, or sought cross-referral between panels. We call these submissions boundary critical:
  • overall, boundary critical submissions receive a 0.5 point lower rating than non-critical submissions
  • in the former polytechnics and colleges, boundary critical and non-critical submissions received almost equal ratings (3.3 and 3.2 respectively)
  • in established universities, boundary critical submissions received an average rating of 4.6 compared with 5.1 for non-critical submissions.
Although small, these differences in ratings are of concern because they suggest that departments suffer during assessment if they are not organised in a way that approximates to the RAE panels.

Objective 4: Strategies

To identify cost-effective RAE and institutional strategies for ensuring proper assessment of interdisciplinary research. Our conclusions under this objective are concerned with building confidence in RAE 2001 and ensuring that necessary improvements are identified. We call this Agenda 2001. Our aim is to ensure that institutions and researchers are not deterred from investment in interdisciplinary research and that they can have confidence that their work will be equitably treated in the RAE. We believe that the problems we have identified with assessment in 1996 will be addressed by changes proposed by the Funding Bodies and by our recommendations (Objective 5). We identify improvements that will strengthen the consistency of assessment in the RAE. Specifically:
  • the range of panel members should:
  • reflect all sub-areas of the units of assessment (UOAs)
  • represent significant cognate areas
  • have experience of single and interdisciplinary working.
  • panels should explicitly state the criteria they intend to apply to interdisciplinary research, and examine and monitor methods that might potentially discriminate against interdisciplinarity
  • new procedures should be introduced for documenting and actively managing boundary critical submissions
  • the new feedback and reporting mechanisms introduced by the Funding Bodies for RAE 2001 should explicitly address how interdisciplinary research was treated by each panel.

Objective 5: Recommendations

If necessary, to recommend revisions to the RAE and provide advice for institutions submitting interdisciplinary research. We have made five specific recommendations:
  1. The Funding Bodies and panel chairs should take steps to ensure the representativeness of panels across all forms and areas of research, including interdisciplinary research.
  2. Panel methodologies should specifically address assessment of interdisciplinarity.
  3. New mechanisms for boundary critical submissions should be introduced based on a new proforma (5C).
  4. The Funding Bodies should introduce monitoring mechanisms to ensure the effectiveness of cross-referral.
  5. Feedback and reporting should embrace interdisciplinary research.
Following our interim report in November 1998, the Funding Bodies accepted these recommendations.

Objective 6: The Longer Term

To advise on:
  • monitoring the volume of interdisciplinary research in the long term
  • making comparisons with other countries
  • monitoring the impact of any recommendations.
The RAE has become one of the most important features of academic life. It is likely to remain so while higher education research is dual funded. Its unpopularity with many researchers should not detract from its development as a long-term vehicle for research assessment. For this reason, we believe that the debate for RAE 2006 should commence now ¨ even though the Funding Bodies are unlikely to confirm before 2003 whether that Exercise will proceed. Researchers, funders and policy makers need to more fully understand and reach a consensus on:
  • the impacts, positive and negative, of the RAE
  • the nature, extent and quality of interdisciplinary research
  • how interdisciplinary research interlocks with single disciplinary research
  • the relative contributions of single and interdisciplinary research to development of fundamental knowledge and to the relevance and application ('worldliness') of research.

[ Home | About the RAE2001 | Panels | Guidance for panel members | Guidance for institutions | Data collection | Publications | Contacts ]