RAE2001 logo

Contents

Another UoA

Section III: Panelsí Criteria and Working Methods


3.30 Economics and Econometrics, UOA 38

UoA Descriptor

3.30.1 The Unit of Assessment (UoA) comprises all aspects of economics and econometrics, whether theoretical or applied (including, where appropriate, economic history).

Purpose

3.30.2 The Panel has drawn up these criteria within the published arrangements for the 2001 Research Assessment Exercise, to provide more detailed guidance in the formation of judgements about the quality of research in the Economics and Econometrics UoA.

UoA Boundaries

3.30.3 The Panel will, where necessary, seek advice from Panels in cognate disciplines via the exchange of submissions. These will include the following UoAs: Business and Management Studies, Agriculture, Accounting and Finance, Statistics and Operational Research, Town and Country Planning, History, Social Policy and Administration, Sociology, Philosophy, Psychology, Politics and International Studies, Law, Environmental Sciences, Geography, and various area studies' UoAs including Asian Studies. Submissions may be cross-referred to another Panel if this is deemed appropriate.

3.30.4 Where submissions made to other UoAs contain significant bodies of economics and/or econometrics research, it is expected that the other panels concerned shall refer them to this Panel for advice. This applies particularly to UoA 43 Business and Management Studies.

Sub-Panels

3.30.5 The Panel will make use of specialist advisers if this is felt to be appropriate following initial consideration of submissions. However, responsibility for the determination of final ratings rests wholly with the Panel. It is not planned to establish any sub-panels, although relevant submissions will be referred to the Development Studies Sub-Panel of the Geography UoA, which includes a member of the Economics and Econometrics Panel.

Interdisciplinary Research

3.30.6 Full credit will be given to interdisciplinary works, for example, those within the area of Development Studies. The Panel includes interdisciplinary practitioners to facilitate this. Where appropriate, interdisciplinary work will be referred to other relevant Panels or, if necessary, to specialist advisers for consideration.

3.30.7 The Panel Chair will be a member of the Umbrella Panel comprising the Chairs of UoAs in the social sciences. Discussions at this forum will further inform the evaluation of work crossing the boundaries of UoAs.

Joint Submissions

3.30.8 The Panel expects that all submissions to the UoA will be from a single institution. Should, however, two or more institutions elect to make a joint submission, detailed information as set out in Section 4 of the RAE Guidance on Submissions (RAE 2/99)

should be provided, together with an explanation of the rationale for a joint submission. The Panel will treat such submissions as single, for the purposes of assessment.

Treatment of Evidence

3.30.9 International and national excellence will be defined in terms of:

  1. Overall quality of nominated published outputs which will be assessed according to criteria which will include:
  1. Substantive contribution in the broad sense, including research contributions to theory, methodology, policy and practice;
  2. Originality;
  3. Technical excellence.
  1. Research depth, vitality and prospects which will be assessed taking into account:
  1. Postgraduate research activity;
  2. Indicators of peer-review esteem;
  3. Development and staff profile of department;
  4. External research income.

3.30.10 The Panel will base its assessment of submissions primarily on its professionally informed judgement of the quality of research outputs.

3.30.11 International excellence will be interpreted within a wide international context. This will be defined by reference to high quality research activity, where it has been identified, around the world. (An international or comparative focus for research does not of itself signify international excellence. Conversely, work which has a national or local focus is capable of attaining standards of international excellence.)

Overall Quality of Nominated Research Outputs

3.30.12 The Panel will review all nominated research outputs and will read extensively from the full works cited in form RA2. Outputs which have been through a rigorous editorial and refereeing process may be judged of a certain quality without being read in detail. However, in addition to works already known to members, the Panel collectively will examine in detail at least 50% of all nominated outputs.

3.30.13 Whilst attaching importance to the peer refereeing process, the Panel recognises that some research of high quality may well be published in less prominent journals or via non-refereed electronic media (for example, particular specialisms, major invited papers or work which is highly innovative). The Panel recognises that high quality research output may be produced for governmental or commercial organisations.

3.30.14 The Panel's reading will be targeted in the following ways:

  1. Where material cited is not previously familiar to the Panel member;
  2. Where material has been published in a less well-known form or output medium, including work cited as discussion papers;
  3. Where the quality rating under consideration is judged to be marginal or otherwise crucial to the rating decision, eg close to the boundary between two points on the rating scale;
  4. Where the submission demonstrates that special circumstances require additional material to be read to enable a fair judgement to be made;
  5. Where cross-referral to another Panel is judged to require consideration and the submission has not specifically requested cross-referral.

3.30.15 The Panel's judgement on the quality of research outputs will be set in the context of its assessment of all the evidence presented in the submission in terms of research depth, vitality and prospects, as outlined in the paragraphs which follow.

Research Depth, Vitality and Prospects

3.30.16 The Panel will look for evidence of a research culture. The Panel will look at the plans for future research (included in form RA5) in the light of the attainment of the plans submitted in the 1996 RAE return. Well defined realistic objectives will be viewed as more impressive than descriptions of intent.

3.30.17 The Panel will take equal note of the contribution of category A, A* and C staff when making an assessment of the quality of research, as long as the main focus of the research of Category C staff is at the department in question. The research record of staff in categories B and D will be considered as further evidence of a research culture. To assist this, departments are invited to provide information in the textual commentary regarding the research output of staff who have left the institution during the assessment period (outside of those in category A*).

3.30.18 Where institutions select low numbers of staff for assessment, the Panel will need to consider whether the department has sufficient 'critical mass' to sustain a research culture and to be able to achieve its research plans and objectives. Small submissions (where fewer than five staff are submitted as research active) should include a full description in the textual commentary regarding any aspects of the research environment, within or beyond their department, which help sustain and support the research activities of those submitted for assessment.

3.30.19 The inclusion of fewer than four research outputs (two outputs for category A*) in a submission for staff who have been research active throughout the assessment period may be regarded as an indicator of lack of depth. However, where it is stated that a member of research active staff has taken a career break for family reasons during the period of assessment, the Panel accepts that a lower number of publications may be appropriate. In addition, the Panel will have regard to textual information presented in form RA6 regarding other individual circumstances (for example, whether an individual is a new entrant (see paragraph 3.30.20 below); is in part-time employment; has been ill during the period of assessment; is engaged on long-term projects; or has been involved in initiatives such as the Teaching and Learning Technology Programme) when making this assessment.

3.30.20 It is recognised that less established members of staff (in particular those new to academic careers within the research assessment period) may well have an output which does not equate with what would reasonably be expected of a more experienced researcher. The Panel will take this into account. It will evaluate the quality of such work in terms of what is reasonably attainable by an active researcher in the early stage of his or her career, and it will set this research activity in the context of the research culture demonstrated in the department as a whole. The Panel would wish to have due regard to potential and departments may wish to indicate in the textual commentary where they have a policy of encouraging inexperienced researchers, and give information on research active staff who have completed doctoral theses within the assessment period.

3.30.21 While the Panel does not view either single or joint authorship of a piece of output as indicating anything about quality, it recognises that joint authorship within an institution may be evidence of a healthy research culture. Nevertheless, multiple citation of the same output for more than one researcher in a submission may be indicative of a lack of depth or productivity. The Panel will consider each case of multiple citation of the same output on its merits and, where appropriate, take it into account in the overall assessment of a submission. Where this will assist the Panel in its deliberations, departments are invited to provide a rationale in the textual commentary for the inclusion of multiple citations of the same output.

3.30.22 The Panel will look carefully at the staff profile for each department, having regard to staff turnover, career development and growth. Clear explanations of the role and involvement of joint appointments, whether with another UK or overseas institution, and those on fixed term appointments from overseas institutions, will be required.

3.30.23 Departments should indicate in form RA6 evidence of peer review esteem, such as editing and refereeing activities and major honours given to individuals in recognition of their research achievements. Those involved in editing major internationally circulated journals and monographed series will be given due recognition.

Research Output (RA2)

3.30.24 The Panel will read and review the following, in no hierarchical order, subject to the appropriate sampling:

  1. Articles in refereed academic, professional and other journals
  2. Books
  3. Chapters in books
  4. Refereed conference papers
  5. Electronic publications
  6. Government reports
  7. Departmental discussion/working papers, where these can be shown to have been made widely available in paper based or electronic form outside the submitting institution
  8. Software, where this can be shown to have been made widely available outside the submitting institution.

3.30.25 Where these can be shown to include the outcome of the writer's research, the production of teaching materials, textbooks and other subject-related pedagogy is recognised as a valid and valued form of research activity. In such cases, the Panel will assess these in a similar way to other published output. Where necessary, submissions will be cross-referred to the Education UoA or specialist advice will be sought.

3.30.26 All types of outputs will be assessed according to criteria set out above.

Research Students and Studentships (RA3)

3.30.27 Postgraduate research activity will be assessed taking account of the number of and growth in research students and research studentships as well as the number of postgraduate research degrees awarded as evidenced by form RA3.

External Research Income (RA4)

3.30.28 The Panel recognises that external research income, as recorded in form RA4, is an input not an output measure. This will therefore be viewed by the Panel as evidence in assessing research depth, vitality and prospects. The Panel will place greater emphasis on external research income which is peer reviewed. In general, the pattern (size and origin) of grants will be accorded greater weight than the absolute value. Institutions are asked to describe in RA5 how the pattern of funding relates to their research activity.

Textual Commentary

RA5

3.30.29 Submitting departments should address the following matters in the textual commentary in RA5:

Research Structure & Environment

  1. Describe the University context in which the research takes place, including the nature and quality of the research infrastructure, as well as any facilities for research students;
  2. Explain the departmental mechanisms and practices for promoting research and sustaining and developing an active and vital research culture;
  3. Give information on research groupings where there are particular areas of strength, indicating who belongs to them (referring to form RA1), their prime activities, how they operate and their main achievements;
  4. Give a commentary on relationships with industry and commerce, government departments, official agencies or other research users and, where appropriate, on the account taken of Government policy initiatives and objectives;
  5. Give information regarding how the pattern (number, size and origin) of external funding relates to research activity;
  6. Describe any arrangements which are in place for supporting interdisciplinary or collaborative research and give information about collaboration at national or international level;
  7. Provide a rationale for the inclusion of any multiple citations of the same outputs in RA2;
  8. List other UoAs to which related work has been submitted and detail any difficulties of fit between departmental structure and the UoA framework.

Staffing Policy

  1. Describe the arrangements for the development and support of the research work of staff, including young or inexperienced researchers and their integration into a wider, supportive research culture;
  2. Describe arrangements for supporting and developing research students and achievements of research students;
  3. Where appropriate explain the role and contribution of category A* who have joined the department;
  4. Where appropriate, comment on how the departure of staff in categories A*, B and D has affected the strength, coherence and research culture of the department at the census date. Information regarding the published output of staff who have left the institution during the assessment period (outside of those in category A*) may be included;
  5. Give an explanation of the involvement in the department's research of any individuals holding joint appointments, whether with another UK or overseas institution, and those on fixed term appointments from overseas institutions.

Research Strategy

  1. Provide a statement about the main objectives and activities in research over the next five years;
  2. Give a commentary on the development of the department during the period since the last RAE, with reference to the research plans indicated in the 1996 RAE submission. This should provide the context for a self-assessment of performance in relation to the issues detailed above in the textual commentary.

RA6

3.30.30 Submitting departments may wish to address the following matters in the textual commentary in RA6:

Evidence of Esteem

  1. List indicators of peer esteem which relate to the staff submitted.

Individual Staff Circumstances

  1. Note any individual staff circumstances which have significantly affected their contribution to the submission;
  2. Give an explanation for any unusually rapid staff turnover.

Contributions by Non Research-active Staff

  1. Institutions may also wish to discuss the relative contribution made to research by staff who have not been returned as research active.

Additional Information

  1. Provide information regarding research active staff who have completed PhD/DPhil theses during the period of assessment;
  2. Where appropriate, provide an explanation of critical mass;
  3. Give indicators of international visibility;
  4. Where appropriate, give further information regarding forthcoming publications. This information should not be included in form RA2;
  5. Indicate other items of special note.

Working Methods

3.30.31 All departmental submissions will be reviewed by all members of the Panel for a preliminary independent assessment.

3.30.32 The Panel will then arrive at preliminary ratings and all submissions will subsequently be considered in greater detail.

3.30.33 Two academic members of the Panel will be assigned to assess in detail all components of particular submissions. The two members will be assigned primarily with regard to their declared expertise and overall workload. No Panel member shall rate any submission in which they have declared an interest.

3.30.34 An interim rating on the RAE scale will be given to every activity and, from this, submission. At this stage, the Panel will view each individual person named in the submission as an 'activity'. The rating will be informed by a wider view of the evidence presented in other parts of the submission, against the criteria regarding research depth, vitality and prospects given in paragraph 3.30.9 b. above.

3.30.35 The interim evaluation of each submission will be further informed by an appraisal from one of the User Members and the Panel Chair.

3.30.36 The interim assessments and other recommendations of Panel members and recommendations from other Panels or specialist advisers will be collated by the Secretary and each rating will be discussed and confirmed by the full Panel. If significant differences arise submissions will be revisited.

3.30.37 Before confirming the award of the highest ratings, the Panel will consult with a group of up to five non-UK based individuals, one of whom will be a user, with expertise to cover a range of areas in economics and econometrics and from a range of countries outside the UK. The role of this group is purely advisory and responsibility for the determination of ratings rests wholly with the Panel.

3.30.38 In general, all quantifiable parts of a submission (for example, external research income and numbers of research students) will be normalised to allow for the number of staff contributing to that part. Thus, the size of the department per se will not directly affect the outcome, although it is recognised that successful research departments are likely to have grown over time and there may be considerations of the critical mass necessary to sustain a vital research culture.

3.30.39 A quantitative dimension will inform the Panel's deliberations in the assessment of ratings (for example in the determination of 'majority' and 'virtually all'). However, the process will ultimately be driven by the professional judgement of the Panel.

3.30.40 The Panel will aim to reach consensus on each departmental rating, ensuring that judgements are made against the stated criteria. A vote will be taken if necessary, based on a simple majority of Panel members. If required, the Chair will exercise a casting vote.


Last updated 17 April 2000

[ Home | About the RAE2001 | Panels | Guidance for panel members | Guidance for institutions | Data collection | Publications | Contacts ]