RAE 2001 : Panels' criteria and working methods

Panels’ Criteria and Working Methods

From Circular 5/99 section III as corrected by Circular 2/00.

3.58 Music, UoA 67

UoA Descriptor

3.58.1 The UoA includes composition and performance (including classical, commercial, and popular); history and criticism of music; ethnomusicology; theory and analysis, including empirical approaches; technology and computer applications.

UoA Boundaries and Interdisciplinary Research

3.58.2 The Panel understands its remit as encompassing academic and practice-based research across the widest range of musical fields.

3.58.3 The Panel is conscious of the multidisciplinary nature of much music research, and will refer to other panels or specialist advisers wherever it is appropriate to do so (for example in relation to research in music education, audio engineering, popular and commercial music practice, or music therapy); final decisions on ratings will rest with the Panel for Music and will be reached against its criteria. The formation of sub-panels is not anticipated.

Joint Submissions

3.58.4 Joint submissions between institutions, if received, will be evaluated as a whole, with supporting data being aggregated and a single rating awarded to the submission.

Treatment of Evidence

3.58.5 Assessment will be based primarily on work listed in RA2, interpreted within the context of the relevant research environment and strategy. Whereas input measures such as external research funding and studentships will be considered as evidence of the research environment, the Panel's overriding concern will be with the quality of research output. Where reference is made to quantitative measures, these will be normalized as appropriate on the basis of the number of research active staff (including category C staff). In the case of fractional and category C staff, the Panel will wish to be satisfied as to the extent and value of the contribution made to the overall research environment, and the extent to which they in turn draw upon it.

3.58.6 The size of departments will not be regarded as an indicator of research quality, nor will breadth of research expertise across a wide range of topics be regarded as necessarily superior or inferior to a concentration of research within a narrower focus.

Practice-based Research

3.58.7 The Panel recognizes the importance of practice-based research to the research cultures and activities of many departments and individual researchers. Practical output which demonstrably embodies original research as defined for the RAE will be assessed on an equal basis with other outputs.

3.58.8 Those submitting practice as research may include (in the 'other relevant details' field of form RA2) a succinct statement of not more than 300 words for each item in this category listed under RA2; for this purpose practice is defined as all outputs listed in paragraph 3.58.12 b) to e) below. These statements should make clear how the practice embodies research as defined in the RAE. They may, where appropriate, include an indication of the aims, methods, procedures, innovation, significance, and context of the practice. It should be noted that the submission of such statements is not a requirement; the Panel will not expect them when the status of the practice as research is self-evident.

3.58.9 Where practice results in multiple outcomes (eg scores, recordings, and performances of the same composition or musical edition), one 'lead' outcome should be identified for inclusion in RA2; other outcomes may be identified in the 'Other relevant details' field of form RA2. Practice-based research originally disseminated prior to 1994 and submitted as such for a previous research assessment exercise may not be submitted in another form for the present exercise (except in the case of revisions, in which case credit will be given only for new elements).

Research Outputs (RA2)

3.58.10 Research in music results in assessable public output of various kinds including, for example, live performances, broadcasts, recordings, films, videos, and computer media as well as conventional published materials. Where appropriate, as in the case of live performances, evidence may be sought that the output was in fact accessible to the public (for instance by virtue of being announced as such). Electronically-disseminated publications (such as internet journal contributions) will be assessed on the same basis as printed materials; where applicable the full URL must be stated in RA2.

3.58.11 In all cases reference will be made to the general definition of research applicable to the RAE, with the following being regarded where relevant as positive qualities: originality or innovation in method, interpretation, or insight; rigour, imagination, scope, substance, or significance; evidence of national or international impact. In reaching a view on the quality of submitted material the Panel will, where appropriate, take account of factors associated with the publication or other dissemination of the work, for example the extent to which it has involved rigorous peer review, but work not subject to such review will not be automatically assumed to be of lower quality; the Panel will be aware of relevant developments affecting, for instance, the music publishing, broadcasting, and recording industries, as well as variations in public support for practice-based research, and will treat such indicators with appropriate sensitivity.

3.58.12 The following listing of research output should be understood as indicative rather than exclusive:

Text-based (literary) publications: no collective ranking of publishers, series, or journals will be made but the Panel may be guided by its knowledge of editorial and refereeing policy. Similar criteria may be applied to conference contributions. Editorial activity (including the editing of period texts as well as the bringing together and preparation for publication of contemporary scholars' work) will be evaluated on the basis of its research content as defined for the RAE; editors of collaborative volumes submitting their own contributions(s) may not wish to include the volume as a separate item in RA2, but their editorial role will be taken into account in evaluation. Teaching material may be submitted where it conforms with the RAE definition of research in terms of disciplinary content or pedagogical approach. Where material published before 1994 is re-issued (eg in a revised edition, a collection of the author's papers, or translation), assessment will be based only on new elements in this publication. Institutions may comment, in the 'Other relevant details' field of RA2, on the research content of edited, teaching, and reissued materials.
Editions of music will normally take the form of published scores. However unpublished editions which have been publicly disseminated through performance, broadcast, or recording may also be submitted; in such cases, institutions must be able to supply a score for assessment purposes. Reference may be made to such factors as the performer(s) associated with the work and the venues(s) in which it is performed, commissioning bodies, prizes and other marks of recognition, and the standing of organizations involved in its dissemination (publishers, broadcasting organizations, record companies); relevant information should be provided in the 'Other relevant details' field of form RA2.
Compositions: all forms of public output are eligible for submission, including published scores, publicly disseminated audio and video recordings, broadcasts, and public performances. In the case of works submitted in the form of broadcasts or public performances, institutions must be able to supply a score or (where the nature of the composition makes this inappropriate) recording for assessment purposes; it is not a requirement that these supporting materials should themselves be in the public domain. Reference may be made to such factors as the performer(s) associated with the work and the venue(s) in which it is performed, commissioning bodies, prizes and other marks of recognition, and the standing of organizations involved in its dissemination (publishers, broadcasting organizations, record companies); relevant information should be provided in the 'Other relevant details' field of form RA2.
Performances: in accordance with the RAE definition of research, performance will be accepted as research where it applies or embodies new or substantially improved knowledge or insights, for instance in terms of interpretation, historical performance practice, or technical innovation. Performance is understood to include conducting and direction as well as instrumental or vocal execution; all forms of public output are eligible for submission, including publicly disseminated live or studio recordings, broadcasts, and public performances. In the case of broadcasts and public performances, institutions must be able to supply a recording (which need not be in the public domain). Reference may be made to such factors as the venue of the performance, the standing of broadcasting organizations or record companies involved in its dissemination, and prizes or other marks of recognition); relevant information should be provided in the 'Other relevant details' field of form RA2.
Computer applications: databases, programs, and equivalent items may be submitted provided they are publicly accessible and embody, in whole or in part, the results of research as defined for the RAE. Responsibility for software specification, design, and/or coding will be recognized where the end product falls clearly within the Music UoA by virtue of its function and/or information content. Information relating to the dissemination of such outputs should be provided in the 'Other relevant details' field on form RA2.
Any other form of public output. These should be accompanied by a succinct statement as defined in 3.58.8.

Research Students and Research Studentships (RA3)

3.58.13 In assessing the overall research environment, the Panel will take account of established patterns of successful research supervision and other indications of high-quality research preparation and support. While institutionally funded studentships will be seen as evidence of commitment to the research infrastructure, studentships awarded on the basis of external specialist review will be regarded as indicators of esteem. The Panel recognizes the uneven patterns of funding across the discipline (for instance in relation to composition and performance) and will not adopt a simply quantitative approach.

External Research Income (RA4)

3.58.14 The Panel will be primarily concerned with the quality of output resulting from externally funded research rather than with quantitative measures of funding. In addition, the Panel recognizes the uneven patterns of access to funding across the discipline. It will however consider external research income, particularly when resulting from a process of specialist peer review, as one of a range of positive indicators relating to the departmental research environment. Institutions may wish to comment in RA5 or RA6 on income which supported research during the assessment period but does not appear in RA4.

Textual Commentary

3.58.15 The Panel will wish to understand submitted work in the context of the departmental research strategy and arrangements for its support (RA5) and indications of esteem (RA6). These sections should be organized according to the following sub-headings. The suggested content is indicative and should be supplemented by other material to which institutions wish to draw the Panel's attention. Commentaries should incorporate an element of self-assessment (i.e. should be critical rather than merely descriptive, identifying measures taken to address perceived problems), but no rating may be proposed.

RA5

Research Structure and Environment

3.58.16 Institutions should provide an explanation of the context within which research is conducted. This should include the nature and quality of the research infrastructure, and the means by which research is promoted and an active research culture maintained. Where appropriate information should be provided concerning any mismatches between departmental and UoA structure; research themes or groupings where these play a significant role in the management of research; any arrangements for supporting interdisciplinary research; and relationships with any external users of research.

Staffing Policy>

3.58.17 Information should be provided on staff development and support, including arrangements for developing the research of younger and/or new researchers and for integrating them into a wider, supportive research culture. The Panel will form a view as to what the balance of staff on permanent and short-term contracts tells them about the research culture of the department; institutions are invited to comment on the role and contribution of staff on short-term contracts, as well as of category A* staff who have joined the department. Comment may also be made on how the departure of staff in categories A*, B and D has affected the strength, coherence, and research culture of the department at the census date.

Research Strategy

3.58.18 Institutions should provide an overview of research activities and achievements during the assessment period (organized as appropriate by individuals, themes, or research groups) and explain the department's research plan and priorities. While assessment will be based on the items listed in RA2, institutions may list other significant public output within the assessment period where they consider that this contributes to an understanding of the overall pattern of activity. Where appropriate, comment should be made on the relationship of externally funded research to the departmental research strategy, and on any major deviations from research plans put forward in the 1996 RAE.

3.58.19 Institutions should provide a statement about the main objectives and anticipated activities in research over the next five years; well defined objectives and measures for their realization will be viewed as more impressive than generalized descriptions of intent. Mention should be made of significant investment expected to result in research outcomes during this period, and of ongoing research which has not yielded major publication or other outputs within the period of assessment; anticipated modes and dates of dissemination should be indicated.

RA6

Evidence of Esteem

3.58.20 Evidence of esteem relating to submitted staff may include, for example, visiting or research appointments, journal or book series editorships, membership of editorial boards and national or international committees, translations of published research into other languages, keynote addresses and invited papers at international conferences, direction of major conferences or festivals, commissions, recording contracts, prizes and other marks of recognition.

Individual Staff Circumstances

3.58.21 The Panel recognises that there are circumstances under which researchers may reasonably submit less than four items of output but will wish to be informed of these circumstances. Information should be provided (with dates and concise details) concerning staff who have completed doctorates or other professional training and/or initiated a research-active career during the assessment period; where such staff are unable to submit significant output, the Panel will expect to see alternative evidence of their research activity, including for instance details of doctoral dissertations (which are recognised as representing significant research), conference presentations, and forthcoming publications. Information should also be provided concerning any individual staff circumstances which have significantly affected their contribution to the submission (e.g. periods of sick leave, career breaks, engagement on long term projects, or participation in projects initiated by the HE funding bodies).

Contributions by Non Research-active Staff

3.58.22 Comment is invited on the role within the department of staff not submitted as research active, including any contribution made towards overall departmental research activity.

Working Methods

3.58.23 Assessment will begin with the evaluation of the work of individual researchers, each of whom will be assigned to two panel members primarily on the basis of subject expertise. The Panel collectively will examine at least 25% of submitted items (one by each individual researcher). Further detailed examination will be carried out selectively and it is anticipated that the overall proportion examined will be in the order of 50%. The Panel will concentrate in particular on those which have not been subject to rigorous peer review. Ratings will be presented to and agreed by the whole Panel.

3.58.24 A preliminary rating of departments will be derived from the ratings of individuals through a process of calculation involving reference to the RAE rating descriptions. The contribution of researchers on fractional contracts will be normalized on the basis of FTE value. In the case of departments including in their submission staff members who have begun research-active careers within the assessment period, this exercise will be carried out twice: once including, and once excluding such staff. This is to ensure that departments are not penalized for appointing and developing entry-level staff; any discrepancy between the resultant preliminary ratings will be viewed in the light of individual circumstances, including details provided of research activity by such staff, and the department's arrangements for and track record in the development of junior staff.

3.58.25 Following preliminary rating, each department will be assigned to two panel members, who will review the preliminary rating(s) in light of the contribution of category C staff, departmental and institutional research environment and strategy, and other information provided in RA5 and 6. Their recommendations will be presented to the entire Panel, all members of which will read all submissions and will be collectively responsible for all ratings awarded. Agreed ratings will be subsequently reviewed in order to moderate between submissions rated at different times. It is anticipated that final decisions will normally be made by consensus, but where this is not possible ratings will be determined by voting.

3.58.26 The Panel will devote particular attention to the boundaries between different ratings. RAE profiling criteria for these turn upon the distinction between research of national and international quality; it should be noted that 'national' and 'international' do not refer to the nature or scope of particular disciplinary areas, nor necessarily to quantitative measures of dissemination. In general, individual work assessed as of international standard will be equivalent to the best work in the same field undertaken anywhere in the world, and will where appropriate demonstrate most or all of the positive qualities mentioned in paragraph 3.58.11 above; work assessed as of national standard will be at least satisfactory in all relevant respects and may be outstanding in some. Again, internationally excellent work is likely to contribute to the formation of new paradigms or the significant extension of existing ones, as compared with work of national standard which may be more limited in its significance. At both the individual and departmental level, the highest levels of achievement will be associated with disciplinary leadership.

3.58.27 The entire submissions (RA1-6) of all departments provisionally rated 5 and 5*, together with submissions from a sample of departments provisionally rated 4, will be forwarded to a group of non-UK based experts who will be requested to comment on the extent to which the departments in question meet international standards of excellence. Final decisions on ratings will however remain the sole responsibility of the Panel.