Research Assessment Exercise 2008: Verification and Audit
Overview of data verification and audit for the RAE 2008
1. The purpose of data verification in the RAE was twofold. Firstly, it was necessary that data in submissions were correct so that panels could make informed assessments on the quality of each submission based on accurate data. Secondly, it was important for the principle of fairness that no institution should gain advantage in the assessment or subsequent funding calculation on the basis of inaccurate data.
2. As stated in RAE 03/2005 'Guidance on submissions', in principle, any part of a submission could be audited; all data should be verifiable through accurate records maintained by higher education institutions (HEIs).
3. The RAE data verification process and any subsequent physical audit of data adhered to principles of good regulation: proportionality, accountability, consistency, and transparency. The audit process balanced the needs of the funding bodies and assessment panels so that the data in submissions was of reliable quality to inform both a quality assessment and subsequent funding calculations; whilst taking account of the desire not to burden institutions unnecessarily with further data requests. Proportionality was understood in terms of the likely risk and consequences of HEIs submitting poor quality or inaccurate data. As a consequence, any auditing of data was targeted.
4. The mechanisms for collecting electronically available outputs using Digital Object Identifiers (DOIs) reduced the chances of HEIs making data input errors on bibliographic data on form RA2 than in the previous RAEs: HEIs could use the RAE 2008 data collection system and CrossRef to cross reference the bibliographic data with the DOI and vice-versa.
5. The submission process through the data collection system automatically validated data: the RAE Team did not send back printouts of submissions for HEIs to check and 'sign-off' as accurate. Submission functionality was added to the data collection system in June 2007, and full details of how this facility operates were provided at that time.
6. It remains the intention of the funding bodies to publish the majority of the data contained in submissions on the internet in 2009. This will include the names of selected staff and the listings of research output, but exclude personal and contractual details and details of research outputs in the category 'confidential report'. We will present aggregate data on supervised research students, supervised research assistants and research income. The data collection system included a facility for institutions to mark as confidential the names of any staff or other data that should be omitted from the published data, for reasons of personal or commercial security or future institutional strategy. Institutions should have ensured that any item that they wished to exclude on these grounds from the published data was flagged before the submission was made.
RAE data verification procedures (Updated 15 November 2007)
7. Further information on arrangements for audits and data verification following the submission deadline of noon on 30 November 2007 was sent to designated RAE contacts at HEIs on 15 November 2007 and is reproduced below.
HEIs should be aware that the planned data verification process has four main strands. The following provides further information about the likely timing of this work to help HEIs in their planning during 2008, though HEIs should bear in mind that the RAE Team may request further information to verify submissions at any time.
- Routine checking
We will audit a proportion of submissions from each HEI as part of the routine checking of RAE data. At least one and up to four UOAs will be selected randomly from each HEIs submissions, depending on the number of UOAs submitted by an HEI and the total number of FTE Cat A staff submitted. Letters requesting evidence to support data included in RAE submissions will be sent to HEIs on or around 7 December 2007. HEIs will then have approximately 25 working days to send the requested documentation to the RAE team by 22 January 2008. The types of evidence HEIs may be asked to provide will include, for example, extracts from payroll or personnel records (such as copies of staff contracts, with salary information deleted); student registration or graduation records (such as extracts from published lists of higher degrees awarded); research grant and contract documentation, and other similarly detailed information. Our actual requests for evidence will list the acceptable forms to be supplied in each case.
- Data comparison
Data from submissions from all HEIs will be subject to comparison with other available datasets, and may include checking RAE data against data held in other datasets, such as Higher Education Statistics Agency data and Research Activity Surveys. This will also include internal consistency checks across all submissions. This work is likely to be undertaken immediately after submissions are received and we expect to contact institutions about queries raised as a result of these checks from December 2007 to April 2008.
- Panel instigated audit
Any data across the full range of submissions may be subject to verification at the instigation of the relevant main and/or sub-panels. This strand of work will be directed by the requests of panels, and as such it may happen at any time throughout 2008. However, based on the broad timetable of the panels it is likely that requests to verify data queries raised by the panels will peak between April and August 2008.
- Targeted audit
In the event that the checks described above indicate the submission of poor quality data, the RAE team may seek further information in relation to the full range of submissions made by an HEI. We would expect to undertake this work from March 2008 onwards. Institutions may be required to provide information of the types described above or other appropriate evidence depending on the nature of the enquiry. Our actual requests for evidence will list the acceptable forms to be supplied in each case.
8. The RAE Team will contact the designated institutional RAE contact in the first instance. The routine checking requests identified above will be sent by post. Audit requests instigated as a result of queries from panels or as a result of our other data comparisons may be instigated by e-mail or letter as appropriate. Except for the routine checks described in paragraph 7a above (where we hope to provide a longer lead time to take account of the Christmas and New Year break) we will require a response to any such requests within 10 working days. If data in any RAE submission are found to be inaccurate following verification and liaison between the RAE Team and the nominated RAE contact at your institution, the RAE Team will notify the nominated RAE contact of any amendments that have been made to the data that was originally submitted and that is held by the funding bodies. The RAE Team will also notify panels of any data amendments that we make so that these can be taken into account in the assessment process. In the event that serious/persistent inaccuracies are discovered, the funding bodies may take the action described in paragraph 10 below.
9. Other than in the course of the verification procedures described above, the RAE Team will make changes to submitted data only in the event that outputs listed as 'pending' in RA2 are not published during the publication period, or if HEIs discover inaccuracies in their submitted data after the submission deadline, but prior to the commencement of substantive discussions by the panels in January 2008. In such cases, the incorrect data will be withdrawn from the submission and therefore not be considered in the assessment process.
Sanctions in the event of serious discrepancies
10. HEIs are reminded that in line with their financial memoranda with the funding bodies, the relevant funding body reserves the right to recover costs from grant should they or their agents need to visit HEIs to collect or review data following targeted or panel instigated audits.
11. In the event of persistent/serious discrepancies in the data contained within submissions, the RAE Team will inform the relevant funding body of the nature and extent of the errors. The funding bodies reserve the right to impose proportionate sanctions if discrepancies are identified which could lead to a material impact on future grants or the outcome of the research assessment exercise.