The Southgate Review - Action taken on recommendations
This report from the Academic Registrar sets out the action taken on each recommendation in the Southgate Review, dealing first with the recommendations that address immediate threats followed by those that require action over a longer period.
Immediate threats
R6. Test security needs urgent review and immediate institution of good practice.
Faculty Response: This recommendation referred principally to the practice of submission of both draft and completed questions by examiners, and requests to examiners for redrafting from the Faculty, being conducted via email. This practice has now ceased, and all transactions are now conducted on a hand-to-hand basis or by post.
R7. Enforcement of deadlines for test construction and finalising the content of papers is a priority. A senior examiner must take responsibility for proofreading the paper and signing it off for printing. This includes ensuring that all ancillary material necessary for answering papers will be available for candidates on the day.
Faculty Response: Deadlines for test construction and finalisation of content have now been set and are being enforced. From the beginning of 2006 the papers have been/will be proofread by the Chair of Examiners, the Academic Registrar and Assistant Academic Registrar of the Faculty and, in addition, are passed to the Irish Faculty of Public Health Medicine, which runs an exam with a similar format, for checking.
R8. A senior examiner should be present at the examination.
Faculty Response: Although at one time it was the case that a senior examiner was always present at the exam this practice has ceased in recent years, possibly reflecting increasing pressures on examiners. A senior examiner was present at the January 2006 sitting and another will be present at the June 2006 sitting and henceforward.
Requiring action over a longer period
R1. The Faculty should make a clear statement of the primary purpose of the Part 1 examination. It is a test of the early knowledge and skill base of trainees for the practice of public health in the UK. In so doing it should make clear the implications for other candidates who take it for different purposes and from different settings.
Faculty Response: It has always been an implicit function of the exam that it is a test of the early knowledge and basic skills of trainees in public health in the UK. The construction of a new curriculum will allow us to explicitly acknowledge and inform others of this function of the exam.
R2. The Faculty should press on with writing the curriculum for UK training in accordance with PMETB standards. The Part 1 examination should assess content defined in the curriculum to the standard defined within the curriculum for the early stages of training.
Faculty Response: The writing of the curriculum has commenced, although there is a lot of work left to do. The standard expected of candidates will be set out in that document.
R3. The examination should be blueprinted against the curriculum within an overall blueprint which takes account of which aspects are best tested in a national examination, and which are best assessed in the workplace (RITA competencies). MPH programmes should be considered within this framework and steps taken to ensure that programme directors around the country are aware of this approach.
Faculty Response: As part of the construction of the curriculum, methods of assessing each competency and its knowledge base are being devised, in consultation with educationalists. As part of this process the existing Part A syllabus will be mapped against the new competency framework. It is part of the job description of the incoming Chair of Part A Examiners to undertake or delegate and supervise this task with assistance from the Faculty.
R4. There should be urgent consideration given to the test formats chosen for assessing knowledge. The construction of a test bank is a good beginning but the quality of the test material needs review. The Faculty should consider asking a consultant on writing examination questions to undertake a detailed review and to run some item-writing workshops for examiners.
Faculty Response: A question bank for Paper 1 of the exam has been in existence for four years. It is part of the job description of the incoming Chair of Part A Examiners to undertake or initiate an ongoing review of the bank, with advice from a consultant on writing examination questions, and to set up workshops on writing examination questions for the examiners.
R5. The test bank should contain completed questions with model answers periodically reviewed for accuracy and currency. Material should address areas that are stable, which underpin the practice of public health, and which are not susceptible to changes in healthcare policy.
Faculty Response: The question bank contains questions with key points, rather than model answers, and clarification will be sought as to the requirements of the PMETB in regard to the provision of model answers. In other respects this recommendation poses some challenges, as health policy is one of the areas set out in the exam syllabus, but, as curriculum construction proceeds, careful thought will be given to whether some aspects of health policy are better tested in the workplace.
R9. The Faculty should take steps to adopt criterion referenced (content-based) standard-setting procedures as part of its commitment to the development of the examination in line with PMETB QA principles.
Faculty Response: The Faculty will be taking advice from PMETB as to its requirements, and from educationalists as to the feasibility of this recommendation. The Part A has never had more than 200 applicants in a single year, and often a lot fewer. The requirements for statistical precision around proposed standards may not be achievable given the numbers involved and the frequency of use of each test (question). Accordingly, thought will have to be given to possible alternatives to the Part A in its current format if this recommendation is to be followed.
R10. The resource for the examination department, and support for senior examiners must be reviewed. There should be no reliance on untrained or temporary staff to undertake tasks where high-level judgement or knowledge of public health is required.
Faculty Response: The resource for the Education and Training Department has been increased by the appointment of a modernisation administrator at a higher grade than that normally used when appointing new staff. In addition, specific funds will be set aside for educationalist advice and input. The situation will be kept under review.