Southgate Review
Response to Dame Lesley Southgate's Review of the Part 1 MFPH examination
Download the full Southgate Review.
In 2005 the Faculty of Public Health commissioned a review of the MFPH Part 1/Part A examination in direct response to an incident which occurred at the May 2005 sitting.
At the time FPH decided that the opportunity should be taken to have a comprehensive external review of the entire examination rather than simply consider the narrow question as to what went wrong in that particular examination sitting. Dame Lesley Southgate, one of the experts who had led the changes at PMETB (now merged with the GMC), was asked to carry out this complete review.
Inevitably such a review takes time, but it has now been discussed by the board of FPH and can be published.
Some of the recommendations that came from the subsequent report demanded immediate action, which was taken. Other recommendations required considerable work and, in some cases, deeper research to find the most satisfactory way forward.
Below is a report, updated in September 2010, which sets out the action that has been taken on each recommendation.
Action taken on recommendations
R1. The Faculty should make a clear statement of the primary purpose of the Part 1 examination.
It is a test of the early knowledge and skill base of trainees for the practice of public health in the UK. In so doing it should make clear the implications for other candidates who take it for different purposes and from different settings.
FPH Response: It has always been a function of the Part A MFPH examination that it is a test of the early knowledge and basic skills of trainees in public health in the UK. The implementation of a new curriculum in 2007 (updated in 2010) explicitly acknowledges and informs others of this function of the examination.
See http://www.fph.org.uk/uploads/2010MASTERPHCurriculum0610b.pdf and
http://www.fph.org.uk/uploads/training_pathway_diagram.pdf
The Part A examination is intended to test candidates' knowledge and understanding of the scientific bases of public health, and their ability to apply their knowledge and skills to the practice of public health. It consists of two written papers (Paper I and Paper II). Both Papers I and II are split into two parts/components, A and B (Paper IA, Paper IB, Paper IIA and Paper IIB), and taken over two consecutive days.
Candidates are expected to have acquired specialist knowledge and skills in public health and show a clear understanding of the principles and methods of related disciplines - notably applied statistics, behavioural sciences, health economics and management.
See http://www.fph.org.uk/part_a_exam
R2. The Faculty should press on with writing the curriculum for UK training in accordance with PMETB standards.
The Part 1 examination should assess content defined in the curriculum to the standard defined within the curriculum for the early stages of training.
FPH Response: The standard expected of Part A MFPH candidates is set out in the Public Health Curriculum, the revised 2010 version having been approved by the General Medical Council (GMC) and UK Public Health Register (UKPHR).
The Part A Matrix illustrates the syllabus knowledge requirements mapped to the Key Areas of the curriculum.
http://www.fph.org.uk/uploads/PartA_Matrix_linked_to_KAs.xls
The Assessment blueprint also illustrates the learning outcomes which are assessed via the Part A examination.
See http://www.fph.org.uk/uploads/FPH_assessment_blueprint.pdf and
http://www.fph.org.uk/uploads/PartA_syllabus.pdf
R3. The examination should be blueprinted against the curriculum within an overall blueprint which takes account of which aspects are best tested in a national examination, and which are best assessed in the workplace (RITA competencies).
MPH programmes should be considered within this framework and steps taken to ensure that programme directors around the country are aware of this approach.
FPH Response: As part of the construction of the curriculum, methods of assessing each competency and its knowledge base were devised, in consultation with educationalists. As part of this process the existing Part A syllabus was mapped against the new competency framework. The Part A Matrix illustrates the syllabus knowledge requirements mapped to the Key Areas of the curriculum.
http://www.fph.org.uk/uploads/PartA_Matrix_linked_to_KAs.xls
The Part A examination is structured to address each area of the syllabus, and is set by the examiners taking into account the whole breadth of the syllabus and its different sections. The questions selected for each examination are also checked against the syllabus.
Additionally, the full public health assessment framework is blueprinted onto the learning outcomes in the curriculum via the Assessment Blueprint.
http://www.fph.org.uk/uploads/FPH_assessment_blueprint.pdf
R4. There should be urgent consideration given to the test formats chosen for assessing knowledge.
The construction of a test bank is a good beginning but the quality of the test material needs review. The Faculty should consider asking a consultant on writing examination questions to undertake a detailed review and to run some item writing workshops for examiners.
FPH Response: A question bank for Paper I of the examination has been in existence for over seven years. The bank is reviewed annually by examiners during the question setting process, with existing questions updated and new questions added to the question bank.
During 2007/2008 consideration was given to the feasibility of introducing alternative and practical test formats which still enable the assessment and testing of the Part A syllabus. MCQ, EMQ, short answer questions and computer-based examinations were all considered as part of the programme of work.
MCQ and EMQ questions were not implemented due to the feasibility of developing a valid item bank across the breadth of the Part A syllabus; it was agreed that this was not an appropriate means of testing the knowledge application or skills required at Part A. Various other factors, including the size of the Part A candidate base and issues of cost, ruled out a move towards a computer-based examination.
However, consideration was given to an evolutionary approach of changing an element of the Part A (Paper IIB). In January 2010, a new format Paper IIB was introduced in response to this recommendation and recommendation 9 (see below). Questions for this new paper are of short answer format and are also drawn from a re-usable question bank, which is constantly being updated and developed.
All MFPH examiners are required to undertake training prior to becoming an active examiner, including having to mark a series of training scripts in order to gain experience and to standardise their marks against those of previous examiners. All MFPH examiners receive specific training on examination procedure and guidance on examiner marking criteria. FPH has also begun to explore providing specialist input into question writing.
R5. The test bank should contain completed questions with model answers periodically reviewed for accuracy and currency.
Material should address areas that are stable, which underpin the practice of public health, and which are not susceptible to changes in healthcare policy.
FPH Response: FPH works with the regulators (GMC and UKPHR) to ensure standards for feedback and examination formats are met.
The Part A question bank contains questions with key points, rather than model answers. Key points illustrate the main points that examiners mark against and that they expect to be covered in a good answer. The provision of model answers has previously been discussed and ruled out, as there is no single way to answer most of the questions posed in the Part A examination.
FPH is currently reviewing the structure of the published key points and examiner comments in order to achieve a standardised means of providing feedback.
R6. Test security needs urgent review and immediate institution of good practice.
FPH Response: Strict procedures governing test security have been considerably enhanced, with due regard to data protection legislation and the security of the examination as a whole.
R7. Enforcement of deadlines for test construction and finalising the content of papers is a priority.
A senior examiner must take responsibility for proofreading the paper and signing it off for printing. This includes ensuring that all ancillary material necessary for answering papers will be available for candidates on the day.
FPH Response: Deadline for question construction and finalisation of content are set and are strictly enforced, and dual-proofreading of the paper by the Chair of Examiners and the FPH Academic Registrar was introduced following the Southgate report. FPH process has now gone further than this with additional proofreading by: a lay proof reader/editor, the Chief Censor of the Hong Kong College of Community Medicine and, in addition, the examination papers are passed to the Irish Faculty of Public Health Medicine, which runs an examination with a similar format, for checking.
R8. A senior examiner should be present at the examination.
FPH Response: The Chair of Examiners was present at each examination sitting between June 2006 and June 2009. This format was reviewed and it was felt that the same level of advice and support could be achieved from senior examiners as long as they were contactable via phone (on call) during the examination. It was therefore agreed that physical presence of senior examiners was not necessary, as long as they were instantly able to be contacted via phone. FPH therefore ensures that a minimum of two senior examiners are available to be contacted on each day of the examination.
Clear protocols are in place to deal with unexpected incidents at the examination, with both invigilators and FPH staff briefed on these beforehand.
The Head of Education and Training, the Senior Examinations Administrator and the Chair of Examiners are all available via phone for candidates sitting the examination in Hong Kong.
R9. The Faculty should take steps to adopt criterion referenced (content-based) standard-setting procedures as part of its commitment to the development of the examination in line with PMETB QA principles.
Faculty Response: The Part A has never had more than 200 applicants in a single year, and often a lot fewer. Educationalist advice received states that the requirements for a statistical precision around proposed standards are difficult to be achieved given the numbers involved and the frequency of use of each test (question). Accordingly, thought has been given to possible alternatives to the Part A in its current format if this recommendation is to be followed.
Over the past four years the Part A Development Group has reviewed the entire examination in light of the Southgate Review and the PMETB recommendations. The Part A Development Group concluded that the modifications to the format of one element of the examination offered the principal scope for improving the standardisation of the examination. Paper IIB has been restructured as a consequence.
The new format of Paper IIB was introduced at the January 2010 sitting of the examination. The new structure is designed specifically to test the range of skills required and to enable the examination board to assess the correlation of performance on Paper IIB against that on other papers at the same sitting, and to assess the average performance, across sittings, on the one question which was common to both examinations. With two sittings now available, it is possible to begin to carry out these analyses and FPH will continue to monitor these statistics at future sittings, which will expand the opportunities for cross-sitting comparisons. Such comparisons were, indeed, one of the principal reasons why the change in format was proposed, and were strongly favoured by PMETB (now the GMC).
R10. The resource for the Education and Training Department, and support for senior examiners must be reviewed.
There should be no reliance on untrained or temporary staff to undertake tasks where high-level judgement or knowledge of public health is required.
FPH Response: The resource for the Education and Training Department has been increased by the appointment of a senior examinations administrator at a higher grade than that normally used when appointing new staff. In addition, specific funds will be set aside for educationalist advice and input.
Attachments
Documents
