PRACTICAL ADVICE

Year : 2011  |  Volume : 24  |  Issue : 3  |  Page : 641

Adapting the World Federation for Medical Education standards for use in a self-audit of an eye care training programme

B Tousignant, R Du Toit
Fred Hollows Foundation, Newmarket, Auckland, NewZealand

Correspondence Address:
B Tousignant
Fred Hollows Foundation, Newmarket, Auckland
NewZealand

Source of Support: None, Conflict of Interest: None

Check

1

PMID: 22267360

Introduction: In 2006, a Postgraduate Diploma in Eye Care (PGDEC) for mid-level health personnel was initiated in Papua New Guinea, in partnership with The Fred Hollows Foundation New Zealand, the local government and Divine Word University. In the absence of national accreditation and with limited resources, an interim evaluation was needed.
Methods: We adapted the World Federation for Medical Education (WFME) standards to use in a self-audit to evaluate nine areas and 38 subareas of programme structure, processes and implementation. We developed a rating system: each area and subarea was scored for partial or complete attainment of basic or quality development levels. Ratings were referenced with supporting documents. Data were gathered internally, through document census and meetings between stakeholders.
Findings: A qualitative and quantitative portrait emerged: all nine programme areas completely attained at least basic level and two completely attained the quality development level. Twenty-six (68%) subareas completely attained the quality development level. Key successes included the administration of the PGDEC, synergies between the partnership's stakeholders and its relationship with the public health system.
Discussion and Conclusions: This self-audit adapted from WFME standards provided a simple, yet systematic and largely objective evaluation. It proved beneficial to further develop the programme, highlighting strengths and areas for improvement.

Keywords: Allied health personnel, education, health occupations, Keywords: Evaluation studies, ophthalmology, Pacific islands, quality control

Context and Objectives 

To address the dearth of qualified eye care providers in Papua New Guinea, a wide range of stakeholders met in 2006 to discuss training options for eye care personnel. Representatives included academic, government, eye care providers and non-governmental organisations active in the field.

The only formal eye care course available at that time was for doctors, specifically a Master’s course in ophthalmology at the University of Papua New Guinea, with few graduates per year¹. Only short, ad hoc training courses were provided for mid-level personnel. The global initiative of Vision 2020: the Right to Sight is aligned with training mid-level eye care personnel to ease the population burden of avoidable blindness^2,3. To meet Pacific eye care needs, a Postgraduate Diploma in Eye Care (PGDEC) had been developed by The Fred Hollows Foundation New Zealand (FHFNZ) and provided at the Pacific Eye Institute in Fiji. This one-year, competency-based course leads to a formal academic qualification. It includes a clinical strand (Refraction, Essential Eye Care and Operating Theatre) and a public health strand (Community Eye Care, Quality Management and Health Promotion)⁴.

A partnership was developed between the Papua New Guinea government, FHFNZ and Divine Word University (DWU) to implement the PGDEC in Papua New Guinea as of 2007. The programme is presented at the Modilon teaching hospital and graduates are awarded the PGDEC qualification by DWU. Aiming for programme sustainability⁵ and professional recognition of the graduates as specialist eye care personnel⁶, academic recognition from the DWU Academic Committee was sought and obtained and in 2007. Applications for recognition of the qualification by national Nursing and Medical Boards were also submitted, but as of 2009 they were still being processed.

In 2009, an interim assessment of the implementation and quality of the PGDEC was required as part of ongoing programme evaluation. To guide this evaluation we looked to the World Federation for Medical Education (WFME) standards^7,8, which exist as a framework to facilitate quality improvement in medical education in line with international recommendations⁹. They have been adapted for both basic and postgraduate medical education^7,8 and also for nursing and midwifery¹⁰. The World Health Organization in the Western Pacific region has adapted and endorsed these for the region¹¹. These standards are intended for use in national and regional accreditation, institutional self-evaluation¹¹ or medical programme reform¹².

This paper describes how, against this background and with limited resources, the WFME standards were quantified and the process used in a self-audit of the PGDEC in Papua New Guinea.

Methods

The evaluation framework consisted of the nine areas and 38 subareas of the WFME standards (Table 1). Each subarea has two levels of quality: a basic level, which 'must' be met and a quality development level, which corresponds to best practice – what a programme 'should' strive for (Table 2).



Table 1: List of areas and subareas of World Federation for Medical Education standards used for Postgraduate Diploma in Eye Care self-audit









Table 2: Example of World Federation for Medical Education area and subarea, with Basic and Quality Development levels







No specific data collection methodology is specified in the WFME standards. A series of questions is included to guide qualitative data collection in each subarea. We added a quantitative component by devising a rating system (Table 3). Each subarea was rated between 0 and 4.0: from having no required elements for the basic level, to having all elements of the quality development level, with intermediate ratings given for levels partially attained due to incomplete, poor quality or underutilized elements. A spreadsheet evaluation grid was designed to collect the qualitative and quantitative data. 



Table 3: Quantitative rating scale used to evaluate the Postgraduate Diploma in Eye Care in each World Federation for Medical Education subarea







The self-audit was led by head of the DWU Eye Care Department. Input was obtained through a census of policies, manuals, reports, minutes of meetings and other relevant administrative and educational documents relating to the PGDEC and the partnership stakeholders. Meetings were held with key representatives (DWU academic faculty, FHFNZ programme staff and administrative and clinical staff at Modilon hospital), guided by WFME questions. The Head of Department transformed notes from these meetings and document searches into ratings for each of the areas and subareas. Each was referenced with details of supporting documents and of the main contributing stakeholders.

Analysis of the evaluation grid included deriving, ranking and plotting mean area and subarea ratings. Ratings and justifications were circulated to key PGDEC stakeholders, along with commendation for the strengths and suggestions for improving low-scoring subareas.

Findings

Combining the qualitative questions from the WFME standards with a rating system provided a comprehensive portrait of the structure, processes and implementation of the PGDEC.

Achievements in implementing the PGDEC are visible across all WFME areas. The mean area rating for each of the nine areas attained at least the basic level (rating of 2.0) and two fully attained the Quality Development level (rating of 4.0) (Figure 1).







Figure 1: Mean rating for main World Federation for Medical Education areas in self-audit of Postgraduate Diploma in Eye Care

Looking specifically at the subareas, 35 (92%) attained at least the basic level and the remaining 3 (8%) subareas partially attained it. The quality development level was completely attained in 26 (68%) subareas or partially attained in 8 (21%) subareas. 

The range of subareas ratings is illustrated in Table 4. An example of a high-scoring subarea is Interaction with the health sector. With a rating of 4.0, both basic level and quality development level were deemed attained: formalised collaborations exist in the form of memorandums of understanding between DWU/FHFNZ and with relevant stakeholders within the public health sector. These include formal agreements with government for the use of hospital facilities for the clinical training of the PGDEC, and with employers (public hospitals administrations, faith-based and private providers in the health care system) for graduates to provide eye care after training. 



Table 4: Range of World Federation for Medical Education subarea ratings achieved during Postgraduate Diploma in Eye Care self-audit







The lowest rating (1.0) was obtained in three subareas. For example, in the subarea of training in other settings, clinical rotations for students are currently limited to the main hospital site and community outreaches within the province. Resources limit access to accredited sites elsewhere in the country or internationally.

Many of the successful subareas were linked to one another, sometimes spanning across multiple WFME areas. Examining subareas attaining quality development level partially or completely (rating 3.0 or 4.0), key strengths of the PGDEC were identified, along with common factors leading to this outcome. These strengths relate to administration of the PGDEC, the relationship between the PGDEC and the health system, the quality of the practical training, PGDEC educational aspects and PGDEC ongoing evaluations (Table 5).



Table 5: Key successes of Postgraduate Diploma in Eye Care self-audit, grouped by high scoring subareas







Discussion

Benefits of the Adapted WFME Standards as a Self-Audit Tool

The WFME standards provided a wide-ranging but user-friendly framework for the evaluation of the implementation of a training programme, and the quality of its structure and processes. We added a quantitative rating scale and a recording of the sources of evidence to the qualitative questions of the WFME standards. This provided not only a more comprehensive picture but also added an element of objectivity to this self-audit. This rating system allowed for clear identification of programme strengths and weaknesses and simplified reporting results to stakeholders. The process we used was effective and systematic, yet simple and inexpensive.

This tool allows a process oriented evaluation, commencing from the programme’s initial needs assessment and examining processes following its evolution^13,14. It may also allow a stakeholder oriented process, taking into account the local health system and other actors affected by the course of the training¹⁵.

This tool has face validity: the WFME standards enjoy widespread recognition and use¹⁶. The comprehensive nature of the standards bestows content validity on the tool. Its construct validity is reflected in its convergence with other methods of programme evaluation. Indeed, the 2009 FHFNZ evaluation reports to funders, which used individual interviews and focus groups with stakeholders within the partnership (students, graduates, DWU administration) corroborates the complementary strengths as significant elements in the success of the programme’s implementation¹⁷. The audit findings of meeting educational goals is supported by all PGDEC students successfully fulfilling the assessments and DWU’s academic requirements, in the year of the audit¹⁸.

Lessons Learned: Strong Implementation through Partnership

Overall, this self-audit described the PGDEC fully attained the WFME basic standards. It also identified the areas in need of quality development to meet best practice standards.

This self-audit highlighted key strengths that may assist similar programmes. The findings underscore the benefits of using a health systems strengthening approach¹⁸. These are apparent in the synergies within the partnership between the government, FHFNZ and DWU. In particular, high ratings were assigned to the administration of the PGDEC, integration of the training into the health system and strong educational and evaluative components. These ratings can be ascribed to the sharing of complementary resources and expertise by the stakeholders. For example, the government provides personnel for training and subsequent deployment at public hospitals. DWU provides administrative and student support and awards qualifications. FHFNZ refurbishes and manages Modilon hospital’s eye unit as a clinical training centre for nearby DWU. FHFNZ, with funding from the Australian and New Zealand governments, supplies academic, financial, administrative and staffing support to DWU’s Eye Care department and to the hospital.

The results highlight the dependence of the programme on external/expatriate intervention—educational and clinical staff, clinical and equipment funding, etc.. This first phase was necessary to allow for capacity development of local staff. Transition towards local ownership should however increasingly be supported in order to ensure sustainability.

High ratings relating to governance, leadership and management testify to the benefit of creating a Head of Department for Eye Care at DWU. It allowed a full-time, in-country presence at the centre of the partnership, providing sufficient executive authority at the local level to successfully implement necessary programme components in tune with stakeholder needs, realities and resources.

Finally, the self-audit identified subareas in need of improvement and initiated reflection on potential improvements. For example, to improve the research contents of the programme, the PGDEC could more emphasise students collecting, analysing and using monitoring data to improve quality of care. Formal research components could be incorporated in a master’s programme available to graduates in the future.

Limitations and Suggestions for Future Studies

The WFME questions, rating system and specification of validation material of this tool facilitated objectivity. However, the inherent potential for subjectivity and bias of a self-audit remains, in particular with only one person assigning the ratings. This could in part be overcome by peer checking or by involving key stakeholders in the rating process.

An internal evaluation has obvious benefits, such as contextual knowledge of the evaluator. In this instance, acquaintance with programme structures and processes allowed the collection of information from different sources for triangulation. Also, logistical arrangements and the timeframe associated with planning and execution were minimised. Costs were essentially limited to the time of the Head of Department for the document census and the meetings, which all took place locally thanks to proximity of stakeholders. We, however, concur with the WFME recommendation of an external evaluation in addition to a self-evaluation to make the process more valuable¹⁹. A panel of medical educationalists⁷, for example, may bring a higher level of objectivity and different viewpoints, but they would be more removed from the operational context. 

Ongoing PGDEC programme evaluation could benefit from regular self-audits by one or more individuals associated with the programme. Further ratings could be used to set, and thereafter to assess performance benchmarks.

Conclusions

This self-audit using WFME standards with a rating system provided a simple, low-cost and systematic process for evaluating the quality of the PGDEC as a training programme. The rating system provides some objectivity to an internal evaluation and simplifies reporting. This process identified programme elements requiring improvement and those contributing to its success. Regular self-audits undertaken by an individual, or with key stakeholders, could be valuable for ongoing programme evaluation. The addition of a periodic external evaluation using this process may reduce bias and provide different perspectives.

Acknowledgements

The authors would like to acknowledge the financial support of the Fred Hollows Foundation New Zealand. Many thanks to the academic and administrative staff of Divine Word University and of Modilon General Hospital Eye Unit for contributions to the self-audit.

References

1. Du Toit R, Palagyi A, Williams C, Ramke J, Brian G. Education of eye health professionals to meet the needs of the Pacific. New Zealand Medical Journal. 2009; 122(1291):69–76.

2. World Health Organization. Global initiative for the elimination of avoidable blindness. Geneva: WHO publications; 2000.

3. Du Toit R, Brian G. Mid-level cadre providing eye care in the context of Vision 2020. New Zealand Medical Journal. 2009; 122:77–88.

4. Du Toit R, Palagyi A, Brian G. The development of competency-based education for mid-level eye care professionals: a process to foster an appropriate, widely accepted and socially accountable initiative. Education for Health. 2010; 23(2):368.

5. Du Toit R, Faal H, Courtright P, Mason I. Checklist for setting up an education programme. Moshi, Tanzania: Kilimanjaro Centre for Community Ophthalmology; 2007.

6. Du Toit R, Hughes F, Mason I, Tousignant B. Facilitating the quality of care in a specialist Pacific ophthalmic nursing workforce. International nursing review. 2011; 58(1):79–88.

7. World Federation for Medical Education. Basic Medical Education - WFME Global Standards for Quality Improvement. WFME Office: University of Copenhagen; 2003.

8. World Federation for Medical Education. Postgraduate Medical Education - WFME Global Standards for Quality Improvement. WFME Office: University of Copenhagen; 2003.

9. Grant J, Marshall J, Gary NE. Pilot evaluation of the World Federation for Medical Education’s global standards for basic medical education. Medical education. 2005; 39(3):245–246.

10. World Health Organization Regional Office for Africa. Guidelines for Evaluating Basic Nursing and Midwifery Education and Training Programmes in the African Region. Brazzaville: WHO publications; 2007.

11. World Health Organization Regional Office for the Western Pacific. WHO Guidelines for Quality Assurance of Basic Medical Education in the Western Pacific Region. Manila, Philippines: WHO publications; 2001.

12. Karle H. Global standards and accreditation in medical education: a view from the WFME. Academic medicine?: journal of the Association of American Medical Colleges. 2006; 81(12 Suppl):S43–48.

13. Galukande M, van Berkel H, Wolfhagen I. Developing a tool for measuring quality of medical education. Education for Health. 2009; 22(2):308.

14. Maccarrick G, Conroy R, Cathal K. Preparing for an institutional self review using the WFME standards – An International Medical School case study. Medical Teacher. 2010; 32:e227–232.

15. Cook DA. Twelve tips for evaluating educational programs. Medical Teacher. 2010; 32(4):296–301.

16. Wilkes M, Bligh J. Evaluating educational interventions. British Medical Journal. 1999; 318(8):1269–1272.

17. Fred Hollows Foundation of New Zealand. Monitoring visit report, Papua New Guinea. Auckland, New Zealand:

18. Divine Word University. Academic Board Proceedings. Madang, Papua New Guinea: Divine Word University.

19. World Health Organization. Everybody business?: strengthening health systems to improve health outcomes?: WHO’s framework for action. Geneva: WHO publications; 2007.