|
 
 |
| PRACTICAL ADVICE |
|
| Year : 2012 | Volume
: 25
| Issue : 3 | Page : 195-197 |
|
Tips for Developing a Valid and Reliable Bank of Multiple Choice Questions (MCQs)
Shazia Sadaf1, Sadaf Khan2, Syeda Kauser Ali3
1 Department of Surgery and Educational Development, Aga Khan University, Karachi, Pakistan
2 Department of Surgery, Aga Khan University, Karachi, Pakistan
3 Department for Educational Development, Aga Khan University, Karachi, Pakistan
| Date of Web Publication | 29-Mar-2013 |
Correspondence Address:
Shazia Sadaf
Senior Instructor - Education Coordinator, Department of Surgery and Educational Development, Faculty Offices, Link Building, Aga Khan University, P.O. Box 3500, Stadium Road, Karachi - 74800
Pakistan
 Source of Support: None, Conflict of Interest: None  | Check |
DOI: 10.4103/1357-6283.109786
Introduction: Multiple choice questions (MCQs) are one of the most frequently used written assessments particularly when there is a large body of material to be tested and a large number of students to be assessed. MCQ examinations have manageable logistics, are easy to administer, can be scored rapidly and difficulty indices and the discriminatory value of each item can be easily calculated, facilitating standard application. Methods of ensuring quality assurance are essential when assessment is used for the purposes of certification. Methods: Developing MCQs requires a concerted effort from curriculum planners, content experts, course coordinators and medical educators. This article aims at outlining a stepwise approach towards ensuring quality assurance in written assessments in an integrated curriculum from aligning assessment with the learning to ensuring banking of good quality MCQs. Discussion: Coordinated efforts and a stepwise approach towards item development, coupled with focused faculty development exercises, is a cost-effective means of developing a huge 'Question Data Bank' of valid and reliable test items which can serve as a National Resource for the Professional Regulatory Councils in the country for their Licensure Examination. Keywords: Faculty development, multidisciplinary, multiple choice questions, post hoc analysis, Question Bank
How to cite this article:
Sadaf S, Khan S, Ali SK. Tips for Developing a Valid and Reliable Bank of Multiple Choice Questions (MCQs). Educ Health 2012;25:195-7 |
| Introduction | |  |
Multiple choice questions (MCQs) are one of the most frequently used written assessments. This tool is particularly efficient when there is a large body of material to be tested and when there are a large number of students to be assessed. [1],[2],[3],[4] MCQ examinations usually have manageable logistics, are easy to administer and can be scored rapidly. [1] Difficulty indices and the discriminatory value of each item can be easily calculated, facilitating standard application. [1],[3] Methods of ensuring quality assurance are essential when assessment is used for the purposes of certification. [1] We outline a stepwise approach towards ensuring quality assurance in written assessments in an integrated curriculum from aligning assessment with the learning to ensuring banking of good quality MCQs.
| Methods | | |
Step 1 - Join forces
The major investment of resources in developing MCQs is in the time taken to write valid items. Developing MCQs requires a concerted effort from curriculum planners, content experts, course coordinators and medical educators. Curriculum planners define the scope of the assessment and approve the assessment modalities; content experts authenticate the knowledge assessed; course coordinators give insight into the actual learning experiences provided, which do not necessarily match with the expectations of curriculum developers and planners; and medical educators provide the technical input in item construction. Combining the efforts of all these individuals requires a fair amount of dedicated administrative support and involvement of departmental leadership, who play a very important role as enablers.
Step 2 - Develop an examination blueprint
Validity and reliability can be termed as the 'Litmus Test' of an assessment tool. For any examination to be reliable and valid, it is essential that it be designed according to a predetermined examination blueprint. [2] This will ensure that all concepts that are considered essential and important are given fair and balanced representation on the examination. [3] This strategy will eliminate wasting testing time and resources on assessing the student's knowledge of non-essential and trivial information. It will also ensure that the examination is not unduly weighted towards a certain topic/subspecialty area/objective. [4]
Once the blueprint has been outlined, it should be compared to the available testing material. [2] Essential concepts will be tested year after year. For the examination to be reliable, it will be necessary to modify the data or information provided in the vignette ensuring a variation in the MCQs used to test the same concept. If the same item is used frequently, it will decay and will become useless as a measure of students' understanding of that concept. [2]
Step 3 - Critical review of the existing question bank
Coordinators involved in selecting items for the examination paper are most likely to recognize faulty items, gaps and deficits in the existing Question Bank (QB). New questions are developed addressing these gaps and the existing questions need to be reviewed and revised for construct and content. [3] This exercise should continue on an ongoing basis, so that each item is reviewed at least every five years. Since medicine is a rapidly developing field, the options need to be reviewed critically in light of new information, thus ensuring content and construct validity. The construction of the question should be reviewed as part of pre and post hoc analysis.
For each discipline, we suggest a critical review of the previous 4-5 years of MCQ examinations by coordinators, content experts and medical educators. This review serves several purposes:
- Determine how well previous examinations matched with the examination blueprint.
- Determine how frequently an item was used over the previous five years. As mentioned previously, if an item is used repeatedly, it may result in low reliability.
- Review of difficulty and discrimination indices of individual items as well as the entire examination. Changing patterns of these indices over the years will serve as indicators of changing teaching strategies, altered exposure to various clinical situations and item decay.
- Identify flaws in MCQ construction that would interfere with using the items as a valid assessment tool. [1]
Step 4 - Faculty development
To ensure a broad-based contribution, all faculty members that will be involved in assessment should receive basic training in the development and critique of MCQs through frequent workshops conducted by experts in medical education. [4] These experts should be available and accessible for periodic review of items and should be able to provide constructive feedback that will allow continuing development of item construction skills. Criticism, without direction, will only serve to discourage further attempts and increase the burden on a select few.
Apart from the gifted 'test-makers', the most common barrier to MCQ development is writer's block. We feel that most faculty members find it hard to develop an MCQ out of context and without direction. Therefore, after reviewing the examination blueprint, the course director/clerkship coordinator should develop a precise list of items that need to be created. For example, if on review of the existing pool of questions, there is no item that addresses common pathogens in the biliary system and their pharmacologic management, the coordinator then assigns these very specific learning objectives to a content/subject expert to develop the MCQs.
Step 5 - Peer and multidisciplinary reviews of new MCQs
Once the MCQs have been constructed, the item should be reviewed by content experts, with often diverse areas of interest. [1] In our setting, all new MCQs go through two cycles of reviews before they are accepted as good quality questions. First is the internal departmental review, comprised of peers within the department. For example, questions related to renal pathology created by a nephrologist are reviewed by a panel comprised of internists, cardiologists, infectious disease physicians, etc. within the Department of Medicine. This ensures that the item is understood in the same manner when read, as was envisioned by its developer and that the information being tested is pertinent and the item is 'age appropriate'. This first review, if done conscientiously, greatly contributes to a high quality item. The exact mechanism of how this review process is carried can vary considerably. In larger departments, a question review committee may serve this purpose. In smaller departments/sections, a series of meetings that can be attended by different faculty members, depending on their availability, serve the purpose well.
The second level of review is the Multidisciplinary Review, which is essential to ensure that all items are in line with the overall objectives of the curriculum. We are cognizant of the fact that no concept in medicine stands alone. Each fraction of information is supported by a multitude of facts and each of these facts may be taught by different experts. Therefore, a valid item will have input from more than one discipline. If an item for a paediatric examination tests an embryology concept, it is only common sense that the embryologist has an opportunity to review and enhance the MCQ. This exercise also allows an estimation of overlapped testing. For instance, the management of myocardial infarction is valid in both an internal medicine and an emergency medicine setting, and indeed, testing of this important concept is valid in both areas. This review is an ideal opportunity to modify these common items so that they subtly assess differing objectives and allows development of a comprehensive MCQ pool.
At our institution, we organize annual retreats involving all chairpersons as a resource to encourage active participation of members of their departments. Clinical faculty, basic scientists, external examiners and medical educators are invited to participate in the retreat, which is held outside the university premises on a weekend. [1] This helps ensure maximum participation and in our experience we have seen the participation grow from 60 faculty members to 145 over the past three years.
Multidisciplinary groups are formed according to the disciplines being reviewed. For example, a group that will review MCQs from obstetrics and gynaecology will have representative content experts as well as representatives from paediatrics, family medicine, surgery, endocrinology, physiology and a medical educator as a resource person within their group. This way, faculty members have an opportunity to develop question-writing skills through a multidisciplinary collaborative learning experience. [1]
Step 6 - Defining item attributes and banking the questions in a question bank (QB)
In order to ensure user friendly storage or 'Banking' of reviewed MCQs, it is essential that as the items are reviewed, revised and accepted, they are assigned attributes that will facilitate extraction from the QB. These attributes include categories such as clinical presentation, discipline, organ system, disease process and specific objectives based on the institution's philosophy on teaching and assessment and the curricular framework. [5] For each item reviewed, an 'MCQ attribute sheet' should be completed, which facilitates the process of feeding the MCQs into the QB. Institutions differ in the systems and programs they employ for maintaining a QB. Our QB is Excel-based with defined attributes, psychometric details including difficulty and a discriminatory index, date of administration, and is isolated from the internet with effective virus/worm protection and secure data back-up.
Step 7 - Selecting items for the examination paper
Putting together a set of 100 or more questions, to test higher levels of cognition, like application and problem-solving, requires careful selection of 'good quality items' and adherence to the examination blueprint. Selecting items for the examination paper therefore is a scientific exercise, where the coordinators, based on the examination blueprint, select 70% of items with moderate difficulty and approximately 10-15% of the items from the easy and difficult indices. Once items are correctly entered into the QB, item retrieval is a simple matter of sorting by attributes and psychometric details leading to relatively easier and fool-proof selection of items for the examination paper. [5]
Step 8 - Post hoc analysis
This step is the last, but perhaps the most important. It provides a measurable outcome of all the time and effort put into the above-described exercise. A detailed post hoc analysis includes review of the item analysis where difficulty and discriminatory indices of each item are determined, making decisions regarding the items based on the item analysis and the subject/content expert feedback. [1] The indices obtained are used to determine whether an item was valid and reliable. If problems are identified, the item is suspended from the QB until it is reviewed and corrected. If the item is deemed to be appropriate, the teaching/learning strategy that addresses that particular objective must be re-evaluated. The indices for each item after each use are entered into the QB. This information adds another facet to developing balanced examinations. [3]
| Discussion | | |
Developing a good a bank of valid and reliable multiple-choice test items targeting higher cognitive abilities and conforming to item construction guidelines presents a big challenge to the item developer. In our experience, this is the culmination of many months of effort by a team. Coordinated efforts and a stepwise approach towards item development, as described above, coupled with focused faculty development exercises, is a cost-effective means of developing a huge 'Question Data Bank' of valid and reliable test items, which can serve as a National Resource for the Professional Regulatory Councils in the country for their Licensure Examinations.
| References | | |
| 1. | Case S.M., Swanson D.B. Constructing Written Test Questions for the Basic and Clinical Sciences. 3 rd edition, National Board of Medical Examiners; Philadelphia 2001. 
|
| 2. | Wass V, Van der Vleuten C, Shatzer J, Jones R. Assessment of clinical competence. Lancet 2001;357:945-9.
|
| 3. | Mc Coubrie P. Improving the fairness of multiple-choice questions: A literature review. Med Teach 2004;26:709-12.
|
| 4. | Mc Coubrie P, McKnight L. Single best answer MCQs: A new format for the FRCR part 2a exam. Clin Radiol 2008;63:506-10.
|
| 5. | Vale, C. D. Computerized item banking. In S. M. Downing & T. M. Haladyna (Eds.), Handbook of test development. Mahwah NJ: Erlbaum 2006.
|
| This article has been cited by | | 1 |
Healthcare providers have insufficient up-to-date knowledge of lower limb sports injuries, and their knowledge is similar to that of athletes |
|
| Asaf Weisman, Eshed Lin, Tomer Yona, Uri Gottlieb, Franco M. Impellizzeri, Youssef Masharawi | | Musculoskeletal Science and Practice. 2023; 65: 102750 | | [Pubmed] | [DOI] | | | 2 |
Improving assessment quality in professional higher education: Could external peer review of items be the answer? |
|
| Susanne Skjervold Smeby,Børge Lillebo,Vidar Gynnild,Eivind Samstad,Rune Standal,Heidi Knobel,Anne Vik,Tobias S. Slørdahl,Anne Spurkland | | Cogent Medicine. 2019; 6(1) | | [Pubmed] | [DOI] | | | 3 |
The journeys of three ASPIRE winning medical schools toward excellence in student assessment |
|
| Rukhsana W. Zuberi,Debra L. Klamen,Jennifer Hallam,Naveed Yousuf,Austin M. Beason,Evyn L. Neumeister,Rob Lane,Jason Ward | | Medical Teacher. 2018; : 1 | | [Pubmed] | [DOI] | | | 4 |
Near-peer question writing and teaching programme |
|
| Omar Musbahi,Fuzail Nawab,Nishat I Dewan,Alexander J Hoffer,James FCC Ung,Muhammed Talha Suleman | | The Clinical Teacher. 2017; | | [Pubmed] | [DOI] | | | 5 |
Climbing Bloomæs taxonomy pyramid: Lessons from a graduate histology course |
|
| Nikki B. Zaidi,Charles Hwang,Sara Scott,Stefanie Stallard,Joel Purkiss,Michael Hortsch | | Anatomical Sciences Education. 2017; | | [Pubmed] | [DOI] | | | 6 |
When passing fails: Designing multiple choice assessments to control for false positives |
|
| David N. Dubins,Gregory M.K. Poon,Lalitha Raman-Wilms | | Currents in Pharmacy Teaching and Learning. 2016; | | [Pubmed] | [DOI] | | | 7 |
The influence of question design on the response to self-assessment in www.elearnSCI.org: a submodule pilot study |
|
| N Liu,X-W Li,M-W Zhou,F Biering-Sørensen | | Spinal Cord. 2015; 53(8): 604 | | [Pubmed] | [DOI] | | | 8 |
Medical school 2.0: How we developed a student-generated question bank using small group learning |
|
| Adrian C.C. Gooi,Connor S. Sommerfeld | | Medical Teacher. 2015; 37(10): 892 | | [Pubmed] | [DOI] | |
|
 |
|
|
|
Search Pubmed for