|ORIGINAL RESEARCH PAPER
|Year : 2010 | Volume
| Issue : 1 | Page : 151
Information-seeking Practices of Senior Medical Students: The Impact of an Evidence-based Medicine Training Programme
NM Lai1, S Nalliah2
1 School of Medicine and Health Sciences, Monash University, Sunway Campus, Johor Bahru, Malaysia
2 School of Medicine and Health Sciences, Monash University, Sunway Campus, Johor Bahru, Malaysia, Malaysia
|Date of Submission||12-Dec-2007|
|Date of Acceptance||22-Jan-2010|
|Date of Web Publication||09-Apr-2010|
N M Lai
JKR 1235, Bukit Azah, 80100, Johor Bahru
Source of Support: None, Conflict of Interest: None
Context: The practice of Evidence-based Medicine (EBM) involves physicians regularly accessing and appraising clinical
information. Few prior studies have assessed the information-seeking behaviours of medical undergraduates. At the International Medical University (IMU), Malaysia, senior medical students receive clinically-integrated EBM training to facilitate their future practice of EBM.
Objectives: We assessed whether EBM training in the final six months of medical training changes our students' information seeking practices and their confidence in understanding and appraising clinical evidence.
Methods: Between September 2005 and February 2006, self-administered questionnaires were distributed to 65 senior medical students at the beginning and again at the end of their clerkship training during which there was a clinically-integrated EBM curriculum. The questionnaires covered the topics of their preferred sources of clinical information, online search frequencies, estimated time to retrieve an abstract, and their understanding and confidence in their critical appraisal skills.
Findings: Sixty-four (98%) students completed the initial survey and 63 (97%) completed the follow-up survey. The majority indicated that they preferred to first consult another individual (colleagues, lecturers, hospital staff) for their clinical queries (60.9% in the initial survey and 61.9% in the follow-up survey), with no change in their overall preference following the EBM curriculum six months later (p=0.144). There were significant increases in search activities following the curriculum, for example, students who searched PubMed or Medline for more than three times per week increased from 9.7% to 31.7% (p < 0.001). Students reported that they more often accessed single journals than databases. Despite significant improvements in students' reported understanding of journals and their confidence in critical appraisal (p < 0.001), there was no improvement in reported search speed, with 48.4% in the initial survey and 49.2% in the follow-up survey reporting to take 30 minutes or less to trace an abstract of interest (p=0.979).
Conclusions: Our EBM training, offered within a supportive curriculum, increased our students' confidence and activity related to EBM, but failed to change students' reported information-seeking behaviours. Other factors influencing medical students' information-seeking practice need to be explored.
Keywords: Evidence-based medicine, medical students, clerkship, information-seeking behaviour
|How to cite this article:|
Lai N M, Nalliah S. Information-seeking Practices of Senior Medical Students: The Impact of an Evidence-based Medicine Training Programme. Educ Health 2010;23:151
For practitioners to make evidence-based clinical decisions, they need regular access to reliable clinical information resources. Although formally defined less than two decades ago1, Evidence-based Medicine (EBM) has become a core aspect of the curricula of many medical schools. Instruction in EBM for physicians has been shown to increase their knowledge and application of EBM2-4, and data suggest that knowledge and skills in EBM acquired by practicing physicians can change their clinical practice5,6. Comparatively lacking are assessments of the impact of EBM training on medical undergraduates in terms of their knowledge, application and behaviours, although limited evidence shows that medical students appear to learn more specific EBM-related skills when they are taught than do practicing physicians2. Good information-seeking practice and confidence in handling clinical evidence, if established at the undergraduate level, might equip students with the critical skills needed to practice EBM in the future. It is therefore important to evaluate the information-seeking behaviours of medical undergraduates and understand how these are influenced by EBM training.
At International Medical University (IMU), Malaysia, a basic level of competence in EBM is expected of all medical graduates. To achieve this, students are exposed progressively to the concepts and practice of EBM, starting in the pre-clinical years with lectures and a Problem-based Learning (PBL) approach, progressing in the clinical years to expectations to complete a research project, a short EBM report and then a clinically-integrated curriculum that incorporates all major steps of EBM in the final six months of medical training, known as the senior clerkship. This curriculum covers the major tenets of EBM, from how to ask appropriate clinical questions to methods of critical appraisal, aiming to equip students with the key skills needed to access and appraise various types of clinical articles.
In this study, we assessed the effectiveness of our EBM curriculum in fostering positive changes in students’ EBM-related learning activities, which will have implications for similar EBM curricula in other schools. We evaluated our senior medical students on their information-seeking practice and confidence in handling clinical evidence before and after the clinically-integrated EBM training within their final six months of training. We hypothesise that following this training programme, students would more often seek answers to their clinical questions from higher-level evidence found in journal articles than other sources. We also expect that they would report more confidence in reading and critically appraising journal articles.
Educational intervention: EBM curriculum in senior clerkship, IMU
In the final six months of undergraduate medical training at IMU, called the senior clerkship, students acquire clinical experience through direct participation in patient management under the supervision of faculty members who are actively involved in providing clinical services. Alongside ward-based learning through direct observation of clinicians and role-modelling, students develop in-depth records of selected patients, with reflections on various issues in their care, including a compulsory segment on EBM. These in-depth learning records, known as the “portfolios”, provide students with a means of focused learning to compliment their general exposure to patient care in their ward work. Portfolios also serve as documents on which students are assessed during their final oral evaluation at the end of clerkship.
Incorporated within the clerkship training is a clinically-integrated EBM curriculum, which consists of two principal parts: i), introductory, plenary lectures and i). small-group, intensive training. The programme is detailed as follows:
In the first week of clerkship, students attend two plenary lectures covering an overview of EBM, methods of searching and critical appraisal of articles on therapy and prognosis. Each subgroup of 13 to 15 students receives the small-group training when they rotate on the Paediatric posting. This includes bedside clinical teaching sessions with question-formulating exercises during which they learn how to convert relevant issues in patient care into focused, answerable clinical questions. The questions range from the utility of diagnostic tests to the effectiveness of treatment and prognosis of disease. Students are coached by their supervisor and peers to frame their question as specifically as possible, following the PICO framework (P: patient; I: intervention, prognostic indicator or index test; C: comparison; O: outcome)7. There are a total of six two-hour clinical sessions in the four-week Paediatric posting. Having formulated these clinical questions, students access common clinical resources in search of answers. They present the clinical articles they acquire at the journal club sessions. Each journal club presentation includes a clinical question, search strategies, and a critical appraisal of the article retrieved, followed by a discussion of how the findings apply to patients. The journal club presentations are adapted by the students as EBM reports in their portfolios. Each student is required to develop at least five EBM reports out of a total of 14 portfolios across all six disciplines in their senior clerkship.
In the introductory plenary lectures, students are directed on the use of EBM learning resources, including the critical appraisal tools with comprehensive explanatory notes, which are made available in the virtual storage (“I” Drive) of the clinical school freely accessible by students and staff. As such, we expect all students to have had similar level of competence in EBM at the end of the clerkship, regardless of when they received the small-group training.
This EBM curriculum was developed by one of the authors (NML) in May 2003 using the principles delineated by Sackett et al.8 This author, then the resident coordinator and assigned teacher of EBM in senior clerkship, oversaw all teaching-learning activities within the programme. He conducted all formal EBM training sessions, including the introductory lectures and small-group training. He also instructed resident supervisors from other disciplines in senior clerkship on EBM training approaches.
Self-administered questionnaires were distributed twice, at the beginning (September 2005) and again at the end of senior clerkship (January 2006), to 65 final-year medical students at IMU, Clinical School Batu Pahat, Malaysia.
The questionnaire was comprised of five items that assess the following five EBM-relevant topics: students’ preferred sources of information to answer clinical queries; frequency with which students access selected clinical information resources; estimated time to locate an abstract of interest; self-perceived understanding of journal articles; and confidence in critical appraisal. The questionnaire is included in the Appendix.
An initial draft of the questionnaire, comprised of items one, two, three and five, was drafted by the first author in June 2005. After piloting with 12 students, item one was re-worded. The revised questionnaire was then administered to a larger group of students (n=50) in July 2005. Internal reliability (Cronbach’s alpha) obtained from combining items two, three and five, i.e. items with ordinal responses, was 0.84 (95% CI for intraclass correlation coefficient:0.79 to 0.88). Item one was excluded because the responses were nominal in nature and hence unsuitable for reliability analysis. This questionnaire was assessed further for its face validity by the Professional Education Advisory Committee (PEAC) of the University in August 2005. The PEAC was comprised of five experts in medical education from our partnering universities overseas and one representative from Malaysia. Following their assessment, another item (“Understanding of journal articles”, item four) was added to the questionnaire.
Prior to fielding the survey, students were given a written notice about the study. On the day the questionnaires were administered, students were also briefed on its purpose by the first author who then left the premises. In the written notice as well as in the briefing, students were told that participation in this study was voluntary and that whether they participated or not and the information they reported would have no bearing on their grades or standing in the university. An administrative staff member who was not otherwise involved in the study then distributed the questionnaires. Completion of the questionnaires was taken as consent to participate in the study. None of the authors were present during the conduct of the study. This study was approved by the Centre for Medical Education (CtME), IMU, the official body approving and overseeing all medical education projects in the university. The study was also approved by our Clinical School Deanery.
We performed the following statistical analyses: Internal reliability for items two through five was expressed as Cronbach’s alpha and 95% CI of intraclass correlation coefficients. Chi-square tests were used to analyse students’ preferred clinical information resources, and the Mann-Whitney-U test was used to analyse the Likert-scaled responses (Statistical Package for Social Sciences (SPSS) version 11 (Chicago, IL, USA)).
Sixty-four out of 65 students (98%) returned the initial questionnaire and 63 (97%) returned the follow-up questionnaire. Items two through five showed good internal reliability, identical to that obtained during initial validation, i.e. alpha of 0.84 (95% CI intraclass correlation coefficient: 0.79 to 0.88). Some questionnaires were incomplete yielding missing data, as indicated by the differing total number of responses for the various items shown in the tables.
There was no difference in the distribution of students’ preferred sources of information for clinical questions between the two surveys (p=0.144) (Figure 1). The majority of the students (60.9% and 61.9% in the initial and follow-up survey respectively) preferred first seeking information from another person, followed by manuals or pocket references. Electronic sources (journal articles and health-related internet documents) replaced textbooks or pocket references as the preferred sources of information among a relatively small number of students by the end of their clerkship. Very few students (3.1% in the initial survey and 1.6% in the follow-up survey) indicated that they preferred to keep the queries to themselves, i.e. without referring to sources to answer their queries. As shown in Table 1, students’ use of common online clinical resources increased significantly during the clerkship. Specifically, students who reported that they searched PubMed or Medline more than three times per week increased from 9.7% to 31.7%, and those who searched the Cochrane Library at least once a week increased from 8.1% to 17.5% (p < 0.001 for all items). Collectively, students accessed single journals subscribed by our university, like New England Journal of Medicine and British Medical Journal, more often than databases like PubMed or Ovid, as detailed in Table 1.
Figure 1 – Students’ preferred sources of information for clinical queries
Table 1 – Reported frequency in accessing common online databases/journals
Students’ estimated time to locate an abstract of interest did not lessen over the course of the clerkship. Almost half of the students (48.4% and 49.2% at the initial and follow-up survey respectively) reported to take 30 minutes or less to trace an abstract (p = 0.979) (Figure 2). Nonetheless, students who reported that they understood articles without help from others increased from 17.5% to 39.7% (p < 0.01) (Figure 3), and those who felt confident in their critical appraisal skills at least for certain types of study improved from 50.8% to 96.8% (p < 0.01) (Figure 4).
Figure 2 – Reported time required to trace an abstract of interest
Figure 3 – Students’ reported understanding journal articles
Figure 4 – Students’ reported confidence on critical appraisal
Although EBM training that relates to actual clinical practice has been shown to produce sustained benefits technically and cognitively4,9,10, most EBM teaching to date takes place within the classroom, while few schools have integrated EBM training into bedside clinical teaching2,11,12. This study provides evidence of the responsiveness of medical students’ information-seeking behaviours to a period of clinically-integrated training that incorporates EBM as a major element in students’ training and assessment. It is reassuring to note that, by our students’ self reports, almost all would seek answers to their clinical queries and not let their questions go unanswered. Following the EBM training, some students evidently switched from printed references (concise references and textbooks) to electronic sources as their preferred sources for answering their queries. There was also a change in the rank order of popularity for “journal articles” from fourth to third place at the end of clerkship, displacing textbooks as the preferred source of information. These findings suggest that exposure to EBM increased some students’ awareness of the importance of up-to-date clinical information and limitations of printed references.
However, contrary to our expectation, students’ habit of consulting another individual first for clinical queries remained unchanged after exposure to EBM. This tendency could well be attributed to students’ general approach to learning, as novices navigating through a huge field of knowledge. As a result, they might focus on acquiring core clinical facts instead of specific, up-to-date clinical information, and choose the easiest path for this purpose. Peers, hospital staff and lecturers, as convenient sources of information, would understandably remain popular. Moreover, the apprenticeship model of student training, as adopted in our senior clerkship where role-modelling of senior colleagues is advocated as the chief means of learning, has perhaps discouraged independent information-seeking, especially if the role-models also refer to another person first for their clinical queries. It was also possible that the students’ earlier medical training, including the EBM curriculum, had not put sufficient emphasis on critical thinking, independent question-formulation and information-seeking, and this was difficult to change in the final six months even following an intensive and clinically-integrated EBM training. On the other hand, the tendency to consult someone else might also reflect a general lack of confidence by junior members of the clinical hierarchy in making independent clinical decisions, as a similar information-seeking pattern has also been observed in hospital residents13. With the current training approach and students’ learning needs in our clerkship, even though students might be receptive to EBM training2, it is perhaps unrealistic to expect this training to produce a quick change in students’ information-seeking behaviours until later in their careers when they have the capacity to exercise critical thinking and the authority to make independent and specific clinical decisions14. As senior clinicians are most often the key influences in students’ clinical learning and practice culture, increasing the awareness and application of EBM among senior clinicians may also facilitate the development of appropriate evidence-seeking practice by students and junior staff6,11,15. It would be of great interest to assess how senior medical staff deal with their own clinical queries.
The majority of our students reported that they accessed journals from once a month to three times a week, a finding comparable to a study by McKibbon et al.16 where physicians searched sources 2.5 to 3.5 monthly with and without EBM instruction, respectively. We included the New England Journal of Medicine and the British Medical Journal within our queried search items because our school subscribes to these journals and their full-text articles are available to students free of cost. Our students preferred these single journals to medical databases, suggesting a tendency to opt for sources that could be found quickly to save time. This practice raises concerns, as it might limit search yields and lead students to miss important and relevant articles. The finding behooves a reinforcement of instructions on search techniques, in particular database-searching, and highlights the need to implement a formal documentation in student portfolios of their search paths leading to the identification of the articles found.
We fail to find any improvement in the time students estimate that it takes them to retrieve a relevant abstract despite significant improvement in their confidence in reading and appraising journal articles. Several explanations are possible on this observation. First, students’ tendency to search single journals, as highlighted previously, might have led to more initial failed search attempts, requiring additional searches before coming onto a suitable article. Next, improved critical appraisal skills might have led to more stringent criteria in article selection, requiring more search time filtering out unwanted articles, offsetting any possible improvement in search efficiency gained from their EBM training. The influence of students’ earlier EBM training might also have conferred them some proficiency in searching, and as such, the EBM curriculum in the final six months of training might not have given them extra skills that translate into faster search process. Alternatively, students might have already reached a limit in their search efficiency at this stage of training, and additional clinical and search experience might be needed to produce further improvement. This phenomenon was observed in another cohort of IMU students17 as well as in other studies on undergraduates18,19, and shorter search times have been observed in resident doctors20.
We acknowledge the following limitations in our study. First, it is unclear whether the improvements reported in search frequencies, understanding of journal and critical appraisal were due to our EBM training or to the fact that EBM was part of students’ compulsory assignment, which was also included in their final summative assessment. Second, we are unable to decipher which factors most likely accounted for the lack of changes in some aspects of students’ EBM practice, as detailed above, because we only assessed a very short period in the overall training of our students. Next, we examined self-reports of practice and confidence, which might more reflect our students’ attitude to EBM than their true competence. Although students’ confidence and attitude to EBM, as assessed here, may play an important part in their motivation to practice EBM, their true competence in carrying out the steps of EBM is best measured by objective and validated tools21-23.
While our senior students reported increases in their search activities and confidence in EBM after a period of EBM training, we did not find great changes in their information-seeking behaviours. It might not be realistic to expect major changes in how they seek information to answer their clinical questions until later in their careers. We also found that our students need more guidance to become aware of the limitations of single journals as information sources, and more encouragement and support to access clinical evidence via more comprehensive resources such as databases. However, due to a study sample limited to our own students and the various limitations of our study, we cannot say how generalisable our findings are to students of other institutions with similar EBM curricula. Future studies should explore the other factors that might influence students’ information-seeking practices, like providing up-to-date clinical information resources at the point-of-care.
List of abbreviations
EBM – Evidence-based Medicine
SPSS – Statistical Package for Social Science
IMU – International Medical University
1. Guyatt G. Evidence-based medicine. ACP Journal Club (Annals of Internal Medicine). 1991; 114(suppl 2):A16.
2. Dobbie A E, Scheider FD, Anderson AD, Littlefield J. What evidence supports teaching evidence-based medicine? Academic Medicine. 2000; 75:1184-1185.
3. Del Mar C, Glasziou P, Mayer D. Teaching evidence based Medicine. British Medical Journal. 2004; 329:989-990.
4. Dinkevich, E, Markinson A, Ahsan S, Lawrence B. Effect of a brief intervention on evidence based medicine skills of pediatric residents. BMC Medical Education. 2006; 6:1. Retrieved 1 May 2006, from URL: http://www.biomedcentral.com/1472-6920/6/1
5. Thom DH, Haugen J, Sommers PS, Lovett P. Description and evaluation of an EBM curriculum using a block rotation. BMC Medical Education. 2004; 4:19. Retrieved 1 May 2006, from URL: http://www.biomedcentral.com/1472-6920/4/19
6. Straus SE, Ball C, Balcome N, Sheldon J, McAlister FA. Teaching evidence-based medicine skills can change practice in a community hospital. Journal of General and Internal Medicine. 2005; 20:340-343.
7. Richardson WS, Wilson MC, Nishikawa J, Hayward RSA. The well-built clinical question: a key to evidence-based decisions. ACP Journal Club; 1995; 123:A12-13.
8. Sackett DL, Straus SE, Richardson WE, Rosenberg WM, Haynes RB. Evidence-based medicine. How to practice and teach EBM (2nd ed.). Edinburgh: Churchill Livingston; 2000.
9. Smith CA, Ganschow PS, Reilly BM, Evans AT, McNutt RA., Osei A., et al. Teaching residents evidence-based medicine skills: a controlled trial of effectiveness and assessment of durability. Journal of General and Internal Medicine. 2000; 15:710-715.
10. Bradley DR, Rana GK, Martin PW, Schumacher RE. Real-time, evidence-based medicine instruction: a randomised controlled trial in a neonatal intensive care unit. Journal of Medical Librarian Association. 2002; 90:194-201.
11. Korenstein D, Dunn A, McGinn T. Mixing it up: integrating evidence-based medicine and patient care. Academic Medicine. 2002; 77:741-742.
12. Slawson DC, Shaughnessy AF. Teaching evidence-based medicine: should we be teaching information management instead? Academic Medicine. 2005; 80:685-689.
13. Ramos K, Linscheid R, Schafer S. Real-time information-seeking behaviour of residency physicians. Family Medicine. 2003; 35(4):257-260. Retrieved 4 May 2006, from http://www.stfm.org/fmhub/fm2003/apr03/sa2.pdf
14. Sackett DL, Strauss SE, for firm A of the Nuffield Department of Medicine. Finding and applying evidence during clinical rounds: The 'evidence' cart. Journal of American Medical Association. 1998; 280:1336-1338.
15. Guyatt G, Cook D, Haynes RB. Evidence based medicine has come a long way. British Medical Journal. 2004; 329:990-991.
16. McKibbon KA, Hayne RB, Johnston ME, Walker CJ. A study to enhance clinical end-user MEDLINE search skills: design and baseline findings. Proceedings of the Fifteenth Annual Symposium on Computer Applications in Medical Care. November 17-20, New York, USA; 1991.
17. Lai NM, Ramesh CJ. The product of outcome-based undergraduate medical education: competencies and readiness for internship. Singapore Medical Journal. 2006; 47:1053-1062.
18. Shelstad KR, Clevenger FW. On-line search strategies of third year medical students: perception vs fact. Journal of Surgical Research. 1994; 56:338-344.
19. Badgett RG, Paukert JL, Levy LS. Teaching clinical informatics to third-year medical students: negative results from two controlled trials. BMC Medical Education. 2001; 1:3. Retrieved 1 May 2006, from URL: http://www.biomedcentral.com/1472-6920/1/3
20. Schwartz K, Northrup J, Israel N, Crowell K, Lauder N, Neale AV. Use of On-line Evidence based Resources at the Point of Care. Family Medicine. 2003; 35(4):251-256. Retrieved 4 May 2006, from URL: http://www.stfm.org/fmhub/fm2003/apr03/sa1.pdf
21. Fritsche L, Greenhalgh T, Falck-Ytter Y, Neumayer H-H., Kunz R. Do short courses in evidence based medicine improve knowledge and skills? Validation of Berlin questionnaire and before and after study of courses in evidence based medicine. British Medical Journal. 2002; 325:1338-1341.
22. Ramos K, Schafer S, Tracz SM. Validation of the Fresno test of competence in evidence based medicine. British Medical Journal. 2003; 326:318-321.
23. Lai NM, Teng CL. (2009). Competence in evidence-based medicine of senior medical students following a clinically integrated training programme. Hong Kong Medical Journal. 2009; 15:332-338.