|
 
 |
| ORIGINAL RESEARCH ARTICLE |
|
| Year : 2015 | Volume
: 28
| Issue : 1 | Page : 29-34 |
|
Comparison of resident performance in interpreting mammography results using a probabilistic or a natural frequency presentation: A multi-institutional randomized experimental study
Pavlos Msaouel1, Theocharis Kappos2, Athanasios Tasoulis3, Alexandros P Apostolopoulos4, Ioannis Lekkas5, Elli-Sophia Tripodaki6, Nikolaos C Keramaris1
1 Greek Junior Doctors and Health Scientists Society, Athens, Greece
2 2nd Department of Surgery, "Metaxa" Cancer Memorial Hospital, Pireaus, Greece
3 Department of Clinical Therapeutics, "Alexandra" Hospital, Voula, Greece
4 Department of Orthopaedics, "Asklipiio" Hospital, Voula, Greece
5 Otorhinolaryngology Clinic, "251 G.N.A." Hospital, Athens, Greece
6 1st Department of Internal Medicine, "Evangelismos" Hospital, Athens, Greece
| Date of Web Publication | 31-Jul-2015 |
Correspondence Address:
Pavlos Msaouel
Chair, Greek Junior Doctors and Health Scientists Society, Athens
Greece
 Source of Support: None, Conflict of Interest: None  | Check |
DOI: 10.4103/1357-6283.161846
Background: Residents are being increasingly challenged on how best to integrate diagnostic information in making decisions about patient care. The aim of this study is to assess the ability of residents to accurately integrate statistical data from a screening mammography test in order to estimate breast cancer probability and to investigate whether a simple alteration of the representation mode of probabilities into natural frequencies facilitates these computations. Methods: A multi-institutional randomized controlled study of residents was performed in eight major hospitals in the city of Athens. Residents were asked to estimate the positive predictive value of the screening mammography test given its sensitivity and 1-specificity as well as the prevalence of breast cancer in the relevant population. One version of the scenario was presented in the single-event probability format that is commonly used in the medical literature, while the other used the natural frequency representation. The two questionnaire versions were randomly assigned to the participants. Results: Out of 200 residents, 153 completed and returned the questionnaire (response rate 76.5%). Although more than one-third of the residents reported excellent or close to excellent familiarity with sensitivity and positive predictive value, the majority of responses (79.1%) were incorrect. However, a significantly higher proportion of residents in the natural frequency group (n = 88) selected the correct response compared with residents (n = 65) in the single-event probability group (28.4% vs 10.8%; 95% confidence intervals of the difference between the two proportions = 5.6-29.7%; P < 0.01). Discussion: Residents more often correctly understand test performance accuracy when test characteristics are presented to them as natural frequency representations than the more common approach of presenting single event probabilities. Educators and journal editors should be aware of this facilitative effect. Keywords: Base rate neglect, Bayesian inference, natural frequencies, probabilities, statistical reasoning, positive predictive values
How to cite this article:
Msaouel P, Kappos T, Tasoulis A, Apostolopoulos AP, Lekkas I, Tripodaki ES, Keramaris NC. Comparison of resident performance in interpreting mammography results using a probabilistic or a natural frequency presentation: A multi-institutional randomized experimental study. Educ Health 2015;28:29-34 |
How to cite this URL:
Msaouel P, Kappos T, Tasoulis A, Apostolopoulos AP, Lekkas I, Tripodaki ES, Keramaris NC. Comparison of resident performance in interpreting mammography results using a probabilistic or a natural frequency presentation: A multi-institutional randomized experimental study. Educ Health [serial online] 2015 [cited 2023 Jun 7];28:29-34. Available from: https://educationforhealth.net//text.asp?2015/28/1/29/161846 |
| Background | |  |
Biostatistical information can be represented as either single-event probabilities (e.g., if a woman has breast cancer then the probability that she will get a positive mammography is 80%) or as natural frequencies (e.g., 8 out of 10 women with breast cancer will get a positive mammography). The accuracy of diagnostic tests is commonly reported in terms of sensitivity, specificity and positive predictive value (PPV) using the conditional probability format which does not specify the reference class used for each calculation: The reference class used to determine a test's sensitivity is people with illness, whereas the reference class used for specificity is people without illness. This switch of reference class makes conditional and single-event probability representations of data confusing as they require physicians to perform a larger number of demanding arithmetic operations (multiplication, addition or division) particularly in clinical scenarios requiring simple Bayesian calculations. [1],[2],[3],[4] Thus, such information may be systematically miscalculated by physicians and subsequently miscommunicated to patients resulting in erroneous choices of medical therapy, inappropriate utilization of medical resources and disproportionate health care expenditures. [5],[6]
The two versions of the diagnostic scenario presented in [Table 1] provide data on the prevalence of a disease (breast cancer) in a population (women aged >40 years). This background information, called base rate, must be taken into account when determining the PPV of a diagnostic test. However, clinicians often neglect base rate, which makes it a common cognitive bias that leads to miscalculations when making inferences about probabilities in Bayesian reasoning problems such as the mammography scenario described in [Table 1]. Bayes' mathematical formula is essential for such computations when data are presented as probabilities. However, this formula can be difficult to learn and memorize. On the other hand, conversion of probabilities to the natural frequency format can make Bayesian reasoning easier. Although both formats are mathematically equivalent, the natural frequency presentation is computationally and conceptually simpler as it automatically incorporates base rate information and utilizes an information format that may be more aligned with what human minds were evolved to understand and process. [3],[4],[7],[8],[9],[10],[11],[12]
A number of reports have indicated that medical trainees at different levels may have distinct learning needs and preferences. [13],[14],[15],[16],[17],[18] It is therefore important to determine the effectiveness of specific learning techniques during particular stages of medical training. The present study focused on residents and investigated potential differences in their statistical reasoning performance when data from a mammography screening problem are presented either as probabilities or as natural frequencies [Table 1]. The mammography screening problem tests an elementary but important Bayesian calculation that lies at the heart of many everyday clinical interactions. [3],[4],[19],[20]
| Methods | | |
Participants and data collection
[Figure 1] shows a summary of the study profile. The study complied with the Standards of the Declaration of Helsinki as well as Greek requirements for survey studies. The study was conducted from January to March 2010 and the participants were chosen from eight major Greek hospitals in the city of Athens [Table 2] that train residents who had graduated from all seven Greek medical schools. The Greek system does not allow official residency training in private hospitals and therefore all institutions surveyed in this study were public. Every resident was given a number according to official enrolment lists that had been kindly provided by each hospital. Out of 1272 eligible residents in all hospitals, 200 (25 from each hospital) were randomly selected by a computerized method into the sample, a number that represents approximately 2% of the total number of residents training in Greece. [21] The probability version of the mammography problem was included in questionnaires seen by half of the sample, while the other half received the natural frequencies version. The two versions were randomly assigned to participants by blindly placing each questionnaire into A4 envelopes, which were then hand shuffled by the researcher. The residents were asked to return the completed questionnaires to a sealed box provided in each hospital. | Figure 1: Profile of the study on the effect of data format on Bayesian reasoning performance
Click here to view |
Survey measures
The first seven questions queried residents' socio-demographics, specialty choices, previous education outside Greece, current training level, residency year and past training in biostatistics of residents. We combined residencies according to their conceptual and occupational relations to form three different medical "fields": (a) Internal medicine (n = 71; dermatology, pediatrics, neurology and general practice were also included in this group) (b) surgical specialties (n = 67) (c) diagnostic and laboratory specialties (n = 15). The next set of questions assessed residents' familiarity with the statistical concepts of sensitivity and PPV by asking them to rate their familiarity with each concept on a 7-point Likert scale ranging from 'none' (score of 1) to 'excellent' (score of 7). We next presented the mammography scenario, derived from the one used by Gigerenzer and Hoffrage, [3] either in the probability or the natural frequency form [Table 1] and asked respondents to calculate and select, in a multiple choice format, the probability (given the sensitivity and 1-specificity of the test and the prevalence of the disease) that a positive mammogram result meant that the woman has breast cancer, which corresponds to the PPV of the screening test. The correct estimate is approximately 7.8%, thus the only valid response in the multiple choices provided in the questionnaire [Table 1] would be 'Approximately 8%'. We chose the multiple-choice format because it elicits higher response rates and is the preferred response format among residents as it requires less response time compared with free-response items. [22],[23] The distractor items were chosen based on the response patterns observed in previous studies that tested the mammography screening scenario on physicians. [19],[20] One easy distractor ('100%') and two difficult distractors ('approximately 70%' and 'higher than 80%') were used in addition to the correct response.
Sample size
Sample size was calculated according to Cohen's guidelines by calculating Cohen's effect size w measure for Chi-square tests. [24] Previous studies have reported relatively large effect sizes w ranging from 0.45 to 0.92. [3],[11],[12] In order to detect an even smaller effect size w = 0.3 (which is a medium effect size by Cohen's criteria [24] ), at α=0.05 with 95% statistical power we would require a minimum sample size of 145 respondents. Prior response rates in surveys conducted by our group ranged between 77.8% and 90.9%. [25],[26] We therefore administered the questionnaire to 200 residents expecting that even a lower than anticipated response rate of 75% would provide 150 completed responses.
Statistical analysis
A priori sample size calculations were performed using the G*Power 3.0.5 software. [27] Data analysis was performed using R (Foundation for Statistical Computing, Vienna, Austria). [28] Median differences between two groups were analyzed using the Mann-Whitney U test for two non-paired data. Categorical variables were compared using Pearson's Chi-square or Fisher's exact tests where appropriate. Differences between two proportions and median differences between two groups are represented as 95% confidence intervals (CI). Bivariate analyses were performed to identify individual resident characteristics that may be associated with performance on the mammography problem. Variables with P < 0.05 on bivariate comparisons were further included in a multivariate analysis performed with a stepwise binary logistic regression test using the dichotomous coding of responses (correct/incorrect) as the dependent variable. A two-sided P value of < 0.05 was considered statistically significant.
| Results | | |
A total of 153 completed questionnaires were returned (response rate 76.5%), with 112 males, 41 females, and a median age 32 years. Sixty-five of the completed questionnaires included the probability version of the Bayesian inference mammography test, while 88 completed questionnaires contained the natural frequency version. A comparison of respondents' characteristics between the two randomized groups is provided in [Table 2]. No significant differences in resident socio-demographic, educational and residency profile characteristics were found between the two groups. There was a significant difference in the response rates between the two groups as more residents completed and returned the natural frequency questionnaires (95% CI = 11.7-34.3%, P < 0.01). Perceived familiarity with the statistical concepts of sensitivity and PPV was rated as "excellent" or "close to excellent" by 88/153 (57.5%) and 60/153 residents (39.2%), respectively. The vast majority of residents (119 out of 153) reported that they received biostatistics training in lecture courses during medical school. Only six respondents (3.9%) received formal biostatistics training during residency.
The distribution of the responses in the probability and natural frequency groups is shown in [Figure 2]. The two representation groups differed significantly with regard to the proportions of the five answers (P < 0.01). The majority of residents (n = 121; 79.1%) answered the mammography problem incorrectly. However, a significantly higher proportion of residents in the natural frequency group selected the correct response compared with residents in the probability group (28.4% vs 10.8%; 95% CI = 5.6-29.7%, P < 0.01). The most popular answer was 'higher than 80%' while the least common incorrect response was '100%'. Furthermore, the probability group tended to select 'I don't know' more frequently [Figure 2]. | Figure 2: Distribution of responses in the probability and natural frequency versions of the mammography problem. The asterisks denote the correct answer. Residents in the natural frequency format group were significantly more likely to choose the correct response (P<0.01)
Click here to view |
Bivariate analyses indicated that only residents' gender was significantly associated with performance on the mammography problem. More specifically, men were more likely than women to correctly answer the mammography problem (25.9% vs 7.3%; 95% CI = 7.2-30%, P = 0.012). Multivariate analysis was further performed in order to estimate the independent main effects of data presentation format and residents' gender on Bayesian reasoning performance [Table 3]. As shown in [Table 3], both the data presentation format and gender were independently associated with residents' performance in the mammography scenario (OR = 0.20, 95% CI = 0.06-0.72, P = 0.013 for females compared with males and OR = 3.62, 95% CI = 1.43-9.16, P = 0.007 for the natural frequency format compared with the probabilistic format). Thus, women were less likely to correctly identify the correct answer than men regardless of the statistical presentation format used. | Table 3: Multivariable logistic regression model of the relationship between resident gender and data presentation format with correct responses in the mammography scenario
Click here to view |
| Discussion | | |
The present study demonstrated that residents are significantly more accurate at integrating statistical data for solving a Bayesian diagnostic test scenario when the information is presented as frequencies rather than percentages. This effect was elicited despite the fact that the answer selections were given as percentages in both cases. Although more than one-third of the residents reported excellent or close to excellent knowledge of sensitivity and PPV, and despite having daily experience reporting diagnostic test characteristics to patients, only a minority of residents could correctly apply this information to calculate the PPV of a very common screening test. Previous reports have shown that experienced physicians, including faculty members of Harvard Medical School, had equally great difficulties with similar medical scenarios posed as single-event probabilities. [19],[20],[29],[30]
Notably, residents were more likely to complete and return the natural frequency version of the questionnaire, suggesting that this format may also reduce the cognitive effort and psychological intimidation of the Bayesian problem. This may be further supported by the higher tendency of residents in the probability group to declare that they did not know the answer to the mammography scenario.
The natural frequency representation has previously been shown to improve the statistical reasoning of physicians and medical students. [11],[12],[31] However, a number of investigations have demonstrated significant differences in learning needs, styles and preferences between undergraduate medical students and residents as well as between residents and specialists. [13],[14],[15],[16],[17] The present study extended to residents previous observations of improved statistical inference using natural frequency scenarios. [19],[20],[30],[31] The majority of residents miscalculated the PPV of the mammography test. Thus, our results raise the question to what extent misinterpretations of the diagnostic value of screening tests contribute to erroneous diagnosis and misuse of laboratory and imaging tests as well as invasive procedures. Identifying educational tools to improve perception and understanding of statistical data is fundamental for both physicians and patients.
It should be noted that although the natural frequency format significantly improved the Bayesian reasoning performance of residents, less than one-third of respondents in the natural frequency group chose the correct answer. This performance is lower compared to what was observed in the seminal study by Gigerenzer et al. [3] that used the natural frequency version of the mammography screening problem in university students or in a study that used natural frequency presentations of different Bayesian reasoning tasks on obstetricians. [11] Furthermore, midwives have shown no benefit from natural frequency representations of diagnostic test data [11] indicating that the effects of such representation modifications may not be consistently strong across different healthcare groups.
Further analysis of the effects of individual resident variables in Bayesian reasoning indicated a main effect of residents' gender as women were less likely to select the correct answer compared with men. A previous survey of internal medicine residents reported that male residents were more likely to correctly answer biostatistics knowledge quizzes. [32] However, to our knowledge no previous study has specifically investigated the effect of gender on the percentage of correct inferences in the mammography or other similar Bayesian scenarios. [3],[8],[9],[10],[11] The results of the present study, although limited by the small sample of women residents surveyed, may indicate the effect of gender-specific social influences, cross-cultural variations, participant recruitment methods, intrinsic cognitive differences or other unknown factors and biases [33],[34],[35],[36] on the relationship between residents' gender and Bayesian reasoning. In addition, the fact that the mammography screening problem tested a strongly gender-specific disease (breast cancer) may also have influenced our results. Due to the small numbers of female respondents and correct answers, there was insufficient power to evaluate the interaction between gender and representation format.
Limitations
While the response rate in the present study was considerably higher than what is typical for physician studies, [37] the possibility of response biases cannot be excluded. In order to fully protect residents' anonymity, we were unable to collect any further data on non-respondents. Residents who felt less comfortable with statistical data may have been less willing to complete and return the questionnaire. It should also be noted that the two tested representations of the mammography scenario have some word ordering differences. Although it is unlikely, the possibility that such subtle variances may have also affected the results cannot be excluded. Furthermore, the fact that all three distractors markedly overestimate the PPV of the mammography test may provide clues to the correct answer. Consequently, test taking skills of residents may affect the results. However, such guessing biases due to individual differences in test taking skills are compensated by the randomization process used in the present study.
| Conclusions | | |
The present data demonstrate that most residents are susceptible to misinterpreting test results due to base rate neglect bias. However, the natural frequency format can make residents more accurate in solving a simple Bayesian problem, which is frequently encountered in daily clinical practice. Medical trainees can be taught how to effectively translate single-event probabilities into natural frequencies using simple classroom tutoring. [20],[31] Nevertheless, a substantial number of residents in the natural frequency group still failed to reach the correct answer. Therefore, this simple technique should be supplemented with further educational interventions as well as mechanical aids aimed at minimizing the effect of base rate neglect and other statistical biases.
| References | | |
| 1. | Peters E, Vastfjall D, Slovic P, Mertz CK, Mazzocco K, Dickert S. Numeracy and decision making. Psychol Sci 2006;17:407-13.  |
| 2. | Gigerenzer G, Hoffrage U, Kleinbolting H. Probabilistic mental models: A Brunswikian theory of confidence. Psychol Rev 1991;98:506-28. |
| 3. | Gigerenzer G, Hoffrage U. How to Improve Bayesian Reasoning Without Instruction: Frequency Formats. Psychol Rev 1995;102:684-704. |
| 4. | Reyna V, Brainerd C. Numeracy, ratio bias, and denominator neglect in judgments of risk and probability. Learn Individ Differ 2008;18:89-107. |
| 5. | Cahan A, Gilon D, Manor O, Paltiel O. Probabilistic reasoning and clinical decision-making: Do doctors overestimate diagnostic probabilities? QJM 2003;96:763-9. |
| 6. | Ghosh AK, Ghosh K, Erwin PJ. Do medical students and physicians understand probability? QJM 2004;97:53-5. |
| 7. | Misuraca R, Carmeci FA, Pravettoni G, Cardaci M. Facilitating effect of natural frequencies: Size does not matter. Percept Mot Skills 2009;108:422-30. |
| 8. | Galesic M, Gigerenzer G, Straubinger N. Natural frequencies help older adults and people with low numeracy to evaluate medical screening tests. Med Decis Mak 2009;29:368-71. |
| 9. | Zhu L, Gigerenzer G. Children can solve Bayesian problems: The role of representation in mental computation. Cognition 2006;98:287-308. |
| 10. | Chapman GB, Liu J. Numeracy, frequency, and Bayesian reasoning. Judgm Decis Mak 2009;4:34-40. |
| 11. | Bramwell R, West H, Salmon P. Health professionals′ and service users′ interpretation of screening test results: Experimental study. BMJ 2006;333:284. |
| 12. | Hoffrage U, Lindsey S, Hertwig R, Gigerenzer G. Medicine. Communicating statistical information. Science 2000;290:2261-2. |
| 13. | van de Wiel MW, Boshuizen HP, Schmidt HG, Schaper NC. The Explanation of Clinical Concepts by Expert Physicians, Clerks, and Advanced Students. Teach Learn Med 1999;11:153-63. |
| 14. | Engels PT, de Gara C. Learning styles of medical students, general surgery residents, and general surgeons: Implications for surgical education. BMC Med Educ 2010;10:51. |
| 15. | Schultz KW, Kirby J, Delva D, Godwin M, Verma S, Birtwhistle R, et al. Medical Students′ and Residents′ preferred site characteristics and preceptor behaviours for learning in the ambulatory setting: A cross-sectional survey. BMC Med Educ 2004;4:12. |
| 16. | Kosower E, Berman N. Comparison of pediatric resident and faculty learning styles: Implications for medical education. Am J Med Sci 1996;312:214-8. |
| 17. | Stritter FT, Baker RM. Resident preferences for the clinical teaching of ambulatory care. J Med Educ 1982;57:33-41. |
| 18. | Richard RD, Deegan BF, Klena JC. The learning styles of orthopedic residents, faculty, and applicants at an academic program. J Surg Educ 2014;71:110-8. |
| 19. | Eddy DM. Probabilistic Reasoning in Clinical Medicine: Problems and Opportunities. In: Kahneman D, Slovic S, Tversky A, editors. Judgement under Uncertainty: Heuristics and Biases. New York: Cambridge University Press; 1982; p. 249-67. |
| 20. | Gigerenzer G, Gaissmaier W, Kurz-Milcke E, Schwartz LM, Woloshin S. Helping Doctors and Patients Make Sense of Health Statistics. Psychol Sci Public Interest 2007;8:53-96. |
| 21. | |
| 22. | Collins J. Education techniques for lifelong learning: Writing multiple-choice questions for continuing medical education activities and self-assessment modules. Radiographics 2006;26:543-51. |
| 23. | Harel Z, Riggs S, Vaz R, Flanagan P, Harel D. A session of multiple choice questions in the orientation process to the adolescent medicine rotation. Teach Learn Med 2004;16:365-7. |
| 24. | Cohen J. Statistical Power Analysis for the Behavioral Sciences (2 nd Edition). Routledge Academic; 1988. |
| 25. | Avgerinos ED, Msaouel P, Koussidis GA, Keramaris NC, Bessas Z, Gourgoulianis K. Greek medical students′ career choices indicate strong tendency towards specialization and training abroad. Health Policy 2006;79:101-6. |
| 26. | Msaouel P, Keramaris NC, Tasoulis A, Kolokythas D, Syrmos N, Pararas N, et al. Burnout and training satisfaction of medical residents in Greece: Will the European Work Time Directive make a difference? Hum Resour Health 2010;8:16. |
| 27. | Faul F, Erdfelder E, Lang AG, Buchner A. GFNx01Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behav Res Methods 2007;39:175-91. |
| 28. | R Development Core Team. R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. 2012 |
| 29. | Casscells W, Schoenberger A, Graboys TB. Interpretation by physicians of clinical laboratory results. N Engl J Med 1978;299:999-1001. |
| 30. | Hoffrage U, Gigerenzer G. Using natural frequencies to improve diagnostic inferences. Acad Med 1998;73:538-40. |
| 31. | Kurzenhauser S, Hoffrage U. Teaching Bayesian reasoning: An evaluation of a classroom tutorial for medical students. Med Teach 2002;24:516-21. |
| 32. | Windish DM, Huot SJ, Green ML. Medicine residents′ understanding of the biostatistics and results in the medical literature. JAMA 2007;298:1010-22. |
| 33. | Kozakowski SM. The impact of resident gender on educational outcomes. Fam Med 2010;42:133-4. |
| 34. | Brase GL, Fiddick L, Harries C. Participant recruitment methods and statistical reasoning performance. Q J Exp Psychol (Colchester) 2006;59:965-76. |
| 35. | Nomura K, Yano E, Fukui T. Gender differences in clinical confidence: A nationwide survey of resident physicians in Japan. Acad Med 2010;85:647-53. |
| 36. | Carnes M. Commentary: deconstructing gender difference. Acad Med 2010;85:575-7. |
| 37. | Kellerman SE, Herold J. Physician response to surveys. A review of the literature. Am J Prev Med 2001;20:61-7. |
[Figure 1], [Figure 2]
[Table 1], [Table 2], [Table 3]
|
Search Pubmed for