Fall webinar report: Null Hypothesis Significance Testing: Should We, or Shouldn’t We?
Dr. Philip M. Sedgwick of St. George’s, University of London discussed the challenges of and shared his experiences teaching null hypothesis significance testing (NHST) in this webinar: “Trials and Tribulations of Teaching NHST in the Health Sciences.”
Challenges in learning and teaching NHST
Dr. Sedgwick began with an introduction to teaching statistics in the health sciences among non-specialists. He emphasized the biggest challenge of teaching NHST is students’ lack of understanding of the underlying principles of statistics coupled with the absence of such topics in textbooks. Other challenges he has faced include:
students’ original intentions in enrolling and lack of ability,
limited curriculum time,
abstract concepts that are not simple or intuitive,
a shift in curriculum design to focus on statistical thinking through real world application as opposed to calculations, formulas, and theory, and
students’ failure to apply statistics and inference appropriately.
Dr. Sedgwick raised the debate of whether one should use NHST, which has been intensified around the misinterpretation of the p-value. He acknowledged the widespread confusion around inference, distinguishing between statistical versus contextual significance and commenting that teaching NHST has become a mechanistic approach where p<0.05 has become the cornerstone of decision making. Dr. Sedgwick’s solution was to reflect on his teaching.
Experiences and approaches to teaching NHST
Dr. Sedgwick discussed his own approaches to learning and teaching NHST using examples, including his personal struggles as a student and a teacher. In the classroom, he does so in a visual way. Dr. Sedgwick emphasized the importance of giving students a historical perspective. He introduced the tale of two theories, including Fisher’s introduction of the null hypothesis and p-value and Neyman-Pearson’s approach to hypothesis testing. He recognized NHST as a combination of these theories, adding that this was not the intention of these statisticians. He questioned why nothing has changed in the last 70 years.
He stressed the importance of developing a culture of uncertainty, underlining that data is evidence and nothing more and the p-value is the strength of the evidence in support of null hypothesis. He aims to steer away from the misunderstanding in healthcare that “a statistically significant p-value is the end of the search for truth.”
Exploring NHST in the research community
Dr. Sedgwick continued with a discussion of statistics reform, noting that basic and applied social psychology discouraged the use of NHST and some others banned NHST. Instead, they placed an emphasis on descriptive statistics, graphs, and effect sizes focusing on estimates and magnitude of effects. Dr. Sedgwick expressed his desire for his students to read literature and be aware of what is happening in the research community. He discussed barriers to change, including publication of only statistically significant results, nonexpert teachers in the classroom, and the erosion of good practice.
During his presentation, Dr. Sedgwick reminded us of several influential quotes:
“Too difficult to teach (introductory courses), encourages dichotomous thinking.” -Hak (2014)
“The emphasis on tests of significance, and the consideration of the results of each experiment in isolation, have had the unfortunate consequence that scientific workers have often regarded the execution of a test of significance on an experiment as the ultimate objective.” -Yates (1951)
“Absence of evidence is not evidence of absence.” -Altman and Bland (1995)
“Evidence of difference is not evidence of no difference.” -Sedgwick (2021)
“The significance of ‘significance’ becomes less significant when significance is used four times in a sentence.” -Gordon
In conclusion, Dr. Sedgwick left us with the idea that assessments sometimes drive learning, as he highlighted some student feedback from his course. Dr. Sedgwick encouraged us to use stories to inform change, and he reminded us that “to be able to teach it, you need to understand it!”
Do you agree? We welcome your thoughts in the comments below.
Comments