#ReadyResources: Talking about inappropriate analysis requests
Do you want to:
teach good research habits by talking about "what not to do?"
create a "safe space" to talk about research misconduct without putting learners on the defensive?
help health scientists recognize troublesome behaviors as misconduct by providing a trusted source?
talk about potential response bias on a survey with sensitive data?
If you said "Yes" to any of these, then this October 2018 paper by Wang, Yan, and Katz in the Annals of Internal Medicine [1] might be just what you're looking for.
The study team invited 800 consulting biostatisticians from ASA to take a survey on inappropriate requests made by researchers with regards to the statistical analysis. Five-hundred twenty-two opened the email invitation and 74% percent responded.
There were 18 different "inappropriate requests" considered in the Bioethical Issues in Biostatistical Consulting (BIBC) Questionnaire used by the team. The survey asked about requests that were made of them as well as those they had heard about from others. Respondents indicated their perception of the ‘severity of [the] Bioethical Violation’ as well as on the frequency of it in the last 5 years.
Data were assessed by age, gender, and race/ethnicity. It was observed that inappropriate requests were reported most often by younger biostatisticians.
The benefits to using a published study like this are that it gives an opportunity to talk about "what not to do." This list of 18 items are not extreme behaviors. While there are cute terms for some inappropriate analysis practices (e.g., "data dredging," "fishing," "p-hacking") what is nice about this list is that there are very specific behaviors called out (e.g., "Modify a measurement scale for some desired results"). Novice analysts or folks who are non-majors in statistics could be more vulnerable to these types of requests. By discussing this paper, the learners have the opportunity to identify and consider counters for these types of requests.
The fact that the paper is published in a well-known and respected medical journal, Annals of Internal Medicine, may provide credibility to the concerns being raised. By publishing this article, the editorial team and peer reviewers are sending the message that this topic is important for their clinical audience.
Of course, talking about inappropriate requests for analysis can be met with hesitation. Learners may feel uncomfortable talking about personal experience, either making or getting such requests, for fear of judgement of their peers or instructor, or for fear of outing a colleague. By using a published journal article, these items can be discussed in the context of the survey results and the experiences of the respondents, thus creating a safer space for discussion.
However, the sensitivity of the topic brings up a potential concern for response bias, an issue noted by the authors. There are many types of sensitive topics that a researcher may be interested in studying. This topic of inappropriate conduct in research may be relatable to a broad audience. Understanding the limitations of different data collection formats and the implications of missing data on the results is important.
For even more discussion, consider the target population for the survey. Would statisticians who are not declared "consultants" respond differently? What about researchers or assistants whose roles are not primarily statistical?
Do you plan to use this resource in your classroom? Tell us how it goes in the comments below.
Want to take it further to discuss publication bias and the drive for "significant" findings? Consider this opinion piece about the AIM article "No Wonder Scientists Ask Statisticians to Cook the Data" in the Bloomberg Opinion column on Oct 26, 2018.
1. Wang MQ, Yan AF, Katz RV. (2018). Researcher Requests for Inappropriate Analysis and Reporting: A U.S. Survey of Consulting Biostatisticians. Annals of Internal Medicine. October 2018.