Statistical Vs. Practical Significance: the Bangladesh Mask Study & Dementia Medicine

Let’s go back to late 2021.

My nursing home was just starting to come out of the inhumane nightmare of window visits we had forced on our residents, while at around this time my local county had just announced they were re-imposing indoor mask mandates again, after easing them for several weeks.

For me - like I’ve said many times, in the beginning when fear was everywhere, confusion reigned, all social media and news media seemed to be sending a unified message of fear - I supported mask mandates (temporarily), to “buy time” while vaccines were being rushed into development.

But I was starting to get really antsy at this point.

I *knew* they were an intervention w/ at best a negligible effect, and I was looking forward to them finally going away (at this point I thought we had long past the point where mask mandates made any sense).

When I first saw the headlines months and months ago on the now-infamous “Bangladesh mask study,” particularly when paired with the major, breathless ballyhooing in the media - I confess I was taken aback at first.

Study co-author, Yale professor of economics Jason Abaluck, triumphantly gassed in the media:

“This should basically end any scientific debate about whether masks can be effective in combating covid at the population level,” Jason Abaluck... who helped lead the study said... calling it “a nail in the coffin.”

I remember being actually somewhat disoriented by reading these pronouncements. I skimmed over a few of the news studies, and then also looked over the report itself, and wrote one of my first GeroDoc articles which basically catalogued my understanding of what common forms of facial PPE (“Personal Protective Equipment”) can do, and can’t do.

Cliff’s Notes Version of the Abaluck Mask Study

The 2021 Bangladesh mask study was a very large, very expensive randomized controlled trial aimed at evaluating the effectiveness of masks in preventing COVID-19 transmission in rural Bangladesh. In the study, which included a total of 342,183 participants across 600 villages in rural Bangladesh, participants were given cloth or procedure masks (or left alone) and in the masked groups, were variously ‘encouraged’ (often using local police, reportedly) to get people to be compliant with masking instructions:

Megan Mansell @mamasaurusMeg

@doc_gero @Jabaluck Lest we forget that the current edition entirely omits the police presence aspect of their methods!

Again - the study was massive, with an “N” of 342,183.

The study found that the provision of masks (and level of reported mask adherence) reported reduced the incidence of symptomatic COVID-19 infections by around 11 percentage points in the treatment group relative to the control group, which is modest, but statistically significant result - albeit one that was apparently realized by the contributions of approximately 20 study participants out of the 342,183 total:

Alistair Haimes @AlistairHaimes

It's fascinating that we're still debating masks. Though I wonder how relevant the setting of Bagladeshi rural life really is to people taking a half hour trip on the District Line at rush hour? telegraph.co.uk/news/2021/11/2…

More on that later.

The Bangladesh Mask Study Since Then

If you want a detailed demolishing of the various problems with how Abaluck and his study partners committed various data & inferential crimes in conducting their multi-million dollar mask trial, I’d first consult the mysterious Bad Cat:

In his (or her? cats are always ‘she’ to me anyways) the illustrious cat goes through all of the problems with it - starting with the study not even properly accounting for COVID-19 seroprevalence in their study population as they distributed test subjects into the various arms of their study… which Bad Cat makes a very convincing argument simply nukes the study’s credibility out of the gate.

Beyond that, there’s the issue of so-called “ascertainment bias,” which, although aren’t mentioned in kitty’s piece (worth a read) - are definitely at play when you read how the study was conducted. Basically - this wasn’t just a study about masks, this was a study about masking instructions, if not a study of a full-court press on incentivizing, begging, bribing, and possibly intimidating subjects into masking.

Regarding ascertainment bias - the technical definition of it is “when some members of a target population is more likely to be included in a sample than others.” Princeton molecular bio professor Maria Chikina and others in fact looked at Abaluck’s study and found telltale signs via their re-analysis of the study data that ascertainment bias had occurred.

Aspects that could have caused this were recruiting subjects from social media and email, and of course the fact that study subjects were by definition individuals willing to wear masks - so not representative of an average person who might be subject to a mask mandate (this is super important to remember, given that this Bangladesh study was so often used to justify mask mandates - the idea of forcing often unwilling people to wear masks). Also, the study was unblinded - which may seem unfair as a criticism (after all, how can you do a blinded study of masking?) - but it seems obviously likely led to all sorts of other things that gave the treatment group an edge. For example, subjects were given all sorts of additional education, encouragement, and possibly even a little intimidation by local police - to wear masks.

If you follow Jason Abaluck on Twitter (who otherwise seems like a perfectly unremarkable Yale economist), in February he spent quite a bit of time on defense about his mask study, particularly in light of the Princeton re-analysis, but also questions about how he and his study partners handled the question of seroprevalence, and lingering questions about some ethical issues in how the study was conducted.

It’s an open question as to how he acquitted himself. I believe that Abaluck himself has also framed his study as one not necessarily of masks wearing only, but of the effectiveness of mask promotion itself, which, fair enough.

I understand the spirited defense. Abaluck is clearly a young guy (good god he looks like a kid) and his Bangladesh study likely represents somewhat of a careermaking achievement for him (he apparently was granted a full professorship in July 2021, just prior to the study being published).

Moreover, I’d say more than any one “media event” of over the last few years - the Abaluck et al. mask study was probably more responsible than any other for keeping the myriad of mask mandates in place since December of 2021 up until now. That 11% risk reduction, combined with the enormous study sample size (N= 341,842), knocked everyone’s socks off.

The Case of Aricept (Donepezil)

In 1996, the FDA approved the first drug developed for treating Alzheimer’s disease, something called a cholinesterase inhibitor, known as Aricept, also known by its generic name donepezil.

Prior to this, there really were no FDA approved medications for Alzheimer’s disease, so this was a big deal. Like any drug that’s approved by the FDA via the normal approval process, it needs data showing that it’s 1) safe, and 2) effective.

One of the key ways that Aricept was proven to be effective was in the 1996 study in the journal, Dementia by authors Sharon Rogers & Lawrence Friedhoff of the Donepezil Study Group.

Not only did they showed the drug was fairly well tolerated - importantly they showed showed a durable improvement across the study period in cognitive testing scores for participants when compared to controls on something called the Alzheimer’s Disease Assessment Scale, or “ADAS-Cog,” an established research assessment that looks broadly at memory, orientation, naming, and a variety of other cognitive domains in an 11 item package.

The chart above is pretty self-explanatory. Essentially, 5 mg of Aricept results in a two-point, and statistically significant difference between those that took the drug, and control subjects on the ADAS-Cog (as an aside, we use 10mg as a therapeutic dose in prescribing).

Great! The only problem is - what does this two-point difference mean?

On the ADAS-Cog, which is a measure with a total potential score of 70, a score of 0-7 is considered normal or no cognitive impairment, a score of 8-18 is considered mild cognitive impairment, and a score of 19-70 is considered moderate to severe cognitive impairment. While this 2-3 point difference on the ADAS-Cog was statistically significant - it’s unclear what the practical significance is.

Does it represent the ability to, say, remember how to dress yourself, or remember a loved one’s name? Does it represent anything clinically meaningful? It’s honestly hard to know - and although this measure does correlate to functional status in dementia patients according to previous research - such a small difference (2 points) may not represent much of anything of practical value.

That being said - we do prescribe Aricept to dementia patients, and we use it even in our nursing home. It seems like for some patients, according to later research that has been done - it may delay decline of individual patients by about six months (e.g., in other words, your Alzheimer’s disease may simply stop in it’s tracks for a half-year before resuming it’s decline).

While this isn’t by any means a cure, much less an impressive treatment, Aricept is something - and given the low side effect profile and relatively low cost (a few hundred dollars a month overall - much less with insurance), we use it. But it wasn’t the above 1996 FDA study noted above that sealed the deal - I would argue in that case, we really didn’t have any idea as to whether Aricept was of practical use.

All we knew was that it was well-tolerated, and produced a statistically significant result in the therapeutic direction.

Back to the Abaluck Mask Study and that 11% Reduction

So let’s take the Bangladesh study at face value. Their mask intervention approach yielded an 11% reduction of COVID-19 incidence over the 24 months that they studied participants. Moreover, given it had such a huge sample size, the results are much less likely to have been the product of spurious, chance results seen in much smaller groups of studied individuals (all other things being equal).

That should be enough, right? This is the “nail in the coffin” for mask deniers, correct?

*This* is what stopped me when I first read about Abaluck’s study back in late 2021. I saw the 314K+ study participants, and the 11% reduction number, and I thought - what does this mean, practically speaking? Particularly when you realize:

“a recent reappraisal found the difference between the intervention and control villages was just 20 cases out of more than 2,000 infections over eight weeks.”

Let’s be clear. Absent mandates, absent my bluster on Twitter, absent my own personal feelings about the matter - I fully support people if they wish to wear whatever facial adornments they wish in public, masks or otherwise.

I also don’t consider the poor acquittal of the Bangladesh study, nor the fact that the venerable Cochrane Collaborative after reviewing 78 separate randomized controlled trials on masks for airborne viral illness (including the Bangladesh study) found no reliable evidence of effectiveness of masks in reducing viral illness spread - I don’t consider any of this as evidence that “masks don’t work” (after all, science does not do well with ‘proving a negative.’)

None of this rules out the possibility that masks can be helpful or useful in *some* circumstances, for some particularly motivated individuals.

What it does seem to be is that just like prior to 2019 - the reality we knew then is basically unchanged from now. On an aggregate level - the practical benefit of community masking by asymptomatic people in non-clinical situations is unknown.

If you Can’t Establish Practical Benefit - Maybe you shouldn’t Mandate it?

It’s hard to look at the 340K+ Bangladeshi subjects in the study by Abaluck and his study partners, the massive amount of cost, study staff, materials, and work involved, and imagine easily how the amount of money and effort expended to delay (and I say “delay” deliberately - because this intervention does not prevent anything) the spread of COVID across a population by 20 cases over 8 weeks is of practical use.

Given that - is there any evidence anywhere that could justify mandating masks on a population ever again?

I would say - no.