The significance of statistical significance

Drawing of otter waiting on trial results
Drawing of an otter waiting on trial results

I recently analysed the results of a clinical trial. It was a small study, but i had been involved in from the beginning and so i was looking forward to it. Receiving a new dataset is an exciting time for a statistician and in this case, the data was in such a good format that i was soon able to start analysing.

I started working through my statistical analysis plan and after examining some descriptives, was soon finished with my analysis of the primary effectiveness outcome. Lo and behold, there was a treatment effect on this outcome (a proxy for fitness since you ask). The effect seemed large enough to be clinically important, the confidence interval around that effect was removed from zero, and formal testing revealed that the p-value was quite small indeed. I was delighted. I allowed myself a quiet smile  and progressed to investigating the long list of secondary outcomes that we had pre-specified.

A week later i presented my results to the trial team. I didn’t particularly try to keep the results secret until this meeting, but the chief investigator was not local and so the opportunity to tell people didn’t arise. Everyone would find out at the same time.

I started the meeting by describing the data and presented some  descriptives of both baseline and follow-up measures. When i got around to presenting the primary outcome there was a large cheer from the group and a palpable sense of elation and relief. There was energy and excitement coursing through the room. As i moved on to the secondary outcomes i noticed tears in the eyes of the chief investigator who was visibly moved.

It was a reminder about exactly how much researchers put into delivering their interventions. I mean, i consider myself one of them. However, the diligence, commitment and sheer graft that chief investigators, trial managers, and data managers pour into their studies is incredible and this experience highlighted the huge responsibility placed in the hands of the final analyst. This team had provided a dataset to me a week before, and while i worked as quickly as i could, I clearly hadn’t gauged the keen anticipation for the results correctly. This team had been losing sleep over the results (one of many studies I work on). The statistician really cannot overestimate how important the final analysis is and it really is a position of privilege.

As an aside, noted the chief investigator was particularly thankful to me for finding a was statistically significant outcome. I protested, saying that all the credit for the treatment effect lay with the intervention deliverers. It did make me wonder what emotions remain unexpressed when i return  less definitive findings!



Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s