Can you trust medical research?
Research helps us find answers to problems such as heart attack, stroke and dementia. Find out how the BHF is making sure those answers are correct.
At a prestigious medical institution in Sweden, a surgeon called Paolo Macchiarini made headlines when he gave people ‘new’ windpipes built from stem cells. But in 2016 it came to light that rules and procedures hadn’t been followed. Most of the patients who received these windpipes had died.
As the nation’s largest funder of heart and circulatory disease research, the BHF takes serious steps to avoid scandals like this harming patients and lowering the reputation of scientific research.
Our committees receive more than 500 funding applications a year from researchers. These are reviewed by external experts (UK and international) as part of a rigorous process.
Professor Jeremy Pearson, BHF Associate Medical Director (Research), says: “No funding body can guarantee that their research is correct 100 per cent of the time. But the processes we have in place and the experts we have on our funding panels are good enough that the research process is sound. It will be rigorous science. Something that is not designed properly doesn’t get funding.”
Building on other scientists' work
The BHF and the scientists we fund are always looking to make improvements. One of our top researchers, David Eisner, BHF Professor of Cardiac Physiology at the University of Manchester, published a paper on the importance of judging research quality. He wants to see scientists given more training on designing experiments, and more studies trying to repeat other people’s experiments.
“Scientific progress is based on someone doing research and publishing a study, then someone else building on it and improving it,” says Professor Eisner. “But there is concern that a large fraction of scientific papers aren’t reproducible – if someone else does the same thing, they will get a different answer.” There are many reasons why this happens – ranging from mistakes in the study design, to the experiment being carried out differently, to fraud (in a small minority of cases). To prevent any of these, Professor Eisner says scientists must record every detail of how the study has been carried out.
The other issue is the role of scientific journals. “Typically, journals don’t want to publish negative results,” he says. “It is much more interesting to say something does something, than that it doesn’t do anything. But that isn’t always helpful. This is avoidable by making sure journals publish results of studies, whether they are negative or positive.” For clinical trials (studies on human beings), this is already happening. If a drug is found to have no effect, the study is published so the information is available to others. Professor Eisner wants to see this in other areas of science.
At the BHF, Professor Pearson (who has served on editorial boards of several academic HM journals) agrees this is an issue. “Scientific journals are like newspapers,” he says. “They like new things and stories that are interesting. It should be easier to publish negative papers – I think that is gradually happening.”
Guidelines for carrying out good studies do exist. They recommend including a clear summary of your aim, how you will do it, what the sample size will be and how you will measure the significance of your findings.
“Not all journals follow these guidelines, but more and more are doing so,” says Professor Pearson. “We make sure our research committee members are aware of the guidelines and the importance of understanding the statistical methods being used.”
He predicts that over the next 10 years guidelines will become more rigorously enforced, while challenges to existing ideas will become more welcome.
Back in Sweden, Paolo Macchiarini was dismissed from his post at the Karolinska Institute and several senior figures resigned.
Stem cell research continues – including many BHF-funded studies. Professor Suwan Jayasinghe, from University College London, is using stem cells to create replacement blood vessels. “We are going to take a long time to get there,” he says. “But we would rather do that and sleep with a conscience than find we have killed somebody."
“One of the reasons I love science is that it works in checks and balances. You do something and if someone else can’t replicate it, then they know there is a problem.”
Is it good or bad science?
If there’s a research story in the news, how do you know whether it’s reliable? “It can be hard to tell,” says Professor Pearson. You can look at the size of the study: “If it’s based on a small number of people, or animals, that should ring alarm bells. You can’t draw any firm conclusions from it.”
The journal it’s published in can also indicate quality – it’s usually a bad sign if a study isn’t published in a scientific journal at all. Although not perfect, all reputable journals have a process of checking research with other scientists. Stories based on surveys or speeches haven’t been checked as carefully.
Something that is not designed properly doesn’t get funding
Professor Jeremy Pearson
BHF Associate Medical Director (Research)
Professor Pearson also advises looking at who funded the research. “A drug company funded trial can be trustworthy, as long as it is independently designed by a university researcher and the results are published without interference,” he says. “But if there is a study showing that cheese is good for you and it is designed by a cheese manufacturer, take it with a pinch of salt.”