"Research Shows" - how spurious is this claim?

Post Reply New Topic

A lot of new research is often posted by well-meaning members on WP, usually on the GAD forum. I entirely understand the excitement of feeling that a new discovery has come along which seems to have the potential to make a really valid and meaningful difference to our lives. However when I read the links I am all too often appalled at the sloppy design, over-reaching claims, poor sampling, self-serving claims by the researchers, invalid conclusions, gross biases of omission in the discussion section, and most of all concerning issues of validity and reliability. And as I have sadly said in previous threads, the amount of cheating in science now, particuarly the soft sciences like psychology, (neuroscience is highly suspect too, IMO, lately) would truly astonish the ordinary person in the street if they knew of the extent of it.

Today I read an article in the Guardian on a study of replicability that suggests 64% of studies published in reputable journals are not replicable. That's shockingly high, even to a sceptical evaluator like me. It makes me wonder how far the rot has gone, and the exploitation of misplaced faith by the public makes me sad. The media too has a lot to answer for in promoting spurious claims as the "big breakthrough" or next big thing. Many of these journalists are no more than scientifically ignorant, myth-promoting PR people, wittingly or unwittingly. There is a bigger picture in this too - about the kind of time and ethos that Western science and political culture has fallen into. I can only hope that the HFA scientists have higher ethical commitments and are resisting this vile trend of self-promotion at any cost; hopefully their integrity is less compromised.

http://www.theguardian.com/science/2015 ... are_btn_fb

Gotta keep dem funds flowing, man.

But yeah, these are shocking numbers indeed.
At the same time I'm really not surprised at the amount of bias, because people are generally biased and it's difficult for most to be objective.
Anyway thanks for the heads-up.

I think the problem is with the media culture. Very early studies with potential implications are published by the media. It would be too long and complex to report the study and what it actually means and that stage it is in, so the media oversimplifies it and picks the potential implications it may or may not have since those would sell better and make them more money.

That's why you need to look for multiple studies that show the same thing. One study can often be wrong. Ten studies saying the same thing? Much less likely to be wrong.

It's precisely the reason I can't take the news seriously. You just can't take a single observational study with high levels of bias and a small sample size and generalise it to the extent I frequently see in the news. You need other studies to back it up, and those studies should be designed in a way that accounts for extraneous variables (among other things). I get concerned about the people who may not consider these things.

I think a lot of these studies are promoted in order to make us feel optimistic or to justify common health habits. Every week there seems to be a new 'cancer breakthrough' or research that suggests that 'drinking X number of coffees per day can lead to a decreased risk of A, B and C'. Then there might be another study done that contradicts it, but we may not hear about it in the news because it's something we won't want to hear. It's no wonder people get confused as to what they should do for their health.

I have been thinking about this since the first news stories about the high rate of non-reproducible results came out.

It hardly seems surprising given the forces at play in academia: publish or perish and don't rock the boat of the larger real estate concern that bears the university name. There is every incentive to cheat and few controls to balance those incentives. It's a sad state of affairs.

It seems that the ordinary person in the street doesn't have much faith in pronouncements in the name of science anyway, valid or invalid. Interpreting the real work is too hard, the nontechnical explanations are lost in a sea of lies and hucksterism and many prefer myths and faith to knowledge in any case.

From that somewhat bleak basic vantage point, I find this study encouraging and reason to be hopeful.

This study is a challenge to the status quo and a force that will push toward necessary reform, much as Thomas Insel's provocative response to the DSM 5 was, though he later partially retracted that harsh critique in the spirit of "going along to get along." It's a small step in the process of self correction that will advance these studies toward real knowledge, despite the desires of entrenched interests vested in the current system.

Whatever happens in the world of academic science, the primary source of messages about science that most people will be exposed to is from advertisers:

These 'researchers' get paid to research, not to find the truth. I am very sceptical of any research on any subject.

I think that's a bit simplistic and unfair to some researchers.

Some research is paid for by large companies who clearly want a certain result and it's entirely rational to be skeptical of that research.

But other research is paid for by things like National Science Foundation grants, and the researcher has no direct financial pressure to find any particular result. The more insidious pressure comes from a ridiculous bias against publishing negative findings in journals and a sense that negative results are somehow failures in some academic circles. This almost certainly pressures some researchers into exaggerating results.

But there is still a lot of good research. An important thing not to forget is written in the Guardian piece this way:


Another good story about this with a slightly different take is here:
http://nymag.com/scienceofus/2015/08/ma ... false.html

Any time you see a statement that begins "Research shows..." and ends with a single, clearly explainable result - then you can bet good money that it isn't true - or at least it's a gross oversimplification. Even similarly designed research comes up with different results - because the question is a little different, the study cohort is different, etc. etc.

I do research for a living - social research, but still... Most researchers are trying to do a pretty good job but they are fallible and subject to influence. That being said my last but one project was paid for by a big company and I was very critical of their industry and they didn't say anything and let it be published. My current project is publicly funded and I'm currently being pressured to remove information that's worrying but of public interest. So, it's not always straightforward that industry money = bad research. But you should be more sceptical!

These are very interesting comments, thank you. I so second your important point Adamantium about publishing negative results, and regarding them as the important information that they in fact are. This is, in a sense, the biggest bias of all, and it is related to the other old and entrenched scandal of repeating statistical analyses on raw experimental data from one design (say with t testing for example) and then only reporting/publishing the ones the fit your original claims, omitting all reference to the number of negative results. (I say "claims" deliberately here, instead of theory)..

This appalling reporting of selective testing has gone on for decades and until the "reputable" journals force researchers somehow to hand over all their raw data, to be kept in some sort of storage for independent analysis if indicated later, or employ specialist statisticians themselves, it will continue. No sign of that happening, even though there have been a growing number of exposed scandals in the top journals over the last couple of years - like the outrageous "peer review" scam Nature exposed last year.

Newspapers have an almost complete silence on the important negative results (from meta-analysis studies) and that silence is not in the public interest at all, it is actually a kind of deliberate omission that is harmful to the public interest.

At least in a tiny way we can share and hopefully spread and raise some awareness here and from here. The whole thing has reached a scandalous level and it is just exploitative of people who fund universities and research outfits through donations, taxes etc in good faith.

I don't fully understand how it works, but researchers who perform meta-analyses (combining results from multiple studies) often do a statistical correction for the anti-null bias. Luckily, in most cases, opposite correlations to those expected are still more likely to get published than a null finding, so if you see many studies getting opposite results from each other, it suggests that there is actually no correlation. They can also estimate how many unpublished null reports would be needed to negate the correlation found in a meta-analysis, and the larger that number is, the more likely it is that what they've found is reliable.

As a physical scientist, my take is that there are some problems with the way data is experimentally handled and interpreted in certain types of research in science. (Certain areas are easier to misinterpret than others, but that is the nature of the beast.) Unfortunately, it can be difficult to easily separate the good from the bad. You would think that peer reviewed journal editors would always be able to do this, but alas, they make mistakes also. There is a good old boys club built in certain journals...

What can complicate this issue more is that there has been an uptake in pay to publish science journals, promoting an easy way to get your research out there in print. Some of them are getting $3500-5000 for each article that goes into their publication that issue. Because of the demands of higher education, some professors do take the short cut route, which blocks the normal peer review process from working. I know of this because I have been pestered by these journals trying to get me to submit to this scheme of theirs. But, I will not.

In grad school, I was trained on how to spot a good peer reviewed article vs. a not so good one in my area. It was part of the cume exam process in a way. This is important thing to know as scientists use data from these articles to build experiments for their own research. If the data is flawed, then it can set the next group of scientists back years trying to figure out why their ideas failed when doing a comparison. There are some retired professors that constantly double check research data in recently published articles to find the ones that are wrong. They then publish about what they found on the data and get another publication on their list. You do not want to be one of their targets.

As for a means to fix this entire process, I really do not have any ideas that could work 100%. Any means of direct change to the peer review process will cause the old guard to fight it. I do think that the pay per publish journals need to be controlled a bit more to prevent some of the stuff that is being called "scientific research". Unfortunately, some of them are published in countries that protect the journals from this. However if nothing is done, the future cost to science could be severe.

Technology has been part of the peer review problem. Cunning, unscrupulous and ambitious researchers learned of weaknesses in the software of reputable journals and exploited it to "peer review" their own studies and the staff at Nature were shocked to learn how often this had happened, even to them. And they had unwittingly colluded with the cheats by relying on computer communications rather than making direct contact with peer reviewers and having dialogue with them personally.

Rather analogous to an employer taking all sorts of claims in a CV at face value and regarding cooked up references etc as the real McCoy.

You have to wonder how far the rot has gone and what hasn't been uncovered. I suspect that if we knew the full extent of it we would sit here in shock, our mouths gaping open..

Unless they are all funded by the same or associated sources, with an underlying agenda to show a certain result.

Ditto the above - my research shows that research can show whatever the people who paid for the research want it to show.

"Experts claim . . . "
"Scientists / doctors say . . . "
"Studies suggest . . . "
"Latest research shows . . . "

You might enjoy this website: SkepticalRaptor

http://www.skepticalraptor.com/skeptica ... ence-news/