Author Topic: Experiment Reproducibility -- Nature Article  (Read 4451 times)

Offline G

  • talky crow
  • ****
  • Posts: 33
  • Karma: +0/-0
    • View Profile
Experiment Reproducibility -- Nature Article
« on: August 16, 2012, 01:44:39 AM »
http://www.nature.com/news/independent-labs-to-verify-high-profile-papers-1.11176

It's an interesting article to read through.  The quick and dirty: it highlights the recent attention of late to papers not having their results reproduced (either through inaction or retesting).  So, now there's a new initiative to incentivize folks (with certifications and additional publications) to attempt to reproduce experiments. 

I find the last paragraph a bit disturbing and seemingly counterproductive to the goals of reproducibility.

Quote
So what if the results don't reproduce? "That's going to be very interesting," Iorns says. "We won't force anyone to publish [validation study] results."

WHY NOT?  Don't we learn just as much, if not more, when an experiment fails?  What purpose is there in hiding failed retests?  If anything, wouldn't that potentially exaggerate the positive results from other retests?

Offline Daneel

  • Miocene Terror Bird
  • *******
  • Posts: 202
  • Karma: +0/-0
    • View Profile
Re: Experiment Reproducibility -- Nature Article
« Reply #1 on: August 16, 2012, 06:29:19 PM »
I agree! Publication bias is a big problem, specially in medicine. Negative studies have little chance to be published, positive studies get published twice as often.
Do you remember Daryl Bem's paper about precognition? The same journal that published his result didn't allow for the publication of 2 (failed) replications, not because it wasn't interesting or the quality of the studies was questionable, but because it's the policy of the journal not to publish replications. Say whaaat?

Offline G

  • talky crow
  • ****
  • Posts: 33
  • Karma: +0/-0
    • View Profile
Re: Experiment Reproducibility -- Nature Article
« Reply #2 on: August 17, 2012, 12:36:52 AM »
Exactly!  I feel like there should at the very least be a permanent, visible, online presence for each study in which all replications can be submitted for peer review and forever linked to the original.  Some sort of reproducibility percent would then be visibly attached to the original study, as well.

Failures really should be published!  If there were flaws in the design or execution of the reproduction, it wouldn't be held against the original, but at least the knowledge of the mistakes that were made could assist and prevent others from making similar mistakes in the future.
« Last Edit: August 17, 2012, 12:38:43 AM by G »

Offline jfpohl

  • talky crow
  • ****
  • Posts: 32
  • Karma: +0/-0
    • View Profile
Re: Experiment Reproducibility -- Nature Article
« Reply #3 on: August 22, 2012, 06:55:53 AM »
As someone who has published in medical journals, I definitely see publication bias in my field.  Negative results are extremely difficult to publish.  If a researcher ever works with a pharmaceutical company for a drug study, a clinical trial can get great results for medications that really help people.  However, pharmaceutical companies will always have a self interest and are not keen on publishing results when drugs don't work as planned.  Not good.   :P
"Doctors will have more lives to answer for in the next world than even we generals." Napoleon Bonaparte

Offline LukeSouthan

  • Miocene Terror Bird
  • *******
  • Posts: 208
  • Karma: +0/-0
    • View Profile
Re: Experiment Reproducibility -- Nature Article
« Reply #4 on: August 22, 2012, 07:50:14 AM »
My dissertation tutor was super into getting my dissertation published as I was doing the experiment and then when all the data came in it wasn't quite statistically significant to give a positive result, but did give some significant negative results and she suddenly lost any interest in getting it published, it was pretty sad.

Offline Daneel

  • Miocene Terror Bird
  • *******
  • Posts: 202
  • Karma: +0/-0
    • View Profile
Re: Experiment Reproducibility -- Nature Article
« Reply #5 on: August 22, 2012, 10:19:53 AM »
Makes you keep things into perspective and remember that science is a human endeavour, doesn't it?

Offline jfpohl

  • talky crow
  • ****
  • Posts: 32
  • Karma: +0/-0
    • View Profile
Re: Experiment Reproducibility -- Nature Article
« Reply #6 on: August 22, 2012, 02:37:22 PM »
I absolutely hate stories like that.  I'm sorry that happened to you.
"Doctors will have more lives to answer for in the next world than even we generals." Napoleon Bonaparte

Offline Cortexture

  • talky crow
  • ****
  • Posts: 21
  • Karma: +0/-0
    • View Profile
Re: Experiment Reproducibility -- Nature Article
« Reply #7 on: September 01, 2012, 01:51:41 PM »



It seems there are two issues here.  Publication bias has already been mentioned, and this seems to have it's origins in the business models of journals that lead to selection bias, but as has been noted elsewhere, scientists rarely give reviewers the opportunity to exercise such bias.  The other issue, at least in some fields, relates to the logic of the analysis methods, in particular weak uses of null hypothesis significance tests. 

Offline Ed Lolington

  • Miocene Terror Bird
  • *******
  • Posts: 241
  • Karma: +0/-0
  • Honorary adjudicator, daytime tweeter.
    • View Profile
    • www.lolington.com
Re: Experiment Reproducibility -- Nature Article
« Reply #8 on: September 06, 2012, 10:55:17 AM »



It seems there are two issues here.  Publication bias has already been mentioned, and this seems to have it's origins in the business models of journals that lead to selection bias, but as has been noted elsewhere, scientists rarely give reviewers the opportunity to exercise such bias.  The other issue, at least in some fields, relates to the logic of the analysis methods, in particular weak uses of null hypothesis significance tests.

I know what a "null hypothesis" is but I'm not sure I understand your last sentence. Could you explain what you mean?

Offline Daneel

  • Miocene Terror Bird
  • *******
  • Posts: 202
  • Karma: +0/-0
    • View Profile
Re: Experiment Reproducibility -- Nature Article
« Reply #9 on: September 06, 2012, 03:30:34 PM »
I think he's saying that the standard statistical tests used in many fields are bias towards the rejection of the null hypothesis.