Obama admin. addresses 'Replication Crisis' in US taxpayer funded science. Current system driven by academic publishing model doesn't yield best science. Fed. Register seeks comments. "Registered Reports" conducts peer review before results are known-NY Times, Nyhan

.
9/18/14, "To Get More Out of Science, Show the Rejected Research," NY Times, The Upshot, Brendan Nyhan, Assistant Professor, Dartmouth

"In 2013, the federal government spent over $30 billion to support basic scientific research. These funds help create knowledge and stimulate greater productivity and commercial activity, but could we get an even better return on our investment?

The problem is that the research conducted using federal funds is driven — and distorted — by the academic publishing model. The intense competition for space in top journals creates strong pressures for novel, statistically significant effects. As a result, studies that do not turn out as planned or find no evidence of effects claimed in previous research often go unpublished, even though their findings can be important and informative.

For instance, a top psychology journal refused to consider studies that failed to replicate a disputed publication claiming to find evidence of extrasensory perception. In addition, the findings that do get published in these journals 

often just barely reach the statistical significance thresholds required for publication — a pattern that suggests selective reporting and publishing of results. 
.

Not surprisingly, other scientists often cannot reproduce published findings, which undermines trust in research and wastes huge amounts of time and money. These practices also create a shaky knowledge base for science, preventing scholars from effectively building on prior research.
.

This pattern of publication bias and failed replications, which is drawing attention in fields from psychology to medicine, has prompted great alarm within the scientific community. Now there are signs that these concerns have spread to policy makers. 
.

The Obama administration has asked for public comment on how the federal government can “leverage its role as a significant funder of scientific research to most effectively address” the replication crisis in science— a question that should be carefully considered given the evidence that current policies are not working.
.

One approach is to require researchers to share data, especially from studies conducted with public support. For instance, the National Institutes of Health and the National Science Foundation already require grantees to share data from their research. These sorts of requirements encourage transparency  

but, even if widely adopted, are unlikely to appreciably reduce bias in which studies are actually published.
.

Others advocate requiring the registration of trials before data has been collected. For instance, some social scientists have voluntarily begun to preregister analysis plans for experiments to minimize concerns about selective reporting. 

Unfortunately, the demand for statistically significant results is still likely to create publication bias.

For example, federal law and journal policies now require registration of clinical trials, but publishing of trial results has been found to be selective, to frequently deviate from protocols and to emphasize significant results. Access to trial data could be increased,

but such a step is again unlikely to change which studies are published in the most influential journals.

Instead, my colleagues and I propose a radically different publishing model: Ask journal editors and scientific peers to review study designs and analysis plans and commit to publish the results if the study is conducted and reported in a professional manner (which will be ensured by a second round of peer review).

This procedure encourages authors and reviewers to develop the strongest possible designs — including those that replicate previously published studies — and eliminates perverse incentives to find or emphasize significant results after the fact. A new scientific format called Registered Reports using this approach has already been adopted at numerous journals across the social and natural sciences.
In a new white paper, I propose that the American Political Science Association offer options for articles in a Registered Reports-style format. Researchers in other academic disciplines and scientific associations are starting to do the same.

.

Unfortunately, overcoming the inertia of the current system will be difficult, which is why altering the incentives created by federal science policy is so important.

Scientists would change their ways much more rapidly if federal funding encouraged publishing in journals that used Registered Reports or other formats intended to minimize publication bias. 

Conversely, journals would be more likely to change their policies if it would help them attract research from top scientists. Appropriately enough, the best way to encourage scientific innovation might be to rethink how we organize the scientific enterprise itself."

 
"Brendan Nyhan is an assistant professor of government at Dartmouth College. Follow him on Twitter at @BrendanNyhan."

===================================

From Federal Register:

7/29/14, "Strategy for American Innovation," Federal Register

"A Notice by the Science and Technology Policy Office and the National Economic Council on 07/29/2014"

Subhead, "Science, Technology, and R&D Priorities".

(scroll down): "(11) Given recent evidence of the irreproducibility of a surprising number of published scientific findings, how can the Federal Government leverage its role as a significant funder of scientific research  

to most effectively address the problem?"


================================ 

"Registered Reports"

"Registered Reports emphasize the importance of the research question and the quality of methodology by conducting the peer review prior to data collection and analysis. Accepted papers then are virtually guaranteed publication in the journal if the authors follow through with the registered methodology."

"“Registered Reports eliminates the bias against negative results in publishing because the results are not known at the time of review” said Daniel Simons, Professor at University of Illinois, Urbana-Champaign and co-Editor of Registered Replication Reports at Perspectives on Psychological Science. Chris Chambers, Professor at Cardiff University, section editor at Cortex and AIMS Neuroscience, and chair of the Registered Reports Committee supported by the Center for Open Science (COS) adds, “Because the study is accepted in advance, the incentives for authors change from producing the most beautiful story to producing the most accurate one.”"