Researchers familiar with the National Institutes of Health’s (NIH’s) peer-review process can cite many cases where the reputation of a scientist or their workplace seemed to count for more than the strength of their ideas. There was the scientist who moved from the Ivy League to a large public university and saw scores on his grant applications drop. The investigators from historically Black institutions who always seemed to do worse than applicants from majority white schools. And the big names from research powerhouses whose proposals won stellar ratings without much scrutiny.
“Anyone who actually attends a study section sees [reputational bias] happening,” says Noni Byrnes, director of NIH’s Center for Scientific Review (CSR), referring to NIH peer-review panels. Now, in a bid to reduce that bias, her center is proposing the first overhaul of NIH’s scoring system in 14 years. But the idea is getting a mixed reception.
The proposed reform was prompted, in part, by a striking pattern in NIH awards: About 70% of grants go to just 10% of all NIH-funded institutions. NIH says both anecdotes and data suggest reputational bias is partly responsible. So, in 2020, a working group of CSR’s advisory council began to examine ways to reduce this bias, as well as ease the workload for reviewers.
Currently, reviewers rate proposals on significance, investigator, innovation, approach, and environment. The reform proposal, released last month, would reorganize those five criteria into three factors. Two would be scored: importance, and rigor and feasibility. Reviewers would also consider, but not directly score, a third factor: expertise and resources. If the reviewer sees problems in these areas—because the applicant lacks relevant experience, for example, or access to key equipment—the reviewer could adjust the overall score and write a brief explanation for NIH staff.
Some researchers worry these changes would go too far. To Shelley Berger, an epigenetics researcher at the University of Pennsylvania and a member of the NIH director’s advisory panel, de-emphasizing an applicant’s background is a mistake. “There is value in some objective score to try to help to assess the investigator,” she said at the panel’s December 2022 meeting. Rather than remake the scores, NIH could use “other methods to try to help underrepresented investigators and institutions to get into the NIH funding system,” she wrote in an email.
In contrast, others think the changes don’t go far enough. They say NIH should simply blind reviewers by preventing them from seeing an applicant’s identifying information, at least initially. NIH is already experimenting with such blinded reviews for a limited number of grants. In the first year, the approach appeared to result in a more diverse pool of applicants for those grants and greater racial diversity among awardees. Some other U.S. research agencies, such as NASA, are also testing blinded reviews.
At NIH, Byrnes says implementing blinded reviews on a large scale would be impractical, because anonymizing proposals is “unwieldy” and reviews take longer. In addition, the agency has found in some tests of blinding that reviewers could still identify about 20% of the applicants.
NIH is now collecting comments on the proposal, with a 10 March deadline. CSR expects to release the final review criteria this summer.
If NIH adopts the new system, it would launch in 2025. The agency should monitor whether it is having any impact, says drug abuse researcher Michael Taffe of the University of California, San Diego, who has long criticized NIH for not doing enough to narrow a funding success gap between Black and white scientists. “NIH has a tendency to put these policy changes in place and not … determine if they are producing the desired outcome,” he says.
Researchers familiar with the National Institutes of Health’s (NIH’s) peer-review process can cite many cases where the reputation of a scientist or their workplace seemed to count for more than the strength of their ideas. There was the scientist who moved from the Ivy League to a large public university and saw scores on his grant applications drop. The investigators from historically Black institutions who always seemed to do worse than applicants from majority white schools. And the big names from research powerhouses whose proposals won stellar ratings without much scrutiny.
“Anyone who actually attends a study section sees [reputational bias] happening,” says Noni Byrnes, director of NIH’s Center for Scientific Review (CSR), referring to NIH peer-review panels. Now, in a bid to reduce that bias, her center is proposing the first overhaul of NIH’s scoring system in 14 years. But the idea is getting a mixed reception.
The proposed reform was prompted, in part, by a striking pattern in NIH awards: About 70% of grants go to just 10% of all NIH-funded institutions. NIH says both anecdotes and data suggest reputational bias is partly responsible. So, in 2020, a working group of CSR’s advisory council began to examine ways to reduce this bias, as well as ease the workload for reviewers.
Currently, reviewers rate proposals on significance, investigator, innovation, approach, and environment. The reform proposal, released last month, would reorganize those five criteria into three factors. Two would be scored: importance, and rigor and feasibility. Reviewers would also consider, but not directly score, a third factor: expertise and resources. If the reviewer sees problems in these areas—because the applicant lacks relevant experience, for example, or access to key equipment—the reviewer could adjust the overall score and write a brief explanation for NIH staff.
Some researchers worry these changes would go too far. To Shelley Berger, an epigenetics researcher at the University of Pennsylvania and a member of the NIH director’s advisory panel, de-emphasizing an applicant’s background is a mistake. “There is value in some objective score to try to help to assess the investigator,” she said at the panel’s December 2022 meeting. Rather than remake the scores, NIH could use “other methods to try to help underrepresented investigators and institutions to get into the NIH funding system,” she wrote in an email.
In contrast, others think the changes don’t go far enough. They say NIH should simply blind reviewers by preventing them from seeing an applicant’s identifying information, at least initially. NIH is already experimenting with such blinded reviews for a limited number of grants. In the first year, the approach appeared to result in a more diverse pool of applicants for those grants and greater racial diversity among awardees. Some other U.S. research agencies, such as NASA, are also testing blinded reviews.
At NIH, Byrnes says implementing blinded reviews on a large scale would be impractical, because anonymizing proposals is “unwieldy” and reviews take longer. In addition, the agency has found in some tests of blinding that reviewers could still identify about 20% of the applicants.
NIH is now collecting comments on the proposal, with a 10 March deadline. CSR expects to release the final review criteria this summer.
If NIH adopts the new system, it would launch in 2025. The agency should monitor whether it is having any impact, says drug abuse researcher Michael Taffe of the University of California, San Diego, who has long criticized NIH for not doing enough to narrow a funding success gap between Black and white scientists. “NIH has a tendency to put these policy changes in place and not … determine if they are producing the desired outcome,” he says.