On October 20, 2026, the metascience organization Research on Research Institute (RoRI) released a working paper evaluating the Volkswagen Foundation's two-year “distributed peer review” pilot in Germany. In the pilot, applicants reviewed one another's proposals in the same round while an expert panel reviewed them independently at the same time. The evaluation shows fairly high acceptance among applicants and a lighter burden for each reviewer. However, the two tracks selected very few of the same projects, and reviewer expertise and strategic scoring remain contested.
Parallel tracks: applicant peer review and expert review
The rules of distributed peer review (hereafter DPR) are simple. Submitting an application to a funding round also commits the applicant to scoring and commenting on other applications in the same round, and the funder then ranks the applications by their aggregated scores. Instead of relying on a small number of experts to read every application, DPR spreads the review work across all applicants.
The Volkswagen Foundation (VolkswagenStiftung) is a private German funding body. Since 2024 it has trialed DPR in its humanities and cultural studies funding program “Open Up – New Research Spaces for the Humanities and Cultural Studies” (German name: Aufbruch), while keeping its existing expert panel review, so the two tracks run in parallel. According to the process the foundation has published, applicants submit anonymous short proposals and commit to reviewing about 5 other applications. After formal checks and conflict-of-interest screening, applicants use a quick assessment form to score and comment on the applications assigned to them, and about 10 funding recommendations are produced from the scores. On the other track, foundation staff first screen applications internally against the funding criteria, then pass them to an interdisciplinary expert panel for written review and discussion at a meeting, which likewise yields about 10 recommendations. Both sets of recommendations go to the foundation's Executive Board and its 14-member Board of Trustees (Kuratorium) for a decision. Regardless of the outcome, all applicants receive written DPR feedback.
Details of the first round (2024) are in a published paper. All 140 applications went through both tracks simultaneously, and 323 applicants and co-applicants each reviewed 4 to 5 applications. The allocation algorithm avoided pairing reviewers with applications from their own institution and prioritized matches in related disciplines. Each application received an average of 9.92 reviews, and the ranking used a trimmed mean. On the expert track, staff first screened out about half of the applications in each discipline, after which 8 panel members reviewed the rest. The foundation committed to funding the projects selected by either track. RoRI's latest press release says that 301 applications took part across the two rounds in 2024 and 2025, that each project received 9 to 10 independent reviews, and that the highest and lowest scores were dropped before averaging.
Main findings of the two-year evaluation
The new working paper is titled “Distributed Peer Review at the Volkswagen Foundation.” Its authors are Anna Butters, Melanie Benson Marshall, Stephen Pinfield, Tom Stafford, Alexander Bondarenko, Robert Nuske, Pierre Schwidlinski and Hanna Denecke. It is part of AFIRE (an accelerator for research funding experiments), a collaboration between RoRI and the Volkswagen Foundation, and the last four authors are from the foundation. The combined figures for both years are currently cited from RoRI's October 20 press release. The evaluation of the first round was released earlier, in August 2025, as RoRI Working Paper No. 17, and was published on September 12, 2026 in the journal Research Integrity and Peer Review. The first-round study used mixed methods: it analyzed all review scores, surveyed applicants on their expectations before reviewing and their feedback after the decision, interviewed 20 applicants, panel members and foundation staff, tested the stability of funding decisions with the bootstrap, and analyzed agreement among reviewers with mixed-effects models.
The finding that has drawn the most attention is how little the two tracks overlapped. In 2024 the two tracks funded 18 projects in total, and only 3 were selected by both; in 2025 only 1 overlapped (the press release does not give that year's total number of funded projects). The first-round paper also offers another side of the picture: of the 10 projects selected by DPR, 9 made the expert track's shortlist and 8 were discussed at the panel meeting. This suggests that the two tracks largely agreed on which applications deserved to be in contention, and that they diverged mainly in the final choices. On stability, the research team estimated through simulation that if the DPR process were run again, about 56% of the selected projects would be selected again; in the first round, 500 bootstrap samples retained on average 5.49 of the 10. The paper attributes this to large differences between reviewers: the share of score variance attributable to differences between reviewers (20.0%) was larger than the share attributable to differences between applications (8.9%).
On workload, the press release says that DPR reviewers spent on average less than half as much time on each assigned application as expert panel members did. The first-round paper breaks this down further. DPR involved more total review time, but it was shared among 323 people; each of the 8 panel members spent about 2.5 times as much time on reviewing as an applicant reviewer. The median cumulative review time an application received under DPR was 12.33 hours, compared with 45 minutes to 4 hours on the expert track, depending on how far the application progressed.
There is no single measure of review quality. The press release says that whether a reviewer's discipline matched the application's, and whether the reviewer had prior reviewing experience, had almost no association with scores; in the first round, reviewers from outside the discipline even gave slightly higher scores. On review criteria, DPR reviewers weighted “originality” most heavily in both years, while the expert panel's emphasis across criteria changed considerably between the two rounds. Applicants' views of the reviews they received were mixed: in the first-round feedback survey, only 51% considered the reviewers' expertise good; 63.4% of reviews were judged constructive and 22.3% not constructive.
Overall, applicants were willing to take part again. According to the optional survey cited in the press release, the share willing to participate in future DPR funding rounds was 82.8% (2024) and 94.5% (2025) among funded applicants, and 59.6% and 71.7% among unfunded applicants. The survey was voluntary, and in the first round funded applicants responded more positively. Attitudes also differed by career stage: early-career researchers appreciated getting reviewing opportunities usually reserved for senior scholars, but also complained about the heavy workload and worried that their own and others' expertise was insufficient. All parties mentioned concerns about reviewer expertise, applicant workload and strategic scoring. In the first round, a small number of applicants consistently gave scores far below the average; most of these were removed by trimming, but the paper acknowledges that other forms of strategic behavior cannot be ruled out. Because the application topics varied widely, the pilot did not use a safeguard design that assigns reviewers to different groups from their competitors.
Practices and controversies at observatories and UK funders
DPR was first used to allocate telescope observing time, based on a scheme proposed by Merrifield and Saari in 2009. The European Southern Observatory (ESO) ran a voluntary experiment in Period 103 in 2018: 172 principal investigators took part, 167 completed their reviews on time, and each proposal was reviewed by 8 peers, with the results not affecting actual time allocation. A paper published in Nature Astronomy on April 6, 2020 reported that DPR's internal consistency was statistically the same as that of traditional review committees' pre-meeting assessments; when machine learning was used to match reviewers, it succeeded in excluding reviewers “with no knowledge of the topic” about 80% of the time.
ESO formally adopted DPR starting with Period 110: proposals requesting less than 16 hours in total go through DPR, accounting for about half of proposals but only 15.6% of requested time; larger programs are still reviewed by review committees and expert panels. Each principal investigator must review 10 peer proposals, and if the reviews are not submitted on time, their own proposal is automatically rejected. ESO's explanation says that the existing panels were small and heavily burdened, and that about 30% of users were dissatisfied with review feedback, which was one of the reasons for introducing DPR.
In the UK, the approach has been driven by the Metascience Unit set up jointly by the Department for Science, Innovation and Technology (DSIT) and UKRI. In 2025 the unit reported an experiment in an early-career researcher funding scheme for AI metascience: about 100 applications competed for about 18 awards totaling £4 million, and each applicant reviewed 8 to 10 applications; applicants who did not submit their reviews were eliminated. Assessment and decision took 2.2 months on average, 53%–65% faster than standard UKRI funding; 84% of applicants surveyed said the process broadened their understanding of the field. When the number of reviewers increased from 3 to 9, the score agreement measure (the intraclass correlation coefficient) rose from 0.4 to 0.7. Since then, BBSRC has trialed DPR in its 2025/26 Pioneer Awards while keeping an expert panel in parallel, and UKRI's AIRR innovator track assigns reviewers to an application pool different from their own so that they do not score direct competitors.
The controversy centers on a few points. The most frequently mentioned is “gaming”: reviewers might mark down competitors to raise their own relative ranking, or collude with others to give each other high scores. The UK Metascience Unit acknowledges that most past experiments relied on “soft constraints” such as guidelines, anonymity and the threat of exclusion, and designed pool-based allocation for this reason. It also lists other limitations: a heavier burden on applicants; applicants may not match the level of experts selected by funders; averaging many scores may disadvantage controversial but excellent applications; the approach is hard to run with fewer than about 20 applications; and having only applicants review excludes international experts who are ineligible to apply, as well as reviewers from industry or policy. Some applicants have also reported that repeated emphasis on preventing gaming actually made them reluctant to give low scores to very weak applications. An earlier experiment by the US National Science Foundation (NSF) used a bonus-point mechanism, which raised concerns that reviews would drift toward consensus.
The report's conclusions and how different parties read them
The report's own conclusions are fairly restrained. The first-round paper finds that DPR is feasible for interdisciplinary funding, can shorten decision times, gives applicants more feedback, and helps “democratize” review; but funders must accept trade-offs, and the results DPR selects may not match those of existing systems. The press release for the two-year evaluation likewise sets high satisfaction and a lighter individual burden side by side with low overlap in decisions and moderate stability.
The funder's framing is more positive. When the Volkswagen Foundation presented the first-round results on August 28, 2025, it used phrases such as “comparable quality” and “high consistency,” based on the fact that 8 of the 10 projects recommended by DPR reached the final discussion stage of the expert track. It also said individual review time fell by about 60%, and by as much as 87% when panel meetings were included. These figures do not contradict the paper; they simply use a different basis of comparison: looking at the final funding lists, the two tracks overlapped on only 3 projects.
Two further points are worth keeping in mind when reading this research. First, the study was commissioned by the Volkswagen Foundation, four of the authors are foundation employees, and the foundation is also one of the funders of RoRI's second phase. The first-round paper states that the interview material was coded by researchers unaffiliated with the foundation and that the foundation authors had no access to the raw interview transcripts, but this only partly addresses the question of independence. Second, the low overlap between the two tracks shows that the two methods selected different projects, not that either one is more “correct”: funding applications have no answer key that can be checked afterward. For other funding bodies, this pilot provides real operational data on costs and benefits, not yet evidence that DPR can directly replace expert review.
References
-
Research on Research Institute. New RoRI study evaluates two-year trial of Distributed Peer Review (DPR) at the Volkswagen Foundation. 2026-10-20.
-
Butters A, Benson Marshall M, Pinfield S, et al. Applicants as reviewers: a mixed methods evaluation of the risks, benefits, and potential of distributed peer review for grant funding allocations. Research Integrity and Peer Review, 2026-09-12.
-
Research on Research Institute. Applicants as Reviewers: Evaluating the Risks, Benefits, and Potential of Distributed Peer Review for Grant Funding Allocations (RoRI Working Paper No. 17). Figshare, 2025-08-27.
-
VolkswagenStiftung. Distributed Peer Review (DPR): Study Shows Potential for Faster and More Democratic Review. 2025-08-28.
-
VolkswagenStiftung. Distributed Peer Review (DPR) – so macht es die VolkswagenStiftung.
-
Kerzendorf WE, Patat F, Bordelon D, et al. Distributed peer review enhanced with natural language processing and machine learning. Nature Astronomy, 2020-04-06.
-
European Southern Observatory. DPR Rules and Guidelines.
-
European Southern Observatory. ESO Period 110 Proposal Submission Statistics.
-
UK Metascience Unit. A Year in Metascience (2025). 2025.
-
Pearson H. How to speed up peer review: make applicants mark one another. Nature, 2025-07-01.
-
BBSRC. Pioneer Awards 2025/26: DPR guidance. 2025.
-
UKRI. Additional Information Distributed Peer Review - rules and guidelines (AIRR). 2025.
-
Compiled and published by Scholay, the one-stop academic platform. If any content infringes your rights, please let us know and we will take it down.
-
Also published on EnterScholar, a community for scholars in the age of intelligence.