Recognizing Excellence in ACL Rolling Review - Celebrating Outstanding Peer Reviewers

About ARR Great Reviewers

Motivation #

Most reviews get written, submitted, and forgotten. But across all ARR cycles, thousands of reviews were so exceptional that Area Chairs flagged them as "Great Reviews." These aren't participation trophies -- they're the reviews that saved papers from rejection, caught fatal flaws others missed, or suggested the key experiment that made everything click. Yet these recognitions end up buried at the bottom of stats pages while the reviewers who wrote them remain invisible.

⭐ What Makes a Great Review? ⭐

"We would like to highlight the importance of quality reviews in the ACL rolling review process. If a review is deemed strong, decisive, helpful, and well-written by Area Chairs, it will be marked as a Great Review. Please note that a review not marked as great does not imply that it is poor; rather, it may simply not meet the exceptional standards set for this designation. Thank you for your continued contributions to the review process. Your efforts are invaluable in maintaining the quality and integrity of our evaluations."

— ARR Stats, April 2024

ARR Great Reviewers brings together all the "Great Review" recognitions from across ARR's cycle pages into one place. It allows reviewers to see their cumulative impact -- how many authors they've helped, how their reviewing has evolved, and where they stand among peers. It's a small way to acknowledge those who pour effort into making others' research better.

The entire analysis is transparent and open-source, with code available at GitHub, covering all ARR cycles where "Great Review" data is available.

Methodology #

This section explains the data collection and processing for ARR great reviewers.

Data Collection #

The dataset comes from the ACL Rolling Review public statistics endpoints. We fetch JSON files for each iteration listed in config/data_sources.toml. Each entry includes reviewer name, institution, counts of reviews and recognized reviews, and the percentage of recognized reviews.

Downloaded files are validated against a reference schema and stored under data/raw/. We then concatenate these files and clean numeric fields to build a unified DataFrame. Aggregations produce metrics for top reviewers and institutions which are exported as JSON under data/metrics/. These metrics feed the static site visualisations.

Ranking and Tie-Breaking #

When ranking reviewers and institutions, ties are resolved using the following hierarchy:

For Recognition Count Rankings: #
  1. Primary: Total number of recognized reviews (descending)
  2. Tie-breaker 1: Recognition rate (recognized/total reviews) (descending)
  3. Tie-breaker 2: Total number of reviews (descending)
For Recognition Rate Rankings: #
  1. Primary: Recognition percentage (descending)
  2. Tie-breaker 1: Total number of recognized reviews (descending)
  3. Tie-breaker 2: Total number of reviews (descending)

This ensures consistent and fair rankings where reviewers with identical primary metrics are ordered by meaningful secondary criteria that reflect their overall contribution and review volume.

Data Quality #

We match reviewers across cycles using their OpenReview IDs to handle name changes and typos. When someone changes institutions, we use their current affiliation but count all their past reviews.

We only rank reviewers with at least 5 total reviews and institutions with at least 3 reviewers. This reduces noise from small samples.

What This Doesn't Capture #

  • Review quality beyond ARR's "recognition" metric
  • Reviewers who don't use ARR
  • Historical institutional affiliations (we use current ones)
  • Differences between research areas or reviewer experience levels

Privacy #

All data comes from public ARR statistics. Reviewers can ask ARR organizers to opt out of future rankings.

Limitations #

ARR defines "recognized" reviews internally -- we don't control this criteria. Our data only goes back to April 2024 and only covers ARR participants.

Recognition doesn't equal review quality. A helpful review might not be "recognized" and vice versa.

Reproducibility #

Raw data files and processing code are available on GitHub. Data schemas are validated on download, and the entire analysis pipeline is open-source and reproducible.

Note: This project is not affiliated with ARR in any way.