A PR Review Gate That Fails Closed: Free Models in CI

Most PR reviews run on trust. A reviewer says LGTM. The code merges. Nobody tests the reviewer.

AI-assisted review changes the cost, not the risk. Free models can review every PR. But an unchecked reviewer is still unchecked. The fix is a gate that fails closed.

This article builds a reproducible PR review gate. It extracts the diff. It runs the tests. It asks a free model to classify findings. It blocks the merge on blocking findings. Every step is scriptable. Every step costs $0.

The workflow fits maintainers of small open-source projects. It fits teams that want a second pass without a second human. It does not replace a human reviewer. It catches the obvious failures first.

The Problem: Reviewers Are Untested

Teams review the code. They rarely review the reviewer. A reviewer misses a bug. The bug ships. The author takes the blame.

AI agents made this worse. Developers now review more code than they write. The reviewer role expanded. The verification step did not. The trend is real: AI promoted every developer to reviewer. Nobody tested the reviewer.

Free models change the economics. A review pass costs tokens, not salaries. But free models hallucinate. They praise broken code. They block good code. The gate must be deterministic. The gate must fail closed.

The Workflow: Four Steps, One Exit Code

The gate runs four steps. Each step writes a file. Each file feeds the next step.

1. Extract the diff.
2. Run the test suite.
3. Generate a structured review.
4. Classify findings and decide.

The exit code is the contract. Zero means merge. Non-zero means stop. A CI job reads the exit code. A human reads the report.

Step 1: Extract the Diff

A review starts with the change. Not the PR description. Not the commit messages. The diff.

#!/usr/bin/env bash
set -euo pipefail

PR_NUMBER="${1:?usage: pr-gate.sh <pr-number>}"
BASE="${2:-origin/main}"

gh pr diff "$PR_NUMBER" > /tmp/pr.diff
git diff --stat "$BASE"...HEAD > /tmp/pr.stats
wc -l /tmp/pr.diff

The gh CLI wraps the GitHub API. The diff lands in /tmp/pr.diff. The stats file shows the blast radius. A 2,000-line PR should trigger a human review. The gate can enforce that rule too.

Step 2: Run the Test Suite

A review without tests is speculation. Run the suite first. Capture the output. The output becomes context for the model.

if [ -f "pytest.ini" ] || [ -d "tests" ]; then
  python -m pytest -q > /tmp/test.log 2>&1 || true
fi

The || true is deliberate. The gate does not fail here. The test log becomes evidence. The model reads the failures. It can spot weakened tests. It can spot skipped suites. It can spot deleted assertions.

Step 3: Generate a Structured Review

Raw diffs confuse models. Structured prompts get structured answers. Build a prompt from the diff, the stats, and the test log.

cat > /tmp/review_prompt.md <<'EOF'
You are a senior reviewer for an open-source project.
Classify every finding as BLOCK, ASK, or NIT.
- BLOCK: bug, security issue, breaking change, weakened test
- ASK: missing context, unclear intent, design question
- NIT: style, naming, formatting
Return one finding per line. Start with the class in uppercase.

Diff:
EOF

cat /tmp/pr.diff >> /tmp/review_prompt.md
echo "Test log:" >> /tmp/review_prompt.md
cat /tmp/test.log >> /tmp/review_prompt.md

The prompt enforces a fixed format. Fixed formats are parseable. Parseable output becomes a gate.

This is where free model access matters. MonkeyCode is an open-source project. It provides free model access and a free server option. The free 10M token allowance covers a meaningful review volume. Quotas and terms change. Check the current documentation before relying on them.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The command below is a placeholder. Replace it with the actual CLI or API call from the project's documentation.

monkeycode-review --prompt /tmp/review_prompt.md > /tmp/review.out

Step 4: Classify and Decide

The output drives the exit code. BLOCK findings fail the gate. ASK findings go to a human. NIT findings never block.

if grep -q "BLOCK" /tmp/review.out; then
  echo 'Gate failed: blocking findings present.'
  cat /tmp/review.out
  exit 1
fi

echo 'Gate passed: no blocking findings.'
exit 0

The logic is simple. The policy is the hard part. The decision table defines the policy.

The Decision Table

Two rules matter. First, BLOCK always fails. Second, NIT never fails. Teams that let NIT block merges train the model to stay silent. Stay strict on BLOCK. Stay silent on NIT.

Benchmark the Gate Before You Trust It

A gate needs calibration. The model scored well in demos. The harness scored well in benchmarks. The gate still needs a local test.

Create a throwaway PR. Insert three known bugs. One security issue. One weakened test. One contract break. Run the gate. Check the classes.

The gate passes calibration when all three are BLOCK. It fails when any seed becomes ASK or NIT. Re-run the calibration after every model or prompt change.

When This Gate Lies

Free models have limits. The gate inherits them.

• Context windows truncate large diffs. The middle of a 500-line diff disappears.
• Models hallucinate file paths. Verify every referenced path.
• Output is non-deterministic. The same diff can produce different classes.
• Test logs can leak secrets. Redact before sending.
• The model cannot run the code. It reasons about the diff only.

The gate is a filter, not a verdict. It catches obvious failures. It does not catch design rot. It does not catch subtle race conditions. It does not catch business logic errors.

Who Should Not Use This

This gate is not for everyone.

• Teams under compliance review need human sign-off. A model cannot sign.
• Repos with secrets in test output need redaction first.
• Tiny hobby PRs do not need a gate. The overhead beats the value.
• Projects with flaky tests will see false BLOCKs. Fix the flake first.

Use the gate where review volume is high and budget is zero. Use it as a first pass. Keep a human as the final gate.

The Full Script

The complete script is below. It is a starting point, not a product. Adjust the prompt. Adjust the policy. Run it on a real PR before trusting it.

#!/usr/bin/env bash
# pr-gate.sh — a review gate that fails closed
# Usage: ./pr-gate.sh <pr-number> [base-branch]
set -euo pipefail

PR_NUMBER="${1:?usage: pr-gate.sh <pr-number>}"
BASE="${2:-origin/main}"

echo '[1/4] Extracting diff'
gh pr diff "$PR_NUMBER" > /tmp/pr.diff
git diff --stat "$BASE"...HEAD > /tmp/pr.stats
wc -l /tmp/pr.diff

echo '[2/4] Running tests'
if [ -f "pytest.ini" ] || [ -d "tests" ]; then
  python -m pytest -q > /tmp/test.log 2>&1 || true
else
  echo 'no test suite found' > /tmp/test.log
fi

echo '[3/4] Generating review'
cat > /tmp/review_prompt.md <<'EOF'
You are a senior reviewer for an open-source project.
Classify every finding as BLOCK, ASK, or NIT.
- BLOCK: bug, security issue, breaking change, weakened test
- ASK: missing context, unclear intent, design question
- NIT: style, naming, formatting
Return one finding per line. Start with the class in uppercase.
Diff:
EOF
cat /tmp/pr.diff >> /tmp/review_prompt.md
echo 'Test log:' >> /tmp/review_prompt.md
cat /tmp/test.log >> /tmp/review_prompt.md

# Placeholder: replace with the actual CLI or API call
# from your provider's documentation.
monkeycode-review --prompt /tmp/review_prompt.md > /tmp/review.out

echo '[4/4] Deciding'
if grep -q "BLOCK" /tmp/review.out; then
  echo 'Gate failed: blocking findings present.'
  cat /tmp/review.out
  exit 1
fi

echo 'Gate passed: no blocking findings.'
exit 0

The Next Step

Run the gate on your last merged PR. See what it catches. Then decide what your reviewers need. The free tier is enough to start. The reviewer deserves testing too.