008 — Evaluate Qodo Merge / PR-Agent as a CodeRabbit Replacement#

Context#

.github/workflows/pr-compliance-gate.yml uses coderabbitai/openai-pr-reviewer@latest as the AI-compliance reviewer (job ai-compliance-review). Two operational pains have surfaced:

The action defaulted to OpenAI model names (gpt-3.5-turbo / gpt-4) and silently produced "nothing obtained from openai" against Azure AI Foundry. Fix required adding OPENAI_API_VERSION plus openai_light_model / openai_heavy_model set to the Azure deployment name — undocumented enough to cost real time.
The coderabbitai/openai-pr-reviewer repo is essentially unmaintained (last release 2023, no Azure-specific docs); the active CodeRabbit product is now a paid GitHub App.

Question: is PR-Agent / Qodo Merge a better fit for our Azure AI Foundry + GDPR/NIS2/CRA compliance review?

Findings#

Dimension	`coderabbitai/openai-pr-reviewer@latest` (current)	PR-Agent (`qodo-ai/pr-agent`)
License	MIT	Apache 2.0 (briefly AGPL, reverted April 2026)
Maintenance	Unmaintained OSS shim (CodeRabbit moved to commercial app)	Active; community-owned `The-PR-Agent` org from April 2026
Hosting	Self-hosted in GitHub Actions	Self-hosted action or Qodo Merge SaaS GitHub App
Azure OpenAI support	Works but undocumented (env: `OPENAI_API_VERSION`; inputs: `openai_*_model` = deployment name)	First-class. Env: `OPENAI.API_BASE`, `OPENAI.API_TYPE=azure`, `OPENAI.API_VERSION`, `OPENAI.DEPLOYMENT_ID`
Custom system prompt	`with: system_message:` (one block)	`.pr_agent.toml` `[pr_reviewer] extra_instructions=…` plus per-tool prompts (`/review`, `/improve`, `/describe`, `/ask`)
Commands available	Single auto-review	`/describe`, `/review`, `/improve`, `/ask`, `/test`, `/update_changelog`, `/help` — comment-driven
Static analysis	None — pure LLM	Pure LLM in OSS; SaaS adds static analysis to reduce hallucinations
BYOK	Yes (Azure / OpenAI / Anthropic)	Yes (Azure / OpenAI / Anthropic / Gemini / local)
Required secrets in our setup	`AZURE_OPENAI_API_KEY`, `AZURE_OPENAI_API_VERSION`, `AZURE_OPENAI_ENDPOINT`, `AZURE_OPENAI_DEPLOYMENT`	Same four (different names, same values)

Behavioural fit for our compliance prompt#

Our current system_message is a "European Compliance Officer auditing for GDPR / NIS2 / CRA". PR-Agent's extra_instructions slot accepts the same persona text and also lets us target only the /review tool, which is what we want — we don't need PR description rewrites or auto-generated tests on a regulated repo. CodeRabbit's single block applies to everything the action produces.

Cost / privacy#

Both run inside our GitHub Actions runners hitting our Azure deployment in Sweden Central → equivalent privacy posture; data stays in our tenant.
Both are zero-cost beyond Azure token spend when used as a GitHub Action.
Qodo Merge SaaS is a separate product (managed app, paid) — explicitly out of scope here; we want the OSS action.

Recommendation#

Migrate the ai-compliance-review job to qodo-ai/pr-agent (pinned tag).

Why: - Active maintenance vs. archived shim — the recent Azure breakage we just patched is a symptom of the latter. - Azure OpenAI is a documented first-class path, not an undocumented hack. - Comment-driven tools (/review, /ask) give reviewers a way to interrogate findings without re-running CI — useful on a compliance gate where "why did you flag this" matters. - Configuration moves from inline YAML strings into a checked-in .pr_agent.toml, which composes better with our other gate config (Semgrep, Checkov, PSRule).

Risks / open items: - Pin to a specific commit SHA or release tag, not @main, to satisfy supply-chain rules (NIS2). Track the upcoming qodo-ai/pr-agent → The-PR-Agent/pr-agent rename — pin and uses: line will need updating once the repo moves. - The cost-saving gate (if: github.event.action == 'opened' || ...) must be preserved or PR-Agent will run on every push.

Critical files to modify#

File	Change
`.github/workflows/pr-compliance-gate.yml`	Replace the `ai-compliance-review` job step. Keep the `if: opened/reopened` guard.
`.pr_agent.toml` (new, repo root)	Custom `extra_instructions` carrying the GDPR/NIS2/CRA persona. Disable tools we don't want (`/improve`, auto-`/describe`).
`.gitleaks.toml`	No change — `.pr_agent.toml` is non-sensitive.
`CHANGELOG.md`	Note the swap under `[Unreleased]`.

Replacement step (sketch)#

ai-compliance-review:
  runs-on: ubuntu-latest
  if: github.event.action == 'opened' || github.event.action == 'reopened'
  permissions:
    issues: write
    pull-requests: write
    contents: read
  steps:
    - uses: actions/checkout@v4
    - name: PR-Agent compliance review
      uses: qodo-ai/pr-agent@v0.29  # pin; revisit after The-PR-Agent rename
      env:
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        OPENAI.API_TYPE: "azure"
        OPENAI.API_BASE: ${{ secrets.AZURE_OPENAI_ENDPOINT }}
        OPENAI.API_VERSION: ${{ secrets.AZURE_OPENAI_API_VERSION }}
        OPENAI.DEPLOYMENT_ID: ${{ secrets.AZURE_OPENAI_DEPLOYMENT }}
        OPENAI.KEY: ${{ secrets.AZURE_OPENAI_API_KEY }}
        CONFIG.MODEL: "azure/${{ secrets.AZURE_OPENAI_DEPLOYMENT }}"
        GITHUB_ACTION_CONFIG.AUTO_REVIEW: "true"
        GITHUB_ACTION_CONFIG.AUTO_DESCRIBE: "false"
        GITHUB_ACTION_CONFIG.AUTO_IMPROVE: "false"

`.pr_agent.toml` (sketch)#

[pr_reviewer]
extra_instructions = """
You are a European Compliance Officer auditing a .NET/C# system for BenefitManager.
Review this PR against:
1. GDPR: Flag PII handling, lack of pseudonymization, or unencrypted sensitive fields.
2. NIS2: Identify supply chain risks or hardcoded credentials.
3. CRA: Ensure input validation for all external API parameters.
Only provide actionable security feedback. Cite line numbers.
"""
require_tests_review = false
require_security_review = true
num_code_suggestions = 0  # Semgrep/CodeQL handle code suggestions

Migration steps#

Confirm the Azure AZURE_OPENAI_DEPLOYMENT and AZURE_OPENAI_API_VERSION repo secrets are set (already required by current setup).
Add .pr_agent.toml at repo root with the compliance prompt.
Replace the ai-compliance-review job step in .github/workflows/pr-compliance-gate.yml per the sketch above; keep the if: guard and the permissions: block.
Pin to a specific PR-Agent release tag (current: v0.29.x) — never @main in a compliance gate.
Delete the now-unused coderabbitai/openai-pr-reviewer step.
Open a draft PR; observe PR-Agent posts a single review comment.
Comment /ask Are there any GDPR PII fields in this diff? on the PR — verify interactive mode reaches the Azure deployment.
Update CHANGELOG.md [Unreleased] → "Changed: AI compliance reviewer migrated from coderabbitai/openai-pr-reviewer to qodo-ai/pr-agent (active maintenance, native Azure OpenAI support)."

Verification#

Smoke test: open a throwaway PR with a deliberately PII-leaking diff (e.g., logging a bare email address). PR-Agent should flag it with a citation to the line and reference GDPR.
Cost guard: confirm the if: opened/reopened filter still suppresses runs on synchronize events (push to PR branch).
Auth: confirm Azure deployment receives the request (Azure portal → AI Foundry → deployment metrics show a hit during the run).
No secret leakage: gitleaks pre-commit + repo scan stay clean (the toml has no secrets).
Rollback: if PR-Agent fails or output regresses, revert the workflow step in a single commit; no schema/state changes required.

Out of scope#

Adopting the Qodo Merge SaaS GitHub App (separate product, paid, requires routing source through Qodo's tenant — fails our "data stays in our Azure tenant" rule).
Replacing Semgrep / CodeQL / Checkov / PSRule. PR-Agent is positioned as a reviewer, not a SAST tool; we keep deterministic scanners for the non-LLM gates.
Self-hosting a PR-Agent server (overkill — the GitHub Action mode is sufficient).