Eight months inside a GitHub Actions merge queue teaches you something that feature comparison matrices never will. A framework can ship fifty metrics, gorgeous dashboards, and citations from respected research labs. If it blocks your deploy because a "vibe check" score drifted from 0.72 to 0.68 against identical code, it is worse than useless. It becomes an active threat to your shipping velocity.
That is the filter most LLM evaluation roundups miss. They count capabilities. They rarely ask the only question that matters in a merge queue: does this check pass and fail the exact same way every single time it runs?
I learned this by doing the uncomfortable work. I wired six open-source LLM eval frameworks into a real CI pipeline. They ran against live production pull requests for eight months. Two earned the right to remain as gatekeepers. The rest were demoted to advisory dashboards, moved to nightly jobs, or removed entirely. The lesson was sharp and expensive: deterministic structure beats probabilistic quality when you are guarding the main branch.
The Real Job of a Merge Gate
A CI gate is not a research environment. It is a bouncer. Its entire purpose is to look at a specific change and answer yes or no. Yes, this PR can join the main branch. No, it cannot. That answer needs to arrive in seconds, cost pennies, and never flip retroactively. If you rerun the same pipeline against the same commit on a quiet Tuesday and a frantic Friday, the outcome must be identical.
This is where most LLM eval frameworks stumble. They are built by data scientists for data scientists. They optimize for insight, exploration, and nuanced scoring. A merge queue optimizes for binary decisions, speed, and zero flakiness. Those two goals only partially overlap.
Why LLM-as-Judge Breaks the Queue
The tools that failed in my test shared a single design sin: they relied too heavily on LLM-as-judge calls as the primary gate mechanism.
An LLM-as-judge prompt asks a model to score an output on a scale of one to ten, or to pick the better of two responses, or to rate factual correctness. The approach is powerful for understanding quality trends. It is poison for a blocking CI check. The same input can produce different scores on different days because temperature, model versioning, and prompt formatting all introduce noise. When that score is tied to a hard threshold and a hard exit code, your queue blocks on ghosts.
The failures cascade quickly. A nondeterministic check creates queue backups. Engineers learn to retry until the number lands favorably, which trains the team to ignore red builds. Token costs pile up because every retry burns more API credits. Worst of all, the signal becomes meaningless. A red build should mean "you introduced a bug." If it means "the judge model woke up picky today," trust erodes.
What the Survivors Do Differently
Promptfoo and DeepEval survived because they treat deterministic checks as first-class citizens and LLM judge scores as secondary, non-blocking signals. They understand that a gate needs an exit code, not a floating-point number with an opinion.
Promptfoo, released under the MIT license, is built for the command line. It runs assertions like regex matches, JSON schema validation, contains checks, and exact string comparisons. These are not fancy. They are glorified grep and jq commands. That is exactly why they work in CI. A regex either matches or it does not. A JSON schema either validates or it throws. Promptfoo returns standard Unix exit codes, so GitHub Actions natively understands when to stop a merge. It is language-agnostic because it operates as a CLI tool. You do not need to install a Python ecosystem inside a Node.js service repo just to validate outputs.
DeepEval, licensed under Apache 2.0, is the choice for Python teams. It integrates like pytest. You write tests in familiar syntax, and a failure blocks the suite naturally. DeepEval offers a huge catalog of metrics, but the critical detail is that you must use them carefully. Lean on deterministic or heuristic metrics for gates. If you pull in G-Eval or other judge-based scorers, wrap them in non-blocking report generators rather than hard asserts. When used this way, DeepEval gives you the ergonomics of a testing framework without the flakiness of a research notebook.
Where the Other Four Fit
The four frameworks that did not survive as gates still have value. They simply belong elsewhere in your toolchain.
Future AGI (Apache 2.0) ships over fifty metrics and targets teams building custom SDKs. The metrics are thorough. The problem is that the tool expects you to write your own harness to drive it in a CI queue. In a research context, that is a reasonable trade. In a merge queue, every layer of custom wiring is a new source of instability. It is a capable evaluation engine, but not a ready gatekeeper.
RAGAS (Apache 2.0) excels at measuring retrieval-augmented generation quality. Its faithfulness and answer relevance metrics are genuinely useful for understanding how a knowledge base performs over time. Unfortunately, those metrics lean heavily on LLM judges. They are excellent for a nightly quality job that posts trends to Slack. They are poor bouncers for a pull request. Move RAGAS to your scheduled analysis pipeline, not your merge blockers.
Arize Phoenix carries the Elastic License 2.0 and sits at a different intersection entirely. It connects distributed tracing with evaluation, giving you observability into why a model behaved a certain way. You want this when you are debugging a production incident or tracing a hallucination back to a bad retrieval chunk. You do not want a tracing tool deciding whether a junior developer’s feature branch can ship. Its architecture is built for insight, not binary gates.
MLflow Evaluate (Apache 2.0) inherits its pedigree from experiment tracking. It is heavy. Pulling it into a lean CI image adds startup time and dependencies that slow down every single job. If you absolutely must use it inside a pipeline, stick to its heuristic metrics for structural checks. Even then, you are fighting the framework’s fundamental design. MLflow wants to log runs and compare experiments across weeks. A merge queue wants a verdict in under a minute.
Practical Rules for Gating
If you take nothing else from this experiment, take these three rules.
First, gate structure, not vibe. You can enforce that an output is valid JSON. You can enforce that it contains required keys. You can enforce that a classification label belongs to an allowed enum. These checks are fast, cheap, and deterministic. You cannot reliably enforce that a summary is "friendly" or that a rewrite is "creative." Those qualities belong in human review or periodic batch evaluation, not in automated gates.
Second, if a score moves on unchanged input, demote it immediately. Run your evaluation suite twice against the exact same artifact. If any metric flips from pass to fail, it has lost its right to block a merge. Promote it to an advisory dashboard where variance is expected and tolerable.
Third, respect the exit code. A pretty HTML report with a red banner does not stop a merge. A nonzero exit code does. Your evaluation tool must speak the native language of your CI platform. Standard out is for humans. Exit codes are for machines.
The Takeaway
We are still early in figuring out how to test LLM-powered applications. The temptation is to treat evaluation like a human grading rubric: nuanced, contextual, and slightly subjective. That works in a research paper. It collapses in a merge queue.
After eight months of production traffic, my pipeline now runs Promptfoo for structural and schema assertions across services, and DeepEval for Python-side behavioral checks that map cleanly to pass-fail conditions. Everything else reports to nightly dashboards. The queue is stable. The signal is clean. The team trusts a red build again.
You do not need more metrics at your gate. You need fewer metrics that tell the truth every single time.
Based on original testing and write-up shared on Dev.to. For more discussions on building reliable AI systems, join the GyaanSetu community on Telegram.