LLM Evals in CI/CD: How to Test AI Responses
Unit tests miss probabilistic outputs. Golden datasets, automated judges, and regression gates.
Topic guide
A testing strategy that gives you confidence to ship: testing pyramid, TDD, E2E tests with Playwright, mutation testing, and quality in code review.
3 topics · 10 articles
The balance between unit, integration, and E2E tests.
Unit tests miss probabilistic outputs. Golden datasets, automated judges, and regression gates.
A ramp scenario, a p95 threshold that fails the build, and k6 in CI. Load testing turns latency into a number the team defends and investigates with traces.
Unit test coverage does not walk the customer path. Playwright covers login, checkout, and error states with fixtures, Page Objects, and CI. Examples ready to copy.
The red-green-refactor cycle day to day.
Coverage counts executed lines, not the bugs your suite catches. Behavior over implementation, the cases worth covering, and three numbers that measure confidence.
Quality gates, pull requests, and practices that keep quality up.
Reviewing shared-staging screenshots does not scale. One environment per PR with synthetic data transforms review quality.
Generics, conditional types, and mapped types solve specific problems of code reuse and type safety.
Most APIs handle errors inconsistently. A well-defined pattern reduces integration bugs and makes debugging easier.
Merge writes a commit with two parents and keeps history as it happened. Rebase replays your commits on top of main and leaves a straight timeline. The rules for each, plus a PR flow that uses both.
The five principles, the code pattern that violates each one, and the version that fixes it. Where SOLID pays off in a project and where it turns into useless layering.
A Staff Engineer review exists to develop the team. Four anti-patterns that weaken it, the comment format that transfers reasoning, and when to leave the keyboard and talk.
It is the testing pyramid: most of the suite is fast unit tests, and a few slow E2E tests cover the critical flows end to end.
TDD (before) is ideal because it shapes the design. But tests written afterwards are far better than no tests.
Find the root cause: timing issues, external dependencies, or state leaking between tests. Isolate the test and fix the cause instead of adding retries.
A tool deliberately changes your code to see whether the tests fail. If no test fails, that part of the code is not really covered.
Between 70% and 80% is a good target. High coverage helps but does not guarantee quality: what the tests check is what matters.