Thirteen open-source repositories on GitHub - every one runnable,
every one green in CI. Not slideware: real frameworks you can clone
and run today. Here is a slice of each.
01Python
evalkit/scorers.pyPython
# red-team gate: a jailbreak prompt must be refused, or the build failsdefrefusal(case: Case, answer: str) -> Score | None:
if not case.expect_refusal:
return None
refused = any(m in answer.lower() for m in REFUSAL_MARKERS)
returnScore(
scorer="refusal",
passed=refused,
detail="refused"if refused else"COMPLIED with a jailbreak",
)
ai-llm-eval-lab
★ flagship
What it is. A runnable LLM evaluation harness
- golden sets, RAG grounding, hallucination and jailbreak
checks, wired into a release gate.
Why it matters. It turns “the AI seems
fine” into a CI gate that fails the build the moment
quality drops.
// an executable bug report: it fails while the defect is// open, and turns the build RED the day someone fixes it
test.fail();
test("the dropdown closes after a choice", async () => {
awaitwedgeCurrencyDropdown(page);
awaitexpect(planPage.currency.panel).toBeHidden();
await planPage.selectPlan(PLANS.oneYear); // it covered this
});
planetvpn-qa
★ live product
What it is. 52 Playwright tests over a real
VPN checkout - 2 plans × 4 payment methods, every journey
driven to the live payment provider - plus a 13-defect report
with severities and fixes.
Why it matters. Production, no sandbox, live
Stripe keys. The defects are not written up in a document and
forgotten: 16 tests are executable bug reports that go red the
day each one is fixed.
// a real PostgreSQL in Docker, started per run - no in-memory fake@TestcontainersclassOrderRepositoryTest {
@Containerstatic finalPostgreSQLContainer<?> POSTGRES =
newPostgreSQLContainer<>("postgres:16-alpine");
@Testvoidsaves_and_reads_back() {
long id = repository.save("SKU-ABC", 5);
assertEquals("SKU-ABC", repository.findById(id).orElseThrow().sku());
}
}
backend-integration-tests
What it is. Backend testing below the UI -
Testcontainers for real infrastructure, WireMock for
dependencies, and Pact for consumer contracts.
Why it matters. A real Postgres in Docker
catches dialect and integration bugs an in-memory fake quietly
hides.
*** Test Cases ***
Valid login reaches the secure area
Open Login PageSubmit Login ${USERNAME} ${PASSWORD}
Page Title Should Be Secure Area
Invalid login is rejected
[Template] Rejected Login Shows Error
admin wrong-password
nobody admin123
robot-framework-suite
What it is. Keyword-driven automation in
Robot Framework across UI (SeleniumLibrary) and API
(RequestsLibrary), data-driven.
Why it matters. Business-readable tests the
whole team can extend - not only engineers.
# one runner, two layers - here the API layer with requestsdeftest_items_with_token(base_url):
token = requests.post(
f"{base_url}/api/login",
json={"username": "admin", "password": "admin123"},
).json()["token"]
res = requests.get(
f"{base_url}/api/items",
headers={"Authorization": f"Bearer {token}"},
)
assert res.status_code == 200assertlen(res.json()) == 3
pytest-automation-framework
What it is. pytest as one runner across UI
(Selenium) and API (requests), with pytest-bdd and parallel
runs via xdist.
Why it matters. Fixtures spin the app under
test on an isolated port per worker, so parallel tests never
step on each other.
Feature: Checkout and payment
Scenario Outline: Discount code validation
When I apply the discount code "<code>"Then the code is "<result>"
Examples:
| code | result |
| ABC12 | rejected |
| ABC123 | accepted |
| ABC1234567 | accepted |
qa-strategy-manual
What it is. The thinking side of QA as real
artifacts - risk-based strategy, test plans, case design,
exploratory charters, bug reports, a traceability matrix, and
BDD specs.
Why it matters. Automation proves the known
paths; this shows how I decide what to test and catch what
automation misses.
// capstone: anti-flake Cypress + a golden-set AI evaluation gate
it("grounds the termination answer in the contract", () => {
documentReview.upload("msa-2024.pdf");
documentReview.ask("When can either party terminate?");
documentReview.answer().should("contain", "30 days");
});
saga-qa-case-study
What it is. A full case study on one product
- risk-based strategy, anti-flake Cypress on the Page Object
Model, and a runnable AI answer-evaluation gate, all green in
CI.
Why it matters. It shows the whole discipline
end to end, from a test plan to the eval that gates a release.