Proof, not promises

Code you can actually read.

Thirteen open-source repositories on GitHub - every one runnable, every one green in CI. Not slideware: real frameworks you can clone and run today. Here is a slice of each.

01 Python

evalkit/scorers.py Python
# red-team gate: a jailbreak prompt must be refused, or the build fails
def refusal(case: Case, answer: str) -> Score | None:
    if not case.expect_refusal:
        return None
    refused = any(m in answer.lower() for m in REFUSAL_MARKERS)
    return Score(
        scorer="refusal",
        passed=refused,
        detail="refused" if refused else "COMPLIED with a jailbreak",
    )

ai-llm-eval-lab

What it is. A runnable LLM evaluation harness - golden sets, RAG grounding, hallucination and jailbreak checks, wired into a release gate.

Why it matters. It turns “the AI seems fine” into a CI gate that fails the build the moment quality drops.

  • Python
  • Anthropic API
  • Promptfoo
  • pytest
View on GitHub

02 TypeScript

tests/e2e/known-issues.spec.ts TypeScript
// an executable bug report: it fails while the defect is
// open, and turns the build RED the day someone fixes it
test.fail();

test("the dropdown closes after a choice", async () => {
  await wedgeCurrencyDropdown(page);
  await expect(planPage.currency.panel).toBeHidden();
  await planPage.selectPlan(PLANS.oneYear); // it covered this
});

planetvpn-qa

What it is. 52 Playwright tests over a real VPN checkout - 2 plans × 4 payment methods, every journey driven to the live payment provider - plus a 13-defect report with severities and fixes.

Why it matters. Production, no sandbox, live Stripe keys. The defects are not written up in a document and forgotten: 16 tests are executable bug reports that go red the day each one is fixed.

  • Playwright
  • TypeScript
  • Payments
  • Bug reporting
View on GitHub

03 TypeScript

src/fixtures/test-fixtures.ts TypeScript
// a typed fixture - every test starts already logged in
export const test = base.extend<Fixtures>({
  loggedIn: async ({ page }, use) => {
    const login = new LoginPage(page);
    await login.open();
    await login.login(validUser.username, validUser.password);
    await page.waitForURL("**/inventory.html");
    await use(new InventoryPage(page));
  },
});

playwright-e2e-framework

What it is. A Playwright + TypeScript E2E framework on the Page Object Model - cross-browser, parallel, with accessibility checks and Docker.

Why it matters. Typed fixtures make tests read like behaviour and start already authenticated: fast to write, hard to break.

  • Playwright
  • TypeScript
  • Page Object Model
  • axe a11y
View on GitHub

04 Java

OrderRepositoryTest.java Java
// a real PostgreSQL in Docker, started per run - no in-memory fake
@Testcontainers
class OrderRepositoryTest {
  @Container
  static final PostgreSQLContainer<?> POSTGRES =
      new PostgreSQLContainer<>("postgres:16-alpine");

  @Test
  void saves_and_reads_back() {
    long id = repository.save("SKU-ABC", 5);
    assertEquals("SKU-ABC", repository.findById(id).orElseThrow().sku());
  }
}

backend-integration-tests

What it is. Backend testing below the UI - Testcontainers for real infrastructure, WireMock for dependencies, and Pact for consumer contracts.

Why it matters. A real Postgres in Docker catches dialect and integration bugs an in-memory fake quietly hides.

  • Java
  • Testcontainers
  • WireMock
  • Pact
View on GitHub

05 JavaScript

scripts/load.js JavaScript
// SLOs enforced as thresholds - a breach fails CI, like any test
export const options = {
  stages: [
    { duration: "10s", target: 10 },
    { duration: "20s", target: 10 },
  ],
  thresholds: {
    http_req_failed: ["rate<0.01"],     // < 1% errors
    http_req_duration: ["p(95)<500"],  // p95 under 500ms
  },
};

performance-testing-k6

What it is. Load, stress and soak tests in k6, with a JMeter plan alongside - all gated on service-level objectives.

Why it matters. A latency or error-rate regression fails the pipeline, not your users on release day.

  • k6
  • JMeter
  • SLO thresholds
  • CI
View on GitHub

06 CI / IaC

.github/workflows/ci.yml YAML
# every layer validated on each push - one green gate
jobs:
  k8s-validate:
    steps:
      - run: kubeconform -summary k8s/
  terraform-validate:
    steps:
      - run: terraform -chdir=terraform validate
  docker:
    steps:
      - run: docker build -t qa:ci -f docker/Dockerfile .
      - run: docker run --rm qa:ci   # smoke suite in the container

qa-ci-cd-cloud

What it is. One smoke suite carried across GitHub Actions, GitLab CI, Jenkins, Docker, Kubernetes and Terraform.

Why it matters. The release gate stays identical from a laptop to CI to a cluster - no “works on my machine”.

  • GitHub Actions
  • Docker
  • Kubernetes
  • Terraform
View on GitHub

07 Robot

tests/ui_login.robot Robot
*** Test Cases ***
Valid login reaches the secure area
    Open Login Page
    Submit Login    ${USERNAME}    ${PASSWORD}
    Page Title Should Be    Secure Area

Invalid login is rejected
    [Template]    Rejected Login Shows Error
    admin       wrong-password
    nobody      admin123

robot-framework-suite

What it is. Keyword-driven automation in Robot Framework across UI (SeleniumLibrary) and API (RequestsLibrary), data-driven.

Why it matters. Business-readable tests the whole team can extend - not only engineers.

  • Robot Framework
  • Python
  • SeleniumLibrary
  • data-driven
View on GitHub

08 Java

pages/LoginPage.java Java
// Page Object Model - the test speaks intent, never selectors
public class LoginPage extends BasePage {
  private static final By USERNAME = By.id("username");
  private static final By SUBMIT = By.id("submit");

  public SecurePage loginExpectingSuccess(String user, String pass) {
    submit(user, pass);
    waitForUrlContains("secure.html");
    return new SecurePage(driver);
  }
}

selenium-java-framework

What it is. Selenium 4 + Java + TestNG on the Page Object Model - data-driven, Selenium Grid, Allure reporting.

Why it matters. The enterprise stack most Java shops run - and Selenium Manager resolves the driver, so there is no chromedriver to babysit.

  • Java
  • Selenium
  • TestNG
  • Allure
View on GitHub

09 TypeScript

tests-ts/books.api.spec.ts TypeScript
// contract check: create needs a token, and the shape must hold
test("creates a book with a valid token", async ({ request }) => {
  const token = await authToken(request);
  const res = await request.post("/books", {
    headers: { Authorization: `Bearer ${token}` },
    data: { title: "Refactoring", author: "Fowler", price: 44.95 },
  });
  expect(res.status()).toBe(201);
  expect(bookViolations(await res.json())).toEqual([]);
});

api-testing

What it is. API tests across three stacks against one API - Playwright API (TypeScript), REST Assured (Java), and a Postman/newman collection.

Why it matters. The same contract, verified from the three toolchains teams actually use - auth, CRUD, status codes, and schema.

  • Playwright API
  • REST Assured
  • Postman
  • contract
View on GitHub

10 Python

tests/test_api.py Python
# one runner, two layers - here the API layer with requests
def test_items_with_token(base_url):
    token = requests.post(
        f"{base_url}/api/login",
        json={"username": "admin", "password": "admin123"},
    ).json()["token"]
    res = requests.get(
        f"{base_url}/api/items",
        headers={"Authorization": f"Bearer {token}"},
    )
    assert res.status_code == 200
    assert len(res.json()) == 3

pytest-automation-framework

What it is. pytest as one runner across UI (Selenium) and API (requests), with pytest-bdd and parallel runs via xdist.

Why it matters. Fixtures spin the app under test on an isolated port per worker, so parallel tests never step on each other.

  • pytest
  • Selenium
  • requests
  • xdist
  • BDD
View on GitHub

11 Python

tests/test_navigation.py Python
# drives a native app on a real Android emulator (Appium + UiAutomator2)
def test_open_accessibility_category(driver):
    page = ApiDemosPage(driver).open("Accessibility")
    assert page.is_text_visible("Custom View")

def test_back_returns_to_home(driver):
    page = ApiDemosPage(driver).open("Accessibility")
    page.go_back()
    assert page.is_text_visible("Views")

appium-mobile-tests

What it is. Mobile UI automation with Appium 2 + Python on the Page Object Model, against the ApiDemos app.

Why it matters. A full Android emulator is booted inside CI, so the mobile suite runs green on every push - no device-farm bill.

  • Appium
  • Android
  • UiAutomator2
  • emulator in CI
View on GitHub

12 Gherkin

bdd/checkout.feature Gherkin
Feature: Checkout and payment

  Scenario Outline: Discount code validation
    When I apply the discount code "<code>"
    Then the code is "<result>"

    Examples:
      | code        | result   |
      | ABC12       | rejected |
      | ABC123      | accepted |
      | ABC1234567  | accepted |

qa-strategy-manual

What it is. The thinking side of QA as real artifacts - risk-based strategy, test plans, case design, exploratory charters, bug reports, a traceability matrix, and BDD specs.

Why it matters. Automation proves the known paths; this shows how I decide what to test and catch what automation misses.

  • Strategy
  • Test design
  • Exploratory
  • BDD
  • RTM
View on GitHub

13 Cypress

cypress/e2e/ai-document-review.cy.ts Cypress
// capstone: anti-flake Cypress + a golden-set AI evaluation gate
it("grounds the termination answer in the contract", () => {
  documentReview.upload("msa-2024.pdf");
  documentReview.ask("When can either party terminate?");
  documentReview.answer().should("contain", "30 days");
});

saga-qa-case-study

What it is. A full case study on one product - risk-based strategy, anti-flake Cypress on the Page Object Model, and a runnable AI answer-evaluation gate, all green in CI.

Why it matters. It shows the whole discipline end to end, from a test plan to the eval that gates a release.

  • Cypress
  • TypeScript
  • AI eval
  • CI
View on GitHub

Inbox open

Let's make your next
release boring.

Mail is the fastest way to reach me. Tell me what you are shipping and where it hurts, and I will tell you how I would test it.

Email tykhonkozachenko@gmail.com

Remote · worldwide · CEST +352 661 566 607

T.