bench get-app https://github.com/frappe/release_tests

[![Listed on Frappe Gems](https://frappegems.com/api/method/frappe_gems.seo.badge?app=frappe%2Frelease_tests)](https://frappegems.com/gems/apps/frappe/release_tests)

release_tests

External HTTP release-test harness for Frappe sites. Given a running site, it logs in over the REST API and exercises a defined set of flows per installed app, adapting to whatever version is present (develop / v16 / v15), then reports pass/fail per app and step.

It is a standalone client — not a Frappe app installed on the site. It only speaks HTTP, so the same harness runs against a local bench site today and a remote / press-deployed site tomorrow with no change.

Why

Frappe ships many app releases monthly, and regressions — including cross-app breakage (e.g. CRM or India-compliance changes breaking ERPNext) — are often catchable by simple release flows: log in, create master data, push one transaction through. This tool automates exactly that across versions.

Install

uv venv && uv pip install -e ".[dev]"   # dev extras add pytest/responses/ruff

Configure targets

Copy the example and fill in credentials (the real file is gitignored):

cp targets.example.toml targets.toml
export RELEASE_ADMIN_PW='...'             # values like "$RELEASE_ADMIN_PW" resolve from env

Each [[target]] is one site: a URL plus either login (username/password) or token (API key/secret) auth. The client sends an explicit Host header so local multi-tenant bench sites route correctly.

Use

release-tests list-suites                       # registered suites
release-tests detect  --target v16-local        # installed apps + versions
release-tests run     --target v16-local        # run all applicable suites
release-tests run     --target all --suite erpnext

run prints a per-target table, writes a JSON report to results/, and exits non-zero if any step failed (CI-friendly). Suites whose app isn't installed are skipped automatically.

Suites

Suite	App	Status
core_frappe	frappe	login, whoami, ToDo CRUD round-trip
erpnext (+ erpnext_*)	erpnext	masters → submit Sales Invoice → Payment Entry → settled; selling/buying/stock/accounts/service/serialised/batched/manufacturing
hrms (+ hrms_*)	hrms	Employee, org, leave, expense flows
crm	crm	create Lead → Deal → Customer
india_compliance	india_compliance	GST masters → GST Sales Invoice → best-effort e-invoice
insights	insights (+erpnext)	Workbook → query on Sales Invoice → charts → dashboard
helpdesk	helpdesk	create Agent → open Ticket
webshop	webshop	publish Website Item → top-bar link → item page loads
builder	builder	author page (hero + cards) → publish → page loads
learning, ksa_compliance	respective	stub — gated reachability probe; fill in real flows

helpdesk, webshop, and builder are written against each app's schema but not yet validated end-to-end here (those apps weren't installable in the dev bench). Every suite auto-skips when its app is absent, so they're exercised on the first target that has the app.

Adding / extending a suite

Subclass ReleaseSuite, set name + required_app, and return ordered Steps from build_steps. Steps share a context dict so later steps reuse earlier records. Use the idempotent helpers in release_tests/factories.py. For version-specific behavior, branch with gating.at_least(versions, app, "16").

Testing the harness

uv run pytest          # fully mocked — no live site required

Roadmap

• v2 ERPNext: full Quotation → Sales Order → Delivery Note → Sales Invoice → Payment → reconciliation chain.
• Press path: provision a site + install an app set, then feed its URL in as a runtime target — the runner already accepts arbitrary target URLs.
• Flesh out the stub suites as those apps become available to test against.