Skip to navigation
Automate Testing

GitHub Pull Request Testing

View as Markdown

Availability: assisted pilot. Dynamic PR Tests are being validated with selected Workspaces. They are not generally available. Ask QA.army for pilot access before following these steps.

Dynamic PR Tests identify changed user behavior, run up to three generated Tests on the matching preview, and report what passed, failed, or could not be verified. GitHub checks are advisory and do not change branch protection.

Configure a Project

  1. Connect GitHub and select the repository in Project → Integrations.
  2. Connect Vercel and select the Vercel Project. Configure preview protection access when needed.
  3. Configure sandbox test accounts for authenticated behavior. Confirm that previews use disposable sandbox data and cannot make real payments, send external messages, or perform destructive account operations.
  4. Enable Dynamic PR Tests and choose a limit from one to three Tests per PR revision.

Regression Test Groups remain independently configured. Dynamic Tests can run while regression execution is paused. Draft and fork PRs are excluded from the pilot.

Select regression Test Groups with an agent

A claimed agent acting for a Workspace Owner can read and update an existing connected GitHub deployment watch using CLI 0.2.11 or MCP 0.1.4. Reuse the same CLI authorization; no second API key is required for this operation. Connecting an integration or approving repository access remains a separate user operation.

Read the current watch first:

qa-army prs watch --integration int_...

Keep its repository ID and every selected Test Group you want to retain. Save the complete desired selection as watch.json; replace these example values with IDs returned by QA.army:

{
  "repository_id": "123456789",
  "test_group_ids": [
    "tgr_0123456789abcdef0123456789abcdef",
    "tgr_fedcba9876543210fedcba9876543210"
  ],
  "enabled": true
}
qa-army prs watch-update --integration int_... --input "$(cat watch.json)"
qa-army prs watch --integration int_...

The update replaces the full selection. The repository must remain the same, and every selected Group must belong to this Project. Preserve the existing enabled value unless you intend to change it. Compare the readback with the intended repository, Groups, and enabled state. There is no version precondition or request key: coordinate one writer and reconcile a conflict or uncertain response from a fresh read instead of automatically repeating the update.

MCP uses prs.watch with integration_id. prs.watch_update accepts integration_id, repository_id, test_group_ids, and enabled together.

Saving the watch does not start a manual Run or establish a passing result. The next matching qualified deployment requests regression coverage. Review the actual Runs and evidence for that revision. Disabled Tests stay disabled, and missing account or execution prerequisites still need to be resolved. This operation does not configure Dynamic PR Tests.

Read the report

Open Project → Tests → Pull Requests. Each report identifies the exact PR revision, planned behavior, expected results, coverage gaps, execution status, and links to canonical Run evidence. Inferred requirements appear as review suggestions and do not execute automatically.

  • Passed: the executed scenarios passed with finalized evidence.
  • Failed: an assertion found a product failure.
  • Error: configuration, infrastructure, or evidence prevented a trustworthy result.
  • No applicable Tests: no supported change required dynamic coverage; this is not a pass.
  • Superseded / cancelled: the attempt is obsolete or was cancelled.

A new commit supersedes pending work. A preview must belong to the selected Vercel Project and match the PR commit. Missing previews time out after 30 minutes. A setup error can be corrected and explicitly rerun.

Use Save as regression to retain a useful scenario in a chosen Test Group. This creates an editable copy; the original verification and evidence stay unchanged.

Understand usage

Planning is included. Each completed generated Test with a trustworthy passed or failed result consumes one Run from the shared Workspace allowance. Errors, cancellations, planning retries and report retries do not consume Runs.

Three dynamic Tests and two regression Tests can consume five Runs. If the complete dynamic plan does not fit the remaining allowance, it pauses. There are no automatic overages. Explicit reruns may consume new Runs; review that cost before confirming.

Pilot enrollment and the agreed plan are arranged with QA.army. No new checkout is required.

CLI and MCP

The pilot adapters use the same REST authorization and results:

qa-army prs list --project prj_...
qa-army prs get --verification prv_...
qa-army prs usage --workspace wsp_...
qa-army prs cancel --verification prv_... --request-key cancel-pr-001

MCP exposes prs.list, prs.get, prs.settings, prs.configure, prs.usage, prs.cancel, prs.rerun, and prs.promote. A governed claimed WorkOS agent can use prs.list, prs.get, and explicit prs.rerun within the claiming user’s authorized Workspace. CLI uses prs list, prs get, and prs rerun --verification prv_... --request-key rerun-pr-001. The server retains preview qualification, current-revision checks, idempotency and Run limits. Dynamic settings, configuration, usage, cancellation and promotion remain outside this claimed-agent permission; use a supported user session or profile credential for those operations. Deployment-watch permissions are separate.