Run real-harness tests

Use this guide when you need regression coverage against a real agent CLI. The current suite uses Cursor (cursor-agent) and Composer 2.5, especially for generated workflow artifacts and CLI orchard generate output.

The default pnpm test and pnpm validate commands stay deterministic and do not require Cursor auth, network access, or Composer quota.

When to run

Run the real-harness suite before merging changes to:

• packages/cli/src/workflow-artifacts.ts
• CLI orchard generate behavior
• Cursor harness repair or source-contract validation
• Codex generation guidance or defaults, unless a separate Codex real-generation check is documented with the change

Skip it for documentation-only changes unless they affect real-harness docs.

Prerequisites

• cursor-agent on PATH and authenticated for your machine
• Network access to Cursor services
• Composer 2.5 quota (non-fast model slug: composer-2.5)
• Built workspace packages (pnpm build)

Commands

From the repository root:

pnpm build
ORCHARD_REAL_HARNESS=1 pnpm test:real-harness

Package-scoped runs:

ORCHARD_REAL_HARNESS=1 pnpm test:real-harness:core
ORCHARD_REAL_HARNESS=1 pnpm test:real-harness:cli

Real-harness tests live under packages/*/test/real-harness/*.real.test.ts and are excluded from default package Vitest runs.

Serialization and concurrency

Real Cursor calls are serialized by:

• root Vitest config (fileParallelism: false, single fork)
• an in-process mutex in packages/core/test/real-harness/cursor-serial.ts

Do not enable write-capable Cursor parallelism in this suite. An optional quarantined concurrent test runs only when ORCHARD_REAL_HARNESS_CONCURRENT=1 is set and documents Cursor CLI config race risk.

Interpreting failures

Symptom	Likely cause
Auth / login errors	Cursor CLI not authenticated; fix locally, not an Orchard regression
Quota / rate-limit errors	Environmental; retry later
Model ignored adversarial first-turn prompt	Environmental flakiness; real-harness prompt triggers are best-effort
Stable contract/dry-run error after repair budget	Orchard regression in artifact generation or source-contract checks
WorkflowArtifactError with retryable: false	Terminal artifact failure; fix the generated artifact instead of retrying unchanged

Tests redact bearer tokens and API key material from failure messages. Never commit Cursor credentials.

Related

• Generate a workflow with CLI
• Run a harness