hecks

Deploy and smoke scripts become commands on the Deploy chapter

Status: Accepted; every row of the table is built, shaped or retired. Date: 2026-10-05. Continues ADR 0080 (a script becomes a command with its outside work behind a port) for the scripts a deployed project keeps, and ADR 0085 (the AwsBox kind that generates most of them). Four slices are built: smoke_run.run, the two rolls, service_roll.run and box_roll.run, the data copy, data_copy.restore and data_copy.verify, and the last three scripts, bluebook_diff.run, preview_run.<verb> and companion_roll.run. The generated make deploy and make deploy-service call the rolls, whose policy requests the smoke.

Context

A project deployed to an AwsBox runs a mix of shell it cannot see into:

• Seven scripts the projection generates from templates: deploy-box.sh, deploy-service.sh, fetch-secrets.sh, render-compose.sh, restore-to-rds.sh, smoke-after-deploy.sh and verify-copy.sh. They are tested against stand-in aws, docker and gh programs, but a run leaves no record: no journal entry, no query for “when did the last smoke fail”, no refusal the model checker can see.
• Three hand-written scripts in a client’s platform directory, read for this ADR:
  • bluebooks-diff.sh prints which bluebook releases a domain deploy changes. It reads the running image’s ai.embryonaut.bluebooks label out of ECR with the AWS CLI alone (read-only, no docker pull: the active task definition’s image tag, its manifest, the arm64 entry of an index, the config blob), reads the same label from the local image about to be pushed (or two JSON files given as --old and --new, no AWS), and compares name, version, shape and digest per bluebook. It marks a new era, and a same-version change in content. It is informational: it always exits 0, because the era check gates a deploy, not this report.
  • preview.sh keeps one isolated stack per git branch (deploy, destroy, list, url, name, login). It derives every name from the branch (with a hash when the slug was altered or truncated), refuses main and master, only touches stacks named <prefix>-*, pushes the already-built images, creates the branch’s database, deploys a separate preview template, and can mint a signup token that signs the caller in as the preview’s first administrator. deploy and destroy write to AWS; name, url, list do not.
  • deploy-umami.sh rolls a second Compose project (Umami and its tunnel) onto the app box beside the app’s project, so an app deploy never removes it. It reads one container’s image, environment and secret references from an ECS task definition, builds compose.json and secrets.json on the laptop, sends them over SSM with the generated fetch-secrets.sh (secret values never travel; the box resolves them), pulls and starts the project, then runs a check on the box: heartbeat, a bad login answering 401, and a registered tunnel connection.

ADR 0080 settled the shape: a rule becomes a given, a side effect sits behind a port, a result is an event. The Deploy chapter already follows it for CostCheck, MakefileCheck and TemplateComparison: a request command, a policy that asks a port, and a pass or flag command that records the answer.

Decision

1. Each script that runs from a laptop or CI gets an aggregate in the Deploy chapter whose Run command takes the script’s inputs, asks the DeployToolchain port, and records passed or flagged. The aggregate is named for the thing that ran (a roll, a smoke), so its journal reads as a history of rolls and smokes.
2. A command takes the project, not a file path. It finds the generated script itself: beside the project’s Makefile, else the only one beneath the project; none or several is a refusal that names script=<path>, the optional override. A project passes the same directory it generated into.
3. Where the work stays shell and where it moves.
   • Scripts that run on the EC2 box (fetch-secrets.sh, render-compose.sh, and the checks a command sends over SSM) stay shell. The box has no Ruby, and these are small enough that a generated, golden-tested script is the right shape. They are steps inside a command, never commands of their own.
   • Orchestrators that run from a laptop or CI become commands. While a script’s logic is only worth running as it is, the adapter runs the generated script with the inputs as the variables it documents and maps its documented exit statuses to a reason; the command and the script share one logic. When a command’s specs hold the behaviour, its logic can move into a Ruby adapter and the template can go. Which of the two a command is at any time is not visible to its callers.
4. Rules about the inputs move to givens (a task definition is a family name; exactly one of two options). Rules about the live world (a stack’s status, a container’s state) stay where the world is read.
5. Every such command is hecks deploy <aggregate>.run and is added to settled in hecks.world, so a flagged run is exit 1 without --wait.
6. A roll and its smoke are two aggregates joined by a policy. A roll records rolled and a policy requests a smoke under the roll’s own run key, unless the roll was given skip_smoke=true; a smoke can still be run alone. Built: ServiceRoll and BoxRoll are the two roll aggregates, and SmokeAfterServiceRoll and SmokeAfterBoxRoll are the policies.
7. make deploy calls the command, and the record is persisted. The generated Makefile’s deploy runs $(HECKS) deploy box_roll.run project="$(CURDIR)" run="$$run" --wait where it ran deploy-box.sh, and (with the hosting scripts) hosting.mk’s deploy-service runs service_roll.run where it ran deploy-service.sh. The commands run in the normal environment, so the Hecks domain binds to HECKS_DATABASE (default postgres://hecks@localhost/hecks) and each deploy’s BoxRoll or ServiceRoll and SmokeRun are durable and queryable.
   • A failed roll or smoke is exit 1, with the script’s own status in the message (the table below). SKIP_POST_DEPLOY_SMOKE=1 becomes skip_smoke=true. The target reads nothing after the command: its exit covers the whole deploy (decision 8).
   • One-time setup. A machine that runs make deploy, a laptop or a CI runner, needs a database and a non-superuser role that owns it (decision 21): HECKS_DATABASE pointing at it, or the default postgres://hecks@localhost/hecks. The first run creates the tables. A CI job that runs make deploy needs a Postgres service and the variable too.
   • A missing database does not stop the deploy. The command cannot start without its database, so the target detects the launcher’s cannot open Hecks failure, prints that error and the setup step, runs the generated script itself (and, with the hosting scripts, the smoke script after it), and exits non-zero: with the script’s own status when it failed, or 24 when everything passed but the record could not be written (make reports it as Error 24; make’s own status is 2). The database error and the deploy’s outcome are stated separately.
   • smoke-after-deploy as a target of its own still runs smoke_run.run, so a smoke can be run alone.
8. The roll record also holds the smoke’s outcome. The roll and the smoke stay two aggregates, joined by policies both ways. The roll’s Answered event requests the smoke (declared after the policy that records the roll, because a reaction chain is bounded at five), carrying after (service_roll or box_roll; alone for a smoke a person runs), which the smoke’s events keep. Policies on SmokePassed and SmokeFlagged whose after names the roll then reach back to the roll by the shared run key: Verify moves rolled to verified (the smoke passed), and FlagSmoke moves rolled to flagged with the smoke’s reason in refusal and in smoke. A flagged roll is a failure state, so a failed smoke makes box_roll.run and service_roll.run exit 1 under --wait, and flagged lists it. A roll with no smoke stays rolled, and its smoke field says why: smoke skipped: skip_smoke=true or smoke skipped: no smoke script.
9. Every project deploys through box_roll.run. A project without hosting_scripts true has no smoke-after-deploy.sh; the adapter looks for one (beside the Makefile, else the only one beneath the project) and, finding none, answers that the smoke is not to be run, which the policy honours. Its Makefile gets HECKS ?= hecks and the same recipe, so each of its deploys leaves a BoxRoll that stays rolled with smoke skipped: no smoke script; only the hosting projects have a deploy-service.

10. Specs run the generated golden script against the stand-in programs of spec/support/box_hosting_stubs.rb, through the launcher. The goldens come from the generator.

11. A command that overwrites live data refuses unless it is confirmed. data_copy.restore replaces the target database’s schemas, so the adapter refuses without confirm=true, as a flagged record whose reason names the schemas, the database and the host it would overwrite, and runs nothing (not even a secret read). dry_run=true answers that same plan as a planned record, runs nothing, and requests no verification, with or without confirm. force=true (drop the target’s schemas first) is the script’s own option and still needs confirm=true.
12. One command name per aggregate, so data_copy.verify is on DataCopy. The launcher resolves hecks deploy <aggregate>.<command>, and a names alias does not reach an attached chapter. DataCopy therefore has both public commands, Restore and Verify, and CopyVerification is the aggregate that runs the comparison. Both commands end in the same policy, CopyVerification.Run under the copy’s run key, and policies on CopyMatched, CopyDrifted and CopyUnverified move the DataCopy to verified, drifted or flagged. After a restore the move is from restored; for data_copy.verify alone, from requested. drifted is an answer, not a refusal: the adapter answers drifted on verify-copy.sh’s status 50, and any other failed status is a refusal.

13. The restore script leaves the comparison to the policy. Run by hand, restore-to-rds.sh still ends by running verify-copy.sh. The command sets VERIFY_BY_COMMAND=1, which ends it after the copy, so the one comparison is the policy’s and is recorded. The generated Makefile calls neither script, so no recipe changed.

14. Which scripts still exist was checked against the client’s current tree before the last slice. A client’s recent change replaced several hand-written scripts with verbs of this gem (a manifest check, a CloudFront comparison, the hosting scripts, the host build), but kept the three this ADR reads: the bluebook report, the preview script and the companion roll. None is retired. The companion script targets the generated box’s stack, not the old one deleted at the cutover, so the roll is current; it stays a client script, found by name.
15. The bluebook report is pure where it can be and never fails. Given old and new (two package.verify outputs, given together or not at all) the comparison is made in Ruby by BluebookReport, with no script and no AWS. Given neither, the adapter runs the project’s bluebooks-diff.sh, which reads the running image’s label out of the registry (read-only), and classifies what it printed. The outcomes are unchanged, changed and unavailable, all exit 0; unavailable covers a lookup that failed, a missing label, a missing script and a script that ended non-zero, each with its reason in the report. Only a malformed request (one of old and new) is flagged, exit 1. The registry read stays in the script because its names (cluster, repository, label) are the client’s; moving them behind the port needs them declared in the project’s world (see “What remains”).
16. One aggregate, six commands, for the preview. PreviewRun has Name, Url, List, Deploy, Destroy and Login, which all ask one port ask, RunPreview, distinguished by an action each command defaults. This keeps one record per run and one set of exit statuses, and the launcher resolves hecks deploy preview_run.<verb> for each (decision 12). The outcomes are named, located, listed, deployed, destroyed, signed_in, planned and flagged, each recorded by its own policy on the answer’s outcome.
17. Writes refuse unless confirmed, and a login counts as one. preview_run.deploy, .destroy and .login and companion_roll.run refuse without confirm=true, as flagged with the plan in the reason, running nothing. login reads a preview’s session secret and mints an eight-hour administrator session, so it is held to the same rule as a write. dry_run=true answers planned and runs nothing, with or without confirm; on the reads it changes nothing. The plan is read from the script’s own text (its stack prefix, region, box stack and directory), so saying it runs no program.
18. A branch is a given. Branch refuses main and master as an invariant, so branch=main is refused by the launcher before any record or script. With no branch=, the script’s own check refuses the checked-out branch when it is one of them, as flagged.
19. A companion is named, not hard-coded. companion_roll.run <project> taskdef= [companion=umami] finds deploy-<companion>.sh and passes the task definition as its one argument. The adapter keeps nothing of Umami; a second companion is a second script of the same shape.
20. Exit statuses. The three scripts end 0 or non-zero with no distinct codes of their own (the preview script ends 1 on every refusal and failure and 2 on a bad verb; the companion script ends 1 on a refused roll, a failed roll and an unhealthy companion; the report script ends 0 always). The adapter does not invent codes: a non-zero status is flagged, exit 1, with the status, its meaning in the table below and the script’s output in the reason.
21. The record’s database needs a non-superuser role, so createdb hecks alone is not enough. Running hecks deploy smoke_run.run project=<dir> --wait against the default persistent environment failed with cannot open Hecks: cannot boot Hecks: PostgresEra's era write-fence is row-level security, and this connection's role "hecks" is a superuser. The era write-fence is row-level security, and Postgres exempts a superuser from every policy, so the persisted record refuses to boot as one. A developer machine whose default hecks role is a superuser (a Homebrew or Postgres.app install makes the installing user one) must use a separate non-superuser role for deploy records and point HECKS_DATABASE at it, for example CREATE ROLE hecks_deploy NOSUPERUSER LOGIN; and CREATE DATABASE hecks_deploy OWNER hecks_deploy; with HECKS_DATABASE=postgres://hecks_deploy@localhost/hecks_deploy. The generated Makefile’s missing-database message names both steps. Its behaviour is unchanged: the script and smoke still run, their result is printed, and the exit is 24 when they passed but nothing was recorded.

The command table

Script	Command	Givens	Outcomes	Generated or adapter
smoke-after-deploy.sh	smoke_run.run <project> (built)	taskdef is a family or family:revision	passed; flagged with exit 20 (roll did not settle), 21 (no gh), 22 (smoke failed), 23 (unknown)	the script stays generated; the adapter runs it
deploy-box.sh	box_roll.run <project> [taskdef=\|tags=] [skip_smoke=] (built)	taskdef is a family or family:revision; tags are name=tag words; skip_smoke is a switch	verified (rolled, and the smoke passed), rolled (no smoke was requested); flagged with exit 1 and the reason 40 (no instance), 41 (the roll did not succeed on the box), 42 (the box is not healthy after it); a policy requests the smoke, and a failed smoke flags the roll	the script stays generated; the adapter runs it
deploy-service.sh	service_roll.run <project> service= [existing_tag=] [local_image=] [skip_smoke=] (built)	service is a container name (the script refuses one the world lacks); existing_tag redeploys a tag already in ECR	verified or rolled (with the tag and task definition); flagged with exit 2 (unknown service), 30 (existing tag not in ECR), 31 (fresh tag already in ECR), 32 (the box’s Compose file unreadable), 33 (no instance), 34 (the stack has no such parameter), 35 (the stack update failed or did not settle), 36 (another parameter changed), 37 (the task definition lacks the pushed image), 38 (no such container in the task definition), 40 to 42 (the box roll’s); a policy requests the smoke, and a failed smoke flags the roll	the script stays generated; the adapter runs it with SMOKE_BY_COMMAND=1
fetch-secrets.sh, render-compose.sh	none: steps inside the rolls			stay shell, run on the box
restore-to-rds.sh	data_copy.restore <project> bastion= source= source_secret= target= target_secret= [source_db=] [target_db=] [force=] [confirm=] [dry_run=] [skip_verify=] (built)	bastion is an instance id; the hosts, secrets and databases are named; confirm=true or the run is refused (decision 11)	verified (restored, and the comparison matched), drifted (restored, the databases differ), restored (skip_verify=true), planned (dry_run=true); flagged with exit 1 and the reason 60 (client older than 16), 61 (the target has a schema), 62 (unexpected restore errors), or the refusal to run unconfirmed; a policy requests the comparison	the script stays generated; the adapter runs it with VERIFY_BY_COMMAND=1
verify-copy.sh	data_copy.verify <project> bastion= source= source_secret= target= target_secret= [source_db=] [target_db=] (built)	same	verified; drifted with the differences (status 50); flagged when the comparison could not be made	the script stays generated; a policy runs it for either command (decision 12)
bluebooks-diff.sh	bluebook_diff.run <project> [old=] [new=] (built, decision 15)	old and new are given together or not at all (else flagged, exit 1)	unchanged, changed (each bluebook added, removed, re-versioned, new era, or same version with different content), unavailable (a lookup failed, with its reason). All exit 0	the comparison of two files is Ruby; the live read stays the client’s script, run read-only
preview.sh name, url, list	preview_run.name, .url, .list (built, decision 16)	the branch is not main or master	named, located, listed; flagged with exit 1 (script status 1: no such preview, a refused branch, an AWS error)	wraps the script; name derivation stays in it (shaped: it moves to Ruby when a preview is declared in the world)
preview.sh deploy, destroy, login	preview_run.deploy, .destroy, .login (built, decisions 17 and 18)	same; confirm=true or the run is refused; dry_run=true prints the plan	deployed, destroyed, signed_in (the script’s output holds where the sign-in line was written), planned; flagged with exit 1	wraps the script behind the port a spec replaces with stand-in programs
deploy-umami.sh	companion_roll.run <project> taskdef= [companion=] (built, decisions 17 and 19)	taskdef is a family or family:revision; confirm=true or the run is refused; the script refuses a missing container or box	rolled (the containers’ status and the box check’s result are in the report), planned; flagged with exit 1 (no box, roll not successful, companion not healthy)	wraps the script, which stays an orchestrator over SSM; the box check is a script it sends to the box

Built in this change

Slice 1: SmokeRun (smoke_run.run <project> [script=] [taskdef=] [skip=true] [async=true] [dry_run=true]), the Smoke ask on the DeployToolchain port, its adapter method, and the generated Makefile calling it.

Slice 2, the rolls: ServiceRoll (service_roll.run) and BoxRoll (box_roll.run), the RollService and RollBox asks and their adapter methods, a policy on each Answered event that requests a SmokeRun (the same run key, the roll’s task definition when it named one), and two policies per roll that record the smoke’s outcome on the roll (decision 8). Both wrap the generated script. The scripts changed in three ways, all regenerated into the goldens: deploy-box.sh and deploy-service.sh end with distinct statuses (table above) instead of 1; deploy-service.sh ends after the roll, printing rolled taskdef=<td> tag=<tag>, when SMOKE_BY_COMMAND=1 (the command sets it, the policy runs the smoke); and the Makefile and hosting.mk recipes call the commands (decisions 7 and 9). Specs run the golden scripts against the stand-in programs, and the recipes against a stand-in hecks.

Slice 3, the data copy: DataCopy (data_copy.restore, data_copy.verify) and CopyVerification, the CopyData and CompareCopy asks and their adapter methods (copy_data, compare_copy), the policies from a restore and from a verify request to the comparison, and three policies that record its outcome on the copy (decisions 11 to 13). Both wrap the generated scripts, which gained distinct statuses (50 differ; 60, 61, 62 for the restore) and VERIFY_BY_COMMAND; goldens are regenerated. Specs run the golden scripts against stand-in aws, psql, pg_dump and pg_restore programs.

Slice 4, the last three: BluebookDiff (bluebook_diff.run), PreviewRun (preview_run.name, .url, .list, .deploy, .destroy, .login) and CompanionRoll (companion_roll.run), the CompareBluebooks, RunPreview and RollCompanion asks and their adapter methods, policies that record each outcome, BluebookReport (the pure comparison) and DeployPlans (the plan a refused or dry write prints). The verbs are settled, so a flagged run is exit 1 without --wait. The scripts are the client’s own and are not changed; specs run stand-in scripts of the same shape against a stand-in aws, and never reach AWS, a registry, SSM or a database.

Command	Exit	Meaning
bluebook_diff.run	0	unchanged, changed or unavailable
bluebook_diff.run	1	only one of old and new was given
preview_run.*	0	the script ended 0, or the run was a dry run
preview_run.*	1	not confirmed, no script, or the script ended 1 (a refusal or failure it reports) or 2 (a bad verb); main and master are refused by the launcher before the run
companion_roll.run	0	the roll and the check on the box passed, or a dry run
companion_roll.run	1	not confirmed, no script, or the script ended 1 (no box instance, no such container, the roll did not succeed, the companion is not healthy on the box)

Still scripts, unchanged: running the scripts by hand, which still smokes at the end of deploy-service.sh.

Consequences

• A smoke becomes a record in the run’s journal, and smoke_run.flagged lists the ones that failed; the MCP door can run it. A roll is a record too: box_roll.flagged and service_roll.flagged list the rolls that failed, with the script’s status in the reason.
• A deploy that writes to AWS is now one hecks deploy command from the MCP door as well; a door that exposes it should treat service_roll.run and box_roll.run like any other command that changes a live system.
• make deploy now needs a hecks and a Hecks database on the machine that deploys (decision 7). Which machines and workflows need one is a deployment fact: this repository’s workflows do not run make deploy.
• A project must regenerate its recipe to pick up the new Makefile and scripts; until then it keeps calling the scripts, and make deploy leaves no record.
• A script whose logic moves into Ruby needs stand-ins for aws and gh at the adapter boundary, not on PATH.

Out of scope

• Deploying anything from this change: the roll commands write to AWS when a person runs them, and this change’s specs only ever run them against stand-in programs.
• Changing the on-box scripts, or the generated scripts beyond their exit statuses and the SMOKE_BY_COMMAND end.
• The Fargate and Lambda kinds’ scripts; they follow when a project asks.
• Scheduling a smoke or a cost check (ADR 0081 lists scheduled commands as unbuilt).
• Cutover steps a person performs (bastion creation, CDN origin switch), which stay in the migration runbook.

Open items

• A persistent journal for deploy commands: decided, persisted (decision 7). Retention of old SmokeRun, BoxRoll and ServiceRoll rows is not decided.
• Whether preview_run belongs to the Deploy chapter or to a Fargate-specific one, since the preview stack is a separate template from the live one. It is in the Deploy chapter for now, wrapping whichever preview.sh the project has, generated or hand-written.

What remains

Every script of the table is now a command. What is left is moving logic, not scripts:

• The registry read of bluebooks-diff.sh (the running image’s label from the active task definition) and the local-image read can move behind a port, read-only, once a project declares its cluster, repository, family and label in its world rather than in the script.
• Preview name derivation (branch to stack, with a hash when the slug was altered) is pure and can move to Ruby; it stays in the script until the preview’s prefix and limits are declared in the world.
• The companion roll could be projected by the generator from a declared companion (image, secrets, tunnel) instead of read from a hand-written script, as the box scripts are.
• Retention of old SmokeRun, BoxRoll, ServiceRoll, DataCopy, PreviewRun and CompanionRoll rows.
• Scheduling a smoke or a cost check (ADR 0081).

This site is open source. Improve this page.