CFn schema refresh runbook

A scheduled job keeps tests/fixtures/cfn-schemas/*.json current with what AWS publishes. A property missing from them still reaches AWS, because it routes through Cloud Control; the fixtures are what let an SDK provider take a property over and what keep the read-only, create-only, nested-required and property-shape data current. This page is the operator's side of it: what arrives, and what to do with it.

The reasoning behind the design — why a scheduled PR rather than a CI check, why the job is allowed to fail — is in Provider Rules.

The short version

Do not wait for CI to tell you. GitHub holds the workflows on a bot-created pull request at action_required until a maintainer approves them, on every push — measured on this repository's release pull requests, which have been merged many times and are still held, so merging one does not earn the bot an exemption. Until you approve, the checks that would catch a schema decision have not started.

The title and the label are the signal, and they are not a proxy for CI. The job runs every fixture-driven check itself, inside the run, and the count in the title is that result — it exists before the pull request's own CI is allowed to start. Approving the workflows re-runs the same checks; it does not tell you anything the title has not already said.

The job keeps at most one pull request open. While one is open, later runs push the new drift onto that same branch and comment with what it added, so nothing waits for a merge.

Your commits on that branch are safe while the pull request is open: the push is additive and never forced, everything the job rewrites is derived and recomputed from your current sources, and the classification you write does not live in those files. Once a pull request is CLOSED without merging, its branch loses that protection — a same-day re-run reuses the name and replaces it. If you want to keep working on a closed one, rename the branch first.

What the job does

Daily, on bot/cfn-schema-refresh/<YYYY-MM-DD>:

1. Switches to the open refresh pull request's branch, if there is one, so drift is measured against what that PR already carries.

2. Downloads AWS's public CloudFormation schema bundle. No AWS credentials are involved.

3. Rewrites only the fixtures that actually changed. A capture-date-only difference is not a change.

4. Stops if nothing drifted, and no refresh pull request is open — no branch, no pull request, no comment.

5. Otherwise regenerates the derived artifacts, then either opens a pull request or adds a commit and a diagnosis comment to the open one.

6. Commits a changelog fragment under changelog.d/entries/ when the cycle added a writable property, because that changes what the shipped binary does. The text is DERIVED — which type gained which property, that it is now a silent drop routed through Cloud Control, whether the type pins provisionedBy: 'cc-api' one way, and whether the snapshot marks it create-only — and nothing that needs a reading of WHY is in it. Edit it if the delta deserves more; the job never overwrites a fragment it already wrote for this cycle, a rename included, and a later cycle writes its own file rather than touching yours. EDIT it rather than deleting it — a deletion is the one thing a same-day re-dispatch would undo.

   A missing fragment or a lost push never fails the job, which is the opposite of the push rule below, deliberately. By the time this step runs the drift is committed and the pull request exists, so the only thing such a failure can still cost is the marking step behind it — and trading the signal a human reads for a changelog line is the wrong way round. Either case leaves a ::warning:: in the run and no entry: write one by hand before merging. A third warning says a fragment for this cycle already exists and differs from what the run rendered — a second batch of drift arrived the same day, and the existing entry does not describe it. The one arm that IS loud is an unsubstituted __PR_NUMBER__ or __CYCLE__ placeholder, which means the renderer is broken rather than a race being lost, and would otherwise commit an entry citing no pull request.

   A cycle that only REMOVES properties gets an entry too, because it ships a delta of its own: a withdrawn property loses the row that made it route through Cloud Control and becomes unrecognized. A new resource, or a changed value, still routes and Cloud Control rejects the key AWS withdrew; an existing resource carrying it unchanged stays on the SDK path with a warn. The entry says which of the two things follows for each type — it keeps auto-routing on another property, or a new resource of that type without the withdrawn key returns to the SDK path while one already recorded provisionedBy: 'cc-api' stays where it is. A property the provider DECLARES is not in that population: its removal makes the declaration bogus, which the pull request escalates as a decision instead.

7. Marks the pull request with how many decisions are left, or clears the marking when there are none. This is the step that also runs on a no-drift day, so a decision you have settled stops being advertised.

Nothing that encodes a judgement is touched: no unhandledByDesign, no bogusTolerated, no NESTED_KEY_ALLOW_LIST, no provider code.

The standing backfill checklist is not written by this job. A separate workflow, backfill-umbrella-sync.yml, rewrites it — one row per resource type, inside the umbrella issue's body — whenever src/provisioning/property-coverage.generated.ts changes on main: a refresh merge, a hand-written backfill, a revert. Driving it from main is what keeps them describing something that actually exists: written during the refresh run instead, they described the job's own workspace, so closing a refresh PR unmerged left the campaign listing properties main does not have.

If a push onto an open pull request would not fast-forward — someone pushed while the job ran — it gives up for that cycle with a warning rather than forcing. The next run recomputes the same drift. A push that fails while the branch has NOT moved is a different thing (a permission or branch-protection problem), and fails the job loudly instead: a green daily run that lands nothing, forever, is the failure mode worth being noisy about.

A green pull request

Merge it. Properties were added and nothing else.

A template using one of those properties already routes through Cloud Control before the merge, as an unrecognized property. The merge records them as known properties the SDK provider does not wire, which puts them on the backfill list and brings their create-only, nested-required and property-shape data in.

Newly unaccounted writable properties are listed in the pull request. They reach the standing backfill checklist when this PR MERGES, not when the job runs — backfill-umbrella-sync.yml rewrites the umbrella's generated block from main, into the row for each property's own resource type, adding one if that type had none. Wiring them into an SDK provider is separate, unhurried work.

Writable matters here: a schema property AWS computes and returns (readOnlyProperties — an Arn, a DomainName) can never be a dropped value, because there was nothing to send. Only settable properties represent work. In a typical cycle most additions are read-only.

How a decision reaches you

A refresh PR that needs one is assigned to you, labelled needs-decision, titled with the count, and opened by a one-line verdict at the top of its body. The verdict names a range — D1–D5 — and every decision in the body carries its label, running straight through the sections. They are spread across sections because they arrive from different checks, so the labels are what let you reach the number in the title; each decision section also carries its own count in the heading. Of those, only the assignment sends a notification — it is applied first for that reason — while the label and the title suffix are what the PR list can still tell you afterwards, including that a PR was settled. The verdict line is rewritten every cycle, so it does not age the way the rest of the body does.

GitHub holds CI on a bot-created pull request at action_required until a maintainer approves the workflows — on every push, and merging an earlier bot pull request does not change that. So a decision-carrying PR does not show a red check; it shows no check at all. Before this marking existed it was indistinguishable, in every list view, from a refresh that needed nothing.

Two classes aside, you cannot merge one by mistake, and the marking is not what stops you. ci-ok is a required status check on main, so the held state above already blocks the merge button — a required check that has not reported is not a pass. Approve the workflows and the block holds for the opposite reason: nearly every decision the count can carry also reddens a check that ci-ok waits on (a nested key leaves audit:nested-key-coverage:check at exit 1, an unsettled removal leaves a declaration property-coverage fails, and the rest of the count IS that family of checks). So the label and the title are there to tell you the work exists, not to hold the gate.

Unsettled is the load-bearing word in that sentence, and it was wrong until issue #3005. A property listed in _todo-backfill.json's bogusTolerated is settled — that is what the entry's rationale says — so property-coverage is green for it, and AWS removing it again is not a decision. The count used to disagree, because it subtracted only what the CURRENT cycle wrote: a tolerance committed on an earlier cycle left the removal counted, and the PR was titled "1 decision needed" with nothing red and nothing to do. Both sides now read the tolerance FILE, so they cannot disagree.

Two classes are still counted without reddening anything, and they are the two exceptions to "you cannot merge one by mistake" above — approve the workflows on such a cycle and CI goes green with the count non-zero, so there the marking is the only signal. Both mean the RUN is broken rather than that AWS moved:

• a fixture the diagnosis could not read from HEAD. The PR body names them. The narrowness is the point — an unparseable fixture in the WORKING TREE is counted the same way but does redden CI, because every fixture-reading check fails to load it.
• a check the refresh run could not EXECUTE — an empty log, a missing task, a killed process. Two of the counted classes read a refresh-side exit code, so CI running the same tasks independently is green while the refresh's failure is what was counted. The run log says which check and why. (A check that ran and genuinely failed is not this: that reddens CI too.)

The marking is cleared by the next run once you have committed the classifications: the job recomputes the count while the PR is open, even on a day AWS changed nothing, then drops the label, restores the plain title and rewrites the verdict line. The assignee stays — the PR is still yours to merge, and un-assigning it would drop it out of your assigned view at the moment it became mergeable. So a standing needs-decision means work is genuinely outstanding, not that nobody tidied up.

On a cycle that publishes nothing, the count can only CLEAR the marking — it never changes a non-zero one in either direction. Removals are computed by diffing against the branch's committed fixtures, so with nothing to refresh that half of the count is empty by construction and a "2 decisions needed" PR would otherwise be retitled "1" with nothing settled. The same run also refuses to clear when regenerating produces changes the branch has not committed — otherwise fixing a provider without running vp run gen:all-matrices would clear the label while the PR's own CI stayed red.

If the label is missing entirely and the run log says it could not add it, create it once — gh label create needs-decision — and re-run. The job deliberately does not create labels, so that deleting one stays a decision.

A red pull request

Red is the hand-off, not a defect. The pull request names which class fired, on which type, which provider lines mention it, and what the AWS SDK still models — so the research is already done and what is left is the decision.

Two of them need a judgement and have their own sections below: a removed or renamed property, and a nested-key divergence.

The rest are CI checks that read the schema fixtures, and the pull request gives each one that failed its own section with the commands to settle it — so this page does not enumerate them. What they have in common is worth knowing: a plain schema addition reaches every one, which is the shape easiest to wave through.

A fourth section, "Fixtures this report could not read", means the comparison itself failed for those types — neither their removals nor their additions are accounted for anywhere in the pull request, so read the refresh job's log.

A property was removed or renamed

A provider still declares a property AWS no longer publishes.

Removal from the schema does not mean the API rejects it. The schema is a published model; the API is the behaviour.

Most of these never reach you. The job settles a removal itself when two facts hold together — the type's own AWS SDK client still declares a member of that name, and the provider actually wires the property rather than only listing it — and it writes the tolerance with those facts as the reason. Both are read from the checkout, and where both hold, "keep sending it" is the only answer either one permits.

A removal reaches this section when one of them fails, or when the same refresh added a name that could be the same field renamed. The pull request says which.

The rename test is deliberately narrow — it flags an addition whose spelling extends or is extended by the removed name, so a rename that changes the middle of a name is not flagged, and the absence of a flag is not evidence there was no rename.

# After editing, this names anything still bogus — and any tolerance AWS has
# since made stale by re-adding the property.
vp test run property-coverage

aws cloudformation describe-type is corroboration, not the authority. A removal here is about an SDK Provider's declaration, and cdkd's SDK Providers call service APIs directly — the CFn type registry is the API being served only for Cloud Control-routed types. It also adds no independent information: the public bundle this job already read was measured byte-identical to the authenticated capture on every type probed.

A nested key diverged

A NESTED_KEY_TARGETS type gained a nested key whose spelling does not match the AWS SDK model. The check reports divergences, not staleness, so regenerating never clears one.

A case-divergence needs no judgement: the SDK models the key under a different capitalisation, so rename it in the provider.

A no-write-evidence is not an SDK question at all: the SDK models the member and this provider builds fresh SDK objects, so the finding says the write pass could not see cdkd assemble the field at that path.

That is usually because it genuinely never does — but not always, and the difference is a judgement. The pass has documented false-positive classes (a RENAMED intermediate segment, a member delivered one wrapper level below the CFn chain, a request-HOISTED member), enumerated in .claude/rules/provider-nested-key-divergence.md; AWS::S3::Bucket alone already carries reviewed passes: ['write'] allow-list entries settling findings that way. So confirm the member is not already delivered by one of those shapes BEFORE writing a second forward for it — a duplicate write is the failure mode of following this section too literally.

When it really is never assembled, write it, per member. Two things go with that, and neither is optional: a member the provider SENDS but its readCurrentState does not READ is permanent phantom drift that cdkd drift --revert re-applies forever, so the readback lands in the same change; and an OBJECT-valued member is forwarded only when the template declares one, or every resource of that type starts sending an empty block it never sent before. A bump can also MOVE a finding here — a no-sdk-member whose newer client models the member becomes a no-write-evidence, which reads like no progress and is the diagnosis changing from "AWS may not have this" to "cdkd does not send it". PR #3002 is the worked example (AWS::S3::Bucket's DefaultRetention.DefaultEventHold).

For no-sdk-member / definition-member-missing, rule out the installed SDK simply lagging the service before allow-listing anything — an entry added over a stale SDK hides a real dropped value.

For a definition-member-missing the job usually rules this out for you. It downloads the published client and re-asks that finding's own question there — does interface I declare member M? — so one reaching this section is one the published client does not resolve either. A finding the bump does resolve appears in its own pull-request section instead, under "SDK bumps that resolve nested-key divergences", naming the client and the version. Merge that bump; no allow-list entry is called for.

The exception is named in the pull request whenever it happens: if the published client could not be downloaded, that finding's SDK-lag reading is unknown, not ruled out, and the procedure lists it as such. Bump and re-check those by hand before allow-listing them.

A no-sdk-member carries no interface, so there is nothing to re-ask automatically — it is a member-index question over the whole client. Check it by hand:

# What is installed, versus what npm publishes today.
node -p "require('@aws-sdk/client-<service>/package.json').version"
npm view @aws-sdk/client-<service> version

# If a newer one exists, bump it and re-check before deciding.
vp run audit:nested-key-coverage:check

Only when the SDK is current and still lacks the member — and the service's own API reference agrees — add a NESTED_KEY_ALLOW_LIST entry with a reason.

Why this is not decided for you

Most of the work is done for you. AWS publishes a third description of each service — the SDK — and the job consults it, so the pull request already says whether a removed property is still modelled there — searched case-insensitively across every client the provider imports, because SDK and CFn capitalisation routinely differ. That is the evidence the decision turns on, and it usually settles the question.

For a nested-key divergence it also reports whether the installed client is behind npm. If it is current, the "the SDK just lags" reading is eliminated outright. If it is behind, that reading is live, and for a definition-member-missing the job settles it: whether a bump fixes the divergence is an interface-level question, so it downloads the published client and asks the interface — a name lookup would answer a different question and could contradict the check. Those findings are grouped into one bump per client rather than left as one decision each.

What is left is genuinely undecidable from the repository:

• A rename looks exactly like a removal at the name level, which is why this repository carries hand-maintained rename maps at all.
• no-sdk-member means the installed SDK version lacks the member. That can be the SDK lagging the service rather than the service lacking it, and the finding carries no interface for the job to re-ask.
• Adding an allow-list entry is a promise that cdkd will never send the value. That is a policy choice, not a lookup.

The asymmetry matters more than the ambiguity: the silencing option (bogusTolerated, NESTED_KEY_ALLOW_LIST) is always available and always turns CI green. Anything choosing automatically under uncertainty converges on it, and that quietly disables the checks that exist to catch dropped values.

Running it by hand

# Pull a refresh forward, or check what AWS has changed. No credentials needed.
vp run gen:cfn-schemas-from-zip

# One type, through the authenticated path. Required for types the public
# bundle does not carry.
node scripts/refresh-cfn-schemas.mjs 'AWS::Service::Type'

# See what changed, then let the coverage test name what is unaccounted.
git diff tests/fixtures/cfn-schemas/
vp test run property-coverage

Trigger the scheduled job without waiting for the next day from the repository's Actions tab, or with gh workflow run cfn-schema-refresh.yml.

Branch names are reserved

The open-pull-request guard matches on the branch prefix bot/cfn-schema-refresh/ in this repository. A pull request of your own from a branch under that prefix reads as the open refresh PR, and the job will push its drift onto your branch. Fork PRs are excluded, so this only applies to branches here.

When there is nothing to read

Every other section of this page starts from a pull request. These are the two states where none exists, and from the outside they look identical to a quiet day.

Tell them apart from the run list, not from the absence.

gh run list --workflow cfn-schema-refresh.yml --limit 5
gh run list --workflow backfill-umbrella-sync.yml --limit 5

• a schedule run, success, no pull request → no drift. Nothing to do. The run log says 0 drifted, N unchanged and No fixture drift — nothing to do.
• a run with failure → the job broke. Read its log; the failing step names the cause.
• no run at all for today → see the cadence below before concluding anything.

The schedule runs late, and that is normal

The cron is 04:37 UTC, deliberately off the hour because the top of the hour is contended on GitHub's shared pool. Both scheduled fires measured so far landed hours after it:

Three points inside seven minutes of each other, so read them as "the delay is real and has been about four and a half hours" — still not a bound, since nothing here constrains GitHub's queue, but consistent enough that a run arriving at the nominal time would be the surprise. An absence at 04:37 means nothing; an absence late in the day is worth a look.

Running either job by hand

Both carry workflow_dispatch for this, and it is the documented recovery path rather than a convenience.

# capture, compare, open or update the PR
gh workflow run cfn-schema-refresh.yml

# re-render the campaign's checklist from main
gh workflow run backfill-umbrella-sync.yml

cfn-schema-refresh behaves exactly as a scheduled run: no drift means no pull request. It is also the only way to exercise the job before its first scheduled fire, and the reactive path when someone reports a dropped property mid-cycle rather than waiting for tomorrow.

backfill-umbrella-sync normally fires only on a push to main that touches src/provisioning/property-coverage.generated.ts, so on a quiet week it may not run for days — and if it failed on the last such push, the campaign's checklist keeps its previous contents until the next one. A dispatch closes that gap. Measured on 2026-09-09, on the flat per-property checklist this job wrote before the per-type restructure: a dispatch took the umbrella from 285 rows to 288 in under a minute, leaving 7,655 characters of provenance intact.

What it rewrites is ONE region of ONE issue:

• the umbrella — the single open issue carrying the backfill-umbrella label, found by that label and never by a number. Its body must hold the two markers <!-- backfill-types:start --> and <!-- backfill-types:end -->, and the job rewrites what sits between them: one row per resource type with silently-dropped properties left, - [ ] `AWS::RDS::DBInstance` — 62 properties: `AllocatedStorage`, …. An umbrella that has never been synced carries neither marker, and the job REFUSES rather than appending — so the first coverage-map push after this design lands fails the workflow until someone pastes the block in once (--render-block, below).
• everything else on that page is yours. The audit provenance, the procedure, which pull request closed which slice: all of it sits outside the two markers and is carried through byte for byte. Hand edits INSIDE the block are overwritten on the next sync, and the checkboxes are always rendered unchecked — what removes a row is the property leaving silentDrop, not a tick.

A pull request wiring a type writes Refs, not Closes: the umbrella stays open for every other type, and the row for the type disappears on its own once the coverage map says it is done.

The per-type issues labelled backfill-type are legacy. go-to-k/cdkd#2949 generated one per resource type and this job reconciled them; their content was folded back into the block above because 44 of the repository's 240 open issues were bot-filed slices of one campaign, which no reader of a public issue count can tell from defects. Nothing generates the label any more, and the issues are retired by a one-shot migration run by hand:

REPO=<owner/repo> node scripts/sync-backfill-umbrella.ts --close-legacy --dry-run
REPO=<owner/repo> node scripts/sync-backfill-umbrella.ts --close-legacy

It closes each one as not planned — the properties are still unwired; what ended is the issue — and SKIPS any labelled issue whose body carries no <!-- backfill-type: … --> marker, since that is something a person filed rather than a generated slice. The label stays on what it closes.

When the reconciler refuses

Unlike the umbrella-lookup warnings, every one of these fails the run and writes nothing. They guard a write onto the campaign's only public page, where a green run that changed nothing would be no notification at all.

• the plan carries no resource types while the umbrella's checklist holds N row(s) — a finished campaign and a coverage-map parse that stopped recognising silentDrop produce the same empty plan, and the second would wipe the whole checklist. If the campaign really is complete, confirm it once: PARENT=<n> REPO=<owner/repo> node scripts/sync-backfill-umbrella.ts plan.json --allow-empty-plan.

• the umbrella issue's body carries no '<!-- backfill-types:start -->' marker (or no <!-- backfill-types:end -->) — the block has no region to be written into, which is also the state a brand-new umbrella is in. Paste the block in once, where it belongs on the page, and every run after that rewrites it:

  node scripts/diagnose-schema-refresh.mjs --umbrella-types > /tmp/plan.json
  node scripts/sync-backfill-umbrella.ts /tmp/plan.json --render-block
  

• the umbrella issue's body carries '…' N times — a copy-paste left two markers; the region is ambiguous and splicing one pair would strand the other's content on the page. Leave exactly one of each.

• the rewritten body would be N characters, past GitHub's limit — the campaign has outgrown this shape and needs a different one. Nothing is truncated, because a list that stops partway reads as finished.

To see what a run WOULD do without touching anything:

node scripts/diagnose-schema-refresh.mjs --umbrella-types > /tmp/plan.json
REPO=<owner/repo> PARENT=<umbrella issue number> \
  node scripts/sync-backfill-umbrella.ts /tmp/plan.json --dry-run

What a dispatch will not fix

• A schedule GitHub has suspended. GitHub stops running scheduled workflows on a repository that has gone inactive, and a manual run does not re-enable them — a push to the default branch does. (The inactivity threshold is GitHub's and is not restated here; check their docs rather than a number copied into this page, which would go stale silently.)
• A wrong answer. A dispatch re-runs the job; it does not re-decide anything. If a pull request already carries a decision you disagree with, the sections above are the path, not another run.
• A closed refresh pull request. Re-running reuses the branch name and replaces it, which is the same-day-replacement case described above.

Related

• Provider Rules — why the job is shaped this way, and the bogusTolerated conventions
• Provider Development — adding a provider, including its first fixture capture
• State Management — what a routing change does to an existing resource's state record