Deployment reference

How an operator deploys and runs its own Phala Pay instance. Phala Pay is self-hosted: each operator deploys a verified release with the settings of its own environment repository, to its own Phala Cloud workspace, on its own domain; the self-hosting guide is the order of the steps, and this document their reference. Names in $VARIABLES are the operator's own values; Phala's are given only as labelled examples, and Phala's own instance, with the demo that only Phala runs, is in Phala's instance.

Every CVM is deployed by GitHub Actions, by the release's Deploy workflow called from the environment repository's main; nothing is deployed from a laptop. Steps outside the workflows that change a registry, Phala Cloud, a CVM, a Safe, a contract, DNS, or a secret are marked HUMAN-ONLY. Backup and restore: RESTORE.md. Incident response: runbooks. Contracts: CONTRACTS.md.

What runs where#

Production CVMs have no SSH, no logs, and no database access. Everything an operator sees comes from the public endpoints, the admin API, Sentry, and the chain.

KMS#

Every CVM uses Phala Cloud's KMS (--kms phala, owner decision): no DstackApp contract and no on-chain compose-hash approval. Fund safety does not depend on upgrade governance: the service holds no key that moves funds and sends no transactions, and a forwarder pays only the treasury its address commits to. Credits are what the attested service signs with each account's webhook key, and merchants pin their keys from attestation and may cap or verify credits on their own node. A malicious upgrade could cause downtime, read service data, or sign false credits within what each merchant's own limits accept; the compose hash in the attestation, verified after every deploy, makes it detectable.

OS image#

The approved OS image is dstack-0.6.1, non-dev, with verified image hash ec43cb8c5e04b6bb6c53b33b16076a01d83d01f7f4b703f2f78a0d947d6247e6. The reviewed os-image-hash file in each environment directory pins this hash, including both Phala staging and production topup configurations. Staging must run this production image before acceptance; a dev image fails verification. Changing the approved pin requires a reviewed PR. Both provision and upgrade pass the pin to verify-attestation.sh, which fails if the official verifier's attested OS hash differs or is missing. preflight.sh accepts only that name and, online, requires a node of the workspace to offer it. The service keeps the dstack v0 guest API (dstack-sdk = "=0.1.3"); the local simulator is built from the 0.6.1 guest release. As documented in the upstream 0.6 upgrade guide, v0 GetKey, the app ID, socket paths, disk key and disk format carry over. An in-place upgrade keeping the same CVM/app/KMS identity retains webhook keys, backup encryption keys and database passwords; merchants do not need to re-pin keys or re-onboard, and backup prefixes stay the same. Only the opt-in v1 GetKey derives different keys for the same name; adopting it needs a separate key migration. On TDX, instance identity also stays the same.

Take and verify a fresh backup before the image upgrade: upstream does not test moving data disks between image versions. Retain the previous kit and deployment record. OS rollback uses PATCH /cvms/<id>/os-image with {"os_image_name":"dstack-0.5.12"}, then the previous kit to verify the old image and recover compose under the rollback compatibility policy. Keep the existing app-compose manifest version; changing it to "3" prevents rollback to 0.5.x.

Approval evidence for dstack-0.6.1#

Checked on 2026-10-16 against Phala Cloud CLI 1.1.22 and the official mkosi-os-v0.6.1 release:

• phala os-images --prod --all --json lists name: dstack-0.6.1, version: 0.6.1, is_dev: false, requires_gpu: false, and os_image_hash: ec43cb8c5e04b6bb6c53b33b16076a01d83d01f7f4b703f2f78a0d947d6247e6. phala api /teepods/available offers the same non-dev hash on every listed node, including prod5.
• The release's dstack-0.6.1.tar.gz has SHA-256 7583f51c717011b8086c2dd4d1e73fbbe7bf2d6d2420b90858d538c26efcca41.
• After extraction, sha256sum dstack-0.6.1/sha256sum.txt equals the approved image hash; (cd dstack-0.6.1 && sha256sum -c sha256sum.txt) passes for all nine listed files: ovmf.fd, bzImage, initramfs.cpio.gz, metadata.json, measurement.tdx.cbor, measurement.snp.cbor, measurement.gcp.cbor, measurement.aws.cbor, and measurement.aws.replay.json. The verified metadata says version: 0.6.1, is_dev: false, ovmf_variant: pre202505, and kernel_header_normalized: true.

dstack-verifier.sh first fetches the measurement archive from download.dstack.org/os-images/mr_<approved-hash>.tar.gz (404 at approval time). When unavailable, it uses the release asset above, verifies its archive checksum, then authenticates sha256sum.txt against the approved hash and verifies every listed file. The authenticated manifest and all nine verified files enter a fresh tmpfs mounted at /tmp/dstack-verifier/cache/images/<approved-hash>; unlisted release files and old measurement caches are never reused. This works with remote Docker daemons because the request and files are streamed through stdin.

The fallback downloads the full OS archive (~617 MB) on each invocation while the measurement archive is unavailable. Budget temporary storage and network time; for manual checks, set TMPDIR="$XDG_RUNTIME_DIR" to use runtime storage and remove it on exit.

Use verifier image ghcr.io/dstack-tee/dstack-verifier:0.6.1 at digest sha256:9f2c07a94efd3d8a1a89268e5faa35e91e3cb345b9c4ab859c86d35142f4f788 for the mkosi image's measurement documents. Live quote/TCB verification remains a deployment acceptance check; approving these bytes does not claim a live CVM was upgraded or attested.

The simulator pins guest commit 7f143148cc49f4a09799958835d310e050137cb3 with source-archive SHA-256 14ae67bcc941b9ec1af9ee17db9a0fc325d1e302c59f28f18695cc73bdc6a276.

One-time setup (HUMAN-ONLY, repository owner)#

In the operator's environment repository (self-hosting, "Your environment repository"); Phala's is this repository, through Deploy Phala's instance. Workflows run on GitHub-hosted ubuntu-latest runners.

1. Environments. Repository Settings > Environments: production and, for a pre-production instance with test routes only, staging (the names Deploy offers), deployment branches main only. Required reviewers are optional (Phala's repository has none: its plan does not offer them). Whoever dispatches Deploy is accountable; the run's actor, summary, and uploaded record are the audit trail.

2. Phala Cloud. Create an API key for the Environment's workspace and store it as the Environment secret PHALA_CLOUD_API_KEY, the only secret GitHub holds (a repository secret for a caller in another organisation, Deploy).

3. Object storage. Create a bucket (or prefix) per Environment in an S3-compatible store, such as Cloudflare R2, that the other Environment's keys cannot reach, and a read-write API token for it. The token is sealed into the CVM (below), never stored in GitHub.

4. Environment variables, per Environment: only deployment state, none of it attested.

   Name	Value
   PHALA_WORKSPACE	display name of the API key's workspace (preflight checks it)
   TOPUP_CVM_ID	empty until the first provisioning, then the CVM id from the run summary
   STAGING_PRODUCT_CVM_ID	staging only, for the optional staging reference product

   Every setting of the CVM is committed instead, in the Environment's directory, for example production/topup/ (Phala's: deploy/environments/phala-network/staging/topup), copied from the kit's deploy/environments/example (Attested settings): the domain, the object store's endpoint and backup prefix, the admin public key (from topup-sdk keygen --keyid admin/<Environment>-v1, a separate key per Environment; the seed stays with the admin), the RPC providers, and the routes. Deploy renders that directory and nothing else, so a setting changes only through a reviewed commit and a Deploy upgrade. The example ships production-eligible Chainlink USDC/USDT routes on Sepolia. PHA is staging/noncommercial only (price sources).

   No setting names a treasury or a transaction-signing key: treasuries are each account's own, set through the API, and the service sends no transactions.

5. Sentry (project admin; the Crons monitors create themselves on their first check-in):

   • Settings > Security & Privacy: keep Data Scrubber and Use Default Scrubbers on; turn Prevent Storing of IP Addresses on.
   • An alert for the environments staging and production (not *-restore) that notifies the on-call owner when an issue is created or regresses, with no level filter (most alert lines are warning events). Confirm that the Crons monitors are listed as connected.
   • Phala's pilot has Sentry Uptime disabled at the owner's request. External /healthz is checked by pilot_watch every two staffed hours for 72 h, then twice daily. The owner accepts the risk between checks. Other operators choose their own external availability monitoring.
   • The project DSN (Settings > Client Keys) is sealed as SENTRY_DSN (Sealing the secrets).

Release and deploy#

Releases#

A release is a v<version> tag of Phala Pay on a commit of main, the Cargo workspace version, which the SDKs share (CONTRIBUTING.md, "Releasing"). Prerequisites (repository settings, HUMAN-ONLY, admin; both in place): a refs/tags/v* ruleset restricting creation, update, and deletion to admins, and immutable releases, so a published v<version> always names the same commit and assets.

CI runs the real-service make cvm-rehearsal on every pull request and push to main, and again before Release builds anything. The payer submits a tx-hash hint through the reference product after broadcasting; hint failure retains scanner discovery. It asserts a credited payment and an acknowledged, signed event redelivery with the receiver's credit unchanged. Prices use local Chainlink, Uniswap V2 and sequencer fixtures plus hermetic exchange responses. The Ethereum mainnet price Anvil resumes twelve-second mining after seeding the thirty-minute TWAP window, keeping its baseline within EVM BLOCKHASH history. No live deployment, public price RPC or production credentials are involved.

The tag runs Release. It runs the whole CI workflow on the commit first, then builds each image on its own GitHub-hosted runner with verify-image.sh (Buildx v0.37.1 and BuildKit v0.33.0 pinned by digest; make verify-image rebuilds the same way), pushes it to ghcr.io/phala-network/ tagged v<version> (public packages: CVMs pull without credentials), and smoke-tests and attests it by the digest BuildKit reports for the push (--metadata-file), never by the tag. All three images are reproducible: the pushed build must have the digest of an earlier build, and anyone rebuilds it from the tag. postgres-walg pins the Debian snapshot and direct package versions, removes apt/dpkg timestamp logs and ldconfig's auxiliary cache, and uses BuildKit's filesystem timestamp normalization. gosu and WAL-G are rebuilt with a pinned, patched Go toolchain and dependencies; WAL-G keeps libsodium encryption and Brotli/LZO restore support. After comparison, Trivy scans each final digest and generates an SPDX SBOM; actions/attest-sbom signs it and attaches it to the digest in GHCR, alongside build provenance. verify-release.sh requires both attestations with the release workflow identity, source commit, and GitHub-hosted runner constraints. A stable release then publishes the SDKs at its version, @phala/pay, @phala/pay-react, and @phala/pay-server to npm, plus phala-pay to PyPI, with trusted publishing. The GitHub release's assets, each attested, are:

• images.json, each image's repository@sha256 by name (render.sh's --images);
• the deploy kit phala-pay-deploy-v<version>.tar.gz (build-kit.sh: LICENSE, deploy/, and docs/ of the tag, a tar identical to their git archive);
• phala-cloud-template.yml (The Phala Cloud template variant);
• deploy.sh, deploy.sh set to deploy this release, which generates an admin key with its Python SDK, the one-command deploy that https://pay.phala.com/deploy.sh (the latest release) and /deploy/v<version>.sh redirect to (self-hosting, "One-command deploy");
• SHA256SUMS.

verify-release.sh is the one verification of a release, run by Deploy and by hand (self-hosting, "Verify a release"). It stops at the first failure: the tag's commit must be in main's history, the assets must match SHA256SUMS, and every asset and image must have a provenance attestation of release.yml at the tag, on a GitHub-hosted runner, for that commit (gh attestation verify --source-digest). Each image must also have an SPDX 2.3 SBOM attestation under the same identity and source constraints. GitHub metadata reads and attestation downloads retry transport failures and HTTP 5xx up to four attempts with exponential backoff, within the 600-second verification budget. Metadata HTTP 429 responses retry only with a numeric Retry-After header; attestation downloads fail on 429 because gh does not expose its retry headers. Attestations are downloaded separately and verified once using gh attestation verify --bundle; checksum, signature, digest, and policy rejections fail immediately.

Deploy#

Deploy is the source of truth for what a deployment checks and does. It is a reusable workflow, which a repository calls at a release, with the same version:

uses: Phala-Network/phala-pay/.github/workflows/deploy.yml@<release commit or tag>

Its inputs are version, environment (the GitHub Environment), target (topup, or product for Phala's reference product), mode (provision or upgrade), and environment_dir (the target's directory in the caller's repository). The caller grants contents: read and attestations: read, and Deploy reads PHALA_CLOUD_API_KEY plus TOPUP_MAINTENANCE_PRIVATE_KEY_PEM for topup upgrade admission control. A caller in the Phala-Network organisation passes secrets: inherit, and the key is the Environment's secret (an Environment secret resolves empty in a called workflow without it, actions/runner#4453); a caller in another organisation, where inherit is not supported, passes it from a repository secret (self-hosting, step 5). It runs verify-release.sh of its own commit, which must be the release's, then only the verified kit's scripts: render, preflight, the deploy with the kit's pre-launch script, and the verification of the attested compose with the official dstack verifier; it uploads the rendered compose and the verification as the run's record. Deploy Phala's instance is Phala's caller, the same way; adopting a release is a pull request that changes its release.

The optional GitHub Environment variable PHALA_NODE_ID pins a new CVM to a Phala Cloud node: empty or unset lets Phala Cloud select it; otherwise the value must contain only digits, the node id from deploy/phala nodes list with that workspace's API key. Deploy passes --node-id only for mode: provision; an upgrade keeps the CVM's node. The run summary records the requested node (or auto) and the actual node's id, name and region from cvm.json. The CLI's JSON output is logged even if deployment fails, with its exit status preserved.

Planned upgrade admission and downtime#

Before a topup upgrade calls Phala Cloud, the workflow admin-signs POST /v1/admin/instance/pause with scope mutations, its run/attempt owner, and a 900-second lease. Configure the protected GitHub Environment's TOPUP_MAINTENANCE_PRIVATE_KEY_PEM secret (Ed25519 PEM) and TOPUP_MAINTENANCE_KEY_ID variable to match an entry in the running instance's attested maintenance_keys. External callers pass the declared signing secret alongside the Cloud key; organisation callers inherit both. This separate key authorizes only POST /v1/admin/instance/pause and POST /v1/admin/instance/resume; all other admin routes, including maintenance inspection, return audited 403 permission_denied. The operator's full admin key still works, but stays with the operator and is never a deployment secret. Restrict Environment access and deployment branches as for the Cloud key. Key material is temporary, never part of the uploaded deployment record, and is deleted after each signed request.

The pause and resume helper retries transport errors and HTTP 408, 429, 500, 502, 503, and 504 responses; each attempt is signed fresh, delays are at least 1 second, and the operation is bounded by a 60-second deadline.

Upgrade preflight also reads GET /cvms/<id>/available-os-images, which filters for the existing CVM's node, KMS allowlist and production/GPU rules. It requires an enabled production dstack-0.6.1 entry before any pause or upgrade call. If cvm-before.json already reports that image, Deploy skips the OS-image operation; new CVMs provision directly on it. Otherwise, after the initial pause, the locked CLI's raw API sends PATCH /cvms/<id>/os-image with {"os_image_name":"dstack-0.6.1"} and waits for the image name, running status and no operation in progress. This reboots the same CVM without changing compose, app id, instance id, node or sealed env. The poll must first observe a busy or non-running state: a new image name on an immediately idle, running CVM does not prove a reboot completed. If the transition is never observed, the bounded poll fails rather than submitting compose while the image operation may be pending. Deploy waits for the old compose's /healthz, then topup takes a fresh 900-second lease immediately before the compose deployment. The first lease disappears on process exit; between OS recovery and the second pause the old compose serves normally, a safe ordinary state. Product upgrades use the same image/health checks without topup's maintenance API.

For planning, assume two restart envelopes of about three minutes each: the prior compose-only staging upgrade took 163 seconds (deployment study). The OS-image reboot has not been measured by this PR, so six minutes total is an assumption, not a downtime guarantee. Each reboot has its own process pause/startup protection; the two 900-second leases are not one continuous pause and cannot be extended across process exit. The upgrade monitor keeps sampling through both reboots and the healthy interval until the step's stop file appears, bounded by its 3,600-second budget. It records each outage window and their total. Convergence ceilings are 15 minutes for OS change (including its PATCH), 10 minutes for old-compose health, 60 seconds for the fresh pause, and 5/15/10 minutes for compose submission/convergence/health. The deployment step allows 60 minutes; the overall job still stops after 75 minutes. A timeout requires inspection and recovery; it does not prove the image change was undone.

Generate an independent key per Environment on the owner's machine using the installed Python SDK CLI. Convert its seed to the PEM the workflow helper accepts:

topup-sdk keygen --keyid maintenance/production-v1 --seed-out maintenance.seed &&(umask 077 && { printf '302e020100300506032b657004220420'; tr -d '\n' < maintenance.seed; } \  | xxd -r -p | openssl pkey -inform DER -out maintenance.pem)

The CLI prints keyid and standard-base64 public_key. Commit only those public values in the Environment's topup/topup.yaml, next to admin_key, before adopting this workflow:

maintenance_keys:  - id: maintenance/production-v1    public_key: <public_key printed by topup-sdk keygen>

The list is optional (empty by default), at most eight keys. IDs and public keys must be unique and distinct from the admin key; missing/invalid public keys refuse configuration. Configure TOPUP_MAINTENANCE_PRIVATE_KEY_PEM from the complete maintenance.pem through the owner's secret-management flow, and TOPUP_MAINTENANCE_KEY_ID=maintenance/production-v1. Never upload the admin seed or PEM. Maintenance signatures use the same origin, digest, five-minute freshness, and single-use replay checks as admin signatures; successful changes record the actual key id.

Rotation: generate v2 independently, retain v1 and add v2's public entry in a reviewed config upgrade, using v1 to drain that upgrade. After health and resume pass, switch the Environment's maintenance PEM/key id to v2, verify a signed maintenance operation, then remove v1's public entry in another reviewed upgrade using v2. Keep the old credential until the first upgrade's cleanup completes. Do not rotate credentials during a running deployment.

New authenticated mutations return 503 service_maintenance, Retry-After: 5, before idempotent execution; reads and already admitted requests continue. Scanners and outbox workers continue until the existing graceful shutdown drains them. A replacement process starts with a startup mutation pause, lifted only by its first successful /healthz database check; the old process's health probes never lift its explicit deployment drain. The workflow then confirms resume with the deployment owner after health checks. Failed runs attempt to resume the serving process. Each process lease expires after at most 900 seconds according to its monotonic clock, even without a runner or expiry worker. Process exit discards its lease; startup health protects the replacement, so a failed deploy cannot persist maintenance. Pause/resume are audited using the existing audit table; maintenance adds no database migration or persistent lease to carry across process replacement. The release's rollback compatibility policy still governs recovery. Account, customer, treasury, and route incident pauses remain. See the stuck-maintenance runbook for inspection and manual clear.

Upgrade probes run from before the upgrade call through service readiness, sampling /healthz every 2 seconds with a 3-second timeout. Recovery closes a window and sampling continues. unavailability.json records each failure/recovery pair in outage_windows and their summed unavailability_seconds; first_failed_at and first_healthy_at retain the first outage's timestamps. The job summary shows all windows in PT and UTC and their total. Maintenance admission and clear responses are separate artifacts. An open window, cancellation or the monitor's time limit leaves the total unknown; observed_unavailability_seconds is then a lower bound from the samples already taken. Interrupted sampling is not a complete deployment measurement, and no observed failure is not a claim of zero downtime. Staging run 37215125136 took about 160 seconds (upgrade step to service readiness), which motivates the SDK's opt-in 300-second budget. This masks transient failures for clients; the CVM still restarts and raw HTTP callers must implement retries.

1. Merge the change to the environment repository's main: a setting, or a new version.
2. First deployment: Deploy with mode: provision, then set TOPUP_CVM_ID (or, for the product, STAGING_PRODUCT_CVM_ID) to the CVM id in the summary, seal the secrets, and create the DNS records it lists. A provision only creates the CVM, which must boot but shows error until it is sealed, so it proves nothing about its health: the acceptance step is the mode: upgrade run with the same release after the sealing, which requires the CVM running, /healthz, the attested compose, and the certificate evidence. If a provision run fails after the summary shows a CVM id, set the variable, seal the secrets, and run that upgrade; never provision twice.
3. Every later change: Deploy with mode: upgrade. An upgrade sends only the compose, so the sealed env stays. Rollback supports the immediately previous release (N-1) only, after the release rollback smoke passes. Keep the upgraded schema; never run down migrations. See Rollback compatibility before changing the version.

Production with live routes additionally needs the factory at its deterministic address on each of their chains (CONTRACTS.md, HUMAN-ONLY, only where it is missing) and a reviewed route PR putting the live routes into the compose. One production deployment serves both modes: live routes on mainnets and test routes (livemode: false) on test networks such as Sepolia. Deploy runs check-route-modes.sh on the rendered compose's routes, as topup reads them, and refuses a route whose livemode does not match its chain, a chain on neither of its lists, a local development chain, and any live route in staging. After the first deploy, in order: seal the secrets; verify the attestation; have the operator's Finance, Risk, and Operations approve the pilot limits (route bounds, each account's caps (limits: open quotes and their credit per account and per customer) and max_unfinalized_credit; architecture §14) and a passed restore drill (RESTORE.md); then onboard accounts with charges_enabled (Phala's instance adds its own onboarding policy).

Sealing the secrets#

The CVM's encrypted env holds the rendered compose's sealed names, its ${NAME:-} references. On initial deployment, the Phala Cloud CLI makes the env file's names the CVM's allowed_envs. instances add retains the source app-compose's allowlist; the restore variant uses the same names with its own read-only storage token in an explicit env file. In either case, verify-attestation.sh requires them to be among the sealed names (compose-policy.jq, allowed_envs_violations): another name is refused, since it would put in the env a value the compose does not reference. A sealed name left out is unset, like an empty one; a required secret is enforced where it is used, on every start: the postgres-walg entrypoint refuses S3 storage unless both AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are set, so neither PostgreSQL nor the backup job starts and the acceptance upgrade's /healthz fails, and topup refuses a keyed provider without its key. The names are part of the app-compose, so sealing with the names a CVM already allows keeps its compose hash. The sealed names are:

• AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, the object store's token (declared in compose.yaml);
• SENTRY_DSN, required for production; staging and templates may leave it empty to turn Sentry off;
• TOPUP_RPC_ANKR_KEY, shared by production and staging; TOPUP_RPC_INFURA_KEY in staging only (including its mainnet price chains); TOPUP_RPC_BLOCKPI_ETHEREUM_KEY and TOPUP_RPC_BLOCKPI_BASE_KEY in production only. Each replaces {key} in its endpoint's attested URL and is declared in that environment's compose.yaml and restore-check mapping.

Each line is NAME=VALUE, the value as is. The Phala Cloud CLI reads the file as dotenv does, which would cut a value at # and strip quotes and surrounding whitespace, so preflight refuses a value with any of them. A key is at least 8 characters of A-Z a-z 0-9 - . _ ~. topup refuses a provider whose URL and key disagree: a {key} without a key, or a key without a {key}. Redaction keeps the URL and the key out of every log line and error. A new CVM waits for its secrets: PostgreSQL initializes a cluster only after listing an empty backup prefix (RESTORE.md). deploy/validate-compose.sh fails if staging's sealed names change, since that needs the re-seal below.

HUMAN-ONLY, owner, from their own machine, with the deployed release's verified kit in kit/, the environment repository at the deployed commit, and the rendered compose from the run's artifact (the provision summary prints these commands):

docker pull "<the compose's phala-pay image>"   # --offline pulls nothing; it runs topup config check in itkit/deploy/preflight.sh --env .env.ENV --compose docker-compose.ENV.yml \  --environment-dir ENV/topup --offline      # .env.ENV: mode 0600kit/deploy/phala envs update "$TOPUP_CVM_ID" -e .env.ENV

Offline, preflight checks every provider's key against its URL (topup config check --secrets) without printing it. Deploy's preflight has no keys, so with a keyed provider it skips the asset chain checks; run preflight without --offline (with --workspace and --os-image) to run them with the keys.

The CVM restarts, /healthz answers, and backups have started once a WAL segment younger than two minutes is listed (aws s3 ls "${WALG_S3_PREFIX%/}/wal_005/" --endpoint-url "$AWS_ENDPOINT" | tail -1). Re-seal the complete secret set the same way whenever a secret changes, including unchanged names. A partial update can unset another chain's credentials. Never change an attested setting with envs update.

A new sealed name (a new keyed provider's TOPUP_RPC_<ID>_KEY) is not among the CVM's allowed names, which Deploy upgrade keeps, so after the upgrade it would be unset and topup would refuse to start (a {key} without a key). Once, before that upgrade, run only the envs update above with .env.ENV holding the new compose's sealed names: the running compose ignores the new name, and preflight against it would refuse it. Every configured endpoint requires its explicitly declared sealed key.

Attested settings#

A value in the encrypted env is outside the attestation: whoever can run phala envs update could change it without changing the compose hash. So the env holds only the secrets above, and every other setting is written into the compose that the attestation covers, from reviewed files. There are four kinds of input, each with one home (design):

render.sh renders them with Docker Compose v2.26.0 for stable attested bytes (the 0.6.1 CVM runs v2.26.1), pinned by sha256 (pinned-compose.sh):

kit/deploy/render.sh --images images.json --gateway-domain gateway.dstack-pha-prod5.phala.network \  production/topup >docker-compose.production.yml

• It merges the stack, the environment overlay, and the variant overlay with config --no-interpolate, which keeps the sealed ${NAME:-} references. The overlay may set only the settings in the table above: the merge with it may differ from the merge without it only in those environment keys, so it can change no image, command, entrypoint, config, mount, port, or service.
• It pins each image to the release's digest, and every image must then be one of the release's or one the kit pins by digest (dstack-ingress).
• It inlines each config file (the environment's topup.yaml, the PostgreSQL init script) as content named after its digest. A changed file therefore changes the definition of exactly the services that mount it, and Compose recreates them.
• It prints Compose's canonical YAML under the project name dstack, which is the name dstack gives the stack it runs in /dstack, so the volumes keep their names.
• Before printing, it applies compose-policy.jq, the policy that validate-compose.sh, preflight, and verify-attestation.sh apply to the same artifact:
  • the variant's exact services;
  • the only published port;
  • the dstack socket;
  • each credential volume on tmpfs, mounted by exactly its services, read-only but for keys;
  • smokescreen with its exact deny list;
  • restore isolation;
  • each sealed name only as the whole value of its own environment key, so a sealed value can never fill the origin, the admin key, or an RPC host.

topup.yaml itself is validated by topup config check, which reads no secret. To change a setting, change its file by pull request and run Deploy upgrade. Phala's reference product renders the same way from deploy/environments/phala-network/staging/product (its config.json and domain).

The Phala Cloud template variant#

render.sh --template renders the Phala Cloud template, a one-click testnet quick start (self-hosting, "The Phala Cloud template"), from environments/phala-cloud-template: Phala's staging routes and the same sealed Ankr and Infura endpoint keys. Each release publishes it as phala-cloud-template.yml, and Phala Cloud's template, once published, is that file byte for byte. compose.template.yaml removes dstack-ingress and restore-check, and topup publishes 80:8080, which the Phala Cloud gateway serves as https://<app-id>.<gateway-domain>.

A template's per-deployment values come from the CVM's env, never from topup.yaml, whose $ are escaped like every environment's. The policy's template variant allows a runtime reference as the whole value of exactly these environment keys, and nowhere else:

The backup and restore helpers also accept these operational limits (the defaults are deliberately finite and are passed to both postgres and backup; an environment may override them):

The heartbeat commits every 15 seconds so WAL upload latency does not consume the full 60-second recovery point budget. These names are the limits used by the committed WAL-G backup/restore helpers.

The template's topup.yaml leaves out public_origin and admin_key.public_key; topup run refuses to start unless each comes from exactly one place, and the policy fixes topup's command, so a service compose can never take either from its env. These values are not attested: whoever controls the workspace can change them without changing the compose hash. Everything else is: the services, the routes and providers, smokescreen, the credential volumes, and the sealed names. verify-attestation.sh checks a template CVM with the variant template. The Phala Cloud CLI, and the deploy form likewise, puts in allowed_envs only the values it is given, so a template CVM deployed without the optional SENTRY_DSN has the other sealed names only; the verifier accepts that (Sealing the secrets) and refuses any other name, DSTACK_APP_DOMAIN included. A missing form value is refused at startup by the service that reads it (the table above). An instance with merchants uses the service variant.

No restore-check path. A template instance cannot be restored through RESTORE.md's restore-check variant. That variant renders from an environment whose backup prefix, origin, and admin key are attested literals, and its guarantees rest on them: the verification instance proves which prefix it restored, reads it only with its own read-only credentials, and is served under an explicit origin. The template's prefix, origin, and key are runtime values, so no attested restore-check compose could prove any of that, and render.sh refuses --template with --restore-check. A template instance's data survives only as its app's backups: another instance of the same app, deployed with the same form values, restores the newest backup when it first starts, without a read-only verification first. That is acceptable for a testnet quick start and is why an instance with merchants uses the service path.

RPC providers#

Every chain used for payments or prices has one read endpoint (Ankr) and one verify endpoint (BlockPI in production, Infura in staging), configured in top-level rpc. Both independently derive money evidence; state calls use canonical EIP-1898 hash pins. There is no in-process failover.

URLs use {key} placeholders and explicit sealed_key names. Staging uses TOPUP_RPC_ANKR_KEY and TOPUP_RPC_INFURA_KEY; production uses Ankr plus TOPUP_RPC_BLOCKPI_ETHEREUM_KEY and TOPUP_RPC_BLOCKPI_BASE_KEY from one BlockPI account. BlockPI uses max_log_blocks: 1000 below its measured free 1,024-block limit. Seal the complete secret set; both topup and restore-check declare the environment's keys in compose mappings.

topup config check --secrets FILE validates the strict public schema and keys without network access. topup rpc check --config FILE checks both endpoints, complete recipient log filters, typed receipts, canonical state and reviewed contracts. Deploy preflight runs it through the candidate rendered compose and sealed env, including restore-check. Missing keys or mappings block preflight. Endpoint failure pauses only its chain or affected price feature.

A dual-source agreed contract-code mismatch freezes its chain with contract_code_mismatch. Disagreement or failed requests make it not-ready without freezing. A fresh passing dual contract check is required before the existing audited admin lift. Other chains and /healthz keep running. See configuration and the RPC runbook.

Custom domain#

The API has one public origin, public_origin (https://$DOMAIN, where $DOMAIN is dstack-ingress's DOMAIN in the environment overlay), a stable name the owner controls, not the CVM's gateway URL, which changes with the node and the app id: merchants call it and pin its attestation, treasury challenges (EIP-4361) name it as their domain and uri, and the admin API verifies every signed @target-uri against it. The official dstack-ingress 2.8 (ghcr.io/dstack-tee/dstack-ingress, pinned by the digest of its release notes; gh attestation verify oci://ghcr.io/dstack-tee/dstack-ingress@sha256:60e682449534715a2f2e5b72326ca4229209c8e26c931c6f34d27e02200ac086 --owner Dstack-TEE shows it was built from tag dstack-ingress-v2.8) publishes the compose's only port, 443. The gateway passes the TLS connection for the domain through to it, it terminates TLS inside the CVM, and forwards the stream with PROXY v2 to topup:8082. It gets the Let's Encrypt certificate with tls-alpn-01 through that same port, so the CVM holds no DNS credentials; the ACME contact is unset because the account document is published. Like keys and topup it mounts the dstack socket (its instance id and the evidence quote), which is why it is pinned by digest and attested with the compose.

A reference-product CVM is served the same way on its own overlay's DOMAIN (Phala's instance, "Staging reference product").

Client address passthrough (v0.10.9). The service sets gateway pp:true on 443, ACCEPT_PROXY_PROTOCOL: "true", SEND_PROXY_PROTOCOL: "true", and TARGET_ENDPOINT: topup:8082 together in one attested app-compose. Topup runs --proxy-protocol-bind 0.0.0.0:8082; this private listener requires PROXY v2 and preserves its source as Axum connection information. It publishes no port. Only ingress sends traffic to it; internal topup healthcheck and other direct HTTP checks keep using plain HTTP on 8080. Public /healthz, upgrade-monitor, and certificate-evidence verification keep calling https://$DOMAIN through the gateway and ingress; their clients send ordinary HTTPS and the gateway supplies the header.

The product keeps product:8089, with both PROXY switches omitted. Both variants pin ingress 2.8. The configured 5s / 30s / 30s HAProxy deadlines and numeric ports pass 2.8's strict validation. Renewal settings are omitted, so upstream uses integer RENEW_INTERVAL=43200 and an unset RENEW_DAYS_BEFORE; explicit overrides must be positive whole numbers, not 12h or 30d. Boolean settings accept only exact true / false, including ACME_STAGING / CERTBOT_STAGING and DETERMINISTIC_TLS_KEY; those and the omitted PROXY booleans default to false. render.sh requires and sets GATEWAY_DOMAIN for both service and product variants, as 2.8 requires. Neither variant sets ROUTING_MAP; each uses the single TARGET_ENDPOINT above. In tls-alpn-01 mode, ingress preserves the client address across its loopback TLS frontend; its ACME responder and evidence HTTP server receive no PROXY header, so issuance, renewal, and /evidences/ retain their normal protocols. See the pinned upstream README's Client address section.

phala-cvm.sh's compose-update CVM_ID COMPOSE PRE_LAUNCH PORT_POLICY_JSON OUT reads GET /cvms/{id}/compose_file, replaces the compose, pre-launch script, port policy and private-log flags, then POSTs /cvms/{id}/compose_file/provision and PATCHes /cvms/{id}/compose_file with only the returned compose_hash. Before the update it reads the CVM's current compose hash with cvms get; if the provisioned hash matches, it reports unchanged; nothing to commit and skips the PATCH. A registered hash alone does not skip the commit. It keeps the current allowed secret names and KMS/storage settings, sends no encrypted env or environment-update fields, and preserves the owner-sealed values. Calls share a five-minute stage deadline and each API call is capped at 30 seconds; failures print the CLI/API diagnostic. OUT records the provision response, including its hash; callers wait for that exact hash to settle before verifying attestation or probing the upgraded service. Deploy upgrades and both provisioning follow-ups use this path because CLI 1.1.22 cannot set port_policy directly. The service policy is {"ports":[{"port":443,"pp":true}],"restrict_mode":false}; restore-check inherits that same policy through instances add. PROXY on 443 is inert there: no ingress or listener runs on 443, and only plain HTTP on 8081 is published. Product and template keep pp:false with the same port and restrict_mode:false. This does not restrict the template's 80 or restore-check's 8081. Attestation verification checks the variant's exact app-compose field; changing it changes the attested compose hash.

Log privacy exception. HAProxy TCP access logs can contain client IPs, including real client IPs with service passthrough enabled. This is an accepted exception for private CVM logs, with public_logs=false; do not export those access logs to Sentry. Topup's tracing and Sentry still never contain client IPs. The topup PROXY listener only carries connection information; there are no new per-IP limits, including for transaction-hash hints.

Rollout and rollback. First deploy and verify v0.10.8 (dstack 0.6.1, ingress 2.8, and phase 1 with PROXY off) on staging and production. Then release phase 2 as v0.10.9 and follow the staging verification recipe before production topup. Product and template keep PROXY off; restore-check inherits the service's policy on its unbound 443. Gateway pp:true, ingress ACCEPT/SEND, the 8082 target, and topup's private listener must change together in one app-compose. Enabling only one hop breaks every connection, including health and certificate checks.

Rolling back from v0.10.9 to an earlier release requires the target release's compose-update path from a v0.10.8 or newer kit, which explicitly restores pp:false together with PROXY-off ingress settings. Never roll back directly to v0.10.7 or older: their deployment path can leave gateway PROXY enabled against an ingress that expects ordinary TLS. Verify health, attestation, and ingress evidence after rollback.

HUMAN-ONLY, owner of the domain's DNS zone (any DNS provider; the Deploy summary's wording assumes Cloudflare, as in Phala's instance), once per CVM instance. Every Deploy run lists the records for its target's domain ($DOMAIN, the DOMAIN of its environment overlay), in the tls-alpn-01 format of the pinned README:

• Not proxied (on Cloudflare, DNS only, grey cloud): a proxied name resolves to the proxy, so neither the CA nor a client reaches the gateway.
• The TXT names the instance, not the app: the CA's validation must reach the one instance that holds the ACME order. An upgrade keeps the instance id; a new instance (Resume) serves the domain only after the TXT carries its id.
• Until both records resolve, dstack-ingress serves a self-signed placeholder and requests no certificate, so Deploy's /healthz wait on the domain fails.
• CAA is optional. An existing CAA record on the domain or a parent domain must permit tls-alpn-01 (Phala's phala.com has none as of 2026-09-26). Pinning accounturi to the account that verify-ingress-evidence.sh prints also means that losing the ingress_certs volume (a new account) blocks renewal until the record is updated.

Certificate evidence. dstack-ingress publishes, at https://$DOMAIN/evidences/, the ACME account, the certificate, sha256sum.txt over both, and a TDX quote whose report_data is the hash of sha256sum.txt. Its evidence server listens on the ingress container's loopback (port 80) and is reached only through 443, where HAProxy routes /evidences and /evidences/... to it (not paths such as /evidences-report), so there is no evidence port to publish. verify-ingress-evidence.sh checks the chain with the official verifier (the quote is app APP_ID's and binds the files) and that the domain serves exactly that certificate; Deploy runs it after every topup upgrade:

kit/deploy/verify-ingress-evidence.sh "$DOMAIN" "$APP_ID"

Ingress 2.8 republishes evidence before HAProxy starts and after issuance. If regeneration fails, it withdraws the previous evidence (/evidences returns HTTP 404) and retries without reloading HAProxy. The verifier treats 404 as not available yet and retries each file up to three times, with the existing 30-second request limit; Deploy retries the whole check within its existing ten-minute deadline. Ingress also publishes attestation.json through /v1/Attest on the pinned dstack 0.6.1. Older guests fall back to quote.json, which the verifier above continues to check. The quote's compose hash records the app-compose at publication time.

Database credentials#

PostgreSQL runs inside the CVM, so its passwords are derived there, never supplied. keys (topup keys) derives the WAL-G key (get_key("backup/v1")) and the owner and application passwords (hex of get_key("db/owner/v1"), get_key("db/app/v1")) into tmpfs volumes; each service mounts only what it needs, read-only, and validate-compose.sh enforces it:

This is separation by mount and by database role, not by KMS: keys, topup, restore-check, and dstack-ingress mount the dstack socket, and any holder of it can derive any key path. PostgreSQL, WAL-G, and migrate never see the socket, and topup never gets the owner login's files. migrate and restore-check refuse any login but the database owner (DATABASE_URL).

The same app id derives the same passwords, so a replacement or restored instance logs in unchanged. Rotation is an ALTER ROLE to a db/*/v2 value and a new compose.

Sentry#

The service reports to Sentry itself, only while SENTRY_DSN is non-empty (a malformed DSN stops topup run). The release is the source commit compiled into the image (SOURCE_COMMIT), and the environment is topup.yaml's environment, both attested.

• Events: every ERROR line and panic, grouped by message, at most one event per issue every 10 minutes. An event holds the log line's fields minus account_id; no request data, no RPC URL, no spans, send_default_pii off.

• Alerts: lines tagged with an alert name, fingerprinted by the name and its grouping tags (route, state, check, chain, scope), with a runbook tag. The names and their runbooks are in the runbooks index.

• Crons: each loop checks in and so creates its monitor:

  Monitor	Checks in	Margin
  topup-fast-scanner-<chain_id>	after fast discovery (every five minutes); three failures alert	2 min
  topup-checkpoint-scanner-<chain_id>	after each independent dual checkpoint check (every ten minutes); one failure alerts	2 min
  topup-coverage-scanner-<chain_id>	after each dual coverage round (hourly), error while coverage fails	2 min
  topup-pump-<n>, topup-outbox-test, topup-outbox-live	each iteration or poll, every minute	5 min
  topup-lock-expiry	after each successful expiry scan, every minute	5 min
  topup-finality-watch	after each finalized advance's passes, and every minute	5 min
  topup-reconciler	ok after a complete round or one skipped because nothing newly finalized, error after failed checks, every 10 min	10 min
  topup-backup	ok while the WAL-G success marker is at most 120 s old, else error; 3 errors open an issue	2 min

  After upgrading, delete or mute the old topup-scanner-<chain> monitors. Independent dual checkpoint checks publish every ten minutes for finality, reversal and refund completion; hourly coverage does not set their cadence. Custody remains hourly. The coverage-lag warning is two hours; see scanner lag.

• External health: Phala's pilot uses manual pilot_watch rather than Sentry Uptime, at the owner's request. A down CVM also misses Crons check-ins. A metrics collector with a metrics-only key is the planned replacement for manual metrics evaluation.

• Egress: topup sends HTTPS to the DSN's ingest host.

A restore-check instance runs no loop and reports as <environment>-restore, so it never checks in or raises an alert of the live environment.

Measuring RPC usage#

Every JSON-RPC call the service sends is counted by configured provider id (provider-a, provider-b, never a URL), chain id, and method (16 named methods; any other counts as other), from process start. The admin-signed GET /v1/admin/metrics returns the counters in the Prometheus text format; with the admin helper from the runbooks:

admin GET /v1/admin/metrics

topup_rpc_calls_total{provider="provider-a",chain_id="11155111",method="eth_blockNumber"} 8012topup_rpc_calls_total{provider="provider-a",chain_id="11155111",method="eth_getLogs"} 7390topup_rpc_calls_total{provider="provider-b",chain_id="11155111",method="eth_getTransactionReceipt"} 214topup_rpc_calls_since_seconds 1790500000

A day's usage is the difference of two readings a day apart (or a counter over now − topup_rpc_calls_since_seconds, scaled to a day); counters restart at zero with the process. A production CVM has no collector, so read it on staging, or occasionally on production, with the admin key.

Cost formula. For one provider and one chain, with n(m) the calls a day of method m and p(m) the provider's price of one call of m (compute units, credits, or currency: the provider's current price list is the input, not this document):

cost per day   = Σ_m n(m) × p(m)cost per month = 30 × cost per day

Chain reads §5.2 accounts for fixed 300-second discovery, independent 600-second checkpoints, 3,600-second coverage, bounded backfill and six-hour catch-up. Compare measured call counters with each provider's dashboard. Production reads with Ankr and verifies with BlockPI; staging stays on Ankr + Infura. Merge and preflight the production allocation and complete sealed mappings before launch §2; no dRPC move or staging upgrade is required by this allocation. Production uses the approved D=115/H=200/Q=150/factory=150/Safe=10 caps; staging keeps its existing caps. Later increases require a separately reviewed budget. Production-attested comments are updated before provisioning (launch §1.8). Each fresh quote snapshot costs one read call, 240 Infura verify credits in staging or 60 BlockPI RU in production; the DB enforces the configured quote_snapshots_per_day (150 production, 60 staging) per price chain per UTC day. RPC operations includes hourly dual custody on all nine chain/token routes (three production, six staging), refunds, unresolved finality stock, slow confirmation lane L and proof work. Parallel-mode production caps are D=115/day, H=200 hint tasks/UTC day and Q=150 fresh snapshots/price chain/UTC day, with factory=150/day and Safe=10/day. Staging retains D=20/H=80/Q=60/factory=20/Safe=2. H/Q default to 80/60 when unset. D, factory and Safe counts are operational caps with monitoring and stop actions. L accepts normal-window exhaustion solely from height lag, credits without a checkpoint wait and does not count toward S. Current L stock and rolling-24-hour entries are each ≤1 per environment across payment chains. Monitor topup_confirmation_slow and topup_confirmation_slow_entries_24h: L remaining non-empty for ten minutes alerts on provider lag; either sum above one triggers its capacity alert and affected-chain quote pause. See the confirmation budget and retry caveat.

Attestation, ingress, and egress#

HUMAN-ONLY, verifier, before onboarding any account. Deploy already verifies the attested compose; to re-check a CVM with the deployed release's verified kit in kit/, the run's rendered compose, and PHALA_CLOUD_API_KEY and DOMAIN (the host of topup.yaml's public_origin) exported:

kit/deploy/phala cvms get "$CVM_ID" --json > cvm.jsonkit/deploy/phala cvms attestation "$CVM_ID" --json > attestation.jsonAPP_ID=$(jq -er '.app_id' cvm.json) && GATEWAY_DOMAIN=$(jq -er '.gateway.base_domain' cvm.json)curl -fsS "https://${APP_ID#0x}-8090.$GATEWAY_DOMAIN/prpc/Info" > info.jsonEXPECTED_OS_IMAGE_HASH="$(cat production/topup/os-image-hash)"kit/deploy/verify-attestation.sh attestation.json info.json "$APP_ID" docker-compose.ENV.yml service \  "$EXPECTED_OS_IMAGE_HASH"export ORIGIN="https://$DOMAIN"   # topup.yaml's public_originkit/deploy/verify-ingress-evidence.sh "$DOMAIN" "$APP_ID"

verify-attestation.sh runs the official dstack verifier (dstack-verifier.sh, ghcr.io/dstack-tee/dstack-verifier:0.6.1 pinned by digest: TDX quote and TCB, RTMR3 event-log replay, OS image measurements). It requires the approved OS image hash, TCB UpToDate, the app id, and a compose hash whose app-compose holds exactly the rendered compose. The compose hash is the hash of the full app-compose JSON the Phala CLI builds (with allowed_envs and the CVM options), not of the YAML file, so every upgrade's hash is new and goes to merchants (docs/integration.md §5.3). It then requires allowed_envs to name only the compose's sealed names, and the variant's compose-policy.jq: for the service, the only published port is dstack-ingress on 443 (tls-alpn-01, PROXY v2 forwarding to topup:8082, for the host of public_origin); for the restore-check variant it is topup on 8081, with no ingress.

The app-compose also carries a pre-launch script, which the guest sources before docker compose up. Deploy sends the kit's phala-cloud-pre-launch.sh with --pre-launch-script (Phala Cloud's own v0.0.20, byte for byte, reviewed from an attested app-compose; its source is not published), and verify-attestation.sh requires exactly that file.

An account's webhook keys come only from the nonce-bound attestation, fetched with a secret key of that account and mode (merchants run the same check, docs/integration.md §5.3):

export NONCE="$(openssl rand -hex 32)"curl -fsS -H "Authorization: Bearer $SECRET_KEY" \  "$ORIGIN/v1/attestation?nonce=$NONCE" > public-attestation.jsonjq '{quote: null, attestation: .tdx_quote}' public-attestation.json |  deploy/dstack-verifier.sh > public-verification.jsonjq -e --arg app "$(jq -r '.app_id | ltrimstr("0x") | ascii_downcase' cvm.json)" \  --arg compose "$(jq -j '.compose_file' attestation.json | sha256sum | cut -d' ' -f1)" \  --arg report_data "$(jq -r '.report_data' public-attestation.json)" '  .details.tcb_status == "UpToDate" and .details.app_info.app_id == $app  and .details.app_info.compose_hash == $compose  and .details.report_data == $report_data + ("0" * 64)' public-verification.json

Then check that report_data binds the nonce, the account, the mode, and the listed webhook keys with the Python SDK's topup_sdk.verify_attestation_binding(response, nonce, expected_account=…, expected_livemode=…) (architecture §14 defines the construction; TopupClient.attestation runs it on every fetch).

Ingress: the attested compose must publish only dstack-ingress on 443; confirm /openapi.json at public_origin with a valid certificate, its certificate evidence, and that PostgreSQL and topup's port 8080 are unreachable. The admin API verifies every signed @target-uri against public_origin, so a correctly signed admin request answered 401 usually means the URL differs from it. Egress (HUMAN-ONLY, cloud network authority; dstack has no hostname allow-list): restrict outbound traffic to the RPC providers' hosts, the price sources, sanctionslistservice.ofac.treas.gov, wc2h-sls-prod-public-published.s3.us-gov-west-1.amazonaws.com, the object storage host, the Sentry ingest host, DNS, the Phala/dstack platform endpoints, and public addresses on ports 443 and 80 for webhooks (merchants register their own endpoints, so their hosts cannot be listed; webhook egress filters the addresses), and record the rules.

Webhook egress#

Merchants register their own webhook URLs (/v1/webhook_endpoints), so a URL may name any address, including the CVM's own network or a cloud metadata service. Every delivery therefore leaves through the smokescreen sidecar (--webhook-proxy http://smokescreen:4750), Stripe's smokescreen and the only IP filter (design §8): it resolves the host itself and refuses, with 407 before connecting, every address that is not publicly routable. The service checks only a URL's scheme and port when it is registered (https on 443; in test mode also http on 80), follows no redirect, and times out after 20 s; run refuses to start without the proxy unless its own origin is http (local stacks).

• Binary. Stripe publishes no image, so the phala-pay Dockerfile builds smokescreen v0.1.0 (commit 609eb8931420453daf5893509be0b25b21bd9edb) with its vendored modules in golang:1.27-trixie pinned by digest, reproducibly, and the sidecar runs it from the phala-pay image: it is pinned and attested with the service's digest.
• Policy. Its defaults refuse loopback, private (10/8, 172.16/12, 192.168/16, fc00::/7, which holds AWS's IPv6 metadata fd00:ec2::254), link-local (169.254/16 with the metadata address 169.254.169.254, fe80::/10), CGNAT (100.64/10, with Alibaba's metadata 100.100.100.200), multicast, unspecified, and IPv6 that embeds IPv4 (NAT64 64:ff9b::/96, 6to4, Teredo); an IPv4-mapped IPv6 address (::ffff:a.b.c.d) is checked as the IPv4 address it maps. The compose adds --deny-range for what those defaults count as global: 0.0.0.0/8, 192.0.0.0/24, 198.18.0.0/15, and 240.0.0.0/4, and repeats 100.64.0.0/10 and 169.254.0.0/16. A --deny-range of ::ffff:0:0/96 must never be added: Go matches it against every IPv4 address and it would refuse all webhooks. No allow-range, ACL, or --unsafe-allow-private-ranges is set, and validate-compose.sh fails if one is.
• Test. tests/smokescreen.sh runs the sidecar from an image with the compose's own command and checks, through plain requests and CONNECT tunnels, that private, loopback, metadata, CGNAT, 0/8, IPv4-mapped, NAT64, and unique-local targets are refused and a public address is not; CI runs it on every image build.
• Local stacks (the sandbox, the CVM rehearsal) deliver directly (no --webhook-proxy, under an http --public-origin), because their receivers listen on private compose addresses.
• A merchant's URL that resolves to a refused address fails like an unreachable one: retried with backoff (probed about once an hour), never disabled, and visible to the merchant as the endpoint's pending_deliveries, oldest_pending_at, and last_attempt, and as each event's pending_webhooks (GET /v1/events?delivery_success=false); the daily report lists endpoints failing longer than failing_for_hours.

Contracts#

The forwarder factory is permissionless and deterministic (design D3): the committed build deploys, through the Arachnid proxy, the factory 0x45466D37587E6E46DC35eB96b74ba3D3b1E5b747 and its implementation 0x49F2F1F1a25269Ea0C6FF2AB1C7B09dCBE9c5bA9 at the same addresses on every chain (contracts/local-test-vectors.json; the build since the per-target gas bounds of #202). A route names only the factory; topup run refuses to start unless the chain holds exactly that build's code there, so the factory is deployed on a chain before any compose with a route on it.

An operator reuses the factory wherever it is deployed (it is on Sepolia and Base Sepolia; verify-deployment.sh below checks a chain read-only) and deploys it only on a chain where it is missing. HUMAN-ONLY, deployer with a funded throwaway EOA, once per chain, from a clone of Phala Pay at the release's tag with its submodules, since deploying builds the contracts (CONTRACTS.md has the checks each script makes; $SEPOLIA_RPC_A and $SEPOLIA_RPC_B are two providers):

read -rsp "Deployer private key: " PRIVATE_KEY && printf '\n' && export PRIVATE_KEYdeploy/contracts/deploy-proxy.sh --rpc-url "$SEPOLIA_RPC_A"          # the Arachnid proxy existsdeploy/contracts/deploy-factory.sh --rpc sepolia/a="$SEPOLIA_RPC_A" --dry-run   # prints both addressesdeploy/contracts/deploy-factory.sh --rpc sepolia/a="$SEPOLIA_RPC_A" --broadcastunset PRIVATE_KEYdeploy/contracts/verify-deployment.sh --rpc sepolia/a="$SEPOLIA_RPC_A" \  --rpc sepolia/b="$SEPOLIA_RPC_B" > sepolia-contract-verification.jsonjq -e '.passed == true' sepolia-contract-verification.json

The dry run must print the two addresses above; if the factory already has that code (anyone may deploy it), the broadcast sends nothing. Verify contracts (Actions) runs the existing verify-deployment.sh daily in independent staging and production matrix jobs, with fail-fast: false. It derives chains and contract addresses from the resolved configs, maps chains through contracts/networks.json, and uses the explicit keyless public RPC mapping, separate from the service's keyed read/verify pairs. Proxy bytecode, implementation/factory bindings and sample forwarders are checked, then route addresses must match the report. It also runs verify-safe.sh daily for the committed Sepolia treasury Safe. Retain independent operator checks for all production Safes.

The reviewed PENDING_DEPLOYMENT_CHAINS list (now empty) permits a chain only when both configured addresses return exactly 0x, with a warning and job-summary line on every such run. Any non-empty code goes through full verification. Remove each chain in the deployment-evidence PR required by launch §1.5, which also updates CONTRACTS.md. Repeat operator checks for mainnet deployment.

Operator onboarding#

Accounts are created only by the operator, after due diligence done offline (design D8); there is no signup, dashboard, or email. The operator creates the account, decides live access, sets platform limits, and issues first or recovery keys; it never sets a treasury, registers a merchant's endpoint, moves funds, records refunds, or resends a merchant's events. Every admin call is RFC 9421-signed with the admin key and audited, and each change to an account is announced to it as an event (account.updated, api_key.created). With the runbooks' admin helper, in order:

1. Due diligence, offline (HUMAN-ONLY, operator): the business, its owners, sanctions screening of the entity, owners, and intended treasuries, jurisdiction, and the signed merchant agreement, under the operator's own policy (Phala's: onboarding policy). The product stores only a reference, the date, and the reviewer.

2. Create (HUMAN-ONLY, admin key holder, after attestation). The answer holds the account id acct_… and, in api_keys, its first secret test key (ppay_sk_test_…) and, with charges_enabled, live key; each secret is shown only here:

   sh

   (umask 077 && admin POST /v1/admin/accounts "$(jq -cn '{name: "<merchant name>",    contact: {name: "<name>", email: "<security email>"},    due_diligence: {reference: "<review reference>", reviewed_at: "<YYYY-MM-DD>",                    reviewed_by: "<reviewer>"},    charges_enabled: false, reason: "<why>"}')" > ~/onboarding/account.json)

   The admin seed converts to the PEM the helper signs with once: (umask 077 && { printf '302e020100300506032b657004220420'; tr -d '\n' < admin.seed; } | xxd -r -p | openssl pkey -inform DER -out admin.pem).

3. Key hand-over (HUMAN-ONLY): send the account id and key to the recorded contact through an encrypted channel, then delete account.json. The merchant rolls the key at once (POST /v1/api_keys/{id}/roll), keeps secret keys offline for administration, and runs its servers with a restricted key (POST /v1/api_keys {"type": "restricted", "permissions": […]}, ppay_rk_…), which cannot manage keys, treasuries, endpoints, webhook keys, or account settings (docs/integration.md §5.4). Everything else the merchant does itself through the API or SDKs: treasury, payment settings (a new account accepts nothing until it configures them), webhook endpoint and webhook key pinning, quotes pause, refunds, and sweeps. The operator sets no merchant term: it bounds them in each route's merchant section (docs/architecture.md §14), and reads an account's settings and effective config with admin GET "/v1/admin/accounts/$ACCOUNT".

4. Live enablement (HUMAN-ONLY, admin key holder), when the review allows it: the answer holds the first live key, handed over as in step 3. Live payments also need a proven live treasury.

   sh

   admin POST "/v1/admin/accounts/$ACCOUNT" '{"charges_enabled": true, "reason": "<review reference>"}'

5. Exposure cap: max_unfinalized_credit caps, in cents and per mode, the credit of the account's deposits credited before they are final (default 100 000, $1 000; design, ledger correctness amendment); a deposit past it is credited at finality instead, and 0 credits every deposit at finality. Raise it on the merchant's request once its volume and its ability to claw back justify it:

   sh

   admin POST "/v1/admin/accounts/$ACCOUNT" '{"max_unfinalized_credit": 500000, "reason": "<why>"}'

A merchant that lost every key or cannot win a leak by rolling asks for a recovery key, verified with the recorded contact (API key compromise and key recovery).

Merchant setup#

These are the merchant's steps, done with its own secret key against the operator's https://$DOMAIN. An operator that is also a merchant of its own instance does them as the merchant, never with the admin key (Phala's staff, for Phala Cloud and the staging reference product). The integration guide is the full reference.

Treasury setup#

Follow Treasuries to prove a treasury per chain and mode, using an EOA or Safe. Update the merchant's pinned treasury when a change takes effect; previously issued addresses keep paying the old treasury.

Payment settings#

New accounts accept nothing. Configure the chains, assets, and terms with the merchant's secret key as described in Payment settings. GET /v1/config then lists the effective assets on chains with an active treasury.

Webhook endpoint#

Pin the account's webhook keys, then register its endpoint. Credit from verified events using the balance rule.

Sweeping#

The merchant sweeps with its own wallet or Safe, paying gas. The SDKs build calls offline from the forwarder export; see Balance, sweeps, and the forwarder export. Failed FlushFailed targets remain the merchant's to resolve.

After a restore: the merchant notice#

A restored service starts frozen in restore mode (design §13): every merchant request with an API key, reads included, answers 503 service_restoring with Retry-After, and nothing credits or delivers until the operator's reconciliation and unfreeze (Reconciliation after a restore). HUMAN-ONLY, operator: before reconciling, send every account's recorded contact the notice: the restore point (the backup time) and the time the restore was detected; that API requests, reads included, answer 503 until further notice, so retry with the same Idempotency-Key; and the records the operator needs from the merchant, received after the restore point: key revocations (id, or prefix and last four), treasury cancellations and crediting pauses, deleted endpoints, deposit addresses and quotes given to customers, and the raw deliveries of deposit events with their webhook headers; a quote or delivery it cannot produce is lost (a payment to the quote is not found; the deposit is re-valued). After the unfreeze, send a second notice: service resumed, any re-valued deposit flagged, events after the restore point delivered again, and that its payment settings are held until it sends its complete configuration again with POST /v1/payment_settings (Payment settings). The runbook Incident communication has the channels.

Local verification#

• make up / make down: the attested compose rendered from the local environment (local/environment.sh: staging's routes, placeholder providers) plus the local overlay (Garage S3, the dstack simulator, a mock product); run manual commands through deploy/local/compose.sh -p PROJECT.
• make cvm-rehearsal: a staging-shaped artifact against Anvil, Garage, and the simulator, from the unsealed boot through sealing, a configuration upgrade (topup recreated, PostgreSQL not), operator onboarding of the product's account, its treasury proof and webhook endpoint, and one credited deposit submitted with a tx-hash hint and a signed event redelivery without a second credit. Mainnet prices use local fixtures; the Ethereum price Anvil mines every twelve seconds after TWAP seeding, while the Base price Anvil mines every second.
• make restore-drill: RESTORE.md.
• make sandbox-local: the integrator sandbox (sandbox/README.md).
• deploy/validate-compose.sh: every committed environment rendered and checked against compose-policy.jq, as CI enforces.

Phala's instance#

Phala's own deployment (its staging routes, the staging reset, the reference product behind the demo, the website, and its onboarding policy) is in Phala's instance.

Production preflight passes --require-sentry to topup config check, forwarding SENTRY_DSN from the preflight environment file; a missing or malformed DSN blocks rollout. Configure the production GitHub Environment secret SENTRY_DSN for this read-only validation; the CVM retains its separately owner-sealed DSN.

Ingress uses the supported dstack-ingress TCP settings TIMEOUT_CONNECT=5s, TIMEOUT_CLIENT=30s, and TIMEOUT_SERVER=30s, allowing the API deadline response to complete.

Rollback compatibility#

Before rolling back to a release without hints_per_day and quote_snapshots_per_day, render that release's kit with its verified configuration and use its original RPC alert rules. Its strict parser rejects the new fields, including defaulted fields printed by newer topup config show. Preserve each release's configuration and operating allocation alongside its verified images. Database compatibility does not make the older binary honor the new caps. The client address passthrough rollback requirements still apply.

Migrations must be expand-only and preserve N-1 reads, writes, defaults and constraints. Under the migration advisory lock, protocol-aware binaries validate every newer applied migration against public.topup_migration_compatibility: the applied checksum must match its owner-written ledger entry, its compatibility floor must be no greater than the binary's maximum known migration, and it must have succeeded. Missing metadata, a higher floor or mismatched checksums fail closed. SQLx continues checking all known migration checksums and dirty state. The application role cannot write the ledger. A rollback never lowers existing floors or edits migration history.

CI resolves the latest stable release as N-1 automatically and requires v0.9.0 or later; currently it resolves v0.10.0. This is the rollback target for a later compatible release, while the initial production v0.10.0 launch has no earlier production target (launch rollback). For any N-1, an exact no rollback to <N-1 version>; restore required declaration in ### Breaking (operators) under ## [Unreleased] or the dated section matching the workspace version (including the release PR before tagging) selects declared mode: CI skips the image smoke and prints the declaration in its job summary. COMPATIBILITY_FLOOR must also exceed the value in N-1's tag; CI fails if the floor is unchanged or lower. Without a declaration, N-1 runs the real image rollback smoke. Breaking migrations raise the floor to the new schema's maximum; retain a tested pre-upgrade restore/reconciliation plan and obtain owner acceptance.

Before upgrading to the read/verify pilot, run this check against a verified pre-upgrade restore copy with PostgreSQL access (the production CVM exposes no database port):

psql "$RESTORED_DATABASE_URL" -v ON_ERROR_STOP=1 -f deploy/check-address-capacity.sql

It reports every chain's historical issued-address count and the permanent 1,000-address cap, then refuses if any count is already over the cap. At exactly 1,000 the upgrade is safe, but new issuance returns 422 address_capacity_reached. Review the address capacity runbook at 70% and 90%. Preserve this output in the upgrade evidence; never delete rows to pass the check.

Before upgrade, preserve the verified N-1 deploy kit, image digests and configuration and a tested pre-upgrade recovery point. To roll back, verify N-1's release again, render its compose/config, and use Deploy mode: upgrade; require migration completion, service health, attestation and TLS checks. If N's changelog declares no rollback, stop writes and follow the restore runbook and reconciliation using the pre-upgrade backup instead. Account for external effects already executed after that backup; restoring blindly can repeat them. Never bypass checksum or compatibility errors. Use a forward repair release if there is no safe recovery point.

Disk pressure#

Retain all application data for seven years, including append-only tables and undelivered events. Fund storage growth; no age-based deletion job is introduced. Seven retained full backups is a recovery rotation, not a seven-year data purge policy.

The read-only capacity probe samples the filesystems containing pgdata and observability every 30 seconds. /metrics exposes topup_disk_used_ratio{volume="pgdata|observability"}, topup_unarchived_wal_bytes, topup_unarchived_wal_age_seconds, and topup_capacity_sample_timestamp_seconds. Ratios include reserved space; when volumes share a filesystem they deliberately report the same pressure. Completed WAL segments with .ready markers count as unarchived; .done segments do not.

The backup monitor sends Sentry-compatible alerts: TopupDiskWarning at >=75%, TopupDiskCritical at >=90%, and TopupUnarchivedWalPressure at >=1 GiB pending WAL or oldest age

=120 seconds. Missing/invalid samples or age >120 seconds emit TopupCapacityProbeFailed. Configure SENTRY_DSN and route these issues to operations; with no DSN they remain local logs. These alerts are sampled from the running service. Crons detect missed check-ins when the service or host is down; external /healthz checks detect ingress failures. Phala's pilot uses the owner-accepted manual pilot_watch cadence above.

1. At warning, inspect growth rates, filesystem free space, largest relations/indexes and observability files. Expand the CVM disk/filesystem before headroom is exhausted; estimate time to full using WAL growth as well as table growth. Verify actual expansion and new samples.
2. At critical, escalate immediately. If expansion cannot complete safely, use the established instance maintenance pause and stop API writes and workers in a controlled incident procedure. Preserve DB and backup continuity. Inspect PostgreSQL archiver errors, credentials, object-store availability and backup lag; restore archiving before resuming writes.
3. Never delete undelivered events, financial/audit records, pg_wal files, .ready markers or unarchived WAL. Never disable archiving to hide pressure, truncate tables or run VACUUM FULL on a nearly full disk. Only remove independently verified disposable logs/cache under their approved retention policy; preserve incident evidence. Expand storage or restore on a larger fenced instance if necessary.
4. Verify WAL draining, newest archive, fresh samples, both volumes below 75%, service/worker progress and pending deliveries before resolving the incident.

Recovery across independent domains remains a proposal in recovery-domains.md, pending the owner's cost decision.

Deployment time budgets#

Operator-side scripts require GNU timeout, or gtimeout from coreutils on macOS, alongside Bash 4.4 or newer. The quick-start bootstrap downloads the verifier and its deadline helper from the same release tag; release checksums and provenance verification still run before kit scripts.

Deploy stages have workflow step deadlines (10 minutes, 20 for preflight/attestation/health, 60 for deployment including an optional OS-image reboot, and 25 for provision's gateway update), in addition to the 75-minute job ceiling. Shell stages use absolute elapsed-time deadlines: anonymous pulls share 600 seconds (each pull <=180 seconds, three attempts); Phala preflight shares 120 seconds (each call <=30 seconds); deployment submission <=300 seconds; CVM convergence <=900 seconds; OS-image PATCH and convergence share <=900 seconds, followed by old-compose health <=600 seconds and a fresh maintenance pause <=60 seconds before compose deployment; attestation and health convergence <=600 seconds each. Each call is capped by remaining stage time, with TERM and a two-second KILL grace. Poll sleeps cannot extend the stage. Diagnostics name the stage, elapsed/remaining seconds and exit status without credentials. Contract and release verification remain mandatory and are also bounded by workflow step deadlines.

OFAC SLS host or publication contract changes fail closed: verification does not advance, and stale negative decisions hold. See the sanctions runbook.