Skip to main content

Marketplace Stack Validation Runbook

v3.0.6 boundary: the only approved retained buyer target is agentic3.tutelacloud.com. agentic.tutelacloud.com is a protected older-release environment and must not be modified or used for this run. Complete buyer validation and retain the evidence for manual owner review; do not submit or publish to Marketplace without later explicit authorization.

Use this runbook before asking a customer to retest AWS Marketplace launch paths. The goal is to validate the published templates and image tags with the lowest cost path first, then proceed to more expensive shapes only after shared certificate, DNS, entitlement, and routing dependencies pass.

Sequence​

  1. Confirm the target GitHub release and Marketplace artifact prefix exist.
  2. Run static validation:
    • bash scripts/marketplace/test-audit-delivery-options.sh
    • bash scripts/marketplace/test-estimate-infrastructure-costs.sh
    • bash scripts/marketplace/test-render-marketplace-artifacts.sh
    • bash scripts/marketplace/test-publish-marketplace-images.sh
    • bash scripts/marketplace/test-validate-marketplace-copy.sh
    • bash scripts/marketplace/test-submit-marketplace-version.sh
    • bash scripts/marketplace/test-update-marketplace-usage-instructions.sh
    • bash scripts/marketplace/test-buyer-release-certification.sh
    • bash scripts/marketplace/test-marketplace-launch-template.sh
    • cfn-lint -i W1011 -t <all eight Marketplace and nested templates>
    • aws cloudformation validate-template for all eight templates in us-west-2 and us-east-1
  3. Upgrade agentic3.tutelacloud.com in buyer account 641059604274 to the exact immutable release and run the post-SSM runtime convergence gate.
  4. Run the release-candidate browser and extension simulation without skips or flakes, full buyer certification, the 64-case policy-action matrix, and the approved/rejected/expired human-approval lifecycle matrix. Every report must be green and metadata-only.
  5. Create and accept the hash-bound certificate described in Marketplace Buyer Release Certification (available in the signed-in Help Center).
  6. Only after buyer acceptance, run the internal Marketplace install smoke harness against staged versioned S3 template URLs:
    • SMOKE_OPTION=standard-public-slim
    • SMOKE_OPTION=standard-public
    • SMOKE_OPTION=standard-private
    • SMOKE_OPTION=vllm-public
    • SMOKE_OPTION=vllm-private
  7. Delete each stack before starting the next run. The harness does this by default unless KEEP_STACK=true.
  8. Stop at the first failure, keep the evidence under artifacts/marketplace-install-smoke/, and fix the template before moving to the next option.

When a stack fails, capture both legacy stack events and CloudFormation operation failed events for the root and nested stack IDs:

aws cloudformation describe-events \
  --region <deployment-region> \
  --stack-name <root-or-nested-stack-name> \
  --filters FailedEvents=true \
  --output json

Use the describe-events output first for CloudFormation Early Validation errors such as NAME_CONFLICT_VALIDATION; it includes validation paths and identifiers that can be absent from describe-stack-events.

Required Buyer Inputs​

Set these variables before running the harness:

export RELEASE_VERSION=vX.Y.Z
export AWS_REGION=us-west-2
export MARKETPLACE_SMOKE_DOMAIN=<customer-owned-domain>
export MARKETPLACE_SMOKE_HOSTED_ZONE_ID=<route53-hosted-zone-id>
export MARKETPLACE_SMOKE_AZ1=us-west-2a
export MARKETPLACE_SMOKE_AZ2=us-west-2b
export MARKETPLACE_SMOKE_DB_PASSWORD=<redacted-strong-password>

Private-network tests also require either MARKETPLACE_SMOKE_PRIVATE_CA_ARN or MARKETPLACE_SMOKE_CERTIFICATE_MODE=create-private-ca with MARKETPLACE_SMOKE_PRIVATE_CA_ACK=I_ACKNOWLEDGE_PRIVATE_CA_COSTS_AND_TRUST_BOOTSTRAP.

vLLM tests require MARKETPLACE_SMOKE_GPU_ACK=I_ACKNOWLEDGE_GPU_AND_MODEL_DOWNLOAD_COSTS.

Notes​

  • standard-public-slim uses the Tutela Marketplace Installer with AccessMode=public, EnableLocalModelRuntime=false, and DeploymentProfile=slim_training. It is the cheapest end-to-end launch path and validates the shared public DNS, public ACM certificate, ALB routing, entitlement, and runtime readiness path before creating managed MSK or NAT.

  • The base launcher preflight creates or verifies required service-linked roles for ACM, ELB, and License Manager before expensive nested stacks proceed.

  • The base launcher preflight also reuses a deterministic retained foundation artifacts bucket only when it is same-account, same-region, encrypted, and has full S3 public access block enabled. Other deterministic leftovers fail preflight with the conflicting resource names so nested stacks do not fail generically.

  • The harness validates the actual published Marketplace artifacts. Do not use local templates for final buyer readiness evidence.

  • No AddDeliveryOptions submission is allowed before the exact buyer release has an accepted certification. Marketplace artifact staging is not Catalog publication.

  • Marketplace S3 artifacts and Marketplace ECR images can exist even when the Catalog AddDeliveryOptions submission failed. Audit Catalog visibility with audit-delivery-options.sh (available in the signed-in Help Center):

    scripts/marketplace/audit-delivery-options.sh \
      --desired-public-count 1 \
      --version-cap-threshold 100 \
      --require-version <latest>

    The desired policy is latest one accepted semver release public and older accepted releases restricted, so AWS Console buyers see only the current public launch path. If the audit reports catalog_missing_required_versions after artifacts and images are verified, submit the missing release when the audit reports available headroom. If the audit instead reports version_cap_headroom_blocked, open AWS Marketplace Seller Operations to archive enough old restricted product versions or confirm a new enforced maximum before resubmitting; AWS reports this condition as EXCEEDS_MAX_VERSIONS. Only after Catalog accepts those versions can restrict-old-delivery-options.sh --visibility public --keep-latest-public-count 1 enforce that only the latest accepted release remains public.

Metadata-only usage-copy repair​

Correct buyer-facing Usage Instructions on an existing public version with update-marketplace-usage-instructions.sh. This is distinct from a release: the generated UpdateDeliveryOptions request contains only the existing delivery-option ID and its UsageInstructions field. It does not add a version, change images or deployment resources, change targeting or visibility, or run the Marketplace publish workflow.

First create and review a preview:

scripts/marketplace/update-marketplace-usage-instructions.sh \
  --dry-run \
  --release-version <current-public-version> \
  --output /tmp/marketplace-usage-copy-repair.json

The preview prints the exact entity revision, delivery-option ID, and SHA-256 of the current Usage Instructions. Confirm the generated JSON contains one UpdateDeliveryOptions change and only UsageInstructions, then submit with all three optimistic-lock values:

scripts/marketplace/update-marketplace-usage-instructions.sh \
  --submit \
  --release-version <current-public-version> \
  --expected-entity-identifier <product-id@revision> \
  --expected-delivery-option-id <delivery-option-id> \
  --expected-current-usage-sha256 <sha256-from-preview> \
  --output /tmp/marketplace-usage-copy-repair.json

Stop if any lock changed. After the Catalog change succeeds, describe the product again and confirm that exactly one public delivery option remains, its version and resources are unchanged, and its Usage Instructions match the safe source. Finish with buyer-account UI verification of the Usage tab and subscription dialog. Do not use this repair command to publish a pending release.

Last updated on