Skip to main content

Topics and subscriptions

This guide details the Pub/Sub resource model: topic configurations, storage quota mechanics, retention rules, subscription types, and resource modification options.

Environment setup​

Set the required environment variables for API requests:

export CAI_PUBSUB_API="https://api.codyhill.net"
export CAI_PROJECT="<your-project-id>"
export CAI_TOKEN="<your-api-key-or-session-token>"

Topics: Named messaging channels​

A topic is a named messaging channel where producers publish messages.

Topic settings​

PropertyRequiredDefaultDescription
nameYes—Immutable. Lowercase letters, digits, and hyphens (1–63 characters). Must start with a letter.
display_nameNoEmptyEditable human-readable label.
max_bytesNo16MiReclaimed storage budget reserved for this topic's unacknowledged backlog. Editable; lowering it to at or below the live backlog is refused unless you ask for it explicitly.
max_ageNoNoneMaximum retention period for messages (e.g. 24h, 30m).
discardNooldCapacity overflow behavior: old drops the oldest unacknowledged messages; new rejects new publishes with HTTP 409.

Creating a topic​

platformctl pubsub topics create orders --display-name "Order Events" \
  --max-bytes 64Mi --max-age 24h --discard new

Storage budget allocation​

Each project has a Pub/Sub message storage budget (default 1 GiB).

  • Claimed Budget: A topic claims its max_bytes allocation immediately upon creation, even when empty.
  • Capacity Management: Deleting unused topics or lowering max_bytes frees storage budget for other topics. A topic can only give back budget it is not using — see lowering max_bytes below what the topic is holding.
  • Monitoring Capacity: Check quota usage anytime via CLI, API, or Console.
platformctl pubsub quota

Subscriptions: Durable readers​

Subscriptions track reader progress and retain unacknowledged messages for processing.

Subscription configuration​

PropertyRequiredDefaultDescription
nameYes—Unique identifier within the topic.
typeNosharedOrdering and consumer model: shared, key-shared, exclusive, failover. Immutable after creation.
ack_deadline_secondsNo30Time window (1–600 seconds) for consumers to acknowledge pulled messages before redelivery.
max_deliverNo5Maximum delivery attempts before forwarding to a dead-letter topic (push subscriptions).
start_fromNoallInitial read position: all (earliest available) or new (from subscription creation onwards). Immutable.
dead_letterNoNoneTarget topic configuration for unprocessable messages: {"topic": "dead-letter-topic", "after_attempts": 5}.

Subscription ordering models​

  • shared: Competing consumer model. Messages are distributed across active workers for parallel processing.
  • key-shared: Partitioned queue. Messages sharing the same key attribute are delivered in strict order to the same consumer instance.
  • exclusive: Single active consumer stream with strict total ordering.
  • failover: Primary single consumer stream with automatic failover to standby consumers.

Creating a subscription​

platformctl pubsub subscriptions create workers --topic orders \
  --type key-shared --ack-deadline-seconds 60 --max-ack-pending 500 --start-from new

Updating resource settings​

Dynamic topic and subscription settings can be modified in place. Immutable properties (name, type, start_from) require deleting and recreating the resource.

Updating a topic limit​

platformctl pubsub topics update orders --max-bytes 128Mi

Lowering max_bytes below what the topic is holding​

Raising max_bytes is checked against the project's storage budget and nothing else, and restating the same size in different units is not a lowering at all. Lowering it is checked against the topic's live backlog — the unacknowledged bytes it is holding right now — because a ceiling below the backlog cannot be honoured without something giving way, and what gives way is not the same on every topic.

A PATCH that would put max_bytes at or below the current backlog is refused with 409. A topic sitting exactly at its ceiling drops or rejects on its very next message, so an equal pair is refused as well. The refusal names both numbers — what the topic is holding and the ceiling you asked for — and spells out the consequence for that topic's discard setting, because the two settings fail in opposite directions:

discardWhat lowering below the backlog would do
old (the default)The broker drops the oldest unacknowledged messages until the backlog fits under the new ceiling. Immediate, permanent, and nothing is dead-lettered on the way out.
newNothing is dropped. Publishing starts failing instead, until consumers drain the backlog below the new ceiling.

Both of those used to happen silently, on a plain edit, at every surface. Neither does now.

Read the backlog before you lower the ceiling. It is the same figure the check reads, so there is no second number to reconcile:

platformctl pubsub usage

The per-topic table has a BACKLOG column — what the topic is still holding unacknowledged.

The override is ?allow_discard=true. Dropping the oldest messages is sometimes exactly what you want — a topic holding a week of events nobody will ever process is claiming budget for nothing. Repeat the call with ?allow_discard=true and the lowering goes through, with the consequence its discard setting implies. This is the same shape as ?cascade=true on project delete: refused by default, explicit when you mean it, never inferred from the plain call.

platformctl pubsub topics update orders --max-bytes 16Mi --allow-discard

The flag is named for what it costs you, not for the check it lifts, and it is the same word as the ?allow_discard=true it sends — the way --cascade matches ?cascade=true — so there is nothing to translate between the curl above and the command here. On a discard: new topic nothing already stored is discarded; publishes start failing instead, which the command's --help spells out. It only means anything alongside --max-bytes; passing it on its own is an error rather than a no-op.

On a discard: new topic the flag is still what unlocks the change, and it is worth reading twice there: it discards nothing, it accepts a ceiling your backlog is already above, and publishes fail until consumers bring the backlog back under it.

An acknowledged shrink is recorded as one. The audit row for that edit carries allow_discard: true and the backlog_bytes reading it overrode, so "how much was at stake when someone waved this through" is answerable afterwards. When the topic was not reporting a backlog, the row records null rather than 0 — an override against an unknown backlog is not an override that risked nothing.

If the backlog cannot be read, the lowering is refused. A topic that is not reporting live figures yet has no backlog to compare against — and a backlog_bytes of 0 on a topic that is not ready means unknown, not empty. The refusal names the state it found rather than guessing: an unread number is not a small number, and a guess here is how an edit becomes silent message loss. ?allow_discard=true overrides this case too, which is what keeps a topic whose figures cannot be read from being un-shrinkable forever. Raising max_bytes is never held up by any of it — it cannot discard anything, so it is not checked against the backlog at all.

Summary best practices​

  1. Pre-create Subscriptions: Ensure subscriptions exist prior to publishing messages so no events are missed.
  2. Configure Dead-Letter Topics: Set up dead-letter targets on push subscriptions to capture and isolate unprocessable messages.
  3. Select Appropriate Subscription Types: Use shared for parallel task queues and key-shared for entity-ordered stream processing.