Skip to main content

Pub/Sub guide

This guide covers everything you need to build and manage messaging workflows with Crusoe AI Platform Pub/Sub.

Service overview​

Pub/Sub decoupling enables producers to publish events asynchronously while consumers process messages independently.

Key use cases:

  • Event-driven microservices: Broadcast state changes across independent downstream microservices.
  • Asynchronous worker queues: Buffer incoming job requests for background processing fleets.
  • Webhook distribution: Replicate inbound webhooks to multiple application endpoints.
  • Serverless automation: Automatically trigger functions when new messages arrive. A deployed workload publishes with a different credential and a different address than you do from your laptop — see publishing from a deployed workload.

Quick start in five commands​

# 1. Log in to your platform account
platformctl login

# 2. Create a messaging topic
platformctl pubsub topics create events

# 3. Create a shared worker subscription
platformctl pubsub subscriptions create worker-q --topic events

# 4. Publish an event message
platformctl pubsub topics publish events --message "order-created"

# 5. Pull and acknowledge the event
platformctl pubsub subscriptions pull worker-q --topic events --max 1 --ack

Two shapes to internalize now rather than after a confusing error. Publishing lives under topics, and its payload flag is --message (which also takes @file or - for stdin). Subscriptions are nested under a topic, so every subscriptions command takes --topic; the subscription name is the only positional argument, and omitting the topic is refused with --topic is required (subscriptions are nested under a topic).

Key concepts​

  • Topic: A named channel to which messages are published.
  • Subscription: A durable reader attached to a specific topic that receives published messages.
  • Message: Data payload plus optional key-value string attributes.
  • Pull vs Push: Pull subscriptions allow applications to poll for messages; push subscriptions POST CloudEvents directly to HTTP endpoints.
  • Acknowledgement: Signal sent by a consumer to remove processed messages from the subscription queue.
  • Dead-Letter Topic: Secondary topic where failed push messages are routed after exceeding maximum retries.

API operations​

Set API credentials in your environment:

export PS="https://api.codyhill.dev"
export PROJ="<your-project-id>"
export TOK="<your-api-key-or-token>"

Create topic​

curl -sX POST "$PS/v1/projects/$PROJ/topics" \
  -H "Authorization: Bearer $TOK" -H 'Content-Type: application/json' \
  -d '{"name":"events","max_bytes":"64Mi","max_age":"168h"}'

Create subscription​

curl -sX POST "$PS/v1/projects/$PROJ/topics/events/subscriptions" \
  -H "Authorization: Bearer $TOK" -H 'Content-Type: application/json' \
  -d '{"name":"worker-q","ack_deadline_seconds":30}'

Publish message​

curl -sX POST "$PS/v1/projects/$PROJ/topics/events:publish" \
  -H "Authorization: Bearer $TOK" -H 'Content-Type: application/json' \
  -d '{"messages":[{"data":"aGVsbG8=","attributes":{"kind":"demo"}}]}'

Pull and acknowledge​

curl -sX POST "$PS/v1/projects/$PROJ/topics/events/subscriptions/worker-q:pull" \
  -H "Authorization: Bearer $TOK" -H 'Content-Type: application/json' \
  -d '{"max_messages":10}'

curl -sX POST "$PS/v1/projects/$PROJ/topics/events/subscriptions/worker-q:ack" \
  -H "Authorization: Bearer $TOK" -H 'Content-Type: application/json' \
  -d '{"ack_ids":["<id-from-pull>"]}'

CLI operations​

# Topic commands
platformctl pubsub topics list
platformctl pubsub topics create events --max-bytes 64Mi --max-age 168h
platformctl pubsub topics get events
platformctl pubsub topics delete events

# Subscription commands
platformctl pubsub subscriptions create worker-q --topic events --ack-deadline-seconds 30
platformctl pubsub subscriptions list --topic events
platformctl pubsub subscriptions delete worker-q --topic events

# Message commands
platformctl pubsub topics publish events --message "hello" --attribute kind=demo
platformctl pubsub subscriptions pull worker-q --topic events --max 10 --ack

# Quota and credentials
platformctl pubsub quota
platformctl pubsub credentials

Console UI walkthrough​

  1. Open the Crusoe Console and select your project.
  2. Go to Messaging → Pub/Sub.
  3. Under Topics, click Create Topic, enter a name, and set storage allocation.
  4. Click into the topic and select Subscriptions → Create Subscription.
  5. Choose delivery mode (Pull or Push).
  6. Use the Publish action on the topic page to test sending messages.
  7. Use the Pull action on the subscription page to inspect and acknowledge messages.

Service limits​

ResourceDefault Quota
Topics per project100
Concurrent producers per topic100
Concurrent consumers per topic100
Maximum request body size4 MiB
Default project storage budget1 GiB
Ack deadline range1–600 seconds
Maximum messages per publish request100

There is no cap on the number of subscriptions a topic may have. The limits that exist bound connections, and a subscription object consumes one only while something is actually reading it — a dormant subscription costs nothing against the producer and consumer counts.

Troubleshooting common issues​

SymptomCauseResolution
Pulled message keeps returningMessage was not acknowledged before ack_deadline_seconds.Ensure consumer sends acknowledgment before deadline expires or increase deadline.
Publish returns 409Topic capacity reached (discard: new) or topic still initializing.Consume unacknowledged messages, increase max_bytes, or set discard: old.
Push messages failingEndpoint URL unreachable or non-2xx status returned.Verify push URL endpoint health and CloudEvent payload handling.
Dead-letter topic fillingMessages failing delivery beyond max_deliver threshold.Inspect consumer error logs and resolve underlying processing failures.

Security controls​

  • Project Scoping: All topics and subscriptions are isolated within their parent project.
  • Role-Based Access Control: Project members can publish and pull; creating/deleting topics requires admin role.
  • Audit Logging: All topic management, subscription changes, and administrative credential requests are logged to the project audit trail.