Skip to main content

Pub/Sub overview

Crusoe AI Platform Pub/Sub is a fully managed, high-throughput publish/subscribe messaging service. It lets microservices, AI agents, and background workers communicate asynchronously without direct dependencies or waiting for immediate responses.

Core architecture​

Pub/Sub relies on three main components:

  • Topics: Named channels where publisher applications send messages.
  • Subscriptions: Durable message queues or streams attached to topics. Each subscription receives its own copy of published messages and tracks its own position.
  • Publishers and Consumers: Applications that send messages to topics (publishers) or retrieve messages from subscriptions (consumers).

Messaging models​

Pub/Sub supports two delivery mechanisms:

  • Pull Delivery: Your consumer requests waiting messages over HTTP, processes them, and acknowledges completion to remove them from the queue.
  • Push Delivery: Pub/Sub automatically POSTs messages as CloudEvents to an HTTP endpoint in your project. If delivery fails, Pub/Sub retries automatically with exponential backoff and can route unprocessable messages to a dead-letter topic.

Key use cases​

Use Pub/Sub whenever you need to:

  • Decouple microservices: Keep publishers operating smoothly even when consumers are offline or processing slow tasks.
  • Fan out events: Send a single event from one publisher to multiple independent subscriber services.
  • Buffer workload spikes: Queue rapid bursts of incoming tasks so worker fleets can drain them at a steady pace.
  • Trigger serverless functions: Automatically invoke HTTP services or functions whenever new events arrive.

Multi-tenant project isolation​

Pub/Sub enforces complete project-level isolation. Topics, subscriptions, storage quotas, and credentials live entirely within your project scope:

  • Default Storage Quota: Each project receives an allocated storage budget (1 GiB by default) for retained messages.
  • Secure Boundaries: Topics and messages in one project cannot be accessed or viewed by other projects.
  • Flexible Management: Configure topics and subscriptions using the Crusoe Console, platformctl CLI, or REST API.

Delivery guarantees​

  • At-Least-Once Delivery: Every published message is guaranteed to be delivered at least once to each active subscription. Design your consumers to handle duplicate messages harmlessly (idempotency).
  • Message Ordering: Ordering can be configured per subscription (such as per-key ordering with key-shared subscriptions).

Cloud platform comparison​

FeatureCrusoe AI Platform Pub/SubGCP Pub/SubAWS SNS + SQSAzure Service Bus
Primary ModelTopics + SubscriptionsTopics + SubscriptionsSNS Topics + SQS QueuesService Bus Topics + Subscriptions
Delivery ModesPull & HTTP PushPull & PushSQS Polling / HTTP EndpointPull & Push
Dead-Letter HandlingManaged Dead-Letter TopicsDead-Letter TopicsDead-Letter QueuesDead-Letter Sub-queues
ProtocolsHTTP REST & Native Wire ProtocolHTTP REST & gRPCHTTP RESTAMQP & HTTP
API Endpoint

Pub/Sub is available on the public API at https://api.codyhill.net. See the API Overview for details on base URLs and authentication.

Put it on the internet​

Publish an endpoint for this project's broker through the API Gateway:

platformctl gateway publish pubsub --allow-cidr 203.0.113.0/24

No resource name — the endpoint covers this project's whole broker.

It is a raw connection, not web requests. Clients speak the broker's binary protocol, so there are no requests for the gateway to read:

  • --auth and --rate-limit are not available, and asking for them is refused rather than accepted and ignored.
  • --allow-cidr is available, and is the only network control there is.
  • Your project's messaging credential is what actually lets a caller in.

Treat the allow list as a coarse net in front of that credential, never as the boundary itself.

See What you can publish.

Next steps​