Advanced: document ingestion pipeline (Python)

Drop a Markdown file in a bucket; a minute later it is searchable. This guide builds that, and it is the pipeline the platform's own documentation search runs on — so every number below is measured on a real corpus rather than estimated.

bucket drop
   │  object-store trigger (polls, up to 60 s)
   ▼
doc-ingest ──── publishes one message per chunk ────▶  Pub/Sub topic
   │                                                        │
   │  reads the object, names it, splits it                 │ subscription
   │                                                        ▼
   │                                              chunk-embedder
   │                                                   │ embeds
   │                                                   ▼
   │                                               VectorDB upsert

Source: demo/doc-ingest and demo/chunk-embedder.

What you need

• A project, platformctl login, and an S3-compatible bucket you control. The platform does not host the store — the trigger needs your endpoint and a credential.
• About 30 minutes.

Why two functions and not one

The split is on the fan-out. One document becomes many chunks, and chunks embed independently — so the second stage scales with the number of chunks while the first scales with the number of documents. Putting both in one function means a 50-chunk document holds one instance for the whole embedding pass, and a burst of uploads queues behind it.

Pub/Sub between them also makes the expensive half retryable on its own. An embedding call that fails does not re-read the object or re-split it.

Stage 1: the object-store trigger

platformctl serverless triggers create docs-ingest \
  --type objectstore \
  --target doc-ingest \
  --bucket my-docs \
  --endpoint https://s3.us-east-1.amazonaws.com \
  --credentials-secret docs-bucket-creds \
  --after-read none

Three things bite here, in this order

1. --endpoint is required and there is no default. The platform does not host your object store, so it cannot guess where it is.

2. --credentials-secret names a secret in this project, not a key. Create it first, and remember a bound secret does nothing until the workload is redeployed.

3. The trigger polls. A drop is picked up within 60 seconds, not instantly. If you upload and immediately check, you will conclude it is broken.

--after-read none leaves the object where it is. The alternatives — move and delete — are covered in the Ruby ETL guide, where consuming the object is the point.

Stage 1: naming the document

This is the part worth reading even if you never build this pipeline, because it is where a plausible design quietly loses data.

A chunk id is <name>#<n>. So whatever names the document decides which chunks overwrite which.

demo/doc-ingest/handler.py

KEY_ATTRS = ('key', 'objectkey', 'object', 'filename', 'name', 'subject')

The object key is not in the event

Measured: this platform's aws2-s3 source sends none of those attributes. Its whole attribute set is the five standard CloudEvent ones, and its subject is the literal string aws-s3-source — the name of the source, not of the object.

Taking subject as the filename made every upload skip on its suffix. The handler therefore believes an attribute only when it looks like an object key.

So the name is read out of the document itself, in this order:

1. An explicit <!-- source: path/to/file.md --> comment. Used verbatim, not slugified, so the citation a reader sees is the path they can open.
2. The frontmatter title:.
3. The first Markdown heading.
4. A hash of the content.

Why frontmatter before heading, with a number attached: on this corpus 137 files have frontmatter and only 84 have an H1. Reading the title first is what keeps those 53 documents from falling through to a hash.

And why it matters at all: the corpus has 17 files called overview.md. Naming them from a title maps all 17 onto one name — and since a chunk id is <name>#<n>, they overwrite each other. The corpus silently ends up holding one of them.

Stage 1: skipping what should not be embedded

demo/doc-ingest/handler.py

TEXT_SUFFIXES = tuple(
    s.strip() for s in os.environ.get('TEXT_SUFFIXES', '.md,.markdown,.txt').split(',')
    if s.strip()
)

A bucket holds whatever someone drops in it. An image embedded as mojibake produces a junk vector that scores against every query — it does not fail, it degrades every search quietly. Skipping is the correct handling, not a limitation.

Stage 2: embedding

demo/chunk-embedder/handler.py

EMBEDDINGS = OpenAIEmbeddings(
    model=EMBED_MODEL,
    check_embedding_ctx_length=False,
    ...
)

check_embedding_ctx_length=False is not an optimisation

Left on, LangChain re-chunks by token count — a second, invisible chunker downstream of the deliberate one — and sends token ids where this server expects text. The symptom is embeddings that work but retrieve badly, which is the hardest kind of bug to see.

The credential

Both functions read PIPELINE_KEY, a service-account key held in the project's secret store and bound to the function.

A workload's own key is deliberately not enough

CAI_PROJECT_KEY reaches inference-api but is denied on project APIs by design. Reading bound secrets is its one power, and the service-account key is what comes back. A pipeline that could write to VectorDB with its ambient identity would be a pipeline any workload could write to VectorDB with.

# The value comes from stdin or a file. There is deliberately no --value flag:
# a secret on the command line lands in your shell history and in the process
# list, where anyone on the machine can read it.
printf '%s' "$SA_KEY" | platformctl secrets put pipeline-key

platformctl secrets bindings set doc-ingest PIPELINE_KEY --secret pipeline-key
platformctl secrets bindings apply doc-ingest

The apply is not optional. A bound secret does nothing until it is applied, because binding records intent and applying rolls the revision that carries it.

Deploy and run it

platformctl functions deploy ./demo/doc-ingest --name doc-ingest
platformctl functions deploy ./demo/chunk-embedder --name chunk-embedder

Then drop a file in the bucket and wait up to a minute.

The numbers to compare against

Measured on the platform's own documentation corpus:

Stage	Number
Documents ingested	5
Chunks produced	18
Vector similarity, best passage	0.5468
Same passage after reranking	0.9489
Second-best passage after reranking	0.0619

That reranking line is the whole argument for the two-stage retrieval this pipeline feeds. Vector search puts the right passage on top at 0.5468 — a score that says "vaguely related" and gives you no way to set a threshold. The reranker reads the query against each candidate and returns 0.9489 against 0.0619 for the runner-up. The ordering barely changed; the separation is what changed, and separation is what lets an agent decide whether it found anything at all.

Traps, at the point you hit them

Trap	What you see	Why
Upload, check immediately	Nothing happened	The trigger polls; allow 60 s.
Every upload skipped	Logs say the suffix did not match	subject is aws-s3-source, not a filename.
Chunks disappear	The corpus holds fewer documents than you uploaded	Many files named overview.md collide on <name>#<n>.
Secret bound, still KeyError	The function cannot see PIPELINE_KEY	Binding is not applying. Run secrets apply.
Retrieval is poor but nothing errors	Answers cite the wrong passage	check_embedding_ctx_length left on, re-chunking behind your chunker.
Index width mismatch	Upsert refused	A vector index's width is fixed at creation. platformctl platform limits prints the model's width — 4096, not 1536.

Teardown

In dependency order — trigger, then functions, then the topic, then the index:

platformctl serverless triggers delete docs-ingest
platformctl delete doc-ingest
platformctl delete chunk-embedder
platformctl pubsub topics delete doc-uploads-topic
platformctl vectordb delete docs-rag-index

What it costs to leave running. The functions scale to zero and cost no compute while idle. What does cost: the Pub/Sub topic reserves its max_bytes from the project budget the moment it exists, used or not, and the VectorDB index holds storage for as long as it exists. The trigger itself polls continuously — that is not free, but it is small.

Where next

• Object-store ETL with move-after-read (Ruby) — consuming objects rather than leaving them.
• Webhook fan-out (Node.js) — the other side of Pub/Sub.
• VectorDB search — querying what this pipeline wrote.