Skip to main content

Tutorial: RAG chatbot over your own documents

By the end of this tutorial you will have a deployed agent that answers questions about your documents. When the documents do not cover a question, it says so instead of making something up. And it names the file every answer came from.

The technique is called RAG: retrieval-augmented generation. Before the model answers, you retrieve the handful of passages most likely to be relevant and hand them to it. The model's job shrinks from "know everything" to "read these three paragraphs and answer the question", which is a job models are good at.

Budget about 40 minutes. Every command is copy-paste-runnable.

What you are building​

Two terms, defined once:

  • An embedding is a list of numbers that captures the meaning of a piece of text. Two texts that mean similar things get numerically similar embeddings, even when they share no words.
  • A vector index stores those number lists and finds the nearest ones fast. That is what VectorDB is.

The important honesty up front: VectorDB stores and searches vectors, it does not create them. You bring your own embeddings, so you choose the embedding model. One rule follows from that, and it is the rule people break: use the same model when loading documents and when asking questions. Use two different models and the numbers will not line up, and your search results will be nonsense.

Before you begin​

You need:

  • A platform account and a project, with the admin role on that project (creating a service account is an admin action). Ask your administrator for an account or an invitation link — see Create an account.
  • platformctl, signed in — see Install the CLI.
  • curl, jq, and python3 (standard library only — nothing to pip install).
  • Your project connected to Crusoe Cloud. Everything you deploy is built into a container image, and that image is stored in a repository in your own Crusoe Cloud Registry — so a project with no Crusoe Cloud credential is refused before anything is built. Connecting is a one-time, project-admin step, and a project that is already connected needs nothing new. See connect your Crusoe Cloud account.
  • An embeddings endpoint. You do not have to bring one. The platform hosts qwen-embedding and serves it on the same public API as everything else on this page — POST $CAI_API/v1/embeddings, authenticated with your own $CAI_TOKEN. platformctl inference models lists what your install serves. Any OpenAI-compatible provider works instead; the one rule is to use the same model for loading and for querying.

Both surfaces this tutorial calls — agents and VectorDB — live on the public API, https://api.codyhill.dev, and platformctl needs neither address set: it defaults to the same one. Only an install that serves VectorDB somewhere else needs a different $VDB (platformctl resolves $CAI_VECTORDB_API first, then $CAI_API, then the public default).

Set your shell up once:

export CAI_API="https://api.codyhill.dev"   # the platform API
export VDB="$CAI_API"                       # VectorDB is served on it too
# $CAI_TOKEN is cached by 'platformctl login'
export CAI_PROJECT="0f7a5c21-...-uuid"      # your project UUID - the ID column below
export INDEX="handbook"

If $CAI_TOKEN is empty (platformctl login caches its token in a file, not in your shell), get one directly — session tokens last 12 hours:

export CAI_TOKEN=$(curl -s "$CAI_API/v1/auth/login" \
  -H 'Content-Type: application/json' \
  -d '{"email":"you@example.com","password":"your-password"}' | jq -r .token)

A UUID is a long unique id, like 0f7a5c21-.... Your project's appears in the console URL when you open the project, in the #/projects/ path segment. Or list it:

platformctl projects list

You should see:

SLUG      NAME      SHORT   ROLE    ID
ml-team   ML Team   ab12cd  admin   0f7a5c21-...-uuid

The ID column is what CAI_PROJECT holds. Last, the embedding model — now that $CAI_TOKEN is in your shell:

export EMBED_BASE_URL="$CAI_API/v1"
export EMBED_API_KEY="$CAI_TOKEN"      # the platform's own endpoint takes your platform credential
export EMBED_MODEL="qwen-embedding"    # 4096 numbers wide - Act 2 measures it rather than trusting this

Pointing these at another provider is a supported swap — change all three here, and set the matching HANDBOOK_EMBED_* on the agent in Act 7, so both halves keep using one model.

Act 1: create the documents​

Real document sets are messy. Start with three tiny files instead, so you can check every answer by eye.

mkdir -p handbook

cat > handbook/pto.md <<'EOF'
Paid time off. Every employee gets 25 days of paid time off each year.
Unused days roll over into the next year, up to a maximum of 5 days.
Request time off at least two weeks in advance.
EOF

cat > handbook/expenses.md <<'EOF'
Expenses. Submit receipts within 30 days of the purchase.
Anything above 500 dollars needs written manager approval before you buy it.
Travel booked through the company portal is billed directly and needs no receipt.
EOF

cat > handbook/laptops.md <<'EOF'
Laptops. The standard issue is a 14-inch laptop, refreshed every 3 years.
Ask IT for an exception if your work needs more memory or a discrete GPU.
Report a lost or stolen laptop to IT within 24 hours.
EOF

Each file is one chunk — one unit that gets embedded and retrieved as a whole. With real documents you would split long files into overlapping chunks of a few hundred words. A chunk is also the unit of text the model reads. Make chunks too large and you drown the model in irrelevant text. Make them too small and you cut answers in half.

At this point you have: three plain-text documents on disk.

Act 2: find out how wide your vectors are​

An index has a fixed vector width, called its dimensions — how many numbers are in each vector. It has to match your embedding model exactly, and it can never be changed after the index is created. So rather than look the number up, measure it.

Save this script — you will use it again in Act 4 to load the documents:

#!/usr/bin/env python3
"""Load a folder of text files into a Crusoe VectorDB index."""
import json
import os
import pathlib
import sys
import urllib.error
import urllib.request

EMBED_BASE_URL = os.environ["EMBED_BASE_URL"].rstrip("/")
EMBED_API_KEY = os.environ["EMBED_API_KEY"]
EMBED_MODEL = os.environ["EMBED_MODEL"]
VDB = os.environ.get("VDB", "").rstrip("/")
TOKEN = os.environ.get("CAI_TOKEN", "")
PROJECT = os.environ.get("CAI_PROJECT", "")
INDEX = os.environ.get("INDEX", "handbook")


def post(url, payload, token):
    request = urllib.request.Request(
        url,
        data=json.dumps(payload).encode(),
        headers={"Content-Type": "application/json",
                 "Authorization": "Bearer " + token},
    )
    try:
        with urllib.request.urlopen(request, timeout=60) as response:
            return json.load(response)
    except urllib.error.HTTPError as error:
        sys.exit("HTTP %s from %s: %s" % (error.code, url, error.read().decode()))


def embed(text):
    body = post(EMBED_BASE_URL + "/embeddings",
                {"model": EMBED_MODEL, "input": text},
                EMBED_API_KEY)
    return body["data"][0]["embedding"]


def main():
    if len(sys.argv) > 1 and sys.argv[1] == "dims":
        print(len(embed("hello")))
        return

    points = []
    for number, path in enumerate(sorted(pathlib.Path("handbook").glob("*.md")), start=1):
        text = path.read_text().strip()
        points.append({
            "id": number,                       # stable id: re-running updates, never duplicates
            "vector": embed(text),
            "payload": {"text": text, "source": path.name},
        })
        print("embedded %s (%d characters)" % (path.name, len(text)))

    result = post("%s/v1/projects/%s/indexes/%s:upsert" % (VDB, PROJECT, INDEX),
                  {"points": points}, TOKEN)
    print(json.dumps(result))


main()

Save it as ingest.py, then measure:

python3 ingest.py dims

You should see a single number, for example:

4096

Record it:

export DIMS=4096   # use whatever your model printed

At this point you have: the exact vector width your embedding model produces.

Act 3: create the index​

Create an index named handbook, with that width and cosine distance. Cosine is the usual choice, because it compares the direction of two vectors — and direction is what "similar meaning" looks like numerically.

platformctl vectordb create "$INDEX" --dimensions "$DIMS" --distance cosine

The index is created in the background, so ask for it again until it reports ready:

platformctl vectordb get "$INDEX"

You should see state ready and ready true, with points_count 0.

If you write points before the index is ready, you get a clear 409: index is not ready yet; its collection has not been created.

Leaving dimensions blank means 1536, which is probably not your width

Omit dimensions — from the console form, from --dimensions, or from the JSON body — and the index is created 1536 wide, the width of the most common OpenAI embedding models. The platform's own qwen-embedding produces 4096. Nothing catches the mismatch at create time: it surfaces at your first upsert, as point 0 has 4096 dimensions; index "handbook" expects 1536, and because an upsert is all-or-nothing the index stays empty. Dimensions cannot be edited afterwards, so the recovery is deleting the index and doing this act again. Always pass the number you measured in Act 2.

Dimensions are permanent

dimensions and distance cannot be changed later. Switching embedding models means creating a new index and loading everything again. The API says so out loud if you try: dimensions is immutable: existing points cannot be re-embedded, so a different width is a new index, not an edit.

At this point you have: an empty, ready index sized for your embedding model.

Act 4: load the documents​

python3 ingest.py

You should see:

embedded expenses.md (198 characters)
embedded laptops.md (191 characters)
embedded pto.md (168 characters)
{"index": "handbook", "upserted_count": 3}

Each document became a point: a vector, plus a JSON payload carrying the original text and its file name. The payload is what makes citations possible later. The vector finds the passage. The payload hands you something a human can read.

Two things the script does on purpose:

  • Stable ids. Point 1 is always expenses.md. Re-run the script after editing a document and it updates that point instead of adding a duplicate.
  • One batch. A single upsert is all-or-nothing. If one vector is the wrong width, the whole batch is rejected, so you never end up half-loaded. You would see point 2 has 1536 dimensions; index "handbook" expects 4096.

Check what landed, without needing a query vector:

platformctl vectordb scroll "$INDEX" --limit 50

You should see the three points listed in id order, each with its payload.

At this point you have: a searchable index containing your documents. You could stop here and build any search UI you like on top of it — see search.

Act 5: make a credential for the agent​

Your agent will call the VectorDB API from inside the platform, so it needs its own credential. Do not hand it your personal one. Create a service account instead: a machine identity that belongs to this project and holds a project role the way a person does.

The member role is exactly enough: members can query an index, but cannot delete one. The same key also covers the embeddings call — the hosted models authorize per principal, not per project role, so any identity the platform recognizes may use them.

platformctl service-accounts create handbook-reader \
  --display-name "RAG chatbot" --role member

Now mint a key:

platformctl service-accounts keys create handbook-reader \
  --display-name handbook-bot --expires-in-days 90

You should see the key id and, once only, the secret itself.

Copy it now

The secret appears in that one response and nowhere else — only a hash is stored. If you lose it, revoke the key and create another.

export SA_KEY="cai_xxxxxxxx_yyyyyyyyyyyy"   # paste yours

Not a project admin? Use a personal API key instead (POST /v1/users/me/keys, available to any signed-in user). Be aware that it acts as you: every call the agent makes carries your identity. A service account is the right shape for a workload. See service accounts and API keys.

At this point you have: a credential scoped to this project, which you can revoke without touching your own account.

Act 6: write the agent​

The agent gets one tool: search_handbook. The model decides when to call it, the tool does the retrieval, and the instruction tells the model to answer only from what came back.

The retrieval logic is identical in all three frameworks — only the way the tool is declared, and the symbol the platform looks for, changes.

mkdir -p rag-chatbot && cat > rag-chatbot/agent.py <<'PYEOF'
import json
import os
import urllib.request

from google.adk.agents import Agent

from crusoe_adk.foundry import foundry_model


def _post(url, payload, token):
    request = urllib.request.Request(
        url,
        data=json.dumps(payload).encode(),
        headers={"Content-Type": "application/json",
                 "Authorization": "Bearer " + token},
    )
    with urllib.request.urlopen(request, timeout=30) as response:
        return json.load(response)


def _retrieve(question):
    # Read configuration HERE, at call time - never at import. See below.
    # Each platform address is injected; the HANDBOOK_* names only override it.
    vectordb_api = (os.environ.get("VECTORDB_API")
                    or os.environ["CAI_VECTORDB_URL"]).rstrip("/")
    embed_base = (os.environ.get("HANDBOOK_EMBED_BASE_URL")
                  or os.environ["EMBED_BASE_URL"]).rstrip("/")
    embed_model = os.environ.get("HANDBOOK_EMBED_MODEL") or os.environ["EMBED_MODEL"]
    index = os.environ.get("HANDBOOK_INDEX", "handbook")
    top_k = int(os.environ.get("HANDBOOK_TOP_K", "3"))

    embedded = _post(
        embed_base + "/embeddings",
        {"model": embed_model, "input": question},
        os.environ["HANDBOOK_EMBED_API_KEY"],
    )
    vector = embedded["data"][0]["embedding"]

    found = _post(
        "%s/v1/projects/%s/indexes/%s:query"
        % (vectordb_api, os.environ["CAI_PROJECT_ID"], index),
        {"vector": vector, "top_k": top_k, "with_payload": True},
        os.environ["VECTORDB_TOKEN"],
    )
    hits = found.get("results", [])
    if not hits:
        return "No matching handbook passage found."
    return "\n\n".join(
        "[%s] %s" % (hit.get("payload", {}).get("source", "handbook"),
                     hit.get("payload", {}).get("text", ""))
        for hit in hits
    )


def search_handbook(question: str) -> str:
    """Search the company handbook for passages that answer a question.

    Args:
        question: The question to look up, in plain English.

    Returns:
        The most relevant handbook passages, each labeled with its source file.
    """
    return _retrieve(question)


root_agent = Agent(
    name="handbook_bot",
    model=foundry_model(),
    instruction=(
        "You answer questions about the company handbook. Always call "
        "search_handbook first, and answer only from the passages it returns. "
        "Name the source file you used. If the passages do not contain the "
        "answer, say so plainly instead of guessing."
    ),
    tools=[search_handbook],
)
PYEOF

The docstring is the interface. ADK builds the tool's schema from the function signature and the docstring, so when the model decides how to call this tool, it reads your words: question: The question to look up, in plain English. Vague docstrings produce vague tool calls.

Four things are true of all three versions:

  • Only the standard library. urllib.request and json ship with Python, so there is no requirements.txt to get wrong.
  • CAI_PROJECT_ID is injected. The platform sets it on every workload, so the agent knows which project's index to query without you hard-coding a UUID.
  • The platform addresses are injected too. CAI_VECTORDB_URL and EMBED_BASE_URL/EMBED_MODEL arrive on every agent, pointing at internal platform addresses — which are not the public addresses your laptop uses. CAI_VECTORDB_URL is locked against being overridden, which is why an override has to have a name of its own; the code above takes VECTORDB_API, HANDBOOK_EMBED_BASE_URL and HANDBOOK_EMBED_MODEL when you set them and the injected values when you do not.
  • Configuration is read inside the tool, at call time — never at import. That is not a style preference. A deploy builds the revision before any of Act 7's configuration exists, so a module-level os.environ["VECTORDB_API"] raises KeyError while the harness is importing your file. The container dies during startup and restarts into the same failure, so the deploy never reaches ready: it ends as failed, or times out after five minutes, and the KeyError itself is only visible in platformctl logs rag-chatbot --history. Read it at call time and the first revision starts cleanly, with the tool the only thing that fails until you finish Act 7.

Deploy it:

platformctl deploy ./rag-chatbot --name rag-chatbot

You should see:

packaging ./rag-chatbot...
uploading rag-chatbot (1.9 KiB, framework=adk)...
build 7d41e0b2-...-uuid accepted
state:  -> building
state: building -> deploying
state: deploying -> ready
rag-chatbot is ready at https://rag-chatbot-ab12cd.apps.codyhill.dev

It is deployed but not yet configured. The addresses it needs are already injected; the credentials it needs are not, so the tool fails on its first call. That is the next act — and note that the agent reached ready regardless, because nothing outside the tool touches configuration.

At this point you have: a running agent whose one tool will fail until you wire it up.

Act 7: wire in the credentials and settings​

Three values go in by the secret route (never readable back) and two by the env route (readable). No address is among them: the agent gets internal VectorDB and inference addresses injected, and those are the ones that route from inside the platform.

The service account key does double duty. Both calls the tool makes — VectorDB and embeddings — are platform APIs that authenticate the same credential, so one key covers both.

Set the secrets first:

platformctl secrets set rag-chatbot \
  VECTORDB_TOKEN="$SA_KEY" \
  HANDBOOK_EMBED_API_KEY="$SA_KEY" \
  TOOL_SANDBOX=false

You should see:

set 3 secret(s) for rag-chatbot

Then the non-secret settings:

platformctl agents env set rag-chatbot \
  HANDBOOK_INDEX="$INDEX" \
  HANDBOOK_EMBED_MODEL="$EMBED_MODEL"
Do not point HANDBOOK_EMBED_BASE_URL at the public API

It is tempting to reuse $EMBED_BASE_URL from your shell here. Do not: that is https://api.codyhill.dev/v1, and the public hostname resolves back to this platform's own public front door — a hairpin that does not route from inside a project. Measured from a running agent, api.codyhill.dev:443 times out while inference-api on its internal address answers. What you see is a 30-second hang inside the tool, not an error naming the address. Leave it unset and the injected EMBED_BASE_URL is used: the same qwen-embedding your loader called from the public side, so the vectors line up.

A third-party provider is a different matter and is fine to set here — an agent may reach the public internet. It is this platform's own public hostname, specifically, that an agent cannot use.

Why TOOL_SANDBOX=false is needed here​

By default the platform runs your own tool code in a single-use sandbox, which is one throwaway container. It is built from your agent's image, but every environment variable is stripped out and private network addresses are blocked. The empty environment is the security boundary. A tool that has been talked into misbehaving by a crafted prompt cannot read your keys or reach internal services.

This tutorial's tool needs exactly the two things that boundary removes: environment variables (the credentials) and an internal address (the VectorDB API). So it has to run inside the agent itself, next to those credentials. That is what TOOL_SANDBOX=false means. Turn the sandbox off only for tool code you wrote and fully trust, as here. The full picture is in built-in tools.

Why changes are not instant​

Every secret, environment, or configuration change creates a brand-new revision — a frozen snapshot of the agent. Traffic moves to it once it starts. A running revision never changes underneath you. Both commands above rolled a new one, so wait for the newest to be serving:

platformctl status rag-chatbot

You should see state ready and ready true. platformctl status does not print the revision name. To confirm which revision is serving, call GET /v1/agents/rag-chatbot and read latest_revision — it should end in -00003. Or open the agent's page in the console. See traffic and revisions.

A tidier home for these credentials

platformctl secrets set writes a value onto one agent, with no versions and no sharing. Do several workloads use the same credential? Store it once in the project secrets manager and bind it to each agent by name. You get versions, rotation, and an audit trail. See use secrets in workloads.

At this point you have: a fully configured RAG agent.

Act 8: ask it things​

Start with a question the documents answer:

platformctl invoke rag-chatbot "How many days of paid time off do I get, and how many roll over?"

You should see:

You get 25 days of paid time off each year, and up to 5 unused days roll over into the next year. (Source: pto.md)
(session: 4b8e1f60-2c7a-4d95-8e13-9a0c5b2d7e41)
  tool_call: search_handbook called with args={'question': 'paid time off days roll over'}

The tool_call line is your receipt that retrieval actually happened. Now ask something phrased with none of the words in the document:

Do I need permission before buying a 900 dollar monitor?

You should see:

Yes. Anything above 500 dollars needs written manager approval before you buy it. (Source: expenses.md)

Note that expenses.md never says "monitor" or "permission". Vector search matched on meaning. That is the whole reason this works better than keyword search.

Now the most important test. Ask something the documents do not cover:

What is the parental leave policy?

You should see a reply saying the handbook does not cover parental leave, rather than an invented policy. Two things produce that behavior: one sentence in the agent's instruction ("say so plainly instead of guessing"), and the fact that the model can see only the retrieved passages. A RAG system that cannot say "I don't know" is worse than no RAG system.

At this point you have: a working, grounded, citing chatbot over your own documents.

Act 9: change a document and watch the answer change​

Retrieval reads live data. So edit a document, re-run the loader, and the next answer changes — with no redeploy:

cat > handbook/pto.md <<'EOF'
Paid time off. Every employee gets 30 days of paid time off each year.
Unused days roll over into the next year, up to a maximum of 10 days.
Request time off at least two weeks in advance.
EOF

python3 ingest.py
platformctl invoke rag-chatbot "How many PTO days do I get?"

You should see the new number, 30. The agent's image, revision, and code are untouched. Only the index changed. That separation is the practical argument for RAG over baking knowledge into a model: your documents change hourly, and model builds do not.

Clean up​

Delete everything you created, in this order: the agent, then the index, then the service account.

platformctl delete rag-chatbot
platformctl vectordb delete "$INDEX"
platformctl service-accounts delete handbook-reader

Deleting an index destroys every vector in it. There are no snapshots and no restore. Service account names stay reserved forever, even after deletion, so a new identity can never inherit an old one's access.

If something breaks​

SymptomCause and fix
index is not ready yet; its collection has not been createdYou wrote points before the index finished being created. Poll GET .../indexes/handbook until "ready": true.
point 0 has 1536 dimensions; index "handbook" expects 4096Your embedding model's output width does not match the index. Re-run python3 ingest.py dims, then create a new index with that number — dimensions cannot be edited.
query vector has 1536 dimensions; index "handbook" expects 4096The agent and the loader are using different embedding models. They must use the same one.
Search returns nothing relevant, but points existAlmost always the same cause as above: loaded with one model, queried with another. Confirm HANDBOOK_EMBED_MODEL on the agent matches EMBED_MODEL in your shell.
The tool hangs for 30 seconds and then reports a timeoutAn address you set on the agent does not route from inside the platform — almost always HANDBOOK_EMBED_BASE_URL or VECTORDB_API pointed at the public API. Remove it and let the injected EMBED_BASE_URL / CAI_VECTORDB_URL be used.
404 from a VectorDB call you believe should work404 means "does not exist or is not yours". The two are deliberately indistinguishable. Check the project UUID and the index name.
The deploy never reaches ready, and platformctl logs rag-chatbot --history shows a KeyErrorYour code reads an environment variable at import time. Act 6's deploy happens before Act 7's configuration exists, so the harness dies importing your file and the revision crash-loops. Read configuration inside the tool, as the examples do.
The tool errors with KeyError on an environment variableThe agent is still serving an older revision, or TOOL_SANDBOX was never set to false. A sandboxed tool call has an empty environment, so every variable lookup fails. Check platformctl status rag-chatbot, and set the switch by the secret route — platformctl secrets set rag-chatbot TOOL_SANDBOX=false; the env route answers 400 for that name.
403 this action requires the project admin roleCreating a service account and deleting an index are admin actions. Ask a project admin.
The agent answers without ever calling search_handbookStrengthen the instruction ("Always call search_handbook first"), and check the tool_calls list on each reply to confirm.
409 an index with that name already existsYou already created it. Move on, or pick another name.

More: Agent troubleshooting and VectorDB search.

What you learned​

IdeaThe one-sentence version
EmbeddingA list of numbers that captures a text's meaning; similar meanings sit close together.
IndexA named container of vectors with one fixed width and one distance metric, both permanent.
PointOne vector plus a JSON payload; the payload is what you show the user.
ChunkThe unit you embed and retrieve — also the unit of context the model reads.
RAGRetrieve first, then let the model answer from what you retrieved.
GroundingInstructing the model to use only the retrieved passages, and to admit when they fall short.
Service accountA project-scoped machine identity, so a workload never carries a person's credential.
RevisionAn immutable snapshot; every secret or config change makes a new one.

Next steps​

Go deeper​

These advanced guides pick up where the quickstarts stop, each exercising a different slice of the platform:

GuideFramework / language
Multi-step research agentLangGraph
Editorial pipeline with a crewCrewAI
Support agent over your own docsADK
Document ingestion pipelinePython
Webhook fan-out, exactly onceNode.js
Scheduled reconciliation jobGo
Object-store ETL with move-after-readRuby