A RAG assistant for a small site on AWS: architecture, evals and cost

The assistant on this site answers questions about my work: experience, services, how a project starts. It is a small retrieval-augmented generation (RAG) system, and it was built the way I build them for clients: measured before it shipped, limited so a bug or an abuser cannot run up the bill, and cheap to keep running. This article walks through the design and the numbers.

What it has to do, and what it must not do

A recruiter or a potential client asks a question; the assistant answers from what is written about me and links the sources it used. The rules that shaped everything else:

• Only from the sources. No guessed dates, clients, rates or availability. If the sources do not cover a question, the answer says so and points to the contact form.
• Answer in the visitor's language. The site is in English, French and Romanian, and visitors ask in all three, often with typos.
• Not a free consultant. "How do I set up an EKS cluster?" gets a pointer to the matching article and the contact form, not a tutorial.
• Bounded cost. Every request costs money, so abuse has to be stopped before it reaches the model.

The corpus: small, so chunk by meaning

The whole corpus is about 7,000 characters of CV, site sections and an FAQ I wrote for the assistant, plus the articles. At that size the usual advice (fixed windows of 300-500 tokens with overlap) does more harm than good: a window would mix two jobs from the CV into one chunk, and the answer could attribute one client's work to another.

Instead, one chunk is one unit of meaning: one job, one site section, one FAQ question, one article section. Chunks range from 160 to 1,450 characters, and each carries a recognisable title ("CV · Allianz Technology"), which is what the visitor sees under the answer.

The text comes from the same source file that generates the site, not from the PDF of my CV. Text extracted from a two-column PDF glues columns together, and a single source means the assistant cannot drift out of sync with the site.

Retrieval without a vector database

Each chunk is embedded once with Amazon Titan Text Embeddings v2 on Bedrock (1,024 dimensions, normalised). The result is a JSON file of a few megabytes (2.4 MB at about a hundred chunks), shipped inside the Lambda package. At query time the question is embedded and compared with every chunk by a dot product in NumPy, which takes about 0.02 milliseconds for a hundred chunks.

A managed vector database would add a service, permissions, a network hop and a monthly minimum for no measurable gain at this size. Packaging the index with the code has a side benefit: code and index are deployed together and rolled back together, so they can never disagree.

No relevance threshold. I planned to drop chunks below a minimum similarity score, then measured. Correct chunks scored between 0.11 and 0.70; questions the corpus cannot answer reached 0.36. The distributions overlap, so any threshold either loses correct answers ("Where did he study?" scores 0.11) or lets unrelated questions through. The top five chunks always go to the model, and the model decides whether they answer the question. That rule lives in the prompt and is covered by the evals.

Generation: short answers with citations

The model receives the top five chunks, numbered, and the visitor's question with the last two exchanges for follow-ups. The system prompt is short and specific: use only the sources, cite each sentence as [n], two to five sentences, third person, the visitor's language, no technical walkthroughs. The [n] markers are removed from the text and turned into the list of links under the answer, so the visitor sees clean prose and can still check every claim.

Follow-up questions such as "and when was that?" have no topic of their own, so retrieval uses the previous question together with the new one.

The fixed instructions are cached between requests, so a repeated system prompt costs a fraction of its first use. Effort is set low: the questions are factual and the corpus is small, so deeper reasoning adds latency without improving answers.

Choosing the model with an eval, not a hunch

I compared two models on the same question set: Claude Opus 5.5 and Claude Haiku 5.5, which is about 40 times cheaper. Both answered all 54 questions correctly. The larger model won on wording: concrete answers, no filler, graceful refusals. It costs about $0.009 per question, which at this site's volume is a few dollars a month, so quality won. On a high-traffic support bot the same eval could just as well have chosen the smaller model.

The eval set now has 75 questions:

• Answerable questions (59), mostly in English with some in Romanian and French, graded on whether the answer contains the expected facts and avoids known wrong ones.
• Questions only I can answer (rates that are not published, a start date), where the correct answer is "I don't have that information" plus the contact form.
• Off-topic and how-to questions, where the assistant must decline or point to an article.
• Typos, in English and Romanian. All passed, so there is no spell-correction step: it would add latency and a failure point without a measured gain.

Retrieval has its own check: for every question, the right chunk must be in the top five. It currently scores 65 out of 65.

That check caught a real problem when more articles arrived. An article has one chunk per section, so a single article could fill all five slots and push out the FAQ entry that says what I have actually done: the answer would recommend the article without mentioning my experience. Now at most two chunks from the same source make the top five.

An eval gate on the way to production

Evals that are not enforced get skipped. The site is deployed by CI, but the assistant's Lambda is built and deployed separately, so a CI job would not stop anything. Instead, the eval scripts record a pass against a fingerprint of everything that shapes an answer: the index, the retrieval and prompt code, the questions, and the model. The packaging script refuses to build if the current fingerprint has no recorded pass. Changing one line of the prompt means running the eval again before it can reach production.

Serving it: CloudFront, Lambda and three layers of limits

Browser -> CloudFront /api/chat (WAF) -> Lambda Function URL (IAM auth, signed by CloudFront)
                                            |-> embeddings (Bedrock)
                                            |-> model API (key in Secrets Manager)
                                            '-> DynamoDB (daily counter, 30-day log)

• Same origin. The API sits behind the site's own CloudFront distribution under /api/*, so there is no CORS and no second domain.
• Only CloudFront can call the function. The Function URL uses IAM authentication, and CloudFront signs requests with Origin Access Control. A direct call to the URL gets a 403. For POST requests the browser has to send a SHA-256 hash of the body, which the widget computes with the Web Crypto API.
• Limits, outermost first. A WAF rate limit per IP (20 requests per 5 minutes) plus AWS's list of IPs with a bad reputation; a global daily cap of 200 questions, counted atomically in DynamoDB; input size limits in the function; and a monthly spend limit on the model API account. Each layer fails with a friendly message that points to the contact form.
• Privacy. Questions and answers are kept for 30 days to improve the FAQ. The visitor's IP is never stored, only a salted hash that changes every day, so a visitor cannot be followed from one day to the next. The privacy page says exactly this.
• Alerts. Errors, throttling and a cost budget alert by email.

Streaming the answer

A complete answer takes three to seven seconds, and that long behind a typing indicator feels slow. The answer is now streamed: the first words appear after one to five seconds, depending on how long the model thinks first, and the rest follows as it is written.

Lambda streams responses natively only on Node.js runtimes. Rather than rewrite the assistant or add an adapter layer, the function runs a small Runtime API loop of its own (about 80 lines of standard-library Python) that sends the response in the Function URL streaming format, through CloudFront to the browser. Two details mattered:

• Citation markers are stripped while streaming. A filter holds back any tail that could still become a [3], so the visitor never sees a marker flash and disappear. A randomised test checks that the streamed text equals the stored answer for thousands of different chunk splits.
• CloudFront passes the stream straight through. The first text reaches the browser about 0.2 seconds after the function sends it.
• The eval runs through the streaming path. The non-streaming function used by the eval is built on the streaming one, so what is measured is what visitors get.

What it costs

At around 900 questions a month, the estimate is:

The largest fixed cost is the firewall, not the model. I accept it as the price of putting an endpoint that costs money per call on the public internet.

What I would do the same way on a client project

• Write the eval set before choosing the model, and let it decide.
• Chunk by meaning when documents have natural units; measure before adding thresholds.
• Keep the index next to the code until the corpus is large enough to need a database.
• Put the eval on the deployment path, not in a report.
• Limit cost in layers, outside the code first.

If you are planning an assistant over your own documentation, support content or internal knowledge, this is the kind of work I do. The AI engineering page describes how a project starts, and the contact form is the quickest way to reach me.