What it has to do, and what it must not do

A recruiter or a potential client asks a question; the assistant answers from what is written about me and links the sources it used. The rules that shaped everything else:

• Only from the sources. No guessed dates, clients, rates or availability. If the sources do not cover a question, the answer says so and points to the contact form.
• Answer in the visitor's language. The site is in English, French and Romanian, and visitors ask in all three, often with typos.
• Not a free consultant. "How do I set up an EKS cluster?" gets a pointer to the matching article and the contact form, not a tutorial.
• Bounded cost. Every request costs money, so abuse has to be stopped before it reaches the model.

The corpus: small, so chunk by meaning

The whole corpus is about 7,000 characters of CV, site sections and an FAQ I wrote for the assistant, plus the articles. At that size the usual advice (fixed windows of 300-500 tokens with overlap) does more harm than good: a window would mix two jobs from the CV into one chunk, and the answer could attribute one client's work to another.

Instead, one chunk is one unit of meaning: one job, one site section, one FAQ question, one article section. Chunks range from 160 to 1,450 characters, and each carries a recognisable title ("CV · Allianz Technology"), which is what the visitor sees under the answer.

The text comes from the same source file that generates the site, not from the PDF of my CV. Text extracted from a two-column PDF glues columns together, and a single source means the assistant cannot drift out of sync with the site.

Retrieval without a vector database

Each chunk is embedded once with Amazon Titan Text Embeddings v2 on Bedrock (1,024 dimensions, normalised). The result is a JSON file of a few megabytes (2.4 MB at about a hundred chunks), shipped inside the Lambda package. At query time the question is embedded and compared with every chunk by a dot product in NumPy, which takes about 0.02 milliseconds for a hundred chunks.

A managed vector database would add a service, permissions, a network hop and a monthly minimum for no measurable gain at this size. Packaging the index with the code has a side benefit: code and index are deployed together and rolled back together, so they can never disagree.

No relevance threshold. I planned to drop chunks below a minimum similarity score, then measured. Correct chunks scored between 0.11 and 0.70; questions the corpus cannot answer reached 0.36. The distributions overlap, so any threshold either loses correct answers ("Where did he study?" scores 0.11) or lets unrelated questions through. The top five chunks always go to the model, and the model decides whether they answer the question. That rule lives in the prompt and is covered by the evals.

Generation: short answers with citations

The model receives the top five chunks, numbered, and the visitor's question with the last two exchanges for follow-ups. The system prompt is short and specific: use only the sources, cite each sentence as [n], two to five sentences, third person, the visitor's language, no technical walkthroughs. The [n] markers are removed from the text and turned into the list of links under the answer, so the visitor sees clean prose and can still check every claim.

Follow-up questions such as "and when was that?" have no topic of their own, so retrieval uses the previous question together with the new one.

The fixed instructions are cached between requests, so a repeated system prompt costs a fraction of its first use. Effort is set low: the questions are factual and the corpus is small, so deeper reasoning adds latency without improving answers.

Choosing the model with an eval, not a hunch

I compared two models on the same question set: Claude Opus 5.5 and Claude Haiku 5.5, which is about 40 times cheaper. Both answered all 54 questions correctly. The larger model won on wording: concrete answers, no filler, graceful refusals. It costs about $0.009 per question, which at this site's volume is a few dollars a month, so quality won. On a high-traffic support bot the same eval could just as well have chosen the smaller model.

The eval set now has 75 questions:

• Answerable questions (59), mostly in English with some in Romanian and French, graded on whether the answer contains the expected facts and avoids known wrong ones.
• Questions only I can answer (rates that are not published, a start date), where the correct answer is "I don't have that information" plus the contact form.
• Off-topic and how-to questions, where the assistant must decline or point to an article.
• Typos, in English and Romanian. All passed, so there is no spell-correction step: it would add latency and a failure point without a measured gain.

Retrieval has its own check: for every question, the right chunk must be in the top five. It currently scores 65 out of 65.

That check caught a real problem when more articles arrived. An article has one chunk per section, so a single article could fill all five slots and push out the FAQ entry that says what I have actually done: the answer would recommend the article without mentioning my experience. Now at most two chunks from the same source make the top five.

An eval gate on the way to production

Evals that are not enforced get skipped. The site is deployed by CI, but the assistant's Lambda is built and deployed separately, so a CI job would not stop anything. Instead, the eval scripts record a pass against a fingerprint of everything that shapes an answer: the index, the retrieval and prompt code, the questions, and the model. The packaging script refuses to build if the current fingerprint has no recorded pass. Changing one line of the prompt means running the eval again before it can reach production.

Serving it: CloudFront, Lambda and three layers of limits

Browser -> CloudFront /api/chat (WAF) -> Lambda Function URL (IAM auth, signed by CloudFront)
                                            |-> embeddings (Bedrock)
                                            |-> model API (key in Secrets Manager)
                                            '-> DynamoDB (daily counter, 30-day log)

• Same origin. The API sits behind the site's own CloudFront distribution under /api/*, so there is no CORS and no second domain.
• Only CloudFront can call the function. The Function URL uses IAM authentication, and CloudFront signs requests with Origin Access Control. A direct call to the URL gets a 403. For POST requests the browser has to send a SHA-256 hash of the body, which the widget computes with the Web Crypto API.
• Limits, outermost first. A WAF rate limit per IP (20 requests per 5 minutes) plus AWS's list of IPs with a bad reputation; a global daily cap of 200 questions, counted atomically in DynamoDB; input size limits in the function; and a monthly spend limit on the model API account. Each layer fails with a friendly message that points to the contact form.
• Privacy. Questions and answers are kept for 30 days to improve the FAQ. The visitor's IP is never stored, only a salted hash that changes every day, so a visitor cannot be followed from one day to the next. The privacy page says exactly this.
• Alerts. Errors, throttling and a cost budget alert by email.

Streaming the answer

A complete answer takes three to seven seconds, and that long behind a typing indicator feels slow. The answer is now streamed: the first words appear after one to five seconds, depending on how long the model thinks first, and the rest follows as it is written.

Lambda streams responses natively only on Node.js runtimes. Rather than rewrite the assistant or add an adapter layer, the function runs a small Runtime API loop of its own (about 80 lines of standard-library Python) that sends the response in the Function URL streaming format, through CloudFront to the browser. Two details mattered:

• Citation markers are stripped while streaming. A filter holds back any tail that could still become a [3], so the visitor never sees a marker flash and disappear. A randomised test checks that the streamed text equals the stored answer for thousands of different chunk splits.
• CloudFront passes the stream straight through. The first text reaches the browser about 0.2 seconds after the function sends it.
• The eval runs through the streaming path. The non-streaming function used by the eval is built on the streaming one, so what is measured is what visitors get.

What it costs

At around 900 questions a month, the estimate is:

The largest fixed cost is the firewall, not the model. I accept it as the price of putting an endpoint that costs money per call on the public internet.

What I would do the same way on a client project

• Write the eval set before choosing the model, and let it decide.
• Chunk by meaning when documents have natural units; measure before adding thresholds.
• Keep the index next to the code until the corpus is large enough to need a database.
• Put the eval on the deployment path, not in a report.
• Limit cost in layers, outside the code first.

If you are planning an assistant over your own documentation, support content or internal knowledge, this is the kind of work I do. The AI engineering page describes how a project starts, and the contact form is the quickest way to reach me.

Start with where the rest of your system lives

Before comparing clusters, look at everything else. A cluster is rarely the only thing a team runs: there are databases, queues, object storage, identity, networking to offices or partners, and often a data platform.

• If most of it is already on AWS, EKS is usually the right call. Pods reach RDS, SQS and S3 over the same VPC, with IAM roles you already manage, and data stays inside one provider's network and bill.
• If you are on Google Cloud, or your data lives in BigQuery, GKE is the natural choice for the same reasons.
• Starting from nothing, both are viable. Then the operational trade-offs below decide it.

Cross-cloud traffic costs money and adds latency, and two IAM systems double the access reviews. Moving a cluster to the "better" Kubernetes service while the databases stay behind is almost never worth it.

What the price list says

The control plane is the easy part to compare. At the time of writing:

Prices change, so check the EKS pricing page and the GKE pricing page before you budget.

Two things in this table matter more than the base fee:

• Extended support is a tax on not upgrading. Running an old Kubernetes version costs six times the normal control plane fee on both platforms. A team that cannot upgrade regularly pays for it every hour. The fix is a tested upgrade procedure, not a cheaper provider.
• The fee is per cluster. One cluster per environment per team multiplies quickly. Three environments for four teams is twelve control planes before a single workload runs. Fewer, shared clusters with namespaces and policies are usually cheaper and easier to keep upgraded.

Where the real money goes

For a mid-size team, the control plane is a rounding error next to these:

• Compute. Nodes are most of the bill on both platforms. What matters is how well they are packed: requests that match real usage, autoscaling that removes idle nodes, and spot or preemptible capacity for workloads that tolerate interruption. A badly sized cluster on the cheaper platform costs more than a well-sized one on the other.
• Networking. NAT gateways, load balancers and cross-zone traffic add up. On AWS in particular, a NAT gateway per zone plus data processing charges can surprise a team that pulls large images or talks to external APIs a lot. VPC endpoints for the services you call most reduce it.
• Observability. Managed logging and metrics on either platform charge by volume. Verbose application logs shipped by default can cost more than the cluster.
• People. The hours your platform engineers spend on upgrades, add-ons and incidents. This is where the two services differ most.

Operational trade-offs

GKE does more out of the box

GKE has been managed Kubernetes for longer, and it shows in the defaults:

• Autopilot runs the nodes for you. You describe pods; Google provisions and patches the capacity, and you pay for what the pods request. For a team without a dedicated platform engineer, this removes a whole category of work.
• Release channels and maintenance windows make upgrades routine. Clusters follow a channel and upgrade inside the windows you set.
• Integrated logging and metrics are on by default.

The trade-off is control. Autopilot restricts privileged workloads, some DaemonSets and node-level tuning. Most business applications never notice; some security and networking agents do.

EKS gives you more pieces to assemble

An EKS cluster is closer to raw Kubernetes. A production setup needs decisions and code for:

• the VPC CNI and IP address planning, since every pod gets a VPC address;
• a node autoscaler, usually Karpenter, which is very good at bin packing and spot usage once configured;
• the AWS Load Balancer Controller, ExternalDNS and certificates;
• pod identity so workloads get AWS permissions through their own roles;
• add-on versions that must be upgraded together with the cluster.

EKS has closed part of the gap with managed add-ons, EKS Auto Mode and Pod Identity, but the cluster still asks more design decisions of you up front. In exchange you get fine control over networking, instance types and how capacity is bought, and everything is in the same AWS account model as the rest of your infrastructure.

Upgrades

Both providers release a new Kubernetes minor version every few months, and both charge extra for staying behind. In practice:

• GKE upgrades the control plane and nodes for you within the release channel and maintenance windows. Your job is to keep workloads compatible: deprecated APIs, Pod Disruption Budgets and enough replicas.
• EKS upgrades the control plane on request; nodes and add-ons are yours to roll. With managed node groups or Karpenter this is a routine, scripted job, but it is a job.

On either platform, the clusters that fall behind are the ones where nobody owns the upgrade. Whatever you choose, put the procedure in writing and run it on a staging cluster first.

How I help a team decide

When a team asks me to choose, I go through these questions:

1. Where do the data and the other services live? This usually decides it.
2. Who will operate the cluster? No dedicated platform person points towards GKE Autopilot, or towards a deliberately small EKS setup built entirely in Terraform with few add-ons.
3. Do you need node-level control? Custom networking, privileged agents or specific instance types favour EKS or GKE Standard.
4. How many clusters do you really need? Usually fewer than planned.
5. What will the first year cost, including compute at realistic utilisation, networking and observability, not only the control plane?

Either way, the cluster goes into Terraform from day one, with the same review process as application code. The checklist from A production-ready Kubernetes cluster on EKS with Terraform applies to GKE almost unchanged.

The short version

• Pick the platform where the rest of your system already lives.
• Starting fresh with a small platform team, GKE Autopilot needs the least operational work.
• If you need fine control or are already deep in AWS, EKS is excellent once the add-ons are in code.
• The control plane fee is the same; compute packing, networking, observability and people decide the bill.
• Upgrade on schedule. Extended support costs six times the standard fee on both.

If you are choosing between the two, or want a review of what an existing cluster really costs, this is the kind of work I do. The contact form is the quickest way to reach me.

Rule one: page on symptoms, not causes

A page should mean users are affected, or will be very soon. That means alerting on what users experience:

• requests failing (error rate);
• requests too slow (latency);
• the service unreachable;
• work not getting done (queue age, jobs not completing).

High CPU, a pod restart, a full-ish disk or a replica down are causes. They might lead to user impact, or they might not: autoscaling, retries and redundancy exist precisely so that they don't. Causes belong on dashboards and in tickets, not in someone's phone at night.

The exception is a cause that will certainly become an outage with no automatic recovery, such as a disk that will be full within hours, or a certificate that expires in days. Those are predictable, so alert on the prediction:

- alert: DiskWillFillIn4Hours
  expr: predict_linear(node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"}[6h], 4 * 3600) < 0
  for: 30m
  labels:
    severity: page
  annotations:
    summary: "{{ $labels.instance }} {{ $labels.mountpoint }} will be full within 4 hours"
    runbook: "https://runbooks.example.com/disk-full"

Two severities, two destinations

I use two levels and resist adding more:

• page: a person must act now, at any hour. Goes to the on-call phone.
• ticket: someone must look within working hours. Goes to a channel or the issue tracker.

Everything else is not an alert. If no one would do anything about it, it is a dashboard panel. A "warning" level that goes to a channel nobody reads is just noise with a nicer name.

Alert on error budgets, not on single spikes

A five-minute error spike that fixes itself does not need a human. A small but steady error rate that will eat a month's reliability target in two days does. Static thresholds handle both badly. SLO burn-rate alerts handle both well.

Start with a target, for example: 99.9% of requests succeed over 30 days. That leaves an error budget of 0.1%. The burn rate is how fast you are spending it: a burn rate of 1 uses the budget in exactly 30 days, and 14.4 uses it in about two days.

First, a recording rule for the error ratio over several windows:

groups:
- name: api-slo
  rules:
  - record: job:request_errors:ratio_rate5m
    expr: |
      sum by (job) (rate(http_requests_total{code=~"5.."}[5m]))
      /
      sum by (job) (rate(http_requests_total[5m]))
  - record: job:request_errors:ratio_rate1h
    expr: |
      sum by (job) (rate(http_requests_total{code=~"5.."}[1h]))
      /
      sum by (job) (rate(http_requests_total[1h]))

Then alert when both a long and a short window burn fast. The long window proves it is sustained; the short one makes the alert stop soon after the problem is fixed:

  - alert: ApiErrorBudgetBurn
    expr: |
      job:request_errors:ratio_rate1h{job="api"} > (14.4 * 0.001)
      and
      job:request_errors:ratio_rate5m{job="api"} > (14.4 * 0.001)
    for: 2m
    labels:
      severity: page
    annotations:
      summary: "API is burning its error budget 14x too fast"
      runbook: "https://runbooks.example.com/api-errors"

A second rule with a lower burn rate over longer windows (for example 6 hours and 30 minutes at a burn rate of 6) catches slower problems. For slow, steady leaks, a ticket-level alert at a burn rate of 1 over three days is enough. This multi-window approach is described in detail in Google's SRE workbook, and it is the single change that removes the most noise in the setups I review.

Make every alert actionable

Each alert rule should answer three questions for the person who receives it:

• What is broken, in the summary, in plain words and with the affected service or instance.
• How bad, through the severity and, where useful, the current value.
• What to do, through a runbook link: how to confirm the problem, the usual causes, and how to mitigate.

An alert without a runbook is a sign the alert is not understood yet. Writing the runbook often shows that the alert should be a dashboard panel instead.

Use for: so that a single bad scrape does not page anyone, but keep it short on paging alerts. The burn-rate windows already smooth out spikes.

Let Alertmanager do the grouping

Many noisy setups are really routing problems. Alertmanager can turn fifty notifications into one:

route:
  receiver: tickets
  group_by: [alertname, cluster, service]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers: [severity="page"]
      receiver: on-call
      repeat_interval: 1h

inhibit_rules:
  - source_matchers: [alertname="ClusterUnreachable"]
    target_matchers: [severity=~"page|ticket"]
    equal: [cluster]

• Grouping sends one notification for all pods of a failing service, not one per pod.
• Inhibition suppresses the symptoms of a bigger failure. If the whole cluster is unreachable, nobody needs forty more alerts about the services in it.
• Silences are for planned maintenance. Create them with an expiry and a comment, never as a permanent mute.

Test alerts like code

Alert rules are code, so they live in Git and go through review. Two tools make that real:

• promtool check rules in CI catches syntax errors before they reach Prometheus.
• promtool test rules runs unit tests: given this series of metric values, this alert must fire after this many minutes, with these labels. It is the only way to be sure an alert fires when it should and stays quiet when it should not.

A dead man's switch completes the picture: an alert that always fires, routed to an external service that pages you if it stops arriving. Otherwise a broken Prometheus or Alertmanager is silent, which looks exactly like a quiet night.

Review the pages every week

The rules above get you a good start. What keeps alerting healthy is a short weekly review of every page from the past week:

• Was it real? Did someone have to act?
• Was it actionable from the alert and the runbook alone?
• Could it have waited until morning?

Every page that fails one of these questions gets a change: delete it, downgrade it to a ticket, raise the threshold, fix the runbook, or fix the underlying problem so it stops happening. A useful target is no more than one or two pages per on-call shift; beyond that, people stop trusting the pager.

Summary

• Page on user-facing symptoms. Causes go on dashboards, except predictable failures.
• Two severities: page now, or ticket for working hours.
• Use SLO burn-rate alerts with a long and a short window instead of static thresholds.
• Every alert has a clear summary and a runbook.
• Group and inhibit in Alertmanager; silences expire.
• Test rules with promtool, and add a dead man's switch.
• Review every page weekly and remove the ones that did not need a human.

If your team is tired of its pager, or you are setting up Prometheus and Grafana for the first time, this is the kind of work I do: dashboards, alerts and SLOs that the team trusts. The contact form is the quickest way to reach me.

The goals

A multi-environment layout should give you:

• One definition, several instances. Staging and production are built from the same code. Differences are explicit inputs, not edits in a copied file.
• Small blast radius. A change to the DNS layer cannot destroy a database. A mistake in dev cannot touch production.
• Promotion. A change is applied to dev, then staging, then production, with the same code at each step.
• Readable plans. A reviewer can see exactly what will change, in which environment, before it happens.

Accounts first

On AWS, I put each environment in its own account: dev, staging and production at least, plus a shared account for things like DNS and container registries. AWS Organizations and IAM Identity Center make this cheap to run.

Separate accounts are the strongest isolation AWS offers. Permissions, quotas and billing are separated by default, and a wrong role in a pipeline cannot reach production resources because it does not have credentials there. Folders and workspaces inside one account do not give you that.

The layout

modules/
  network/          # VPC, subnets, NAT, endpoints
  eks-cluster/      # cluster, node groups, add-ons
  rds-postgres/     # database, parameter group, alarms
  service/          # one application's IAM, queues, buckets
live/
  dev/
    network/        # main.tf calls modules/network with dev inputs
    eks/
    data/
  staging/
    network/
    eks/
    data/
  prod/
    network/
    eks/
    data/

Two kinds of directories, with different rules:

• modules/ contain resources and logic, and no environment names. They take inputs, such as CIDR ranges, instance sizes and the number of replicas, and expose outputs. They have no backend and no provider credentials.
• live/<env>/<component>/ are root modules: the directories you run terraform plan in. Each one is short. It configures the backend and provider for its account, calls one or more modules with that environment's values, and nothing else.

Split state by component

Each directory under live/ has its own state. I split by how often things change and how much damage a mistake can do:

• network: changes rarely, everything depends on it;
• cluster or compute: changes with upgrades;
• data: databases and buckets, protected with prevent_destroy and deletion protection;
• services: change often, owned by application teams.

A small state plans in seconds and can only break what it contains. Components read what they need from each other through outputs, either with terraform_remote_state or, better, by looking resources up by tag or name with data sources, so the dependency is on the resource and not on another state file's layout.

State lives in S3 in each environment's own account, with locking and versioning. Recent Terraform versions lock with a lock file in the same bucket, so a separate DynamoDB table is no longer needed.

Modules worth having

The temptation is to wrap every AWS resource in a module. That produces a thin layer that hides nothing and must be updated with every provider release. I write modules for units that mean something to the team:

• "A network", not "a subnet": VPC, subnets in three zones, routing, NAT and endpoints, with sensible defaults.
• "A Postgres database for a service": the instance or cluster, parameter group, subnet group, security group, backups, alarms and the secret.
• "A service": its IAM role, queues, buckets and the permissions between them.

A good module has few required inputs, safe defaults (encryption on, public access off, deletion protection on for data), and outputs that other components need. Where a well-maintained community module already does the job, such as the VPC or EKS modules, I use it pinned to a version rather than rewriting it.

Versioning and promotion

Modules are versioned. In a mono-repo that can be a Git tag referenced by ?ref=; in a larger organisation, a private module registry. Each environment pins a version:

module "network" {
  source = "git::https://gitlab.example.com/platform/terraform-modules.git//network?ref=network-v1.4.0"

  name       = "prod"
  cidr_block = "10.30.0.0/16"
  azs        = ["eu-west-1a", "eu-west-1b", "eu-west-1c"]
}

Promotion is then a one-line change: dev moves to network-v1.5.0 first, then staging, then production. The diff between environments is visible in Git, and rolling back is reverting that line.

For a small team with one repository, relative paths (source = "../../../modules/network") are fine, as long as changes are still applied environment by environment.

Keeping the root modules short

The copy-paste problem comes back if root modules grow. A few rules keep them thin:

• Environment differences are inputs. Sizes, counts, CIDRs and feature flags go into the module call or a small terraform.tfvars. No if env == "prod" logic inside modules; pass the value instead.
• Providers carry the account. Each root assumes a role in its environment's account and sets default tags (environment, owner, cost centre), so every resource is tagged without anyone remembering to.
• Pin versions of Terraform and providers in every root, and upgrade them deliberately, environment by environment.

Wrappers like Terragrunt can generate the backend and provider blocks for you. They are useful with dozens of roots; with a handful, plain Terraform and a short template are easier for everyone to read.

The review and apply flow

Structure only helps if changes go through it:

1. A merge request changes a module or a root.
2. CI runs terraform fmt -check, validate, a linter and a security scanner, then plan for every root the change touches, and posts the plans on the merge request.
3. A reviewer reads the plan, not only the code.
4. After merge, CI applies the saved plan to dev automatically; staging and production apply after a manual approval, using the plan that was reviewed.

Nobody applies from a laptop, and nobody clicks in the console. If something has to be changed by hand during an incident, it is imported or reverted the next day, and a scheduled drift check (plan with no changes expected) shows anything that was missed.

Moving existing code into this layout

You rarely start from zero. The migration I use keeps resources in place:

1. Pick one environment's copy as the reference, usually production, and turn it into modules with the same resources.
2. Move state, not resources. moved blocks, or terraform state mv for older versions, map old addresses to the new module paths. The plan must show no changes before anything else happens.
3. Bring the other environments onto the same modules one at a time, turning every difference into an explicit input. Each difference you find is a question for the team: intentional, or drift?
4. Import what was created by hand, with import blocks, so it is managed from then on.

Done carefully, every step ends with a plan that shows no changes, and production never notices.

Summary

• One account per environment; state per environment and component.
• Modules for meaningful units, with safe defaults and no environment names.
• Root modules that only wire modules together for one environment.
• Pinned module versions, promoted from dev to production.
• Plans reviewed in merge requests and applied by CI.

If your Terraform has grown into copied directories, or you are setting up AWS environments from scratch, this is work I do often. The contact form is the quickest way to start.

What "production-ready" means

Before writing any Terraform I agree on a short definition with the team. A cluster is ready for production when:

• Everything is in code. The VPC, the cluster, node groups, add-ons and access rules are created by Terraform from a reviewed merge request. Nobody clicks in the console.
• It survives a zone failure. Nodes and load balancers span at least three availability zones, and workloads run more than one replica.
• Access is identity-based. People and pipelines get Kubernetes permissions through IAM, and pods get AWS permissions through their own roles, never through the node role.
• It scales on its own. Pods scale on load, nodes scale on pending pods, and there are limits so a bug cannot scale the bill.
• You can see what it does. Metrics, logs and alerts exist before the first release, not after the first incident.
• It can be upgraded. There is a tested procedure to move to the next Kubernetes version every few months.

Network first

The cluster lives in its own VPC with private subnets for nodes and public subnets only for load balancers. A few decisions here are expensive to change later:

• CIDR size. The VPC CNI gives every pod a VPC IP address. A /16 with /19 private subnets leaves room to grow; a /24 does not. If addresses are tight, prefix delegation or a secondary CIDR for pods are the usual fixes.
• Three availability zones, one private and one public subnet in each.
• NAT gateways. One per zone for resilience, or a single one to save cost in non-production environments. VPC endpoints for ECR, S3 and STS cut NAT traffic for image pulls and API calls.
• Subnet tags so the load balancer controller knows where to place internal and internet-facing load balancers.

Terraform layout

I keep the cluster in its own Terraform root module with remote state and locking, separate from the applications that run on it. Most of the heavy lifting is done by the well-maintained community modules for the VPC and EKS, pinned to a major version:

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 21.0"

  name               = "platform-prod"
  kubernetes_version = var.kubernetes_version

  vpc_id     = module.vpc.vpc_id
  subnet_ids = module.vpc.private_subnets

  endpoint_public_access = false   # API reachable from the VPC / VPN only
  enable_irsa            = true

  eks_managed_node_groups = {
    system = {
      instance_types = ["m7g.large"]
      ami_type       = "AL2023_ARM_64_STANDARD"
      min_size       = 3
      max_size       = 6
      desired_size   = 3
    }
  }
}

A few habits that save trouble later:

• Pin everything: provider versions, module versions and the Kubernetes version. Upgrades become deliberate merge requests instead of surprises.
• One state per environment, with the same code and different variables, so staging really rehearses production.
• Plan in CI, apply after review. The pipeline posts the plan on the merge request; a person reads it before anything changes.

Identity and access

Two separate questions: who can talk to the Kubernetes API, and what AWS resources a pod can reach.

• People and pipelines get access through EKS access entries mapped to IAM roles, with Kubernetes RBAC on top. Engineers assume a role through SSO; the deploy pipeline has its own narrow role.
• Pods get AWS permissions through IAM roles for service accounts (or EKS Pod Identity). Each service gets only what it needs, such as reading one S3 bucket or one secret.
• Nodes keep a minimal role, and IMDSv2 with a hop limit of 1 stops pods from borrowing the node's credentials.

Add-ons that every cluster needs

I manage these with Terraform or Helm, versioned like everything else:

• Core EKS add-ons: VPC CNI, CoreDNS, kube-proxy and the EBS CSI driver for persistent volumes.
• AWS Load Balancer Controller to expose services through ALBs and NLBs from Ingress and Service objects.
• External DNS and cert-manager, if the cluster owns DNS records and certificates.
• External Secrets to sync values from Secrets Manager or Parameter Store into Kubernetes, so secrets never sit in Git.
• metrics-server, which the horizontal pod autoscaler depends on.

Autoscaling and cost

• Pods: horizontal pod autoscalers on CPU, memory or request rate, with sensible minimum replicas and pod disruption budgets.
• Nodes: Karpenter (or the Cluster Autoscaler) adds capacity when pods are pending and removes empty nodes. Graviton instances and Spot capacity for stateless workloads usually cut compute cost noticeably.
• Guardrails: resource requests and limits on every workload, namespace quotas and a maximum node count, so a runaway deployment cannot grow without bounds.

Observability from day one

• Metrics: Prometheus scraping the cluster and the applications, Grafana dashboards per service, and alerts on symptoms users feel (error rate, latency, saturation) rather than on every CPU spike.
• Logs: container logs shipped to a central store with a retention policy.
• Control plane logs: API server and audit logs enabled in EKS, so you can answer "who changed this?".

Security baseline

• Private API endpoint, or a public one restricted to known CIDRs.
• Pod Security Standards enforced per namespace, with restricted as the default for application namespaces.
• Network policies so namespaces cannot talk to each other unless they need to.
• Image scanning in the pipeline, and images pulled only from your own registry.
• Encryption of Kubernetes secrets with a KMS key.

Upgrades

EKS versions leave standard support after about 14 months, so upgrades are routine work, not a project. My usual order is: read the upgrade notes and check for deprecated APIs, upgrade staging, then the control plane in production, then add-ons, then node groups with a rolling replacement that respects pod disruption budgets. Because everything is in Terraform, the upgrade is a short merge request plus a checklist.

Checklist before the first workload

1. VPC across three zones, private nodes, endpoints for ECR, S3 and STS.
2. Cluster, node groups and add-ons created from Terraform in CI, state remote and locked.
3. Access entries and RBAC for people and pipelines; IRSA or Pod Identity for workloads; IMDSv2 enforced.
4. Load balancer controller, External Secrets and metrics-server installed and versioned.
5. Pod and node autoscaling with limits; requests and limits on every workload.
6. Prometheus, Grafana, logs and alerts working, with a test alert delivered.
7. Pod Security Standards, network policies and secret encryption enabled.
8. Upgrade procedure written down and rehearsed on staging.

If your team needs a cluster like this, or a review of one that already runs in production, I can help: I design and build it with you, and hand it over with documentation your team can maintain.