Increase from e2-medium (2 vCPU / 4 GB) to e2-standard-4 to give
the full stack (app + kube-prometheus-stack + nginx ingress) enough
CPU headroom and avoid API rate limiter exhaustion on startup.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

'client rate limiter Wait: context deadline exceeded' happens when the
API server is still warming up on a fresh cluster and queued requests
exhaust the context before getting a rate-limiter slot.

Wait for all nodes to be Ready before any Helm operation so the API
server is fully available. Increase --timeout from 10m to 15m to give
more headroom on slow cluster starts.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

--set with memory values like '90Mi' can be misparsed by Helm's
value parser. Move resources to k8s/ingress-nginx-values.yaml.

Remove --wait: it blocks until the LoadBalancer gets an external IP,
which can exceed 5m on GCP and fail the job. The controller pods
starting is enough; the LB IP is assigned asynchronously.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

kube-prometheus-stack deploys operator, alertmanager, kube-state-metrics
and node-exporter without limits by default, causing CPU contention on
the node. Set requests/limits for every component.

NGINX Ingress Controller also lacked resource constraints; add via
Helm --set flags in the deploy job.

Total CPU requests across all components: ~560m / 1930m allocatable
on e2-medium, leaving headroom for kube-system and traffic spikes.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Full stack (app + monitoring + ingress + kube-system) requires ~860m
CPU and ~1.5GB RAM in requests. e2-small (2GB) had insufficient
headroom; e2-medium (2vCPU/4GB) provides ~45% CPU and ~55% RAM
utilization at request level, comfortable for production.

infra: e2-small → e2-medium
backend:    100m/500m CPU, 128Mi/256Mi RAM
frontend:   50m/200m CPU,  64Mi/128Mi RAM
prometheus: 100m/500m CPU, 400Mi/800Mi RAM
grafana:    50m/200m CPU,  128Mi/256Mi RAM

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Reduce application pod footprint and cap monitoring stack to fit
within ~1.7GB allocatable RAM on a single e2-small node.

backend:  100m/256m CPU, 128Mi/256Mi → 50m/200m CPU, 96Mi/192Mi
frontend: 50m/100m CPU, 64Mi/128Mi  → 20m/80m CPU, 48Mi/96Mi
prometheus: add 50m/300m CPU, 200Mi/400Mi limits
grafana:    add 20m/150m CPU, 80Mi/160Mi limits

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

If DATABASE_URL is defined in GitLab CI variables, use it directly.
Otherwise fetch the Cloud SQL private IP via gcloud and build the URL
from DB_PASSWORD variable, removing the need to hardcode the IP.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Regional cluster (location=us-central1) creates 1 node per zone =
3 nodes total. Switch to zonal (us-central1-a) for a single node.
Also downsize machine_type from e2-medium to e2-small (2vCPU/2GB).

Add var.zone to variables.tf and GCP_ZONE to CI variables.
Update get-credentials to use --zone instead of --region.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

- Restore ingressClassName: nginx and /grafana path in ingress.yaml
  (GCE ingress does not support ExternalName backends needed for
  cross-namespace Grafana routing)
- Install ingress-nginx via Helm in deploy-gke before applying
  manifests; --wait ensures the LoadBalancer IP is ready before
  the Ingress is created

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

NGINX Ingress Controller was not deployed; use GKE's built-in GCE
ingress which provisions a GCP HTTP Load Balancer automatically.

Remove /grafana path: GCE ingress does not support ExternalName
backends. Grafana remains accessible via kubectl port-forward.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

serviceAccountAdmin allows managing SAs but not assigning them to
resources. serviceAccountUser is required to attach a SA to GKE
node pools and other compute resources.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Replace single container.admin binding with for_each over the full
set of roles needed to provision VPC, GKE, Artifact Registry,
Cloud SQL, service accounts, IAM bindings, APIs and VPC peering.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Write GCP_SERVICE_ACCOUNT_KEY to /tmp/gcp-key.json and export
GOOGLE_APPLICATION_CREDENTIALS so the Google provider authenticates
automatically without requiring gcloud auth application-default login.

Also export TF_VAR_project_id from GCP_PROJECT_ID so the variable
is passed to Terraform without needing -var flags in the script.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Required approvals on environments is a Premium/Ultimate feature.
Keep only when:manual as the single gate for terraform-destroy.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Bind the job to the 'destroy-production' environment so GitLab blocks
execution until an authorized user approves it in the UI, in addition
to the existing when:manual gate.

Configure in GitLab: Settings > CI/CD > Protected Environments
  environment: destroy-production, required approvals: 1

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

- New 'destroy' stage at the end of the pipeline
- terraform-destroy job is when:manual so it only runs on explicit
  click in GitLab UI; allow_failure:false blocks accidental continuation
- Extract .terraform_setup anchor (image + before_script with init)
  shared between 'infra' and 'terraform-destroy' to avoid duplication
- Update variables comment to reflect container.admin and Terraform
  backend variables

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

- Section 5: explain the CORS problem that drove API_BASE='', show
  before/after fetch URLs, document the Vite proxy as the local dev
  equivalent
- Section 6: split into 6.1 (route/integration tests) and 6.2 (unit
  tests), describe what each individual test guarantees about route
  behavior and metric instrumentation

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

- Frontend: document API_BASE relative URL approach and Vite dev proxy
- Backend: add section 6 covering Jest/supertest test suite (9 tests)
- Repository structure: add metrics.js, __tests__/, k8s/monitoring/,
  infra/iam.tf to the tree
- Pipeline: add deploy-monitoring job, GRAFANA_ADMIN_PASSWORD variable,
  update SA role to roles/container.admin
- Observability: replace basic GCP logging section with full section 9
  covering Prometheus/Grafana stack, instrumented metrics, dashboard
  panels with PromQL, Grafana access via ExternalName service

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

rate()/$__rate_interval can return no data on stat panels when the
calculated interval doesn't align with scrape points. Switch to
increase()/$__range to match the same approach used by the other
stat panels and reliably capture all errors in the selected window.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Use $__rate_interval instead of fixed 5m, sum() to aggregate across
instances, and 'or on() vector(0)' so the numerator falls back to 0
when no 5xx series exist instead of leaving the panel blank.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Counters displayed with increase() return floats due to Prometheus
interpolation. Force integer display on the three total panels.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

challenges_updated_total has a status label, so increase() alone
returns one series per status value. Wrap with sum() to collapse
all statuses into a single total for the stat panel.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Replace raw counter with increase(...[$__range]) on the three stat
panels so they reflect events within the time window chosen in Grafana
instead of the all-time cumulative value.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Add stat panels for total created/deleted/updated challenges and a
full-width timeseries showing created vs deleted per period using
increase() over $__rate_interval.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>