Perception

Overview

Perception is a dashboard for a market-research consultancy. It takes what people say about brands (reviews, social posts, creator content, survey answers), sorts it into themes and sentiment, and lets analysts compare brands and export the results straight into client decks. A chat assistant answers questions about the data in plain language.

It runs as one installation that serves several client workspaces. Each user sees only the workspaces an admin has granted them, and every row in the database is tagged with the workspace it belongs to.

This is client work under NDA, so this write-up covers the architecture and infrastructure. It leaves out clients, data sources and how the analysis itself works.

My role

A four-person team built it, and I'm the largest contributor by commits. Roughly, I owned:

• Dashboard features: most of the chart tabs analysts use every day, with filters and period controls that follow the active workspace.
• Native exports: every tab exports to PowerPoint as real, editable charts and tables, not screenshots, and to Excel.
• A self-serve survey data-file builder: viewers define their own breaks and download counts and column percentages as a styled spreadsheet.
• CI/CD: GitHub Actions to production through OIDC and SSM, with migrations and health checks on every deploy.
• Hardening: a security and correctness pass on the Go API, an authenticated ingest endpoint, every test running in CI, and the database moved off the public internet.
• Cost: an infrastructure cost audit that cut the monthly AWS bill.
• Docs: documentation good enough that a new engineer can add a client workspace without asking anyone.

Architecture

Everything runs on AWS. One EC2 host runs the whole stack in Docker Compose behind Caddy, which routes by path so the browser only ever talks to one origin and the session cookie stays first-party.

flowchart LR
  subgraph client["Browser"]
    ui["Dashboard - Next.js"]
  end
  subgraph ec2["EC2 host - Docker Compose"]
    caddy["Caddy - TLS and path routing"]
    api["Query API - Go Fiber"]
    etl["Data service - FastAPI"]
    web["Next.js server"]
    jobs["Scheduled collectors"]
  end
  subgraph aws["AWS managed"]
    rds[("RDS PostgreSQL 16 + pgvector")]
    bedrock["Bedrock - LLMs"]
    lambda["Lambda fetchers - multi-region"]
  end
  ui -->|"HTTPS, one origin"| caddy
  caddy -->|"/api/chat, /api/ingest"| etl
  caddy -->|"/api"| api
  caddy -->|"everything else"| web
  api --> rds
  etl --> rds
  etl -->|"assistant"| bedrock
  jobs -->|"fetch pages"| lambda
  jobs -->|"classify"| bedrock
  jobs -->|"write results"| rds

• Caddy terminates TLS and splits /api traffic by specificity: chat and ingest go to the Python service, the rest of /api to Go, and everything else to Next.js. Both backends already serve under /api, so the order of the rules matters.
• The Go query service checks the session (an HTTP-only, signed JWT cookie) and the workspace, then runs the aggregations behind every chart.
• The Python data service takes spreadsheet uploads and runs the assistant, which uses models on Bedrock and semantic search over pgvector embeddings.
• Collectors run on a schedule on the same host. They fetch fresh posts and reviews through small Lambda functions in several regions, ask Bedrock to classify them, and write straight to RDS.
• RDS PostgreSQL 16 holds every workspace. It has no public endpoint; engineers reach it through an SSM port-forwarding tunnel.

System design

Two request paths matter most: a chart load, and a question to the assistant. In both, the server enforces workspace isolation; nothing from the browser is trusted for it.

sequenceDiagram
  participant U as Analyst
  participant C as Caddy
  participant Q as Query API
  participant E as Data service
  participant DB as RDS Postgres
  participant B as Bedrock
  U->>C: Open a tab with filters
  C->>Q: GET /api chart data with session cookie
  Q->>Q: Verify JWT and workspace access
  Q->>DB: Aggregate rows for this workspace only
  DB-->>Q: Results
  Q-->>U: Chart series
  U->>C: Ask the assistant a question
  C->>E: POST /api/chat
  E->>E: Verify session and workspace
  E->>DB: Semantic search with pgvector, scoped to workspace
  DB-->>E: Matching opinions
  E->>B: Prompt with retrieved context
  B-->>E: Answer
  E-->>U: Streamed answer with sources

Deploys

Every push to main builds all three services, and only a green build reaches the server.

flowchart LR
  push["Push to main"] --> build["GitHub Actions build: go vet and build, next build, Python import check"]
  build -->|"passed"| oidc["Assume AWS role via OIDC - no stored keys"]
  oidc --> ssm["SSM send-command to the tagged EC2 host"]
  ssm --> deploy["checkout sha, docker compose build"]
  deploy --> migrate["Apply schema migrations"]
  migrate --> up["docker compose up -d"]
  up --> health["Health-check API, data service, frontend"]

The deploy role can do exactly four things: describe instances, send one SSM document to the instance with the right tag, and read the result. There's no SSH key in GitHub and no port 22 open to the internet.

Decisions & trade-offs

• One EC2 host with Docker Compose, not ECS. For a handful of workspaces it's cheaper, simpler to debug and quick to redeploy. The cost is a single point of failure; the runbook already covers moving to ECS Fargate behind an ALB when uptime matters more than cost.
• One database, every row tagged by workspace. A new client is configuration, not new infrastructure. Every query must filter by workspace, so access checks live in one place in each service.
• Go for reads, Python for data and AI work. Chart queries need to be fast and predictable; ingestion and model calls are easier in Python. Both share the database, so there's no sync layer.
• OIDC and SSM instead of keys and SSH. Nothing long-lived to leak, and nothing listening for the internet to probe.
• Native editable exports. Screenshots would have been quicker to build, but analysts restyle every chart for their decks, so real charts are what they need.

What I'd do next

• Embed new opinions incrementally after each ingestion batch, instead of with a manual script.
• Add a per-workspace query cache, since most tabs are read far more often than the data changes.
• Write the infrastructure down as code; today the runbook is the source of truth.