Open Trace

Overview

Open Trace is a small analytics platform in three parts: a plain JavaScript tracker you drop into any site, an Express and Socket.io backend that stores events in MongoDB, and a React dashboard for metrics, session replay, heatmaps, funnels and a live feed. The repo also ships a demo SaaS site that generates real traffic to look at.

The problem

Tools like Hotjar and FullStory answer "what did users actually do on this page", but they are hosted products with their own pricing and data handling. I wanted to understand how that pipeline works end to end, from capturing clicks in the browser to replaying a session, and build a version I fully control.

What I built

• tracker.js: assigns a session id in localStorage with an inactivity timeout, captures page views, clicks, scroll depth, throttled mouse paths, rage clicks and dead clicks (clicks on non-interactive elements), batches them and flushes every 200 ms, switching to navigator.sendBeacon when the page is hidden.
• An ingest endpoint that validates each event with Zod, drops bad ones without failing the batch, normalizes page URLs, bulk-inserts with insertMany, and upserts one session per session_id with pages visited, max scroll, rage and dead click flags, plus browser, OS, device and country from UAParser and geoip-lite.
• Live streaming: every inserted event is emitted over Socket.io so the dashboard feed updates as you click on the test site.
• Session replay built from stored mouse paths and click coordinates, and heatmaps that overlay clicks per page with rage and dead clicks marked.
• Funnels defined as ordered URL steps, analyzed over a time window against session page lists.
• AI session summaries: the backend formats a session's events into a prompt for an OpenAI-compatible endpoint (NVIDIA NIM, Llama 3.1 8B by default) and caches the result on the session.

Architecture

The tracker writes to one Express service, which serves both the REST API and the WebSocket feed for the dashboard.

flowchart LR
  subgraph client["Client"]
    site["Any website with tracker.js"]
    dash["React dashboard"]
  end
  subgraph server["Backend"]
    api["Express REST API"]
    ingest["Event ingest controller"]
    ws["Socket.io server"]
    agg["Sessions, heatmap, funnel, metrics controllers"]
  end
  subgraph data["Data"]
    db[("MongoDB events and sessions")]
  end
  subgraph ext["External"]
    llm["NVIDIA NIM LLM"]
  end
  site -->|"POST batched events"| api
  api --> ingest
  ingest --> db
  ingest --> ws
  ws -->|"live events"| dash
  dash -->|"REST queries"| api
  api --> agg
  agg --> db
  agg -->|"session summary"| llm

System design

The path of a batch of clicks from a visitor's browser to the live dashboard.

sequenceDiagram
  participant V as Visitor browser
  participant T as tracker.js
  participant A as Express API
  participant D as MongoDB
  participant S as Socket.io
  participant W as Dashboard
  V->>T: Click, scroll, mouse move
  T->>T: Detect rage or dead click and queue event
  loop Every 200 ms
    T->>A: POST api events with batch
  end
  A->>A: Validate each event with Zod and drop invalid
  A->>D: insertMany events
  A->>D: Upsert session with pages, flags, device, geo
  A->>S: Emit each inserted event
  S-->>W: Event appears in live feed
  A-->>T: 202 accepted and dropped counts

Decisions & trade-offs

• Per-event validation instead of rejecting the whole batch. One malformed event from an old tracker version should not lose the rest, so the API returns accepted and dropped counts.
• Writes go straight to MongoDB. That keeps the stack to one service and one database, but a traffic spike lands directly on the database. A queue like Redis or Kafka in front of ingest is the obvious next step.
• Sessions are client-generated ids in localStorage. Simple and cookie-free, but clearing storage starts a new session.
• The funnel check counts a session at a step if it visited every step up to that point, matched by prefix. It does not enforce strict order between visits, which is good enough for simple flows but can overcount.

What I'd do next

• Put a queue between ingest and the database and move session upserts to a worker.
• Enforce step order in funnel analysis using event timestamps rather than the set of pages visited.
• Add per-site API keys so one backend can serve multiple tracked sites safely.