LLM Internal States Monitoring

What this project does

AI Generated text is just an outcome of a model's internal states changing with every forward pass. We, as end-users, cannot see the internal confidence, hesitation, and routing that shaped them. Everything explaining how an answer formed lives one level down in the residual stream. By projecting an intermediate layer's hidden state through the model's final norm and unembedding, the logit lens reveals what the network computed at each step.

LLM-monitoring is an interpretability tool built to make this internal state legible in real time. Running local Hugging Face checkpoints in-process, it attaches PyTorch forward hooks across all decoder layers and streams layer-by-layer metrics to an interactive React UI via SSE. Rather than treating the LLM as a black box API endpoint, it acts as a real-time per-layer X-ray for every token as it is emitted.

What you are looking at

If interpretability is new to you but transformers are not, here is the architecture at a glance before digging into the internals.

You type a prompt. The FastAPI backend loads a causal LM in-process (defaulting to Qwen3-1.7B) and executes an autoregressive decode loop. During each forward pass, three collectors tap into the decoder stack via PyTorch forward hooks:

• Tier 1 (Logit Lens): Layer-wise predictions, rank metrics, and KL divergence.
• Tier 2 (Residual Geometry): Layer cosine similarities, vector norms, and massive activations.
• Tier 3 (Attention Dynamics): Prompt mass and attention sink ratios.

Clicking any token in the generated stream opens a full per-layer diagnostic for that isolated step: where the prediction crystallized (decision depth), how far intermediate distributions sit from the output (KL divergence), whether the network was actively attending to context (prompt mass), and intermediate token candidates. Every run is indexed token $\times$ layer in SQLite for historical review, side-by-side diffing, and cross-checkpoint comparisons.

To keep live streaming viable, all collectors aggregate tensors into scalars directly inside the hook, never forwarding raw activations to the client. The memory footprint of doing otherwise makes the engineering trade-off obvious:

$\text{Payload per token} = L \times d_{\text{model}} \times \text{sizeof}(\text{fp16})$

For an 8B class model ($L = 32$, $d_{\text{model}} = 4096$, 2 bytes per float):

$\text{Per-token footprint} = 32 \times 4096 \times 2\text{ B} \approx 256\text{ KB}$

$\text{1,000-token stream} = 1000 \times 256\text{ KB} \approx 256\text{ MB of raw states}$

Streaming hundreds of megabytes of raw states over SSE would overwhelm browser memory and serialization throughput. Instead, the backend pushes compact scalar events token-by-token while generation runs.

The application runs as a self-contained monolith: Python 3.12 + FastAPI on the backend and React 19 + Vite + Tailwind on the frontend, served from a single process on port 8100.

Main motivation

I didn’t set out to build an observability suite. I was driven by pure curiosity: what does a transformer actually "think" before it commits to an answer?

Static papers on the logit lens proved mid-stack states hold latent predictions, but offline notebooks felt too detached. I wanted an X-ray of the network thinking in real time. Day one was a basic CLI script hooking into Qwen to print layer predictions to stdout. It worked in an afternoon, but answering one isolated question immediately spawned ten more: How does computation evolve across a full answer? When does a token resolve by shallow recall versus deep reasoning? Is context actually being read?

Chasing those questions forced the tool to evolve:

• Taming silent failures: Handling architecture quirks–like Gemma's logit softcapping or tuned lens drift on non-English text–which silently distort telemetry without throwing errors.
• Designing for live intuition: Shipping lightweight scalars over SSE to turn opaque tensors into real-time decision-depth curves and attention heatmaps.

What began as an afternoon experiment grew into a self-validating instrument built to observe what the model is doing beneath fluent prose.

The Analysis screen – turning a single token into datamining tool

The UI has four tabs – Analysis, Comparison, History, Definitions – but Analysis is where you spend most of your time. Generate, click a token, read the right column.

Worth knowing on first open:

• Prompt vs what the model sees. The toggle applies the chat template live. A one-word prompt at Gemma arrives as thirteen tokens; every attention index is against those positions, not your raw string.
• Run settings. Token budget, temperature, raw vs tuned lens, and which collector tiers are on (t1+t2+t3). Lens overhead is reported separately from throughput – 0.16 ms/token on Qwen3-1.7B, measured on an M5 Pro – so the instrument's cost stays checkable.
• Six presets. Each is chosen to make one phenomenon visible, not to show off the model. More on those below.
• Token verdict. "Decided at layer 23 of 28 · 16 changes of mind · half-converged by layer 20" plus a layer trail – what each layer would have emitted if you had stopped there. On the strawberry count, early layers guess punctuation and fragments; the digit only wins late.
• Five per-layer charts. Decision, Convergence (KL divergence to final), Uncertainty (entropy), Attention (prompt mass and attention sink), Residual geometry (cosine similarity to the layer below – near 1 means the layer barely moved the stream).

Below the charts sits the run heatmap: tokens × layers, switchable between rank of the chosen token, KL to final, attention on the prompt, sink mass, and layer similarity. Long runs wrap into side-by-side blocks. This is where run-level shape appears – a band of high rank several tokens before the prose goes wrong is the model losing the thread, visible before you notice in the text.

A forward pass will tell you where a token was decided

Another very importand and exposed number is decision depth: how far up the stack the answer settled. Ask for the capital of Australia and "Canberra" is already the top-1 guess a third of the way up – the fact was retrieved, not derived. Ask for the area of a hexagon with a side of 7 and the digits do not settle until the last two layers.

Attach a forward hook per decoder layer, keep the last position, project:

def hook(_module, _args, output):
    hidden = output[0] if isinstance(output, tuple) else output
    captured.append(hidden[:, -1, :])          # last position only
 
handles = [layer.register_forward_hook(hook) for layer in model.model.layers]
 
# after the step, for every captured layer:
logits = model.lm_head(model.model.norm(h))    # the model's own norm, not a copy
probs = logits.float().softmax(-1)

Skip the final norm and you still get logits, still get a plausible-looking bar chart, and every number in it is wrong: mid-stack activation scales are nothing like what lm_head expects. Nothing crashes. That is why the app reports which layer a measurement came from rather than a single summary score.

Reading the raw depth curve, though, the middle of the stack is suspiciously bad at everything.

Raw logit lens vs tuned logit lens

On Llama- and Qwen-family models the raw logit lens reads intermediate layers poorly – the residual basis drifts, so a straight projection under-reads what the layer knows. The fix is a tuned lens: a small affine translator per layer, trained to map that residual into the output basis first. Roughly three minutes on an M5 Pro:

uv run python -m scripts.train_tuned_lens --steps 600 --seq-len 256

Mean KL divergence reduction against the raw lens: 71.6% on Qwen3-1.7B, 75.6% on Gemma 4 E2B-it. That is a large correction, and the kind of number that invites you to stop asking questions.

Then I ran a Polish prompt through the same lens. The reduction fell from about 74% to about 35%. The translator had been trained on English wikitext. Nothing errored; the curves just got less honest. So the trainer takes a corpus:

uv run python -m scripts.train_tuned_lens --corpus my_corpus.txt --steps 600

A lens is an instrument with a calibration, and the calibration belongs to a language and a format, not to a model. Which raises the obvious follow-up: if the tool can be quietly wrong about what a layer thinks, what does it get right often enough to act on?

Six prompts that fullfill different use cases

The preset catalogue is not a demo reel. Each entry is an instrument calibration – a question you run once to learn what a healthy curve looks like, then read everything else against it.

Default temperature is 0 and top-p is1.0. This is a measuring instrument; silently truncating the distribution would make every sampled run a measurement of a distribution the model did not produce. A warning appears when temperature ≥ 1 meets top-p = 1.

Comparison module – two runs, one question

Once you have a baseline run, Comparison is where model families and prompt variants become legible. Pick two runs – the same prompt on Qwen vs Gemma, or hexagon area run #67 vs #68 – and the view averages per-layer curves over the answer plus a metric table.

The registry is a comparison set, not a menu – three ModelSpec rows, each isolating one variable:

Reference throughput on Apple MPS (M5 Pro): Qwen 8.3 tok/s, Gemma 6.8 tok/s. One model resident at a time – these are 3–14 GB objects, and switching evicts the previous checkpoint.

Boundaries of the application

The hard boundary is architectural. This needs the forward pass in-process, with eager attention so the attention matrix is actually materialized.

That rules out Ollama, llama.cpp and every HTTP backend: they return text. It also rules out GGUF – transformers dequantizes it to fp32 on load, so a 12B checkpoint becomes about 48 GB and the quantization buys nothing. Weight-only schemes that keep hooks working, like GPTQ or AWQ, are fine, and are where the massive activations view earns its place – single residual dimensions running 100–1000× larger than the rest are what breaks quantization, and they are visible directly in tier 2.

The softer boundary is correlational. A layer where the answer settles is not proof that the layer computed it, and this app does not do interventions. Causal claims need ablation and patching – a different tool, a different cost.

Important design choices

In-process hooks over API wrappers. The moment the model runs behind someone else's HTTP endpoint, all of this is gone. The premise of the project is that the instrument and the forward pass share one address space.

Aggregation in the hook, not in the browser. Raw states do not cross the wire. Scalars do. That is a performance choice and an honesty choice – if you cannot afford to store it, you cannot pretend you measured it.

Validation before registry. check_model exists because silent wrong numbers are worse than crashes. The decode-loop diff line is the one that matters.

Presets as calibration, not demos. Six prompts that teach you what healthy curves look like beat a thousand ad-hoc generations you cannot compare.

Depth ratio for cross-model comparison. Raw layer index compares unlike stacks. Ratio does not.

What I would not claim: that a shallow decision depth proves retrieval, that high prompt mass proves correct RAG, or that the tuned lens is ground truth. They are correlates you act on when the alternative is reading fluent text and guessing.

Where it landed

One FastAPI process serves the compiled frontend. 219 backend tests cover collectors, the decode loop, and the model registry; 180 frontend tests cover the React shell. Deep capture writes raw tensor dumps to .npz under a disk budget for offline analysis the UI does not surface yet. Per-head attention matrices are collected but not rendered. Stored runs predating the candidate list field reopen without "what the model nearly said."

What is left is the honest version of a question worth asking often: not "is the model right", which you can check, but "was it ever in doubt" – and, for RAG, "did it ever read what I gave it" – which until you open the forward pass you cannot.

Clone, build, run:

git clone https://github.com/JanBancerewicz/LLM-monitoring.git
cd LLM-monitoring && uv sync
npm install --prefix frontend && npm run build --prefix frontend
uv run uvicorn backend.api:app --port 8100

Open http://localhost:8100. Try it yourself with one of the predefined prompts , then run the same model with different prompts and compare their depth curves. This sequence is what the project was built for.