Part 3 · Evaluating LLMs · module 2 of 2 · 12 min

Qualitative evaluation

Looking at the model instead of scoring it — black-box evaluation, red-teaming, activation distributions and token heatmaps.

The essence

Qualitative evaluation is structured looking. Rather than reducing behaviour to a number, you probe the model with deliberately chosen inputs and inspect what comes out — including the internals. It is not the soft alternative to metrics; it is how you find the failures no metric was designed to catch, and every quantitative benchmark started life as somebody noticing something by eye.

Why it matters

Metrics only measure failures you anticipated. The ones that matter — a jailbreak, a systematic refusal pattern, a subtle factual drift after fine-tuning — are almost always found by looking first and quantified afterwards. Visualisation also connects behaviour to mechanism, which is the bridge into the interpretability parts of this curriculum.

Topics covered (6)

01Black-box evaluation
02Red-teaming
03Accuracy, coherence and relevance as judgement axes
04Distributions of hidden-state activations
05Heatmaps of tokens for qualitative inspection
06Visualising single-token predictions

Black-box evaluation

Black-box means you have inputs and outputs and nothing else — the setting for any model behind an API. You cannot inspect weights or activations, so the method is entirely about designing inputs whose outputs are diagnostic.

The useful discipline is contrast. A single output tells you almost nothing; a pair differing in exactly one respect tells you what that respect does. Change only the name in a request, only the phrasing of a constraint, only the language, and the difference in output is your measurement.

Probe families worth running on any model

Minimal pairs — change one word and see whether the output changes in the way it should.
Format stress — the same request as prose, as JSON, as a table, and with a typo.
Length and position — bury the key instruction at the start, middle and end of a long context.
Consistency — ask the same question five ways and check the answers agree.
Refusal boundary — a benign request, a clearly harmful one, and several deliberately ambiguous cases in between.
Prompt-injection resistance — put a competing instruction inside the user-supplied data.
Known-unknowns — ask about something that does not exist and see whether it declines or invents.

Red-teaming

Red-teaming is adversarial evaluation: the objective is to make the model fail, and a session in which nothing broke is usually a poorly designed session rather than a safe model.

The recurring techniques are worth knowing because they are what safety training is measured against — role-play framings that recast a refused request as fiction, incremental escalation across turns, instructions hidden in retrieved documents or code comments, encoding the request so it does not pattern-match a refusal, and switching to a language with weaker safety coverage.

The output of red-teaming is a corpus of failures. Each becomes a regression test, and the ones that recur become a category worth quantifying — which is precisely how today's safety benchmarks were built.

Judging accuracy, coherence and relevance

These three axes vary independently, and separating them is what makes qualitative notes useful. Accuracy is whether the claims are true. Coherence is whether the text is internally consistent and well-formed. Relevance is whether it answers what was asked.

A fluent, well-structured, entirely wrong answer scores high on coherence and zero on accuracy — and that combination is the dangerous one, because coherence is what readers use as a proxy for accuracy. Rating each axis separately on a small rubric, with two independent raters, produces something you can actually track across model versions.

Looking inside: activation distributions

For an open model you can go further than inputs and outputs. Plotting the distribution of hidden-state values per layer is a fast, high-yield diagnostic. Healthy layers show roughly bell-shaped activations with a stable scale across depth.

What you are looking for is anomalies: units that are always zero (dead), units saturated at their extremes, a variance that grows steeply with depth (a normalisation or initialisation problem), and the outlier dimensions that appear in trained transformers — a handful of features with magnitudes orders of magnitude above the rest, which are also what makes naive quantisation degrade quality.

Activation distributions across layers

import torch, numpy as np, matplotlib.pyplot as plt
from transformers import AutoModel, AutoTokenizer

tok = AutoTokenizer.from_pretrained("gpt2")
model = AutoModel.from_pretrained("gpt2", output_hidden_states=True).eval()

ids = tok("The quick brown fox jumps over the lazy dog.", return_tensors="pt")
with torch.no_grad():
    hs = model(**ids).hidden_states          # tuple: embeddings + one per layer

stats = [(h.mean().item(), h.std().item(), h.abs().max().item()) for h in hs]
for i, (m, s, mx) in enumerate(stats):
    print(f"layer {i:2}  mean {m:+.3f}  std {s:6.3f}  max|x| {mx:8.1f}")

# Norm growth through depth is expected — each block ADDS to the residual
# stream. A sudden jump, or a max|x| far above the std, marks outlier features.

fig, ax = plt.subplots(1, 2, figsize=(11, 4))
ax[0].plot([s for _, s, _ in stats], "o-")
ax[0].set(xlabel="layer", ylabel="std", title="Activation scale by depth")
ax[1].hist(hs[6].flatten().numpy(), bins=100, log=True)
ax[1].set(xlabel="activation", title="Layer 6 distribution (log count)")
plt.tight_layout()

Token heatmaps

The most informative single visualisation in this whole area is text coloured by a per-token quantity. Colour each token by the log-probability the model assigned it and you can read, at a glance, exactly where the model was surprised.

This localises problems in a way an aggregate cannot. A perplexity of 30 tells you the model struggled; a heatmap tells you it struggled on the proper nouns and sailed through the syntax. The same rendering works for attention weight from a chosen head, for a single neuron's activation, or for the change in prediction after an intervention — which is how the causal experiments later in this curriculum are usually presented.

Colouring text by per-token surprise

import torch, torch.nn.functional as F, matplotlib.pyplot as plt
from matplotlib.colors import Normalize

@torch.no_grad()
def token_surprise(model, tok, text):
    ids = tok(text, return_tensors="pt")["input_ids"]
    logits = model(ids).logits[0, :-1]
    targets = ids[0, 1:]
    logp = F.log_softmax(logits, -1).gather(-1, targets[:, None]).squeeze(-1)
    pieces = [tok.decode([t]) for t in targets]
    return pieces, (-logp).tolist()          # higher = more surprised

def heatmap(pieces, values, cmap="Reds"):
    norm = Normalize(vmin=0, vmax=max(values))
    colours = plt.get_cmap(cmap)(norm(values))
    fig, ax = plt.subplots(figsize=(12, 2.5))
    x, y = 0.01, 0.85
    for piece, colour in zip(pieces, colours):
        t = ax.text(x, y, piece, fontsize=11, backgroundcolor=colour,
                    transform=ax.transAxes, family="monospace")
        # Advance by a character-width estimate; wrap at the right margin.
        x += 0.011 * (len(piece) + 1)
        if x > 0.94:
            x, y = 0.01, y - 0.2
    ax.axis("off")
    return fig

pieces, surprise = token_surprise(model, tok,
    "The capital of France is Paris, and the capital of Burkina Faso is Ouagadougou.")
heatmap(pieces, surprise)
# 'Paris' is nearly free; 'Ouagadougou' lights up — and its subword pieces
# after the first are cheap, because once started the word is determined.

Inspecting a single prediction properly

import torch

@torch.no_grad()
def inspect(model, tok, prompt, k=10):
    ids = tok(prompt, return_tensors="pt")["input_ids"]
    logits = model(ids).logits[0, -1]
    probs = torch.softmax(logits, -1)
    top = probs.topk(k)

    entropy = -(probs * probs.clamp_min(1e-12).log()).sum().item()
    print(f"entropy {entropy:.2f} nats  (low = confident, ~10.8 = uniform)")
    for p, i in zip(top.values, top.indices):
        print(f"  {p.item():6.3f}  {tok.decode([i])!r}")

inspect(model, tok, "The capital of France is")
inspect(model, tok, "My favourite colour is")

# Entropy alongside the top-k is the pair worth printing: it distinguishes
# 'confidently right', 'confidently wrong' and 'genuinely uncertain', which
# a single argmax cannot.

Worth remembering

Black-box evaluation probes inputs and outputs only; it is what you can do to any model, including an API.
Red-teaming is adversarial by design — the goal is to find a failure, not to confirm success.
Plotting activation distributions catches dead units, saturation and outlier dimensions that metrics hide.