references/analysis.md
A supporting file of the ia-evaluation skill.
IA study analysis
Card sort
Similarity matrix
For each pair of cards (a, b): similarity(a, b) = participants who placed a and b in the same group / total participants. Display as a matrix ordered
so high-similarity pairs sit together (most tools do this; by hand, sort by
the strongest pair first and grow the cluster).
Clustering
Agglomerative clustering on the similarity matrix produces the dendrogram. Read it at a chosen agreement level (60-70% is the usual first cut): each branch below the cut is a candidate group. Lower cuts merge groups; higher cuts split them. Report the cut used.
Standardisation
Participants name groups differently. Before counting labels, merge synonyms into standardised categories ("Billing", "Payments", "Money" -> "Billing and payments") and keep a map of original wordings with frequencies; the most frequent user wording is the label candidate.
Reporting
- Strong clusters: cards with pairwise similarity above the cut.
- Contested cards: placed in three or more standardised categories with no category above ~40%. Candidates for cross-listing or renaming.
- Label candidates per cluster with wording frequency.
- For closed sorts: per-card placement distribution and the share placed in the intended category.
Tree test
Per task, over participants:
- Success = reached a correct destination (last click) / participants.
- Directness = reached the destination without backing up / participants.
- Time = median seconds to the final click.
- First click = distribution of the first top-level choice; the correct first click strongly predicts success.
- Wrong destinations = where unsuccessful participants ended; group by branch.
Interpretation:
| Success | Directness | Reading |
|---|---|---|
| high | high | Structure and labels work |
| high | low | Found eventually; a sibling label competes or the path is long |
| low | high | Confident wrong path: a label misleads |
| low | low | Structure problem: no obvious home |
Conventional thresholds: success below ~60% or directness below ~50% needs action; between 60-80% success, look at the wrong-path branches before deciding.
First-click test
- Correct first click rate per task; below ~75% the cue on the page is weak.
- Click map: where wrong clicks landed, to tell a label problem from a layout problem.
Comparing two structures
Same tasks, participants split between trees. Compare per-task success and directness; a difference under the sampling noise of ~50 participants per arm (roughly 10-15 points) is not decisive on its own.