ia-evaluation

Information architecture evaluation

Chooses and runs the study that answers a structure question, and turns the numbers into a grouping and labelling decision. Structure definition itself (entities, relationships, content models) belongs to silver-structure; service-level mapping to service-blueprint; the words on the labels to better-writing once chosen.

Inputs

• The question: "how should these N items be grouped", "can users find X in this tree", or "which of two navigations works better".
• The content inventory: the items or the proposed tree, with any constraints (items that must stay together, regulatory sections).
• Access to participants, or the instruction to plan for later recruitment.

Workflow

1. Match the question to the method

Read references/methods.md for the decision table. In short:

• Open card sort when the grouping and the labels are unknown: users make and name groups. Generative.
• Closed card sort when categories exist and the question is whether items fit them. Evaluative.
• Hybrid sort when categories exist but may be incomplete.
• Tree test when a hierarchy exists and the question is findability: participants locate tasks in a text-only tree, so labels and structure are tested without visual design.
• First-click test when a page design exists and the question is whether the first choice on that page is right.

Do not sort more than about 60 cards; split by area instead. Do not tree test with fewer than eight tasks or more than fifteen per participant.

2. Design the study

• Cards / tree: one concept per card, user language, no leading words shared between cards that would sort themselves. For trees, include the full depth users would see and realistic sibling counts.
• Tasks (tree and first-click): describe the goal in the user's words without using label words from the tree ("find out how much you were charged last month", not "find billing history"). Define the correct destination(s) per task in advance.
• Participants: card sorts stabilise around 15-20 participants for pattern detection (Tullis and Wood; NN/g recommends 15+); tree tests need around 50 for stable per-task success rates. Recruit from the target population; note when a convenience sample was used.
• Instrument: record per participant the groups and names (sort) or the path, success, directness and time (tree/first-click).

3. Analyse

Read references/analysis.md for the calculations.

• Card sorts: build the similarity matrix (how often each pair of cards was grouped together), cluster it (dendrogram or agreement view), then standardise participant category names into a shared set before counting. Report the strong clusters (agreement above ~60-70%), the contested cards (spread across clusters), and the label candidates with the frequency of each wording.
• Tree tests: per task, success rate (correct destination reached), directness (reached without backtracking), median time, and the first-click path distribution. A task under ~60% success or under ~50% directness signals a structural or label problem; look at where wrong paths went to tell which.
• First-click: proportion of correct first clicks per task; below ~75% the page-level cue is weak.

4. Decide and document

Propose the structure: groups, labels, depth, and the placement of contested items (cross-list, rename, or move), each tied to the evidence that motivated it. Where results are ambiguous, say what a follow-up study (usually a tree test of the revised tree) would settle.

5. Output

• Study plan (method, cards or tree, tasks with correct answers, participant target, instrument).
• Results: matrices or per-task tables, with the thresholds used.
• Recommendation: sitemap or navigation outline, contested items with rationale, open questions and the follow-up study.

Before you finish

Limitations

Thresholds above are field conventions, not statistical guarantees; small samples give direction, not precision. Remote unmoderated tools automate the matrices; when analysing by hand, the references give the formulas.