Lexi

Problem

Precedent research is about finding the judgments that support an argument, and also the ones that cut against it. Finding only favourable cases is dangerous. An LLM alone is a poor tool for that: it will produce plausible citations that don't exist, and it has no reliable way to know whether it found the precedent that matters.

The prompts also vary widely. "Which cases involve commercial vehicles?" needs a clean lookup, while a client's brief needs supporting precedents, adverse ones, legal risk and a strategy. A single ReAct loop sends both down the same heavy path.

Approach

The design principle is to use deterministic structure where possible, hybrid retrieval for legal recall, and LLMs for interpretation and synthesis, and to expose the whole trace so the system can be evaluated and debugged.

Enrichment happens once, at build time. Claude sub-agents produced the case cards, metadata and chunk labels, which are frozen as artifacts, so the runtime stays fast and consistent. Gemini handles runtime reasoning.

Chunking has three levels: small child chunks for precise retrieval, parent chunks to preserve legal reasoning, and a case card per judgment for document-level questions. That came to 56 documents, 717 parent chunks, 3,647 child chunks and 56 case cards.

Architecture

interpret -> route
  simple:  simple_retrieve -> simple_answer
  clarify: clarify
  deep:    plan -> expand -> support -> adverse -> compensation
           -> select -> analyze -> synthesize -> validate

Each child chunk gets a dense and a BM25 rank, which are fused and then reranked:

RRF         = sum(1 / (60 + rank + 1))
fused_score = rrf_score + metadata_boost

The top candidates go to bge-reranker-v2-m3. Evidence selection then buckets chunks into supporting, adverse and neutral, with limits per document, and attaches the parent context.

Evaluation

The golden set has 25 queries across factual metadata, single-issue, procedural, statutory, no-answer, comparison, ambiguous, multi-hop and cross-document strategy questions.

• 9 are deterministic: exact labels computed from validated metadata.
• 16 are designed: authored and labelled from each document's issue and stance tags.
• Pooling was reviewed but not auto-added. Mean pool recall is 0.94, but adding whatever retrieval found would have made the answer key circular.

Retrieval and answers are scored separately, because a system can retrieve the right documents and synthesize poorly, or write a plausible answer while missing key cases.

Results (v2)

Dimension	Metric	Result
Precision	Precision@5	0.591
Precision	Precision@10	0.481
Precision	Answer precision	0.405
Recall	Recall@10	0.662
Recall	Recall@20	0.770
Recall	MRR	0.807
Recall	nDCG@10	0.714
Recall	Support recall	0.446
Reasoning	Faithfulness	3.88 / 5
Reasoning	Legal reasoning	3.40 / 5
Reasoning	Completeness	2.88 / 5
Reasoning	Issue coverage	0.957
Safety	No invented documents	1.0
Safety	Must-not-claim respected	1.0
Adverse	Adverse recall	0.562
Adverse	Adverse reasoning	3.72 / 5
Adverse	Adverse search executed (deep queries)	0.889
Adverse	Adverse section present	0.889
Agent	Routing accuracy	0.92
Agent	No-answer handled	1.0

What the first evaluation changed

The v1 baseline showed adverse cases being crowded out, so I made two fixes: metadata enumeration for listing queries, and stance-aware adverse retrieval at document level.

Metric	v1	v2	Change
Adverse recall	0.342	0.562	+0.22
Adverse reasoning	3.48	3.72	+0.24
Completeness	2.68	2.88	+0.20
Precision@5	0.53	0.591	+0.06
Recall@20	0.73	0.77	+0.04
MRR	0.765	0.807	+0.04
nDCG@10	0.67	0.714	+0.04
Must-not-claim respected	0.96	1.0	+0.04
Answer precision	0.474	0.405	−0.07

The one regression, answer precision, is the expected cost of surfacing more adverse precedents. For legal research that's the right trade, and no safety property regressed.

Where it still falls short

• Completeness is low at 2.88/5, so multi-part answers sometimes miss a requested facet.
• Support recall is low at 0.446, and adverse recall is still incomplete at 0.562.
• Appellant-role queries fail, because the system doesn't yet track who appealed.
• The router missed 2 of 25: one that should have asked for clarification and one that over-escalated.

Caveats

• Judge scores are single-sample. An earlier iteration measured reasoning-judge spread up to ±0.34, so the small judge deltas above are within noise. The deterministic gains are real.
• Self-judge bias. The judge is Gemini scoring Gemini, offset by objective checks such as invented documents, must-not-claim and issue coverage. A different judge model is the planned upgrade.
• One known borderline label. A contested stance label on one judgment could shift adverse recall if a domain expert flipped it.

Tradeoffs

Decision	Benefit	Cost
LangGraph instead of a single ReAct chain	Control, routing, visible trace, easier debugging	More moving parts
Hybrid retrieval instead of vector search only	Better legal recall for exact terms and paraphrases	More retrieval complexity
In-memory BM25	Simple and fast for 56 documents	Not suitable for 5,000+
Build-time enrichment	Faster runtime, stable metadata	Corpus changes require a rebuild
Parent-child chunking	Precision plus legal context	More artifacts to manage
Document-level adverse stance	Better adverse recall	A small drop in answer precision
Gemini Flash at runtime	Lower latency and cost	Weaker on hard synthesis than a larger model

At 5,000 documents the logic stays and the scale layer changes: a sparse index or OpenSearch instead of in-memory BM25, stronger metadata pre-filters, a proper ingestion DAG with versioned enrichment, a lawyer-reviewed taxonomy and golden set, and caching for embeddings, expansions and reranks.

Outcome

The agent handles both lookups and deep research over the corpus, and scored 0.92 routing accuracy, 0.81 MRR and 0.77 Recall@20 with no invented documents on the 25-query golden set. Adverse recall improved from 0.34 to 0.56 once the evaluation showed where the problem was.

With another week, I'd add a facet checklist before synthesis, appellant-role metadata, better query expansion for support recall, multi-sample judging with a second model, and a deterministic compensation calculator.

An earlier single-agent build over-reasoned and didn't route or filter by prompt type, which is what led to this multi-node design.

What I learned

A demo shows that a system can be right. An evaluation set shows how often it is, and which stage to blame when it isn't. Adverse recall was the clearest case: the prompt already asked for adverse precedents, but only measuring it showed they were being crowded out, and the fix was in retrieval rather than the prompt.