1. Prepare the corpus
Outcome: Documents have owner, version, and access metadata.
Tasks
- →Select 20–50 documents
- →Clean the structure
- →Add metadata
- →Store the source URI
Checks
- ✓Every chunk has a source
- ✓Versioning is reproducible
Core•12–20 hours
Evidence-first Knowledge Assistant
Build a RAG assistant that answers only from verified evidence, shows citations, and measures retrieval quality.
ingestionchunkinghybrid retrievalrerankingcitationsRAG evaluation
Scenario
Task
A team needs an assistant for internal policies. The system must not invent answers, must expose its sources, and must abstain when the evidence is insufficient.
Step-by-step execution
2. Configure retrieval
Outcome: Relevant passages consistently reach the top-k.
Tasks
- →Compare chunk sizes
- →Add lexical + vector search
- →Configure reranking
- →Add an ACL filter
Checks
- ✓A test query set exists
- ✓ACL is covered by negative tests
3. Build the evidence contract
Outcome: The model cannot hide missing evidence.
Tasks
- →Define the citation schema
- →Add abstention
- →Validate source IDs
- →Reject unsupported claims
Checks
- ✓Every claim has a citation or abstention
- ✓An unknown source ID blocks the answer
4. Run evaluation
Outcome: Retrieval and generation quality are measured separately.
Tasks
- →Create a golden set
- →Calculate recall@k
- →Evaluate citation correctness
- →Build a failure taxonomy
Checks
- ✓A baseline exists
- ✓A regression threshold is recorded
Acceptance criteria
- Every citation opens a real source
- ACL prevents data leakage
- Weak evidence triggers abstention
- The evaluation report contains a baseline and failure examples
Assessment rubric
How the result is assessed
Passing score: 70/100 · Distinction: 90/100
Corpus and provenance
Documents, chunks, metadata, versions, and source URIs are reproducible.
Insufficient
Sources lack version or owner metadata, or chunks lose provenance.
Competent
Every chunk has source, metadata, and version.
Strong
Freshness policy, lineage, and change control are implemented.
Evidence required
- ✓ Link to code or artifact
- ✓ Short README explaining decisions
- ✓ Test output or runtime evidence
- ✓ Example document and its chunks
Retrieval quality
Hybrid retrieval, filters, and reranking are measured on a query set.
Insufficient
There is no test set or retrieval is judged subjectively.
Competent
A baseline, query set, and measurable top-k exist.
Strong
Multiple strategies are compared with a regression threshold and failure taxonomy.
Evidence required
- ✓ Link to code or artifact
- ✓ Short README explaining decisions
- ✓ Test output or runtime evidence
- ✓ Recall@k or an equivalent retrieval metric
Grounding and citations
Claims are supported by evidence, and missing evidence causes abstention.
Insufficient
The model can answer without evidence or citations do not open the source.
Competent
Unsupported claims are blocked and citations are valid.
Strong
Groundedness and citation correctness are measured automatically.
Evidence required
- ✓ Link to code or artifact
- ✓ Short README explaining decisions
- ✓ Test output or runtime evidence
- ✓ Examples of correct citation and abstention
Access control and security
ACL is applied during retrieval and verified with negative tests.
Insufficient
Filtering happens after retrieval or leakage remains possible.
Competent
ACL is applied to search and covered by negative tests.
Strong
Tenant isolation, audit logging, and policy tests are implemented.
Evidence required
- ✓ Link to code or artifact
- ✓ Short README explaining decisions
- ✓ Test output or runtime evidence
- ✓ Negative ACL test