Skip to main content
Advanced•10–16 hours

Decision-Ready AI Analysis Dossier

Turn AI-assisted analysis into a decision artifact with verified metrics, an evidence ledger, uncertainty and sensitivity analysis, counter-analysis, reproducibility, and refresh/change controls.

decision analysisevidence managementuncertainty analysisreproducibilityexecutive communicationchange control

Scenario

Task

Leadership asks for a recommendation based on several datasets, SQL extracts, and AI-assisted EDA. The most dangerous outcome is not an obvious hallucination, but a persuasive story with individually correct numbers and weak causality, cherry-picking, or an unstable metric. Deliver a dossier where facts, interpretation, assumptions, and recommendation are separated and the decision can be revisited when data or the metric contract changes.

Step-by-step execution

1. Start with the decision contract, not the dashboard

Outcome: Analysis is optimized for a decision and falsifiable evidence rather than the number of charts.

Tasks

  • →Formulate the decision and alternatives
  • →Define material metrics and thresholds
  • →Record forbidden causal or unsupported inference
  • →Set evidence sufficiency and an abstention condition

Checks

  • ✓Decision owner and horizon are defined
  • ✓No KPI exists without a business definition
  • ✓Insufficient data may produce “we cannot recommend”

2. Build a claim-level evidence ledger

Outcome: Fact, calculation, estimate, assumption, and recommendation are not mixed in one paragraph.

Tasks

  • →Break the narrative into material claims
  • →Link each fact or derived claim to a query or source slice
  • →Mark estimates and assumptions separately
  • →Add freshness, confidence, and verification status

Checks

  • ✓A material claim without evidence fails the gate
  • ✓An AI-generated explanation does not become a source
  • ✓Stale evidence is visible before the recommendation is read

3. Run an independent challenge

Outcome: The recommendation survives alternative explanations and sensitivity analysis.

Tasks

  • →Build at least two counter-hypotheses
  • →Test alternative segments and time windows
  • →Vary key assumptions across a sensitivity range
  • →Check selection, survivorship, leakage, and Simpson-style aggregation risks

Checks

  • ✓The report states conditions under which the recommendation changes
  • ✓Correlation is not called causation without an appropriate design
  • ✓A high-impact conclusion has an independent calculation or reviewer challenge

4. Make the analysis operationally reproducible

Outcome: A month later the same pipeline can safely refresh the decision or explain drift.

Tasks

  • →Version source, query, notebook, and metric contract
  • →Add deterministic regression checks
  • →Define refresh triggers for data, schema, metric, model, or business-policy changes
  • →Record a rollback or hold rule for materially changed results

Checks

  • ✓Refresh does not overwrite previous evidence without a version trace
  • ✓Metric or schema changes trigger revalidation
  • ✓Material regression stops automatic publication until review

Acceptance criteria

  • The decision contract defines question, alternatives, thresholds, owner, and abstention condition
  • Every material claim is classified and has evidence or an explicit assumption label
  • Key metrics have deterministic verification and a reproducible source/query path
  • Counter-analysis and sensitivity show when the recommendation would change
  • Privacy, lineage, freshness, and change-control gates are included in the release decision
  • The final memo separates fact, interpretation, and recommendation and does not hide unresolved uncertainty