From documenting decisions to questioning them

The natural instinct would be to throw the problem at an LLM. "Analyze this proposal." But what comes back is not analysis. It’s a more polished version of whatever the input already concluded. The model follows the framing it was given, fills gaps with fluent prose, and delivers something that reads like confirmation. Research on LLM sycophancy explains why: training on human feedback rewards agreement over accuracy, so models default to confirming the user’s position rather than testing it. The effect is that you’ve added a second source of bias on top of the first.

I wanted something different: not an AI that decides for you, but one that challenges the input before confirming it, a form of structured skepticism.

A question emerged: can you build some kind of structured method, encode it into an AI agent, and have it apply a form of critical thinking to every input it receives?

I didn’t know where to start, but refining my search on decision analysis methods, I stumbled upon unexpected and interesting resources: structured analytical methods built for contexts where getting the analysis wrong has serious consequences: defense, national security, strategic planning.

The most useful reference was Pherson and Heuer’s Structured Analytic Techniques for Intelligence Analysis (3rd ed., 2020). Sixty-six techniques. One stood out: the Key Assumptions Check. The CIA’s Tradecraft Primer (2009) lays out the method in four steps: it forces you to list every assumption in your analysis and check whether each one has evidence behind it.

From this technique and four related research lines, I derived a four-category classification I called FCAU: each claim in the input gets classified as a Fact (supported by evidence), a Constraint (a verified non-negotiable boundary, a deadline claimed as regulatory stays an Assumption until the regulation is cited), an Assumption (presented as fact but lacking evidence), or an Unknown (an acknowledged gap).

Where do the categories come from? The conceptual frame is the 2×2 matrix Rumsfeld used in his 2002 DoD briefing (transcript) — known knowns, known unknowns, unknown unknowns, plus the fourth quadrant he omitted: unknown knowns. Žižek (2004) meant unconscious presuppositions — beliefs we don’t know we hold. The working definition the method uses is narrower: claims accepted as fact but never verified. That’s where most preventable failures start. The operational process comes from the U.S. Army TRADOC’s CLA (revised 2012). The "imagine failure before it happens" technique comes from Klein’s Pre-mortem (2007). And the diagnosis — why we keep falling into this trap — comes from Kahneman (2011): our mind runs two modes, System 1 (fast, intuitive) and System 2 (slow, deliberate). System 1 checks whether a story holds together, not whether anything is missing.

Here’s how FCAU maps to Rumsfeld’s quadrants:

Why the split? Facts describe the situation, constraints bound it. Thanks to the pre-mortem, unknown unknowns might emerge, but there’s no guarantee, so the method excludes them.

The one design choice that matters most: any claim without evidence gets classified as Assumption by default, it has to earn the label of Fact. The main analytical element of the method comes from this single inversion.

In practice, this changes the conversation. The proposer isn’t told they’re wrong — just which claims still need evidence. Most teams can have that conversation without anyone getting defensive.

Once an analytical method is developed, the problem becomes how to port it to an LLM and make it consistently execute the analysis instead of reverting to its natural tendency to agree fluently with whatever it was given. The solution came from agentic design patterns, as documented by Anthropic (2024), Andrew Ng (2024), and the broader research community.

The result comes from composing different individual patterns, of which six turned out to be essential:

Instructions are written to leave as little room for interpretation as possible. They say what to do, not what to avoid. And the output format makes the model fill an FCAU grid and think inside that grid.

Looking ahead, having a prompt opens possibilities I haven’t fully explored yet: running the same prompt on different LLMs and comparing the classifications, or feeding the structured output into a second agent for deeper analysis. The method doesn’t depend on the engine; in principle, it could be transferred.

Take a real-world scenario. Someone proposes migrating an on-premise ERP to the cloud. A vendor quoted a price and a twelve-month timeline. IT supports the move. Finance approved the budget. The choice seems made.

The FCAU classification decomposes this differently.

The validation step forces questions that the proposal didn’t invite.

Where would a skeptic push back hardest? What is the real cost of doing nothing? Is the current ERP actually unsustainable? What overlooked alternative might reframe the choice? A phased approach, perhaps, or a different vendor, or a hybrid architecture.

The claims that matter most for this decision are unverified assumptions, not established facts. The value of the analysis is not the confidence label, it is the itemized list: get a second vendor quote, verify team capacity, estimate migration complexity. Those are the specific conversations the team needs before this becomes a decision. Had the proposer brought benchmarks, a capacity assessment, and a pilot report, those claims would be Facts, and the output would be a MADR, not an exploration report.

When a decision is mature enough, well-defined question, genuinely comparable options, known drivers, manageable uncertainty, the agent produces a structured decision record in the MADR format I started from. When it isn’t, the output is an exploration report that maps the decision space, surfaces gaps, and gives concrete next steps.

The ERP migration, our example, would take the second path: not ready for a decision record, because the foundation isn’t there yet.

The agent isn’t intelligent. It follows a method that forces the same decomposition and the same questions on every input, without cutting corners or getting political.

None of the elements I used is new on its own. The surprise was that wiring them into one sequence produced behaviors that no single ingredient exhibits alone.

Three complementary mechanisms, an automatic depth selector that detects bias in the input, a critical signals checklist, and quality guardrails, reduce the risk of accepting the input’s framing uncritically. An assumption misclassified as a fact in the first step can get caught by the validation questions in the second. Confidence is structurally tied to the classification: if key drivers are assumptions, it’s capped by the taxonomy itself.

It was necessary to build a test suite of 69 scenarios in eight domains: software architecture, business, organization, compliance, procurement, innovation, crisis, and multi-stakeholder decisions. The scenarios served a dual role: regression tests and tuning targets. Each time the output failed in a recognizable way, I tracked the cause to a specific instruction and revised it.

Three examples.

I added web search: in my case, it created context dilution that degraded classification quality, so I removed it. The trade-off is explicit: without search the agent cannot verify claims on its own, so promoting an Assumption to Fact requires the team to bring the evidence.

The agent was quietly smoothing over contradictions in the input. I only caught it when a test with deliberately conflicting claims produced a confident, coherent, wrong analysis. I had to add an explicit contradiction-detection step.

Persuasive inputs (e.g. sales pitches dressed up as technical assessments) needed their own guardrail. The model has to name the framing before it starts the analysis, otherwise it inherits it and the whole output is compromised.

The tool matters less than the method. Strip away the structure and what’s left is enhanced autocomplete. A rigorous method should transfer across models and platforms, and it already transfers to how I read a proposal myself. The value is in the decomposition, not in the inference engine.

What seems to be missing from the agentic AI literature, in day-to-day practice, is guidance on composition: which patterns amplify each other, which create friction, what happens when you tighten the resource budget. That gap is where I think the real work lies.

The work isn’t finished. But the method has already changed how I read proposals and evaluate options, even without running the agent.