Fail-Open: The Anatomy of a Diligence Report That Says Nothing and Passes Anyway

The report had nine rows. Nine rows, nine blanks.

Summary: not provided. Author stance: not assessed. Information points: null. Domain tag: unclassified. Confidence: not evaluated. Time sensitivity: not evaluated. Source quality: not evaluated.

It rendered in a clean grotesque, eleven point, exported to PDF, and went into a committee packet holding forty million dollars of discretion and eleven minutes of agenda time. Nobody stopped it. The document had done what documents do. It existed.

I have seen this artifact four times in eighteen months. Twice as a vendor deliverable. Once as an internal pipeline output pushed to a shared drive with a timestamp and no owner. Once as a paid "deep analysis" that a fund had already circulated to its LPs. In all four cases the failure was identical, and in none of the four was it malicious. The pipeline ran. It exited zero. It wrote a file.

The most dangerous output in security work is not a failing grade. It is a null that renders as a pass.

The automation nobody supervises

Between 2021 and 2026 the crypto diligence stack went from a spreadsheet and a paranoid analyst to six systems, three vendors, and one person who signs. Token scanners. Contract analyzers. On-chain analytics with subgraph backends. Sentiment aggregators. A summarization layer bolted on top because the raw output was too long to read. A templating engine at the end that flattens everything into a two-page PDF suitable for a packet.

The pitch is always the same. Weeks compressed into minutes. Institutional grade. Continuous monitoring.

The current market makes this worse, not better. Sideways price action starves allocators of the one signal they know how to read, which is direction. When direction is absent, they substitute proxies. A dashboard. A score. A letter grade. Anything with a date stamp on it. Chop is for positioning, the saying goes, but positioning requires information, and information is expensive, so the market buys the appearance of information instead. A consolidation market is where diligence debt accumulates quietly, because nobody gets punished for a bad process in a flat tape.

So the stack gets bought. And the stack has a failure mode that no one who purchases it is trained to look for.

The standard pipeline has five stages: fetch, parse, extract, score, render. Nulls are born in stages one and two. They are laundered in stages four and five. The laundering is the part that matters, because a null sitting in a JSON file is harmless — it is visibly, usefully empty. A null that has been through a scoring layer and a templating engine looks exactly like a finding that came back negative.

The specific artifact I examined last quarter had placeholders in the fallback branch of the template. Not errors. Not exceptions. Deliberate defaults, authored by whoever wrote the template, sitting in the code path that executes when the upstream extraction returns nothing. The template was designed to always produce output. It succeeded.

Four ways a pipeline returns nothing

Every silent failure I have audited resolves into one of five buckets, and the first four are boringly mechanical.

Silent parser success. The fetcher got a 200. The parser ran, found no matching selectors because the page layout changed eleven days ago, and returned an empty object instead of raising. The job exited zero. The orchestrator, which only checks exit codes, marked it green. This is the single most common cause and it is a design flaw, not an operational one: exit codes describe whether the process completed, not whether it produced signal. Every pipeline I have reviewed that reports health via exit code is a pipeline that cannot distinguish success from silence.

Schema drift. The extractor was updated to emit information_points. The template still reads info_points. The renderer, written defensively by someone who did not want a KeyError in production, substitutes the default. Now you have a report where the most important field was never populated, generated by two components that each behaved correctly against their own contract.

Empty input. This is the one nobody checks because it is upstream of everything anyone owns. Paywalled article. JavaScript shell with no server-rendered body. Encoding mismatch that turns a UTF-8 document into mojibake that the tokenizer strips. A fetch that followed a redirect to a consent page. The extractor receives three hundred bytes of boilerplate and correctly reports that it found no claims.

Skipped stage. A human shortcut. Someone pastes the template, intending to fill it later, and the file enters the review queue carrying default values that are indistinguishable from evaluated ones. This is the cheapest failure to fix and the most common in small teams.

Wrong document. This one is mine, and I add it to every checklist I write because it is the hardest to notice. The pipeline ran perfectly on the wrong artifact. A cached copy from the previous month. The proxy address when the implementation was the thing that changed. The Ethereum deployment when the incident was on the L2. The extractor was correct, the schema was correct, the renderer was correct, and the entire report describes a system that is not the system under review. I have caught this twice — once in a contract review where the in-scope bytecode hash did not match the deployed bytecode, and once in a treasury audit where the multisig being analyzed had been rotated six weeks prior.

None of these five produce an error. All five produce a document.

The atomic unit, and why refusal is the only correct output

There is a structural reason the empty report could not be analyzed, and it is worth naming precisely, because it is the same reason half the diligence I read cannot be analyzed.

Analysis needs entities. The smallest entity in document forensics is a claim with provenance: a statement, a source, and a type. Statement: the protocol will distribute 40% monthly. Source: whitepaper page four, section titled Guaranteed Returns. Type: prediction presented as fact. That is one information point. It is verifiable, locatable, and classifiable. Ten of those and you can build an argument map, test the logic chain, and separate what is known from what is asserted.

Zero of those and you have nothing to analyze. Not little. Nothing. There is no smaller unit to fall back to. Confidence scores, domain tags, sentiment — all of those are properties of information points. Strip the information points and the metadata describes an empty set.

I learned this at twenty-one, dissecting a whitepaper that promised 40% monthly returns. I did not start with a conclusion about fraud. I started by listing every claim in the document and attempting to attach a verifiable referent to each one. Daily payout mechanism: no contract, no address, no code. Referral structure: present, documented, mathematically consistent with a Ponzi payout schedule. Trading bot: asserted, never specified, no exchange API credentials, no on-chain evidence of trading activity. The finding was not that any single claim was false. The finding was that the majority of claims had no referent at all — and that the document's credibility came entirely from the density of the ones that did.

That is the whole discipline. A claim without a referent is not a weak claim. It is not a claim.

Which is why the correct output of an empty input is a refusal with a diagnostic, and why almost nobody produces one. A summarization model handed an empty document will not refuse. It will produce fluent, confident, structurally valid prose about the absence of findings, because that is what it was optimized to do — produce output. The counterfeit is not the fabrication of facts. It is the fabrication of the shape of an analysis. Form without content is harder to catch than a wrong number, because a wrong number can be checked against reality and a well-formed blank cannot be checked against anything.

The laundering layer

The null becomes dangerous at render time.

Trace the supply chain of a single missing value. The extractor returns information_points: []. The scorer sees an empty array and computes no score, writing null. The template receives null for a field it cannot leave blank, so it substitutes a default string. The renderer, which was written to be readable, styles that default string the same weight and color as every other value. The PDF goes to a reader who has eleven minutes and no reason to open the JSON.

Five hops. Four systems. Zero humans between the empty array and the committee packet.

Now compare the language available to an actual auditor. Financial audit has spent a century building vocabulary for exactly this situation. Scope limitation. Qualified opinion. Disclaimer of opinion. "We were unable to obtain sufficient appropriate evidence." Going concern doubt. These are not hedging words. They are precise instruments that let a professional say I could not verify this in a form that is legally distinct from this is fine and carries different consequences.

The crypto diligence stack has no such vocabulary. It has a green checkmark and a red flag and, apparently, a gray placeholder that renders at the same confidence as both.

I signed a memo in 2024 that existed entirely to make this distinction legible. The subject was the key management architecture inside a spot Bitcoin ETF's custody arrangement. The product was secure. The multisig design was sound, the key ceremony had independent witnesses, the operational controls were better than most DeFi protocols will ever manage. And the architecture was deliberately opaque in ways that had nothing to do with security and everything to do with satisfying a regulator — key shards held by entities selected for their licensing posture, not their operational competence, and a recovery process whose documentation was written for examiners rather than for engineers.

That memo was not a criticism of the product. It was a scope statement. Here is what was verified. Here is what was accepted on representation. Here is what could not be checked by anyone outside the perimeter. Three buckets. Any report that cannot sort its contents into those three buckets is not a report about the thing; it is a report about the report.

The check I run on every document I am handed, and it has never failed me: delete the analysis section. Does the document still look complete? If yes, the analysis section was decoration and the document's real function was to occupy a slot in a packet. That empty nine-row report passed this test in the worst possible way. Remove the findings and nothing changed, because there were no findings, and the layout did not care.

Four instances in production

Abstractions are cheap. Here is where the same failure mode shows up in live systems, and what it costs.

Scope laundering in smart contract audits. I have read audits whose conclusion was "no critical findings" where the in-scope artifact was a proxy contract, forty lines of delegation logic, and the implementation holding every user's funds was explicitly out of scope. The audit was accurate. The report was honest. The conclusion was arithmetic on an empty set. Nobody lied; the deliverable simply had a null where the risk lived, rendered at the same visual weight as the parts that were checked. When I review an audit now, I read the scope section before the findings, and if the scope is not expressed in deployed bytecode hashes, I treat the findings as unverified. Audits are reassurance until you compare the scope against the address.

The indexer that died quietly. A subgraph stops indexing and the TVL chart goes flat. Flat reads as stability. Flat reads as a market that has settled. What actually happened is that the query layer froze at block N and the frontend has been rendering a photograph, not a feed. The detection is trivial and almost nobody does it: query the indexed block height alongside the metric, compare it against the chain head, and alert when the gap exceeds a threshold. A flat chart and a dead indexer are visually identical and semantically opposite. I have seen a lending protocol's borrow-rate dashboard described in a committee memo as "stable" during a period when the indexer had been four days behind and the real utilization curve had moved forty percent.

Oracle freshness has two definitions and dashboards pick the convenient one. In 2020, mapping the bZx exploit, the attack was not a bug in the protocol's code. It was a legitimate sequence of operations against a price source that could not distinguish "recent" from "correct." The oracle had updated within its heartbeat window. It was fresh by the definition the contract checked and stale by the definition that mattered. Every oracle integration I review now gets the same question: which property are you actually validating — timestamp, deviation, or both — and what does your circuit breaker do when one passes and the other fails? Most integrations check one. The other becomes a null that renders as a valid price.

Wallet clustering as the missing evidence. When I reverse-engineered a high-profile NFT launch in 2021, the public narrative was decentralization. The metadata said so. The launch mechanics said so. What the metadata does not contain is wallet clustering, and when I ran funding-graph analysis across the mint transactions, over fifteen percent of supply resolved to addresses with traceable funding paths back to the deployer's operational wallets. The project was not lying about its metadata. It simply never published the analysis that would contradict the narrative, which is the most common form of dishonesty in this industry. NFTs are art until you inspect the metadata hash. Projects are decentralized until you trace the funding graph. Neither claim is false. Both are unverified at the layer that determines whether they matter.

The Terra audit in 2022 was the same shape at ten thousand times the size. The peg held at block granularity on every dashboard that mattered, because the price the dashboards consumed was produced by the system being defended. Circular verification, rendered as a stable chart, in a market that read stability as safety. Forty billion dollars against a null that nobody had labeled.

Designing for the loud null

What I now require from any pipeline before I trust a single row of its output is not accuracy. Accuracy is downstream. I require that it fail loudly, in the schema, in a form a non-engineer can read.

Four constraints, all cheap.

Evidence arrays cannot be empty. If a schema has an evidence field, the validator rejects []. Not warns. Rejects. A claim with no evidence is the exact thing the field exists to catch, and an empty array is a positive assertion that the check found nothing, not a missing value. Encode that distinction or you will lose it.

Confidence enums get no null member. If the allowed values are high, medium, low, and unassessed, then unassessed is a legitimate output and the renderer must style it differently from all three others — not gray it out, but mark it as an unresolved item requiring a named human. The template's fallback branch should be the loudest output in the system, not the quietest.

Every null gets a name. When a field cannot be populated, the pipeline records why and who is accountable: fetch failed with status 403, parser found no matching selectors, source is paywalled, source is a JavaScript shell, human skipped with intent to return. This is a two-column addition and it converts a silent failure into a work item.

The renderer refuses to print a summary on zero information points. This is the entire fix. One conditional. If the entity count is zero, the template emits a diagnostic page instead of a report, and the diagnostic page says, in plain language, that nothing was extracted and no analysis was performed. Not a placeholder. A refusal. The document that reaches the committee should look broken, because it is.

I have watched teams spend six figures on extraction quality and zero on the render layer, which is where the loss actually happens. The best extractor in the world, wired to a template that cannot express absence, produces a PDF that reads as a pass.

Who benefits from the ambiguity

There is a reason this failure persists, and it is not technical.

Vendors invoice on throughput. Reports per month, coverage per quarter, monitored addresses. Nobody's renewal depends on refusal rate, and a vendor whose system returns "I could not analyze this" on twenty percent of inputs looks broken next to a vendor that returns a confident two-pager on all of them. The market selects for rendering, not for extraction. Same structural pressure that makes rating agencies generous and sell-side research bullish.

Analysts face a sharper version of it. Saying "the input was empty and I will not fabricate an analysis" is professionally correct and commercially expensive. It reads as an inability to perform. The person who writes four pages on an empty document looks productive. The person who writes one paragraph saying the document contained nothing verifiable looks like they did not try. I have lost work to this. I expect to again.

And the allocator, the person with eleven minutes, gets plausible deniability. A blank report in a packet means the process was followed. It does not mean the position was examined. In a sideways market, where nothing is going up and nothing is blowing up, process artifacts substitute for judgment everywhere, because there is no P&L event to correct the error. The correction arrives in the next drawdown, when the report that said nothing is discovered to have been accurate and useless.

What the automation skeptics get wrong

The reflexive conclusion here is that automation was the mistake. That a human analyst reading the original document would have caught the emptiness.

I have read enough human analyst reports to disagree. A human handed an empty document produces exactly the same artifact — fluent prose, no referents, confident tone — and does it without a fallback branch to blame. The schema was the only honest part of that broken report. It was the only component that said, in a machine-readable field, that it had found nothing. Every other layer, human or not, was optimized to conceal that fact behind formatting.

Structured extraction is not the enemy of good diligence. It is the only thing in this industry that makes absence visible at all. The rigid template, the enumerated fields, the required arrays — those constraints are what force a system to declare what it does not have. Fluent prose is where absence goes to hide. That is why so much of the confident research in this space cannot be falsified: it has no fields to be wrong in.

Which inverts the usual critique. The pipeline did not fail because it was automated. It failed because the automation stopped at extraction and let a rendering layer, written by someone optimizing for a screenshot, decide how to express a null. The extraction was correct. The extractor reported zero entities. Somewhere downstream, in a template nobody reviewed, a correct report of nothing became an incorrect report of everything being fine.

An audit is a clean bill of health until you compare the scope to the deployed address. A dashboard is data until you check the indexer's block height against the chain head. Both statements are the same sentence wearing different clothes: the output of a monitoring system is a claim about the monitoring system until you verify it against the thing it claims to describe.

The question to ask before the next drawdown

The differentiator in this cycle will not be model quality, extraction accuracy, or coverage. Every vendor will have adequate versions of all three within eighteen months. The differentiator is whether the stack fails loud — whether a null produces a diagnostic page or a green checkmark, whether an empty input stops the process or feeds it.

The test costs nothing and takes a minute. Next time a diligence report lands on your desk, ask what it would look like if it had found nothing at all. If the answer is that it would look exactly like this one, you do not have a report. You have a slot in a packet, and a countdown you cannot see.