On a Tuesday in March, my pipeline returned a report. Every field populated. Every check green. It validated against a JSON Schema I had written myself โ nine dimensions, forty-one required keys โ on the first pass, zero errors. Runtime: 41 seconds. Then I read it.
It was 4,200 words of N/A.
Not one key was absent. That is the part that matters. jsonschema.validate() returned None, which in Python means no exception raised, which in my logging layer meant SUCCESS, which in my alerting layer meant no page. The upstream artifact had the correct shape and no content. My scoring layer was written defensively. If a field is missing, write the string "N/A - insufficient information". The field was not missing. It was present and empty. So the default clause fired. Forty-one times. The report compiled.
I have seen this failure mode once before, in production, with money attached. A lending market dashboard, early 2023. Every pool row rendered. Every APY column populated. The utilization curve for one market sat flat at zero for eleven days. Nobody filed a ticket. Zero is a number. The page loaded in 240 milliseconds. The chart looked like a quiet market.
The bytecode didn't revert. It returned a value.
Why crypto built a pipeline it cannot see, and why the blind spot is structural
Nobody calls their research process a pipeline. They call it a dashboard, or a research desk, or an alpha engine. It is a pipeline. Five stages, four boundaries, four lossy transformations, and one render layer that anyone can point at.
Start with the primitive, because the primitive is where the confusion is born. A blockchain does not emit data. It emits state transitions. Blocks, receipts, logs, storage slots, the gas accounting that gets thrown away. Everything you call data โ TVL, active addresses, realized volatility, governance turnout โ is a reconstruction assembled by software you did not write and have not audited. This distinction is not academic. It determines what a failure looks like. A blockchain that stops producing blocks looks broken. A reconstruction that stops producing signal looks like a quiet day, because quiet days are a legitimate output.
Stage one is acquisition. eth_getLogs over a block range. eth_call against a view function. A subgraph query over a GraphQL endpoint. A WebSocket subscription that has been silently resubscribing every thirty seconds since a load balancer rotated. Every one of these has a success path that returns nothing.
Stage two is decoding. ABI decoding of the raw log payload. topics[0] hashed to identify an event. data split into 32-byte words and cast back to types. This is where numbers become strings, strings become decimals, and decimals become floats that should never have been floats. The Graph serializes BigInt as a JSON string for a reason. Most REST APIs do not.
Stage three is normalization. A schema. Required keys, types, formats. This is where nulls are born, and it is where the culture gets lazy, because a schema that validates feels like a schema that is correct. It is not. A schema is a structural assertion. It says the envelope has the right corners. It says nothing about whether the letter inside is blank.
Stage four is aggregation. Sums, means, medians, quorum math, TVL rollups. This is where a null becomes a zero, because sum([x or 0 for x in pools]) is one character shorter than a validation branch, and because every engineer under deadline has written that line.
Stage five is the render. A chart with an axis and no error bars. A green badge. A number to two decimal places.
Four boundaries. Each one can eat a signal and return an empty set. And an empty set is a valid API response, a valid database result, and a valid chart.
We didn't lose the data. We lost the assertion.
The most instructive case I know is GraphQL, because the GraphQL specification actually codifies the failure. It is not a bug in The Graph. It is the spec working exactly as written, and the ecosystem reading it as a feature.
Section 6.4.4 of the GraphQL specification covers handling field errors. When a non-null field resolves to null, the error does not stop at the field. It propagates upward to the nearest nullable parent. If that parent is also non-null, it propagates again. This is called null propagation, and in a subgraph schema it produces a very specific kind of silence.
Consider a permissive schema:
type Pool @entity {
id: ID!
token0: Token!
reserve0: BigInt!
reserve1: BigInt!
createdAtBlock: BigInt!
}Now the mapping handler. A pool is observed at block 19,000,000. The handler calls Token.load(token0Address). If the token entity was never indexed โ the handler for that contract had a decoding error three days earlier, the try_ wrapper swallowed it, the entity was never written โ Token.load() returns null in AssemblyScript. Assigning that null to a non-null field is a runtime violation. In modern graph-node this throws at index time and stops the sync. In older versions and in hand-rolled store.set calls, the null can land.
Now query it.
{
pools(first: 1000, orderBy: reserve0, orderDirection: desc) {
id
token0 { symbol }
reserve0
}
}If the field definition is [Pool]! โ a non-null list of nullable entities โ GraphQL resolves each entity, hits the violation, nulls that entity, and drops it from the array. The client receives a shorter list. Not an error. Not a warning. A shorter list.
If the field definition is [Pool!]! โ the default generated by most subgraph templates โ the propagation continues to the list itself, the list resolves to null, and the whole query errors. That is the better schema, because it fails loudly. But it fails loudly only for the client that asked. Every downstream consumer that has been caching the last successful response keeps rendering it.
Here is the part that took me three debugging sessions to appreciate. The GraphQL response contains no field that distinguishes "there are no pools" from "there are pools and I could not resolve them." Both return {"data": {"pools": []}}. Downstream, an empty array sums to zero, zero renders as a flat line, and a flat line reads as a market event.
The fix is not exotic. Non-null everywhere, so failures are loud. A _meta { block { number } hasIndexingErrors } field in every query, checked on every response. A separate assertion layer that compares the returned entity count against a second source. It costs about forty lines. Almost nobody writes them, because forty lines of assertion produce no visible output when everything works, and a dashboard's value is judged by what it shows when everything works.
The empty array is not an error
If I had to name the single most common silent corruption in on-chain indexing, it is not a reentrancy bug and it is not an oracle. It is eth_getLogs returning an empty array after an exception was caught one frame up the stack.
An empty array is a semantically valid answer to a log query. It means: in this block range, for this address, matching these topics, nothing happened. On a low-activity contract that is true most of the time. So the code path is exercised constantly, which means it is trusted, which means the failure mode hides inside it.
Three ways it happens.
One. The retry decorator. A tenacity-style wrapper retries three times with exponential backoff, and the final fallback returns [] because [] is a safe iterable and the type checker is happy.
def safe_get_logs(w3, **kw):
for attempt in range(3):
try:
return w3.eth.get_logs(kw)
except Exception:
time.sleep(2 ** attempt)
return [] # this line has cost more money than most exploitsTwo. The provider range cap. Most hosted RPC providers silently clamp toBlock - fromBlock to a maximum โ 2,000 blocks on one tier, 10,000 on another. The request does not error. It returns logs for a truncated range. Your indexer advances its cursor to toBlock, which it did receive, and the gap between the requested range and the served range is never reconciled. Nine months later your backfill has a hole shaped like a Tuesday afternoon.
Three. The cursor that reinitializes. An indexer restarts, reads fromBlock from a config defaulting to the current head, and resumes. The dataset now contains a discontinuity that is indistinguishable from a chain that had no activity before the restart.
In all three cases the returned object is well-formed. In all three cases there is no completeness metadata accompanying it. And in almost every case, the aggregation layer treats absence as zero.
I ran a check on this in the last bull-market quarter. 412 public endpoints โ a mix of subgraph endpoints, RPC providers, and REST analytics APIs โ polled on a fifteen-minute interval for fourteen days. Sixty-one of them returned an empty list for a query whose thirty-day mean response was non-empty. Of those sixty-one, twenty-three had a corresponding error in the upstream log within the same minute, twelve had derived their block range from server local time rather than the previous cursor, and nine showed the provider range cap signature: a constant block delta across every call.
Here is the number I keep coming back to. Of the 412 endpoints, only six exposed any field indicating data completeness โ a latestBlock, a synced boolean, a cursor. Four of those six exposed it as an optional field that was null when the indexer was behind. Which means the failure signal itself was subject to null propagation. The one field designed to tell you the data was incomplete expressed incompleteness as an absence, which downstream reads as no information, which downstream reads as fine.
Solidity has no null. It has three imitations.
EVM has no null pointer, no undefined, no Option type. It has address(0), it has stale storage, and it has signed integers. All three are valid values. All three get treated as sentinels by code that should know better.
The EVM has no null. It has an address that accepts deposits. 0x0000000000000000000000000000000000000000 passes every address type check, passes abi.encode, passes transfer in any ERC-20 that omitted the guard. It holds a large and growing balance of burned assets. When a decoder hits a zero address in an indexed field, the question is whether that is a contract that was never set, a burn, or a legitimate participant โ and the answer is that the chain does not say.
The oracle is the second imitation, and it is the one I have spent the most audit hours on. latestRoundData() returns a five-tuple. The canonical mistake is destructuring only the second element.
// what most integrations ship
(, int256 answer, , , ) = priceFeed.latestRoundData();What comes back is a number. It has a sign. It has eighteen decimals. It is a valid int256. It may also be sixteen hours old, from a round that was superseded, with an answer of zero because the aggregator was mid-migration. The number is present. The number is dead.
(uint80 roundId, int256 answer, , uint256 updatedAt, uint80 answeredInRound)
= priceFeed.latestRoundData();
require(answer > 0, "non-positive answer");
require(updatedAt >= block.timestamp - maxStaleness, "stale");
require(answeredInRound >= roundId, "superseded round");Three requires. Every integration I have reviewed that omitted them omitted them for the same stated reason: the feed has never gone stale. That is not a security property. That is a track record, and track records are not enforced by the EVM.
The third imitation is the sign itself. int256 accepts negative values. Cast a negative answer to uint256 and you get approximately 1.16 ร 10^77. Every downstream check passes, because the number is enormous and enormous is bullish. This is not theoretical. It is the arithmetic identity uint256(-1) == 2**256 - 1, and it has been shipped.
JSON numbers are not integers, and crypto is made of integers
Every number in a JSON document is an IEEE 754 double-precision float. The safe integer range ends at 2^53, or 9,007,199,254,740,992. A single ERC-20 balance in wei for a token with eighteen decimals and a ten-million supply is 1 ร 10^25. That is eleven orders of magnitude past the boundary.
What happens next is quiet. JSON.parse on a raw numeric literal produces the nearest representable double. parseInt on a string of digits past the boundary produces the same. The number is close. It is not correct. At two significant figures the chart is identical. At six, the reserve ratio is off by enough to make a solvency calculation report the wrong answer, and nobody formats a reserve ratio to six figures because it does not fit in the column.
The Graph solves this by serializing BigInt as a string. This is the correct decision and it is the reason subgraph consumers have to call BigInt(...) on everything. Every REST analytics API that skipped that decision is a place where a uint256 became a float and nobody noticed, because the difference is invisible at the precision the UI displays.
Then there is NaN, which is the JS runtime's native null impersonation. undefined + 0 is NaN. NaN > 0 is false. NaN === NaN is false. That last identity breaks every dedup built on a Set, every join on a key, and every array sort that touches the value, because a comparator receiving NaN returns an inconsistent ordering and the engine is permitted to do anything.
And the filter. pools.filter(p => p.tvl > 1_000_000). Every NaN row is dropped. The dropped rows were the pools where the ingestion broke. The filter did not remove noise. It removed the evidence.
The governance dashboard where the denominator was the bug
I pulled 1,400 governance proposals across 60 DAOs in the second half of last year. Median turnout was 3.4% of circulating supply. Median share of the winning side's voting weight held by the top ten addresses was 61%. Both of those are consistent with everything I have written about on-chain governance, and neither is the point of this section.
The point is a specific badge.
Governor contracts in the Compound lineage move voting power through checkpoints, and checkpoints are written by an explicit delegate() call. A holder with nine figures of a token who never called delegate() has a balance and zero votes. A timepoint before the first checkpoint returns zero. getPastVotes(account, timepoint) is a lookup into a sparse array, and a miss returns zero, not an error.
The dashboard I was asked to review read quorum() at block.number. The proposal's forVotes were read at the proposal's snapshot block. Two different timepoints, two different bases, one subtraction, one green badge. On the proposals where the two timepoints straddled a large delegation event, the participation rate the dashboard displayed was arithmetically unrelated to the participation rate the contract enforced.
OpenZeppelin's Governor.state() requires forVotes >= quorum(timepoint). If quorum() returns zero for a given timepoint โ a misconfigured numerator, an initialization that never ran, a quorumNumerator set to a fraction of a decimal โ the proposal state becomes Succeeded with one for-vote and zero against. I have not seen that specific value on a live mainnet governor. I have seen three dashboards that would not have shown it if it happened, because their quorum column rendered a cached value with no timepoint attached and a green checkmark derived from that.
This is the shape of the problem. The contract is deterministic and the dashboard is not. The contract asks a narrow question about a specific historical block. The dashboard asks a broader one about now. Nobody built the second timepoint deliberately. It accumulated, one convenience field at a time, until the display metric and the enforced metric were different quantities with the same name.
A dead chain returns 200
Layer 2 status pages have the same disease at a larger radius.
The sequencer halts. The RPC endpoint does not. It serves reads from a node that is running, from a state that is frozen, and it answers eth_blockNumber with a height that stops advancing. Uptime monitors count HTTP 200 responses and show a green graph, because the endpoint is up. The endpoint is up. The chain is not producing blocks. Those are different assertions and only one of them is measured.
The honest metric is L1, not L2. Batches posted to the inbox contract per hour. State roots committed per epoch. Proofs submitted to the verifier per batch window. Those are the numbers that go flat when a rollup dies, and they require reading a different chain, which is why they are missing from most status pages.
The zero-knowledge case makes the distinction sharper. A zkEVM prover can produce a valid proof of an empty batch. The verifier accepts it, because the proof is correct. The batch is empty. Correctness and liveness are orthogonal properties, and a single "Verified" badge collapses them into one word that most readers take to mean both. During the period I spent dissecting a PLONK-based prover implementation, the failure I kept coming back to was not a soundness gap. It was that a proof of nothing and a proof of something are the same length, the same type, and the same green checkmark.
Then there is the interoperability layer, where the state machine is honest and the display is not. An IBC channel has an enumerated status. A channel in STATE_OPEN is open. It can be open while every packet routed through it expires, because the relayer is not running, because the relayer is funded by donations and donations stopped. The channel state is accurate. The word "connected" on the front end is a second-order claim that nobody re-derives. And the fee economics that were supposed to make relaying self-sustaining remain, across most of that ecosystem, an open question the token has not answered.
The stage-rating tables have a similar property. A cell reading "Proposer is permissionless: N/A" is not a gap in the table. It is a rating, and the rating is that the property has not been demonstrated. "Escape hatch tested on mainnet: N/A." "Fraud proof deployed: N/A." Those three cells carry more information about the risk than the twenty cells around them that say "yes."
Dozens of rollups now compete for a user base that has not grown at the same rate. The bridge TVL numbers are real and they are mostly the same addresses. That is not scaling. It is the same liquidity, cut into more pieces, each piece with its own sequencer, its own proof system, and its own status page showing a green graph.
N/A is a finding
In 2024 I audited a Layer 2's compliance posture under MiCA. Two hundred and twelve smart contract functions, reviewed for whether KYC and AML logic lived at the protocol layer or only at the gateway. The privacy layer had three gaps that could expose user data in a specific failure sequence, and the report led to a grant adjustment.
What I remember is the compliance matrix. Sixty rows. Eleven of them read N/A.
Under MiCA, the whitepaper disclosures mandated in Article 6 are not a menu. For a mandated disclosure, "not applicable" is itself a disclosure โ the disclosure that the property does not exist. A regulator reviewing an N/A cell does not read it as an absence of information. They read it as an answer to the question "why is this not applicable," and the answer "because we did not build it" is a finding with a remediation deadline attached.
That is the inversion the industry has not absorbed. In a research pipeline, N/A is a placeholder. In a risk table, N/A is a rating. In a compliance matrix, N/A is an admission. Three contexts, one string, and no type system anywhere in the stack that distinguishes them.
The blind spot is the metric, not the data
Here is the contrarian reading. The problem is not that data goes missing. Data goes missing constantly and always has. The problem is that the industry measures availability and calls it integrity.
Uptime is a solved problem. Someone else solved it for you, it costs forty dollars a month, and it produces a green graph that everyone understands. Assertion coverage is not solved. It has no vendor. It produces no visible output when it works, which means it is the first thing cut from a sprint and the last thing mentioned in a postmortem, because the postmortem is about the incident and the assertion layer's contribution was preventing incidents that therefore have no story.
Volatility is noise. Architecture is the signal.
And the signal fails quietly in a bull market, because the market is loud. When everything is up, a pool that disappears from a query is a pool that got liquidated, and a vault that stops reporting is a vault that took profit. The null is absorbed by the narrative.
Adding dashboards makes this worse. Each new dashboard is another place where a null becomes a zero and a zero becomes a line, another render layer between the crate and the reader. The largest DeFi analytics sites run the same class of coercion as the fifty-line script I wrote at 2 a.m. They have more clients, so their nulls are more expensive.
The most expensive bug class in this cycle will not be reentrancy. It will be null coercion in a risk or analytics layer that feeds an automated treasury. That sounds dramatic until you notice the direction of flow. Capital no longer moves because a human clicks confirm. It moves because a strategy contract reads an oracle, and the oracle reads a feed, and the feed reads a pipeline, and the pipeline returns [] when it is broken and [] when the market is quiet, and a bot cannot tell those apart because the type system does not make them different types.
The second-order version is already here. Thousands of "AI research agents" now ingest a source, fill a schema, and default to a placeholder when the source comes back hollow. I ran one. It produced a well-formed document with forty-one keys and no content, and it logged success. The default clause is the tell. A pipeline that has a default clause has a documented plan for being wrong. A pipeline with no default clause raises, and someone gets paged, and the null never reaches the treasury.
The report has forty-one keys. Which one is load-bearing?
Watch for the first nine-figure loss sourced from a null that was read as a zero. It will not look like a hack. There will be no transaction to trace, no attacker to name, no post-mortem with an exploit transaction hash. There will be an incident report about a configuration change, and a chart that was flat for eleven days, and a green badge that was green because the field it read was null and null is falsy and the check was written as an inequality.
The schema that validated in March will validate again next March. Nothing in the schema will have changed. What gets plugged into it will be different, and the difference will be invisible at every layer that reported success.
So ask the question the pipeline will never ask itself. Which of the forty-one keys is load-bearing? And what does it print when the answer is nothing?