When the model gets it wrong: four failure modes we've actually seen

The scoring system on this site has been running live signals since early 2025. In that time it has produced a lot of correct calls, some wrong ones, and a few that were confidently, embarrassingly wrong. This piece is about the wrong ones — specifically, the patterns that keep showing up when we look at bad signals in hindsight.

Understanding failure modes is more useful than reading about successes. Anyone can show you the trades that worked.

Failure mode 1: regime change with no warning

The model is trained on data. That data reflects a particular market regime — the volatility range, correlation structure, and average holding periods that existed during training. When the regime changes, the model doesn't know. It keeps making the same calculations against inputs that now mean something different.

The clearest version of this happened when the market transitioned from an aggressive trending phase into a choppy, low-volume sideways grind. Technical indicators that had been reliable — MACD crossovers, moving average compression — started firing constantly on noise. Confidence scores were high. Most signals went nowhere.

There's no clean solution to this. The composite does include a volatility-regime input that tries to detect when conditions have shifted. But regime detection is genuinely hard, and "late detection" is really just "no detection" with extra steps.

Failure mode 2: factor correlation during stress

The composite takes four input categories and weights them. The theory: when technical signals are weak but on-chain is strong, they balance. In practice, during market stress, they tend to move together.

Here's a concrete example. In a fast drawdown, exchange inflows spike (people sending coins to sell), RSI collapses, social sentiment turns negative, and market-cap momentum turns down — simultaneously. Every factor signals the same thing. The score looks highly confident.

And it is confident — just not in a useful way. By the time all four inputs agree on "bad," the price has often already moved. High confidence during stress usually means "things are already bad," not "things are about to get worse." That distinction matters if you're trying to act on the signal rather than just confirm what you can already see on the chart.

Failure mode 3: the validator overfitting on familiar regimes

The neural network gate — the second-stage model that can suppress composite signals it doesn't recognize from training data — performs well on regimes it's seen. Poorly on new ones. That's just what neural networks do.

The specific failure pattern worth naming: it allows signals that look historically familiar but are structurally different. A high-RSI, positive-MACD, low-exchange-flow setup in a bull market looks similar to the same setup six months later in a pre-correction. The NN trained on the prior period might approve both. One of them worked. The other didn't.

The validator reduces bad calls in aggregate. It produces false confidence in specific situations. We try to make this visible by showing the raw composite score alongside the validated signal — if they diverge significantly, that divergence is itself information.

Failure mode 4: social data arrives late

Social sentiment is the noisiest input category by a wide margin. Twitter mention volume, Reddit activity, and Trends data all have meaningful lags. By the time a spike in mentions shows up as a positive sentiment reading, the price move it correlated with has usually already happened — in either direction.

The model handles this by using z-scored deviations from rolling baselines rather than raw counts. Z-scoring doesn't solve the lag problem. It means "high relative to recent baseline" rather than "high in absolute terms," which is a useful frame — but when something trends on crypto Twitter, price moves first and the data catches up later.

This is why social signals are weighted at 20% in the composite, not higher. They're useful for confirming a signal that other factors have already pointed toward. Not for generating one.

What to do with this

The failure modes above don't mean the system is useless. They mean it's a model. Accurate enough across a large sample to be worth paying attention to. Wrong enough in specific situations to cause real losses if followed uncritically.

The most common mistake is treating a 90% confidence score as a 90% probability of the trade working. Confidence measures internal consistency of signals — not accuracy against future price. A score can be internally consistent and completely wrong.

Read the factor breakdown. If everything is pointing the same direction, ask why. Sometimes it means a strong trend. Sometimes it means the model is about to embarrass itself, and in hindsight the reason will be obvious.