Method

How to check a startup forecast after the fact

Diagram on a paper card: six horizontal range bars in shades of blue to the left of a vertical line, a dashed box to the right with empty circles waiting at the end of each row, and a gold seal in the corner

Most forecasts can never be checked because nobody wrote down a range, a date and the reasons. How to record yours, and what a report records for you.

The deck from two years ago

Picture a founder clearing out an old folder and finding the pitch deck from two years ago. This is a made-up example, but a familiar one. The deck said the business would have a certain number of paying customers by now. It has fewer. Was the forecast badly wrong, or roughly right with bad luck? Nobody can say.

The deck gave one number and no range. It did not say which assumptions the number rested on, or how sure anyone was about them. And in the meantime the story has been retold many times: the market was slower, a competitor appeared, the price was changed. Every explanation is plausible. None can be checked, because the forecast was never written in a form that could be checked.

Why most forecasts escape being checked

Three habits make a forecast uncheckable.

  • A single number. "We will have 2,000 customers" is almost certainly wrong in detail, so it cannot tell you whether the thinking behind it was good.
  • No date for the check. Without a stated month, a forecast drifts. A year later it becomes "eventually".
  • No recorded reasons. If the assumptions were not written down, you cannot tell which one failed, so you cannot learn from the miss.

Each of these lets a forecast be reinterpreted after the fact instead of scored.

Write it down before you know

The fix is simple and slightly uncomfortable. Before the outcome is known, write down:

  1. A range, not a point. "Between 400 and 1,600 customers by month 24", with the range as wide as your honest uncertainty.
  2. The month from which it can be checked.
  3. The assumptions it rests on, each with its own range, and where each figure came from.

Then, when the month arrives, ask one question of each range: did the truth land inside it? Over many forecasts this tells you something valuable. If you say "I am fairly sure" and the truth keeps landing outside your ranges, your ranges are too narrow. If it always lands inside, they may be wider than they need to be. This is called calibration, and it can only be measured from forecasts recorded in advance.

What a Startup Intelligence run records

Startup Intelligence takes the same view of its own output. The product's documentation puts its position bluntly: the verdict is never the product; the ordered queue of things that could kill the idea is. A run also keeps two records that make the engine itself checkable.

The ranges, recorded in advance. At the time of the run, the engine writes down the range it stated for each assumption that carries the outcome, with its evidence rung, before anything is known. The report's last page lists these recorded claims with the month from which they can be scored, and says how they will be scored: by whether the truth lands inside the range that was stated.

The settings, recorded on the run. The same page lists the seed, the number of simulated paths and the model settings the result came from, so the simulation can be re-run from the same inputs and give the same distribution. Someone who distrusts a result can check it instead of arguing with it.

What it does not claim yet

This is the part we want to be most careful about. No run has yet reached the month when its ranges can be checked. So there is no record of how well the engine's ranges have matched reality, and the report does not pretend there is. Its own words on that page are that there is no calibration curve yet, and that inventing one would be a lie.

Two more limits belong here. A simulation is a calculator: it computes the consequences of the assumptions it was given and does not check whether they are true. And today the engine consults no outside source during a run, so the recorded ranges rest on the AI model's own knowledge and are filed on the lowest rungs of the evidence ladder. A recorded range makes a forecast checkable; it does not make it right.

Start your own ledger

You do not need any tool to begin. Keep one page per idea with a few columns: the claim, the range you believe today, where the figure came from, and the month you will check it. Seal it in the sense that matters: do not edit old rows. When a result arrives, add a new row instead. As results come in, you learn whether your sense of certainty can be trusted, which is worth more than any single forecast.

Takeaway

A forecast earns trust only when it can be checked. Write down a range, a date and the reasons before the outcome is known, then score the range honestly when the date arrives. To decide in advance what result would make you stop, read kill thresholds, and for why the report shows ranges instead of a single figure, read why the report gives no single score. The evidence grades feature explains the ladder behind every figure, and how to test a startup idea before you build it puts the steps in order.

Reports support planning. They are not investment advice and not a guarantee of results.

Key takeaways

  • A forecast you cannot check is an opinion with decimals.
  • Record a range, a check date and the reasons before the outcome is known.
  • Judge forecasts by whether the truth lands inside the stated ranges, not by memory.

Questions

How do you check whether a forecast was right?

Write down a range, the date by which it can be checked and the assumptions behind it, before the outcome is known. When the date arrives, see whether the truth landed inside the range.

Can Startup Intelligence show how accurate its past forecasts were?

Not yet. Every run records its ranges in advance, but no run has reached the date when its ranges can be checked, so there is no accuracy record to show.

Startup Intelligence: what it does

Related articles

See Startup Intelligence for yourself

Request access