Three rules for a security score that doesn’t lie to you

Haven has always shown you counts. Three critical findings, eight high, forty-seven resolved. Counts are honest and they are useful, and they have one flaw that took us a while to name: they describe the present tense only. They tell you what is wrong right now. They cannot tell you whether you are better off than you were last month.

That turns out to matter more than it sounds, because it is the difference between a tool you use once and a tool you keep.

The second-visit problem

The pattern we kept seeing was this. Someone installs Haven, runs a first scan, gets a genuine shock — a plaintext password in a 2019 email, a tax return shared with a link that still works — fixes the worst of it, and closes the app. Then nothing. There is no reason to come back, because nothing in the product demonstrates that coming back is worth anything.

The counts, of course, went down. But a number that went from 58 to 41 with no memory of the 58 is just a smaller number. Without history, every visit is the first visit.

So we set out to build a single number that carries direction as well as state. And almost immediately ran into the fact that most security scores are quietly dishonest.

Counting is the obvious answer, and it is wrong

The obvious implementation is to add up the findings, weight them by severity, and subtract from 100. We built that first. It produces a number that is stable, explainable, and unfair in a specific way that took a real archive to notice.

Consider two people. One opened an email account last month and has 1,200 messages in it. The other has been using the same address since university and has 140,000. The second person will always have more findings in absolute terms — not because they are careless, but because they have more life stored. Scoring on raw counts tells the person with the longest history that they are the most negligent, which is both untrue and precisely the wrong message to send to the user who needs the product most.

So the score divides. Every open finding contributes its severity weight, that total is measured against the size of everything Haven scanned, and the resulting density is what becomes the number. The question stops being “how many problems do you have” and becomes “of everything you have, how much is exposed.”

The two archives above are not a hypothetical. Those are the scores our shipped code produces for those inputs: the fresh account with three critical findings scores 67, and the fifteen-year archive with forty scores 95. The second person has thirteen times more findings and is, by the only measure that means anything, keeping a tidier house.

Rule one: never punish someone for looking

Here is the failure mode that worried us most. You connect Gmail, get a score, spend a fortnight cleaning it up, and watch the number climb. Pleased with yourself, you connect your old Outlook account — the one you have been avoiding — and the number falls off a cliff, because Haven can now see fifteen years of things you had forgotten.

Nothing got worse. Your exposure did not change; only our visibility did. But the interface just told you that using more of the product made you less safe, and the obvious lesson to draw is: don’t connect the other account. We would have built a machine that teaches people not to look.

So the first rule is that connecting a source can never lower your score. At the moment you connect, Haven records an offset that holds the number exactly where it was, marks the point on your chart as baseline moved, and lets you climb from there.

The subtlety is in the middle panel, and we got it wrong on the first attempt. Our initial implementation simply pinned the score across a connection — held it flat, no matter what. A test caught the problem: if the account you just connected is tidier than what you already had, your overall density genuinely improves, and pinning silently swallows that. The rule protects you from a drop; it was never supposed to suppress real good news.

So the offset only ever grows. If the new account makes things worse, the offset absorbs exactly the damage and your number holds. If it makes things better, the offset stays where it is and the score rises on its own merit. One consequence worth stating plainly: because that offset is never clawed back, your score is only ever comparable to itself over time — never to anybody else’s. Which brings us to the thing we deliberately did not build.

Rules two and three: agree with the user

The other two rules are simpler, and both are about the score not arguing with the person reading it.

Remediating moves the score up immediately — on the action, not on the next scan. If you revoke a public link and the number sits there until tomorrow’s scan, you have learned that the number is not really about you.

Telling Haven that something is normal for you moves it up too. Haven lets you mark a finding as expected — the test credentials in your own dev inbox, the address book you share with your family on purpose. If you have said something is fine and the score keeps counting it against you, then the score is not measuring your exposure, it is measuring our opinion of your exposure. Those are different products.

What we left out on purpose

An early design had a comparison: you’re in better shape than 84% of Haven users. It tested well. We cut it.

Two reasons. The first is structural: shipping that line means every installation has to report its score to us, which means building a channel that carries a compact summary of how exposed a specific person is. That is precisely the kind of data we have spent this product’s entire architecture arranging not to hold. The score is computed on your device, stored on your device, and never transmitted — and a percentile is not worth breaking that for.

The second is that the comparison is an invitation to stop. Being in the 84th percentile is genuinely compatible with having a password sitting in plain text in your inbox. A number that makes you feel finished while you are still exposed is not a feature.

Where the score stays quiet

Two behaviours look like bugs and are not, so they are worth saying out loud.

Small wins do not always move the numeral. Clearing one low-severity finding out of a hundred thousand scanned items is a real improvement and a tiny fraction of a point. The displayed score will not budge. We think that is correct — a number that jumped on every click would be a progress bar, not a measurement — but it does mean the score is the wrong place to look for encouragement on a quiet day. That is why the panel beside it reports what you actually did: findings resolved, new findings, net movement, most-improved source.

Undo lowers it again. If you reverse a remediation, the exposure is genuinely back and the score reflects that. A line that only ever went up would be tracking your clicking rather than your posture.

One more, for the people who like this sort of thing: the density-to-score curve is a saturating hyperbola rather than the more obvious exponential decay, specifically so it can be computed with nothing but addition, multiplication, and division. Floating-point exp is a library call whose final bits are not guaranteed identical across platforms, and “the same data always produces the same score” was a requirement we did not want to hold loosely.

What the work turned up

Building the score meant building the trend chart, and building the trend chart meant shipping the other half of this release: a view of your shared files organised by who can reach them rather than by which file they are. A list of files is the right shape for handling one thing at a time and the wrong shape for “how much does this one person still have?”, which is the question that actually comes up when a contract ends.

Building that view is how we found a bug we had been shipping for months. When a cloud file was shared with several people outside your organisation, Haven collapsed it into a single finding — and kept only the first person’s name. The others were never stored. They were invisible in the audit, and because the revoke path matched on the name we had kept, they were un-revokable from inside Haven. The permissions had been fetched every scan and then discarded.

On its own that is an ordinary bug. What made it urgent is that the fix surfaces those missing people as new findings, which lowers scores — and we were about to ship a product where scores have history. Fix it later and every user sees an unexplained cliff in their chart caused by us getting better at detection. Fix it now, before anybody has a trend line, and it costs nothing. Both shipped together, in the same release, for that reason.

That is the whole argument for writing the rules down before the formula. The formula is twenty lines. The rules are what tell you which twenty.