Last week Stephanie and I toured Majorca on our “Post-Vacation Vacation”, and I can confirm two things. and even without my girls around… I still end up in arts and crafts.

At the end of July, we took the three girls to Lanzarote, then they went off to their respective co-parents and we seized the opportunity to disappear for a week by ourselves. I turned another year older on Monday, which I am reliably informed is fine.

Also I got this pretty neat Lego gift:

Back at the desk now.

This issue is layer one of the Evidence Stack: data.

Governance was about who decides. Data is about what you collect, and… far more importantly — why. It is the layer where I consistently see the most money spent and the least thinking done.

Previous edition: L0 Governance

ProofStack does not typically lead on data. We are not data scientist (but we’ll happily refer you to our friends who are). We advise, but the company owns it.

There are two moments when people call us about this layer.

The first is our least favourite: A company has collected huge amounts of data over several years, has no especially clear idea why, and would now like someone to find the value in it.

The second is our favourite: A company is planning what to collect and has not started yet. Which means we can start with the end in mind.

The distance between those two phone calls is usually about two years and a great deal of money.

Founders add fields to their products on the basis that they might need them later, for some purpose nobody has defined. It feels prudent. It is not.

Every additional thing you ask of a user raises the odds they disengage. And we know from clinical trials, and we know from registry studies, that an enormous proportion of collected data is never analysed at all. It sits there. It was never attached to a question.

So the hard ask we put to people is this: what would actually be different if you had all the data?

Imagine it is five years from now and you have beautiful longitudinal data on ten thousand people. Or a million. What changes?

And here is the thing. When people answer that question honestly, they very rarely describe data. They describe something else.

There are four rungs, and the acronym is DIKW.

Data is the ones and zeros. Date of birth, sex at birth, scores on a depression test. It also includes the metadata: how many times someone opened your app this week, at what hour, for how long.

Information is the why underneath the behaviour. Why is this person ordering food at eleven at night? Because they are stranded at an airport with no other options? Because the power went out and they could not use the cooker? The number alone does not tell you.

Knowledge is the comparative question. How effective is this intervention? How does it compare to a drug? Who is the ideal user for this product?

Wisdom is the rung almost nobody reaches. Wisdom is realising that your engagement figures look the way they do because the "I agree" button sits in the bottom right of the screen, which is exactly where a right-handed person's dominant thumb rests, and your users are simply tapping through your app as fast as they can.

Most founders spend their entire lives between the first two rungs. Let's gather lots of data. Let's see how much we have. Let's look at the averages. Let's make some pie charts, which are the least useful way of presenting data ever devised.

And there it stays.

Founders have been told for years by investors that data is valuable.

We reject the premise. 

Data is not valuable. 

There is far too much of it. 

People say data is the new oil, and to the extent that is true at all… it is true in ways nobody intends. Oil is expensive to extract. Raw crude is messy, dirty, toxic and poisonous. You must be extremely careful how you handle it. And if you have a leak, it is a disaster.

Raw data behaves the same way. Bear in mind that a meaningful proportion of what enters your system is not coming from people at all. It is bots. It could be a competitor. It could be an adversary. As we have covered in recent issues, it could be a journalist or a nonprofit deliberately red-teaming your product with fabricated inputs. You cannot assume any of it is valid.

What people will actually pay for is knowledge and wisdom.

You can turn oil into an electric toothbrush. But it takes a ton of clever people and a great many processes to do it. You have to extract it safely, clean it, refine it, run it through a series of machines and make it pure.

When we do this properly it is usually a two-hour session with the people who actually hold the decisions: the CEO, the research lead, the commercial lead.

We open with a version of the miracle question. It is Monday morning, you have woken up, and a miracle has occurred. You have perfect data. Who are you going to show it to, and how do they react?

The researcher says: we publish in a high-impact journal.

The CEO says: I can tell my investors we are about to be very valuable.

The commercial officer says: nobody is going to buy it anyway, because their contracts are not structured so that this finding changes anything for them.

Those three answers need to be in alignment. When they are not, you can end up with a superb paper in Nature and a company that folds. Or a genuinely important finding that does precisely nothing to your deal cycle.

In my experience, those stakeholders have very rarely vomited forth those harsh realities in a safe setting before. They have assumed that if they collect everything everyone asks for, someone will figure it out later. They will hire a data scientist. It will make sense eventually.

So we build the steelman argument for why you need all this data, and in doing so the assumptions fall out.

Then we get concrete. If a thousand people open your app and half never open it again, and half of those never open it a third time, and the question you want to answer is how OCD changes over two years, you do not need a thousand users. 

You need millions. 

Building that funnel out on a whiteboard tells you very quickly whether your question is answerable at all.

Separately from all of the above, there is a category of problem in digital health that has nothing to do with strategy and everything to do with paperwork.

In a lot of contexts you will want to use a validated questionnaire. Patient-reported outcomes, or PROs. And you may not have permission to.

I have had this conversation more times than I would like. A product owner walks me through their build and mentions they collect everything using the XYZ instrument. I ask what that costs them. They say it is free, they found it on the internet. And I have to explain, gently, that this is very often not the case.

This is why data is a shared layer. 

The Proof Point

There is one thing almost nobody does, and it costs nothing.

Call the people whose minds you are trying to change. The payer. The clinical director. The health system lead. And ask them a single question:

When was the last time you saw data that changed your mind?

Then keep going. How many people was it collected on? How did the decision actually get made? How did the change move through your organisation?

You will get surprises. Sometimes the number that sounds impressive turns out not to have been the thing at all. Sometimes it was a quote. A qualitative piece. A vibe passed along by a member of the clinical team, which then moved the whole organisation.

Because the reason we are doing any of this is to change health, and changing health almost always means changing an organisation. We are not adjusting a price or flipping a switch. We are saying that every nurse should stop doing this and start doing that, and everyone's lives will be better.

Easy to say. Considerably more complicated to do.

Start with the decision you need to influence. Then work backwards to the exact evidence required to move it.

Everything else is exhaust.

Next issue: L2 — Analysis. Where we monitor the evidence.

Since half of you are reading this on a sun lounger: here's my summer reading list.

Five books, all fairly nerdy. No apology here.

Eliane is heading to Nashville.

Over a hundred people commented on Eliane's announcement that she was joining ProofStack. She is already adding real value to our behavioural health practice, and she will be at Behavioral Health Tech in Nashville September 22-24.

If you are going, reply here and I will connect you.