Here’s the uncomfortable truth: in one week you’ve already lost most of them. I don’t know who you are, but I can tell you that your weekly retention curve looks like this:

In Mixpanel’s 2024 Benchmarks Report, they reported that the average one-week retention rate was 28%.3 And it’s getting worse; the amount varied by vertical but every industry had a year-over-year decrease in retention. This tracks with Li’s finding that the first day is the modal point for conversions by an order of magnitude. By the end of the trial, it’s too late for most users.

We Need to Go Deeper…

The single-biggest hurdle in a free trial is about time, but it’s not the duration of the trial: it’s the time to the user’s activation. “Activation” is product analytics jargon for the moment a user gets to the moment of value – the “aha” moment that demonstrates why they need what you’re selling. Activation is a matter of minutes, hours, maybe days, but not almost certainly not weeks. Before you worry about habits, cannibalizing your paid business, or any of that, you have to go a layer deeper than retention.

The starting point, if you haven’t already, is to define and measure your funnel. The funnel is like a zoomed-in version of the retention curve. The retention curve measures the funnel’s global, end-to-end transition percent:

Your global problem is the aggregate of local problems, and the funnel captures local transitions between the four stages.4 Breaking data out in this way provides metrics to optimize, but I think the most productive use of the funnel is actually as a diagnostic tool. Defining and measuring a funnel requires you to be clear about things like:

• What’s our product’s moment of value? If you have to distill the new user experience into one “aha” moment that leads to conversion, what would it be?5 How is it measured? Your product has a many meaningful interactions, but trying to make sense of them all at once is overwhelming and rarely yields actionable insight. Start with focus.

• What steps are absolutely necessary to get to that moment? As products grow in complexity, their setup processes do too. The funnel provides a lens through which you can focus on what’s most essential, and can give you the confidence you need to defer (or cut!) the rest of the setup process.

• How often do you actually expect users to come back? You’ll probably be measuring Daily Active Users, and you should be, but there’s a good chance you don’t expect most users to return every day. Do you expect them to be there within three days? A week? A month?

If you’re looking for a framework to help you get started, I’ve collected some pointers and first-level diagnostics in the Funnel Diagnostic Worksheet. The worksheet is for subscribers only, but I’ve got great news: subscriptions are free!

Two Common Diagnostic Questions

There are loads of diagnostic approaches to take with a funnel, but there are two high-impact questions to start with:

How many steps are in setup? A Userpilot study found that only 81% of users don’t complete their onboarding checklists.6 Remember: your goal is to get them to “aha”. Cut ruthlessly.

What transition has the lowest conversion percent? This is where you have to roll up your sleeves and dig into the data. There are a few questions to ask your data:

• Have we seen any changes in conversion percent in the last 30 days? 90 days? Year?

• What groups of users are converting well? This can give an enlightening view into your product’s personas, and a pointer for product or UX research.

• When they’re not converting, what are they doing instead? Grab your favorite analytics software (or analytics person) and wade into the data marshes. Sam Zaiss wrote a great piece on this. To quote Sam, the common challenges to address at this step are:

• The most insightful user paths aren’t the most common, so we need another way to identify that signal amidst all the noise.

• The most meaningful events aren’t tracked, because they sit on the seams between features or product teams.

• The most common visualizations aren’t designed to let the key insights pop, making them harder to discover.

Sam Zaiss, Deliberate Data Culture

When you view the funnel as a diagnostic tool, it can shine a light on the best places to put your data-savvy folks to work. As you take action on problems, the local transition rates give you a responsive and measurable indicator of progress.

Ready to Dig In?

If you’ve been through the basics and aren’t sure how to go deeper, I’d love to help. Full Rank is a data science consultancy that works with small teams that don’t have full-time data science support. If you complete the Funnel Diagnostic Worksheet, reach out for a free 30-minute consultation.

Last week, I read that a company in Finland is using AI to find the “perfect coffee blend”. And here I am buying an imperfect blend of beans from a local coffee roaster like a sucker.

There’s little question that AI is everywhere. I have a wide network of product managers and data scientists, and the vast majority of them are working on an AI integration or product of some sort. Companies talk about AI with enthusiasm – who doesn’t want “perfect” coffee? The market’s roaring enthusiasm for AI technology doesn’t transfer to individuals, though. Gallup’s 2024 survey found that only 13% of Americans believe AI does more good than harm. On top of that, 77% don’t trust businesses to use AI responsibly.

I’m a data scientist in Seattle, so I wondered: how are local companies approaching the AI trend? Is it really everywhere in the startup bubble, or am I in an even smaller AI bubble? Is the hype out of control? Are they publicly committed to responsibility? (Spoiler: You’re not going to like all the answers). I took a deep dive into the public-facing content of some of the fastest-growing startups in the Pacific Northwest to analyze their AI-related language.

Data and Methods

The Geekwire 200 is a ranked list of 200 fast-growing startups in the Pacific Northwest. I built a crawler to scan the public-facing websites of the companies on that list, and extract their text. After filtering out sites that prohibit automated crawling, I was left with 187 startups. I pulled down the first two “layers” of content from their websites (the homepage, and all pages linked from it), and extracted user-facing text to analyze common phrases.1

After an initial analysis of frequent phrases, I built three lists:

• Markers for AI. This includes things like “AI”, “artificial intelligence”, and “Generative Models”.

• Markers for hyperbole. This includes things like “visionary”, “bleeding-edge”, “revolutionary”, and “perfect”.2

• Markers for AI responsibility. Things like “Responsible AI”, “bias mitigation”, and “AI ethics”.

The way these three lists overlap (or don’t) is revealing. Onto the findings!

All Aboard the B2B AI Train

The first major difference is clear: 73% of B2B companies are pitching AI, but less than half of consumer and R&D startups are.3 B2B startups have focused these pitches on efficiency. When they talk about AI they use words like “faster” and “maximize engagement”. Consumer companies focus on terms like “conversation” and “personalization”, emphasizing the interactivity enabled by Generative AI.

Transformational. Revolutionary. Super Stoked.

I don’t go to company websites expecting humility, but the language associated with AI often goes a few steps beyond optimism. I looked at the overlap of hyperbolic language and text about AI. Pre-market R&D companies use the most of it ( and that might be where it’s most warranted)4, but this language isn’t reserved for the lab. B2B and B2C companies alike are on the “cutting-edge”, building “revolutionary” tech. Speaking just for myself, this language almost always rings hollow, whether it’s about a spreadsheet or the “perfect” AI-generated coffee blend. 🥱

Who’s Taking Responsible AI Seriously?

Gallup’s findings revealed that people don’t have high hopes that companies will use AI responsibly. But there’s a light at the end of the anxiety-tunnel: 57% of respondents reported that their concerns would be reduced if businesses were transparent about how AI is used. I looked at how companies discuss their commitments and actions to responsible AI.

The current state of the world won’t inspire hope in those 57% of Americans. Only 19% of the companies talking about AI shared anything at all about their commitments to responsible AI, model evaluation, or bias mitigation. I didn’t expect a majority, but I was a little taken aback. There wasn’t a high bar for inclusion here: merely stating “StartupCo believes in responsible AI” would’ve counted. When the vast majority of companies aren’t even paying lip-service to responsibility, is it any wonder that people don’t trust them?

It’s Not All Bleak: Bright Spots and Actions

It doesn’t paint a rosy picture, but there are bright spots. Some PNW startups doing this well. Responsive AI’s Committment to Ethical AI and Humanly’s Ethical AI Manifesto outline the design principles and frameworks used to develop their product. Textio’s Building Responsibly page describes some of the procedural and statistical methods used to evaluate models and mitigate bias.5 If you’re a team building an AI product and you’re not sure where to start with responsible AI, you can look to your peers working on this already.

If you’re ready to roll up your sleeves, Data Statements provide a clear place to start. This form of documentation was originally proposed by Emily M. Bender and Batya Friedman in a 2018 paper at UW. As Gallup reported, transparency is a key first step to winning consumer trust. But documentation about the provenance, characteristics, and handlers of the data provide benefits that go beyond building customer confidence. Every data set is biased, whether you’ve quantified it or not. Data statements like these also provide the builders with a framework for identifying gaps and risks. They’re fertile soil for growing your responsible AI efforts.

Faced with overstocked aisles of AI products, consumers will vote with their wallets. When the hype fades, unease about responsible AI may remain. Why should they trust your company? Have you earned that trust? What do you owe to your customers and neighbors, and how can you get started today?

What About Your Product?

If you haven’t started rigorously evaluating your models, now’s the best time to start. If you could use help measuring and mitigating bias in your AI products, I’ve done it before, and I’d love to work with your team. Reach out and let’s talk about how we can work together to understand the performance of your AI features, identify and address biases, and give you (and your customers) peace of mind.