Thanks for reaching out. Be sure to check us out on LinkedIn for all of our current news and announcements.
Oops! Something went wrong while submitting the form.
Part 2 of 5 in a series on Lean Analytics in an AI era. [Start with Part 1 here]
A user signs up for your AI product. They set up their account. Connect their data. Upload a document. Hit run.
Your dashboard lights up green. Activated.
Except the output was mediocre, and they're never coming back.
This is the core problem with AI product metrics. The funnel still completes. The value doesn't always show up. Let’s walk through the first three shifts that break the old playbook.
Shift 1: Time to value collapses.
Traditional SaaS onboarding was a journey. Sign up, take a few steps, eventually see value. Time to value wasn't even a priority for most companies.
Results from 2024 study run by Ben Yoskovitz on TTV.
AI flipped that. People are willing to try, but they're fickle. They know there's hype, so they expect your product to blow them away on the first attempt. If the output quality isn't there, they move on. Fast.
And it's not just about chat interfaces or “prompt - response” interfaces. A user drops in a messy document and expects a polished proposal back. Uploads a spreadsheet and expects clean analysis. Sketches a wireframe and expects a working UI. The input varies. The expectation is constant: fast, high-quality output, first try.
The flip side is that time to competency collapsed too. Non-technical users can now produce expert-level work without the training curve that used to gate them. Your activation curve used to be a learning curve. Now it's an interaction or two.
What to measure: time to first useful output, and the percentage of users who get a useful result on attempt #1, whether that attempt is a prompt, an upload, or a sketch.
Shift 2: Activation isn't deterministic anymore.
In traditional SaaS, activation was a clean event. A + B + C reliably produced D. You could instrument the funnel because the steps guaranteed the outcome.
AI breaks that. A user can complete every step and still get a "meh" result. Your dashboard says activated. They are not.
Activation isn't a binary gate anymore. It's a quality-weighted event.
This matters because everything in product management (and building ventures) is a loop. Nir Eyal's Hooked model (trigger, action, reward, investment) still applies, but with a catch that breaks the math underneath it. In the classic model, the action is deterministic and a reward reliably follows. AI introduces variance on both sides. Users stretch and test the edges of what you built, and the outputs range in quality. Two sources of variance in a single loop is genuinely harder to instrument.
To be clear, this doesn't mean activation has to shrink to a single prompt. Compound, multi-step activation often works better, when the setup actually improves the output. Connecting context, uploading reference material, training the tool on your voice. More setup can mean higher first-run quality. The shift isn't that activation got shorter. It's that finishing the steps no longer guarantees value.
What to measure: keep the old funnel-completion metrics, but pair them with the attempt #1 quality signal. The funnel tells you the user finished. The quality signal tells you whether finishing produced anything worth coming back for. You need both, side by side.
Shift 3: Engagement is directional.
Old wisdom: more time in product is good. Longer sessions, higher DAU, deeper usage. Session length went on every investor deck.
AI changes the question entirely. Stop asking "is engagement up or down?" Start asking "what is the user's time being spent on?"
Time spent struggling (regenerating, re-prompting, tweaking inputs to force a useful output) is bad engagement, every single time. It's a failure dressed up as engagement, and it will flatter your dashboard while your product quietly dies.
Time spent with the AI doing the work (a spreadsheet being restructured, a proposal being generated, a contract being reviewed) is good engagement. Those minutes are AI labor, not user friction.
Time spent exploring or creating is good too. And zero user time with the task completed is the ideal for agent products. The best outcome is invisible.
The same number on your dashboard can mean two opposite things. Your job is to know which one you're looking at.
GitHub Copilot reports the percentage of suggestions accepted as a core metric, sitting around 27-30% industry-wide. That KPI didn't exist in traditional SaaS. It measures "was the AI's work useful?" rather than "did the user stick around?" Huge difference.
Three shifts, one root cause: AI output is probabilistic, not deterministic. Your funnel can't see that on its own.
Which brings the obvious question: if you sliced your engagement metric by "struggle time" versus "AI doing the work," what would it actually tell you? Most dashboards can't answer that yet. Can yours?
