Case Study

The Real Problem Wasn't Acquisition — It Was Activation

Data audit and product analytics for an AI-powered bar exam prep platform

PostHogEdTech / SaaSProduct AnalyticsData Audit

Context

The Client

—AI-powered bar exam prep platform with daily quizzes, practice exams, social features, and gamification
—Freemium model: free trial → monthly or lifetime subscription
—Growing user base with significant paid acquisition spend

The core tension: The team was investing heavily in acquiring users but had no visibility into what happened after signup. Was the product retaining users? Were trial users actually studying?

Problem

What Was Missing

“We have the events. We just don't know what they're telling us.”

—No data quality validation — dev/staging traffic mixed in
—No onboarding funnel analysis
—No engagement segmentation or activity tiers

—No trial-to-paid conversion tracking
—No retention or stickiness metrics
—No feature adoption visibility

Phase 1

The Data Audit

When working with new clients with an existing PostHog implementation, it's always a good idea to start with a comprehensive data audit. I created data audit dashboard that maps every event, every user property, and every data source — surfacing issues that would silently corrupt any analysis built on top.

Data audit insights

~20%

Events from non-production

30+

Custom events mapped

User properties catalogued

Critical finding: Roughly 20% of all events came from staging servers, localhost, and development preview environments. Without filtering, every metric in PostHog was inflated. Internal users were not properly filtered out.

Data audit dashboard in PostHog showing user property coverage, event sources, and host-level traffic analysis

The data audit dashboard — mapping users, events, and data sources before building any analytics

Phase 2

The Product Analytics Dashboard

A 34-chart full-funnel dashboard covering every stage of the user journey — with actionable guidance written into every chart description.

Acquisition

Activation

Engagement

Monetisation

Retention

Support

Consulting standard: Every chart description includes what it measures, what to look for, and what to do if the trend is negative. For example: “If the earned-to-shared ratio stays below 10%, test adding a social proof message to the share prompt.” This is consulting output, not a data catalog.

Finding 1

Two-Thirds of Trial Users Never Engage

The engagement segmentation chart split all trial users by study activity in the last 30 days. The result was stark.

Trial User Engagement (last 30 days)

Inactive (0 questions)66.6%

At-risk (quiet 30d)15.0%

Low (1-9 questions)7.3%

Medium (10-49)6.5%

High (50+)4.7%

413

Inactive trial users

High-engagement users

Active paying subscribers

Key insight: The product doesn't have an acquisition problem — it has an activation problem. 638 accounts created in 90 days, but only 114 (18%) ever answered more than one question. The biggest lever isn't more ad spend; it's getting existing trial users to open the app and start studying.

Finding 2

The Onboarding Funnel Has a Cliff

The onboarding funnel with optional steps revealed where users drop out of the first-session experience.

Account Created

638

Survey Completed (optional)

167 (26%)-74%

Tutorial Completed (optional)

~150-10%

First Quiz Started

189 (30%)--26%

The cliff: Only 30% of new accounts ever start their first quiz. The survey and tutorial steps are optional, and most users skip them — which might mean they never understand the product's value before bouncing. This is the highest-leverage optimization point in the entire funnel.

Finding 3

The Monetisation Gap

The subscription status breakdown told the revenue story at a glance:

Status	Users	Implication
No status / never trialed	5,759	Visitors who never converted to trial
Paused	580	Tried, stopped — re-engagement opportunity
Trialing	129	Currently in free trial
Active (paying)	34	Revenue-generating users

The math: 635 trials started → 542 expired → 34 active subscribers. That's a 5.4% trial-to-paid rate. But 580 users are in “paused” state — a massive re-engagement opportunity that was invisible before the dashboard.

Actions

What We Recommended

1
Fix the first-session experience
70% of users never start a quiz. Test making the first quiz mandatory during onboarding instead of optional
2
Add day-3 and day-7 re-engagement nudges
Week-1 is when habits form or break. Push notifications or emails at these critical moments
3
Target the 580 paused users
These users already know the product. A re-engagement campaign (discount, new features) is cheaper than acquiring new users
4
Instrument the trial expiry banner conversion
The dashboard tracks banner clicks but not the upgrade flow that follows. Close the measurement gap
5
Filter non-production traffic at the source
Configure PostHog to drop staging/localhost events before they enter the pipeline, not after

Impact

What This Engagement Delivered

Before

—Dev/staging traffic inflating every metric by ~20%
—No visibility into user activation or engagement
—No onboarding funnel or drop-off analysis
—No trial-to-paid conversion tracking
—30+ custom events sitting unused in PostHog

After

—Clean data with Internal Users cohort filtering
—66% inactive trial users identified — activation crisis surfaced
—Full onboarding funnel with 70% drop-off point identified
—580 paused users identified as re-engagement opportunity
—34-chart dashboard covering the full user lifecycle

Methodology

Audit First, Build Second

The entire engagement was delivered in three weeks, following a deliberate sequence:

Week 1

Data Audit

17-chart audit dashboard mapping every event, property, and data source. Surfaced ~20% data pollution and created the Internal Users cohort.

Week 2

Paid Acquisition

Connected 4 ad platforms, configured 6 conversion APIs, built the paid acquisition dashboard with cross-platform funnels.

Week 3

Product Analytics

34-chart full-funnel dashboard with engagement segmentation, onboarding analysis, monetisation tracking, and retention metrics.

Why audit first: If we had built the product analytics dashboard without the audit, every chart would have included ~20% noise from dev environments. The engagement segmentation would have shown different numbers. The onboarding funnel would have counted test accounts. Audit first means every insight built on top is trustworthy.

Want results like these?

I help product teams turn their analytics into decisions — not just dashboards. Book a free call and let's talk about your data.

Book a free call