A/B Test Peeking Problem
Checking a test daily and stopping when the p-value dips under 0.05 produces far more false winners than the number suggests; here is what to do instead.
Field notes on GA4, Google Tag Manager, experimentation, and measurement strategy — strategy, team rituals, and lived stories from building and learning in public.
Article catalogue
Filter by level to find what your team needs now — from beginner foundations to advanced strategy, every piece is ready to share.
Checking a test daily and stopping when the p-value dips under 0.05 produces far more false winners than the number suggests; here is what to do instead.
Basic and Advanced Consent Mode differ in what loads before a visitor chooses; that is a measurement trade-off and a legal decision at the same time.
CUPED uses what each user did before the test to cancel out noise, so a decision needs less traffic; here is how it works and where it stops helping.
GA4 now gives AI assistants their own channel; here is how to read it without mistaking a referrer-based label for the full picture.
Google's Analytics MCP server lets an AI agent query GA4 directly; read-only access protects your configuration, not your conclusions.
Per-conversion attribution settings and custom lookback windows change what a conversion number means, so document them or watch two reports disagree.
Google is turning every Google tag into a full Tag Manager container and adding no-code Visual Tagging; here is what to re-check and what to keep doing by hand.
Data Studio's Show reasoning panel tells you how the AI read your question; it does not tell you the answer is right.
A winning short test may only have measured curiosity; a small holdout and a planned re-check tell you whether the lift lasts.
If the split you got is not the split you planned, the experiment is broken before you read a single metric; here is the five-minute check.
Moving variant assignment to the server removes the flicker, but it moves the hard problems to where you randomize, how analytics learns the variant, and caching.
A tracking plan only works if it is short enough to maintain; here is the smallest version that still prevents the usual naming and ownership mess.
Conversational Analytics doesn't remove the workflow from writing SQL against GA4's export — it just moves the judgment from writing the query to verifying someone else's.
Most consent banners handle a reject click just fine and silently ignore the one signal now legally mandatory to honor in 12 US states — this is what it actually takes to fix that.
Server-side GTM isn't a free upgrade — this is what it actually changes, what it costs in ongoing infrastructure ownership, and how to tell if your team is ready for that trade.
If someone forwarded you an article saying third-party cookies are finally gone, here's the plain-language version of what's actually true in 2026.
GA4's AI insights are confident by design — the job left to you is deciding which ones are also correct.
A trustworthy readout only needs four things in it — this is the beginner path to building one in Data Studio (formerly Looker Studio) without copy-pasting GA4 numbers into a spreadsheet every week.
Most GA4 questions don't need BigQuery — this is the workflow I use to tell which ones do, and how to answer them fast once they do.
The readout meeting is live storytelling. The summary is what travels after it. This article is about the written artifact — the one-page document that lands in a stakeholder's inbox, gets forwarded to someone who was not in the room, and needs to work without a presenter explaining the charts.
Help new teammates learn which dashboards to open, which metrics to trust, and when to ask for help.
Decide when to add tools, when to go deeper with what you have, and how to align leadership.
Frame results with context, contrasts, and next steps so partners stay engaged after the share-out.
Replace vague test ideas with hypotheses that name the audience, change, and outcome.
The philosophy of experiment prioritisation is well-covered elsewhere. This article focuses on the mechanics: how each column in the backlog template forces a specific decision, what anti-pattern it prevents, and what failure looks like when it is missing.
Give stakeholders a five-minute read on product health, experiment progress, and data quality.
Use a structured retro to turn experiment results into better hypotheses and smoother delivery.
Run a focused workshop that connects executive goals to product behaviors and experiment priorities.
Track experiment throughput, cycle time, and decision quality so the team keeps learning every sprint.
Consent banner testing has a built-in measurement paradox: the analytics tool you rely on to evaluate your experiment only fires after the user makes the decision you are trying to optimise. This playbook shows how to design, instrument, and analyse consent tests despite that constraint.
Use a launch-week checklist to confirm tracking, consent, and dashboards are ready before you ship.
Build a simple scorecard that grades event coverage, naming, and reliability so product teams trust every metric in reviews.
Design a GA4 Exploration workspace that answers the same handful of product questions every week so you can compare experiments without rebuilding charts.
Connect GA4 and your consent banner in three practical steps so you only capture the data you actually need.
Create a clear list of experiment ideas and score them so your team always knows the next test to run.