How to Measure A/B Tests with Plain Analytics Events

You do not need an experimentation platform to run an honest A/B test. You need three things: random assignment, consistent exposure, and conversion counted per variant. Plain analytics events cover the second and third; ten lines of JavaScript cover the first. What no tool can give you — and where most homegrown tests actually fail — is patience with sample sizes. We will cover that too.

Assignment: random once, sticky thereafter

function getVariant(test, variants = ['a', 'b']) {
  const key = 'exp_' + test;
  let v = localStorage.getItem(key);
  if (!v) {
    v = variants[Math.floor(Math.random() * variants.length)];
    localStorage.setItem(key, v);
  }
  return v;
}

const variant = getVariant('pricing_headline');
// ...render accordingly, then record exposure:
window.webanalytics?.track('experiment_viewed', {
  test: 'pricing_headline', variant,
});

localStorage keeps returning visitors in their lane. Fire the exposure event only when the variant is actually seen — assignment without exposure dilutes the test.

Conversion: tag the existing event

window.webanalytics?.track('signup_completed', {
  exp_pricing_headline: getVariant('pricing_headline'),
});

Adding the variant as a property on your normal conversion event (rather than inventing a parallel event) keeps the funnel intact and makes the readout one comparison: conversion rate of exposed-A vs exposed-B.

The honesty rules

1. Size the test before starting. Detecting a 20% relative lift on a 3% baseline needs roughly 7,000+ visitors per variant. On 200 visits/week, that is months — better to test bigger swings (which need smaller samples) or not test at all and just ship.
2. No peeking-and-stopping. Checking daily and stopping at the first 'significant' flicker is how teams ship noise. Fix the duration in advance; read once.
3. One primary metric. Declare it before launch. If you check six metrics, one will look significant by chance and you will believe it.
4. Mind the variant's speed. A heavier variant B can lose on load time, not message. Clycyo records per-visit load times, so you can check whether the design lost or the kilobytes did.

When to graduate to a real platform

Concurrent tests interacting, server-side assignment needs, or an experimentation culture shipping weekly — that is platform territory, as noted in the PostHog migration guide. For the occasional headline, pricing, or layout test that most startups actually run, events plus discipline produce the same decisions for free — with landing-page variants as the natural first experiment.