All directories
SkillMarketingGrowth

A/B Testing Frameworks

Plan statistically sound A/B tests and growth experimentation programs—hypothesis design, sample size, metrics, ICE prioritization, and a compounding experiment playbook.

Open on GitHubBrowse more skills
01

Overview

A/B Testing Frameworks is a growth-experimentation skill from Corey Haines’ Marketing Skills collection. It turns vague “should we test this?” questions into statistically sound experiment designs: hypothesis, metrics, sample size, variants, traffic allocation, and a reusable experiment playbook. Unlike a one-off CRO tip sheet, this skill is built for both single tests and an ongoing experimentation program—ICE prioritization, velocity targets, weekly/monthly cadence, and documenting winners so learnings compound across your funnel.

SaaSE-commerceOther

Work with LimeDock

Using this skill? LimeDock can wire it into a durable automation you own.

Skills show what's possible. LimeDock builds and runs the owned marketing, sales, and ops automations around them.

We sell owned automations for SaaS teams — live workflows that plug into Slack, CRM, and your internal platform — not just a skill list.

Book a workflow call
02

Link

https://github.com/coreyhaines31/marketingskills/tree/main/skills/ab-testing
03

Installation guide

Option A — Install just this skill (recommended)

npx skills add coreyhaines31/marketingskills --skill ab-testing

Option B — Install the full Marketing Skills collection

npx skills add coreyhaines31/marketingskills

That also unlocks related skills like `cro`, `analytics`, `copywriting`, `emails`, and `programmatic-seo`.

Option C — Manual install (Claude Code)

git clone https://github.com/coreyhaines31/marketingskills.git /tmp/marketingskills
mkdir -p ~/.claude/skills
cp -R /tmp/marketingskills/skills/ab-testing ~/.claude/skills/ab-testing

Claude.ai

Customize → Skills → Upload the `ab-testing` folder as a zip / `SKILL.md` → enable the skill.

Always skim `SKILL.md` before first use so you know its triggers and related skills.

04

How to use it

1. **Trigger it with experiment language** — say “A/B test,” “split test,” “hypothesis,” “ICE score,” “experiment backlog,” or “which version is better.” 2. **Share context up front** — baseline conversion rate, traffic volume, page/surface, and the change you’re considering. If you keep a `product-marketing.md` (or `.agents/product-marketing.md`) in the repo, the skill will read it first. 3. **Let it run the assessment** — it will clarify test context, current state, and constraints before designing the experiment. 4. **Lock the methodology** — primary metric, sample size, variants, and traffic split before launch. Do not peek early. 5. **Close the loop** — analyze results, update the experiment playbook, and feed learnings into the next ICE-ranked hypothesis.

Hypothesis format it expects

Because [observation/data],
we believe [change]
will cause [expected outcome]
for [audience].
We'll know this is true when [metrics].

**Related skills in the same collection:** `cro` (test ideas), `analytics` (measurement), `copywriting` (variant copy).

05

Example prompts

  • “Design an A/B test for our pricing page. Baseline plan selection is 4.2%, ~18k unique visitors/week. We want to test a simplified 2-plan layout vs current 3-plan grid.”
  • “Help me write a strong hypothesis and ICE score for testing social proof next to the signup CTA on /start.”
  • “We have low traffic (~2k visits/week on this page). Recommend test type, MDE, and how long to run before calling a winner.”
  • “Turn our last 6 product insights into an experiment backlog with ICE scores and a 30-day launch order.”
  • “We peeked at day 4 and saw a “winner.” Walk me through why that is dangerous and how to restart correctly.”
  • “Draft the experiment playbook entry for a +12% CTA CTR winner so we can reuse the pattern on checkout.”
06

Use cases and examples

  • Pricing page plan-selection A/B with primary, secondary, and guardrail metrics defined
  • CTA copy / button hierarchy experiment on a SaaS homepage
  • Onboarding step variant test (server-side or client-side) with sample-size math
  • Build a 20+ hypothesis experiment backlog scored with ICE
  • Stand up a weekly/bi-weekly/monthly experimentation cadence for a growth team
  • Document a winning test into a reusable growth playbook pattern
  • Decide A/B vs A/B/n vs MVT vs split-URL based on traffic reality
  • Pre-launch checklist: tracking verification, QA on both variants, peeking prevention
07

Prerequisites

  • Claude Code, Claude.ai Skills, or another Agent Skills–compatible host
  • Ability to change the page/surface under test (or a feature-flag / experimentation tool)
  • Baseline metric + rough traffic numbers (even estimates help)
  • Optional: analytics or experiment tool (PostHog, Optimizely, VWO, LaunchDarkly, etc.)
  • Optional: product marketing context file in the repo for brand/audience defaults
08

Tips

  • Test one meaningful variable. Multi-change tests obscure causality.
  • Pre-commit sample size. Stopping early (“peeking”) inflates false positives.
  • Pick one primary metric, supporting secondary metrics, and guardrails that must not tank.
  • If traffic is thin, prefer bolder changes or longer runtimes over tiny color tweaks.
  • Win rate of 20–30% is normal for mature programs—optimize for learning velocity, not vanity wins.
  • Promote winners into a playbook (“social proof near pricing CTAs”) and reuse across funnels.
  • Pair with `cro` for idea generation and `analytics` for instrumentation.
09

Sources