A/B Testing & Content Optimization

Paid pluginsize Mplanned, not built

Indicative price, not an offer: $35–80/mo depending on test volume and traffic tier

Generated from docs/plan/paid/a-b-testing-content-optimization/PLAN.md in the AstroBaaS repository. Nothing described below is implemented — it is the written plan for it.

A/B Testing & Content Optimization enables merchants to run statistical tests on content variants (headlines, CTAs, images) and measure their impact on clicks and conversions with statistical confidence. Merchants create multiple versions of content, route traffic between them, and review results dashboards showing conversion lift, confidence intervals, and recommendations. This is a paid capability owned by a plugin, not core CMS.

The problem

Merchants change headlines or calls-to-action but have no idea whether the change improved conversions. They are testing blind, making decisions based on gut feel instead of data, and leaving conversion on the table.

What it does

• Store multiple content variants per page or component, tagged with experiment metadata
• Route storefront traffic probabilistically across variants (A/B split, weighted multivariate)
• Capture conversion events and attribute them to the variant shown at click time
• Calculate statistical significance (confidence level, p-value, lift confidence interval)
• Display results dashboard showing conversion rates, confidence, and recommendation to winner
• Schedule experiment auto-conclusion when statistical significance threshold is met
• Export results as CSV for external analysis
• Webhook event on experiment conclusion for external reporting systems
• Variant performance audit log showing every assignment and conversion attribution

What it deliberately does NOT do

Each boundary carries its reason. A boundary without a reason gets crossed by the next person who reads this.

• Machine-learning-driven recommendation for optimal variant — only statistical reporting
• Personal visitor tracking across sessions — experiments use session-level bucketing only, per privacy baseline
• Multivariate tests with more than 16 combinations — combinatorial explosion breaks statistical power; users must run sequential tests instead
• Holdout groups (control arm that never sees any variant) — scope is A/B split only; holdout requires separate feature flag system

Data model

New plugin-owned collection: Experiment (id, name, status, variantA_id, variantB_id, trafficSplit, startDate, endDate, conversionsA, conversionsB, status). No core schema migration; variants stored in existing content collections with experiment_id reference. Audit log tracks assignment and conversion.

API

• POST /api/experiments — create experiment
• GET /api/experiments/:id — fetch experiment and current results
• PATCH /api/experiments/:id — pause/resume/conclude
• POST /api/experiments/:id/convert — record conversion event
• GET /api/experiments/:id/results — current statistical summary (confidence, lift, recommendation)

Every route added here must also appear in src/pages/openapi.json.ts — a test fails the build if it does not.

Admin

Experiments list view showing status, active variants, traffic split, sample size, current lift, and confidence. Create experiment form picks content type, variant A/B, traffic split. Results view displays conversion funnel, p-value, confidence interval, auto-conclusion status. Ability to manually conclude or adjust traffic split mid-experiment.

The seam — why this is paid

Core owns: content storage, session API, event webhooks. Paid plugin owns: variant routing logic, statistical calculation (t-test, chi-square), results dashboard, auto-conclusion scheduler.

Paid owns statistical testing logic and multivariate-test infrastructure. A/B testing for content optimization is an advanced capability beyond core CMS; it requires ongoing analytics and support commitment.

Dependencies

• core-content-versioning (variants stored as versions or separate records)
• core-webhook-system (conversion events triggered via webhook)
• core-audit-log (experiment assignment and conversion tracking)

Acceptance checks

Each of these must be able to fail. Before claiming this is done, break the code deliberately and watch each one go red.

• Two variants of same content can be created and assigned different traffic splits (50/50, 70/30)
• Storefront receives variant ID in response, consistent within session cookie
• Conversion event recorded with variant ID is attributed correctly in results
• Results show p-value < 0.05 with 100 conversions per variant (statistical power reached)
• Experiment auto-concludes on schedule if startDate + durationDays reached
• Merchant cannot manually edit a winning variant mid-experiment without stopping the experiment first
• CSV export includes all results data (sample size, lift, confidence interval, p-value) in standard format

Risks

Careless implementation of traffic routing (biased bucketing, session inconsistency) invalidates statistical results. Incorrect p-value calculation leads to false-positive winner declarations. Attribution lag (conversion recorded after variant expires) causes data loss. Insufficient sample-size validation allows conclusion on n<30 per variant, destroying statistical validity.

Commercial context

The anchor is what the nearest equivalent charges on Shopify or Magento today. It is context for a pricing decision, not the decision.

---

Generated from the commerce plan. See docs/COMMERCE-PLAN.md for the full catalogue and ../../AI-GUIDE.md for how to work on this repository.