A/B tests

Configure A/B-testable types

Document types can be configured to support editor-defined A/B tests. This is done by adding the custom.abTest property to the document type definition. Here's an example:

export const myDocumentType = documentSchema({
  name: 'myDocumentType',
  type: 'document',
  custom: {
    abTest: true,
    // etc...
  },
  // ...
)}

Define the A/B test schema

• A/B tests are separate documents in Sanity, which include:
  • testId: a unique identifier for logging & analyzing the test. Read-only.
  • controlDocument: reference to a document with route
  • variants: array of:
    • allocation: 0-100 percentage of users to show this variant to
    • document: reference to a modified duplication of the control document
• Only documents with custom.abTest: true can be used as control or variants
• Documents with A/B tests include a abTest__meta property, an object with:
  • docType: control or variant
  • If a variant, variantKey: the ID of the variant that is logged to analytics
• The A/B test schema is added in /app/sanity/schemas/documents/index.ts and it takes in the array of all other document types

Query variants for current route

A/B-testable types should wrap their document queries with buildAbTestFragment to fetch existing A/B tests for the current queried document:

const PAGE_ROUTE_QUERY = /* groq */ `
"routeData": *[${DOC_ID_FILTER}][0] {
  ${buildAbTestFragment(`
    ...,
    ${TRANSLATIONS_FRAGMENT}
    body[] {
      ${BLOCKS_BODY_FRAGMENT}
    },
  `)}
}`

buildAbTestFragment works by querying any abTest that includes the current document as its controlDocument and using the routeDataFragment to fetch the data for every variant:

/**
 * Returns the GROQ fragment for fetching the `abTest` property of `MinimalRouteData`
 */
function buildAbTestFragment(routeDataFragment: string) {
  return `
  "abTest": *[_type == "abTest" && ${NON_PRIVATE_FILTER} && controlDocument._ref == ^._id][0]{
    testId,
    variants[] {
      _key,
      allocation,
      _key != "control" => {
        document->{
          ${routeDataFragment}
        },
      },
    },
  },
  ${routeDataFragment}
  `
}

Render a variant and persist the choice on a user-basis

The app/routing/pickAbTest.ts function will pick a variant for the current route if there's any A/B test for it. It will:

• Parse the user's cookies and check if they have a saved variant key that is valid

• If so, return the variant document for that key

• Otherwise, run chooseVariant from @tinloof/js-toolkit

  app/routing/pickAbTest.ts

  const chosenVariant =
    savedVariantKey && allVariantKeys.includes(savedVariantKey)
      ? (variants.find(
          (v) => v._key === savedVariantKey,
        ) as (typeof variants)[0])
      : chooseVariant(abTest.variants)

• Save the chosenVariant's key to user's cookies

• Modify the route data to include the chosen variant's data and simplify the abTest object with just the data we need for analytics:

  app/routing/pickAbTest.ts

  const routeData = {
    ...(chosenVariant.document || data.routeData),
    abTest: {
      testId: abTest.testId,
      chosenVariantKey,
    },
  }

• Return that to routeLoader, which will use this as the final data

  • And skip cache if we have an A/B test
  app/routing/routeLoader.ts

  // ===== 4. PICK A/B TEST IF ANY IS CONFIGURED =====
  const [dataWithAbTest, abTestCookie] = await pickAbTest(data, context)
   
  // Don't cache test variations or preview data
  const skipCache = !!(abTestCookie || previewState !== PreviewState.OFF)
   
  // ===== RETURN FINAL RESULT =====
  return cachedResponse(dataWithAbTest, {
    cdnCacheDuration: skipCache ? 0 : undefined,
    //...
  })

Log the results to Google Analytics

If using Google Analytics, you'll need to set the gaTrackingId in app/config.ts by populating the PUBLIC_GA_TRACKING_ID environment variable in .env.

The app/components/GoogleAnalytics.tsx component loads the scripts and initializes the gtag function. After mounting, it'll log the current variant through a set_variant event, if there's any active test in the current page:

const shouldTrack =
  // Must having a tracking ID
  !!config.gaTrackingId &&
  // And don't track in development
  env.NODE_ENV !== 'development' &&
  // or in the Sanity studio
  !pathname.startsWith('/manage')
 
// LOGGING A/B TEST VARIANTS
const { testId, chosenVariantKey } = props.routeData?.abTest || {}
useEffect(() => {
  if (!testId || !chosenVariantKey || !shouldTrack) return
 
  gtagEvent('set_variant', {
    ab_test_id: testId,
    ab_variant_key: chosenVariantKey,
    // Don't let Analytics treat this as an interaction event that could skew bounce rate calculation
    non_interaction: true,
  })
}, [testId, chosenVariantKey, shouldTrack])

Analyze the results

The parts the toolkit handles

@tinloof/sanity-toolkit exposes a custom component for editing tests more easily and reliably.

When adding a variant, the component will:

• modify controlDocument to add a abTest__meta: { docType: "control" } property
• duplicate controlDocument to use as the variant with abTest__meta: { docType: "variant", variantKey: generateVariantKey() }
• add a reference to this new document to the A/B test document's variants array

When commiting a variant:

• patch controlDocument to remove abTest__meta and include the changed data from the variant
• delete all variant documents
• delete the A/B test document's variants array

It also includes an AbTestDeleteAction document action (opens in a new tab) that deletes variants and removed abTest__meta from the control document before deleting the A/B test document itself.

In the pages browser and in collection indexes, we don't display variant documents by including NON_VARIANT_FILTER:

const NON_VARIANT_FILTER = 'abTest__meta.docType != "variant"'

Finally, @tinloof/js-toolkit exposes a chooseVariant helper that properly distributes the choice of variants according to their accumulation, with statistical significance.