The moons of Verthandi

Drag and drop visual A/B testing on React applications, using pure framework-independent JSX code

A/B testing for web applications

In this post we will introduce a technique and set of open source tools to visually perform A/B testing experiments on Web applications using drag and drop on a test application written in vanilla React and without introducing any code dependencies.

A/B testing is a modern product development technique that is widely used today, according to Wikipedia:

A/B testing (also known as bucket testing or split-run testing) is a user experience research methodology. A/B tests consist of a randomized experiment with two variants, A and B. It includes application of statistical hypothesis testing or “two-sample hypothesis testing” as used in the field of statistics. A/B testing is a way to compare two versions of a single variable, typically by testing a subject’s response to variant A against variant B, and determining which of the two variants is more effective.

Wikipedia on A/B testing

While no silver bullet, it is definitely an essential tool to iterate on web development. A number of mature solutions exist in the market, covering a wide range of use-cases, from pure frontend A/B testing to backend, hybrid frontend and backend, etc., and a number of major companies have rolled out their own in-house systems, like Spotify, Booking, and of course Google and many others.

Web development can be fast-paced as its model allows for rapid iteration (unlike mobile development with mandatory slower store approval rollouts) as is therefore very well suited for leveraging A/B testing to the max. Web applications can be deployed several times a day and there are no limits on how many releases can be done or how frequently.

Many solutions come with some caveats, as we are after all modifying the behaviour of our web app, they either require specific frameworks to be used, need the deployment of specific microservices in the critical path, inject bulky Javascript into the frontend code, introduce risks create client-side issues, and a long etc. A common issue is that they require product people to wait for developers to implement the needed changes and roll them out, even if the change is very small. For instance, changing the size of a logo or button for an experiment is a small change that should not require much development effort if any. There are workarounds to allow product profiles to make those change that but they often need hacks like the blanket injection of 3rd-party Javascript in our web app, can cause UI flicker, increased latency, etc.

In other words:

Wouldn’t it be great if we could have a visual way to create A/B tests without developer intervention?

An imaginary web product owner

But we need to keep our frontend developers in mind:

Wouldn’t it be great if such an A/B system did not interfere with my code and required absolutely no dependencies?

An imaginary sensible frontend developer

All this reminded me of my days working in the fast-paced online news industry, the problems we faced there and how they were solved, so I decided to give this a try and created a system to attack this problem directly.

Desirable properties of our visual A/B testing system

Let’s try to come up with a checklist of desirable properties we would like our web A/B testing system to have:

• Independent of the A/B testing engine – have complete independence on the engine used, namely the system used to split users between variants, the types of tests done (A/A, A/B, multiple variants, etc.), and so on.
• Server-side, client-side and hybrid client-server A/B testing – we should be able to do all three types of A/B tests using the same approach.
• Compatible with modern engineering practices – allow automated testing, version control CI/CD integration, gitops and any other sensible software development, testing, deployment and operation methodologies. Allow for seamless devops operation in times of duress and out of hours incidents.
• No client or server code injection of any kind – no injection of extraneous Javascript in the frontend, or extra microservices in the backend (besides any code mandated by the A/B engine itself to do the split), no added performance issues, UI flicker, library includes, extra latency or other side-effects. No code added in the critical path means no extra bugs are introduced.
• Code in any way you want – impose no artificial restrictions in the frontend code, use any architecture or frameworks you like. Use React, Vue.js, Angular, etc. JSX? TSX? Redux? It should not be a problem.
• Allow visual operation by non-engineers – allow non-developer product and business users to visually create, manage and delete A/B tests in production, having little or no knowledge on the technical internals.

This is an extensive list of sometimes conflicting requirements but let’s see how well we can strike a balance.

A/B for React applications in JSX

Enter Morfeu, an OSS web application to visually manage complex APIs. Morfeu is a generic system designed to manage APIs in YAML, JSON and XML that can be extended to handle other formats. Snow Package is a microservice that adds JSX support so we can use it in application using JSX to handle the frontend, for instance web apps written in React.

JSX is a Javascript syntax extension that is often used to describe the interface of React applications, and as a structured language, it can be parsed and generated to help with specialised use-cases like A/B testing.

Let’s see a short demo of it in action:

How it works

The following diagram illustrates the process:

Morfeu needs an Abstract Syntax Tree to represent the different elements in the interface. The Snow Package application uses the Babel parser to read the original JSX and turn the original code into the appropriate AST. A schema of the possible structure of pages is also needed, written in a subset of XML Schema with a bit of extra metadata thrown in. We call this schema the model.

All this package is read by Morfeu and presented in a ‘simplified visual interface’ to users, which is a logical representation of the underlying site. This interface hides a lot of the complexity which helps non-technical users into focusing on what is really important. Morfeu users can then freely modify that site structure within the constrains of the defined model (it is not a free-for-all as we will see below) and see in realtime what is going on with a direct feedback loop wherever needed.

Typically, the page in question is edited to add content, modify its configuration, delete unwanted stuff and so on, but in this post we are particularly interested in adding A/B tests in the page. A/B experimentation is also a particularly good example of meaningful page manipulation so it is perfect to illustrate the overall concept.

This is the original site we are editing (apologies for the barebones CSS ^^):

Which is presented in simplified form in the Morfeu UI like thus:

The content central part is in generated from the page vanilla JSX code, robustly parsed using a Babel traverse function (docs). The JSX source in question:

const _root = <>
<Header>
  <Menu callCopy="Please enjoy this demo site!" logoCopy="Welcome to snow package!" logoSize="XL"/>
  <Search searchButtonCopy="Search!" searchExamplesCopy="Cars, Android, phones..." startCategory="Phones"/>
</Header>
<Body>
<Title>Need some inspiration for today?</Title>
<Row>
  <Col size="8">
    <ImgText imgURL="/img/photos/houses.png" text="Nice apartments" textColor="light" textSize="M"/>
    <ImgText imgURL="/img/photos/clothes.png" text="Clothes" textColor="white" textSize="XL"/>
  </Col>
  <Col size="4">
    <ImgText imgURL="/img/photos/misc.png" text="Bargains found!" textColor="primary" textSize="XL"/>
    <ImgText imgURL="/img/photos/cars.png" text="Cars!" textColor="dark" textSize="XL"/>
    <ImgText imgURL="/img/photos/phones.png" text="Handsets here!" textColor="primary" textSize="XL"/>
  </Col>
</Row>
</Body>
<Footer>
<Copyright legalCopy="(c) 2020 Snow package test site">
  <ExtraLink link="https://github.com/" text="Github"/>
  <ExtraLink link="https://foo.com" text="foo.com"/>
</Copyright>
</Footer>
</>;

This diagram shows the different elements of the interface:

As shown in the UI screenshot and described in the diagram, on the right hand side we see the model generated from the XML Schema and extra metadata, where we see we can add experiments to the images in the middle section (called IMGTEXT in our model, and nested inside a ROW / COL structure):

We want to create an A/B test where we either show the car photo (variant A) or the phone handset photo (variant B). We drag the A/B experiment element to the page content and move the two photos inside as variant A and B:

We hit ‘save’ and that’s it. Morfeu sends the new AST structure that now includes the experiment, Snow Package converts back it to JSX and is finally serialised to disk. This is the new column structure in the code as a result of making this change (the rest of the file is unmodified):

<Col size="4">
  <ImgText imgURL="/img/photos/misc.png" text="Bargains found!" textColor="primary" textSize="XL"/>
  <Experiment experimentID={1234}>
    <ImgText imgURL="/img/photos/cars.png" text="Cars!" textColor="dark" textSize="XL"/>
    <ImgText imgURL="/img/photos/phones.png" text="Handsets here!" textColor="primary" textSize="XL"/>
  </Experiment>
</Col>

The Experiment element with ID 1234 has been added, and that includes two ImgText React Component instances underneath it with their specific attributes. As expected, the test site has been updated, is executing the Experiment React Component and presents either the car or the handset image variants:

That is all that is needed to create and deploy an A/B testing experiment with the proposed setup. The experiment needs to run its course now, its results analysed, and so forth. Once the experiment is concluded the same user can drag and drop the selected variant out and remove the rest, propose a new test, etc.

Let us look at the code of the different elements, starting with the Experiment React Component (source):

export function Experiment(props) {

  const children = props.children ? props.children : [];
  const variant = Math.random() < 0.5

  // if we only have one child we do an A|A test
  const A = children.length > 0 ? children[0] : '';
  const B = children.length>1 ? children[1] : children[0];
  if (variant) {
    return B;
  }

  return A;

}

This is a trivial non-persistent implementation that randomly selects the variant and shows it to the user and therefore not that useful as an A/B testing engine, but we could easily drop in the react-ab-test React component, our own in-house implementation or a commercial component. As long as we define the right schema model (for instance, in the case of react-ab-test the experiment ID is stored in the attribute name instead of experimentID) we are good to go, the Morfeu setup does not care about that.

What about the rest of the React components? They are just as vanilla, take ImgText for example:

export class ImgText extends React.Component {

constructor(props) {

  super(props);

  this.text = props.text;
  const textClass = 'card-title text-'+(props.textColor ? props.textColor : 'dark');
  switch (props.textSize) {
    case 'S': this.finalText = <h5 className={textClass}>{this.text}</h5>; break;
    case 'L': this.finalText = <h4 className={textClass}>{this.text}</h4>; break;
    case 'XL': this.finalText = <h2 className={textClass}>{this.text}</h2>; break;
    default: this.finalText = <h3 className={textClass}>{this.text}</h3>;
  }
  this.imgURL = props.imgURL;

}

render() {
  return <div className="card text-{this.textColor}">
           <img className="card-img img-white" 
                src={this.imgURL} 
                alt={this.text}
                style={{filter: 'blur(1px)', }}/>
           <div className="card-img-overlay">{this.finalText}</div>
         </div>;
}

}

Please do not mind any bad style or other snafus as this is my first React application ever ^^, but it should work to showcase the concept, it is a plain vanilla React Component that takes the props defined in the model and renders them as you would expect.

What about the model schema that needs to be defined for all this to work? It basically defines the possible structure of the page in question and the attributes we want to expose to our Morfeu users (complete source). This is the model for the ImgText React Component:

<xs:complexType name="imgText">
  <xs:annotation>
    <xs:appinfo>
      <mf:metadata>
        <mf:desc>Static image with a title</mf:desc>
        <mf:thumb>/proxy/site/snowpackage/img/imgtext-thumb.svg</mf:thumb>
        <mf:cell-presentation type="IFRAME">http://localhost:3010/#/preview/ImgText?$_ATTRIBUTES</mf:cell-presentation>
        <mf:category categ="Content" />
        <mf:category attr="@text" categ="Content" />
        <mf:category attr="@textSize" categ="Content" />
        <mf:category attr="@textColor" categ="Content" />
        <mf:category attr="@imgURL" categ="Content" />
        <mf:default-value name="@textSize">M</mf:default-value>
        <mf:default-value name="@imgURL">/img/IMAGE GOES HERE.png</mf:default-value>
        <mf:default-value name="@textColor">black</mf:default-value>
      </mf:metadata>
    </xs:appinfo>
  </xs:annotation>
  <xs:attribute name="text" type="textField" use="required"/>
  <xs:attribute name="textSize" type="sizeList" />
  <xs:attribute name="textColor" type="colorsList" />
  <xs:attribute name="imgURL" type="imgURLTextField" use="required"/>
</xs:complexType>

When editing ImgText elements in the UI, this is what is presented (not pretty but functional):

The model schema for this component is certainly far less complex than the actual component development itself. It is important to note that only the parts of the application we want to handle visually need to be modelled, the rest of the app like nested components, app logic, message passing, state management, the JS code wrapping the JSX structure, other pages, etc., can be safely ignored.

It is also really important to note that by defining our model we are adding a lot of useful semantics, for instance we define the prop textSize of the ImgText component can have the enumerated values S, M, L, XL (in this particular model, another use-case could have totally different values). What CSS class or style properties correspond to each value is up to the React Component implementation. There are a number of good reasons why we keep it this way:

• Letting non-specialists manipulate low level details like CSS classes, div structures without control or assistance is dangerous and will can interfere with the webapp behaviour, frontend logic, etc., so in Morfeu those details are kept in the React Component code and not exposed in the UI, this approach avoids a common source of bugs and frustration.
• It also lets frontend developers evolve the implementation, style, look and feel, etc. without having to modify the top-level JSX structure code or the model schema itself.
• It establishes a clear, formally-defined contract between the frontend developer and the Morfeu users, this is what you can modify, set A/B experiments on, etc., and that is the high-level semantic level you should be operating on. Morfeu presents this interface and no implementation details or any other aspects of the app, keeping a sane separation of concerns.

Today’s applications are complex, to cater to that complexity, the model schema can specify the following aspects of the page:

• Which components go where, in what order an how many of them (min, max, open ended)
• Which components can be nested inside which other components and which do not, which child component elements are mandatory and which optional
• Re-use component definitions in different contexts (ie. we can have only max two ImgText inside an Experiment component but an unlimited number inside a Row / Col structure)
• Make certain components readonly, so they cannot be modified (ie, all pages should have a header, footer, and so on and those cannot be changed).
• Specify allowed component props, and which are mandatory or optional
• Specify basic types of the props (number, string, boolean, enumeration) and check valid values with a regexp (Morfeu will present those as sensible UX elements and interactively check regexp upon editing), it should be easy to add things like color pickers, calendars or more advanced UX editors to the current UI
• Default values of props, practical for mandatory properties
• Specify prop logical categories which in Morfeu are presented in different property tabs for more readability (for instance, we can use an Advanced tab for technical props we do not want non-advanced users to modify).

An extra important feature to consider when letting users manipulate complex components or properties: in the case of simple components, a placeholder like can be used to signal we have a Title component in place but we have the option of a realtime feedback loop for more complex situations, where we have more options or props values can interact in non-trivial ways. A component like ImgText has quite a few options that may not be easy to work with so Morfeu lets you present the outcome of a specific configuration in an iframe showing users how the component will be rendered (either realistically or with a custom made informative representation). The code to add to the React application to do this realtime preview presentation is trivial:

export function Preview(props) {

  const { component } = useParams();
  const query = useQuery(); 
  let params = {};
  query.forEach((v, k) => params[k] = v);
  params._preview = true;

  let preview;
  switch(component) {
    //[...]
    case 'ImgText':
      preview = new ImgText(params).render();
      break;
    //[...]

  }

  return  preview;

}

function useQuery() {
  return new URLSearchParams(useLocation().search);
}

The above implementation just reuses the very ImgText React Component we defined in the first place and renders it as is, we are not adding any extra logic, and with that we can show in realtime to the Morfeu user how the component will look and behave once applied to the page. This creates an instant feedback loop so we need to be careful with the performance of ImgText as we will get a request every time the Morfeu user makes a change to that component or one of its props. We can always revert to using static files if the component is too slow for a realtime loop or provide a simplified view. As another option, we could inherit from any given component and decorate it with extra information to assist Morfeu users in specially complex configurations, provide helpful tips, etc.

All the data flows are summarised in the following diagram:

Last but not least, it is quite common to find instances of having complex components that act in concert, must together or require specific combinations of props configurations. For that very case Morfeu has the feature of snippets, which are pre-created sets of components with all values and children pre-configured, those can be dragged wherever it is relevant and then modified. This is much more convenient and practical than recreating everything from scratch all the time. The snippets are related to a catalogue and thus listed in a plain JSON file, and the snippets themselves are just fragments of JSX code as you would expect.

As a final note, when implementing approaches like this, we can get carried away and thing we can do effective development without writing code and there are solutions out there that attack this particular problem. This is not what Morfeu aims to solve, it focuses instead of product iteration using pre-built features that both product and development have agreed on and developed together.

So how did we fare? Let’s check our original checklist.

Checklist

Given the described approach, let’s review the checklist of objectives:

• Independent of the A/B testing engine ✅ – As we are editing raw JSX code, we are not depending on any specific A/B testing engine or particular experimentation library.
• Server-side, client-side and hybrid client-server A/B testing ✅ – the resulting React code can run in bundled javascript form completely in the client or be rendered in the server, etc. according to your preferences and configuration.
• Compatible with modern engineering practices ✅ – Morfeu treats the original JSX code as source of truth, therefore any kind of gitops, CI/CD process can be done, from code reviews to continuous deployment, etc. Morfeu is completely independent of that. An example approach would be to have a staging environment where changes are pushed directly by changes in the JSX files (with its own branch) and when product is happy with the result a full CI/CD automated testing and continuous deployment cycle is done, with changes tested and pushed to production. You name it.
• No client or server code injection of any kind ✅ – We are modifying JSX code that has no dependencies introduced by Morfeu, there are no libraries or extra microservices to run in the application critical path that could introduce bugs or latency. Morfeu and Snow Package are completely stateless services that can be run in containers and shut down or restarted completely independently of our React applications. The static model files and preview logic are not needed to run any code and can be stripped out for production, staging, etc. We could use a system like Central Dogma to help automate all aspects of such a pipeline.
• Code in any way you want ✅ – The described approach still allows engineers to develop and code the actual application in any way or structure they want, as well as make changes to the pages managed by Morfeu using their editor and environment of choice. As Morfeu and Snow Package treat the JSX as source of truth, any and all manual changes introduced by developers will be parsed and presented to Morfeu users and vice-versa. It is also important to note that we only model the schema of the pages of the application we want product or business to modify using the UI, the rest of the application and React components do not need a model definition. TSX support should trivial thanks to the use of Babel. For Vue.js it should be even simpler, given Vue uses an HTML-based template syntax that is easy to parse and generate. Angular templating language is also easy to parse and it helps templates can be stored separately from application logic. As we are not imposing any restrictions on the code itself, choice of other development libraries like Redux, Jasmine, etc. is also completely up to the frontend developer team.
• Allow visual operation by non-engineers ✅ – As shown, people without deep knowledge of the technical details can visually operate the system, add and manage A/B experiments without the direct intervention of a developer. The way Morfeu is designed, only the previously-agreed properties exposed by the page component developers are there, technical details, subcomponents, technical props and things like specific CSS classes and all sorts properties can be hidden away in any way we want, like being specified in configuration files or hardcoded in the code and in general kept away from unwanted Morfeu user operation.

Try it yourself

A Docker Compose setup is available to make it really easy to start up:

export DOCKERIP=<your docker ip here>

# clone the repos, all from the same folder, checking out the same version
git clone https://github.com/danigiri/morfeu.git
cd morfeu && git fetch && git -c advice.detachedHead=false checkout v0.8.5 && cd ..
git clone https://github.com/danigiri/snow-package.git
cd snow-package && git fetch && git -c advice.detachedHead=false checkout 0.8.1 && cd ..

# clone the React demo site
git clone https://github.com/danigiri/snowpackage-site.git
cd snowpackage-site && git fetch && git -c advice.detachedHead=false checkout 0.8.1

# start the build and the services (this will take a while), remember DOCKERIP needs to have a value 
docker-compose build --build-arg HOSTNAME=$DOCKERIP && docker-compose up

# on another window, jump into the demo site to make live changes as a developer 
# or see how Morfeu changes the JSX files
docker exec -it snowpackage-site /bin/bash

# morfeu should be at http://DOCKERIP:8980/?config=%2Fproxy%2Fsite%2Fsnowpackage%2Fconfig.json
# demo site should be at http://DOCKERIP:3010

# React demo site code is mounted in a volume that will persist between restarts, 
# you need to manually delete it to start from scratch
docker volume ls | grep site

This is what this snippet does step by step:

• We first specify the IP where our Docker host exposes services (if using Docker Machine, the command docker-machine ip <name> will help), if running on a Linux machine with Docker installed, localhost should be OK
• We next clone the repos and checkout a stable version of each one
• Perform the Docker Compose build and start the different containers (this will take a while), specifying the hostname or IP where services will be exposed as a docker compose build argument, it will also create a persistent volume with the React JSX code in it
• Launch a bash shell process to poke around the React application
• We can see Morfeu in action in http://DOCKERIP:8980/?config=%2Fproxy%2Fsite%2Fsnowpackage%2Fconfig.json and the React site in http://DOCKERIP:3010
• The config parameter lets the frontend know where to pick up the configuration for this specific scenario
• A Docker volume called ‘site’ is created that contains the React JSX files, so they will persist across container restarts, showcasing the stateless nature of Morfeu and Snow Package, the command docker volume will help managing this
• The different Docker files are on each repo and can be used to start the microservices in a different configuration.

Next steps

The basic concept demonstrated by the Morfeu and Snow Package combo is completed. We can edit JSX-based React applications and manage experiments on them without a developer and without interfering with the code, not adding extra latency or any extraneous dependencies. Complex web applications in React or other frameworks can be edited visually to handle configuration, iterate them, make small changes or perform all sorts of A/B test experiments. The described setup is quite flexible so extra features like support for TSX could be added, or even new types of UX elements for props like date or color pickers, etc. Morfeu and Snow Package are available under the Apache 2 OSS license.

Thanks for reading up to this ^^, issues and PRs welcome.