Lean Startup Theory advocates the use of ongoing experimentation to find out if customers value your product or not. The use of A/B testing is often used determine if some feature is having a significant positive influence on customers or not. The popularity of A/B testing derives from the fact that it is easy to change some minor feature of a website to see if some success measure is improved or not relative to the control/existing version of the website. The technique is easy and the math is easy (but check this out).

A/B testing is only one experimental technique that might be used and it has some limitations. One limitation is that it is often used to test only one factor or version at a time to see if the factor/version improves success metrics or not. Optimal performance is often a function of the interaction of two or more factors that can not determined by testing one factor at a time. Chemical reactions can occur optimally at a combination of temperature and pressure that is not predicted by studying each factor separately.

Today I want to mention a methodology that is not that well known but which is used in industry to find the factors and the levels of each factor that produces optimal outcomes. That methodology is called Response Surface Methodology (RSM) and is commonly used in chemical industries to find the optimal operating conditions (temp, pressure, catalytic agents, reactants, pH, etc..) for producing a chemical reaction.

Response Surface Methodology begins by listing all the factors that might contribute to the response. It also examine the levels that each of these factors might take on. When you do this you quickly run into a combinatorial explosion of factor levels to test. Where Response Surface Methodology comes in is to help guide you towards a reduced set of factors/levels to interatively test to arrive at an estimate of the optimal factor level settings.

All I can hope to do in this blog is mention a couple of ideas from response surface methods that I found interesting and am still exploring.

If you have, say, 3 factors (temperature, pressure, pH) with 5 levels then to run a full factorial design requires that you measure responses under each of the possible 125 conditions. This is generally not economically feasible so the question becomes whether you can find the optimal condition by studying some subset of conditions, often called a fractional factorial design. One such design that is popular in RSM is called Central Composite Rotable Design. That is an intimidating phrase for a neat idea. Basically you reduce the number of levels by only testing the extreme levels of each factor along with the median level of each factor and interpolating all the values that would fall in between.

So if you wanted to test the growth of a plant as a function of nitrogen and moisture instead of studying all the levels of each factor you would only test the response for the median levels of each factor and the extreme levels of each factor and try to interpolate what the response would be between these levels. Other fractional designs are possible.

The second idea from RSM that is worth thinking about is to plot the levels of your factors on each axis to generate a response surface that depicts our how the variables interact. The combination of moisture and temperature levels will generate a growth response in the plant that can be plotted as a response surface that an educated eye can read to better grasp what the optimal factor levels might be, or what tradeoff might might be best to make. We are all familiar with drawing and interpreting graphs consisting of curves, but not so familiar with drawing and interpreting surfaces and contours so as to understand the interaction of 2 variables. Being able to visualize the interaction of 2 variables as response surfaces can allow our visual system to process the information more thoroughly than a set of numbers would.

I have found that a good starting point for understanding response surfaces is Khan Academy's tutorials on Multivariate Calculus. You don't have to know calculus to learn some useful multivariate skills from the first few tutorials.

I hope this blog has helped to convince you that there are experimental methodologies besides A/B testing that might be applied to discovering the right combination of factors for your product or service. One at a time testing does not provide any insight into the potential interactions of that factor with other factors. For that you need a factorial design and to administer it efficiently you may want to consider response surface methodologies.

The definitive text on RSM is by George Box and Norman Draper with the title Response Surfaces, Mixtures, and Ridge Analyses (2nd, 2007). Not an easy read but worth scanning for ideas and probably very useful if you want to use RSM to optimize your product or service.