Maximum Difference Scaling (MaxDiff)

Market researchprovides important information to identify and analyse market demand, marketsize, market trends and competition, and allows companies to assess theviability of a potential product or service before taking it to market. It is afield that recognises the importance of utilising data to make evidence based decisions.Many statistical and analytical methods have become popular in the field ofquantitative market research.

In our MarketResearch terminology blog series, we discuss a number of common terms used inmarket research analysis and explain what they are used for and how they relateto established statistical techniques. Here we discuss "Maximum DifferenceScaling", but check out our other articles on key driver analysis and customer segmentation, and look out fornew articles on CHAID and TURF, amongst others, coming soon.

What is it for?

MaxDiff (MaximumDifference or Best-Worst Scaling) is a survey method in market research thatwas originally developed in the 1990's and is used to try to gain anunderstanding of consumers' likes and dislikes. Respondents are usually askedto select the most and least important attributes from a subset of productfeatures. The question is then repeated a number of times with the list ofattributes varied so that the respondent selects the best and worst featuresfrom a number of subsets of product characteristics. The goal of the researchis to rank the attributes in terms of their importance to customers on a commonscale, so that comparisons and trade-offs between them can be made. See belowfor an example of a MaxDiff question looking at the attributes of a householdappliance.

An example of a MaxDIff survey question.

The method is easyfor respondents to complete and forces them to make a discriminating choiceamongst attributes. There is no opportunity for bias to occur due todifferences in the use of rating scales (which is commonly seen acrossdifferent countries and cultures) such as those that can occur with afive-point, non-comparative scale from "Not important" to"Extremely important", for example. Furthermore only two selectionsneed to be made from each list, making it arguably more manageable/practicalthan the ranking of each item. When there are four attributes in the list, suchas in the example above, we learn about five of the six pairwise comparisonsbetween the items but from just two customer choices; it is only the comparisonbetween the two attributes which are not selected that remains unknown. Forexample, from the response above we know that:

·        Safety is more important than Design/aesthetic

·        Safety is more important than Speed of boiling

·        Safety is more important than Capacity

·        Design/aesthetic is more important than Capacity

·        Speed of boiling is more important than Capacity

·        Design/aesthetic vs. Speed of boiling unknown

What statistical techniques are used?

Firstly, experimentaldesign is required in MaxDiff to construct the lists ofproduct characteristics to be chosen from, to determine the number andcombinations of attributes per question and to determine the number ofquestions that each respondent must complete. These are chosen so as to get thebest balance of attributes within each question, maximising the informationobtained whilst minimizing the burden to the respondents. Ideally, combinationsare chosen so that each item is shown an equal number of times and pairs ofitems appear together an equal number of times. Most often, so-called balanced incomplete block (BIB), orpartially balanced incomplete block (P-BIB) designs are used.Take a look at our case study"Judging at the Big Bang Fair" for anotherexample of the application of experimental design.

A number ofdifferent approaches are used by market researchers to analyse MaxDiff surveyresults.

Counts analysis

A simple,so-called "Countsanalysis" approach involves calculating the differencebetween the numbers of times each item is chosen as best and worst (termed the"count") and then ranking the attributes based on these differences.This can be done at both the individual respondent level and also aggregatedover all respondents. However, this method fails to take the experimentaldesign of the survey into account and, for example, doesn't use the informationobtained when two items appear together in a list to distinguish between thosewith a tied count. Furthermore, if the experimental design was unbalanced, andso some items appeared more often than others, counts analysis will give biasedestimates as items that appear more frequently will have had more opportunitiesto be chosen as best or worst.

'Tricked' logistic regression models

Alternatively,random utility (or discrete choice) models, suchas logistic regression models, arecommonly applied to MaxDiff data. Logistic regression models are designed topredict the probability of a binary dependent variable (e.g., a yes/noresponse) via a linear combination of independent explanatory variables. TheMaxDiff experiment, though it involves discrete choices, clearly does not fitinto this design. A trick however is therefore used to apply the methodology inthis case.

The"trick" involves separating out the responses for each attribute ineach list as a binary outcome (chosen or not chosen) for the dependent variableand then using dummy variables for the independent variables to indicate whichattribute the response corresponds to and whether it was select as best (1) orworst (-1). The coefficients for each attribute from the fitted model are thendirectly compared to give a rank ordering for the attributes in terms ofcustomer preference. They are often also transformed andinterpreted as estimates of the relative probabilities of each item beingchosen as the best. So, for the example above, we might find that thedesign/aesthetic of the kettle has the highest "share of preference"with approximately 40% chance of being selected as most important compared tothe other attributes in the list.

There are a numberof issues with this analysis approach. Most importantly, the assumption ofresponses being independent, which thelogistic regression model relies upon (and in fact almost all statisticaltechniques do), is clearly violated as each choice will be affected by theattributes that were available to select in the current list, and best andworse choices will clearly be correlated. Therefore, the resulting parameterestimates will be biased and cannot be relied upon.

Rank-ordered logistic regression models

A more robustanalysis that can be applied to MaxDiff involves applying a rank-orderedlogistic regression or "exploded logit" model. This allows usto model the partial rankings obtained from the responses to the MaxDiffquestions (see the bullet point list above, for example), whilst accounting forthe ties. This approach does not violate the independence assumption like thetricked logistic regression model above and, as before, allows you to estimatethe rank ordering of the attributes in terms of customer preference or toestimate probabilities of attributes being selected as the best.

Despite thisapproach being more statistically sound, there are still questions over theinterpretability of the results. In particular, we are only assessing therelative importance/desirability of the attributes and so it is crucial tocarefully consider the product features to be included upfront. The resultsalso don't indicate if any of the features are likely to actually impactcustomer behaviour, and furthermore customers'responses (self-stated importance) won't necessarily reflect what they actuallywant.

Alternative approaches

As the MaxDiffbest and worst selections only depend upon the rank ordering of the attributesand their analysis simply provides estimates of the rank ordering ofattributes, it may be simpler to directly ask consumers to rank the attributesin the first place. Although this approach is slightly more intensive forrespondents, it is simply a case of repeatedly asking for the most importantattribute from a decreasingly long list of items. This also simplifies the datacollection process as we no longer need to generate experimental designs.

The rank-orderedlogistic regression models described above are explicitly designed to analysethese sorts of data and allow us to estimate and test for differences amongitems in respondents' preferences for them. It's also simple to incorporatepredictor variables accounting for respondents' or items' characteristics, orboth, that allow us to investigate what characteristics affect the rankings.

Related Articles

·        What's YourPreference? Asking survey respondents about their preferencescreates new scaling decisions (Steve Cohen & Bryan Orme)

·        The MaxDiffKiller: Rank-Ordered Logit Models

·        Why doesn't Rhave a MaxDiff package?

资料来源：http://www.select-statistics.co.uk/article/blog-post/maximum-difference-scaling-(maxdiff)