Introduction to Impact Measurement: Focus on aggregated data Use Cases
Marketing is a fundamental aspect of every business that aims to promote a product or a service. However, launching a marketing campaign and hoping for the best is not enough. Companies must measure the impact of their actions to determine their effectiveness and make informed decisions about future triggers. Monitoring the impact of actions helps companies to:
- Allocate resources effectively
- Get an actionable vision of ROI
- Identify areas for future improvements and levers that must be interrupted
- Understand customer behavior, knowing what motivates and what drives them to purchase
- Stay competitive
Impact measurement is therefore a crucial topic as every company is trying to streamline marketing costs and optimize their investments. However, measuring impact is a challenging task, and traditional methods like AB testing or MMM (Marketing Mix Modeling) may not always applicable due to their specific requirements.
Our objective is to provide a methodology that is not only accurate but also time-efficient and robust, allowing businesses to quickly assess the effectiveness of their marketing actions.
As, the developed algorithms will strongly depend on the structure of the data (individual data, aggregated, timeseries, …) to best fit the situation and estimate an impact, a distinction between aggregated data and individual one must be made.
Aggregated data refers to data that is averaged, grouped by geographic areas, time periods, service agencies, or by other dimensions (ex: store sales over a defined period of time). In contrast, individual data are associated to a single element (client metadata for example, etc.) and are used to conduct analyses on a finer level such as customer behavior analyses.
Depending on the type of the action to measure, both the dataset and approach used will be different as displayed in Figure 1.
This first article of a series of two will mainly focus on methods applied on aggregated data to reliably measure the impact of various marketing interventions so that informed decisions can be made about which interventions to implement and optimizing the associated investments. Times-series algorithms and causality-based approaches will be described and thoroughly analyzed.
Stay tuned for the upcoming paper about individual data approaches to have a complete overview of impact measurement approaches.
I. Time-based approaches
The dataset used in this paper consists of sales data from 1115 stores over a period of 942 days. The data includes various types of information, such as the number of visiting customers, applied promotions, holidays, and other relevant factors.
Since there is no specific action taking place in any particular store, a first simulated situation based on an artificial trigger was set on an experimental store to get a sense of ground truth. The idea was to randomly isolate a store to which a specific treatment is applied and keep the other stores out.
Two scenarios (i.e. two events) were manually created on the experimental store, from a specific date:
- Scenario 1: +10% on sales with gaussian noise
- Scenario 2: -7% on sales with gaussian noise
In this experiment, two types of data can be distinguished:
- Covariates: all variables available for the experimental store (number of customers, promotions, holidays, day of the week, month, …)
- Other units: observations of the reference stores (sales of stores not affected by the treatment).
This paper will cover different sets of technics using either covariates, other units, or both in order to find the average treatment effect (ATE) of those +10% and -7% on the sales. The ATE is a very important KPI that provides an average measure of the treatment’s impact on the outcome of interest. It represents the average change that can be attributed to the treatment itself, independent of other factors.
A. Synthetic control group
Synthetic control group methods use data from the other units. These methods are used when A/B testing cannot be implemented. A synthetic unit is assembled based on those not affected by the treatment and a difference is deduced between this synthetic unit and the experimental one (which was affected by the treatment).
The created synthetic unit must be fitted as close as possible to the real unit from the pre-treatment period in order to catch all the effects of side variables from the other units and therefore isolate the effect of our trigger (i.e. the treatment).
- Difference in differences
The simplest — but less effective — way to achieve some kind of control group is the Difference in Differences method (or DiD). The synthetic unit is created by taking the average of the other units.
The difference between the experimental and synthetic units is computed before and after the intervention and the impact is deduced from the difference between the two previous results.
This naïve method gives unprecise approximations of the real impact emphasizing the need for more complex approaches.
2. Synthetic control with SparceSC
A more efficient way to measure a treatment impact exclusively with data from other units is to create a synthetic unit based on the weighted sum of other units.
Microsoft developed a state-of-the-art library to perform this fit: SparseSC. Each other unit is given a weight to make the synthetic unit as close as possible to the real unit in the pre-treatment period (before the intervention).
To conclude, these two approaches are interesting if only data from other units without specific treatment are available. The results depend on how close these units are to the experimental unit in terms of behavior before the treatment. Although an impact (ATE) can be observed, it is not the most accurate.
B. Covariate prediction — forecast of the counterfactual
A different approach can also be considered which involves forecasting the counterfactual scenario. This method entails predicting what the outcome would be if no treatment were applied and then calculating the impact based on the difference between the prediction and the actual outcome.
For the methods discussed in the upcoming sections, the input used to measure the impact of a specific action or treatment no longer relies on data from other units, but rather focuses on covariates.
1. CausalImpact
The CausalImpact method fits a Bayesian Structural Time Series (BSTS) model on the pre-intervention data and forecasts what the response would look like had the intervention not happened.
This method shows an accurate estimation of the average treatment effect (ATE), thanks to a close timeseries prediction. The second graph displays a clear 10% shift in the delta between the prediction of the model and the real sales.
2. Prophet
Prophet is a forecasting model that utilizes an additive approach to predict time series data. It effectively captures non-linear trends along with yearly, weekly, and daily seasonality, while also considering the impact of holidays. This model performs exceptionally well when applied to time series data with pronounced seasonal patterns and a substantial history of multiple seasons.
One of Prophet’s strengths is its ability to handle missing data and accommodate trend shifts, making it robust in such scenarios. Additionally, the model generally handles outliers effectively.
The Prophet model clearly shows its strong estimation capabilities in this seasonal use case.
3. XGBoost
Another method to predict the counterfactual scenario consists of using simple regression algorithms like XGBoost.
This method also provides a correct approximation of the ATE with the covariates.
C. Methods summary
This table summarizes the algorithms mentioned above and their corresponding results for each scenario. If the available data consists of other units or entities, then the synthetic control groups should be used. On the other hand, if the data only includes explanatory variables related to the outcome, covariate prediction is the right method. In case of both availabilities, a hybrid model combines the best of both worlds. It is important to note that this scenario represents an ideal case where simulated events only affect the target variable (outcome) without any direct or indirect impact on other variables such as the number of customers or other related factors.
In real life, measured events and treatments often have complex and interconnected effects across multiple variables requiring a deeper understanding of causal relationships.
II. Causality-based approaches
Causality-based approaches unveil the causal links between variables through graphs and equations.
The complexity of these graphs increases with the number of variables and relationships. They illustrate which variables have a direct or indirect impact on which variables and how changing one variable affects the others.
On the dataset used previously, it appears that the sales generally increase when a promotion is run. But an increase by 30% on average is observed in the number of customers. Having this in mind, some questions can be raised:
- Does the promotion really have a direct impact on sales?
- Is the measured impact only due to the promotion, or to other variables?
- How can we quantify the real impact of each of these variables on sales?
This is where causality-based methods come in. They provide a robust estimation of one variable effects on an outcome independently from the other variables. These methods can determine whether a promotion has a direct or indirect impact on sales and the extent of it. A three-steps approach was developed to tackle this challenge.
1. Build a causal graph
Such a graph can be built using domain knowledge (human expertise) and / or automatic algorithms.
2. Estimate the effects
The idea is to decorrelate the effect of a variable on the outcome from any other side effects. Different methods exist to compute the estimand, like the Backdoor estimand and the Structural Causal Model (SCM).
The estimand refers to the target quantity or the causal effect that is to be estimated. It represents the specific parameter of interest that captures the causal relationship between a treatment or intervention and an outcome variable. In our case, the estimand is the average treatment effect (ATE).
3. Check the assumptions
The robustness must be checked by simulating a change in the variables, replacing the treatment variables and selecting random samples / subset of the dataset.
Applying these different methods on a selection of stores from the Rossman Store Sales dataset, gives the following impacts for the promotion variable on sales:
This analysis provides valuable insights into the influence of the promotion on the store’s performance. The impact is highlighted thanks to both approaches and the values are close to the expected target. The DoWhy (backdoor estimate) provides the strongest estimation with the lowest uncertainty.
To be noticed that the expected target value is computed using non-causal time-based approach, as a benchmark in this context. It is not applicable when the data is non-temporal and fails to consider the interconnections between variables.
Conclusion
In conclusion, measuring the impact of marketing actions is crucial for businesses to understand the effectiveness of their campaigns and make informed decisions about future strategies. Focusing on aggregated data, this paper explores various methods including time series approaches, synthetic control groups, and counterfactual forecasting. It also highlights the importance of considering the structure of data, to determine the most appropriate approach.
When only data from other comparison units is available, the synthetic control group using the SparceSC library proves to be the most effective algorithm. For temporal data with explanatory variables, covariate predictions with Prophet offer a reliable way of measuring the impact of an action. Furthermore, in the case of non-temporal data, causal inference algorithms, specifically the Backdoor Estimate from Dowhy is advisable.
Overall, this paper provides a comprehensive introduction to impact measurement in marketing, offering valuable insights for businesses seeking to optimize their marketing investments and remain competitive. A second article will follow, focusing on individual data (at client level for example) to complement the series of methods already developed.
