Data & Intelligence July 31, 2018
How to Predict Future Demand to Optimise your Marketing Effort
Many businesses encounter the issue of both over- and underselling and seek innovative ways to deal with that, as any of these scenarios could have a primarily negative outcome for the company. For example, in the recruitment industry you do not want too little applications for a vacancy because this increases the probability that you cannot deliver new employees. On the other hand, in the travel industry it costs money when a type of accommodation is not booked and thus left unrented. Additionally, you do not want to invest marketing budgets in vacancies or accommodations that will most definitely be filled or sold. Luckily, with demand forecasting this can be avoided.
Forecasting demand is of great value for all businesses for many different reasons, but it is especially important for businesses that are subject to one of the following situations:
- Flexible Supply – Flexible Demand: the future supply of the product or service is flexible and the company has some sort influence to modify it.
- Fixed Supply – Flexible Demand: the future supply of the product or service you are selling is fixed for future time periods and the company has rarely any influence in modifying it.
To tackle the problem of over- or underselling, it is possible to develop a demand forecasting system. It’s a system that predicts the expected demand – hires or accommodation bookings – for the future 52 weeks and incorporates the future supply to determine if actions need to be taken. In other words, the forecasting system predicts the demand for a particular product per week, 52 weeks in the future, which is compared with the supply of the product for each individual future week. When there is a gap between supply and demand, the product is flagged and actions can be taken to proactively prevent over- or underselling.
How does a Forecasting System work?
The ability to forecast future demand accurately is mainly due to machine learning techniques that are applied to historical data to build models that nicely capture seasonality and trends for individual products or services. The modelling pipeline runs on a weekly basis and consists of the following components:
1. Collect data
The first step is to collect historical data that is used to train the machine learning and statistical models. Along with the actual ‘transactional’ data, static, temporal and website data is included. It can be of great influence on the prediction accuracy. This additional data includes parameters such as website traffic, product views, product add to carts, weather, week of the year and other characteristics to the forecasting model.
2. Cold entry management
The next step is to clean the data and convert it into complete time series, which is a complete set of data points of a unique product. In other words, data from all current products over the last, say, four years (at least two years to make sure seasonality is detected) is needed. This data is not available for new products and therefore we need to generate this data using a matching algorithm. Moreover, this algorithm compares the new product’s characteristics (e.g. price range, category, location, etc.) with all products in the database and selects the top 10% matching products of which the historical data points are averaged and used as data for the new product. After applying this algorithm to the dataset, it generates a complete and clean dataset which can be used for modelling.
3. Forecast reservations & Model competition
The third step in this process is the start of the modelling. This step consists of different components. First up: conduct a model competition, in which six models are selected, (see figure 2 below) of which one will be selected as the best performing model. Here a holdout set of the last 52 weeks of all products is used to test which model has the best performance (MAE and RMSE). After this first step, the best performing model (including its hyperparameters) is used as the final model to predict the demand for all products for the future 52 weeks. At this point the expected future demand per product is forecast on the most granular level (e.g. product by week).
4. Predict arrival periods
Predicting the arrival period is an additional time component that is not subject to all industries, but only for those that have a specified delivery or consumption date. In the travel industry, for example, you are not necessarily interested in the week an accommodation is booked, but you want to know for which week or weekend the accommodation is available. Bootstrapping is used to convert the forecasted bookings into forecasted arrival periods.
5. Inventory difference
Now the demand for all products is forecast 52 weeks in the future, it’s simple to deduct this from the available supply and flag a product when there is a potential future surplus. Additionally, this step of the pipeline is very flexible and made available to the end user to play around with. There are many business-specific decisions that can be applied in this step. Therefore, testing is the best approach.
Lastly, the system’s output is two-fold: one, it generates a list of flagged/unflagged products, which is typically used to allocate or withdraw marketing budgets from campaigns related to those products; two, it generates the forecast demand that is often used by managers/directors for budgeting and inventory management. These forecasting systems typically batch processes, meaning that they run at night on a daily, weekly or monthly basis, depending on what scope you are predicting.
The demand forecasting system can be monetised in several ways, depending on the type of business and use-case you have.
- Fixed Supply – Flexible Demand: in this scenario, you can monetise this system by smartly allocating your marketing budgets to those products that need it. Simply subtract the fixed supply from the expected demand and select the products that have a large discrepancy between demand and supply. Additionally, stop all advertising campaigns and efforts that are related with the products for which overselling is expected, and invest more in products for which a surplus is expected
- Flexible Supply – Flexible Demand: in this scenario, you can monetise the forecasting system through improving your inventory management strategies. In more detail, this means that the forecasted demand could simply function as input for the purchasing department when pre-orders are made. An additional situation would be to use the forecasted demand for expectation management when new supply is provided. For example, in the recruitment industry it could be used to give an estimate to the company that is looking for new employees whether the vacancies are likely to be filled on time.
These two situations are working intertwined and one scenario does not exclude the other. The forecasting system is best exploited when it is used for all possible scenarios, of which only a few are mentioned above. All future related activities, from hiring new staff to budget allocation and inventory management can be improved using a forecasting system as described in this article.
More sales, less surplus
To summarise: many businesses encounter the issue of both over- and underselling and seek innovative ways to deal with that, because these scenarios have a mostly negative outcome for the company. A machine-learning forecasting system enables companies to get an accurate understanding about future demand, which is compared with the future supply (stock).
Consequently, the products for which a large discrepancy is expected between supply and demand, actions should be taken to proactively to avoid the difference between the two. There are many scenarios in which companies can monetise from a forecasting system, but the low hanging fruit is to automatically shift marketing budgets from products that are expected to be oversold to products that are expected to be undersold. This will result in more sales and less surplus products, which is both beneficial for the company as well as the customers.