Optimizing Advertisement Campaigns Through MLOps

Author
.
2024

Overview

This case study explores how a major US digital TV streaming company overcame challenges related to the optimization of their advertisement Campaigns. Our client needed a solution to identify optimal bids for serving ad impressions purchased through real-time auctions, occurring at a rate of 3 million requests per second.

Challenge

The solution needed to address several different of challenges:

  • Understanding which model was accountable for each decision.
  • A/B testing and experimentation involved custom code changes, increasing the complexity of the code.
  • Model scoring occasionally experienced delays, leading to reliance on subpar decision-making.
  • Expensive cloud costs running custom Hadoop-based algorithms.
  • Hard to evaluate the business impact of decisions made by various models.
  • One monolithic process launched for training models,  
  • Unexpected changes in underlying data produced sub optimal bid decisions.

Solution

Blend's team developed a solution that automates ML model development for real-time scoring. By leveraging AWS services, Amazon EMR, Amazon Lambda, and Apache Spark, the team capabilities were extended to  support low-latency scoring at the serving layer.

Framework-agnostic MLOps system leveraging MLFlow/ Extension of Apache Spark for low latency scoring/ Tensorflow.

  • Thousands of models trained every day, specific for each Ad Campaign. Different model variants trained in parallel and compete every day. The best model is selected and activated.
  • Complete end-to-end process from raw data to model scoring is part of a CI/CD pipeline with automation, with unit and integration tests.
  • Dashboards with real-time metrics including both ML specific metrics (e.g. AUC) as well as business metrics (e.g CPA, CTR). Alerts for data drift and model drift.
  • Running simultaneous experiments by configuring and scheduling A/B tests in a safe and controlled manner.

Impact

The implementation of these solutions brought about remarkable benefits for the client. There was an impressive 80% reduction in cloud costs. Additionally, bid decisioning now occurs in less than 1 millisecond, ensuring swift and efficient ad serving. The ability to concurrently operate thousands of models through controlled A/B tests has enhanced the company's agility and decision-making capabilities. And lastly, there has been a notable 30% reduction in CPA, signifying improved cost-effectiveness and performance in advertising campaigns.

Key Data Points

  • +80% Cost Reduction in cloud costs  
  • 30% Reduction in CPA