Using machine learning and artificial intelligence to create meaningful engagement

A home internet experiment at Telstra, 2019

My squad and I recognised the value proposition for the new internet plans had evolved since its launch, and for that reason, we engaged analytics experts to identify our winning driver.

Unlimited data with $90 price point (Variation B) was the most successful, with the conversion rate of 0.42% over 51 days

Techniques and tools

• Customer journey map
  Miro
• Online whiteboard
  Miro
• Adobe Audience Manager
  Audience profiles
• Machine leaning and artificial intelligence
  Adobe Auto-Target powered by Adobe Sensei
• Optimisation engine
  Adobe Experience Cloud
• Analysis and report
  Adobe Analytics

Outcome

• The machine learning models performed better
  A lift of 22.66% in banner click-throughs from targeted experiences compared to the randomly delivered experiences of the control group
• The models resonated with our customers
  35% more visits to the internet plans page compared to the BAU banner
• A reduction in acquisition costs
  The model converted 0.39% of visits whereas the control was only 0.32% — a difference of 1,565 click-throughs
• The $90 price point value proposition drove the best business impact, increased revenue
  0.42% better than the overall targeted conversion

Tacit and explicit knowledge unite

Opportunity

Our tacit knowledge of the value proposition for internet plans matured over months of regular customer testing, customer verbatim, web analytics, and call centres’ customer verbatim and data.

This experiment was an opportunity to combine this tacit knowledge with machine learning to inform our models.

The end-to-end customer journey identifies touchpoints from the awareness phase to explore, and purchase

A learning machine puts customer first

Automation

The models use algorithms to automatically identify relationships between customer attributes and learn the best experiences to display based on that data, without having to be explicitly programmed.

The solution: When Adobe Auto-Target starts the experiment, the machine learns and identifies the customer attributes in their segment, and then displays one of the four variations with our best performing value proposition

Personalisation and A/B testing in machine learning experiments are also automated. Using our in-depth customer understanding was essential in picking up the signal from the noise in our data flow, and the models became more robust over time as more information about final outcomes was fed in.

With all the heavy lifting done, we were able to focus on refining our four best performing value propositions and creating consistency throughout the journey from awareness to exploration and purchase.

The Experiment: All website traffic was impacted by the experiment. 50% of the traffic was directed through to the control (training set) with an even distribution between the four available paths. The machine learning gathered data from the control and applied its learning to the targeted, the other 50% (scoring set).

Going the distance

Result

The experiment ran for 51 days and the learning took 34 days. The result was eagerness from the business to relaunch the experiment using similar messaging to maintain the momentum of improving results. The traffic allocated to the control group will increase to 10/90% to deliver more personalised experiences.

Numbers in action

Experiment stats

• 4 million visits to the experiment*
• 202,024 banner impressions**
• 14,967 banner click-throughs*
  7.41% click-through rate
• 2,942 internet orders commenced**
  1.46% impressions to order start
• 333 internet orders completed**
  0.16% impressions to order complete, and 11.83% shop conversion rate

*Adobe Target (10 April–30 May 19)
**Adobe Analytics (10 April–30 May 19)

Order conversion stats

Order conversion rates and orders: $99 connection fee waiver (Variation A; BAU) = 0.16% (51 orders), Unlimited data with $90 price point (Variation B) = 0.19% (200 orders), Entertainment (Variation C) = 0.13% (42 orders), Smart modem (Variation D) = 0.14% (40 orders)

Team

Project owner — Ann-lee Chin
Business Analyst — Laurene Desire
UX designer and researcher — Wendy Cooper
Content writers — Greg Williams & Bethan Jones
Visual designer — Ye Yint Aung