TMW Case Study #001 | Scaling Martech QA with computer vision and robots

Welcome to The Martech Weekly, where every week I review some of the most interesting ideas, research, and latest news. I look to where the industry is going and what you should be paying attention to.

👋 Get TMW every Sunday

TMW is the fastest and easiest way to stay ahead of the Martech industry. Sign up to get TMW newsletters, along with a once-a-month complimentary Sunday Deep Dive. Learn more here.

Using computer vision and robots to scale Martech QA, with Satya Ramachandran of Rappi

Welcome to our very first TMW case study! Kicking off this series, we’re featuring Rappi, the Latin American super-app that connects consumers with merchants that sell a wide variety of products, and drivers that can bring those products to their doorstep. The three-sided business is not only a logistical challenge, but also a Martech challenge.

Rappi’s array of marketing campaigns and offers, driven by a sophisticated deep-linking strategy, is crucial to its success. It did, however, lead to the need for an impossibly large amount of QA to ensure the successful delivery of customer experience workflows, ensuring that would-be customers don’t fall off their buying journey at any point, from clicking on an ad through to landing in the app and making a purchase.

Leading the Martech and Adtech practice at Rappi is Satya Ramachandran, who brings over 12 years of Martech experience to the table, having previously worked as a data engineer building distributed databases. 

In this case study, we’ll walk through how Satya not only scaled the Martech QA process using computer vision and robots, but turned QA into a profit-driving initiative with champions throughout the business, rather than just a cost center.

Satya’s responses have been edited for clarity and congruency.

You can listen to the entire exchange here 🎧

• Part I: Setting and Context
  • Satya’s move from engineering to Martech.
  • Rappi’s three-sided business and the cold start problem.
  • Maturing from growth-at-all-costs to profitability.

• Part II: The Challenge
  • Deep linking and “silent” conversion failures.
  • The limitations of manual testing.
  • The responsibility gap between marketing and product.

• Part III: The Solution
  • Computer vision and robotics.
  • QA as a revenue driver.
  • Implementation challenges.

• Part IV: Reflections
  • The most painful part of the process.
  • Advice for others implementing bleeding edge technology.
  • The future of Rappi.

Part I: Setting and Context

Satya was originally a data engineer. He decided to move into Martech because of his ambition to synergize the technical work he was doing with the commercial impact it was driving. This led Satya through a few organizations, including the publisher Firefly. Next up came DoorDash, which provided his first real taste of marketing at a mature company. His desire for growth and a new challenge then took him to Rappi. 

Rappi was particularly appealing as the company was in a hyper-growth phase. The Martech infrastructure was a blank slate, meaning he had a great opportunity to build it from the ground up. Satya has a broad role at Rappi as Head of Martech and Adtech; he looks after all technology initiatives, with his current focus being on building Acquisition IQ, a homespun acquisition platform that demonstrates his belief in the power of composable Martech. But before he could move onto other projects, Satya had to solve a fundamental problem that was curbing Rappi’s growth.

With Rappi’s three-sided business, there are three distinct business models: B2C with consumers, B2B with small businesses (drivers), and B2B with larger enterprises (merchants). To enable a customer to transact from a merchant and have the product delivered directly to them by a delivery partner takes some serious coordination – not only of data and platforms, but also incentives. Knowing which part of the business to prioritize was not easy, especially during the early days when Rappi was in its growth phase. The company had to overcome a cold start where it had no merchants, no drivers and ultimately, no customers. Led by direct sales, Rappi started to gain traction and maturity as a business, leading to a change of focus towards profitability over growth. This is when measurement became so much more important for Satya; it was an important tool to understand what was driving growth and where customers were dropping out of the buying journey.

Part II: The Challenge

For right or wrong, digital marketing is very focused on tracking and analytics. Everything that can be measured, will be measured and pored over by marketers trying to gain a percentage-point increase in conversion. For Rappi, tracking the performance of the end-to-end journey of consumers was critical, from the moment they see an ad or click somewhere on a website, through to landing in the app and making a transaction. If any step in that journey fails, the customer doesn’t transact, and that means lost revenue and wasted ad spend.

This is where deep linking comes in. You probably don’t think too much about when you click on a CTA on a website or social media and it immediately deep links you to a specific page within a specific app, but there is a lot going on in those milliseconds following your click. This means there’s also a lot that can go wrong in that time, especially at handoff points between different applications.

For Rappi, the risks were particularly pronounced, as they had hundreds of different marketing workflows that included handoffs between websites and ads to the Rappi app. Each of these flows would be tested when first set up, but the marketing campaigns driving traffic to the app were changing all the time, including the offers and customer eligibility, which could change depending on dynamic factors such as the time of day. Satya had a tough problem on his hands, as he couldn’t consistently test these processes manually and wasn’t able to quantify the size of the problem. 

Satya had implemented OneLink by AppsFlyer for deep linking. In testing deep links from outside Rappi, he was only able to see that a user had landed in the app. Separately, the product team would do automated regression testing using Selenium, which would prove that once in the app, users were able to do everything they needed to complete a transaction. The real challenge was testing both of those things together. This is where Satya started to see errors that couldn’t be reproduced, and what he describes as “silent” drops in conversion rate. 

“So basically, marketing is about driving conversions in whatever metric we are trying to achieve... So, what is happening is because of these errors, you get these conversion rate drops, and it's a silent drop. You don't even know why they are dropping. You don't know why they're dropping because there are, for example, temporary errors that aren’t happening all the time. So, you don't realize that there is a problem, and that's causing the loss of conversion.”

Silent drops in conversion… it’s enough to make any marketer break out in a cold sweat. So, not only was the commercial performance of Rappi’s marketing suffering, but it was also happening in an unpredictable way that flew below the radar.

So, how did Satya’s team isolate the problem? Manually at first, but he recognized that this would never reach the scale he needed. At points, Rappi would have close to 500 marketing flows running daily, which would need to be stress tested for real, random human behavior, not just by marketers who know exactly how the flow should work.

Not only was there a huge number of flows that needed testing, but each flow was individually complex to test, with many involving 15 to 20 steps in the process. Many tests even involved setting up real Facebook and search ads as a jumping-off point for the workflow. A big part of the testing bottleneck was structural: Satya had a small development team with no dedicated QA team; the marketing team were skilled at marketing strategy and executing campaigns, but had no background in testing and QA; and the product team were capable testers, but were focused on product outcomes within the app.

For Satya, marketing QA is more complex than product QA. Product QA for an app usually relies on having a logged-in user as the point of origin. Marketing QA, however, can start anywhere across the web and then take a user as deep into an app as the checkout page, with a pre-populated cart and coupon to boot. Not only that, the ads that Rappi was testing were changing all the time, particularly in terms of creative. Because of this complexity, Satya knew he needed something bleeding edge to solve the problem.

Part III: The Solution

So, Satya did some digging and found a company called Mobot that could help him solve the silent conversion problem that Rappi was experiencing. Their pitch was like something out of a science fiction book: they’d use computer vision to view Rappi’s many user flows on an array of devices, and then robotic fingers would physically click the screen exactly like a human would. The promise was that this would allow Rappi to test its user flows across search, display, and social ads every single day and even at different times of day, including accounting for different nuances within each flow.

For Satya, this promised to not only solve the direct conversion problem, but also the challenging split of responsibilities across marketing and product teams, where both view testing within the confines of their specialisms, rather than as an end-to-end process.

Despite how futuristic this all sounds, this type of technology is not new. Computer vision and robotics have been used together for a while to test devices like smartphones before they are released to the general public. Applying this to testing ads and deep links on these devices takes the concept of end-to-end testing to another level.

So how does it actually work? Mobot has a device “farm” set up with all sorts of smartphones, laptops, and tablets covering all major operating systems, screen sizes, and versions. Placed in front of them are robotic fingers that can click on the trackpads, buttons, and screens of the devices. Whereas a lot of automated digital testing is done based on virtually emulating human interactions with devices, Mobot simulates real human behavior, testing the true end-to-end journey that a potential user would go through. To make the magic happen, Rappi must specify the user flows and the expected outcomes. If Mobot finds that a flow doesn’t work as expected, it will instantly trigger an alert to the Rappi team, allowing them to remedy the issue before it costs the company any more revenue.

For Rappi, this has many benefits. Not only can they proactively find bugs in the user flow, but they can pause ad spends that are leading to dead ends, and move them towards ads that are working as expected. The “silent” failures are now heard instantly. 

Satya was an engineer before he got into Martech, and in his own words, he “never thought about the cost of a good solution.” His career change into Martech was driven by a desire to understand the outcomes and impact of his work, which he delivered in spades using Mobot. During pilot, the solution only cost $5K, but achieved an additional $150K in orders that would have otherwise been lost to failed marketing flows without mentioning the additional long-term customer lifetime value. Incredibly, Satya found that close to 50% of deep links were failing, meaning that there was a massive opportunity for QA to not only make an impact, but also uncover invaluable insights about Rappi’s customers and processes.

QA is usually seen as just a functional step in the campaign production process. Build the campaign, test the campaign, launch the campaign, measure the campaign. But Satya turned this idea on its head by framing QA as a genuine revenue driver. He describes QA as insurance that protects companies when accidents happen, although in his conversations with CMOs from other businesses, most of them reveal that they spend nothing on QA. This is in stark contrast to product teams, who generally invest a lot into QA and testing.

Speaking of the product team, Mobot was integrated into their instance of Jira. This means that as soon as a user flow issue is detected, Mobot generates a Jira ticket including a recording of the interactions that led to the issue. Mobot’s AI technology is also able to categorize failures into groups depending on the suspected bug type, so that they don’t overload the product team’s Jira board with similar bugs found in different places.

Mobot’s use of AI is striking. Although it is by far the most dominant trend in tech at the moment, many marketers are yet to find use cases that make a considerable and measurable difference to the bottom line. Rappi’s success with Mobot shows that there are effective use cases for AI out there, but are marketers looking for them in the wrong places? Using AI to create personalized GenAI videos might be a sexier demonstration of the technology, but it’s in scaling up the mundane where it really thrives. For Satya and his team, a massive leap of faith was required from all involved to trust the robots in automating the QA process.

Suspending disbelief was just one challenge that Satya had to overcome to overhaul the marketing QA process. There were challenges with integrations and specifying the desired user flows from a constantly changing set of ads, but the biggest problem – like most tech endeavors – was getting folks internally to buy in to the change. This was partly because it meant that Satya had to find a way to communicate his belief that the product was broken in a lot of places, which wasn’t made any easier when early tests showed a lot of false positives, with flows ending in expected failures. As the QA process was fine-tuned, however, the team started to uncover real bugs that had a clear, quantifiable impact on conversion rates. 

That was the moment the product and paid media teams started to get on board, but there was still another hurdle to overcome: executive buy-in. With Rappi moving from a growth to profitability focus, Satya knew that using futuristic buzzwords like "computer vision” and “robots” would make the execs think that what he was suggesting would be expensive. So, he led with the business outcome first, before explaining the tech to get the project over the line. Now, everyone at Rappi sees the value, so much so that the paid media team agreed to pay for marketing QA out of their budget.

“So, once we started getting these examples, the paid marketing team started putting their wood behind us. And now they are the biggest champions for us. [Paying for Mobot] is now coming out of the media budget. For a growth marketer to sacrifice their media dollars means that it is valuable.”

Part IV: Reflections

Now that implementing computer vision and robots for marketing QA is in the rear-view mirror, Satya can reflect on the process and articulate the most painful part, suggesting that there is a fast-paced culture in marketing that can make it hard to implement Martech, as stakeholders expect results straight away:

“It takes time for you to invest. You have to make sure you put in some money initially, invest some money, invest some resources, and then the results will take some time to happen. So that initial period was a very tough period. First of all, the infrastructure was taking time. We didn't have that much support internally. There was some internal stuff that needed to happen to support the project, but all the resources were prioritized for other projects. So, we had to beg, borrow, and steal to get it over the line.

And then, after a couple of weeks, nothing is happening because the whole thing is still getting set up. So, then it's taking a little bit more time but in marketing, everything happens really fast. It's not like when you're launching campaigns. With technology, you have to get used to how long it takes. So that initial part, until we were able to show value, was a very painful time.”

For anyone looking to go through a similarly transformative, cutting-edge technology adoption, Satya has some familiar advice: start with the outcome, and show a roadmap towards building value. He describes Martech folks as a “little bit special” as they are the bridge between technology and marketing, which means they must not only be able to talk in business terms, but also need to set people’s expectations around the level of investment and time-to-value they can expect from Martech initiatives.

Satya’s success with revolutionizing Rappi’s marketing QA process shows that when you get these things right, not only can you drive great insight and return on investment, but you can also change people’s perceptions from seeing something as a cost center, to seeing it as a growth driver.

🔓 Are you ready to join the PRO's?

Get the entire Wednesday Martech Briefing and all of our newsletters every week with TMW PRO. It's the easiest and fastest way to make sure you're fully informed on how the marketing technology industry is evolving. 

Here’s what some of our existing TMW PRO members have to say;
 
“TWM is a finger on the pulse of what's happening, with the clarity of insight to see the important patterns underneath the noise” – Scott Brinker (VP, Hubspot).

“I recommend TMW to all leaders who are customer-focused and transforming companies via digital technologies” – Kazuki Ohta (CEO, Treasure Data).
 
“I've subscribed to many Marketing newsletters, but TMW is at the top of my list with the most valuable content for marketing professionals" – Lilly Lou (Marketing Director, Icertis).
 
TMW PRO is an invaluable resource that’ll help you to truly transform and future-proof your career in Martech. 

Are you ready to join the PRO's?

If the answer is yes, CLICK HERE.

Stay Curious,

Make sense of marketing technology.

Sign up now to get TMW delivered to your inbox every Sunday evening plus an invite to the slack community.

Want to share something interesting or be featured in The Martech Weekly? Drop me a line at juan@themartechweekly.com.