Effective Marketplace Ad Testing Has to Go Beyond A/B

Ads are a powerful and underutilized profit tool for marketplace operators. A Goldman Sachs Equity Research report noted in February 2021 that “in the US alone we believe retailers are positioned to generate $15-$20 billion of e-com oriented retail media revenue from CPG manufacturers by 2025; the figure should be substantially larger when we contemplate the potential to tap into media streams from electronic, apparel, sporting goods and other manufacturers for some retailers.” The scale here is massive and marketplace operators are taking note.

Despite the eagerness of many to take advantage of the trend, there are many who don’t know how to effectively implement the program. Within any organization, there are competing priorities. There are people responsible for site planning, merchandising, UX, CX, sort, personalization, supplier relationships, shopper support, etc. that all have a vested interest in making the marketplace as compelling and profitable as possible.

The good news is that, when done correctly, ads can do just that:

• Ads truly enhance the shopping experience
• Ads deliver high margin revenue for your you and your suppliers
• Ads improve supplier retention on your marketplace

Marketing Technology News: Helium 10 Announces New Amazon Advertising Platform – Launches ‘Adtomic’ a Breakthrough Solution for Amazon Sellers Looking to Optimize Advertising Campaigns

Are Ads Right for You?

To know if ads are right for you, it is important to look at the objectives of the participants in your marketplace. This is what we generally see:

 

Knowing the objectives of the participants in your marketplace is important to understanding how you can determine if your marketplace is a fit for an ads program. There are a few key things to look at when considering this. Are your organic CVR, CTR, and basket sizes at the right levels to support efficient returns for ads? Will sellers be compelled to participate in the ads program based on their organic performance? Furthermore, is there enough diversity from a merchandising perspective to support a healthy auction? Not enough diversity will not provide incentive for sellers to participate in the program, which could in turn hurt the buyer experience. Even if they do decide to participate, low diversity will lead to low CPCs, lower the upside of the program for you, the marketplace operator.

Once you’ve determined if your organic CVR, CTR, and average basket size are at the right levels and there’s enough diversity from a merchandising perspective the next consideration is: what are the sellers in your marketplace like and where do their budgets come from? Would your marketplace be competitive and compelling to where you would be able to tap into budgets beyond marketing (distribution, brand building, awareness etc.). The most effective marketplace monetization occurs when you are able to tap into multiple budgets, This allows the sellers to increase their demand based on a variety of their own economic goals. This in turn shows buyers the most relevant offers and enhances their shopping experience.

Handling Objections Through Testing

When ads are discussed, the most common objections we hear are that they will:

• Hurt the buyer experience (by distracting them from the core marketplace experience)
• Reduce conversion rate (by promoting potential suboptimal products)
• Alienate sellers (by forcing them to pay another marketplace fee)

The bottom line is that every single marketplace we work with has proven through extensive testing that none of these objections actually play out in the real world. This isn’t an accident – it starts with implementing best practices, but it is really supported by a rigorous testing methodology. The most mature marketplaces are literally running hundreds of tests – from the placement of the ads on the page to the color of a button inside the ad unit. By testing every possible iteration, you learn, you get better, and you drive results.

How You Test

When thinking about how to scale and test an ads program for a marketplace, there are three horizons you should consider that contain different ad inventory types, testing considerations, and desired outcomes.

The first horizon you should consider includes limiting impact to organic activity and core conversion. The ad format to consider in this use case is normally sponsored listings. By design, sponsored listings are native and endemic and through testing have been proven to have little to no impact on organic conversion, click-through rate and aggregate revenue. Generally, sponsored listings programs should test in three waves: alpha, beta, and general availability:

• Alpha – funded by partner, limited rollout and user traffic split
• Beta – funded by advertisers, full rollout, user traffic split
• General Availability – funded by advertisers, full rollout, all users

Limited rollout entails finding one robust marketplace to rollout a test to.

User traffic split simply means creating treatment groups by users vs sessions, ramping up slowly to gauge impact to organic activity, i.e.: 10% of users, 20% of users, 50% of users, 95% of users.

The desired outcomes of sponsored listings include incremental revenue and limited impact to organic performance. It’s safe to say that a sponsored listings test is an effective route to take.

 The second horizon you should consider includes capturing of branding/awareness dollars (CPM) and monetization of additional remnant inventory by way of display ad formats. Display ads allow for creative targeting by utilizing first party data; information you can collect from the users interactions with the site. Different types of display ads that utilize this first party data include banner ads, video, etc. The way to test this approach is by conducting an A/B test for each page that the ad is displayed as well as each ad format. Different ads can be generated from a set of information gathered from the user; A/B testing is important because it allows you to understand which ad is most effective.

The third horizon you should consider is monetization of long-tail inventory by way of additional POSs and linkoff on top level category pages. If done correctly, by capturing the customer that is higher in the purchase path, you can lower impact to core conversion. If done incorrectly, this approach has the potential to capture a customer that is not necessarily low in the funnel and close to conversion. The way to test this approach is by conducting an A/B test for each page that the ad is displayed as well as each ad format. Testing contextual linkoff has the potential to take away traffic from organic portions of the site, however the reward for this approach if rolled out properly is high being that they are extremely incremental helping to capture the customer that was not previously on the radar.

Horizon	Product Features	Testing Considerations	Desired Outcomes
1	Sponsored Listings	– Alpha – funded by partner, limited rollout and user traffic split – Beta – funded by advertisers, full rollout, user traffic split – General Availability- funded by advertisers, full rollout, all users	– Incremental revenue – Limited impact to organic performance
2	Display Ad Formats (Banner, video, etc.)	A/B test for each format and page	– Capture of branding/awareness dollars (CPM) – Monetization of additional remnant inventory
3	Contextual Link Off (expansion to additional POS or sites)	A/B test for each format and page	– Monetization of high level category pages – Monetization of long tail inventory

All this being said, there is no ‘one size fits all’ approach. Deciding the right testing path for you takes thoughtful and strategic consideration of the dynamics of your marketplace and having the right success measurements in place to guide you on your journey.

Marketing Technology News: MarTech Interview with Lauren Weinberg, Global Head of Marketing and Comms at Square

Measuring Success

 When looking at the success of your ads program, there are some quick-hit metrics you should review:

• Impact to organic conversion – Is conversion helped or hurt by ads? How much?
• Ad revenue – How much are you making in total with advertising? This is a measure of total volume.
• CPM (cost per thousand impressions) – How much are you making per impression? This is a measure of the quality of your monetization.
• RPS (revenue per session) – How much money are you making per user session? Is this number higher with or without ads?
• Active advertisers – How are advertisers participating in your ad program? This is a measure of the diversity of demand in your program.

There are several other metrics you’ll want to look at over time as your program matures, but there are two less common ones that we find particularly interesting with new marketplace ads programs:

Seller retention is often improved by ads programs. Why? Ads within a marketplace are a value-add service to your core offering (selling on the marketplace) that further cements your relationship with a seller. The real goal of an ads program is not just to improve revenue, but also to improve the stickiness of your underlying marketplace.

Your profit margin is often improved with ads. Let’s say for example:

Hypothetical Scenario 1 (no ads):

Your marketplace fee on a $50 basket is $15 (30%). After the costs of operating the marketplace itself, let’s say you walk away with a $1.50 profit. Your profit margin is 10% on this order. Let’s say you have 50M orders a year with an ABS (average basket size) of $50 and an average marketplace fee of 30%. Your profit would be $75M (on $750M in revenue) and your profit margin is still 10%.

Hypothetical Scenario 2 (with ads):

Now let’s say you have the same exact marketplace economics with a $100M a year ad business. Your margin on ads is much higher – let’s say 75%. In this case, your ads profit is $75M. On that same 50M orders a year with an ABS (average basket size) of $50 and an average marketplace fee of 30%, your profit would be $150M (on $850B in revenue) and your profit margin is now 18%. Thus, your profit margin would be higher with ads.

In Conclusion

Ads are a powerful and growing profit tool for marketplace operators. Effectively testing ads in your marketplace requires that you understand the objectives or your marketplace participants and have the proper metrics in place to constantly measure success and adjust your strategy accordingly.

Marketing Technology News: Bad Data is Bad News for Marketers