ASO Split Testing: How to Run an App Store Screenshot Test Without Leaving Your Screenshot Tool

Quick answer: ASO split testing means showing a changed version of your App Store product page (usually different screenshots) to a slice of real visitors and comparing how many install. On iOS you do it with Apple's Product Page Optimization: one treatment, a traffic share, App Review, then up to 90 days of data. The slow part has never been the test, it's making the variant and shuffling files into App Store Connect. SnapMonk now runs the whole loop: render a variant, compare it panel by panel with your live page, send it to Apple as a test, and watch your listing's before/after conversion in the same dashboard.

I've written before about why you should A/B test screenshots and how to generate variants with AI. This post is about the mechanics, because the mechanics are why most indie developers run zero tests a year. It's a lot of clicking for a result you'll see in a month.

We connected SnapMonk to App Store Connect this week so that the clicking mostly goes away. Here's the loop as I'd run it now.

What a split test on the App Store actually is

Apple calls it Product Page Optimization (PPO). You keep your live page as the control, create up to three treatments that change the icon, screenshots or preview video, choose what share of traffic sees each, and Apple splits visitors for up to 90 days. Treatments go through App Review before they run, usually a day or two.

Two constraints shape everything else:

1. One variable per test. If you change screenshots and the icon together, you learn nothing about either.
2. Apple's verdict is only in App Store Connect. Apple's API lets tools create and start tests but does not expose per-treatment conversion. Any tool that shows you "treatment B won with 95% confidence" is either scraping or making it up. What the API does give you is your listing's daily impressions, page views and downloads, which is enough to see the before/after.

Step 1: pick the change worth testing

Order of impact, from our own testing and everything I've read: the first screenshot, then the caption on it, then the order of frames two and three, then everything else. Search results show three frames before anyone taps, so frames four to eight barely move installs.

A good first test: same app, same frames, different opening caption. A bad first test: an entire redesign. You can't attribute a redesign to anything.

Step 2: render the variant

Make the treatment in the screenshot generator. One set, one change. Keep the device, background family and frame count identical to your live set so the only difference is the thing you're testing. Save it to your gallery.

Step 3: compare it against the live page before you send it

This is the step I always skipped and always regretted. Open Experiments, pick the app, and SnapMonk reads your current product page from App Store Connect: version, languages, and the live screenshots for the device you're testing. Pick your render and the two sets sit side by side, panel one against panel one, tap to zoom.

You're checking three things:

• the frame count matches (a treatment replaces the whole set for that device and language)
• the change is actually just one change
• nothing in the variant is worse than the live frame it replaces (it happens more than you'd think)

Step 4: send it to Apple

Name the test (that's what shows in App Store Connect), optionally write down what changed and why you expect it to convert better, and send. SnapMonk creates the experiment and treatment at Apple, uploads the screenshot set, and starts it so it goes to App Review. Or leave it un-started and send it from the card later.

Review takes a day or two. The state updates in Experiments every morning, or on refresh.

Under the hood this uses your own App Store Connect API key, stored encrypted; the connecting App Store Connect post covers how that works and why Apple requires the Admin role for tests.

Step 5: read the before and after

Here's where I want to be precise about what you're seeing.

Experiments shows your listing's conversion (downloads over page views, from Apple's daily analytics reports) with the 30 days before Apple approved the test as the baseline and every day since as the test period. The card says "listing +6% so far," or minus, or flat.

That is the effect of running the test on the whole listing, not the treatment's conversion versus the control's. Because Apple only sends the treatment to a slice of visitors, a real treatment win shows up as a smaller lift on the blended number. A blended +6% at 50% traffic suggests the treatment is doing roughly +12% on its own, though the noise at indie traffic volumes is wide.

For the actual verdict, open the test in App Store Connect. Apple runs the stats there. When it declares a winner, apply it in App Store Connect and come back to Experiments to record which page you kept. From then on the card tracks your listing's conversion since the change, so you can see whether the win held.

Step 6: ship, re-baseline, next test

If the treatment won, the new page becomes the control. If you run SnapMonk's send-to-next-version flow the winning render goes to your next editable App Store version with two clicks, live when you submit.

Then pick the next single variable. Most apps have three or four cheap wins in the first three frames before they hit diminishing returns.

How long does a test need?

Apple allows 90 days. Realistically you need enough page views for the difference to stand out from noise. If your page gets a few hundred views a day, two to three weeks is usually enough to see a big effect and not enough to see a small one. If it gets a few dozen, only large changes are testable at all, and you should be testing big swings, not caption tweaks.

Don't stop a test because the first week looks good. Weekday and weekend traffic convert differently, and featuring or a review spike can skew a few days badly.

Where to start if you've never run one

1. Connect App Store Connect (five minutes, an API key)
2. Open Experiments, look at where your listing under-converts by market
3. Render one variant of your first frame with a caption that names the outcome instead of the feature
4. Compare, send, wait two weeks, read the verdict in App Store Connect
5. Record the result and go again

That's a test a month with about an hour of your time each. For most indie apps that's the highest-return hour in ASO.

FAQ

What is ASO split testing? ASO split testing shows a changed version of your app's store page to part of your real visitors and compares install rate against the current page. On the App Store it's done with Apple's Product Page Optimization; on Google Play with store listing experiments. Screenshots are the most commonly tested element because they do most of the converting.

How do I run a split test on App Store screenshots? Create a Product Page Optimization test in App Store Connect with your live page as control and a treatment carrying the changed screenshots, set a traffic share, and submit it for review. Tools like SnapMonk's Experiments can create the test, upload the treatment screenshots and start it for you using your App Store Connect API key.

Can a tool show me which screenshot variant won? Not from Apple's data. The App Store Connect API can create and start product page tests but doesn't expose per-treatment conversion. Apple's verdict lives only in App Store Connect. What tools can legitimately show is your listing's overall before/after conversion from Apple's daily analytics reports.

How long should an App Store split test run? Apple allows up to 90 days. Two to three weeks is a practical minimum for apps with a few hundred daily page views; low-traffic apps should test bigger changes and expect to run longer. Never judge a test on its first week because weekday, weekend and featuring effects distort short windows.

What should I test first in my App Store screenshots? The first screenshot and its caption. Search results show frames one to three before anyone opens your page, so that frame is effectively your ad. Test an outcome-led caption against your current one before testing anything about frames four to eight.

Does ASO split testing affect my keyword rankings? Indirectly, and positively if it works. Apple's search algorithm rewards apps that convert well on a keyword with better positions for it, so a screenshot win tends to show up as more installs from existing traffic and then as more traffic.

Related reading

• A/B testing App Store screenshots: the methodology and what's testable
• A/B test screenshots with AI variants: producing treatments fast
• App Store conversion rate optimization: the elements ranked by impact
• Connect App Store Connect to SnapMonk: the API key setup Experiments relies on
• Experiments: the product page