Experiment analysis should answer whether the treatment improved the business outcome, not only whether the UI got more taps. AppActor analysis is subscription-aware. It combines assignment data with purchase, trial, cancellation, refund, and revenue signals so a variant is not judged only by an early click or local screen event.

​
Metrics to compare

​
Result modes

AppActor can evaluate experiments through different result windows: Use the mode that matches the decision. A pricing experiment often needs a cohort window. A short onboarding-copy experiment may be useful during the live experiment window.

​
Read the result carefully

Before calling a winner, check:
  • whether the variants have enough traffic
  • whether the maturity window is ready
  • whether assignment distribution is close to expected variant weights
  • whether one variant received a different country, app version, or platform mix
  • whether refunds or cancellations erased early conversion lift
  • whether the treatment changed entitlement quality, not just purchase count
Do not invent statistical significance language if the dashboard is showing maturity, lift, distribution, and subscription metrics. Call the result based on the metrics AppActor actually reports.

​
When a winner is real

A winner is ready to roll out when it has a clear metric lift, does not create a support or refund regression, and still matches your app’s entitlement model.

​
Next actions

  • Roll the winner into Remote Config if it is a server-driven payload.
  • Ship an app update if the winning behavior requires new client code.
  • Stop or archive the experiment after the rollout path is clear.