Health metrics settings

Health metrics in Runway are configurable metrics you can use to monitor the health of a release as it rolls out. Runway supports a number of different metric types that are computed using data from your Stability Monitoring, Observability & Analytics and App store integrations.

Runway uses your team’s custom ‘health’ definition on the Rollout page to give you a glanceable, immediately understandable view of how your app is faring post-release.

Once health definitions have been created, they can be leveraged as rules for triggering automations that help manage your rollout, like halting unhealthy phased rollouts and accelerating healthy phased rollouts.

Runway will also send Slack notifications any time your defined health metrics fall outside of their thresholds during a release rollout.

Custom thresholds related to observability & analytics integrations can only be configured for the key events that you have selected in those integrations' settings.

Metric Types

Stability monitoring metric types

Runway polls the connected stability monitoring integration to retrieve updated values for metrics every 10 minutes (approximately) during the course of a release’s rollout. Values are segmented by release version, platform, environment, and build number.

Supported metrics:

• Crash-free sessions

• Crash-free sessions delta from previous version value

• Crash-free sessions delta from previous version as a percentage

• Crash-free users

• Crash-free users delta from previous version value

• Crash-free users delta from previous version as a percentage

Crash-free sessions

The percentage of sessions for the release that have not experienced a crash. Calculated as follows:

(Number of sessions on a given version - number of sessions on a given version that have experienced a crash) / Number of sessions on a given version

Crash-free users

The percentage of users for the release that have not experienced a crash. Crash rate per user is calculated as follows:

(Number of users on a given version - number of users on a given version that have experienced a crash) / Number of users on a given version

Crash-free sessions & users delta from previous version value

The delta between the current release's user stability or session stability value and the previous version's value.

Crash-free sessions & users delta from previous version as a percentage

The delta between the current release's user stability or session stability and the previous version's as a percent change.

Observability & Analytics metric types

Runway polls the connected observability & analytics integration to retrieve updated values for metrics every 10 minutes (approximately) during the course of a release’s rollout. Values are segmented by release version and platform.

Supported metrics:

• Events per DAU

• Events per DAU delta from previous version value

• Events per DAU delta from previous version as a percentage

• Events per session

• Events per session delta from previous version value

• Events per session delta from previous version as a percentage

• Average of property value

• Average of property value delta from previous version value

• Average of property value delta from previous version as a percentge

• Median of property value (supported by Datadog and Amplitude only)

• Median of property value delta from previous version value

• Median of property value delta from previous version as a percentage

• Percentiles of property value (p90, p95, p99) (supported by Datadog and Amplitude only)

• Percentiles of property value (p90, p95, p99) delta from previous version value

• Percentiles of property value (p90, p95, p99) delta from previous version as a percentage

Events per DAU

The number of times an event is triggered per daily active user. Calculated by taking the total number of times an event has been triggered, and dividing that number by the total number of daily active users.

Events per session

The number of times an event is triggered per session. Calculated by taking the total number of times an event has been triggered, and dividing that by the total number of sessions.

Events per DAU and events per session delta from previous version value

The delta between the current release's count of events per DAU / per session and the previous version's value.

Events per DAU and events per session delta from previous version value as a percentage

The delta between the current release's count of events per DAU / per session and the previous version's value as a percentage.

Average of property value

For a given event, the average value of a specified property across all triggered events in the release version. For example:

Event name: "Add to cart"

Event property: "amount"

For all triggered "Add to cart" events in the release version, the average value of the "amount" property.

Median of property value

For a given event, the median value of a specific property across all triggered events in the release version.

Event name: "Add to cart"

Event property: "amount"

For all triggered "Add to cart" events in the release version, the median value of the "amount" property.

Percentiles of property value (p90, p95, p99)

For a given event, the 90th, 95th, or 99th percentile values of a specific property across all triggered events in the release version.

Event name: "Add to cart"

Event property: "amount"

For all triggered "Add to cart" events in the release version, the 90th, 95th, and 99th percentiles of the "amount" property.

Average, median, and percentile of property value delta from previous version

The delta between the current release's average, median, or percentile (p90, p95, p99) of property value and the previous version's values. For example:

Event name: "Add to cart"

Event property: "amount"

Average value: 95.00

Delta from previous version (value): +5.00

Average, median, and percentile of property value delta from previous version as percentage

The delta between the current release's average, median, or percentile (p90, p95, p99) of property value and the previous version's as a percentage. For example:

Event name: "Add to cart"

Event property: "amount"

Average value: 95.00

Delta from previous version (percentage): +5.5%

Ratings metric types

Runway interacts with the relevant app store API to fetch the latest user rating average for a given release and app.

Supported metrics:

• User rating for version average

• User rating for version average delta from previous version

• User rating for version average delta from previous version as a percentage

User rating average

The user rating for the given release version. This value comes directly from the Play Store or App Store integration, and is segmented by release version.

User rating average delta from previous version

The delta between the current release's average user rating and the previous version's value. For example:

Average user rating (current version): 3.5

Delta from previous version (value): -0.5

User rating average delta from previous version as a percentage

The delta between the current release's avg user rating and the previous version's value as a percentage. For example:

Average user rating (current version): 3.5

Delta from previous version (percentage): -10%

Calculated metrics

There is a special metric type that allows you to apply an equation to one or more more existing metrics, with the resulting value becoming a calculated metric that you can use just like any other metric. This can be used for capturing funnels or certain conversion rates, for example.

To add calculated metrics, choose the "Calculated from existing metrics" option in the Add Health Metric modal. You will select one or more existing metrics, and each will be assigned a token (M1, M2, etc.). Use those tokens to build the desired equation using standard mathematical operators, e.g. 100*M1/M2.

Currently supported for Observability & Analytics metrics only.