Regression testing

The Regression testing step is QA's home for communicating the status of regression testing to the rest of the team. Even for teams without dedicated QA, Runway provides an easy-to-use regression testing script tool that can help to catch feature regressions before they become more costly to fix out in the wild.

Regression testing status

Use this dropdown to update the status of regression testing on a specific build. You can choose a status of In progress, Passed, or Failed and specify which build the new status applies to.

Regression testing results can be used to gate submission or release automations, because Runway requires this step to be complete (green) before automatically submitting an update for review to the app stores.

Test runs

If you've connected a regression testing integration, Runway will surface information about relevant test runs for each release within the Regression testing step.

Runway parses test run names for the presence of a version string (e.g. “2.3.0”) and any keywords specified in your regression testing integration settings in Runway to pull in the right test runs for each release.

Note that the overall Regression testing step status will still rely on the build testing status dropdown, so that you can be explicit about the final testing result.

Regression testing lists

From the regression testing step, you can put together a script of testing items that need to be checked by QA or other team members. This list lives across releases, so when you add new items to check, they'll show up as testing items on your next release as well.

Testers can mark an item as In progress, Passed, Failed, or Blocked. A tester can also leave an optional comment to provide additional context.

Team members in the user group QA will by default have the ability to update the status of regression testing items, but items can also be assigned specific owners. If a regression item is assigned to a user group, any user in the group can update the status of the regression item.

Slack reminders can be sent for all incomplete regression items, or individually for specific items to remind owners to update the status of their items.

Managing regression testing lists

You can view the list of all existing test items from the initial Summary tab. From here you can Add items to the list or Edit the existing list.

If you click the Edit button, you'll see additional controls to Delete an item, Edit an item's details, or re-order the list. Hit Done when you're finished making changes.

Click the Add button to create a new regression item.

Regression test item components

Release types

Specify which types of releases (major, minor, point, hotfix) require this test item.

Test item

A name for this test item. This appears on the regression testing list itself, as well as notifications related to updates on this item.

Notify in Slack or Teams when completed

Keep this option selected if you want your team to be notified about updates to this testing item during a release cycle. Runway will post in Slack or Teams when this item is marked as Passed, Failed, or Blocked (or was formerly Passed but reverted), as well as who performed the change and when.

Additional information

Optional - You can add information here that can easily be surfaced in Runway for team members responsible for testing this item. These notes serve as documentation that will be available on every release cycle. Markdown is supported, and there's a built-in viewer.

For example, you might have a state in your app that requires a specific sequence to test. You could leave a note in Additional information that says something like:

Use the coupon code TEST123 to check if the coupon field shows up correctly in the Subtotal table

Owners

Regression items can optionally specify user groups or individuals as owners – specify a regression item owner if a task is intended to be completed by a specific team member or by a team member from a specific group.

When a regression item has an owner, only the individual or a user in one of the specified groups will have permission to update the item's status or comment.