Regression testing

The Regression testing step is QA's home for communicating the status of regression testing to the rest of the team. Even for teams without dedicated QA, Runway provides an easy-to-use regression testing script tool that can help to catch feature regressions before they become more costly to fix out in the wild.
Build testing status
Use this dropdown to update the status of regression testing. You can choose a status of In progress, Passed, or Failed and additionally specify which build the new status applies to.
Regression testing results can be used to gate submission or release automations, because Runway requires this step to be complete (green) before automatically submitting an update for review to the app stores.
TestRail integration
​Click here to learn how to set up Runway's TestRail integration.​
Once you've connected TestRail, Runway will surface information about relevant test runs within the Regression testing step.
For the release linkage, Runway will parse your test run names for the presence of a version string (e.g. “2.3.0”) and any keywords specified in your TestRail integration settings in Runway.
The overall Regression testing step status will still rely on the build testing status dropdown, so that you can be explicit about testing status/result, and the build tested.
Regression testing lists
Here, you can put together a handy script of testing items that need to be checked by QA or other team members. This list lives across releases, so when you add new items to check, they'll show up as testing items on your next release as well.
If you have the TestRail integration connected, you won't see Runway's regression testing lists.
Testers can mark an item as In progress, Passed, Failed, or Blocked. A tester can also leave an optional comment to provide additional context.
Team members with an assigned role of QA will have the ability to update the status of regression testing items, but will otherwise have read-only access to the rest of Runway.
Managing regression testing lists
You can view the list of all existing test items from the initial Summary tab. From here you can Add items to the list or Edit the existing list.
If you click the Edit button, you'll see additional controls to Delete an item, Edit an item's details, or re-order the list. Hit Done when you're finished making changes.
Click the Add button to create a new Approval item.
Test item components
Release types
Specify which types of releases (major, minor, point, hotfix) require this test.
Test item
A name for this test item. This appears on the regression testing list itself, as well as notifications related to updates on this item.
Notify in Slack when completed
Keep this option selected if you want your team to be notified about updates to this testing item during a release cycle. Runway will post in Slack when this item is marked as Passed, Failed, or Blocked (or was formerly Passed but reverted), as well as who performed the change and when.
Additional information
Optional - You can add information here that can easily be surfaced in Runway for team members responsible for testing this item. These notes serve as documentation that will be available on every release cycle. Markdown is supported, and there's a built-in viewer.
For example, you might have a state in your app that requires a specific sequence to test. You could leave a note in Additional information that says something like:
Use the coupon code TEST123 to check if the coupon field shows up correctly in the Subtotal table
Owner
Optional - Specify a role here if a testing item is intended to be evaluated by a team member from a specific group. This may affect which team members signed into Runway have permissions to change this item's status, comment, or to revert a previous status.

Release candidate

Beta testing

Last modified 4mo ago

Copy link

Outline

Build testing status

TestRail integration

Regression testing lists

Managing regression testing lists

Test item components