How to test apps customized for each company (multi-tenant apps)

If you provide one base app to multiple companies and change the logo, colors, text, available features, or connection settings for each company (a so-called multi-tenant or white-label app), creating test cases for each company makes the number of test cases grow with the number of companies. Every time the screens change, you have to make the same fix many times.

This page explains how to use one test case for all tenants and handle the differences between tenants with settings, and which MagicPod features to use.

Note: On this page, each company that you provide a customized app to is called a "tenant."

Table of contents

1. Basic approach
2. List the differences between tenants
3. Create a batch run setting for each tenant
4. When to use shared variables and when to use data patterns
5. Use locators that do not change between tenants
6. Put common steps into shared steps
7. When features or screen flows differ between tenants
8. Which tenants to test, and when
9. How to organize projects
10. Examples of team rules

1. Basic approach

• Create one test case that is shared by all tenants. Do not copy test cases for each tenant.
• Do not write values that differ between tenants (URLs, login information, displayed text, etc.) directly in test cases. Move them out into shared variables.
• Switch tenants with batch run settings, not with test cases (one batch run setting per tenant).

With this approach, when you fix a test case because of a screen change, one fix applies to all tenants. When you add a new tenant, you only need to add one batch run setting.

2. List the differences between tenants

Before you start creating test cases, we recommend listing what differs between tenants. The feature to use depends on the type of difference.

3. Create a batch run setting for each tenant

In the "Advanced settings" tab of the batch run settings, set the following items for each tenant. Giving each setting a name that shows the tenant makes them easier to manage (e.g., "10:TenantA_Staging").

Shared variables

Register values that differ between tenants as shared variables, and refer to them from test cases with ${variable name}.

• Make passwords and API tokens secret shared variables so that they are masked.
• Use the same variable names for all tenants. If the variable names differ between tenants, you cannot share test cases.

Reference: Utilize variables

For browser testing: Base URL

If each tenant has a different domain or subdomain, set the URL for each tenant in the "Base URL" field of the batch run setting, and use root-relative paths (e.g., /login) in the "Open URL" steps of your test cases.

Reference: Switch test execution environments (Use different domains for test cases)

Note: You cannot use variables in the Base URL field. If tenants are identified by the path of the URL (e.g., https://example.com/tenant-a/), include a shared variable in the URL of the "Open URL" step, such as ${TENANT_PATH}/login.

For mobile app testing: app files and app launch arguments

• If you build a separate app for each tenant (with a different bundle ID or package name), specify the app under test for each tenant in the batch run settings dialog.
• If tenants can be switched inside one app, it is also a good idea to ask your development team to make it possible to pass the tenant through app launch arguments or app env variables.

Reference: Setting "App launch arguments" / "App env variables"

When running tests from the API

If you want to switch tenants when running tests from CI or other tools, you can specify a batch run settings number and overwrite only the shared variables.

Reference: Run Tests Using a Batch Run Settings Number While Overwriting Some Settings

4. When to use shared variables and when to use data patterns

"Data patterns," which run one test case repeatedly with different input data, can also be used to handle differences between tenants. Shared variables and data patterns are good at different things, so we recommend using them as follows.

When data patterns are a good fit

• When you want to test different inputs within the same tenant
  Use data patterns for patterns you want to test regardless of the tenant, such as combinations of payment methods (credit card / bank account) and amounts (exactly the limit / over the limit). You can combine them with tenant values (shared variables) in the same test case.
• When you want to run the same simple check for all tenants at once in browser testing
  If the only differences between tenants are the URL and login information, you can make each row of the data pattern one tenant and check all tenants in a row with one test case (e.g., whether each tenant's login screen opens and the logo is displayed). You can also run only a specific tenant. In the "Batch run target" tab of the batch run settings, select "Specify by test case numbers" and enter "test case number[data pattern number]" to run only that data pattern. For example, if the second data pattern of test case number 3 has the values for Tenant B, entering "3[2]" runs only Tenant B.

Notes on storing tenant information in data patterns

• Data patterns are set for each test case. If you write tenant information in the data patterns of many test cases, you will need to fix all of those test cases every time you add a tenant or change login information. Keep tenant information that is used by multiple test cases in shared variables.
• Do not write values such as passwords in data patterns. Use secret shared variables instead.
• For mobile apps with a separate app for each tenant, the app under test is decided by the batch run setting, so switch tenants with batch run settings.

Reference: Utilize Data Patterns (Data-Driven Testing)

5. Use locators that do not change between tenants

In apps where the text and design differ between tenants, locators based on text or images fail as soon as you switch tenants.

• The most effective approach is to ask your development team to add unique IDs that are the same for all tenants (for web, id or data-testid; for mobile, accessibility identifier / resource-id, etc.).
• If you use locators that contain text, you can handle the text for each tenant by using variables in the locator, such as ${SERVICE_TITLE}.

Reference: Checklist of what developers should do for successful test automation

6. Put common steps into shared steps

Make steps that are common to all tenants into shared steps, such as logging in, going to the home screen, and entering payment details. If you use shared variables (such as ${LOGIN_ID}) inside shared steps, one shared step can log in to all tenants.

For parts where the steps differ a lot between tenants (e.g., the available payment methods differ by tenant, so the input screens follow different flows), separate shared steps such as "Payment (credit card)" and "Payment (bank account)" make test cases easier to read.

Reference: Utilize shared steps

7. When features or screen flows differ between tenants

When a feature exists only for some tenants: separate with labels

Add a label with the feature name (e.g., "Feature_PointPayment") to the test cases for that feature, and include the label in the execution targets only in the batch run settings of the tenants that have the feature. Compared with checking the tenant inside test cases, this makes it easier to see in a list which tests run for which tenant.

Reference: Perform batch run only for specific test cases

When only some steps of the same test case differ: conditional branching

Store the tenant's settings in shared variables (e.g., FEATURE_POINT=on), and use the "If variable value equals" command to branch the steps.

Too many branches make test cases hard to read. If you need many branches, consider splitting the test case.

Reference: Utilize conditional branching

8. Which tenants to test, and when

Running every test case for every tenant every time uses a lot of execution time and cloud device usage. We recommend changing the scope of test runs based on what has changed.

• For checking the look and feel, such as logos and colors (e.g., visual diff checks), the expected results differ for each tenant. It is easier to manage if you limit these checks to representative tenants or only the tenants that need them.
• If you use scheduled runs, set different times for the batch run settings of each tenant. This makes it less likely to go over the number of cloud devices that can run at the same time.

Reference: Shorten the time taken to run tests

9. How to organize projects

We basically recommend keeping everything in one project. Shared steps and shared UI elements can only be used within the same project. If you separate projects by tenant, you will need to fix the common parts once for each project.

However, consider separating projects in cases like the following.

• The customization is large, and the screens have very little in common
• Different teams are in charge of testing each tenant, and you want to separate member permissions

Reference: Duplicate test cases to another project

10. Examples of team rules

As the number of tenants grows, so does the number of batch run settings and labels, and it becomes harder to see who manages which tenant's tests and how. To avoid confusion when adding tenants or fixing test cases, we recommend deciding on rules like the following with your team in advance.

• Decide on naming rules that include the tenant name or feature name in batch run setting names and label names
• Write down the steps for adding a tenant (adding a batch run setting, registering shared variables, checking feature labels, and preparing test accounts)
• When you find a bug that only happens for a specific tenant, first check whether you can fix the test case while keeping it shared (whether shared variables or labels can handle it)

Reference: Operational rules for using MagicPod with multiple users