System/UI Tests – The Layer You Should Hate Needing

Part 6 of “The Test Pyramid — Reimagined.” Start with the opener if you missed it.

---

System tests are the tests you should hate needing, write anyway, and keep on the shortest possible leash.

That’s the entire post in one sentence, but I’m going to spend the rest of it convincing you that the three clauses are equally important. Most teams nail one or two and miss the third. The teams that nail all three have suites that take fifteen minutes to run instead of three hours, catch the bugs that only exist post-deploy, and aren’t paging anybody at 3 AM because a flaky locator timed out. The teams that nail none of them have the inverted pyramid I’ve been complaining about for five posts straight: a thin base, almost nothing in the middle, and a hulking pile of UI tests at the top doing work that should have been caught five layers down.

The system layer is where the push-down rule gets tested most. Every shortcut you take here — every test you push up to this layer because it’s where the tooling lives, every Selenium test that exists to verify behavior a unit test would catch in milliseconds — compounds in cost, flake, and noise for as long as the suite exists. Some system tests are non-negotiable. Most of what teams currently have at the system layer is non-essential, and the cost of carrying it is paid every CI run, every release, every on-call rotation.

Let’s get specific.

What I mean by “system test”

Quick terminology note, because this layer suffers from naming-by-accident. At my company, we call these system tests. You’ll hear the same tests called UI tests or E2E tests depending on who set up the suite. I prefer “system test” for two reasons:

• They test the whole system, not just the UI. The UI is the entry point, not the unit under test. What you’re actually exercising is everything behind it — the deployed app, the CDN, the identity provider, the cookies, the load balancer, the network. Calling them “UI tests” puts the emphasis on the layer you care about least and obscures the layer that’s actually paying the bill.
• “E2E” usually means something else. End-to-end tests, in most teams’ usage, are long workflow tests: sign up, configure, run a job, see the result, cancel, churn. Those are valid — somewhere — but they’re a different shape from what belongs at this layer. The tests I’m describing here exist specifically to provide a deploy-gate signal: is the deployed application fundamentally working? They’re shallow, focused, and short. Conflating them with workflow E2E tests is how you end up with a “system suite” that takes three hours to run and gates every release.

I’ll use “system test” throughout. If your team calls them UI tests or E2E tests, mentally substitute — but notice when the name is dragging the suite in a direction it shouldn’t go.

In the opener I gave you the one-line version: the full deployed stack — real auth, real CSS, real third-party JavaScript, real CDN, real DNS, real everything. Let me unpack that.

A system test, in my model:

• Exercises the full deployed application as a real user would. A real browser, driven by Selenium or Playwright or Cypress. A real mobile build, driven by Appium or Espresso or XCUITest. Whatever the user’s actual entry point is — that’s what the system test drives.
• Runs against a deployed environment. Above the line. The application is built, deployed, and reachable at a URL or installed on a device. Nothing about this test runs in-process.
• Includes every layer the user includes. The CDN serving your JavaScript. The CSS your theme injects. The third-party analytics scripts your marketing team added six months ago. The identity provider that redirects on login. The CSP headers. The cookies. The load balancer. The everything.
• Exists primarily to catch bugs that only exist post-deploy. This is the rule that does most of the work. If the bug the test would catch could have been caught at a lower layer, the test belongs at that lower layer. The system layer is for the bugs that cannot be caught anywhere else.

That last bullet is the entire job description. Every system test in your suite either fits it or doesn’t. The ones that don’t fit are the ones eating your release cadence.

What only the system layer can catch

The bugs that justify the system layer’s existence have a specific flavor: they’re emergent from the deployment itself. The code is fine. The integration tests pass. The API tests pass. The pre-deploy pipeline is happy. And then the thing breaks anyway, because being deployed introduced a variable nothing upstream could simulate.

A non-exhaustive list, drawn shamelessly from my own scar tissue:

• The CSP header that blocks your analytics in production but not staging. A new content-security-policy rule got tightened in the production config but not the staging one. Every pre-deploy test passed. The system test that loads the real page in a real browser is the only test that can notice that GA is silently failing to load.
• The third-party CDN serving a different asset version. A vendor pushed an update to their JavaScript SDK. Your staging environment caches the old version. Production fetches the new one. Behavior diverges. Nothing in your code changed. The system test against production catches it; nothing else can.
• The IAM role that’s right in dev and wrong in prod. The dev environment’s service account has permissions the prod one doesn’t. Pre-deploy tests pass against a permissive local config. The deployed prod app can’t reach the secrets manager. The first request hits the floor. A smoke-level system test notices in seconds.
• The auth flow that redirects somewhere unexpected. Your IDP’s behavior in production differs subtly from staging — different consent screen, different scopes, different redirect URL allowlist. The integration tests stub auth entirely. The API tests use a token issued in a test tenant. The system test that actually clicks “log in” and goes through the real flow is the only one that finds out.
• The mobile build that crashes on first launch because of a missing entitlement. The app builds fine. The unit tests pass. The Detox or XCTest tests pass on a simulator with a permissive config. The signed, distributed build hits a real device, asks for a permission you didn’t declare, and crashes. The system test on a real device against a real distribution build is what catches it.
• The cookie domain that’s wrong in the wrong way. Set on app.example.com instead of .example.com. Login works. Then the user navigates to account.example.com and the cookie isn’t there. Pre-deploy tests don’t have multiple subdomains. The system test does.

None of those bugs are findable below the line. All of them are findable in five minutes by a competent system test. That is the layer’s job. Catch the small number of bugs nothing else can, with the smallest number of tests you can get away with.

If your system suite is doing more than that, you’re paying rent on tests that didn’t have to live up there.

What a clean system test looks like

Same idea, two languages — one web, one mobile.

TypeScript + Playwright — a system test of the production login flow, including the real redirect through the identity provider:

import { test, expect } from "@playwright/test";

const APP_URL = process.env.APP_URL; // e.g. https://app.staging.example.com

test("a returning user can log in and reach their dashboard", async ({ page }) => {
  await page.goto(APP_URL);

  await page.getByRole("link", { name: /log in/i }).click();

  // Redirected to the real identity provider here.
  await page.getByLabel("Email").fill(process.env.SMOKE_TEST_USER!);
  await page.getByRole("button", { name: /continue/i }).click();
  await page.getByLabel("Password").fill(process.env.SMOKE_TEST_PASSWORD!);
  await page.getByRole("button", { name: /log in/i }).click();

  // Redirected back to the app. CSP, cookies, third-party scripts all real.
  await expect(page).toHaveURL(new RegExp(`${APP_URL}/dashboard`));
  await expect(page.getByRole("heading", { name: /welcome back/i })).toBeVisible();
});

What this test catches that nothing else can: the IDP redirect, the cookie set on the right domain, the CSP allowing the IDP’s scripts back through, the deployed dashboard rendering without a missing-asset error. Five things that all live above the line, in one test, in well under a minute.

What this test doesn’t do: assert on the shape of the dashboard’s data, validate form-validation edge cases, test the dashboard’s sorting behavior. Those are unit and integration jobs. Doing them here would balloon the suite without adding coverage.

Java + Appium — a system test of a real mobile build, on a real device or emulator, asserting the login flow works end to end:

class LoginSystemTest extends BaseMobileTest {

    @Test
    void aReturningUserCanLogInAndReachTheirDashboard() {
        app.findByAccessibilityId("login-link").click();

        WebDriverWait wait = new WebDriverWait(app, Duration.ofSeconds(15));

        wait.until(visibilityOfElementLocated(By.id("email")))
            .sendKeys(System.getenv("SMOKE_TEST_USER"));
        app.findByAccessibilityId("continue").click();

        wait.until(visibilityOfElementLocated(By.id("password")))
            .sendKeys(System.getenv("SMOKE_TEST_PASSWORD"));
        app.findByAccessibilityId("log-in").click();

        WebElement welcome = wait.until(
            visibilityOfElementLocated(MobileBy.AccessibilityId("welcome-heading"))
        );
        assertThat(welcome.getText()).matches("(?i)welcome back.*");
    }
}

Same job, different surface. The build is signed. The device is real (or a high-fidelity emulator). The IDP is real. The deep-link handler that returns control to the app after the redirect is real. None of those are testable below the line, which is exactly why this test exists.

Both tests share a property worth pulling out: they’re single-flow. One login, one assertion of arrival. Not a battery of dashboard interactions. Not five subsequent navigations. The temptation to “while I’m here, let me also verify…” is exactly the temptation that turns a fifteen-minute system suite into a three-hour one. Resist it.

The word “some”

The opener spent one sentence on this and I want to spend a whole section on it: you need some system tests. The word doing the work is some.

Here’s the calibration I use, and your numbers will vary:

• A handful of critical-path system tests per major user flow. Login. Checkout. The thing your product literally exists to do. If these break, you don’t have a product.
• A small set of smoke-level system tests that run on every deploy, finishing in single-digit minutes. Their job is to catch the deploy itself broke something obvious.
• A modest set of cross-cutting system tests for the things only the real browser stack exhibits — CSP, third-party scripts, real auth, mobile deep links, viewport-sensitive rendering at a couple of representative breakpoints.

That’s it. Tens of tests, not hundreds. If your system suite is in the thousands and growing, you have moved tests up to the system layer that did not need to be there, and your CI is paying for it on every run.

The question I ask in PR review for any new system test: what would have to change about this codebase to delete this test? If the answer is “we’d need to add integration tests for the same behavior” — good, do that, and delete this one. If the answer is “we’d need to also delete the flow it tests” — that’s a system test that earns its slot. The flow it covers is critical enough that its real-world behavior is worth verifying after every deploy.

On flake

The single most damaging property of an unkempt system suite is not its runtime. It’s its flake rate. A suite that fails 5% of the time for reasons that have nothing to do with the code under test trains your team to ignore failures. Once your team is trained to ignore failures, the suite is worse than useless — it’s actively harmful, because the one time the failure is real, nobody believes it.

I have a whole post on flaky tests and I won’t repeat it here, but the short version applies with extra force at the system layer:

• Retry failed tests, but track every retry. A test that fails and passes on retry is still telling you something true about the product; deleting it loses that signal forever. The catch — and this is the part teams skip — is that every retry has to land in a dashboard you actually look at. A retry nobody sees is just a slow way to hide a problem.
• Triage every flake the retry surfaced. Either fix the test, fix the application, or delete the test. Don’t let “it’s just flaky” be the resting state for any test in your suite. The retry dashboard is what makes this triage possible — without it, flakes are invisible and the system layer rots.
• Use the deterministic affordances your test framework gives you. Playwright’s auto-waiting and locator strategies, Appium’s wait-for-element conditions, network mocking at the boundaries where the third-party isn’t what you’re testing. Most flake is avoidable. Most teams have never invested the hour it takes to avoid it on a given test.

One calibration that matters more here than at any lower layer: the more variables a test depends on, the more often something somewhere is going to flake. A unit test has one variable — your code. A system test has dozens — the deployed app, the CDN, the IDP, the network, the browser, the load balancer, every third-party script the page pulls in. Some baseline flake rate at this layer is not a bug in your suite, it’s a property of the universe. If a given test fails-then-retries once or twice a month, that’s noise — log it, move on. If it’s flaking once every few days, that’s signal — stop and fix it, because it’s about to start training your team to ignore failures, and the retry dashboard is showing you exactly which test it is.

A small, fast, deterministic system suite that the team trusts is worth ten times a large, slow, flaky one. Aim for the first. Refuse to settle for the second.

On who owns this layer

The system layer is where the tester-versus-developer trust dynamic I called out in the opener plays out most visibly. The temptation, when a bug ships, is for the tester to add one more system test that would have caught it. Over a couple of years, that habit is how you end up with the inverted pyramid.

The healthier dynamic, the one I’m trying to defend across this whole series:

• When a bug ships, the first question isn’t “what system test should we add?” It’s “what’s the lowest layer that could have caught this?” Most of the time, the honest answer is a unit or integration test that doesn’t exist yet. Add that one. The system test is the fallback, not the default.
• System tests are a shared responsibility — devs write them when the feature genuinely needs them, SDETs maintain the framework and curate the suite. Neither side gets to throw tests over the wall and forget them.
• Pruning the system suite is real, valuable, ongoing work. A senior SDET who deletes ten unnecessary system tests has done more for the team’s release velocity than one who adds ten new ones. Reward the pruning.

The system layer is a shared garden, not a tester landfill. Treat it that way and it stays useful. Treat it like a place to dump every test the team can’t agree on a better home for, and it eats the team.

Where the pyramid sits, six posts in

Six posts in, here’s the thing the whole arc has been pointing at:

The pyramid isn’t dead. It just needed two changes to fit the way modern software actually ships. First, four layers, not three — split out integration from API, because they’re different jobs at different costs on different sides of the line. Second, the line itself — make the pre-deploy / post-deploy boundary explicit, so every test in the suite has to defend its slot relative to it.

That’s it. Everything else above is detail.

If I’ve done my job, you have a model you can put on a whiteboard tomorrow, defend in a strategy meeting next week, and use as a tiebreaker in PR review for the rest of your career. The next time someone announces the pyramid is dead, you’ll have somewhere to point.

What’s next

One thing this whole model assumes — and the “On who owns this layer” section above hinted at — is that developers and testers are actually talking to each other. The four-layer pyramid only works if the team agrees on which layer a given test belongs at, and that agreement only happens if devs and SDETs are operating as one team rather than two adjacent ones throwing tests over a wall. So the next post is about that: what healthy dev-tester communication actually looks like, what breaks when it isn’t there, and the specific habits that turn “the testers” and “the devs” into one group of engineers who happen to specialize differently.

Subscribe, RSS, bookmark — whatever your mechanism is.

Go forth, and test.