Web App Testing with Playwright

---
name: webapp-testing-playwright
description: Drives a local web app in a real browser with Playwright to prove a flow actually works - clicking through the real steps, capturing console and network errors, and saving screenshots as evidence. Use to verify a UI change end to end, reproduce a UI bug, or check whether a page is throwing at runtime.
license: Apache-2.0
metadata:
  source: anthropics/skills
---

# Web App Testing with Playwright

You verify user interface behavior by driving a real browser, not by reading the code and reasoning about what should happen. A flow is working when a browser completed it and the console stayed quiet.

Playwright is a browser automation library that controls Chromium, Firefox and WebKit from a script.

**Language.** The examples here use the Python sync API (`playwright.sync_api`), which is what the bundled helper script expects. The JavaScript API has the same methods in camelCase:

| Python | JavaScript |
| --- | --- |
| `page.get_by_role("button", name="Sign in")` | `page.getByRole('button', { name: 'Sign in' })` |
| `page.wait_for_load_state("networkidle")` | `page.waitForLoadState('networkidle')` |
| `page.on("console", handler)` | `page.on('console', handler)` |
| `expect(locator).to_be_visible()` | `await expect(locator).toBeVisible()` |

Pick whichever matches the project. Never introduce a second language into a repository just to run a test.

## Workflow

1. **Get the flow in user steps.** "Open /signup, fill email and password, submit, expect the dashboard." If the request is vaguer than that, ask what counts as success before writing a line.
2. **Get the app running and reachable.** Confirm the URL responds before automating. A run against a dead server wastes the whole cycle.
3. **Reconnoiter.** On any page you have not automated before, load it, wait, screenshot, and list the interactive elements. Discover selectors from the rendered page rather than guessing from source.
4. **Write the script one step at a time.** Each user action followed immediately by the assertion that proves it worked.
5. **Capture console messages and failed requests** for the whole run, from before the first navigation.
6. **Screenshot the states your assertions depend on**, plus the failure state if there is one.
7. **Run it, headless.** Fix the script if it is the script that is broken; report the app if the app is broken.
8. **Report a verdict with evidence.** Pass or fail per flow, the failing step, console and network output, screenshot paths.

```text
Browser test checklist
[ ] Flow written down as user steps with a success condition
[ ] App reachable at a known URL before the run
[ ] Selectors discovered from the rendered page, by role/label/text
[ ] Every action followed by an assertion on the result
[ ] Console and network listeners attached before the first navigation
[ ] Screenshots taken at the asserted states
[ ] Verdict reported with the first failing step named
```

## Choosing the approach

```text
Is the page static HTML with no client-side rendering?
  yes -> read the file, take selectors from the markup, drive it over a file:// URL
  no  -> is the dev server already running?
           no  -> start it with scripts/with_server.py
           yes -> reconnoiter first, then act
```

## Starting the app

`scripts/with_server.py` starts one or more servers, waits until each accepts connections, runs your automation, then shuts everything down even if the automation fails.

Run `python scripts/with_server.py --help` before using it, and treat it as a black box. Read its source only if you need to change how it behaves.

```bash
python scripts/with_server.py --server "npm run dev" --port 5173 -- python flow_login.py

python scripts/with_server.py \
  --server "python -m app.server" --port 8000 \
  --server "npm run dev" --port 5173 \
  -- python flow_checkout.py
```

Your automation script then assumes the server is up and contains only browser logic. If a server never starts, the helper exits with code 3 and prints the path to that server's log; read the log before blaming the test.

## Reconnaissance before action

On an unfamiliar page, look before you click:

```python
page.goto("http://localhost:5173/signup")
page.wait_for_load_state("networkidle")
page.screenshot(path="artifacts/signup-initial.png", full_page=True)

for button in page.get_by_role("button").all():
    print("button:", button.inner_text())
for field in page.locator("input, textarea, select").all():
    print("field:", field.get_attribute("name"), field.get_attribute("type"))
```

`networkidle` is the right blunt instrument for reconnaissance. In the real flow, wait for the specific thing you care about instead: an element, a response, a URL change.

## Selectors

Prefer handles that describe what a user sees, in this order:

1. `get_by_role("button", name="Add to cart")` - role and accessible name
2. `get_by_label("Email address")` - form fields by their label
3. `get_by_text("Order confirmed")` - visible copy
4. `get_by_test_id("cart-badge")` - an explicit test hook, if the project uses them
5. CSS or XPath - last resort, and only when nothing above works

Role and label selectors survive refactors and quietly check accessibility at the same time: if `get_by_label` cannot find a field, that field has no label and a screen reader user cannot use it either. A selector like `div.sc-bdVaJa > span:nth-child(3)` is broken the moment someone reorders the markup.

## Console and network capture

Attach the listeners before the first `goto`, or the messages from initial load are lost.

```python
console_errors, failed_requests = [], []

page.on("console", lambda msg:
        console_errors.append(f"[{msg.type}] {msg.text}") if msg.type in ("error", "warning") else None)
page.on("pageerror", lambda err: console_errors.append(f"[pageerror] {err}"))
page.on("requestfailed", lambda req:
        failed_requests.append(f"{req.method} {req.url} - {req.failure}"))
page.on("response", lambda res:
        failed_requests.append(f"{res.status} {res.url}") if res.status >= 400 else None)
```

A run with a red console is a fail unless you can name why that specific message is harmless. Write the reason in the report; do not just drop it.

## What counts as a pass

- The asserted element or text is present and visible, not merely in the DOM.
- No console errors and no `pageerror` events, or a written justification for each one.
- No 4xx or 5xx responses on the flow's own requests.
- The screenshot shows the state the assertion claims.

"The script ran without raising" is not a pass. A page that navigates and then renders an error screen will run without raising.

## Report format

```text
Flow:        add to cart from a product page
Verdict:     FAIL
Steps:       1 navigate /p/123          ok
             2 click "Add to cart"      ok
             3 cart badge reads "1"     ok
             4 console clean            FAIL
Console:     [error] TypeError: cart.items is undefined (cart.js:44)
Network:     all 200
Screenshots: artifacts/cart-after-click.png
Repro:       load /p/123, click "Add to cart", watch the console
```

## Example

**Request.** "Check that login works: /login, email and password, submit, expect the dashboard greeting."

**Script** (`flow_login.py`, run through the server helper):

```python
from playwright.sync_api import sync_playwright, expect

console_errors = []

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page(viewport={"width": 1440, "height": 900})
    page.on("console", lambda m: console_errors.append(f"[{m.type}] {m.text}")
            if m.type == "error" else None)
    page.on("pageerror", lambda e: console_errors.append(f"[pageerror] {e}"))

    try:
        page.goto("http://localhost:5173/login")
        page.get_by_label("Email address").fill("test.user@example.com")
        page.get_by_label("Password").fill("correct-horse-battery")
        page.screenshot(path="artifacts/login-filled.png")

        with page.expect_response("**/api/session") as response_info:
            page.get_by_role("button", name="Sign in").click()
        assert response_info.value.status == 200, f"session returned {response_info.value.status}"

        expect(page.get_by_role("heading", name="Welcome back, Test")).to_be_visible(timeout=5000)
        page.screenshot(path="artifacts/login-dashboard.png", full_page=True)
        print("PASS" if not console_errors else "FAIL (console)")
    finally:
        for line in console_errors:
            print("console:", line)
        browser.close()
```

**Run:**

```bash
python scripts/with_server.py --server "npm run dev" --port 5173 -- python flow_login.py
```

**Report:**

```text
Flow:        login with valid credentials
Verdict:     PASS
Steps:       1 navigate /login              ok
             2 fill email and password      ok
             3 submit -> POST /api/session  200
             4 dashboard heading visible    ok
             5 console clean                ok
Screenshots: artifacts/login-filled.png, artifacts/login-dashboard.png
```

The submit step waits on the actual session response rather than on a timer, so a slow machine changes the duration of the test and not its result.

## Common mistakes

- Asserting the page loaded instead of asserting the action worked.
- `wait_for_timeout` sprinkled around to paper over races. It is flaky by construction: too short on a loaded machine, wasted time on a fast one.
- Selecting by a generated class name or `nth-child`, which the next refactor renames.
- Ignoring the console, so a green run hides an exception every real user will hit.
- Screenshotting after the page has moved on, which produces evidence of the wrong moment.
- One flow depending on state another flow left behind. Each flow seeds what it needs and cleans up after itself.
- Reading a helper script's source to learn what it does when `--help` would have answered it.

## Edge cases

- **Authentication in the way.** Log in once, save storage state with `context.storage_state(path=...)`, and reuse it. Do not automate the login form at the start of every unrelated flow.
- **The element genuinely is not there.** Before reporting a bug, screenshot and dump `page.content()`. A missing element and a failed render look identical from a selector error.
- **Third-party widgets and iframes.** Use `frame_locator` for the frame. Console noise from a third-party script is still reported, but it can be justified in the report rather than counted against the flow.
- **Anything destructive.** Never run these tests against production, and never let a flow send real payments, real email, or real webhooks. Local or staging with test data only.
- **Flake you cannot explain.** Rerun ten times and record the rate before calling it flaky. Then find the race; a retry that hides it is a decision to ship the race.
- **No dev server, only a built bundle.** Serve the build directory over a static server and point the helper at it. Testing a `file://` bundle that expects a base path will mislead you.
- **The script is what is broken.** If the script fails on step one against a page that clearly works in a browser, fix the script. Report app bugs only when the browser reproduces them by hand.

## Files

- `scripts/with_server.py` - run this, do not read it. Starts servers, waits for their ports, runs your automation, and shuts everything down afterward. Start with `python scripts/with_server.py --help`.
- `references/examples.md` - three complete runs to read before writing your own: a failing add-to-cart flow with a console error, reconnaissance on an unfamiliar page, and a static HTML page driven over a `file://` URL.

## Attribution

Adapted from the `webapp-testing` skill in [anthropics/skills](https://github.com/anthropics/skills/tree/main/skills/webapp-testing), Apache-2.0. The decision tree, the reconnaissance-then-action pattern and the server helper come from there; the selector guidance, capture harness and report format are ours.

# Worked runs

Three complete runs: the script, what it printed, and the report that went back to the requester. The third one needs no server at all.

## Contents

- [Run 1: add to cart, and why it failed](#run-1-add-to-cart-and-why-it-failed)
- [Run 2: reconnaissance on an unfamiliar page](#run-2-reconnaissance-on-an-unfamiliar-page)
- [Run 3: a static HTML page over file://](#run-3-a-static-html-page-over-file)
- [Habits worth copying](#habits-worth-copying)

---

## Run 1: add to cart, and why it failed

**Request.** "Verify add to cart works on the product page."

**Flow in user steps.** Open `/p/123`, click "Add to cart", the cart badge should read "1".

### Script

```python
# flow_add_to_cart.py
from playwright.sync_api import sync_playwright, expect

console, network = [], []

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page(viewport={"width": 1440, "height": 900})

    # Listeners go on before the first navigation, or load-time errors are lost.
    page.on("console", lambda m: console.append(f"[{m.type}] {m.text}") if m.type == "error" else None)
    page.on("pageerror", lambda e: console.append(f"[pageerror] {e}"))
    page.on("response", lambda r: network.append(f"{r.status} {r.url}") if r.status >= 400 else None)

    failures = []
    try:
        page.goto("http://localhost:5173/p/123")
        expect(page.get_by_role("heading", name="Trail Runner 4")).to_be_visible()

        page.get_by_role("button", name="Add to cart").click()

        badge = page.get_by_test_id("cart-badge")
        expect(badge).to_have_text("1", timeout=5000)
        page.screenshot(path="artifacts/cart-after-click.png", full_page=True)
    except AssertionError as exc:
        failures.append(str(exc))
        page.screenshot(path="artifacts/cart-failure.png", full_page=True)
    finally:
        if console:
            failures.append(f"{len(console)} console error(s)")
        print("VERDICT:", "PASS" if not failures else "FAIL")
        for line in failures:
            print("  failure:", line)
        for line in console:
            print("  console:", line)
        for line in network:
            print("  network:", line)
        browser.close()
```

### Output

```text
VERDICT: FAIL
  failure: 1 console error(s)
  console: [pageerror] TypeError: cart.items is undefined
      at updateMiniCart (cart.js:44:19)
      at HTMLButtonElement.<anonymous> (cart.js:12:5)
```

The badge did read "1", and to a human clicking through, the page looked fine. The failure is real anyway: `updateMiniCart` threw, so the mini-cart never rendered, and the same function runs on the checkout page.

### Report

```text
Flow:        add to cart from the product page
Verdict:     FAIL
Steps:       1 navigate /p/123               ok
             2 product heading visible       ok
             3 click "Add to cart"           ok
             4 cart badge reads "1"          ok
             5 console clean                 FAIL
Console:     [pageerror] TypeError: cart.items is undefined (cart.js:44)
Network:     no 4xx or 5xx
Screenshots: artifacts/cart-after-click.png
Repro:       load /p/123, click "Add to cart", read the console
Note:        the visible outcome is correct; the mini-cart render throws. Reported as a
             fail because updateMiniCart is shared with checkout.
```

**Why the verdict is FAIL.** The assertion the requester asked for passed. Reporting PASS would have shipped a live exception on a shared code path, so the report states both facts and lets the reader see the disagreement.

---

## Run 2: reconnaissance on an unfamiliar page

**Request.** "Test the settings page saves the display name." Nobody knows what the form looks like from the outside, and the markup is generated.

### Script

```python
# recon_settings.py
from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page(viewport={"width": 1440, "height": 900})
    try:
        page.goto("http://localhost:5173/settings")
        page.wait_for_load_state("networkidle")   # blunt, but right for a first look
        page.screenshot(path="artifacts/settings.png", full_page=True)

        print("--- buttons")
        for b in page.get_by_role("button").all():
            print(f"  {b.inner_text()!r} visible={b.is_visible()}")

        print("--- labeled fields")
        for label in page.locator("label").all():
            text = label.inner_text().strip()
            control = label.get_attribute("for")
            print(f"  {text!r} -> #{control}")

        print("--- headings")
        for h in page.get_by_role("heading").all():
            print(f"  {h.inner_text()!r}")
    finally:
        browser.close()
```

### Output

```text
--- buttons
  'Save changes' visible=True
  'Cancel' visible=True
  '' visible=True
--- labeled fields
  'Display name' -> #field-a41
  'Time zone' -> #field-a42
--- headings
  'Account settings'
  'Profile'
```

Three things fall out of this and none of them were guessable from the source:

1. The field ids are generated (`field-a41`), so selecting by id would break on the next build. `get_by_label("Display name")` is stable.
2. There is a button with no accessible name. That is a real accessibility defect worth reporting even though it was not the assignment.
3. There is no visible success message in the initial state, so the test needs a different success signal.

### The flow test that followed

```python
page.get_by_label("Display name").fill("Ada L")
with page.expect_response("**/api/profile") as res:
    page.get_by_role("button", name="Save changes").click()
assert res.value.status == 200, f"profile save returned {res.value.status}"

page.reload()
expect(page.get_by_label("Display name")).to_have_value("Ada L")
```

Reloading and re-reading the field is the strongest available check that the value was actually saved rather than only shown. A toast that appears and fades proves the front end fired, not that the server stored anything.

---

## Run 3: a static HTML page over file://

Some pages have no server at all: an exported report, an email template, a standalone form. Playwright drives them over a `file://` URL, which makes them the cheapest thing in the suite to test.

### Script

```python
# flow_static_form.py
from pathlib import Path
from playwright.sync_api import sync_playwright, expect

page_path = Path("dist/contact.html").resolve()
if not page_path.exists():
    raise SystemExit(f"build the page first: {page_path} does not exist")

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    try:
        page.goto(page_path.as_uri())

        # Submitting empty should surface the browser's own validation.
        page.get_by_role("button", name="Send").click()
        email = page.get_by_label("Your email")
        assert email.evaluate("el => !el.validity.valid"), "empty email was accepted"

        email.fill("not-an-email")
        page.get_by_role("button", name="Send").click()
        assert email.evaluate("el => el.validity.typeMismatch"), "bad email format was accepted"

        email.fill("ada@example.com")
        page.get_by_label("Message").fill("Hello")
        page.get_by_role("button", name="Send").click()
        expect(page.get_by_text("Thanks - we will reply within two working days")).to_be_visible()

        page.screenshot(path="artifacts/contact-success.png", full_page=True)
        print("VERDICT: PASS")
    finally:
        browser.close()
```

### Output

```text
VERDICT: PASS
```

Points worth stealing:

- The script checks the file exists and exits with a clear message instead of failing inside the browser with something unreadable.
- Both invalid cases are asserted before the valid one, so a form that accepts anything fails loudly rather than passing on the happy path.
- Validity is read from the element itself rather than from an error message, which keeps the test working when the copy changes.

---

## Habits worth copying

1. Listeners attached before the first navigation, every time.
2. Every action paired with the assertion that proves it worked, before the next action.
3. Waiting on a response or an element, never on a duration.
4. Screenshots named for the state they show, not `1.png` and `2.png`.
5. Reports that name the first failing step and hand over a reproduction the reader can run by hand.
6. Findings outside the assignment, such as the unnamed button in run 2, reported separately rather than folded into the verdict.

#!/usr/bin/env python3
"""Start one or more dev servers, wait until they accept connections, run a command, shut them down.

Adapted from the webapp-testing skill in anthropics/skills (Apache-2.0). Server output is
written to log files instead of pipes, so a chatty server cannot fill a pipe buffer and hang
the run, and so the log is readable when a server fails to come up.

Usage:
    # one server
    python scripts/with_server.py --server "npm run dev" --port 5173 -- python flow_login.py

    # backend plus frontend
    python scripts/with_server.py \
      --server "python -m app.server" --port 8000 \
      --server "npm run dev" --port 5173 \
      -- python flow_checkout.py

Exit codes:
    0-125  the exit code of the command that was run
    2      bad arguments
    3      a server did not start within the timeout (its log path is printed)
"""

import argparse
import os
import socket
import subprocess
import sys
import tempfile
import time

EXIT_BAD_ARGS = 2
EXIT_SERVER_FAILED = 3
POLL_INTERVAL_SECONDS = 0.5
CONNECT_TIMEOUT_SECONDS = 1.0
SHUTDOWN_GRACE_SECONDS = 5


def wait_for_port(host, port, timeout_seconds):
    """Poll until something accepts TCP connections on host:port. True if it came up."""
    deadline = time.monotonic() + timeout_seconds
    while time.monotonic() < deadline:
        try:
            with socket.create_connection((host, port), timeout=CONNECT_TIMEOUT_SECONDS):
                return True
        except OSError:
            time.sleep(POLL_INTERVAL_SECONDS)
    return False


def start_server(command, log_path):
    """Launch one server with its output redirected to log_path."""
    log_file = open(log_path, "w", encoding="utf-8")
    try:
        process = subprocess.Popen(
            command,
            shell=True,
            stdout=log_file,
            stderr=subprocess.STDOUT,
        )
    except OSError as exc:
        log_file.close()
        raise RuntimeError(f"could not launch {command!r}: {exc}") from exc
    return process, log_file


def stop_server(process, log_file, label):
    """Terminate a server, escalating to kill, and always close its log handle."""
    try:
        if process.poll() is None:
            process.terminate()
            try:
                process.wait(timeout=SHUTDOWN_GRACE_SECONDS)
            except subprocess.TimeoutExpired:
                process.kill()
                process.wait()
        print(f"stopped {label}", file=sys.stderr)
    except OSError as exc:
        print(f"could not stop {label}: {exc}", file=sys.stderr)
    finally:
        log_file.close()


def main():
    parser = argparse.ArgumentParser(
        description="Run a command with one or more servers running.",
        epilog="Ports are matched to servers in the order the flags appear.",
    )
    parser.add_argument("--server", action="append", dest="servers", required=True,
                        help="server command; repeat for more than one server")
    parser.add_argument("--port", action="append", dest="ports", type=int, required=True,
                        help="port each server listens on; repeat to match --server")
    parser.add_argument("--host", default="127.0.0.1", help="host to poll (default 127.0.0.1)")
    parser.add_argument("--timeout", type=int, default=60,
                        help="seconds to wait for each server (default 60)")
    parser.add_argument("--log-dir", default=None,
                        help="where to write server logs (default: a new temp directory)")
    parser.add_argument("command", nargs=argparse.REMAINDER,
                        help="-- followed by the command to run once servers are up")

    args = parser.parse_args()

    command = args.command[1:] if args.command and args.command[0] == "--" else args.command
    if not command:
        parser.error("no command given after --")
    if len(args.servers) != len(args.ports):
        parser.error(f"got {len(args.servers)} --server and {len(args.ports)} --port; they must match")

    log_dir = args.log_dir or tempfile.mkdtemp(prefix="with_server_")
    os.makedirs(log_dir, exist_ok=True)

    running = []
    try:
        for index, (server_command, port) in enumerate(zip(args.servers, args.ports), start=1):
            log_path = os.path.join(log_dir, f"server{index}_port{port}.log")
            print(f"starting server {index}: {server_command} (log: {log_path})", file=sys.stderr)
            process, log_file = start_server(server_command, log_path)
            running.append((process, log_file, f"server {index} on port {port}"))

            if not wait_for_port(args.host, port, args.timeout):
                print(
                    f"server {index} did not accept connections on {args.host}:{port} "
                    f"within {args.timeout}s. Log: {log_path}",
                    file=sys.stderr,
                )
                return EXIT_SERVER_FAILED
            print(f"server {index} ready on port {port}", file=sys.stderr)

        print(f"running: {' '.join(command)}", file=sys.stderr)
        return subprocess.run(command).returncode
    except RuntimeError as exc:
        print(str(exc), file=sys.stderr)
        return EXIT_SERVER_FAILED
    except KeyboardInterrupt:
        print("interrupted", file=sys.stderr)
        return EXIT_SERVER_FAILED
    finally:
        for process, log_file, label in reversed(running):
            stop_server(process, log_file, label)


if __name__ == "__main__":
    sys.exit(main())