Add browser E2E tests for the scenario pages (Playwright, iw-support-e2e-testing) #18

New issue

Open

opened 2026-09-12 17:17:09 +00:00 by mph · 0 comments

mph commented

2026-09-12 17:17:09 +00:00

Owner

Why

The 3.12.0 work verified all 87 components by hand in a browser, and it recorded the result in COMPONENTS.md. That check is not repeatable. ConformanceTest catches a missing tag or attribute against the Web Awesome manifest, but nothing catches a page that renders wrongly, a component that does not upgrade, or a console error. The next Web Awesome upgrade needs a check that CI can run.

What exists today

scenarios/src/Server.scala: object Server extends cask.MainRoutes, override def port: Int = 8086 (hard-coded), initialize() at the end, 84 routes.
scenarios/src/Layout.scala: 79 nav items; private val waCdn = "/wa-pro", served by @cask.staticFiles("/wa-pro") from node_modules/@awesome.me/webawesome-pro/dist-cdn.
Tests: webawesome/jvm/test and webawesome/js/test, both utest. The scenarios module has no test module.
CI (.github/workflows/ci.yml, Forgejo Actions): compile, format, lint, test on JDK 21. No Node, no npm install, no browser.

The support module to use

works.iterative.support::iw-support-e2e-testing, version 0.2.1 (verified in the forge registry: POM and jar both answer 200).

It brings Playwright Java 1.49.0, cucumber-scala 8.23.1, cucumber-junit 7.20.1, Typesafe Config 1.4.3, iw-support-core 0.2.1 and zio 2.1.24, on Scala 3.3.7. Our build already uses Scala 3.3.7 and Mill 1.1.2, so there is no version friction. The suite adds JUnit4 as a second test framework in a new module; it does not change the utest modules.

The build needs the forge repository:

override def repositories = Task {
  super.repositories() ++ Seq("https://code.iterative.works/api/packages/iterative-works/maven")
}

Entry points (package works.iterative.testing.e2e):

Type	Use
`PlaywrightCucumberRunner`	the trait a step class extends; gives `page`, `freshPage()`, `baseUrl`, `config`, `testData`
`CommonStepDefinitions`	about 25 ready Gherkin steps, self-type `PlaywrightCucumberRunner`
`CucumberLifecycleHooks`	`BeforeAll`/`AfterAll`/`After`; it MUST be in the Cucumber glue packages
`PlaywrightTestContext`	holds playwright, browser, context and page for the JVM run
`PageObject` / `BasePageObject`	page-object helpers

Model to copy: iw-support/forms/scenarios/jvm/src/e2e/ (suite entry point, server hooks, step definitions, feature files, application.conf).

Work to do

Make the port configurable. Server.port reads the port from the same place as the tests. In the iw-support pattern, baseUrl in application.conf is the single source, and the hooks derive the port from it.
Make the runtime source configurable. Layout.waCdn becomes configurable, so a page can load Web Awesome either from /wa-pro (local Pro package) or from https://cdn.jsdelivr.net/npm/@awesome.me/webawesome@<version>/dist-cdn (free, public).
Add the test module in build.mill, following the iw-support convention:

object e2e extends ScalaTests with TestModule.Junit4 {
  override def sources = Task.Sources("src/e2e/scala")
  override def resources = Task.Sources("src/e2e/resources")
  def mvnDeps = super.mvnDeps() ++ Seq(
    mvn"works.iterative.support::iw-support-e2e-testing:0.2.1",
    mvn"com.github.sbt:junit-interface:0.13.3"
  )
}

Boot the server in the suite. A hooks object with BeforeAll starts Server in a thread, polls a page until it answers HTTP 200 with a deadline, and AfterAll stops it. Model: ScenariosServerHooks.scala in the iw-support suite.
Write the feature files. Proposed three levels:
- Every page loads. One scenario over a table of all 79 nav routes: the page answers, every wa-* element on it reports :defined, and the console has no error. This alone replaces most of the manual review.
- Behaviour of the interactive components: dialog and drawer open and close, dropdown and submenu, toast.create(), tabs, tree, accordion, pagination, combobox filtering and allowCreate, data grid sort and search, slider, colour picker, carousel.
- The known upstream defects (the changelog section "Known upstream issues"): each one gets a scenario that states the present behaviour, so an upstream fix shows up as a failing test.
Wire CI. A job installs a browser (npx playwright install --with-deps chromium) and runs the module. Decide whether the module belongs inside ./mill __.test or outside it, because a browser job is slower and needs more setup than the other jobs.
Keep the artifacts. Upload the screenshots and the JUnit XML report on failure.

Decision: which runtime does CI render?

Free from jsdelivr (or from public npm): no token, and it covers the 70 free components. The 17 Pro components then stay uncovered, or the suite tags them and skips them.
Pro from the private registry: it covers all 87 components, but CI needs the Cloudsmith token (.npmrc) and an npm ci step.
Both are possible: the free source by default, and the Pro source when the token is present.

Evidence: @awesome.me/webawesome@3.12.0 is public (registry.npmjs.org serves the tarball, and jsdelivr answers 200 for dist-cdn/webawesome.loader.js). The Pro package needs the token.

Traps in the support module

e2e-testing/EXAMPLE.md documents a ZIO API (E2ETestLayers, ZIOPageObject, withPage) that does not exist in the source. Use README.md and USAGE_GUIDE.md.
The Cucumber glue array must contain works.iterative.testing.e2e, or the lifecycle hooks never run.
headless defaults to true. channel = "chrome" (with a PLAYWRIGHT_CHANNEL override) is needed on hosts where the downloaded chromium cannot run, for example nix-profile machines.
The module has no tracing support, and no automatic screenshot on failure. An After hook must add the screenshot if we want one.
E2ETestConfig.timeout is read from the config but never applied to the page.
The suite in iw-support is not in CI yet, so we are the first to wire it.

Definition of done

./mill scenarios.e2e passes locally, headless.
CI runs the suite and keeps the report and the screenshots.
All 79 routes are covered by the load check, and the interactive components have behaviour scenarios.
CONTRIBUTING.md tells how to run the suite and how to add a scenario.
COMPONENTS.md states which check is automatic now, so the manual Review column stops being the only evidence.

## Why The 3.12.0 work verified all 87 components by hand in a browser, and it recorded the result in `COMPONENTS.md`. That check is not repeatable. `ConformanceTest` catches a missing tag or attribute against the Web Awesome manifest, but nothing catches a page that renders wrongly, a component that does not upgrade, or a console error. The next Web Awesome upgrade needs a check that CI can run. ## What exists today - `scenarios/src/Server.scala`: `object Server extends cask.MainRoutes`, `override def port: Int = 8086` (hard-coded), `initialize()` at the end, 84 routes. - `scenarios/src/Layout.scala`: 79 nav items; `private val waCdn = "/wa-pro"`, served by `@cask.staticFiles("/wa-pro")` from `node_modules/@awesome.me/webawesome-pro/dist-cdn`. - Tests: `webawesome/jvm/test` and `webawesome/js/test`, both utest. The `scenarios` module has no test module. - CI (`.github/workflows/ci.yml`, Forgejo Actions): compile, format, lint, test on JDK 21. No Node, no npm install, no browser. ## The support module to use `works.iterative.support::iw-support-e2e-testing`, version `0.2.1` (verified in the forge registry: POM and jar both answer 200). It brings Playwright Java 1.49.0, cucumber-scala 8.23.1, cucumber-junit 7.20.1, Typesafe Config 1.4.3, iw-support-core 0.2.1 and zio 2.1.24, on Scala 3.3.7. Our build already uses Scala 3.3.7 and Mill 1.1.2, so there is no version friction. The suite adds JUnit4 as a second test framework in a new module; it does not change the utest modules. The build needs the forge repository: ```scala override def repositories = Task { super.repositories() ++ Seq("https://code.iterative.works/api/packages/iterative-works/maven") } ``` Entry points (package `works.iterative.testing.e2e`): | Type | Use | |---|---| | `PlaywrightCucumberRunner` | the trait a step class extends; gives `page`, `freshPage()`, `baseUrl`, `config`, `testData` | | `CommonStepDefinitions` | about 25 ready Gherkin steps, self-type `PlaywrightCucumberRunner` | | `CucumberLifecycleHooks` | `BeforeAll`/`AfterAll`/`After`; it MUST be in the Cucumber glue packages | | `PlaywrightTestContext` | holds playwright, browser, context and page for the JVM run | | `PageObject` / `BasePageObject` | page-object helpers | Model to copy: `iw-support/forms/scenarios/jvm/src/e2e/` (suite entry point, server hooks, step definitions, feature files, `application.conf`). ## Work to do 1. **Make the port configurable.** `Server.port` reads the port from the same place as the tests. In the iw-support pattern, `baseUrl` in `application.conf` is the single source, and the hooks derive the port from it. 2. **Make the runtime source configurable.** `Layout.waCdn` becomes configurable, so a page can load Web Awesome either from `/wa-pro` (local Pro package) or from `https://cdn.jsdelivr.net/npm/@awesome.me/webawesome@<version>/dist-cdn` (free, public). 3. **Add the test module** in `build.mill`, following the iw-support convention: ```scala object e2e extends ScalaTests with TestModule.Junit4 { override def sources = Task.Sources("src/e2e/scala") override def resources = Task.Sources("src/e2e/resources") def mvnDeps = super.mvnDeps() ++ Seq( mvn"works.iterative.support::iw-support-e2e-testing:0.2.1", mvn"com.github.sbt:junit-interface:0.13.3" ) } ``` 4. **Boot the server in the suite.** A hooks object with `BeforeAll` starts `Server` in a thread, polls a page until it answers HTTP 200 with a deadline, and `AfterAll` stops it. Model: `ScenariosServerHooks.scala` in the iw-support suite. 5. **Write the feature files.** Proposed three levels: - **Every page loads.** One scenario over a table of all 79 nav routes: the page answers, every `wa-*` element on it reports `:defined`, and the console has no error. This alone replaces most of the manual review. - **Behaviour of the interactive components:** dialog and drawer open and close, dropdown and submenu, `toast.create()`, tabs, tree, accordion, pagination, combobox filtering and `allowCreate`, data grid sort and search, slider, colour picker, carousel. - **The known upstream defects** (the changelog section "Known upstream issues"): each one gets a scenario that states the present behaviour, so an upstream fix shows up as a failing test. 6. **Wire CI.** A job installs a browser (`npx playwright install --with-deps chromium`) and runs the module. Decide whether the module belongs inside `./mill __.test` or outside it, because a browser job is slower and needs more setup than the other jobs. 7. **Keep the artifacts.** Upload the screenshots and the JUnit XML report on failure. ## Decision: which runtime does CI render? - **Free from jsdelivr** (or from public npm): no token, and it covers the 70 free components. The 17 Pro components then stay uncovered, or the suite tags them and skips them. - **Pro from the private registry:** it covers all 87 components, but CI needs the Cloudsmith token (`.npmrc`) and an `npm ci` step. - Both are possible: the free source by default, and the Pro source when the token is present. Evidence: `@awesome.me/webawesome@3.12.0` is public (`registry.npmjs.org` serves the tarball, and jsdelivr answers 200 for `dist-cdn/webawesome.loader.js`). The Pro package needs the token. ## Traps in the support module - `e2e-testing/EXAMPLE.md` documents a ZIO API (`E2ETestLayers`, `ZIOPageObject`, `withPage`) that **does not exist** in the source. Use `README.md` and `USAGE_GUIDE.md`. - The Cucumber `glue` array must contain `works.iterative.testing.e2e`, or the lifecycle hooks never run. - `headless` defaults to true. `channel = "chrome"` (with a `PLAYWRIGHT_CHANNEL` override) is needed on hosts where the downloaded chromium cannot run, for example nix-profile machines. - The module has no tracing support, and no automatic screenshot on failure. An `After` hook must add the screenshot if we want one. - `E2ETestConfig.timeout` is read from the config but never applied to the page. - The suite in iw-support is not in CI yet, so we are the first to wire it. ## Definition of done - `./mill scenarios.e2e` passes locally, headless. - CI runs the suite and keeps the report and the screenshots. - All 79 routes are covered by the load check, and the interactive components have behaviour scenarios. - `CONTRIBUTING.md` tells how to run the suite and how to add a scenario. - `COMPONENTS.md` states which check is automatic now, so the manual Review column stops being the only evidence.