Assertion-discipline guidance: three ways a green test proves nothing #461

New issue

Open

opened 2026-08-27 06:39:32 +00:00 by mph · 1 comment

mph commented

2026-08-27 06:39:32 +00:00

Owner

A suffix mutation cannot falsify a 'contains' assertion.
Proving a column-headers test could fail, the message key was mutated 'zakazky.overview.column.fazovaLhuta' -> '...fazovaLhutaMUTANT'. The suite stayed green at 74/74: the mutated string still contains the original as a substring, so html.contains(original) remained true. Re-mutating to 'zakazky.overview.column.MUTANT' turned exactly one test red. Rule: when the assertion is a substring check, only a mutation that is not a superstring of the pinned value proves anything.
A fixture that writes the read model directly proves the renderer, not the path.
A scenario ('waiting on an external authority does not raise risk') passed in E2E and by hand in the browser, while the state it asserted on was, at that time, unreachable through the application's own commands - the fixture seeded the read model instead of issuing commands. Nothing in the run distinguished 'the feature works' from 'only the fixture works'. Rule: when a fixture seeds by writing a repository, say so in the feature file, so a green run is not read as evidence the path exists.
A filter can be unfalsifiable by construction, whatever the assertion says.
A 'mine only' toggle was tested by a user whose role was absent from fullAccessRoles. Per-record authorization grants such a user exactly the records that the toggle's own definition calls 'mine' - so the visible set is a subset of 'mine' and the toggle can never remove a row. The scenario was vacuous no matter how it was written. Two conditions had to hold to fix it: the 'someone else's' row had to be an entity type the user really can see with the toggle off, and the scenario had to assert the count BOTH before and after the toggle. A single post-toggle count cannot distinguish 'the filter removed it' from 'it was never there'.

The general form of (3) is worth stating on its own: before writing a filter scenario, check whether the authorization already implies the filter's predicate for the chosen user. If it does, no assertion can save the scenario.

Type: feature
Filed from: /home/mph/Devel/projects/medeca-modul-poptavky-MEDECA-378 on branch MEDECA-378

From MEDECA-378 triage - three separate discoveries in one issue, all the same shape: a test that passed and could not have failed. Proposing they join the existing 'prove RED' guidance as named cases. 1. A suffix mutation cannot falsify a 'contains' assertion. Proving a column-headers test could fail, the message key was mutated 'zakazky.overview.column.fazovaLhuta' -> '...fazovaLhutaMUTANT'. The suite stayed green at 74/74: the mutated string still contains the original as a substring, so html.contains(original) remained true. Re-mutating to 'zakazky.overview.column.MUTANT' turned exactly one test red. Rule: when the assertion is a substring check, only a mutation that is not a superstring of the pinned value proves anything. 2. A fixture that writes the read model directly proves the renderer, not the path. A scenario ('waiting on an external authority does not raise risk') passed in E2E and by hand in the browser, while the state it asserted on was, at that time, unreachable through the application's own commands - the fixture seeded the read model instead of issuing commands. Nothing in the run distinguished 'the feature works' from 'only the fixture works'. Rule: when a fixture seeds by writing a repository, say so in the feature file, so a green run is not read as evidence the path exists. 3. A filter can be unfalsifiable by construction, whatever the assertion says. A 'mine only' toggle was tested by a user whose role was absent from fullAccessRoles. Per-record authorization grants such a user exactly the records that the toggle's own definition calls 'mine' - so the visible set is a subset of 'mine' and the toggle can never remove a row. The scenario was vacuous no matter how it was written. Two conditions had to hold to fix it: the 'someone else's' row had to be an entity type the user really can see with the toggle off, and the scenario had to assert the count BOTH before and after the toggle. A single post-toggle count cannot distinguish 'the filter removed it' from 'it was never there'. The general form of (3) is worth stating on its own: before writing a filter scenario, check whether the authorization already implies the filter's predicate for the chosen user. If it does, no assertion can save the scenario. --- **Type:** feature **Filed from:** `/home/mph/Devel/projects/medeca-modul-poptavky-MEDECA-378` on branch `MEDECA-378`

claude commented

2026-10-16 00:38:57 +00:00

Member

Intake: needs-input

1. Where must the three assertion-discipline cases go: dev-docs, iw-cli, or both?

The issue says to add three named cases to "the existing 'prove RED' guidance". That guidance is in the dev-docs repository, in the shared implementation-workflow snippet blueprints/snippets/assertion_discipline.md. iw-cli has no "prove RED" text. Its docs/testing.md has only an iw-cli section, "Asserting a rejection". The examples come from MEDECA (feature files, read-model fixtures, role-based authorization), and iw-cli has none of these.

Prove RED by mutating the exact value the assertion pins and observe the failure

a) Move to dev-docs. File the issue again on dev-docs, and close IW-461 here. The cases then reach every project's implementation workflow.
b) Keep it in iw-cli. Add the cases to iw-cli docs/testing.md. Only iw-cli readers see them, and cases 2 and 3 have no iw-cli example.
c) Both. Add the general rules in dev-docs, and only the substring-mutation case in iw-cli docs/testing.md.

Recommended: a): the guidance the issue names is in dev-docs, and the examples are not about iw-cli.

Answer in a comment on this issue, per number, for example "1: a", or in your own words; intake reads it on its next iteration.

Intake: needs-input ### 1. Where must the three assertion-discipline cases go: dev-docs, iw-cli, or both? The issue says to add three named cases to "the existing 'prove RED' guidance". That guidance is in the dev-docs repository, in the shared implementation-workflow snippet [blueprints/snippets/assertion_discipline.md](https://code.iterative.works/iterative-works/dev-docs/src/branch/main/blueprints/snippets/assertion_discipline.md#L1-L6). iw-cli has no "prove RED" text. Its [docs/testing.md](https://code.iterative.works/iterative-works/iw-cli/src/branch/main/docs/testing.md#L136-L150) has only an iw-cli section, "Asserting a rejection". The examples come from MEDECA (feature files, read-model fixtures, role-based authorization), and iw-cli has none of these. > Prove RED by mutating the exact value the assertion pins and observe the failure - **a) Move to dev-docs.** File the issue again on dev-docs, and close IW-461 here. The cases then reach every project's implementation workflow. - **b) Keep it in iw-cli.** Add the cases to iw-cli `docs/testing.md`. Only iw-cli readers see them, and cases 2 and 3 have no iw-cli example. - **c) Both.** Add the general rules in dev-docs, and only the substring-mutation case in iw-cli `docs/testing.md`. **Recommended: a)**: the guidance the issue names is in dev-docs, and the examples are not about iw-cli. Answer in a comment on this issue, per number, for example "1: a", or in your own words; intake reads it on its next iteration.

claude added the

needs-input

label

2026-10-16 00:38:57 +00:00