Testing

Soft assertions

By default a failure halts the test immediately. Soft assertions collect failures and raise them together at the end, so one run reports every problem.

Each collected failure carries its file:line, so you can jump straight to the assertion that failed:

from assertpy2 import assert_that, soft_assertions

with soft_assertions():
    assert_that("foo").is_length(4)
    assert_that("foo").is_empty()
    assert_that("foo").is_equal_to("bar")

Collected failures raised at the end of the block

assertpy2.AssertionFailure: soft assertion failures:
1. Expected <foo> to be of length <4>, but was <3>.  [test_str.py:12]
2. Expected <foo> to be empty string, but was not.  [test_str.py:13]
3. Expected <foo> to be equal to <bar>, but was not.  [test_str.py:14]

Note

Only assertion failures are collected. Errors like TypeError/ValueError and an explicit fail() halt immediately. Use soft_fail() to collect a forced failure. Soft state is thread-safe and async-safe (independent per thread and per asyncio.Task via contextvars).

The message is a rendering, not the only copy. The raised AssertionFailure carries failures, one record per collected failure, with everything the text had flattened: values, diff, group label and the (file, line) it was collected at. See What a soft block hands back.

An exception from inside the block still wins, and it is left exactly as it was raised: same type, same message, same traceback, so except TimeoutError around the block keeps working. What the block had already collected travels with it as an exception note rather than being dropped:

TimeoutError: service did not answer

soft assertion failures:
1. Expected <1> to be equal to <2>, but was not.  [test_orders.py:14]

On Python 3.10 the failures are attached the same way and stay reachable as exc.__notes__, but that interpreter's traceback does not print notes, so there they are read rather than shown. add_note arrived in 3.11.

One consequence for anyone matching on the message: current pytest searches notes as well, so pytest.raises(..., match="^service did not answer$") around a soft block stops matching once a note is attached. The message itself is unchanged.

Grouped soft assertions

Bind the collector with as sa and group failures by section with sa.group(label):

with soft_assertions() as sa:
    with sa.group("Headers"):
        assert_that(headers["Content-Type"]).is_equal_to("application/json")
    with sa.group("Body"):
        assert_that(body["status"]).is_equal_to("ok")

Grouped output

soft assertion failures:
  [Headers]
    1. Expected <text/html> to be equal to <application/json>, but was not.  [test_api.py:37]
  [Body]
    2. Expected <error> to be equal to <ok>, but was not.  [test_api.py:39]

Note

Soft mode collects assertion failures only.

After a failed raises() / warns() + when_called_with() there is no captured value left to assert on, so the rest of that one chain goes inert and is skipped silently. Independent assertions that follow are collected as usual.

assert_all

A convenience wrapper for inline soft assertions, equivalent to wrapping the calls in with soft_assertions()::

from assertpy2 import assert_all, assert_that

assert_all(
    lambda: assert_that(x).is_positive(),
    lambda: assert_that(y).is_not_none(),
    lambda: assert_that(z).is_length(3),
)

Async assertions

eventually() creates a polling assertion that retries until the condition holds or a timeout is reached, useful for eventual consistency and reactive systems:

async def test_status_converges():
    await assert_that(get_status).eventually().is_equal_to("ready")

By default it polls for 5 seconds every 0.5 seconds. Tune with within() and every():

await assert_that(get_count).eventually().within(10).every(0.2).is_greater_than(100)

The subject is a callable rather than a value, and that is the design rather than a limitation. A value is refused outright, because polling one would re-read nothing: an immutable subject could only spin to the timeout, and the library cannot tell one apart from a live object. The callable says what to read again, so assert_that(lambda: element.text) asks the element each round.

Both sync and async callables work, and any assertion method is available after eventually():

await assert_that(async_get_status).eventually().is_equal_to("done")
await assert_that(get_name).eventually().starts_with("Al")
await assert_that(get_count).eventually().is_between(10, 20)

A chain keeps polling for as long as it is written. Every call on it, whether a navigation step like described_as() or another assertion, is replayed against a fresh probe on each poll, and the whole chain is awaited once at the end:

await assert_that(get_order).eventually(timeout=10).is_instance_of(Order).has_status("PAID")

Negation is part of that chain too, which is how you wait for something to stop being true:

await assert_that(get_status).eventually(timeout=10).not_.is_equal_to("pending")

The await is what runs the chain, so leaving it out polls nothing. A chain that is dropped without being awaited raises a RuntimeWarning, the same way a dropped coroutine does, and --assertpy2-dangling reports it from the source at collection.

Awaiting hands back the ordinary builder over the value that settled, so anything asserted on the result afterwards is a plain assertion against that value rather than a new wait.

By default only a failing assertion is retried. Any other exception raised by the probe propagates immediately.

The one exception is an AssertionError raised by the probe itself. It arrives where a failing assertion arrives and is retried the same way, which is what a probe asserting its own preconditions usually wants. The timeout message says so, naming the type and the count.

When "not ready yet" arrives as an exception, such as a refused connection while a service boots, list those exception types in ignoring:

await assert_that(get_order).eventually(timeout=10, ignoring=ConnectionError).has_status(
    "PAID"
)

# or configure fluently, like within()/every()
await assert_that(get_order).eventually().within(10).ignoring(
    ConnectionError, TimeoutError
).has_status("PAID")

Note

Only AssertionError (plus any ignoring types) is retried. Other exceptions propagate immediately. On timeout the last failure is chained for context. ignoring accepts only Exception subclasses, so KeyboardInterrupt and friends can never be swallowed.

Polling itself is always strict, since retrying requires hard failures. Only the final timeout failure honors the builder's mode: inside soft_assertions() it is collected instead of raised, and under assert_warn() it is logged.

Synchronous polling

eventually_sync() is the same polling assertion without asyncio: assertion methods block the calling thread (via time.sleep) instead of returning coroutines, so it drops straight into plain non-async tests:

def test_status_converges():
    assert_that(get_status).eventually_sync(timeout=10, interval=0.2).is_equal_to("ready")

# within()/every()/ignoring() work the same way
assert_that(get_order).eventually_sync().within(10).ignoring(ConnectionError).has_status(
    "PAID"
)

Retry rules, chaining, negation, soft/warn behavior, and the polling trace are identical to eventually():

assert_that(get_order).eventually_sync(timeout=10).is_instance_of(Order).has_status("PAID")
assert_that(get_status).eventually_sync(timeout=10).not_.is_equal_to("pending")

The one difference is that the probe must be a sync callable. One that returns an awaitable raises TypeError, so poll async probes with eventually() and await.

Reaching for asyncio.run() to call eventually() from a non-async test works only where nothing else owns a loop in that thread. Playwright's sync API is the common counter-example, since it drives a loop of its own and the call fails with RuntimeError: asyncio.run() cannot be called from a running event loop. That configuration is why eventually_sync() exists.

Polling trace

Every poll is recorded, so a timeout failure diagnoses itself instead of just reporting that time ran out. The message opens with a one-line trend that pins the failure mode:

Trend line	What it means
probe raised ConnectionError on all 12 polls	the service never came up
value unchanged across 12 polls	it converged to the wrong value
value changed 3 times; last change 0.4s before the deadline	the timeout is too short
value cycles between 2 states across 12 polls	it keeps returning to earlier values, so waiting longer will not help
value could not be compared across 12 polls	the probe returns something that cannot be rendered, so movement is unknown

Every count in that line is over the whole run, not over the samples the trace kept: a long poll drops middle samples, and a value that moved on every one of a thousand polls has to read as a thousand rather than as the two dozen still in hand. Where a line describes the shape of what was kept rather than counting, it says so (value cycles between 2 states in the 25 polls kept).

The raised AssertionFailure carries the full timeline as .trace, a PollTrace of per-poll samples with identical consecutive polls collapsed.

In the pytest report a Polling Trace section leads with that one-line summary, covering total polls, elapsed time and how the value moved. It then lists every distinct poll with its offset, repeat count and the error or failure it produced:

  t=+0.0s error x2: ConnectionError('boot')
  t=+0.5s fail x2: Expected <'PENDING'> to be equal to <'PAID'>, but was not.
  t=+1.5s fail x5: Expected <'SHIPPED'> to be equal to <'PAID'>, but was not.

Allure receives the same timeline as a typed Polling Trace JSON attachment, with diffs between consecutive distinct samples.

Sample values are point-in-time snapshots, so they stay correct even when the probe mutates and returns the same object. They are capped like other attachments: long polls keep the first 5 and last 20 samples.

In soft/warn modes the message keeps the trend line, and so does the trace object: a timeout collected by a soft block is reachable as failures[i].trace on the aggregate. Warn mode logs the message and keeps nothing, having nothing to keep it on.

The recorder can be switched off per assertion with trace=False, on both eventually() and eventually_sync().

On every poll that fails with a value it reads that value twice, once for the sample and once to tell whether the value moved. The second walk covers the whole value rather than the cut sample, so a change past the hundredth item of a container is still seen.

A poll that passes, or one whose probe raised, walks nothing, and so does a chain that settles on its first probe. What the recorder costs is therefore the cost of failing polls, and it grows with the payload: one failing poll is a few microseconds over a scalar and about 2 ms over two hundred records.

Ten of them, which is what a timeout usually reaches:

Ten failing polls over	recorder on	recorder off
a scalar	118 µs	85 µs
a payload of two hundred records	22 ms	1.2 ms

That second row is most of what a failing poll costs, which is why a tight interval over a large response is where turning the recorder off is worth it.

That is the rare case: a near-zero interval meeting a heavy probed value, where even point-in-time snapshots cost too much. The timeout failure then reports just the last failure.

Every step of a chain is replayed against a fresh probe on each poll, so steps multiply by polls. One scalar probe settling on its eleventh call measured 117 µs polled by a one-step chain and 150 µs by a five-step one. A step that fails ends that poll and the steps behind it do not run, so the cheap check that tells the common failure apart is worth putting first.

Polls that nearly timed out

A poll that converges on its fifth attempt is eventually() doing its job, so a retry on its own says nothing.

Burning most of the budget before converging says a great deal. That run passed, and the next one on a slower machine will not. Under pytest those are collected across the run and listed at the end:

assertpy2 polls that nearly timed out:
  tests/test_orders.py::test_status: converged on attempt 41 at 0.81s of 1.0s (81% of the budget)

Only polls past 70% of their timeout are named, so a healthy suite prints nothing. The report is advisory and never fails a run.

Move that bar or turn the report off with assertpy2_poll_report. A slow CI box converges late on every poll, where the default turns a signal into a line of noise per run.

[tool.pytest.ini_options]
assertpy2_poll_report = "0.9"  # only polls past 90% of their timeout, or "off" to say nothing

Setting it to off also stops the samples being collected, so nothing is measured at the poll site.

Polls are attributed to the test that made them, including those a fixture makes in setup or teardown. Under pytest-xdist the workers ship their findings to the controller, which prints the combined list.

Waiting on a browser

A UI test is the polling case in its purest form: the page is asked something before it is ready, and the answer settles a moment later. eventually_sync() takes any callable, so a driver read is a probe like any other.

def test_the_banner_settles():
    reads = iter(["", "", "Welcome, Alice"])  # a page that is not ready on the first two reads

    assert_that(lambda: next(reads)).eventually_sync(timeout=2, interval=0.1).is_equal_to(
        "Welcome, Alice"
    )

With a real driver the probe is the read itself. Selenium has no assertions of its own, so this replaces a hand-written WebDriverWait and keeps the failure diagnostics:

assert_that(lambda: driver.find_element(By.CSS_SELECTOR, "#banner").text).eventually_sync(
    timeout=10, ignoring=(NoSuchElementException, StaleElementReferenceException)
).is_equal_to("Welcome, Alice")

Use Playwright's own expect() for locator assertions. It auto-waits, it retries, and it knows things about the page that a generic poller cannot. What it does not cover is the data behind the interface, and that is where an assertion library earns its place:

expect(page.get_by_role("heading")).to_have_text("Orders")  # Playwright's job

order = assert_conforms(api.get(f"/orders/{order_id}").json(), OrderModel).value
assert_that(order.total).is_close_to(page_total, 0.01)      # ours

The same split applies to Selenium once you leave the DOM: assert the page with the driver, assert the payload, the database row or the extracted structure with assert_that().

Snapshot testing

Capture a data structure to disk as JSON and compare against it on every run.

assert_that({"a": 1, "b": 2, "c": 3}).snapshot()

On the first run the snapshot file is created, a SnapshotCreatedWarning is emitted, and the test passes. On later runs the value is compared to the stored snapshot and the test fails on any mismatch.

Most Python structures are supported: dict, list, set, objects, numbers, None, complex, datetime/date/time, Decimal, and bytes. Commit the snapshot artifacts (the __snapshots folder) to source control.

Note

The capture warning makes a first run visible. Without it a wrong first capture would silently become the reference.

Under -W error (or filterwarnings = ["error"]) a new capture fails explicitly, which is usually what you want in CI.

One id per case

Without an id, the snapshot is keyed by the line of the snapshot() call. Every case of a parametrised test shares that line, so they share one key: the first case stores its value and the rest are compared against it.

@pytest.mark.parametrize("name", ["alice", "bob", "carol"])
def test_user(name):
    assert_that(load(name)).snapshot(id=f"user-{name}")  # without the id, all three share one key

Sharing a key fails loudly when the values differ, naming two cases that look unrelated. It passes silently when they agree, asserting one case out of however many, which is the worse half and the reason a SnapshotKeyReusedWarning names the key and the source line as soon as a second test reaches it. Raise it to an error if you would rather not rely on reading warnings:

[tool.pytest.ini_options]
filterwarnings = ["error::assertpy2.SnapshotKeyReusedWarning"]

What counts is two tests on one key, not two calls. A helper that snapshots twice inside one test asserts both values and stays silent. Should its second call then fail, the failure says so, since otherwise it reads as two unrelated values compared for no reason:

Expected <{'user': 'bob'}> to be equal to <{'user': 'alice'}>, but was not. This test reached
<__snapshots/snap-helper.json> more than once, so the value above was compared against what an
earlier call in the same test stored. Give each call its own snapshot(id=...).

Under pytest-xdist the cases of one parametrised test may land on different workers, where no single process sees the second one, so that split is caught by a sweep at the end of the run instead.

Updating snapshots

Run pytest with --assertpy2-snapshot-update and every failing snapshot comparison overwrites the stored value instead of failing.

Each overwrite emits a SnapshotUpdatedWarning, so the run reports exactly which snapshots changed. Matching snapshots are left untouched, and the comparison options (ignore, tolerance, ...) are honored when deciding whether a snapshot is stale:

pytest --assertpy2-snapshot-update

For runners other than pytest, set the ASSERTPY2_SNAPSHOT_UPDATE=1 environment variable instead. Deleting the snapshot files and re-running the suite still works too, and each fresh capture emits a SnapshotCreatedWarning.

CI mode

A first run creates a missing snapshot and passes. That is handy locally, but in CI it means a snapshot test whose golden was never committed creates it in the ephemeral workspace, passes, and silently disables drift detection.

Enable CI mode to make a missing snapshot a hard failure instead:

pytest --assertpy2-snapshot-ci

It is also auto-enabled when a CI environment variable is set (the near-universal CI marker), or via ASSERTPY2_SNAPSHOT_CI=1. Disable the autodetection with --assertpy2-snapshot-no-ci or ASSERTPY2_SNAPSHOT_CI=0. Local runs are unaffected.

Obsolete snapshots

When a test is deleted its stored snapshot lingers on disk. At the end of a run assertpy2 reports snapshots it did not touch:

assertpy2 snapshots:
  obsolete snapshot: __snapshots/snap-test_orders.json::42
  obsolete snapshot file: __snapshots/snap-test_legacy.json

Each line carries a short hint on how to remove it. Reporting is always safe.

Removal is deliberately conservative. An obsolete sub-snapshot, meaning a line-number key in a file whose module still ran, is pruned only under update mode on a full run.

That way a run narrowed by -k, -m, --lf or --ff never deletes a snapshot that merely looks unused because its test was deselected. A whole obsolete file is only ever reported, never deleted.

Under pytest-xdist the touched-snapshot sets from all workers are aggregated on the controller first, so a snapshot exercised on another worker is never mistaken for an orphan.

Inline snapshots

An inline snapshot keeps the expected value in the test source instead of a separate file. Call matches_inline() empty, record it once with --assertpy2-snapshot-update, and the literal is written back into the call:

# before recording
assert_that(client.get("/orders/1").json()).matches_inline()

# after `pytest --assertpy2-snapshot-update`
assert_that(client.get("/orders/1").json()).matches_inline(
    {"id": 1, "status": "paid"}
)

Later runs compare against the literal, and update mode overwrites it on drift, just as snapshot() does. The same selective knobs apply, so volatile fields never make the snapshot brittle:

assert_that(order).matches_inline(
    {"id": 0, "total": 42.0}, placeholders={"id": match.is_uuid()}, tolerance=0.01
)

A recorded literal holds the value captured on that run, so a placeholder field shows the captured id rather than the 0 above. The placeholder governs the comparison, not what is written.

Recording needs the [inline] extra (pip install assertpy2[inline]). The comparison does not.

It is a plain equality check, so it runs under pytest-xdist and needs no source introspection or assertion rewriting.

Under xdist the recorded edits are shipped to the controller and applied once, never written by workers in parallel.

Inline snapshots hold source literals, so only JSON-ish values work: a dict, list, tuple or set of scalars.

For a datetime, Decimal, UUID or a custom object use snapshot() instead. The two are complementary and share the same update flag, CI mode, selective comparison and structured diff.

Custom types

Beyond the built-in codec (set, complex, datetime/date/time, Decimal, bytes, UUID, Enum), register a serializer for any other type so snapshot() stores and round-trips it instead of raising:

from assertpy2 import register_snapshot_serializer
import pathlib

register_snapshot_serializer(pathlib.PurePath, str, pathlib.PurePath)

Matching is by isinstance, subclasses included. The registry is consulted before the built-ins, and a later registration wins.

The decode half runs your own code on load, so it is a trusted, explicit opt-in. The automatic instance decode never imports anything.

Contract snapshots

snapshot() compares exact values, so a response full of generated ids and timestamps needs ignore or placeholders to stay stable.

When you care about the response's shape rather than its values, reach for matches_contract_snapshot(). It records paths and type categories, never values, and on later runs fails only on structural drift: a field added, removed, or retyped.

assert_that(response.json()).matches_contract_snapshot()

It is value-tolerant by construction, so dynamic ids, timestamps, and amounts (and 5 vs 5.0) change freely. A real contract change fails with the drifted paths:

Expected <{...}> to match contract snapshot <...>, but the structure drifted:
  + promo_code
  ~ id number -> str

No hand-written model is needed. The contract is inferred from the first response, and it shares the same storage, update mode and CI mode as snapshot().

The model-driven counterpart is assert_conforms(..., exact=True). Reach for that when you already have a pydantic model.

Because a contract is inferred from a single observation it cannot know which fields are optional, so a legitimately sometimes-absent field reads as removed. Re-record with update mode when the contract really changed.

Shape placeholders

comparators and ignore make the comparison tolerate volatile fields, but the golden still stores the arbitrary value captured on the first run. placeholders instead records a self-documenting shape token in the golden and asserts the field's shape on every run:

from assertpy2 import match

assert_that(response).snapshot(
    id="order",
    placeholders={
        "id": match.is_uuid(),
        "created_at": lambda ts: isinstance(ts, str),
    },
)

The golden reads "id": {"__placeholder__": "a valid UUID string"} instead of a specific id, and each run asserts the actual field is present and satisfies the matcher (a Matcher or a callable predicate) rather than comparing it for equality.

Every other field is still compared exactly, so drift outside the placeholders is caught. Placeholders apply to top-level keys of a dict-like value and combine with ignore.

Parameters

Snapshots are keyed by test filename plus line number by default. Override with id or path:

assert_that({"a": 1}).snapshot(id="my-custom-id")
assert_that({"a": 1}).snapshot(path="my-custom-folder")

Volatile fields and float noise

The comparison accepts the same selective options as is_equal_to(): ignore, include, tolerance, and comparators - so timestamps, generated ids, or float jitter don't break snapshots. The snapshot file always stores the full value. The options only shape the comparison:

assert_that(api_response).snapshot(
    id="order", ignore=["created_at", ("user", "session_id")]
)
assert_that(metrics).snapshot(id="latency", tolerance=0.001)
assert_that(payload).snapshot(
    id="user", comparators={"name": lambda a, e: a.lower() == e.lower()}
)

Known limitations

Beyond the JSON natives, these types survive a round-trip:

Type	Stored as
set	a tagged list
complex	a tagged pair
datetime / date / time	ISO text, microseconds and timezone kept
Decimal	exact text, never a float
bytes / bytearray	base64, both compared as bytes
uuid.UUID, Enum members	their canonical text form
any object with a __dict__	its attributes

Anything else raises TypeError on capture. Teach the codec with register_snapshot_serializer(), which is what a frozenset or a domain class needs.

Three more things worth knowing:

• A tuple comes back as a list, since JSON has no tuple. A snapshot of (1, 2) compares as [1, 2] on the next run and fails, so convert tuples before snapshotting.
• Snapshot ids are case-insensitive. Filenames are lower-cased, so two ids differing only by case land in one file.
• The write lock is not crash-safe. A process killed mid-write leaves a stale .lock beside the snapshot. Delete it if snapshot writes start timing out.