Merge branch 'economic-test-suite'
Marco Ricci commited on 2026-07-05 16:57:37
Zeige 0 geänderte Dateien mit 0 Einfügungen und 0 Löschungen.
This topic branch implements steps to make the test suite more
economical (i.e., to ensure a high value-to-code ratio in the test
suite) on the grounds that changes to the main codebase usually cause
changes to the test suite, and thus the test suite needs similar
management to the main codebase regarding technical debt.
1. As the first big step, we divide our tests semantically into
"regression tests" and "correctness tests", serving [the two main
purposes of a test suite][TWO_KINDS]: fast feedback ("regression
tests") vs. high coverage/confidence ("correctness tests").
Specifically:
- During regression testing, random input generation is turned
down severely, currently to three examples. In the long run,
property-based tests without explicit examples will be skipped,
and thus every non-correctness-only hypothesis test should have
at least one explicit example.
- Regression tests that "should" be correctness tests shall be
implemented as property-based tests with explicit examples.
Conversely, property-based tests that implement an exhaustive
testing matrix over some parameter set shall be refactored such
that the testing matrix is expressed with
`pytest.mark.parametrize`. (It is possible for a test to be
a "combined" test using both `pytest.mark.parametrize` and
`hypothesis.given`.)
2. As a second step, avoid using "heavyweight" machinery if more
"lightweight" machinery will do. Specifically, avoid spawning or
interfacing with an external SSH agent if the internal stub SSH
agent works just as well, e.g., because the communication is being
stubbed out anyway.
Beyond these two steps, add several quality-of-life improvements to the
test suite:
- To convert `parametrize`'d tests to hypothesis tests, add
a converter function that takes the same arguments as the
`parametrize` mark and decorates the respective function with the
equivalent explicit examples (and, if not yet a hypothesis test,
also synthesizes a `hypothesis.given`).
- Fix the stub SSH agent to behave more like a real communication
socket (incomplete messages, EBADF after closing), thus also making
it more suitable to model actual connection problems.
- Clean up multiple "magic strings" in the test suite by introducing
appropriate enums.
- Move some common setup code or parametrization into the tests
package.
- Update the tooling: update `ruff`, and use `pyrefly` in addition to
`mypy`.
[TWO_KINDS]: https://blog.nelhage.com/post/two-kinds-of-testing/ "Nelson Elhage: Two kinds of testing"
* economic-test-suite: (28 commits)
Restore the `test_hash_length_expansion`/`test_wrong_initial_hash_length` vault test
Fix some test coverage slipups
Define `use_stub_agent`/`use_stub_agent_with_address` in conftest
Add a heavy-duty test for Sequin internals
Implement missing agent system support via tests.data.SystemSupportAction
Move some setup "action" classes into tests.data
Turn some parametrized unit tests into nominal property-based tests
Rewrite the vault CLI tests for config management to use the stub agent
Fix tiny mistakes across the test suite
Update tooling (mypy/pyrefly, mkdocstrings overrides)
Run the vault utility `key_to_phrase` tests against the stub SSH agent
Add a pytest marker for external SSH agent use
Run vault CLI config management tests against the stub agent where possible
Use the stub agent for SSH agent client communications faking
Harmonize tests for missing SSH agent communication support
Reformat recent test changes for consistent style
Type generators explicitly as generators, not iterators
Refactor the (SSH agent) testing machinery tests for modularity
Re-align the stubbed SSH agent socket with "real world" behavior
Serialize the SSH agent protocol numbers as request/response codes
...