https://git.schokokeks.org/derivepassphrase.git/tree/0bf24eb16ec698d7759657cb810385ec7388f74c Recent commits feed provided by GitList. Some of our tests spawn subprocesses, and without this, the subprocesses fail to import code from the `tests` hierarchy (e.g., `tests.data`.) We have so far sidestepped this issue by requiring `$PYTHONPATH` to be set accordingly, but it turns out that setting this in the pytest settings works even more reliably. In the long run, it might be smarter to use a similar source layout in `tests` as we already do in `src`: a top-level `test_derivepassphrase` package, and the `data`, `machinery` and other tests modules/packages as submodules/subpackages of `test_derivepassphrase`. https://git.schokokeks.org/derivepassphrase.git/commit/0bf24eb16ec698d7759657cb810385ec7388f74c software@the13thletter.info (Marco Ricci) Sun, 05 Jul 2026 21:31:22 +0200 0bf24eb16ec698d7759657cb810385ec7388f74c This topic branch implements steps to make the test suite more economical (i.e., to ensure a high value-to-code ratio in the test suite) on the grounds that changes to the main codebase usually cause changes to the test suite, and thus the test suite needs similar management to the main codebase regarding technical debt. 1. As the first big step, we divide our tests semantically into "regression tests" and "correctness tests", serving [the two main purposes of a test suite][TWO_KINDS]: fast feedback ("regression tests") vs. high coverage/confidence ("correctness tests"). Specifically: - During regression testing, random input generation is turned down severely, currently to three examples. In the long run, property-based tests without explicit examples will be skipped, and thus every non-correctness-only hypothesis test should have at least one explicit example. - Regression tests that "should" be correctness tests shall be implemented as property-based tests with explicit examples. Conversely, property-based tests that implement an exhaustive testing matrix over some parameter set shall be refactored such that the testing matrix is expressed with `pytest.mark.parametrize`. (It is possible for a test to be a "combined" test using both `pytest.mark.parametrize` and `hypothesis.given`.) 2. As a second step, avoid using "heavyweight" machinery if more "lightweight" machinery will do. Specifically, avoid spawning or interfacing with an external SSH agent if the internal stub SSH agent works just as well, e.g., because the communication is being stubbed out anyway. Beyond these two steps, add several quality-of-life improvements to the test suite: - To convert `parametrize`'d tests to hypothesis tests, add a converter function that takes the same arguments as the `parametrize` mark and decorates the respective function with the equivalent explicit examples (and, if not yet a hypothesis test, also synthesizes a `hypothesis.given`). - Fix the stub SSH agent to behave more like a real communication socket (incomplete messages, EBADF after closing), thus also making it more suitable to model actual connection problems. - Clean up multiple "magic strings" in the test suite by introducing appropriate enums. - Move some common setup code or parametrization into the tests package. - Update the tooling: update `ruff`, and use `pyrefly` in addition to `mypy`. [TWO_KINDS]: https://blog.nelhage.com/post/two-kinds-of-testing/ "Nelson Elhage: Two kinds of testing" * economic-test-suite: (28 commits) Restore the `test_hash_length_expansion`/`test_wrong_initial_hash_length` vault test Fix some test coverage slipups Define `use_stub_agent`/`use_stub_agent_with_address` in conftest Add a heavy-duty test for Sequin internals Implement missing agent system support via tests.data.SystemSupportAction Move some setup "action" classes into tests.data Turn some parametrized unit tests into nominal property-based tests Rewrite the vault CLI tests for config management to use the stub agent Fix tiny mistakes across the test suite Update tooling (mypy/pyrefly, mkdocstrings overrides) Run the vault utility `key_to_phrase` tests against the stub SSH agent Add a pytest marker for external SSH agent use Run vault CLI config management tests against the stub agent where possible Use the stub agent for SSH agent client communications faking Harmonize tests for missing SSH agent communication support Reformat recent test changes for consistent style Type generators explicitly as generators, not iterators Refactor the (SSH agent) testing machinery tests for modularity Re-align the stubbed SSH agent socket with "real world" behavior Serialize the SSH agent protocol numbers as request/response codes ... https://git.schokokeks.org/derivepassphrase.git/commit/caa1b1748a9b129e2e1afd684c0d52905e6a2df8 software@the13thletter.info (Marco Ricci) Sun, 05 Jul 2026 16:57:37 +0200 caa1b1748a9b129e2e1afd684c0d52905e6a2df8 This test exercises the "automatically enlarging the sequin input" path of passphrase derivation in vault. We originally removed this test because its original name and description suggested a somewhat different purpose, and necessity. https://git.schokokeks.org/derivepassphrase.git/commit/91953be84921031dfe082fe314c0216baa96fe9a software@the13thletter.info (Marco Ricci) Sun, 05 Jul 2026 16:52:19 +0200 91953be84921031dfe082fe314c0216baa96fe9a This topic branch restores compatibility with click 8.4.2 (and probably with all versions from 8.2.0 onwards) by working around the changes to the test runner since 8.2.0. This is a stopgap measure. In the long run, we intend to move away from click, because the API is both unstable and, at times, asinine. See the "remove-click" wish in the 0.6 wishlist. * click-8.4.2-compatibility: Work around test runner changes in click > 8.1.8 https://git.schokokeks.org/derivepassphrase.git/commit/2140b40a8c44be10ada98172ef366e83471c0fa2 software@the13thletter.info (Marco Ricci) Sun, 05 Jul 2026 16:38:19 +0200 2140b40a8c44be10ada98172ef366e83471c0fa2 https://git.schokokeks.org/derivepassphrase.git/commit/5d1c2297ead22c7dd0ef5e78c994aa674680f6be software@the13thletter.info (Marco Ricci) Sun, 05 Jul 2026 16:21:55 +0200 5d1c2297ead22c7dd0ef5e78c994aa674680f6be More modern versions of click – I believe 8.2.0 and higher, and definitely in 8.4.0 and higher – implement stdout and stderr capture in a completely broken way when prompts are involved and the streams are mixed: mixed streams are not truly interleaved, and prompts cause extra typed-in whitespace from stdin to appear in stderr as well. As a result, the "correct" captured output in a "mixed stdout/stderr" test runner looks neither like an interactive shell session with TTY I/O nor like a non-interactive shell session with redirected I/O; it's a hobgobbled mess of a bit of both. Work around this for now by supplying the correct broken expected output for test runners from newer click versions. But if anything, this reinforces my desire to drop click as a dependency as soon as possible. (See the wishlist item "remove-click" in the 0.6 wishlist. That wish also details *other* API breakage that users have to wrangle with.) https://git.schokokeks.org/derivepassphrase.git/commit/65c53d61f985d4c71cfa6a44bcc236d0ac598927 software@the13thletter.info (Marco Ricci) Sun, 05 Jul 2026 14:28:33 +0200 65c53d61f985d4c71cfa6a44bcc236d0ac598927 This avoids redefining the same fixture in multiple test modules. https://git.schokokeks.org/derivepassphrase.git/commit/3e9162ca6f3d536c73446940c6ea34beb9cf5ddf software@the13thletter.info (Marco Ricci) Sun, 05 Jul 2026 11:13:19 +0200 3e9162ca6f3d536c73446940c6ea34beb9cf5ddf Add a heavy-duty hypothesis test (rule-based state machine) that compares the state of our Sequin class against an as-literal-as-possible port of the original JavaScript sequin module, included inline in the test class. (The heavy-duty marker and the state machine machinery are probably overkill for this application, given that there is only one rule, and that the test runs very fast. Nevertheless, phrasing this as a state machine makes the workings of the test very explicit: what are the state transitions, what are the invariants, etc.) https://git.schokokeks.org/derivepassphrase.git/commit/ac77734f1ee1a32f825aedd28e223ff88e48aa9b software@the13thletter.info (Marco Ricci) Sat, 04 Jul 2026 23:22:58 +0200 ac77734f1ee1a32f825aedd28e223ff88e48aa9b Implement the "missing system support for the SSH agent communication channel" setup code via the now centralized `tests.data.SystemSupportAction` enum, instead of duplicating the setup steps everywhere. This means adding the correct `SystemSupportAction` value to the `MISSING_AGENT_SUPPORT` parametrization set. In particular, since this parametrization set is already used in two different test modules, put it into `tests.machinery.pytest.Parametrize`. https://git.schokokeks.org/derivepassphrase.git/commit/988320655d5241e715792ec81fbf24f9656c0fde software@the13thletter.info (Marco Ricci) Sat, 04 Jul 2026 21:10:03 +0200 988320655d5241e715792ec81fbf24f9656c0fde Move the ListKeysAction, SignAction, SocketAddressAction and SystemSupportAction classes into tests.data. We intend to reuse the SystemSupportAction class for a new purpose next. https://git.schokokeks.org/derivepassphrase.git/commit/a5e9119b7521004707782c16f16e57fe315230df software@the13thletter.info (Marco Ricci) Sat, 04 Jul 2026 21:08:08 +0200 a5e9119b7521004707782c16f16e57fe315230df