derivepassphrase.git

Set up the "hypothesis" testing library
Parent ef6258d
Marco Ricci

Marco Ricci commited on 2024-09-27 17:32:46
Zeige 3 geänderte Dateien mit 30 Einfügungen und 6 Löschungen. 

"hypothesis" is a Python library for property-based testing, i.e. for
asserting that the output of the tested code satisfies some property by
generating valid test inputs and testing these.  For `derivepassphrase`
in particular, this means generating different passphrase constraint
settings and testing that the generated passphrases match these
constraints, no matter what constraints exactly are chosen (within
reason).  The hypothesis library provides methods to generate test input
(test function parameter values, or custom walks for state machine-like
systems) and, for failing examples, shrink those to shorter/simpler ones
exhibiting the same error.

By its nature, hypothesis runs unit test functions multiple times with
different inputs, and so takes much longer than normal unit tests.  (In
the default configuration, 200 examples per test are run.)  It is
therefore sensible to set up alternate testing profiles with more or
less examples per test, and to include a mechanism in the test runner to
switch to these profiles.  The hypothesis documentation provides an
explicit example for profile setup for pytest use, plus three sensible
configurations "ci", "dev" and "debug", which I copied.  The pytest
command-line as called by `hatch test` is also straightforward to adapt:
add `--hypothesis-profile=...` to all pytest invocations.  However, the
shape of the `hatch test` environment matrix no longer matches the
reality of scenarios we'd like to test (because we added a new
dimension: slow, thorough testing, or fast testing).  So use this
opportunity to also clean up the matrix.
.gitignore
Zeige Datei @ bf906f3
... ... 
@@ -4,3 +4,4 @@ html/
4 4 
 __pycache__/
5 5 
 *.swp
6 6 
 .coverage*
7 
+.hypothesis/
pyproject.toml
Zeige Datei @ bf906f3
... ... 
@@ -127,18 +127,33 @@ dependencies = [
127 127
128 128 
 [tool.hatch.envs.hatch-test]
129 129 
 default-args = ['src', 'tests']
130 
+extra-dependencies = [
131 
+    "hypothesis >= 6.0",
132 
+]
133 
+matrix-name-format = '{variable}_{value}'
130 134
131 135 
 [[tool.hatch.envs.hatch-test.matrix]]
132 
-python = ["3.10", "3.11", "3.12", "pypy3.10"]
136 
+python = ["3.12", "3.11", "3.10", "pypy3.10"]
137 
+cryptography = ["no", "yes"]
138 
+hypothesis-profile = ["user-default"]
133 139
134 140 
 [[tool.hatch.envs.hatch-test.matrix]]
135 
-python = ["3.10", "3.11", "3.12", "pypy3.10"]
136 
-feature = ["export"]
141 
+cryptography = ["yes"]
142 
+hypothesis-profile = ["ci"]
137 143
138 144 
 [tool.hatch.envs.hatch-test.overrides]
139 
-matrix.feature.features = [
140 
-    { value = "export", if = ["export"] },
145 
+matrix.cryptography.features = [
146 
+    { value = "export", if = ["yes"] },
141 147 
 ]
148 
+matrix.hypothesis-profile.env-vars = [
149 
+    { key = "HYPOTHESIS_PROFILE", if = ["ci", "default", "dev", "debug"] },
150 
+]
151 
+
152 
+[tool.hatch.envs.hatch-test.scripts]
153 
+run = "pytest --hypothesis-profile={env:HYPOTHESIS_PROFILE:default}{env:HATCH_TEST_ARGS:} {args}"
154 
+run-cov = "coverage run -m pytest --hypothesis-profile={env:HYPOTHESIS_PROFILE:default}{env:HATCH_TEST_ARGS:} {args}"
155 
+cov-combine = "coverage combine"
156 
+cov-report = "coverage report"
142 157
143 158 
 [tool.hatch.envs.release]
144 159 
 extra-dependencies = [
... ... 
@@ -149,6 +164,7 @@ extra-dependencies = [
149 164
150 165 
 [tool.hatch.envs.types]
151 166 
 extra-dependencies = [
167 
+  "hypothesis >= 6.0",
152 168 
   "mypy ~= 1.0",
153 169 
   "pytest ~= 8.1",
154 170 
 ]
tests/conftest.py
Zeige Datei @ bf906f3
... ... 
@@ -14,6 +14,7 @@ import sys
14 14 
 import textwrap
15 15 
 from typing import TYPE_CHECKING, TypeVar
16 16
17 
+import hypothesis
17 18 
 import packaging.version
18 19 
 import pytest
19 20
... ... 
@@ -24,9 +25,15 @@ if TYPE_CHECKING:
24 25 
     from collections.abc import Iterator
25 26 
     from typing import Literal
26 27
27 
-
28 28 
 startup_ssh_auth_sock = os.environ.get('SSH_AUTH_SOCK', None)
29 29
30 
+# https://hypothesis.readthedocs.io/en/latest/settings.html#settings-profiles
31 
+hypothesis.settings.register_profile("ci", max_examples=1000)
32 
+hypothesis.settings.register_profile("dev", max_examples=10)
33 
+hypothesis.settings.register_profile(
34 
+    "debug", max_examples=10, verbosity=hypothesis.Verbosity.verbose
35 
+)
36 
+
30 37
31 38 
 # https://docs.pytest.org/en/stable/explanation/fixtures.html#a-note-about-fixture-cleanup
32 39 
 # https://github.com/pytest-dev/pytest/issues/5243#issuecomment-491522595
33 40