4d92a083bc sha256: speed up writes using multi-block compression (furszy)
0753f8b909 Add API to override SHA256 compression at runtime (furszy)
fdb6a91a5e Introduce hash context to support pluggable SHA256 compression (furszy)
Pull request description:
Tackling the long-standing request #702.
Right now we ship our own SHA256 implementation, a standard baseline version that does not take advantage of any hardware-optimized instruction, and it cannot be accessed by the embedding application - it is for internal usage only.
This means embedding applications often have to implement or include a different version for their use cases, wasting space on constrained environments, and in performance-sensitive setups it forces them to use a slower path than what the platform provides. Many projects already rely on tuned SHA-NI / ARMv8 / or other hardware-optimized code, so always using the baseline implementation we ship within the library is not ideal.
These changes allow users to supply their own SHA256 compression function at runtime, while preserving the existing default behavior for everyone else. This is primarily intended for environments where the available SHA256 implementation is detected dynamically and recompiling the library with a different implementation is not feasible (equivalent build-time functionality will come in a follow-up PR).
It introduces a new API:
```C89
secp256k1_context_set_sha256_transform_callback(ctx, fn_transform)
```
This function installs the optimized SHA256 compression into the `secp256k1_context`, which is then used by all internal computations. Important: The provided function is verified to be output-equivalent to the original one.
As a quick example, using this functionality in Bitcoin-Core will be very straightforward: f68bef06d9
ACKs for top commit:
real-or-random:
ACK 4d92a083bc
w0xlt:
ACK 4d92a083bc
theStack:
ACK 4d92a083bc
Tree-SHA512: 058e2e82071f1ca77254b684458292c621e60d65bbcc5500574429717e7db75bc9f3221129fafd11eb5d33e666a5efec5e9844460d3b194ef3b6b16f2df28fb9
921b9711ea util: introduce and use `ARRAY_SIZE` macro (Sebastian Falbesoner)
Pull request description:
This PR is another tiny improvement found while working on #1765, with the goal to avoid code repetition.
The `ARRAY_SIZE` macro definition is pretty wide-spread in C projects and e.g. matches the one [used in the Linux Kernel](9702969978/include/linux/array_size.h (L11)) (without the additional check to reject pointers, as we would need GNU C for that, see e.g. https://stackoverflow.com/a/19455169; not sure if a useful counterpart exists that only relies on C89). Replacement instances were identified via `$ git grep sizeof.*/.*sizeof`.
ACKs for top commit:
w0xlt:
ACK 921b9711ea
real-or-random:
utACK 921b9711ea
Tree-SHA512: 44b6bf0132cf00fade526a3fc04e03dc896d04874123614c032206b61f97c81f94d139b6cc0c108eceaa699251580c19420d230b3150607303ca2cb7ab9a0bcb
This introduces `secp256k1_context_set_sha256_compression()`,
which allows users to provide their own SHA256 block-compression
function at runtime.
This is useful in setups where the fastest implementation can only
be determined dynamically based on the available CPU features, and
rebuilding the library is not possible.
The callback is installed on the `secp256k1_context` and is then used
by all operations that compute SHA256 hashes. As part of the setup,
the library performs sanity checks to ensure that the supplied
function is equivalent to the default transform.
Passing NULL to the callback setter restores the built-in
implementation.
This is purely a mechanical change with no behavior change.
It introduces a secp256k1_hash_ctx struct inside secp256k1_context
and propagates it to all SHA256-related operations.
This sets up the ability to provide a hardware-optimized SHA256
compression function at runtime in a follow-up commit.
c49c9be504 bench: Update help functions in bench and bench_internal (kevkevinpal)
Pull request description:
### Motivation
This change is motivated by https://github.com/bitcoin-core/secp256k1/pull/1793#pullrequestreview-3644885897
> While aligning implementation across all benchmarks, argv could be passed to the help() in bench.c and bench_internal.c.
### Description
In the `bench` and `bench_internal` `help` functions `argv` was not being passed. In this change, we pass in argv and use it in the help text.
ACKs for top commit:
real-or-random:
ACK c49c9be504
Tree-SHA512: 77184db4bf5c16827f19d888af73939f4139cc2e84ae5256d995cf61f606d5865928480fc009a0185e1a6843f3c38dd1b858d1316e524c9b165459c7367f2318
This removes printing of the "random run = " at the end of the tests. I
haven't seen a single case where this proved to be useful. And as of
48789dafc2, this is anyway printed only at
the end of the exhaustive tests and not the normal tests, so the
probability that this will be useful in the future is very low.
f48b1bfa5d hash: add midstate initializer and use it for tagged hashes (w0xlt)
Pull request description:
Each tagged hash midstate function (e.g., `secp256k1_schnorrsig_sha256_tagged`) calls `secp256k1_sha256_initialize` before immediately overwriting every field it sets: `s[0]` through `s[7]` and `bytes`. The `buf[64]` member does not need initialization either, because `bytes` is set to 64, which means the buffer position (`bytes & 0x3F`) (`= bytes % 64`) is 0, so buf is always written before being read.
Remove the 11 redundant `secp256k1_sha256_initialize` calls across the `schnorrsig`, `ellswift`, and `musig` modules.
ACKs for top commit:
real-or-random:
utACK f48b1bfa5d
theStack:
Code-review ACK f48b1bfa5d
Tree-SHA512: 769beb96f3921cc3c180ed0d17484ffa0dc78041c889a8e56603679d8eaca5fe13e63759ada78f83d8e0ff7aae392e6bcbc1a9fe8b959105ea4a3d8ef51abf15
Introduce secp256k1_sha256_initialize_midstate() in the hash layer and use it at all tagged-hash midstate call sites across schnorrsig, musig, and ellswift.
Document the byte-counter contract at the declaration site in hash.h and add run_sha256_initialize_midstate_tests() to directly verify helper behavior against initialize_tagged.
Also switch the helper to take const uint32_t state[8] to reduce argument-order risk at call sites.
The secp256k1_ellswift_xdh function uses overflow = secp256k1_scalar_is_zero(&s) which overwrites the overflow flag from the preceding secp256k1_scalar_set_b32 call. This means secret keys >= the curve order are silently accepted (reduced mod n) instead of being rejected.
The fix changes = to |=, matching the correct pattern already used in secp256k1_ecdh (main_impl.h, line 51).
The ECDH module's test suite explicitly tests overflow rejection (passes secp256k1_group_order_bytes as a key and checks the function returns 0). The ellswift test suite has no corresponding test, which is why this went undetected.
In the bench and bench_internal help functions argv was not being
passed, in this change we pass in argv[0] and use it in the help text.
Additionally instead of passing all of argv in bench_ecmult we now
just pass argv[0] and is used as the executable_path variable.
f47bbc07f0 test: add unit tests for secp256k1_scalar_check_overflow (Rohit Yadav)
Pull request description:
This Pull Request improves the tests for `secp256k1_scalar_check_overflow` as requested in #1812.
### Changes:
- Removed the redundant "all ones" check from `run_scalar_tests`.
- Added a new dedicated test function `test_scalar_check_overflow`.
- Added static checks for edge cases: `0`, `N-1`, `N`, `N+1`, and `MAX`.
- Added random input tests that verify `check_overflow` against a manual byte comparison.
Fixes#1812.
ACKs for top commit:
theStack:
re-ACK f47bbc07f0
real-or-random:
utACK f47bbc07f0
Tree-SHA512: dad3aa31ecf3f296843c907ac3d9aa5a9b9cb839b36aa3b59e49c853c60c58291412e70dff37dc15f8e14023a8f1e1aba87395065607612d5f6cfa92e14e73b5
97b3c47849 refactor: remove unnecessary `malloc` result casts (Sebastian Falbesoner)
Pull request description:
While working on benchmark code for #1765, I noticed that in some instances we explicitly cast `malloc` results in the codebase. It seems that there is no good reason to do this in C, and it's even considered bad practice, see e.g. https://stackoverflow.com/a/605858.
This commit touches mostly test code, the only two functions used in production are `secp256k1_context_{create,clone}`. Instances were found manually via `$ git grep "malloc("`.
ACKs for top commit:
real-or-random:
Weak Concept ACK && Code Review ACK 97b3c47849
w0xlt:
ACK 97b3c47849
Tree-SHA512: 74aa9f47eb52b7f2a6fcb69deb6aef0c0daa136c5deedfba1228218ef178c722212d8e9936fd2946d2035df932637ca4df49c98ddde488c6b009a74c4d5df316
1bc74a22f8 test: show both Autotools and CMake usage for ctime_tests (8144225309)
Pull request description:
When building with CMake and running `ctime_tests` outside valgrind, users see:
```
Usage: libtool --mode=execute valgrind ./ctime_tests
```
CMake users don't have libtool. Show both commands.
### Before
```
$ ./build/bin/ctime_tests
This test can only usefully be run inside valgrind because it was not compiled under msan.
Usage: libtool --mode=execute valgrind ./ctime_tests
```
### After
```
$ ./build/bin/ctime_tests
This test can only usefully be run inside valgrind because it was not compiled under msan.
Usage: valgrind ./ctime_tests (or with Autotools: libtool --mode=execute valgrind ./ctime_tests)
```
Fixes#1697
ACKs for top commit:
real-or-random:
utACK 1bc74a22f8
Tree-SHA512: d35c332c75fe3df66928cb8b137e11995c67a57744985a50a539d1d9f24cf39ee46f17c6f6a501664a62f67e11b7bb041ba0e1eed6632bf7dccdb57a2c88f9bc
It seems that there is no good reason to do this and it's even
considered bad practice, see e.g. https://stackoverflow.com/a/605858
This commit touches mostly test code, the only two functions used
in production are `secp256k1_context_{create,clone}`.
Instances were found manually via `$ git grep "malloc("`
748c0fdd67 Add CMake build directory patterns to `.gitignore` (Hennadii Stepanov)
7eb86bdb01 autotools: Rename `build-aux` to `autotools-aux` (Hennadii Stepanov)
Pull request description:
Whenever I work on changes that require comparison, such as benchmarking, I end up with two or more build directories that provide different binary variants simultaneously. Adding these build directories to `.gitignore` makes the workflow a bit easier.
Additionally, a trivial refactoring is included to reduce the code.
ACKs for top commit:
real-or-random:
utACK 748c0fdd67
furszy:
ACK 748c0fdd67
Tree-SHA512: 948917dcdc2ec6d5a2227f35ef9208fdbc62c56047db1c60b39f6da632642847aefa18f136986f9f15f08e0b2385964afe9a311346b728536323c54b4f0e3f04
47eb70959a ecmult: Use size_t for array indices in _odd_multiplies_table (Tim Ruffing)
bb1d199de5 ecmult: Use size_t for array indices into tables (Tim Ruffing)
Pull request description:
I don't think the current code is incorrect, but using `size_t` improves readability because the type makes it clear that we're dealing with array indices.
Also, making the result of the `ECMULT_TABLE_SIZE` macro (hopefully) a `size_t` fixes a compiler warning on MSVC, see #1791.
ACKs for top commit:
hebasto:
re-ACK 47eb70959a.
jonasnick:
ACK 47eb70959a
theStack:
ACK 47eb70959a
Tree-SHA512: e484fd610d50e972021c0184a683993364290eb58e09b65f9521b4507ec8d0639b402c67002005630b389bc863a7aa05b75f7224524dbcbafbfa5f9a4812b4a5
In this change the get_iters function was updated to print an error
message and then return 0. In the functions that use get_iters they
print the help text and then EXIT_FAILURE
bd5ced1fe1 doc/bench: added help text for SECP256K1_BENCH_ITERS env var for bench_ecmult (kevkevinpal)
Pull request description:
ACKs for top commit:
real-or-random:
utACK bd5ced1fe1
hebasto:
ACK bd5ced1fe1, I have reviewed the code and it looks OK. Tested on Ubuntu 25.10.
jonasnick:
ACK bd5ced1fe1
Tree-SHA512: 7cfc1a8915717bdfe2901f20f578e23368ece9937a40f36805a0a5b741f97a0502a085c973f6912b96c2bca921ef1654908cfe2c90c0601a7ffa92de4415dc62
2f73e5281d group: Avoid using infinity field directly in other modules (Tim Ruffing)
Pull request description:
Minor refactoring to make the abstraction cleaner
ACKs for top commit:
hebasto:
ACK 2f73e5281d, I have reviewed the code and it looks OK.
theStack:
ACK 2f73e5281d
Tree-SHA512: eae5ad1ce81f491adb48ab1cbf04211f8d43e41255abcacc958fa3dcb1de5021707d56ed1b009a6f3f6c45cd8f20c1f2677891690a3c0a467fc7e064af2512a8
ae00c552df Add VERIFY_CHECKs that flags are 0 or 1 (John Moffett)
Pull request description:
Flags for constant-time masking rely on the values being exactly `0` or `1` rather than `0` or true (any nonzero). One function, `secp256k1_fe_cmov` [documents](e7f7083b53/src/field.h (L315)) and [`VERIFY_CHECK`s](e7f7083b53/src/field_impl.h (L365)) this, but most don't.
This updates the documentation and adds `VERIFY_CHECK`s enforcing `flag == 0 || flag == 1` for:
`secp256k1_fe_storage_cmov`
`secp256k1_gej_cmov`
`secp256k1_ge_storage_cmov`
`secp256k1_scalar_cadd_bit`
`secp256k1_scalar_cond_negate`
`secp256k1_scalar_cmov`
`secp256k1_int_cmov`
ACKs for top commit:
furszy:
ACK ae00c55
hebasto:
re-ACK ae00c552df.
Tree-SHA512: c9d358929d39d93b0aea602d318429f7e82af96bf601f048a1cdeb0621b8adc6d1204648d352aa2060cb0f63db6dcf0da863854375ed313cea44dfad61c19a18
3b5b03f301 doc/bench: Added cmake build options to bench error messages (kevkevinpal)
Pull request description:
## Motivation
I wanted to try and run the benchmarking scripts and I noticed the recovery benchmark in `bench.c`. I wanted to run but I was using `cmake` and the error message telling me to use `./configure -enable-module-recovery` wasn't sufficient.
I figure rather than forcing users to look into the `CMakeLists.txt` file or anywhere else we should add this to the output
## Solution
I appended to the message to include the `-DSECP256K1_ENABLE_MODULE_...=ON` in the message.
ACKs for top commit:
real-or-random:
utACK 3b5b03f301
hebasto:
ACK 3b5b03f301, I have reviewed the code and it looks OK.
Tree-SHA512: 3a6c966b65ab3f0d6dda81e5dd95529087db3f2901f2686af68c16079ec5b323568f3c9acb155d92a6d50b4102faf0844d0d87216df001adbf69cab4ce86dabc
No behavior changes.
Refactors the previously monolithic ElligatorSwift test into isolated,
independent test cases. Doing so allows the test suite to execute
these cases in parallel rather than sequentially.
Overall, seen 35-40% tests time reduction locally.
This is quite useful for the Debug build with no optimizations,
which is noticeably slow.
#### Local Debug-build Results (7 jobs):
- master: 138.0 seconds.
- this PR: 89.3 seconds.
(~1.55× speedup, ~35% reduction)
#### Local Release-build Results (7 jobs):
- master: 9.5 seconds.
- this PR: 5.9 seconds.
(~1.61× speedup, ~38% reduction)
Flags for constant-time masking rely
on the values being exactly 0 or 1 rather
than 0 or true. Add VERIFY_CHECKs to enforce
in VERIFY builds as a preventative
measure and add documentation where relevant.
3daab83a60 refactor: remove ret from secp256k1_ec_pubkey_serialize (kevkevinpal)
Pull request description:
This is a follow-up to https://github.com/bitcoin-core/secp256k1/pull/1774#discussion_r2539737079
It is pretty straightforward to remove `ret` and to just return either `0` or `1`
ACKs for top commit:
real-or-random:
utACK 3daab83a60
theStack:
ACK 3daab83a60
Tree-SHA512: ce598d917455a2d25297436bf2b900a9e88a638617cb79ca22e467135035c334b6815911fe4429ff44dbd877e6d10a346d0b37f2e5a7459e5b35854023832d27
f5e815f430 remove secp256k1_eckey_pubkey_serialize function (Sebastian Falbesoner)
0d3659c547 use new `_eckey_pubkey_serialize{33,65}` functions in modules (ellswift,musig) (Sebastian Falbesoner)
adb76f82ea use new `_eckey_pubkey_serialize{33,65}` functions in public API (Sebastian Falbesoner)
fc7458ca3e introduce `secp256k1_eckey_pubkey_serialize{33,65}` functions (Sebastian Falbesoner)
Pull request description:
This PR splits up the pubkey serialization function `secp256k1_eckey_pubkey_serialize` into two variants for the compressed (33 bytes) and uncompressed (65 bytes) public key output format each, where only non-infinity group elements as input are allowed. The motivation is to simplify call-sites significantly, as they currently need to introduce two variables and a VERIFY_CHECKs on the return value and the in/out size parameter within a pre-processor block, typically leading to 8 lines of code. By using the new functions, the code is reduced to a single line of code that just calls the function (see #1773). This is helpful for already existing modules on master (ellswift, musig) and upcoming ones (silentpayments, see #1765).
One drawback is that the public API function `secp256k1_ec_pubkey_serialize` is now slightly more complex (we now call one of two functions instead of a single one, depending on whether the compressed flag is set or not), but that should hopefully not be a problem.
The commits are intentionally kept small to ease review, happy to squash them if that is preferred.
(Kudos to w0xlt for the initial idea (https://github.com/bitcoin-core/secp256k1/pull/1765#pullrequestreview-3462461331) and to real-or-random for the suggestion to split the already existing function (https://github.com/bitcoin-core/secp256k1/issues/1773#issuecomment-3540461718).)
ACKs for top commit:
real-or-random:
utACK f5e815f430
w0xlt:
ACK f5e815f430
Tree-SHA512: da576bbeae477f31ba76c0001f8df08b51fe5e31d67b422a238348ead3341bf37f0c1509ad9d0a93b63e6d61c152707c85beabd02f4eac3b3bdcff129e0ea750
This change improves separation from CMake build directories, which
typically use the "build" prefix.
Additionally, corresponding `.gitignore` entries have been refactored.
26166c4f5f ecmult_multi: reduce strauss memory usage by 30% (Jonas Nick)
Pull request description:
This is a draft because I'm not sure about the cleanest way to implement it.
ACKs for top commit:
real-or-random:
ACK 26166c4f5f benchmarks show no significant difference (only tried low point counts)
siv2r:
tACK 26166c4
hebasto:
ACK 26166c4f5f, I have reviewed the code and it looks OK.
Tree-SHA512: f289daee0b0b51451331eefdd99200a78bd83539365d38465c038dc0e6ad940daf821119f7161b08a2390cf046e3859a8f950f2fe881a427aba16353031def7d
