e6e622e5a0e22c2ac1b50b96af818e412d67ac54 Implement O(1) OP_IF/NOTIF/ELSE/ENDIF logic (Pieter Wuille)
d0e8f4d5d8ddaccb37f98b7989fb944081e41ab8 [refactor] interpreter: define interface for vfExec (Anthony Towns)
89fb241c54fc85befacfa3703d8e21bf3b8a76eb Benchmark script verification with 100 nested IFs (Pieter Wuille)
Pull request description:
While investigating what mechanisms are possible to maximize the per-opcode verification cost of scripts, I noticed that the logic for determining whether a particular opcode is to be executed is O(n) in the nesting depth. This issue was also pointed out by Sergio Demian Lerner in https://bitslog.wordpress.com/2017/04/17/new-quadratic-delays-in-bitcoin-scripts/, and this PR implements a variant of the O(1) algorithm suggested there.
This is not a problem currently, because even with a nesting depth of 100 (the maximum possible right now due to the 201 ops limit), the slowdown caused by this on my machine is around 70 ns per opcode (or 0.25 s per block) at worst, far lower than what is possible with other opcodes.
This PR mostly serves as a proof of concept that it's possible to avoid it, which may be relevant in discussions around increasing the opcode limits in future script versions. Without it, the execution time of scripts can grow quadratically with the nesting depth, which very quickly becomes unreasonable.
This improves upon #14245 by completely removing the `vfExec` vector.
ACKs for top commit:
jnewbery:
Code review ACK e6e622e5a0e22c2ac1b50b96af818e412d67ac54
MarcoFalke:
ACK e6e622e5a0e22c2ac1b50b96af818e412d67ac54 🐴
fjahr:
ACK e6e622e5a0e22c2ac1b50b96af818e412d67ac54
ajtowns:
ACK e6e622e5a0e22c2ac1b50b96af818e412d67ac54
laanwj:
concept and code review ACK e6e622e5a0e22c2ac1b50b96af818e412d67ac54
jonatack:
ACK e6e622e5a0e22c2ac1b50b96af818e412d67ac54 code review, build, benches, fuzzing
Tree-SHA512: 1dcfac3411ff04773de461959298a177f951cb5f706caa2734073bcec62224d7cd103767cfeef85cd129813e70c14c74fa8f1e38e4da70ec38a0f615aab1f7f7
This replaces one remaining instance of the literal "BTC" string with
the CURRENCY_UNIT constant, as is done in most of the codebase already.
The other remaining instance (which is just part of a log message and thus
not really user-visible) is just removed.
After this change, no instance of literal "BTC" remains anywhere in the
non-Qt and non-test codebase.
09e25071f40c564af08a1386c39c4f2d8eb484b6 Cache parent xpub inside of BIP32PubkeyProvider (Andrew Chow)
deb791c7ba057a3765d09b12bf3e55547a5298e4 Only cache xpubs that have a hardened last step (Andrew Chow)
f76733eda5f4c161e9eb47c74b949582ab8f448a Cache the immediate derivation parent xpub (Andrew Chow)
58f54b686f663e4c46a2cf7a64560409007c7eb3 Add DescriptorCache* read_cache and DescriptorCache* write_cache to Expand and GetPubKey (Andrew Chow)
66c2cadc91d26074b89e5ada68350b5c8676efac Rename BIP32PubkeyProvider.m_extkey to m_root_extkey (Andrew Chow)
df55d44d0de2174ba74ed3a28bef5e83b0a51b47 Track the index of the key expression in PubkeyProvider (Andrew Chow)
474ea3b927ddc67e64ae78e08c20c9264817e84d Introduce DescriptorCache struct which caches xpubs (Andrew Chow)
Pull request description:
Improves the descriptor cache by changing it from a `std::vector<unsigned char>` to a newly introduced `DescriptorCache` class. Instead of serializing pubkeys and whatever else we would want to cache in a way that may not be backwards compatible, we instead create a `DescriptorCache` object and populate it. This object contains only an xpub cache. Since the only `PubkeyProvider` that used the cache is the `BIP32PubkeyProvider` we just have it store the xpubs instead of the pubkeys. This allows us to have both the parent xpub and the child xpubs in the same container. The map is keyed by `KeyOriginInfo`.
Sine we are caching `CExtPubKey`s in `DescriptorCache`, `BIP32PubKeyProviders` can use the cached parent xpubs to derive the children if unhardened derivation is used in the last step. This also means that we can still derive the keys for a `BIP32PubkeyProvider` that has hardened derivation steps. When combined with descriptor wallets, this should allow us to be able to import a descriptor with an `xprv` and hardened steps and still be able to derive from it. In that sense, this is an alternative to #18163
To test that this works, the tests have been updated to do an additional `Expand` at the `i + 1` position. This expansion is not cached. We then do an `ExpandFromCache` at `i + 1` and use the cache that was produced by the expansion at `i`. This way, we won't have the child xpubs for `i + 1` but we will have the parent xpubs. So this checks whether the parent xpubs are being stored and can be used to derive the child keys. Descriptors that have a hardened last step are skipped for this part of the test because that will always require private keys.
ACKs for top commit:
instagibbs:
code review re-re-ACK 09e25071f4
Sjors:
re-ACK 09e25071f40c564af08a1386c39c4f2d8eb484b6
Tree-SHA512: 95c8d0092274cdf115ce39f6d49dec767679abf3758d5b9e418afc308deca9dc6f67167980195bcc036cd9c09890bbbb39ec1dacffbfacdc03efd72a7e23b276
c8e24ddce31a8de6255b23c19d958c1cd44a8847 [REFACTOR] Abstract out script execution out of VerifyWitnessProgram() (Pieter Wuille)
Pull request description:
This is a refactoring cherry-picked out of #17977. As it touches consensus code, I don't think this would ordinarily meet the bar for review cost vs benefit. However, it simplifies the changes for Taproot significantly, and if it's going to be necessitated by inclusion of that code, I may as well give it some additional attention by PRing it independently.
ACKs for top commit:
fjahr:
Re-ACK c8e24ddce31a8de6255b23c19d958c1cd44a8847
theStack:
re-ACK c8e24ddce3
Empact:
Code Review Re-ACK c8e24ddce3
ajtowns:
ACK c8e24ddce31a8de6255b23c19d958c1cd44a8847
jnewbery:
ACK c8e24ddce31a8de6255b23c19d958c1cd44a8847
jonatack:
ACK c8e24dd
Tree-SHA512: 96c2aa5d2f9c7c802bcc008f5cde55b1dfedfaf42e34101331e6c0d594acdf6437661102dc939718f0877c20451336855dfbaa8aa8f57d9e722a7fa7329e3a46
a33cffbeabcc42137c4a66aa19b7dd1d300e6d73 util: HelpExampleRpc formatting fixup (Jon Atack)
Pull request description:
Minor visual fixup of the HelpExampleRpc template; conforms to the JSON-RPC spec as per https://www.jsonrpc.org/specification#examples. (I'm... somewhat embarassed to open such a minor change, but this is what is shown in all the CLI/RPC help docs.)
ACKs for top commit:
laanwj:
ACK a33cffbeabcc42137c4a66aa19b7dd1d300e6d73
Tree-SHA512: 8f1dee080c224742fff60a33fec6f5fb1d59c9fa51f3f2a67bf2e1837dbfa25f12a69e34518936588940013b0e61f55378b4f1a571c47c3cb081ca5b245e1091
When the user doesn't specificy inputs, it makes sense to automatically select them. But when the user does specify inputs, we now fail if the amount is insufficient, unless addInputs is set to true.
NotifyEntryAdded never had any subscribers so can be removed.
Since ConnectTrace no longer subscribes to NotifyEntryRemoved, there are
now no subscribers.
The CValidationInterface TransactionAddedToMempool and
TransactionRemovedFromMempool methods can now provide this
functionality. There's no need for a special notifications framework for
the mempool.
ConnectTrace used to subscribe to the mempool's NotifyEntryRemoved
callback to be notified of transactions removed for conflict. Since
PerBlockConnectTrace no longer tracks conflicted transactions,
ConnectTrace no longer requires these notifications.
The only CValidationInterface client that cares about transactions that
are removed from the mempool because of CONFLICT is the wallet.
Start using the TransactionRemovedFromMempool method to notify about
conflicted transactions instead of using the vtxConflicted vector in
BlockConnected.
8a2a652e6fab5eb8224beefcc07d9011b61865a8 Remove redundant type information from rpc docs (David O'Callaghan)
Pull request description:
Simple edit of the RPC calls to remove redundant text ("A json object/array ...") from the beginning of help.
Fixes: #18258
Top commit has no ACKs.
Tree-SHA512: cbbf760e0b7b4eda61c40b420ed77f5d878318e37b0eb13e63567212240b2c4ecc15d84030e98075e21c9ae9016539adfd201e5661ea824166a76d335180c32f
3e32499909ca8127baaa9b40ad113b25ee151bbd Change example addresses to bech32 (Yusuf Sahin HAMZA)
Pull request description:
This is a follow-up PR to #18197 that fixes RPCExamples.
Fixes #18185.
ACKs for top commit:
MarcoFalke:
ACK 3e32499909ca8127baaa9b40ad113b25ee151bbd
jonatack:
ACK 3e32499
Tree-SHA512: c7a6410ef8b6e169016c2c5eac3e6b9501caabd0e8a0871ec31e56bfc44589f056d3f5cb55b5a13bba36f6c15136c2352f883e30e4dcc0997ffd36b27f9173b9
fab7d14ea5a4305317d66f35beb3225a07823d42 test: Check that wait_until returns if time point is in the past (MarcoFalke)
Pull request description:
Add an explicit regression test for the condvar bug (#18227), so that this doesn't happen again
ACKs for top commit:
laanwj:
ACK fab7d14ea5a4305317d66f35beb3225a07823d42
Tree-SHA512: 6ec0d0b3945cae87a001e367af34cca1953a8082b4a0d9f8a20d30acd1f36363e98035d4eb173ff786cf6692d352d41f960633415c46394af042eb44e3b5ad71
9220a0fdd0f3dc2c8dd7cbeefac7d11106451b51 tests: Add one specialized ProcessMessage(...) fuzzing binary per message type for optimal results when using coverage-guided fuzzing (practicalswift)
fd1dae10b4a549ba9292d837235d59bd9eebbed3 tests: Add fuzzing harness for ProcessMessage(...) (practicalswift)
Pull request description:
Add fuzzing harness for `ProcessMessage(...)`. Enables high-level fuzzing of the P2P layer.
All code paths reachable from this fuzzer can be assumed to be reachable for an untrusted peer.
Seeded from thin air (an empty corpus) this fuzzer reaches roughly 20 000 lines of code.
To test this PR:
```
$ make distclean
$ ./autogen.sh
$ CC=clang CXX=clang++ ./configure --enable-fuzz \
--with-sanitizers=address,fuzzer,undefined
$ make
$ src/test/fuzz/process_message
…
```
Worth noting about this fuzzing harness:
* To achieve a reasonable number of executions per seconds the state of the fuzzer is unfortunately not entirely reset between `test_one_input` calls. The set-up (`FuzzingSetup` ctor) and tear-down (`~FuzzingSetup`) work is simply too costly to be run on every iteration. There is a trade-off to handle here between a.) achieving high executions/second and b.) giving the fuzzer a totally blank slate for each call. Please let me know if you have any suggestion on how to improve this situation while maintaining >1000 executions/second.
* To achieve optimal results when using coverage-guided fuzzing I've chosen to create one specialised fuzzing binary per message type (`process_message_addr`, `process_message_block`, `process_message_blocktxn `, etc.) and one general fuzzing binary (`process_message`) which handles all messages types. The latter general fuzzer can be seeded with inputs generated by the former specialised fuzzers.
Happy fuzzing friends!
ACKs for top commit:
MarcoFalke:
ACK 9220a0fdd0 🏊
Tree-SHA512: c314ef12b0db17b53cbf3abfb9ecc10ce420fb45b17c1db0b34cabe7c30e453947b3ae462020b0c9f30e2c67a7ef1df68826238687dc2479cd816f0addb530e5
