fac4be30482c21ac330e09ef8756c49e37faa6fa fuzz: Configure check for main function (take 2) (MarcoFalke)
Pull request description:
Actually fix https://github.com/google/honggfuzz/issues/336#issuecomment-702972138
Follow-up to #20065
Steps to test: `honggfuzz` section in doc/fuzzing.md
ACKs for top commit:
practicalswift:
cr ACK fac4be30482c21ac330e09ef8756c49e37faa6fa: patch looks correct!
Tree-SHA512: 893768c80439fe5d90b883ade89dc02f5bb80e27637916cf5575b6a9ed0b1c04942ff851342f5bbabb8666e6696715427feeb98f5301ad23c7b87b09e5872f97
faf3b4b53359850e0c47e67ead1c2b6dccc55531 refactor: Treat ArgsManager::Flags as uint32_t explicitly (MarcoFalke)
Pull request description:
The underlying type might be implementation defined, which is probably why the sanitizer kills the fuzz tests.
Fix that by pinning the underlying type.
This refactor does not change behaviour and only affects the sanitizer in tests.
ACKs for top commit:
practicalswift:
cr ACK faf3b4b53359850e0c47e67ead1c2b6dccc55531
Tree-SHA512: d446824836e1037b4200ba3630c8628090678cfad45559866275d8e06349f7c8cdb7e816619f5afb35f9f65299cc00e046d2f81b73cd8eb843e2e15676b647d5
590bda79e876d9b959083105b8c7c41dd87706eb scripted-diff: Remove setup_clean_chain if default is not changed (Fabian Jahr)
98892f39e3d079c73bff7f2a5d5420fa95270497 doc: Improve setup_clean_chain documentation (Fabian Jahr)
Pull request description:
The first commit improves documentation on setup_clean_chain which is misunderstood quite frequently. Most importantly it fixes the TestShell docs which are simply incorrect.
The second commit removes the instances of `setup_clean_clain` in functional tests where it is not changing the default.
This used to be part of #19168 which also sought to rename`setup_clean_chain`.
ACKs for top commit:
jonatack:
ACK 590bda79e876d9b959083105b8c7c41dd87706eb
Tree-SHA512: a7881186e65d31160b8f84107fb185973b37c6e50f190a85c6e2906a13a7472bb4efa9440bd37fe0a9ac5cd2d1e8559870a7e4380632d9a249eca8980b945f3e
20677ffa22e93e7408daadbd15d433f1e42faa86 validation: Guard all chainstates with cs_main (Carl Dong)
Pull request description:
```
This avoids a potential race-condition where a thread is reading the
ChainstateManager::m_active_chainstate pointer while another one is
writing to it. There is no portable guarantee that reading/writing the
pointer is thread-safe.
This is also done in way that mimics ::ChainstateActive(), so the
transition from that function to this method is easy.
More discussion:
1. https://github.com/bitcoin/bitcoin/pull/20749#discussion_r559544027
2. https://github.com/bitcoin/bitcoin/pull/19806#discussion_r561023961
3. https://github.com/bitcoin/bitcoin/pull/19806#issuecomment-768946522
4. https://github.com/bitcoin/bitcoin/pull/19806#issuecomment-768955695
```
Basically this PR removes the loaded-but-unfired footgun, which:
- Is multiplied (but still unshot) in the chainman deglobalization PRs (#20158)
- Is shot in the test framework in the au.activate PR (#19806)
ACKs for top commit:
jnewbery:
code review ACK 20677ffa22e93e7408daadbd15d433f1e42faa86. I've verified by eye that neither of these members are accessed without cs_main.
ryanofsky:
Code review ACK 20677ffa22e93e7408daadbd15d433f1e42faa86. It is safer to have these new `GUARDED_BY` annotations and locks than not to have them, but in the longer run I think every `LOCK(cs_main)` added here and added earlier in f92dc6557a153b390a1ae1d0808ff7ed5d02c66e from #20749 should be removed and replaced with `EXCLUSIVE_LOCKS_REQUIRED(cs_main)` on the accessor methods instead. `cs_main` is a high level lock that should be explicitly acquired at a high level to prevent the chain state from changing. It shouldn't be acquired recursively in low-level methods just to read pointer values atomically.
Tree-SHA512: 68a3a46d79a407b774eab77e1d682a97e95f1672db0a5fcb877572e188bec09f3a7b47c5d0cc1f2769ea276896dcbe97cb35c861acf7d8e3e513e955dc773f89
fa61b9d1a68820758f9540653920deaeae6abe79 util: Add ArgsManager::GetCommand() and use it in bitcoin-wallet (MarcoFalke)
7777105a24a36b62df35d12ecf6c6370671568c8 refactor: Move all command dependend checks to ExecuteWalletToolFunc (MarcoFalke)
fa06bce4ac17f93decd4ee38c956e7aa55983f0d test: Add tests (MarcoFalke)
fac05ccdade8b34c969b9cd9b37b355bc0aabf9c wallet: [refactor] Pass ArgsManager to WalletAppInit (MarcoFalke)
Pull request description:
This not only moves the parsing responsibility out from the wallet tool, but it also makes it easier to implement bitcoin-util #19937
Fixes: #20902
ACKs for top commit:
ajtowns:
ACK fa61b9d1a68820758f9540653920deaeae6abe79
fjahr:
Code review ACK fa61b9d1a68820758f9540653920deaeae6abe79
Tree-SHA512: 79622b806e8bf9dcd0dc24a8a6687345710df57720992e83a41cd8d6762a6dc112044ebc58fcf6e8fbf45de29a79b04873c5b8c2494a1eaaf902a2884703e47b
f22a3ec1403293f45a06558d105f0da624c5ebed build: make macOS HOST in download-osx generic (fanquake)
Pull request description:
This was missed in #20419, and the update before that, so just make this non-versioned so that we don't have to worry about it. This is fine, because it's just for downloading sources.
ACKs for top commit:
RandyMcMillan:
ACK f22a3ec1403293f45a06558d105f0da624c5ebed
dongcarl:
utACK f22a3ec1403293f45a06558d105f0da624c5ebed
Tree-SHA512: 3a6993a69594a793a5185e4ba48858443a1002a37b96ff881d39ca7719c79432b35d709bd9a9379f8046bdbeb716c5e1598f273a7e7e3f3bf528b6a807abe5ec
Environment variables and search paths can drastically effect the
operation of build tools.
Include these in our id string to mitigate against false cache hits.
When building with g++-10 (or 8) on Focal, binaries are being produced
with noexecstack by default, so we can remove the workaround of
explicitly passing "-Wl,-z,noexecstack" for risvc46 and powerpc64le.
When building for powerpc64 this is still required.
Compilers used change as follows:
Linux native GCC 7.5 -> GCC 8.4
Linux cross GCC 8.4 -> GCC 8.4
Windows mingw-w64 7.3 -> mingw-w64 9.3
macOS Clang 8.0.0 -> Clang 8.0.0
The macOS and Win cross builds in the CI are updated to use Focal, and
per the op, running the security tests is disabled in the Windows
build.
the i2p peer counts column is displayed iff the node is connected
to at least one i2p peer, so this doesn't add clutter for users
who are not running an i2p service
572fd0f7382bd0e6c7acc27dc354fae8489ab0a0 doc: More precise -debug and -debugexclude doc (wodry)
Pull request description:
I wondered how one could enable debug logging with `-debug=<category>` for multiple categories. Found out solution is to specify that option multiple times for each wanted category.
This PR documents this behavior and uses the same wording for the same behavior of `-debugexclude=<category>` to make that also clear and stringent.
ACKs for top commit:
laanwj:
ACK 572fd0f7382bd0e6c7acc27dc354fae8489ab0a0
MarcoFalke:
ACK 572fd0f7382bd0e6c7acc27dc354fae8489ab0a0
theStack:
ACK 572fd0f7382bd0e6c7acc27dc354fae8489ab0a0
Tree-SHA512: 8d93db37602fd5ff4247e7c11478e55b99c0e3d47eaa2bb901937805d8f2a466b3a198b713b760981c5411576b74c52e2909c46c6d3f2e0e04215fd521b65cf7
This was missed in #20419, and the update before that, so just
make this un-versioned so that we don't have to worry about it.
This is fine, because it's just for downloading sources.
bff7c66e67aa2f18ef70139338643656a54444fe Add documentation to contrib folder (Troy Giorshev)
381f77be858d7417209b6de0b7cd23cb7eb99261 Add Message Capture Test (Troy Giorshev)
e4f378a505922c0f544b4cfbfdb169e884e02be9 Add capture parser (Troy Giorshev)
4d1a582549bc982d55e24585b0ba06f92f21e9da Call CaptureMessage at appropriate locations (Troy Giorshev)
f2a77ff97bec09dd5fcc043d8659d8ec5dfb87c2 Add CaptureMessage (Troy Giorshev)
dbf779d5deb04f55c6e8493ce4e12ed4628638f3 Clean PushMessage and ProcessMessages (Troy Giorshev)
Pull request description:
This PR introduces per-peer message capture into Bitcoin Core. 📓
## Purpose
The purpose and scope of this feature is intentionally limited. It answers a question anyone new to Bitcoin's P2P protocol has had: "Can I see what messages my node is sending and receiving?".
## Functionality
When a new debug-only command line argument `capturemessages` is set, any message that the node receives or sends is captured. The capture occurs in the MessageHandler thread. When receiving a message, it is captured as soon as the MessageHandler thread takes the message off of the vProcessMsg queue. When sending, the message is captured just before the message is pushed onto the vSendMsg queue.
The message capture is as minimal as possible to reduce the performance impact on the node. Messages are captured to a new `message_capture` folder in the datadir. Each node has their own subfolder named with their IP address and port. Inside, received and sent messages are captured into two binary files, msgs_recv.dat and msgs_sent.dat, like so:
```
message_capture/203.0.113.7:56072/msgs_recv.dat
message_capture/203.0.113.7:56072/msgs_sent.dat
```
Because the messages are raw binary dumps, included in this PR is a Python parsing tool to convert the binary files into human-readable JSON. This script has been placed on its own and out of the way in the new `contrib/message-capture` folder. Its usage is simple and easily discovered by the autogenerated `-h` option.
## Future Maintenance
I sympathize greatly with anyone who says "the best code is no code".
The future maintenance of this feature will be minimal. The logic to deserialize the payload of the p2p messages exists in our testing framework. As long as our testing framework works, so will this tool.
Additionally, I hope that the simplicity of this tool will mean that it gets used frequently, so that problems will be discovered and solved when they are small.
## FAQ
"Why not just use Wireshark"
Yes, Wireshark has the ability to filter and decode Bitcoin messages. However, the purpose of the message capture added in this PR is to assist with debugging, primarily for new developers looking to improve their knowledge of the Bitcoin Protocol. This drives the design in a different direction than Wireshark, in two different ways. First, this tool must be convenient and simple to use. Using an external tool, like Wireshark, requires setup and interpretation of the results. To a new user who doesn't necessarily know what to expect, this is unnecessary difficulty. This tool, on the other hand, "just works". Turn on the command line flag, run your node, run the script, read the JSON. Second, because this tool is being used for debugging, we want it to be as close to the true behavior of the node as possible. A lot can happen in the SocketHandler thread that would be missed by Wireshark.
Additionally, if we are to use Wireshark, we are at the mercy of whoever it maintaining the protocol in Wireshark, both as to it being accurate and recent. As can be seen by the **many** previous attempts to include Bitcoin in Wireshark (google "bitcoin dissector") this is easier said than done.
Lastly, I truly believe that this tool will be used significantly more by being included in the codebase. It's just that much more discoverable.
ACKs for top commit:
MarcoFalke:
re-ACK bff7c66e67aa2f18ef70139338643656a54444fe only some minor changes: 👚
jnewbery:
utACK bff7c66e67aa2f18ef70139338643656a54444fe
theStack:
re-ACK bff7c66e67aa2f18ef70139338643656a54444fe
Tree-SHA512: e59e3160422269221f70f98720b47842775781c247c064071d546c24fa7a35a0e5534e8baa4b4591a750d7eb16de6b4ecf54cbee6d193b261f4f104e28c15f47
