This is more code, but a much more elegant solution than having
`ElectrumExt` methods return `SyncResult`/`FullScanResult` and having an
`ElectrumResultExt` extention trait.
`ElectrumResultExt` trait is also introduced that adds methods which can
convert the `Anchor` type for the update `TxGraph`.
We also make use of the new `TxCache` fields in
`SyncRequest`/`FullScanRequest`. This way, we can avoid re-fetching full
transactions from Electrum if not needed.
Examples and tests are updated to use the new `ElectrumExt` API.
This PR removes `RelevantTxids` from the electrum crate and tracks
transactions in a `TxGraph`. This removes the need to separately
construct a `TxGraph` after a `full_scan` or `sync`.
86711d4f46f467c651238ad3804fdbe1d22a8600 doc(chain): add section for non-recommended K to descriptor assignments (Daniela Brozzoni)
de53d721913537f56281a134270eafd356f908ad test: Only the highest ord keychain is returned (Daniela Brozzoni)
9d8023bf56a693f1cb2ba340ed024c654307c069 fix(chain): introduce keychain-variant-ranking to `KeychainTxOutIndex` (志宇)
6c8748124fd40e0fee37f78ca30457441b13fbcb chore(chain): move `use` in `indexed_tx_graph.rs` so clippy is happy (志宇)
537aa03ae0f8bec4dc799d33738e9bb7977bdac1 chore(chain): update test so clippy does not complain (志宇)
ed117de7a5b1756482b2e6487855b80e97c597ba test(chain): applying changesets one-by-one vs aggregate should be same (志宇)
6a3fb849e86e0bc21086519ae0201b95ddde5bf4 fix(chain): simplify `Append::append` impl for `keychain::ChangeSet` (志宇)
1d294b734dd6f4639075cba271e2b40f437f998f fix: Run tests only if the miniscript feature is.. ..enabled, enable it by default (Daniela Brozzoni)
0e3e136f6fa7215f6391dbcc1c4781262111ce64 doc(bdk): Add instructions for manually inserting... ...secret keys in the wallet in Wallet::load (Daniela Brozzoni)
76afccc555feff084867d6f9406e4e74bee938cc fix(wallet): add expected descriptors as signers after creating from wallet::ChangeSet (Steve Myers)
4f05441a00b921efd661da0dff94d9c28e38b70d keychain::ChangeSet includes the descriptor (Daniela Brozzoni)
8ff99f27dfe45643bf401409ca72429a9b812873 ref(chain): Define test descriptors, use them... ...everywhere (Daniela Brozzoni)
b9902936a0d17498eec8866233d89b2882d0af8f ref(chain): move `keychain::ChangeSet` into `txout_index.rs` (志宇)
Pull request description:
Fixes#1101
- Moves keychain::ChangeSet inside `keychain/txout_index.rs` as now the `ChangeSet` depends on miniscript
- Slightly cleans up tests by introducing some constant descriptors
- The KeychainTxOutIndex's internal SpkIterator now uses DescriptorId
instead of K. The DescriptorId -> K translation is made at the
KeychainTxOutIndex level.
- The keychain::Changeset is now a struct, which includes a map for last
revealed indexes, and one for newly added keychains and their
descriptor.
### Changelog notice
API changes in bdk:
- Wallet::keychains returns a `impl Iterator` instead of `BTreeMap`
- Wallet::load doesn't take descriptors anymore, since they're stored in the db
- Wallet::new_or_load checks if the loaded descriptor from db is the same as the provided one
API changes in bdk_chain:
- `ChangeSet` is now a struct, which includes a map for last revealed
indexes, and one for keychains and descriptors.
- `KeychainTxOutIndex::inner` returns a `SpkIterator<(DescriptorId, u32)>`
- `KeychainTxOutIndex::outpoints` returns a `BTreeSet` instead of `&BTreeSet`
- `KeychainTxOutIndex::keychains` returns a `impl Iterator` instead of
`&BTreeMap`
- `KeychainTxOutIndex::txouts` doesn't return a ExactSizeIterator anymore
- `KeychainTxOutIndex::last_revealed_indices` returns a `BTreeMap`
instead of `&BTreeMap`
- `KeychainTxOutIndex::add_keychain` has been renamed to `KeychainTxOutIndex::insert_descriptor`, and now it returns a ChangeSet
- `KeychainTxOutIndex::reveal_next_spk` returns Option
- `KeychainTxOutIndex::next_unused_spk` returns Option
- `KeychainTxOutIndex::unbounded_spk_iter` returns Option
- `KeychainTxOutIndex::next_index` returns Option
- `KeychainTxOutIndex::reveal_to_target` returns Option
- `KeychainTxOutIndex::revealed_keychain_spks` returns Option
- `KeychainTxOutIndex::unused_keychain_spks` returns Option
- `KeychainTxOutIndex::last_revealed_index` returns Option
- `KeychainTxOutIndex::keychain_outpoints` returns Option
- `KeychainTxOutIndex::keychain_outpoints_in_range` returns Option
- `KeychainTxOutIndex::last_used_index` returns None if the keychain has never been used, or if it doesn't exist
### Checklists
#### All Submissions:
* [x] I've signed all my commits
* [x] I followed the [contribution guidelines](https://github.com/bitcoindevkit/bdk/blob/master/CONTRIBUTING.md)
* [x] I ran `cargo fmt` and `cargo clippy` before committing
#### New Features:
* [x] I've added tests for the new feature
* [x] I've added docs for the new feature
ACKs for top commit:
evanlinjin:
ACK 86711d4f46f467c651238ad3804fdbe1d22a8600
Tree-SHA512: 4b1c9a31951f67b18037b7dd9837acbc35823f21de644ab833754b74d20f5373549f81e66965ecd3953ebf4f99644c9fd834812acfa65f9188950f1bda17ab60
de2763a4b88e65b1c67a06148b4f9e012e561c5a ci: Pin clippy to rust 1.78.0 (valued mammal)
Pull request description:
This PR pins clippy check in CI to the rust 1.78 toolchain, which prevents new lints in stable releases from interrupting the usual workflow. Because rust versions are released on a predictable schedule, we can revisit this setting in the future as needed (say 3-6 months).
### Checklists
#### All Submissions:
* [x] I've signed all my commits
* [x] I followed the [contribution guidelines](https://github.com/bitcoindevkit/bdk/blob/master/CONTRIBUTING.md)
* [ ] I ran `cargo fmt` and `cargo clippy` before committing
ACKs for top commit:
danielabrozzoni:
ACK de2763a4b88e65b1c67a06148b4f9e012e561c5a
storopoli:
ACK de2763a4b88e65b1c67a06148b4f9e012e561c5a
notmandatory:
ACK de2763a4b88e65b1c67a06148b4f9e012e561c5a
oleonardolima:
ACK de2763a4b88e65b1c67a06148b4f9e012e561c5a
Tree-SHA512: 73cad29a5ff437290aca8f85a011c4f5fc4d9ff5755f3d3ef9fa1820f5631eda857b1a67955adfc6ef98145958c674cc09f7613b96f38cc30c75a656d872edbc
This fixes the bug with changesets not being monotone. Previously, the
result of applying changesets individually v.s. applying the aggregate
of changesets may result in different `KeychainTxOutIndex` states.
The nature of the changeset allows different keychain types to share the
same descriptor. However, the previous design did not take this into
account. To do this properly, we should keep track of all keychains
currently associated with a given descriptor. However, the API only
allows returning one keychain per spk/txout/outpoint (which is a good
API).
Therefore, we rank keychain variants by `Ord`. Earlier keychain variants
have a higher rank, and the first keychain will be returned.
We only need to loop though entries of `other`. The logic before was
wasteful because we were also looping though all entries of `self` even
if we do not need to modify the `self` entry.
- The KeychainTxOutIndex's internal SpkIterator now uses DescriptorId
instead of K. The DescriptorId -> K translation is made at the
KeychainTxOutIndex level.
- The keychain::Changeset is now a struct, which includes a map for last
revealed indexes, and one for newly added keychains and their
descriptor.
API changes in bdk:
- Wallet::keychains returns a `impl Iterator` instead of `BTreeMap`
- Wallet::load doesn't take descriptors anymore, since they're stored in
the db
- Wallet::new_or_load checks if the loaded descriptor from db is the
same as the provided one
API changes in bdk_chain:
- `ChangeSet` is now a struct, which includes a map for last revealed
indexes, and one for keychains and descriptors.
- `KeychainTxOutIndex::inner` returns a `SpkIterator<(DescriptorId, u32)>`
- `KeychainTxOutIndex::outpoints` returns a `impl Iterator` instead of `&BTreeSet`
- `KeychainTxOutIndex::keychains` returns a `impl Iterator` instead of
`&BTreeMap`
- `KeychainTxOutIndex::txouts` doesn't return a ExactSizeIterator
anymore
- `KeychainTxOutIndex::unbounded_spk_iter` returns an `Option`
- `KeychainTxOutIndex::next_index` returns an `Option`
- `KeychainTxOutIndex::last_revealed_indices` returns a `BTreeMap`
instead of `&BTreeMap`
- `KeychainTxOutIndex::reveal_to_target` returns an `Option`
- `KeychainTxOutIndex::reveal_next_spk` returns an `Option`
- `KeychainTxOutIndex::next_unused_spk` returns an `Option`
- `KeychainTxOutIndex::add_keychain` has been renamed to
`KeychainTxOutIndex::insert_descriptor`, and now it returns a
ChangeSet
We plan to record `Descriptor` additions into persistence. Hence, we
need to add `Descriptor`s to the changeset. This depends on
`miniscript`. Moving this into `txout_index.rs` makes sense as this is
consistent with all the other files. The only reason why this wasn't
this way before, is because the changeset didn't need miniscript.
Co-Authored-By: Daniela Brozzoni <danielabrozzoni@protonmail.com>
a577c22b12f4edf0b2f2f266fb13656caf1d21c4 fix(persist): add default feature to enable bdk_chain/std (Steve Myers)
Pull request description:
### Description
This PR adds a `default` feature to `bdk_persist` so it can be build on its own. Once #1422 is done we can remove the `default`again.
### Notes to the reviewers
I need to be able to build `bdk_persist` on its own so I can publish it to crates.io.
### Checklists
#### All Submissions:
* [x] I've signed all my commits
* [x] I followed the [contribution guidelines](https://github.com/bitcoindevkit/bdk/blob/master/CONTRIBUTING.md)
* [x] I ran `cargo fmt` and `cargo clippy` before committing
ACKs for top commit:
ValuedMammal:
ACK a577c22b12f4edf0b2f2f266fb13656caf1d21c4
oleonardolima:
ACK a577c22b12f4edf0b2f2f266fb13656caf1d21c4
storopoli:
ACK a577c22b12f4edf0b2f2f266fb13656caf1d21c4
Tree-SHA512: 8b07a9e4974dec8812ca19ce7226dcaece064270a0be8b83d3c326fdf1e89b051eb0bd8aa0eda9362b2c8233ecd6003b70c92ee046603973d8d40611418c3841
22aa534d7648e5808414ea3adfcfb702572bd6c9 feat: use `Amount` on `TxBuilder::add_recipient` (Leonardo Lima)
d5c0e7200cba0c3b4d3e3fbea168cd07ee6c1d2c feat: use `Amount` on `spk_txout_index` and related (Leonardo Lima)
8a33d98db977a07e130ad57fa9c658a5c90d4a4b feat: update `wallet::Balance` to use `bitcoin::Amount` (Leonardo Lima)
Pull request description:
fixes#823
<!-- You can erase any parts of this template not applicable to your Pull Request. -->
### Description
It's being used on `Balance`, and throughout the code, an `u64` represents the amount, which relies on the user to infer its sats, not millisats, or any other representation.
It updates the usage of `u64` on `Balance`, and other APIs:
- `TxParams::add_recipient`
- `KeyChainTxOutIndex::sent_and_received`, `KeyChainTxOutIndex::net_value`
- `SpkTxOutIndex::sent_and_received`, `SpkTxOutIndex::net_value`
<!-- Describe the purpose of this PR, what's being adding and/or fixed -->
### Notes to the reviewers
<!-- In this section you can include notes directed to the reviewers, like explaining why some parts
of the PR were done in a specific way -->
It updates some of the APIs to expect the `bitcoin::Amount`, but it does not update internal usage of u64, such as `TxParams` still expects and uses `u64`, please see the PR comments for related discussion.
### Changelog notice
<!-- Notice the release manager should include in the release tag message changelog -->
<!-- See https://keepachangelog.com/en/1.0.0/ for examples -->
- Changed the `keychain::Balance` struct fields to use `Amount` instead of `u64`.
- Changed the `add_recipient` method on `TxBuilder` implementation to expect `bitcoin::Amount`.
- Changed the `sent_and_received`, and `net_value` methods on `KeyChainTxOutIndex` to expect `bitcoin::Amount`.
- Changed the `sent_and_received`, and `net_value` methods on `SpkTxOutIndex` to expect `bitcoin::Amount`.
### Checklists
#### All Submissions:
* [x] I've signed all my commits
* [x] I followed the [contribution guidelines](https://github.com/bitcoindevkit/bdk/blob/master/CONTRIBUTING.md)
* [x] I ran `cargo fmt` and `cargo clippy` before committing
#### New Features:
* [x] I've added tests for the new feature
* [x] I've added docs for the new feature
#### Bugfixes:
* [x] This pull request breaks the existing API
* [ ] I've added tests to reproduce the issue which are now passing
* [x] I'm linking the issue being fixed by this PR
ACKs for top commit:
evanlinjin:
ACK 22aa534d7648e5808414ea3adfcfb702572bd6c9
Tree-SHA512: c4e8198d96c0d66cc3d2e4149e8a56bb7565b9cd49ff42113eaebd24b1d7bfeecd7124db0b06524b78b8891ee1bde1546705b80afad408f48495cf3c02446d02
f6218e4741a4b036d33a9f075ce5feb484e9bd87 chore: reexport crates in `TestEnv` (Wei Chen)
125959976f63a1e12fc139f048dc2856e993efe5 chore: remove `anyhow` dev dependency from `electrum`, `esplora`, and `bitcoind_rpc` (Wei Chen)
Pull request description:
<!-- You can erase any parts of this template not applicable to your Pull Request. -->
### Description
Reexports `electrsd` in `TestEnv` to remove the `electrsd` dev depedency out of `bdk_electrum` and `bdk_esplora`.
Credit to @ValuedMammal for the idea.
Since `bitcoind` reexports `anyhow`, this dev dependency was also removed from `bdk_electrum`, `bdk_esplora`, and `bdk_bitcoind_rpc`. `bitcoind`, `bitcoincore_rpc` and `electrum_client` were also reexported for convenience.
### Changelog notice
* Change `bdk_testenv` to re-export internally used crates.
### Checklists
#### All Submissions:
* [x] I've signed all my commits
* [x] I followed the [contribution guidelines](https://github.com/bitcoindevkit/bdk/blob/master/CONTRIBUTING.md)
* [x] I ran `cargo fmt` and `cargo clippy` before committing
ACKs for top commit:
evanlinjin:
ACK f6218e4741a4b036d33a9f075ce5feb484e9bd87
Tree-SHA512: c7645eb91d08d4ccb80982a992f691b5a8c0df39df506f6b361bc6f2bb076d62cbe5bb5d88b4c684c36e22464c0674f21f6ef4e23733f89b03aa12ec43a67cba
f6dc6890c37752e80c86c68ae3ca3cc4fac6245e docs(esplora): fixed `full_scan` and `sync` documentation (Wei Chen)
Pull request description:
<!-- You can erase any parts of this template not applicable to your Pull Request. -->
### Description
Fixed documentation for `full_scan` and `sync` in `bdk_esplora`.
### Changelog notice
<!-- Notice the release manager should include in the release tag message changelog -->
<!-- See https://keepachangelog.com/en/1.0.0/ for examples -->
* Updated documentation for `full_scan` and `sync` in `bdk_esplora`.
### Checklists
#### All Submissions:
* [x] I've signed all my commits
* [x] I followed the [contribution guidelines](https://github.com/bitcoindevkit/bdk/blob/master/CONTRIBUTING.md)
* [x] I ran `cargo fmt` and `cargo clippy` before committing
ACKs for top commit:
evanlinjin:
ACK f6dc6890c37752e80c86c68ae3ca3cc4fac6245e
storopoli:
ACK f6dc6890c37752e80c86c68ae3ca3cc4fac6245e
Tree-SHA512: 900fb1a2839379af867a6effad32ec4bdfb897330a72ee1e1ec203299e7f3d5fae576550aeed8fd93c5c70a13ad2b0e898033d8b45b490319b5d74216b93f332
- update all fields `immature`, ` trusted_pending`, `unstrusted_pending`
and `confirmed` to use the `bitcoin::Amount` instead of `u64`
- update all `impl Balance` methods to use `bitcoin::Amount`
- update all tests that relies on `keychain::Balance`
db4734747240d2e1cc72bbfd7be6c49baa8d2967 test(wallet): add thread safety test (Rob N)
Pull request description:
### Description
`Wallet` auto-implements `Send` and `Sync` after removing the generic. This test is a compile time error if there are changes to `Wallet` in the future that make it unsafe to send between threads. See #1387 for discussion.
### Checklists
#### All Submissions:
* [x] I've signed all my commits
* [x] I followed the [contribution guidelines](https://github.com/bitcoindevkit/bdk/blob/master/CONTRIBUTING.md)
* [x] I ran `cargo fmt` and `cargo clippy` before committing
#### New Features:
* [x] I've added tests for the new feature
* [ ] I've added docs for the new feature
#### Bugfixes:
* [ ] This pull request breaks the existing API
* [ ] I've added tests to reproduce the issue which are now passing
* [ ] I'm linking the issue being fixed by this PR
ACKs for top commit:
evanlinjin:
ACK db4734747240d2e1cc72bbfd7be6c49baa8d2967
Tree-SHA512: 490e666bc503f15286268db7e5e2f75ee44ad2f80251d6f7a01af2a435023b87607eee33623712433ea8d27511be63c6c1e9cad4159b3fe66a4644cfa9e344fb
2cda9f44ee1807898e4f2afffc95774839fee678 Bump bdk version to 1.0.0-alpha.10 (Daniela Brozzoni)
Pull request description:
### Description
bdk_chain to 0.13.0
bdk_bitcoind_rpc to 0.9.0
bdk_electrum to 0.12.0
bdk_esplora to 0.12.0
bdk_file_store to 0.10.0
bdk_testenv to 0.3.0
bdk_persist to 0.2.0
### Checklists
#### All Submissions:
- [x] I've signed all my commits
- [x] I followed the [contribution guidelines](https://github.com/bitcoindevkit/bdk/blob/master/CONTRIBUTING.md)
- [x] I ran cargo fmt and cargo clippy before committing
ACKs for top commit:
storopoli:
ACK 2cda9f44ee1807898e4f2afffc95774839fee678
Tree-SHA512: 7d3e5f2c9b9da13713e3bb1e6a11d07e9c381221c837a002aefb780698b1d45d64f2582bd0445ecdf7432bf3fe0ba5d6dadd43aa413cf4e5e557f7334a02fa06
bdk_chain to 0.13.0
bdk_bitcoind_rpc to 0.9.0
bdk_electrum to 0.12.0
bdk_esplora to 0.12.0
bdk_file_store to 0.10.0
bdk_testenv to 0.3.0
bdk_persist to 0.2.0
a5fb7fdf505a53b5865ed83d53745e7efa5e7818 fix: Cargo clippy lints after rust 1.78 (Daniela Brozzoni)
Pull request description:
<!-- You can erase any parts of this template not applicable to your Pull Request. -->
### Description
Caught when trying to release (#1420), clippy failed randomly although it worked on master, this happened because rust 1.78 had just been release and we use clippy stable. IMHO we should pin the clippy version in CI and bump it manually at each new rust release.
### Checklists
#### All Submissions:
* [x] I've signed all my commits
* [x] I followed the [contribution guidelines](https://github.com/bitcoindevkit/bdk/blob/master/CONTRIBUTING.md)
* [x] I ran `cargo fmt` and `cargo clippy` before committing
ACKs for top commit:
notmandatory:
ACK a5fb7fdf505a53b5865ed83d53745e7efa5e7818
Tree-SHA512: c803366367576224f9e9690cdee2c0161fc083550355415f9174e93ada2f597440f54ac966bb3ebecdc916824d43de17ac72801e4ef0f75c8a1df640fe40df6d
c0374a0eeb155f1cb4f9d99223f6bc65d24f7df8 feat(chain): `SyncRequest` now uses `ExactSizeIterator`s (志宇)
0f94f24aaf6374fe2d7c2abf32a870d3849fb8cc feat(esplora)!: update to use new sync/full-scan structures (志宇)
4c52f3e08e85e73db379ee3411cceb8dbcec92e1 feat(wallet): make wallet compatible with sync/full-scan structures (志宇)
cdfec5f90726f2313963c3a71e5d18bb10624736 feat(chain): add sync/full-scan structures for spk-based syncing (志宇)
Pull request description:
Fixes#1153
Replaces #1194
### Description
Introduce universal structures that represent sync/full-scan requests/results.
### Notes to the reviewers
This is based on #1194 but is different in the following ways:
* The functionality to print scan/sync progress is not reduced.
* `SyncRequest` and `FullScanRequest` is simplified and fields are exposed for more flexibility.
### Changelog notice
* Add universal structures for initiating/receiving sync/full-scan requests/results for spk-based syncing.
* Updated `bdk_esplora` chain-source to make use of new universal sync/full-scan structures.
### Checklists
#### All Submissions:
* [x] I've signed all my commits
* [x] I followed the [contribution guidelines](https://github.com/bitcoindevkit/bdk/blob/master/CONTRIBUTING.md)
* [x] I ran `cargo fmt` and `cargo clippy` before committing
#### New Features:
* [x] I've added tests for the new feature
* [x] I've added docs for the new feature
ACKs for top commit:
notmandatory:
tACK c0374a0eeb155f1cb4f9d99223f6bc65d24f7df8
Tree-SHA512: c2ad66d972a6785079bca615dfd128edcedf6b7a02670651a0ab1ce5b5174dd96f54644680eedbf55e3f1955fe5c34f632eadbd3f71d7ffde658753c6c6d42be
81de8f60516899fad2ec99cbd15d6b8d02385ebb feat(bdk-persist): extract persistence traits to new crate (Rob N)
Pull request description:
### Description
#1387 introduced `anyhow` as a dependency to remove generics from `Wallet`. Introducing a new crate for persistence types removes the dependency on `anyhow` for `bdk_chain`. Resolves#1409, as well as removing the old documentation for "tracker".
### Notes to the reviewers
Open for any comments.
### Changelog notice
- Introduce `bdk-persist` crate
### Checklists
#### All Submissions:
* [x] I've signed all my commits
* [x] I followed the [contribution guidelines](https://github.com/bitcoindevkit/bdk/blob/master/CONTRIBUTING.md)
* [x] I ran `cargo fmt` and `cargo clippy` before committing
#### New Features:
* [ ] I've added tests for the new feature
* [ ] I've added docs for the new feature
#### Bugfixes:
* [ ] This pull request breaks the existing API
* [ ] I've added tests to reproduce the issue which are now passing
* [x] I'm linking the issue being fixed by this PR
ACKs for top commit:
evanlinjin:
ACK 81de8f60516899fad2ec99cbd15d6b8d02385ebb
Tree-SHA512: 29b192b13f3951cc67c06bec7f788d8d7a4aeaf2ffcbf9476d4a6567529d284a93594c8d94b69741a68a9aadfdc9f6c4178084a2298c505e8e0d505219400382
* Changed `Wallet::apply_update` to also take in anything that
implements `Into<Update>`. This allows us to directly apply a
`FullScanResult` or `SyncResult`.
* Added `start_full_scan` and `start_sync_with_revealed_spks` methods to
`Wallet`.
Co-authored-by: Steve Myers <steve@notmandatory.org>
