a4f28c079e3bc4500e8329aeed230d8100d38617 chore: improve LocalChain::apply_header_connected_to doc (LLFourn)
8ec65f0b8ef7d452a0bdba6760c46bd8511c91ff feat(example): add RPC wallet example (Vladimir Fomene)
a7d01dc39acb3ff7f4bc8e237c7862d019a91cd1 feat(chain)!: make `IndexedTxGraph::apply_block_relevant` more efficient (志宇)
e0512acf9476fa3fa7da9cc28a222432f974aad5 feat(bitcoind_rpc)!: emissions include checkpoint and connected_to data (志宇)
8f2d4d9d400e398b54cf1bad36057f3f8f4a69a4 test(chain): `LocalChain` test for update that is shorter than original (志宇)
9467cad55d68354fe037d57bceeee57c2032bd51 feat(wallet): introduce block-by-block api (Vladimir Fomene)
d3e5095df10d21ff5cb7f8ce4dab7922f3cb6e35 feat(chain): add `apply_header..` methods to `LocalChain` (志宇)
2b61a122ff40253f2e67c0ae6824ce881c9f12a1 feat(chain): add `CheckPoint::from_block_ids` convenience method (志宇)
Pull request description:
### Description
Introduce block-by-block API for `bdk::Wallet`. A `wallet_rpc` example is added to demonstrate syncing `bdk::Wallet` with the `bdk_bitcoind_rpc` chain-source crate.
The API of `bdk_bitcoind_rpc::Emitter` is changed so the receiver knows how to connect to the block emitted.
### Notes to the reviewers
### Changelog notice
Added
* `Wallet` methods to apply full blocks (`apply_block` and `apply_block_connected_to`) and a method to apply a batch of unconfirmed transactions (`apply_unconfirmed_txs`).
* `CheckPoint::from_block_ids` convenience method.
* `LocalChain` methods to apply a block header (`apply_header` and `apply_header_connected_to`).
* Test to show that `LocalChain` can apply updates that are shorter than original. This will happen during reorgs if we sync wallet with `bdk_bitcoind_rpc::Emitter`.
Fixed
* `InsertTxError` now implements `std::error::Error`.
#### All Submissions:
* [x] I've signed all my commits
* [x] I followed the [contribution guidelines](https://github.com/bitcoindevkit/bdk/blob/master/CONTRIBUTING.md)
* [x] I ran `cargo fmt` and `cargo clippy` before committing
#### New Features:
* [x] I've added tests for the new feature
* [x] I've added docs for the new feature
ACKs for top commit:
LLFourn:
self-ACK: a4f28c079e3bc4500e8329aeed230d8100d38617
evanlinjin:
ACK a4f28c079e3bc4500e8329aeed230d8100d38617
Tree-SHA512: e39fb65b4e69c0a6748d64eab12913dc9cfe5eb8355ab8fb68f60a37c3bb2e1489ddd8f2f138c6470135344f40e3dc671928f65d303fd41fb63f577b30895b60
71fff1613d1c167d180636857063e181f6acedf1 feat(chain): add txout methods to `KeychainTxOutIndex` (志宇)
83e7b7ec402bef27c3f83876f346ec2abd23eff1 docs(chain): improve `KeychainTxOutIndex` docs (志宇)
9294e30943f3f79bb36915d69652d9086323131f docs(wallet): improve docs for unbounded spk iterator methods (志宇)
b74c2e262255a39e9dc904aad469e307bf7d0151 fix(wallet): use efficient peek address logic (志宇)
81aeaba48a980a7e7add76ac2d24e896bedaa1d2 feat(chain): add `SpkIterator::descriptor` method (志宇)
c7b47af72f278ba73abea58db49cff1245428f97 refactor(chain)!: revamp `KeychainTxOutIndex` API (志宇)
705690ee8fddba8517d907183b7ddfeafb633609 feat(chain): make output of `SpkTxOutIndex::unused_spks` cloneable (志宇)
Pull request description:
Closes#1268
### Description
Previously `SpkTxOutIndex` methods can be called from `KeychainTxOutIndex` due to the `DeRef` implementation. However, the internal `SpkTxOut` will also contain lookahead spks resulting in an error-prone API.
`SpkTxOutIndex` methods are now not directly callable from `KeychainTxOutIndex`. Methods of `KeychainTxOutIndex` are renamed for clarity. I.e. methods that return an unbounded spk iter are prefixed with `unbounded`.
In addition to this, I also optimized the peek-address logic of `bdk::Wallet` using the optimized `<SpkIterator as Iterator>::nth` implementation.
### Notes to the reviewers
This is mostly refactoring, but can also be considered a bug-fix (as the API before was very problematic).
### Changelog notice
Changed
* Wallet's peek-address logic is optimized by making use of `<SpkIterator as Iterator>::nth`.
* `KeychainTxOutIndex` API is refactored to better differentiate between methods that return unbounded vs stored spks.
* `KeychainTxOutIndex` no longer directly exposes `SpkTxOutIndex` methods via `DeRef`. This was problematic because `SpkTxOutIndex` also contains lookahead spks which we want to hide.
Added
* `SpkIterator::descriptor` method which gets a reference to the internal descriptor.
### Checklists
#### All Submissions:
* [x] I've signed all my commits
* [x] I followed the [contribution guidelines](https://github.com/bitcoindevkit/bdk/blob/master/CONTRIBUTING.md)
* [x] I ran `cargo fmt` and `cargo clippy` before committing
#### New Features:
~* [ ] I've added tests for the new feature~
* [x] I've added docs for the new feature
#### Bugfixes:
* [x] This pull request breaks the existing API
* [ ] I've added tests to reproduce the issue which are now passing
* [x] I'm linking the issue being fixed by this PR
ACKs for top commit:
danielabrozzoni:
ACK 71fff1613d1c167d180636857063e181f6acedf1
Tree-SHA512: f29c7d2311d0e81c4fe29b8f57c219c24db958194fad5de82bb6d42d562d37fd5d152be7ee03a3f00843be5760569ad29b848250267a548d7d15320fd5292a8f
Previously `SpkTxOutIndex` methods can be called from
`KeychainTxOutIndex` due to the `DeRef` implementation. However, the
internal `SpkTxOut` will also contain lookahead spks resulting in an
error-prone API.
`SpkTxOutIndex` methods are now not directly callable from
`KeychainTxOutIndex`. Methods of `KeychainTxOutIndex` are renamed for
clarity. I.e. methods that return an unbounded spk iter are prefixed
with `unbounded`.
The wallet is currently created without setting any lookahead value for
the keychain. This implicitly makes it a lookahead of 0. As this is a
high-level interface we should avoid footguns and aim for a reasonable
default.
Instead of simply patching it for wallet, we alter `KeychainTxOutIndex`
to have a default lookahead value. Additionally, instead of a
per-keychain lookahead, the constructor asks for a `lookahead` value.
This avoids the footguns of having methods which allows the caller the
decrease the `lookahead` (and therefore panicing). This also simplifies
the API.
Co-authored-by: Antoine Poisot <darosior@protonmail.com>
Co-authored-by: 志宇 <hello@evanlinjin.me>
Coinbase transactions cannot exist in the mempool and be unconfirmed.
`TxGraph::try_get_chain_position` should always return `None` for coinbase
transactions not anchored in best chain.
For `IndexedTxGraph`:
- Remove `InsertTxItem` type (this is too complex).
- `batch_insert_relevant` now uses a simple tuple `(&tx, anchors)`.
- `batch_insert` is now also removed, as the same functionality can be
done elsewhere.
- Add internal helper method `index_tx_graph_changeset` so we don't need
to create a seprate `TxGraph` update in each method.
- `batch_insert_<relevant>_unconfirmed` no longer takes in an option of
last_seen.
- `batch_insert_unconfirmed` no longer takes a reference of a
transaction (since we apply all transactions anyway, so there is no
need to clone).
For `TxGraph`:
- Add `batch_insert_unconfirmed` method.
..into account
When you pass a non-wildcard descriptor in `new_with_range`, we make
sure that the range length is at most 1; if that's not the case, we
shorten it.
We would previously use `new_with_range` without this check and with a
non-wildcard descriptor in `spks_of_all_keychains`, this meant creating
a spkiterator that would go on producing the same spks over and over
again, causing some issues with syncing on electrum/esplora.
To reproduce the bug, run in `example-crates/example_electrum`:
```
cargo run "sh(wsh(or_d(c:pk_k(cPGudvRLDSgeV4hH9NUofLvYxYBSRjju3cpiXmBg9K8G9k1ikCMp),c:pk_k(cSBSBHRrzqSXFmrBhLkZMzQB9q4P9MnAq92v8d9a5UveBc9sLX32))))#zp9pcfs9" scan
```
b206a985cffaa9b614841219371faa53ba23dfc3 fix: Even more refactoring to code and documentation (志宇)
bea8e5aff4f1e4d61db3970a6efaec86e686dbc3 fix: `TxGraph::missing_blocks` logic (志宇)
db15e03bdce78c6321f906f390b10b3d9e7501b2 fix: improve more docs and more refactoring (志宇)
95312d4d05618b4c464acc0fdff49fb17405ec88 fix: docs and some minor refactoring (志宇)
8bf7a997f70fdffd072fd37e12c385e731728c5a Refactor `debug_assertions` checks for `LocalChain` (志宇)
315e7e0b4b373d7175f21a48ff6480b6e919a2c6 fix: rm duplicate `bdk_tmp_plan` module (志宇)
af705da1a846214f104df8886201a23cfa4b6b74 Add exclusion of example cli `*.db` files in `.gitignore` (志宇)
eabeb6ccb169b32f7b7541c9dc6481693bdeeb8a Implement linked-list `LocalChain` and update chain-src crates/examples (志宇)
Pull request description:
Fixes#997
Replaces #1002
### Description
This PR changes the `LocalChain` implementation to have blocks stored as a linked-list. This allows the data-src thread to hold a shared ref to a single checkpoint and have access to the whole history of checkpoints without cloning or keeping a lock on `LocalChain`.
The APIs of `bdk::Wallet`, `esplora` and `electrum` are also updated to reflect these changes. Note that the `esplora` crate is rewritten to anchor txs in the confirmation block (using the esplora API's tx status block_hash). This guarantees 100% consistency between anchor blocks and their transactions (instead of anchoring txs to the latest tip). `ExploraExt` now has separate methods for updating the `TxGraph` and `LocalChain`.
A new method `TxGraph::missing_blocks` is introduced for finding "floating anchors" of a `TxGraph` update (given a chain).
Additional changes:
* `test_local_chain.rs` is refactored to make test cases easier to write. Additional tests are also added.
* Examples are updated.
* Exclude example-cli `*.db` files in `.gitignore`.
* Rm duplicate `bdk_tmp_plan` module.
### Notes to the reviewers
This is the smallest possible division of #1002 without resulting in PRs that do not compile. Since we have changed the API of `LocalChain`, we also need to change `esplora`, `electrum` crates and examples alongside `bdk::Wallet`.
### Changelog notice
* Implement linked-list `LocalChain`. This allows the data-src thread to hold a shared ref to a single checkpoint and have access to the whole history of checkpoints without cloning or keeping a lock on `LocalChain`.
* Rewrote `esplora` chain-src crate to anchor txs to their confirmation blocks (using esplora API's tx-status `block_hash`).
### Checklists
#### All Submissions:
* [x] I've signed all my commits
* [x] I followed the [contribution guidelines](https://github.com/bitcoindevkit/bdk/blob/master/CONTRIBUTING.md)
* [x] I ran `cargo fmt` and `cargo clippy` before committing
#### New Features:
* [x] I've added tests for the new feature
* [x] I've added docs for the new feature
ACKs for top commit:
LLFourn:
ACK b206a985cffaa9b614841219371faa53ba23dfc3
Tree-SHA512: a513eecb4f1aae6a5c06a69854e4492961424312a75a42d74377d363b364e3d52415bc81b4aa3fbc3f369ded19bddd07ab895130ebba288e8a43e9d6186e9fcc
Thank you @vladimirfomene for these suggestions.
* Rename `LocalUpdate::keychain` to `LocalUpdate::last_active_indices`.
* Change docs for `CheckPoint` to be more descriptive.
* Fix incorrect logic in `update_local_chain` for `EsploraExt` and
`EsploraAsyncExt`.
Shout out to @LLFourn for these suggestions.
* Improve/fix `LocalChain` documentation
* Refactor `TxGraph::missing_blocks` to make it more explicit that
`last_block` has state.
* `update_local_chain` method of `EsploraExt` and `EsploraAsyncExt` now
returns a `local_chain::Update` instead of just a `CheckPoint`.
This commit changes the `LocalChain` implementation to have blocks
stored as a linked-list. This allows the data-src thread to hold a
shared ref to a single checkpoint and have access to the whole history
of checkpoints without cloning or keeping a lock on `LocalChain`.
The APIs of `bdk::Wallet`, `esplora` and `electrum` are also updated to
reflect these changes. Note that the `esplora` crate is rewritten to
anchor txs in the confirmation block (using the esplora API's tx status
block_hash). This guarantees 100% consistency between anchor blocks and
their transactions (instead of anchoring txs to the latest tip).
`ExploraExt` now has separate methods for updating the `TxGraph` and
`LocalChain`.
A new method `TxGraph::missing_blocks` is introduced for finding
"floating anchors" of a `TxGraph` update (given a chain).
Additional changes:
* `test_local_chain.rs` is refactored to make test cases easier to
write. Additional tests are also added.
* Examples are updated.
* Fix `tempfile` dev dependency of `bdk_file_store` to work with MSRV
Co-authored-by: LLFourn <lloyd.fourn@gmail.com>
Other changes:
* The `async-https` feature of `bdk_esplora` is no longer default.
* Rename `ObservedAs` to `ChainPosition`.
* Set temporary MSRV to 1.60.0 to compile all workspace members will all
features.
Instead of relying on a `OwnedIndexer` trait to filter for relevant
txouts, we move the listing and balance methods from `IndexedTxGraph` to
`TxGraph` and add an additional input (list of relevant outpoints) to
these methods.
This provides a simpler implementation and a more flexible API.
* Introduce `LocalChain::inner` method to get the inner map of block
height to hash.
* Replace `LocalChain::get_block` (which outputted `BlockId`, thus able
to return invalid representation) with `get_blockhash` that just
returns a `BlockHash`.
* Remove `TODO` comments that should be github tickets.
If `BlockId` implements `Anchor`, the meaning is ambiguous. We cannot
tell whether it means the tx is anchors at the block, or whether it also
means the tx is confirmed at that block.
Instead, `ConfirmationHeightAnchor` and `ConfirmationTimeAnchor` structs
are introduced as non-ambiguous `Anchor` implementations.
Additionally, `TxGraph::relevant_heights` is removed because it is also
ambiguous. What heights are deemed relevant? A simpler and more flexible
method `TxGraph::all_anchors` is introduced instead.
`TxGraph::try_get_chain_position` used to always exclude unconfirmed
transactions with last_seen value of 0. However, what is the point of
including a transaction in the graph if it cannot be part of the chain
history? Additionally, maybe sometimes we don't wish to use the
last_seen field at all.
The new behavior will consider unconfirmed transactions with last_seen
of 0.
* `Additions` now implements `Append` and uses `Append` to implement
`append()`.
* `append()` logic enforces that `last_seen` values should only
increase.
* Test written for `append()` with `last_seen` behaviour.
Instead of forcing all transactions inserted to use the same anchors, we
change the API to have unique anchors per transaction.
This allows for more flexibility in general. For example, use `Anchor`
implementations that contain the position in a block of a transaction.
The `insert_relevant_txs` test has also been changed to used
`KeychainTxOutIndex` so that index additions can be checked
(`SpkTxOutIndex` has no additions).
Additionally, generic bounds of some `IndexedTxGraph` list methods have
been fixed.
