96a9aa6e63474dbd93a2ef969eef5b07c79e6491 feat(chain): refactor `merge_chains` (志宇)
2f22987c9e924800f8682b2dcbdde60fd26b069a chore(chain): fix comment (志宇)
daf588f016ec3118c875db8ed6b55fa03683f0f6 feat(chain): optimize `merge_chains` (志宇)
77d35954c1f3a18f767267e9097f63ca11c709ec feat(chain)!: rm `local_chain::Update` (志宇)
1269b0610efb7bd86d92a909800f9330568c797a test(chain): fix incorrect test case (志宇)
72fe65b65f297ebb7160eee6859c46e29c2d9528 feat(esplora)!: simplify chain update logic (志宇)
eded1a7ea0c6a4b9664826df4f77b714cbad0bcc feat(chain): introduce `CheckPoint::insert` (志宇)
519cd75d23fbb72321b0b189dca12afbfd78c0c7 test(esplora): move esplora tests into src files (志宇)
a6e613e6b978b995abf6c92a16df0300b113aa2c test(esplora): add `test_finalize_chain_update` (志宇)
494d253493f1bc914adba16a28ccf1bc0a0f4ec8 feat(testenv): add `genesis_hash` method (志宇)
886d72e3d541d088320bbdad6804057f32aca684 chore(chain)!: rm `missing_heights` and `missing_heights_from` methods (志宇)
bd62aa0fe199d676710c9909617198d62f4897c0 feat(esplora)!: remove `EsploraExt::update_local_chain` (志宇)
1e997939837e9c1f0c087d6d28ac12e373c8c05f feat(testenv): add `make_checkpoint_tip` (志宇)
Pull request description:
Fixes#1354
### Description
Built on top of both #1369 and #1373, we simplify the `EsploraExt` API by removing the `update_local_chain` method and having `full_scan` and `sync` update the local chain in the same call. The `full_scan` and `sync` methods now takes in an additional input (`local_tip`) which provides us with the view of the `LocalChain` before the update. These methods now return structs `FullScanUpdate` and `SyncUpdate`.
The examples are updated to use this new API. `TxGraph::missing_heights` and `tx_graph::ChangeSet::missing_heights_from` are no longer needed, therefore they are removed.
Additionally, we used this opportunity to simplify the logic which updates `LocalChain`. We got rid of the `local_chain::Update` struct (which contained the update `CheckPoint` tip and a `bool` which signaled whether we want to introduce blocks below point of agreement). It turns out we can use something like `CheckPoint::insert` so the chain source can craft an update based on the old tip. This way, we can make better use of `merge_chains`' optimization that compares the `Arc` pointers of the local and update chain (before we were crafting the update chain NOT based on top of the previous local chain). With this, we no longer need the `Update::introduce_older_block` field since the logic will naturally break when we reach a matching `Arc` pointer.
### Notes to the reviewers
* Obtaining the `LocalChain`'s update now happens within `EsploraExt::full_scan` and `EsploraExt::sync`. Creating the `LocalChain` update is now split into two methods (`fetch_latest_blocks` and `chain_update`) that are called before and after fetching transactions and anchors.
* We need to duplicate code for `bdk_esplora`. One for blocking and one for async.
### Changelog notice
* Changed `EsploraExt` API so that sync only requires one round of fetching data. The `local_chain_update` method is removed and the `local_tip` parameter is added to the `full_scan` and `sync` methods.
* Removed `TxGraph::missing_heights` and `tx_graph::ChangeSet::missing_heights_from` methods.
* Introduced `CheckPoint::insert` which allows convenient checkpoint-insertion. This is intended for use by chain-sources when crafting an update.
* Refactored `merge_chains` to also return the resultant `CheckPoint` tip.
* Optimized the update `LocalChain` logic - use the update `CheckPoint` as the new `CheckPoint` tip when possible.
### Checklists
#### All Submissions:
* [x] I've signed all my commits
* [x] I followed the [contribution guidelines](https://github.com/bitcoindevkit/bdk/blob/master/CONTRIBUTING.md)
* [x] I ran `cargo fmt` and `cargo clippy` before committing
#### New Features:
* [x] I've added tests for the new feature
* [x] I've added docs for the new feature
ACKs for top commit:
LLFourn:
ACK 96a9aa6e63474dbd93a2ef969eef5b07c79e6491
Tree-SHA512: 3d4f2eab08a1fe94eb578c594126e99679f72e231680b2edd4bfb018ba1d998ca123b07acb2d19c644d5887fc36b8e42badba91cd09853df421ded04de45bf69
The intention is to remove the `Update::introduce_older_blocks`
parameter and update the local chain directly with `CheckPoint`.
This simplifies the API and there is a way to do this efficiently.
Ensure that `CheckPoint::range` returns expected values by comparing
against values returned from a primitive approach.
I think it is a good idea to start writing proptests for this crate.
a2a64ffb6e92baf46a912f36294f3f4f521a528a fix(electrum)!: Remove `seen_at` param from `into_tx_graph` (valued mammal)
37fca35ddede6cbc9d9428a2722eff82a405b1b2 feat(tx_graph): Add method update_last_seen_unconfirmed (valued mammal)
Pull request description:
A method `update_last_seen_unconfirmed` is added for `TxGraph` that allows inserting a `seen_at` time for all unanchored transactions.
fixes#1336
### Changelog notice
Changed:
- Methods `into_tx_graph` and `into_confirmation_time_tx_graph`for `RelevantTxids` are changed to no longer accept a `seen_at` parameter.
Added:
- Added method `update_last_seen_unconfirmed` for `TxGraph`.
### Checklists
#### All Submissions:
* [x] I've signed all my commits
* [x] I followed the [contribution guidelines](https://github.com/bitcoindevkit/bdk/blob/master/CONTRIBUTING.md)
* [x] I ran `cargo fmt` and `cargo clippy` before committing
ACKs for top commit:
evanlinjin:
ACK a2a64ffb6e92baf46a912f36294f3f4f521a528a
Tree-SHA512: 9011e63314b0e3ffcd50dbf5024f82b365bab1cc834c0455d7410b682338339ed5284caa721ffc267c65fa26d35ff6ee86cde6052e82a6a79768547fbb7a45eb
Upgrade:
- bitcoin to v0.31.0
- miniscript to v11.0.0
Note: The bitcoin upgrade includes improvements to the
`Transaction::weight()` function, it appears those guys did good, we
no longer need to add the 2 additional weight units "just in case".
That accepts a `u64` as param representing the latest timestamp
and internally calls `insert_seen_at` for all transactions in
graph that aren't yet anchored in a confirmed block.
These methods allow us to query for checkpoints contained within the
linked list by height and height range. This is useful to determine
checkpoints to fetch for chain sources without having to refer back to
the `LocalChain`.
Currently this is not implemented efficiently, but in the future, we
will change `CheckPoint` to use a skip list structure.
Wrapping transactions as `Arc<Transaction>` allows us to share
transactions cheaply between the chain-source and receiving structures.
Therefore the chain-source can keep already-fetched transactions (save
bandwidth) and have a shared pointer to the transactions (save memory).
This is better than the current way we do things, which is to refer back
to the receiving structures mid-sync.
Documentation for `TxGraph` is also updated.
b290b2950221578d73c3194a1accfd080f83631a test(chain): change test case comments to docstring (志宇)
c151d8fd230fb87f27aadb965157f94f01f021a2 fix(chain): `KeychainTxOutIndex::lookahead_to_target` (志宇)
Pull request description:
### Description
This method was not used (so it was untested) and it was not working. This fixes it.
The old implementation used `.next_store_index` which returned the keychain's last index stored in `.inner` (which include lookahead spks). This is WRONG because `.replenish_lookahead` needs the difference from last revealed.
### Changelog notice
Fix `KeychainTxOutIndex::lookahead_to_target`
### Checklists
#### All Submissions:
* [x] I've signed all my commits
* [x] I followed the [contribution guidelines](https://github.com/bitcoindevkit/bdk/blob/master/CONTRIBUTING.md)
* [x] I ran `cargo fmt` and `cargo clippy` before committing
#### Bugfixes:
* [x] I've added tests to reproduce the issue which are now passing
ACKs for top commit:
notmandatory:
ACK b290b2950221578d73c3194a1accfd080f83631a
Tree-SHA512: af50c6af18b6b57494cfa37f89b0236674fa331091d791e858f67b7d0b3a1e4e11e7422029bd6a2dc1c795914cdf6d592a14b42a62ca7c7c475ba6ed37182539
5489f905a434ecc06867603c7c421e3e50d993ca feat(chain): add `map_anchors` for `TxGraph` and `ChangeSet` (Antonio Yang)
022d5a21cff6c46fb869f8fd538b4026e531ee57 test(chain) use `Anchor` generic on `init_graph` (Antonio Yang)
Pull request description:
### Description
Fix#1295
### Checklists
#### All Submissions:
* [X] I've signed all my commits
* [X] I followed the [contribution guidelines](https://github.com/bitcoindevkit/bdk/blob/master/CONTRIBUTING.md)
* [X] I ran `cargo fmt` and `cargo clippy` before committing
#### New Features:
* [ ] I've added tests for the new feature
* [ ] I've added docs for the new feature
#### Bugfixes:
* [ ] This pull request breaks the existing API
* [ ] I've added tests to reproduce the issue which are now passing
* [ ] I'm linking the issue being fixed by this PR
ACKs for top commit:
evanlinjin:
ACK 5489f905a434ecc06867603c7c421e3e50d993ca
LLFourn:
ACK 5489f905a434ecc06867603c7c421e3e50d993ca
Tree-SHA512: c8327f2e7035a46208eb32c6da1f9f0bc3e8625168450c5b0b39f695268e42b0b9053b6eb97805b116328195d77af7ca9edb1f12206c50513fbe295dded542e7
a4f28c079e3bc4500e8329aeed230d8100d38617 chore: improve LocalChain::apply_header_connected_to doc (LLFourn)
8ec65f0b8ef7d452a0bdba6760c46bd8511c91ff feat(example): add RPC wallet example (Vladimir Fomene)
a7d01dc39acb3ff7f4bc8e237c7862d019a91cd1 feat(chain)!: make `IndexedTxGraph::apply_block_relevant` more efficient (志宇)
e0512acf9476fa3fa7da9cc28a222432f974aad5 feat(bitcoind_rpc)!: emissions include checkpoint and connected_to data (志宇)
8f2d4d9d400e398b54cf1bad36057f3f8f4a69a4 test(chain): `LocalChain` test for update that is shorter than original (志宇)
9467cad55d68354fe037d57bceeee57c2032bd51 feat(wallet): introduce block-by-block api (Vladimir Fomene)
d3e5095df10d21ff5cb7f8ce4dab7922f3cb6e35 feat(chain): add `apply_header..` methods to `LocalChain` (志宇)
2b61a122ff40253f2e67c0ae6824ce881c9f12a1 feat(chain): add `CheckPoint::from_block_ids` convenience method (志宇)
Pull request description:
### Description
Introduce block-by-block API for `bdk::Wallet`. A `wallet_rpc` example is added to demonstrate syncing `bdk::Wallet` with the `bdk_bitcoind_rpc` chain-source crate.
The API of `bdk_bitcoind_rpc::Emitter` is changed so the receiver knows how to connect to the block emitted.
### Notes to the reviewers
### Changelog notice
Added
* `Wallet` methods to apply full blocks (`apply_block` and `apply_block_connected_to`) and a method to apply a batch of unconfirmed transactions (`apply_unconfirmed_txs`).
* `CheckPoint::from_block_ids` convenience method.
* `LocalChain` methods to apply a block header (`apply_header` and `apply_header_connected_to`).
* Test to show that `LocalChain` can apply updates that are shorter than original. This will happen during reorgs if we sync wallet with `bdk_bitcoind_rpc::Emitter`.
Fixed
* `InsertTxError` now implements `std::error::Error`.
#### All Submissions:
* [x] I've signed all my commits
* [x] I followed the [contribution guidelines](https://github.com/bitcoindevkit/bdk/blob/master/CONTRIBUTING.md)
* [x] I ran `cargo fmt` and `cargo clippy` before committing
#### New Features:
* [x] I've added tests for the new feature
* [x] I've added docs for the new feature
ACKs for top commit:
LLFourn:
self-ACK: a4f28c079e3bc4500e8329aeed230d8100d38617
evanlinjin:
ACK a4f28c079e3bc4500e8329aeed230d8100d38617
Tree-SHA512: e39fb65b4e69c0a6748d64eab12913dc9cfe5eb8355ab8fb68f60a37c3bb2e1489ddd8f2f138c6470135344f40e3dc671928f65d303fd41fb63f577b30895b60
71fff1613d1c167d180636857063e181f6acedf1 feat(chain): add txout methods to `KeychainTxOutIndex` (志宇)
83e7b7ec402bef27c3f83876f346ec2abd23eff1 docs(chain): improve `KeychainTxOutIndex` docs (志宇)
9294e30943f3f79bb36915d69652d9086323131f docs(wallet): improve docs for unbounded spk iterator methods (志宇)
b74c2e262255a39e9dc904aad469e307bf7d0151 fix(wallet): use efficient peek address logic (志宇)
81aeaba48a980a7e7add76ac2d24e896bedaa1d2 feat(chain): add `SpkIterator::descriptor` method (志宇)
c7b47af72f278ba73abea58db49cff1245428f97 refactor(chain)!: revamp `KeychainTxOutIndex` API (志宇)
705690ee8fddba8517d907183b7ddfeafb633609 feat(chain): make output of `SpkTxOutIndex::unused_spks` cloneable (志宇)
Pull request description:
Closes#1268
### Description
Previously `SpkTxOutIndex` methods can be called from `KeychainTxOutIndex` due to the `DeRef` implementation. However, the internal `SpkTxOut` will also contain lookahead spks resulting in an error-prone API.
`SpkTxOutIndex` methods are now not directly callable from `KeychainTxOutIndex`. Methods of `KeychainTxOutIndex` are renamed for clarity. I.e. methods that return an unbounded spk iter are prefixed with `unbounded`.
In addition to this, I also optimized the peek-address logic of `bdk::Wallet` using the optimized `<SpkIterator as Iterator>::nth` implementation.
### Notes to the reviewers
This is mostly refactoring, but can also be considered a bug-fix (as the API before was very problematic).
### Changelog notice
Changed
* Wallet's peek-address logic is optimized by making use of `<SpkIterator as Iterator>::nth`.
* `KeychainTxOutIndex` API is refactored to better differentiate between methods that return unbounded vs stored spks.
* `KeychainTxOutIndex` no longer directly exposes `SpkTxOutIndex` methods via `DeRef`. This was problematic because `SpkTxOutIndex` also contains lookahead spks which we want to hide.
Added
* `SpkIterator::descriptor` method which gets a reference to the internal descriptor.
### Checklists
#### All Submissions:
* [x] I've signed all my commits
* [x] I followed the [contribution guidelines](https://github.com/bitcoindevkit/bdk/blob/master/CONTRIBUTING.md)
* [x] I ran `cargo fmt` and `cargo clippy` before committing
#### New Features:
~* [ ] I've added tests for the new feature~
* [x] I've added docs for the new feature
#### Bugfixes:
* [x] This pull request breaks the existing API
* [ ] I've added tests to reproduce the issue which are now passing
* [x] I'm linking the issue being fixed by this PR
ACKs for top commit:
danielabrozzoni:
ACK 71fff1613d1c167d180636857063e181f6acedf1
Tree-SHA512: f29c7d2311d0e81c4fe29b8f57c219c24db958194fad5de82bb6d42d562d37fd5d152be7ee03a3f00843be5760569ad29b848250267a548d7d15320fd5292a8f
Previously `SpkTxOutIndex` methods can be called from
`KeychainTxOutIndex` due to the `DeRef` implementation. However, the
internal `SpkTxOut` will also contain lookahead spks resulting in an
error-prone API.
`SpkTxOutIndex` methods are now not directly callable from
`KeychainTxOutIndex`. Methods of `KeychainTxOutIndex` are renamed for
clarity. I.e. methods that return an unbounded spk iter are prefixed
with `unbounded`.
The wallet is currently created without setting any lookahead value for
the keychain. This implicitly makes it a lookahead of 0. As this is a
high-level interface we should avoid footguns and aim for a reasonable
default.
Instead of simply patching it for wallet, we alter `KeychainTxOutIndex`
to have a default lookahead value. Additionally, instead of a
per-keychain lookahead, the constructor asks for a `lookahead` value.
This avoids the footguns of having methods which allows the caller the
decrease the `lookahead` (and therefore panicing). This also simplifies
the API.
Co-authored-by: Antoine Poisot <darosior@protonmail.com>
Co-authored-by: 志宇 <hello@evanlinjin.me>
Coinbase transactions cannot exist in the mempool and be unconfirmed.
`TxGraph::try_get_chain_position` should always return `None` for coinbase
transactions not anchored in best chain.
For `IndexedTxGraph`:
- Remove `InsertTxItem` type (this is too complex).
- `batch_insert_relevant` now uses a simple tuple `(&tx, anchors)`.
- `batch_insert` is now also removed, as the same functionality can be
done elsewhere.
- Add internal helper method `index_tx_graph_changeset` so we don't need
to create a seprate `TxGraph` update in each method.
- `batch_insert_<relevant>_unconfirmed` no longer takes in an option of
last_seen.
- `batch_insert_unconfirmed` no longer takes a reference of a
transaction (since we apply all transactions anyway, so there is no
need to clone).
For `TxGraph`:
- Add `batch_insert_unconfirmed` method.
..into account
When you pass a non-wildcard descriptor in `new_with_range`, we make
sure that the range length is at most 1; if that's not the case, we
shorten it.
We would previously use `new_with_range` without this check and with a
non-wildcard descriptor in `spks_of_all_keychains`, this meant creating
a spkiterator that would go on producing the same spks over and over
again, causing some issues with syncing on electrum/esplora.
To reproduce the bug, run in `example-crates/example_electrum`:
```
cargo run "sh(wsh(or_d(c:pk_k(cPGudvRLDSgeV4hH9NUofLvYxYBSRjju3cpiXmBg9K8G9k1ikCMp),c:pk_k(cSBSBHRrzqSXFmrBhLkZMzQB9q4P9MnAq92v8d9a5UveBc9sLX32))))#zp9pcfs9" scan
```
b206a985cffaa9b614841219371faa53ba23dfc3 fix: Even more refactoring to code and documentation (志宇)
bea8e5aff4f1e4d61db3970a6efaec86e686dbc3 fix: `TxGraph::missing_blocks` logic (志宇)
db15e03bdce78c6321f906f390b10b3d9e7501b2 fix: improve more docs and more refactoring (志宇)
95312d4d05618b4c464acc0fdff49fb17405ec88 fix: docs and some minor refactoring (志宇)
8bf7a997f70fdffd072fd37e12c385e731728c5a Refactor `debug_assertions` checks for `LocalChain` (志宇)
315e7e0b4b373d7175f21a48ff6480b6e919a2c6 fix: rm duplicate `bdk_tmp_plan` module (志宇)
af705da1a846214f104df8886201a23cfa4b6b74 Add exclusion of example cli `*.db` files in `.gitignore` (志宇)
eabeb6ccb169b32f7b7541c9dc6481693bdeeb8a Implement linked-list `LocalChain` and update chain-src crates/examples (志宇)
Pull request description:
Fixes#997
Replaces #1002
### Description
This PR changes the `LocalChain` implementation to have blocks stored as a linked-list. This allows the data-src thread to hold a shared ref to a single checkpoint and have access to the whole history of checkpoints without cloning or keeping a lock on `LocalChain`.
The APIs of `bdk::Wallet`, `esplora` and `electrum` are also updated to reflect these changes. Note that the `esplora` crate is rewritten to anchor txs in the confirmation block (using the esplora API's tx status block_hash). This guarantees 100% consistency between anchor blocks and their transactions (instead of anchoring txs to the latest tip). `ExploraExt` now has separate methods for updating the `TxGraph` and `LocalChain`.
A new method `TxGraph::missing_blocks` is introduced for finding "floating anchors" of a `TxGraph` update (given a chain).
Additional changes:
* `test_local_chain.rs` is refactored to make test cases easier to write. Additional tests are also added.
* Examples are updated.
* Exclude example-cli `*.db` files in `.gitignore`.
* Rm duplicate `bdk_tmp_plan` module.
### Notes to the reviewers
This is the smallest possible division of #1002 without resulting in PRs that do not compile. Since we have changed the API of `LocalChain`, we also need to change `esplora`, `electrum` crates and examples alongside `bdk::Wallet`.
### Changelog notice
* Implement linked-list `LocalChain`. This allows the data-src thread to hold a shared ref to a single checkpoint and have access to the whole history of checkpoints without cloning or keeping a lock on `LocalChain`.
* Rewrote `esplora` chain-src crate to anchor txs to their confirmation blocks (using esplora API's tx-status `block_hash`).
### Checklists
#### All Submissions:
* [x] I've signed all my commits
* [x] I followed the [contribution guidelines](https://github.com/bitcoindevkit/bdk/blob/master/CONTRIBUTING.md)
* [x] I ran `cargo fmt` and `cargo clippy` before committing
#### New Features:
* [x] I've added tests for the new feature
* [x] I've added docs for the new feature
ACKs for top commit:
LLFourn:
ACK b206a985cffaa9b614841219371faa53ba23dfc3
Tree-SHA512: a513eecb4f1aae6a5c06a69854e4492961424312a75a42d74377d363b364e3d52415bc81b4aa3fbc3f369ded19bddd07ab895130ebba288e8a43e9d6186e9fcc
