Also fixup `test_list_owned_txouts` to check that the right
outputs, utxos, and balance are returned at different local
chain heights.
This fixes an issue where unbroadcast and otherwise non-canonical
transactions were returned from methods `list_chain_txs` and
`Wallet::transactions` because every tx inserted had a last_seen
of 0 making it appear unconfirmed.
Note this commit changes the way `Balance` is represented due to
new logic in `try_get_chain_position` that no longer considers
txs with non-canonical anchors. Before this change, a tx anchored
to a block that is reorged out had a permanent effect on the
pending balance, and now only txs with a last_seen time or an
anchor confirmed in the best chain will return a `ChainPosition`.
The previous commit b9c5b9d08b040faf6c6b2d9b3745918031555b72 added
IndexSpk. This goes further and adds `Indexed` and `KeychainIndexed`
type alises (IndexSpk is Indexed<ScriptBuf>) and attempts to standardize
the structure of return types more generally.
The underlying SpkTxOutIndex should not use DescriptorIds to index
because this loses the ordering relationship of the spks so queries on
subranges of keychains work.
Along with that we enforce that there is a strict 1-to-1 relationship
between descriptors and keychains. Violating this leads to an error in
insert_descriptor now.
In general I try to make the translation layer between the SpkTxOutIndex
and the KeychainTxOutIndex thinner. Ergonomics of this will be improved
in next commit.
The test from the previous commit passes.
deps(chain): bump `bitcoin` to `0.32.0`, miniscript to `12.0.0`
fix(chain): use `minimal_non_dust()` instead of `dust_value()`
fix(chain): use `compute_txid()` instead of `txid`
deps(testenv): bump `electrsd` to `0.28.0`
deps(electrum): bump `electrum-client` to `0.20.0`
fix(electrum): use `compute_txid()` instead of `txid`
deps(esplora): bump `esplora-client` to `0.8.0`
deps(bitcoind_rpc): bump `bitcoin` to `0.32.0`, `bitcoincore-rpc` to
`0.19.0`
fix(bitcoind_rpc): use `compute_txid()` instead of `txid`
fix(nursery/tmp_plan): use proper `sighash` errors, and fix the expected
`Signature` fields
fix(sqlite): use `compute_txid()` instead of `txid`
deps(hwi): bump `hwi` to `0.9.0`
deps(wallet): bump `bitcoin` to `0.32.0`, miniscript to `12.0.0`
fix(wallet): use `compute_txid()` and `minimal_non_dust()`
- update to use `compute_txid()` instead of deprecated `txid()`
- update to use `minimal_non_dust()` instead of `dust_value()`
- remove unused `bitcoin::hex::FromHex`.
fix(wallet): uses `.into` conversion on `Network` for `NetworkKind`
- uses `.into()` when appropriate, otherwise use the explicit
`NetworkKind`, and it's `.is_mainnet()` method.
fix(wallet): add P2wpkh, Taproot, InputsIndex errors to `SignerError`
fix(wallet): fields on taproot, and ecdsa `Signature` structure
fix(wallet/wallet): convert `Weight` to `usize` for now
- converts the `bitcoin-units::Weight` type to `usize` with help of
`to_wu()` method.
- it should be updated/refactored in the future to handle the `Weight`
type throughout the code instead of current `usize`, only converting
it for now.
- allows the usage of deprecated `is_provably_unspendable()`, needs
further discussion if suggested `is_op_return` is suitable.
- update the expect field to `signature`, as it was renamed from `sig`.
fix(wallet/wallet): use `is_op_return` instead of
`is_provably_unspendable`
fix(wallet/wallet): use `relative::Locktime` instead of `Sequence`
fix(wallet/descriptor): use `ParsePublicKeyError`
fix(wallet/descriptor): use `.into()` to convert from `AbsLockTime` and
`RelLockTime` to `absolute::LockTime` and `relative::LockTime`
fix(wallet/wallet): use `Message::from_digest()` instead of relying on
deprecated `ThirtyTwoByteHash` trait.
fix(wallet/descriptor+wallet): expect `Threshold` type, and handle it
internally
fix(wallet/wallet): remove `0x` prefix from expected `TxId` display
fix(examples): use `compute_txid()` instead of `txid`
fix(ci): remove usage of `bitcoin/no-std` feature
- remove comment: `# The `no-std` feature it's implied when the `std` feature is disabled.`
- update to use `bitcoin::Amount` on `CreateTxError::FeeTooLow` variant.
- update to use `bitcoin::Amount` on `Wallet::calculate_fee()`.
- update to use `bitcoin::Amount` on `FeePolicy::fee_absolute()`.
- update to use `bitcoin::SignedAmount` on
`CalculateFeeError::NegativeFee` variant.
- update to use `bitcoin::Amount` on `TxGraph::calculate_fee()`.
- update to use `bitcoin::Amount` on `PsbUtils::fee_amount()`
- The KeychainTxOutIndex's internal SpkIterator now uses DescriptorId
instead of K. The DescriptorId -> K translation is made at the
KeychainTxOutIndex level.
- The keychain::Changeset is now a struct, which includes a map for last
revealed indexes, and one for newly added keychains and their
descriptor.
API changes in bdk:
- Wallet::keychains returns a `impl Iterator` instead of `BTreeMap`
- Wallet::load doesn't take descriptors anymore, since they're stored in
the db
- Wallet::new_or_load checks if the loaded descriptor from db is the
same as the provided one
API changes in bdk_chain:
- `ChangeSet` is now a struct, which includes a map for last revealed
indexes, and one for keychains and descriptors.
- `KeychainTxOutIndex::inner` returns a `SpkIterator<(DescriptorId, u32)>`
- `KeychainTxOutIndex::outpoints` returns a `impl Iterator` instead of `&BTreeSet`
- `KeychainTxOutIndex::keychains` returns a `impl Iterator` instead of
`&BTreeMap`
- `KeychainTxOutIndex::txouts` doesn't return a ExactSizeIterator
anymore
- `KeychainTxOutIndex::unbounded_spk_iter` returns an `Option`
- `KeychainTxOutIndex::next_index` returns an `Option`
- `KeychainTxOutIndex::last_revealed_indices` returns a `BTreeMap`
instead of `&BTreeMap`
- `KeychainTxOutIndex::reveal_to_target` returns an `Option`
- `KeychainTxOutIndex::reveal_next_spk` returns an `Option`
- `KeychainTxOutIndex::next_unused_spk` returns an `Option`
- `KeychainTxOutIndex::add_keychain` has been renamed to
`KeychainTxOutIndex::insert_descriptor`, and now it returns a
ChangeSet
We plan to record `Descriptor` additions into persistence. Hence, we
need to add `Descriptor`s to the changeset. This depends on
`miniscript`. Moving this into `txout_index.rs` makes sense as this is
consistent with all the other files. The only reason why this wasn't
this way before, is because the changeset didn't need miniscript.
Co-Authored-By: Daniela Brozzoni <danielabrozzoni@protonmail.com>
- update all fields `immature`, ` trusted_pending`, `unstrusted_pending`
and `confirmed` to use the `bitcoin::Amount` instead of `u64`
- update all `impl Balance` methods to use `bitcoin::Amount`
- update all tests that relies on `keychain::Balance`
96a9aa6e63474dbd93a2ef969eef5b07c79e6491 feat(chain): refactor `merge_chains` (志宇)
2f22987c9e924800f8682b2dcbdde60fd26b069a chore(chain): fix comment (志宇)
daf588f016ec3118c875db8ed6b55fa03683f0f6 feat(chain): optimize `merge_chains` (志宇)
77d35954c1f3a18f767267e9097f63ca11c709ec feat(chain)!: rm `local_chain::Update` (志宇)
1269b0610efb7bd86d92a909800f9330568c797a test(chain): fix incorrect test case (志宇)
72fe65b65f297ebb7160eee6859c46e29c2d9528 feat(esplora)!: simplify chain update logic (志宇)
eded1a7ea0c6a4b9664826df4f77b714cbad0bcc feat(chain): introduce `CheckPoint::insert` (志宇)
519cd75d23fbb72321b0b189dca12afbfd78c0c7 test(esplora): move esplora tests into src files (志宇)
a6e613e6b978b995abf6c92a16df0300b113aa2c test(esplora): add `test_finalize_chain_update` (志宇)
494d253493f1bc914adba16a28ccf1bc0a0f4ec8 feat(testenv): add `genesis_hash` method (志宇)
886d72e3d541d088320bbdad6804057f32aca684 chore(chain)!: rm `missing_heights` and `missing_heights_from` methods (志宇)
bd62aa0fe199d676710c9909617198d62f4897c0 feat(esplora)!: remove `EsploraExt::update_local_chain` (志宇)
1e997939837e9c1f0c087d6d28ac12e373c8c05f feat(testenv): add `make_checkpoint_tip` (志宇)
Pull request description:
Fixes#1354
### Description
Built on top of both #1369 and #1373, we simplify the `EsploraExt` API by removing the `update_local_chain` method and having `full_scan` and `sync` update the local chain in the same call. The `full_scan` and `sync` methods now takes in an additional input (`local_tip`) which provides us with the view of the `LocalChain` before the update. These methods now return structs `FullScanUpdate` and `SyncUpdate`.
The examples are updated to use this new API. `TxGraph::missing_heights` and `tx_graph::ChangeSet::missing_heights_from` are no longer needed, therefore they are removed.
Additionally, we used this opportunity to simplify the logic which updates `LocalChain`. We got rid of the `local_chain::Update` struct (which contained the update `CheckPoint` tip and a `bool` which signaled whether we want to introduce blocks below point of agreement). It turns out we can use something like `CheckPoint::insert` so the chain source can craft an update based on the old tip. This way, we can make better use of `merge_chains`' optimization that compares the `Arc` pointers of the local and update chain (before we were crafting the update chain NOT based on top of the previous local chain). With this, we no longer need the `Update::introduce_older_block` field since the logic will naturally break when we reach a matching `Arc` pointer.
### Notes to the reviewers
* Obtaining the `LocalChain`'s update now happens within `EsploraExt::full_scan` and `EsploraExt::sync`. Creating the `LocalChain` update is now split into two methods (`fetch_latest_blocks` and `chain_update`) that are called before and after fetching transactions and anchors.
* We need to duplicate code for `bdk_esplora`. One for blocking and one for async.
### Changelog notice
* Changed `EsploraExt` API so that sync only requires one round of fetching data. The `local_chain_update` method is removed and the `local_tip` parameter is added to the `full_scan` and `sync` methods.
* Removed `TxGraph::missing_heights` and `tx_graph::ChangeSet::missing_heights_from` methods.
* Introduced `CheckPoint::insert` which allows convenient checkpoint-insertion. This is intended for use by chain-sources when crafting an update.
* Refactored `merge_chains` to also return the resultant `CheckPoint` tip.
* Optimized the update `LocalChain` logic - use the update `CheckPoint` as the new `CheckPoint` tip when possible.
### Checklists
#### All Submissions:
* [x] I've signed all my commits
* [x] I followed the [contribution guidelines](https://github.com/bitcoindevkit/bdk/blob/master/CONTRIBUTING.md)
* [x] I ran `cargo fmt` and `cargo clippy` before committing
#### New Features:
* [x] I've added tests for the new feature
* [x] I've added docs for the new feature
ACKs for top commit:
LLFourn:
ACK 96a9aa6e63474dbd93a2ef969eef5b07c79e6491
Tree-SHA512: 3d4f2eab08a1fe94eb578c594126e99679f72e231680b2edd4bfb018ba1d998ca123b07acb2d19c644d5887fc36b8e42badba91cd09853df421ded04de45bf69
The intention is to remove the `Update::introduce_older_blocks`
parameter and update the local chain directly with `CheckPoint`.
This simplifies the API and there is a way to do this efficiently.
Ensure that `CheckPoint::range` returns expected values by comparing
against values returned from a primitive approach.
I think it is a good idea to start writing proptests for this crate.
a2a64ffb6e92baf46a912f36294f3f4f521a528a fix(electrum)!: Remove `seen_at` param from `into_tx_graph` (valued mammal)
37fca35ddede6cbc9d9428a2722eff82a405b1b2 feat(tx_graph): Add method update_last_seen_unconfirmed (valued mammal)
Pull request description:
A method `update_last_seen_unconfirmed` is added for `TxGraph` that allows inserting a `seen_at` time for all unanchored transactions.
fixes#1336
### Changelog notice
Changed:
- Methods `into_tx_graph` and `into_confirmation_time_tx_graph`for `RelevantTxids` are changed to no longer accept a `seen_at` parameter.
Added:
- Added method `update_last_seen_unconfirmed` for `TxGraph`.
### Checklists
#### All Submissions:
* [x] I've signed all my commits
* [x] I followed the [contribution guidelines](https://github.com/bitcoindevkit/bdk/blob/master/CONTRIBUTING.md)
* [x] I ran `cargo fmt` and `cargo clippy` before committing
ACKs for top commit:
evanlinjin:
ACK a2a64ffb6e92baf46a912f36294f3f4f521a528a
Tree-SHA512: 9011e63314b0e3ffcd50dbf5024f82b365bab1cc834c0455d7410b682338339ed5284caa721ffc267c65fa26d35ff6ee86cde6052e82a6a79768547fbb7a45eb
Upgrade:
- bitcoin to v0.31.0
- miniscript to v11.0.0
Note: The bitcoin upgrade includes improvements to the
`Transaction::weight()` function, it appears those guys did good, we
no longer need to add the 2 additional weight units "just in case".
That accepts a `u64` as param representing the latest timestamp
and internally calls `insert_seen_at` for all transactions in
graph that aren't yet anchored in a confirmed block.
These methods allow us to query for checkpoints contained within the
linked list by height and height range. This is useful to determine
checkpoints to fetch for chain sources without having to refer back to
the `LocalChain`.
Currently this is not implemented efficiently, but in the future, we
will change `CheckPoint` to use a skip list structure.
Wrapping transactions as `Arc<Transaction>` allows us to share
transactions cheaply between the chain-source and receiving structures.
Therefore the chain-source can keep already-fetched transactions (save
bandwidth) and have a shared pointer to the transactions (save memory).
This is better than the current way we do things, which is to refer back
to the receiving structures mid-sync.
Documentation for `TxGraph` is also updated.
b290b2950221578d73c3194a1accfd080f83631a test(chain): change test case comments to docstring (志宇)
c151d8fd230fb87f27aadb965157f94f01f021a2 fix(chain): `KeychainTxOutIndex::lookahead_to_target` (志宇)
Pull request description:
### Description
This method was not used (so it was untested) and it was not working. This fixes it.
The old implementation used `.next_store_index` which returned the keychain's last index stored in `.inner` (which include lookahead spks). This is WRONG because `.replenish_lookahead` needs the difference from last revealed.
### Changelog notice
Fix `KeychainTxOutIndex::lookahead_to_target`
### Checklists
#### All Submissions:
* [x] I've signed all my commits
* [x] I followed the [contribution guidelines](https://github.com/bitcoindevkit/bdk/blob/master/CONTRIBUTING.md)
* [x] I ran `cargo fmt` and `cargo clippy` before committing
#### Bugfixes:
* [x] I've added tests to reproduce the issue which are now passing
ACKs for top commit:
notmandatory:
ACK b290b2950221578d73c3194a1accfd080f83631a
Tree-SHA512: af50c6af18b6b57494cfa37f89b0236674fa331091d791e858f67b7d0b3a1e4e11e7422029bd6a2dc1c795914cdf6d592a14b42a62ca7c7c475ba6ed37182539
5489f905a434ecc06867603c7c421e3e50d993ca feat(chain): add `map_anchors` for `TxGraph` and `ChangeSet` (Antonio Yang)
022d5a21cff6c46fb869f8fd538b4026e531ee57 test(chain) use `Anchor` generic on `init_graph` (Antonio Yang)
Pull request description:
### Description
Fix#1295
### Checklists
#### All Submissions:
* [X] I've signed all my commits
* [X] I followed the [contribution guidelines](https://github.com/bitcoindevkit/bdk/blob/master/CONTRIBUTING.md)
* [X] I ran `cargo fmt` and `cargo clippy` before committing
#### New Features:
* [ ] I've added tests for the new feature
* [ ] I've added docs for the new feature
#### Bugfixes:
* [ ] This pull request breaks the existing API
* [ ] I've added tests to reproduce the issue which are now passing
* [ ] I'm linking the issue being fixed by this PR
ACKs for top commit:
evanlinjin:
ACK 5489f905a434ecc06867603c7c421e3e50d993ca
LLFourn:
ACK 5489f905a434ecc06867603c7c421e3e50d993ca
Tree-SHA512: c8327f2e7035a46208eb32c6da1f9f0bc3e8625168450c5b0b39f695268e42b0b9053b6eb97805b116328195d77af7ca9edb1f12206c50513fbe295dded542e7
a4f28c079e3bc4500e8329aeed230d8100d38617 chore: improve LocalChain::apply_header_connected_to doc (LLFourn)
8ec65f0b8ef7d452a0bdba6760c46bd8511c91ff feat(example): add RPC wallet example (Vladimir Fomene)
a7d01dc39acb3ff7f4bc8e237c7862d019a91cd1 feat(chain)!: make `IndexedTxGraph::apply_block_relevant` more efficient (志宇)
e0512acf9476fa3fa7da9cc28a222432f974aad5 feat(bitcoind_rpc)!: emissions include checkpoint and connected_to data (志宇)
8f2d4d9d400e398b54cf1bad36057f3f8f4a69a4 test(chain): `LocalChain` test for update that is shorter than original (志宇)
9467cad55d68354fe037d57bceeee57c2032bd51 feat(wallet): introduce block-by-block api (Vladimir Fomene)
d3e5095df10d21ff5cb7f8ce4dab7922f3cb6e35 feat(chain): add `apply_header..` methods to `LocalChain` (志宇)
2b61a122ff40253f2e67c0ae6824ce881c9f12a1 feat(chain): add `CheckPoint::from_block_ids` convenience method (志宇)
Pull request description:
### Description
Introduce block-by-block API for `bdk::Wallet`. A `wallet_rpc` example is added to demonstrate syncing `bdk::Wallet` with the `bdk_bitcoind_rpc` chain-source crate.
The API of `bdk_bitcoind_rpc::Emitter` is changed so the receiver knows how to connect to the block emitted.
### Notes to the reviewers
### Changelog notice
Added
* `Wallet` methods to apply full blocks (`apply_block` and `apply_block_connected_to`) and a method to apply a batch of unconfirmed transactions (`apply_unconfirmed_txs`).
* `CheckPoint::from_block_ids` convenience method.
* `LocalChain` methods to apply a block header (`apply_header` and `apply_header_connected_to`).
* Test to show that `LocalChain` can apply updates that are shorter than original. This will happen during reorgs if we sync wallet with `bdk_bitcoind_rpc::Emitter`.
Fixed
* `InsertTxError` now implements `std::error::Error`.
#### All Submissions:
* [x] I've signed all my commits
* [x] I followed the [contribution guidelines](https://github.com/bitcoindevkit/bdk/blob/master/CONTRIBUTING.md)
* [x] I ran `cargo fmt` and `cargo clippy` before committing
#### New Features:
* [x] I've added tests for the new feature
* [x] I've added docs for the new feature
ACKs for top commit:
LLFourn:
self-ACK: a4f28c079e3bc4500e8329aeed230d8100d38617
evanlinjin:
ACK a4f28c079e3bc4500e8329aeed230d8100d38617
Tree-SHA512: e39fb65b4e69c0a6748d64eab12913dc9cfe5eb8355ab8fb68f60a37c3bb2e1489ddd8f2f138c6470135344f40e3dc671928f65d303fd41fb63f577b30895b60
71fff1613d1c167d180636857063e181f6acedf1 feat(chain): add txout methods to `KeychainTxOutIndex` (志宇)
83e7b7ec402bef27c3f83876f346ec2abd23eff1 docs(chain): improve `KeychainTxOutIndex` docs (志宇)
9294e30943f3f79bb36915d69652d9086323131f docs(wallet): improve docs for unbounded spk iterator methods (志宇)
b74c2e262255a39e9dc904aad469e307bf7d0151 fix(wallet): use efficient peek address logic (志宇)
81aeaba48a980a7e7add76ac2d24e896bedaa1d2 feat(chain): add `SpkIterator::descriptor` method (志宇)
c7b47af72f278ba73abea58db49cff1245428f97 refactor(chain)!: revamp `KeychainTxOutIndex` API (志宇)
705690ee8fddba8517d907183b7ddfeafb633609 feat(chain): make output of `SpkTxOutIndex::unused_spks` cloneable (志宇)
Pull request description:
Closes#1268
### Description
Previously `SpkTxOutIndex` methods can be called from `KeychainTxOutIndex` due to the `DeRef` implementation. However, the internal `SpkTxOut` will also contain lookahead spks resulting in an error-prone API.
`SpkTxOutIndex` methods are now not directly callable from `KeychainTxOutIndex`. Methods of `KeychainTxOutIndex` are renamed for clarity. I.e. methods that return an unbounded spk iter are prefixed with `unbounded`.
In addition to this, I also optimized the peek-address logic of `bdk::Wallet` using the optimized `<SpkIterator as Iterator>::nth` implementation.
### Notes to the reviewers
This is mostly refactoring, but can also be considered a bug-fix (as the API before was very problematic).
### Changelog notice
Changed
* Wallet's peek-address logic is optimized by making use of `<SpkIterator as Iterator>::nth`.
* `KeychainTxOutIndex` API is refactored to better differentiate between methods that return unbounded vs stored spks.
* `KeychainTxOutIndex` no longer directly exposes `SpkTxOutIndex` methods via `DeRef`. This was problematic because `SpkTxOutIndex` also contains lookahead spks which we want to hide.
Added
* `SpkIterator::descriptor` method which gets a reference to the internal descriptor.
### Checklists
#### All Submissions:
* [x] I've signed all my commits
* [x] I followed the [contribution guidelines](https://github.com/bitcoindevkit/bdk/blob/master/CONTRIBUTING.md)
* [x] I ran `cargo fmt` and `cargo clippy` before committing
#### New Features:
~* [ ] I've added tests for the new feature~
* [x] I've added docs for the new feature
#### Bugfixes:
* [x] This pull request breaks the existing API
* [ ] I've added tests to reproduce the issue which are now passing
* [x] I'm linking the issue being fixed by this PR
ACKs for top commit:
danielabrozzoni:
ACK 71fff1613d1c167d180636857063e181f6acedf1
Tree-SHA512: f29c7d2311d0e81c4fe29b8f57c219c24db958194fad5de82bb6d42d562d37fd5d152be7ee03a3f00843be5760569ad29b848250267a548d7d15320fd5292a8f
Previously `SpkTxOutIndex` methods can be called from
`KeychainTxOutIndex` due to the `DeRef` implementation. However, the
internal `SpkTxOut` will also contain lookahead spks resulting in an
error-prone API.
`SpkTxOutIndex` methods are now not directly callable from
`KeychainTxOutIndex`. Methods of `KeychainTxOutIndex` are renamed for
clarity. I.e. methods that return an unbounded spk iter are prefixed
with `unbounded`.
The wallet is currently created without setting any lookahead value for
the keychain. This implicitly makes it a lookahead of 0. As this is a
high-level interface we should avoid footguns and aim for a reasonable
default.
Instead of simply patching it for wallet, we alter `KeychainTxOutIndex`
to have a default lookahead value. Additionally, instead of a
per-keychain lookahead, the constructor asks for a `lookahead` value.
This avoids the footguns of having methods which allows the caller the
decrease the `lookahead` (and therefore panicing). This also simplifies
the API.
Co-authored-by: Antoine Poisot <darosior@protonmail.com>
Co-authored-by: 志宇 <hello@evanlinjin.me>
