Rework sqlite: Instead of only supported one schema (defined in
`bdk_sqlite`), we have a schema per changeset type for more flexiblity.
* rm `bdk_sqlite` crate (as we don't need `bdk_sqlite::Store` anymore).
* add `sqlite` feature on `bdk_chain` which adds methods on each
changeset type for initializing tables, loading the changeset and
writing.
Rework changesets: Some callers may want to use `KeychainTxOutIndex`
where `K` may change per descriptor on every run. So we only want to
persist the last revealed indices by `DescriptorId` (which uniquely-ish
identifies the descriptor).
* rm `keychain_added` field from `keychain_txout`'s changeset.
* Add `keychain_added` to `CombinedChangeSet` (which is renamed to
`WalletChangeSet`).
Rework persistence: add back some safety and convenience when persisting
our types. Working with changeset directly (as we were doing before) can
be cumbersome.
* Intoduce `struct Persisted<T>` which wraps a type `T` which stores
staged changes to it. This adds safety when creating and or loading
`T` from db.
* `struct Persisted<T>` methods, `create`, `load` and `persist`, are
avaliable if `trait PersistWith<Db>` is implemented for `T`. `Db`
represents the database connection and `PersistWith` should be
implemented per database-type.
* For async, we have `trait PersistedAsyncWith<Db>`.
* `Wallet` has impls of `PersistedWith<rusqlite::Connection>`,
`PersistedWith<rusqlite::Transaction>` and
`PersistedWith<bdk_file_store::Store>` by default.
Rework wallet-construction: Before, we had multiple methods for loading
and creating with different input-counts so it would be unwieldly to add
more parameters in the future. This also makes it difficult to impl
`PersistWith` (which has a single method for `load` that takes in
`PersistWith::LoadParams` and a single method for `create` that takes in
`PersistWith::CreateParams`).
* Introduce a builder pattern when constructing a `Wallet`. For loading
from persistence or `ChangeSet`, we have `LoadParams`. For creating a
new wallet, we have `CreateParams`.
Both `bdk_electrum` and `bdk_esplora` now report the exact block
that the transaction is in, which removes the need for having the
old `ConfirmationTimeHeightAnchor` and `ConfirmationHeightAnchor`.
This PR introduces a new, simpler anchor type that can be modified
to support additional data in the future.
Remove `PersistBackend`, `PersistBackendAsync`, `StageExt` and
`StageExtAsync`. Remove `async` feature flag and dependency. Update
examples and wallet.
The previous commit b9c5b9d08b040faf6c6b2d9b3745918031555b72 added
IndexSpk. This goes further and adds `Indexed` and `KeychainIndexed`
type alises (IndexSpk is Indexed<ScriptBuf>) and attempts to standardize
the structure of return types more generally.
The underlying SpkTxOutIndex should not use DescriptorIds to index
because this loses the ordering relationship of the spks so queries on
subranges of keychains work.
Along with that we enforce that there is a strict 1-to-1 relationship
between descriptors and keychains. Violating this leads to an error in
insert_descriptor now.
In general I try to make the translation layer between the SpkTxOutIndex
and the KeychainTxOutIndex thinner. Ergonomics of this will be improved
in next commit.
The test from the previous commit passes.
deps(chain): bump `bitcoin` to `0.32.0`, miniscript to `12.0.0`
fix(chain): use `minimal_non_dust()` instead of `dust_value()`
fix(chain): use `compute_txid()` instead of `txid`
deps(testenv): bump `electrsd` to `0.28.0`
deps(electrum): bump `electrum-client` to `0.20.0`
fix(electrum): use `compute_txid()` instead of `txid`
deps(esplora): bump `esplora-client` to `0.8.0`
deps(bitcoind_rpc): bump `bitcoin` to `0.32.0`, `bitcoincore-rpc` to
`0.19.0`
fix(bitcoind_rpc): use `compute_txid()` instead of `txid`
fix(nursery/tmp_plan): use proper `sighash` errors, and fix the expected
`Signature` fields
fix(sqlite): use `compute_txid()` instead of `txid`
deps(hwi): bump `hwi` to `0.9.0`
deps(wallet): bump `bitcoin` to `0.32.0`, miniscript to `12.0.0`
fix(wallet): use `compute_txid()` and `minimal_non_dust()`
- update to use `compute_txid()` instead of deprecated `txid()`
- update to use `minimal_non_dust()` instead of `dust_value()`
- remove unused `bitcoin::hex::FromHex`.
fix(wallet): uses `.into` conversion on `Network` for `NetworkKind`
- uses `.into()` when appropriate, otherwise use the explicit
`NetworkKind`, and it's `.is_mainnet()` method.
fix(wallet): add P2wpkh, Taproot, InputsIndex errors to `SignerError`
fix(wallet): fields on taproot, and ecdsa `Signature` structure
fix(wallet/wallet): convert `Weight` to `usize` for now
- converts the `bitcoin-units::Weight` type to `usize` with help of
`to_wu()` method.
- it should be updated/refactored in the future to handle the `Weight`
type throughout the code instead of current `usize`, only converting
it for now.
- allows the usage of deprecated `is_provably_unspendable()`, needs
further discussion if suggested `is_op_return` is suitable.
- update the expect field to `signature`, as it was renamed from `sig`.
fix(wallet/wallet): use `is_op_return` instead of
`is_provably_unspendable`
fix(wallet/wallet): use `relative::Locktime` instead of `Sequence`
fix(wallet/descriptor): use `ParsePublicKeyError`
fix(wallet/descriptor): use `.into()` to convert from `AbsLockTime` and
`RelLockTime` to `absolute::LockTime` and `relative::LockTime`
fix(wallet/wallet): use `Message::from_digest()` instead of relying on
deprecated `ThirtyTwoByteHash` trait.
fix(wallet/descriptor+wallet): expect `Threshold` type, and handle it
internally
fix(wallet/wallet): remove `0x` prefix from expected `TxId` display
fix(examples): use `compute_txid()` instead of `txid`
fix(ci): remove usage of `bitcoin/no-std` feature
- remove comment: `# The `no-std` feature it's implied when the `std` feature is disabled.`
All `Wallet` constructors are modified to require a change
descriptor, where previously it was optional. Additionally
we enforce uniqueness of the change descriptor to avoid
ambiguity when deriving scripts and ensure the wallet will
always have two distinct keystores.
Notable changes
* Add error DescriptorError::ExternalAndInternalAreTheSame
* Remove error CreateTxError::ChangePolicyDescriptor
* No longer rely on `map_keychain`
2d2656acfa83ab4c4846c0aab14072efb64c5cc3 feat(electrum): re-export `transaction_broadcast` method (志宇)
53fa35096fbeea2b82b28987e8ef6f7d39ffc80b refactor(electrum)!: put the tx cache in electrum (LLFourn)
Pull request description:
Previously there was a `TxCache` that you passed in as part of the sync request. There are lots of downsides to this:
1. If the user forgets to do this you cache nothing
2. where are you meant to keep this cache? The example shows it being recreated every time which seems very suboptimal.
3. More API and documentation surface area.
Instead just do a plain old simple cache inside the electrum client. This way at least you only download transactions once. You can pre-populate the cache with a method also and I did this in the examples.
* [x] This pull request breaks the existing API
ACKs for top commit:
evanlinjin:
self-ACK 2d2656acfa83ab4c4846c0aab14072efb64c5cc3
notmandatory:
ACK 2d2656acfa83ab4c4846c0aab14072efb64c5cc3
Tree-SHA512: 6c29fd4f99ea5bd66234d5cdaf4b157a192ddd3baacc91076e402d8df0de7010bc482e24895e85fcb2f805ec6d1ce6cdb7654f8f552c90ba75ed35f80a00b856
Previously there was a tx cache that you passed in as part of the sync
request. This seems bad and the example show'd that you should copy all
your transactions from the transaction graph into the sync request every
time you sync'd. If you forgot to do this then you would always download everything.
Instead just do a plain old simple cache inside the electrum client.
This way at least you only download transactions once. You can
pre-populate the cache with a method also and I did this in the examples.
The previous `TxOut` for transactions received from an external
wallet may be optionally added as floating `TxOut`s to `TxGraph`
to allow for fee calculation.
This is more code, but a much more elegant solution than having
`ElectrumExt` methods return `SyncResult`/`FullScanResult` and having an
`ElectrumResultExt` extention trait.
`ElectrumResultExt` trait is also introduced that adds methods which can
convert the `Anchor` type for the update `TxGraph`.
We also make use of the new `TxCache` fields in
`SyncRequest`/`FullScanRequest`. This way, we can avoid re-fetching full
transactions from Electrum if not needed.
Examples and tests are updated to use the new `ElectrumExt` API.
This PR removes `RelevantTxids` from the electrum crate and tracks
transactions in a `TxGraph`. This removes the need to separately
construct a `TxGraph` after a `full_scan` or `sync`.
- The KeychainTxOutIndex's internal SpkIterator now uses DescriptorId
instead of K. The DescriptorId -> K translation is made at the
KeychainTxOutIndex level.
- The keychain::Changeset is now a struct, which includes a map for last
revealed indexes, and one for newly added keychains and their
descriptor.
API changes in bdk:
- Wallet::keychains returns a `impl Iterator` instead of `BTreeMap`
- Wallet::load doesn't take descriptors anymore, since they're stored in
the db
- Wallet::new_or_load checks if the loaded descriptor from db is the
same as the provided one
API changes in bdk_chain:
- `ChangeSet` is now a struct, which includes a map for last revealed
indexes, and one for keychains and descriptors.
- `KeychainTxOutIndex::inner` returns a `SpkIterator<(DescriptorId, u32)>`
- `KeychainTxOutIndex::outpoints` returns a `impl Iterator` instead of `&BTreeSet`
- `KeychainTxOutIndex::keychains` returns a `impl Iterator` instead of
`&BTreeMap`
- `KeychainTxOutIndex::txouts` doesn't return a ExactSizeIterator
anymore
- `KeychainTxOutIndex::unbounded_spk_iter` returns an `Option`
- `KeychainTxOutIndex::next_index` returns an `Option`
- `KeychainTxOutIndex::last_revealed_indices` returns a `BTreeMap`
instead of `&BTreeMap`
- `KeychainTxOutIndex::reveal_to_target` returns an `Option`
- `KeychainTxOutIndex::reveal_next_spk` returns an `Option`
- `KeychainTxOutIndex::next_unused_spk` returns an `Option`
- `KeychainTxOutIndex::add_keychain` has been renamed to
`KeychainTxOutIndex::insert_descriptor`, and now it returns a
ChangeSet
- update all fields `immature`, ` trusted_pending`, `unstrusted_pending`
and `confirmed` to use the `bitcoin::Amount` instead of `u64`
- update all `impl Balance` methods to use `bitcoin::Amount`
- update all tests that relies on `keychain::Balance`
c0374a0eeb155f1cb4f9d99223f6bc65d24f7df8 feat(chain): `SyncRequest` now uses `ExactSizeIterator`s (志宇)
0f94f24aaf6374fe2d7c2abf32a870d3849fb8cc feat(esplora)!: update to use new sync/full-scan structures (志宇)
4c52f3e08e85e73db379ee3411cceb8dbcec92e1 feat(wallet): make wallet compatible with sync/full-scan structures (志宇)
cdfec5f90726f2313963c3a71e5d18bb10624736 feat(chain): add sync/full-scan structures for spk-based syncing (志宇)
Pull request description:
Fixes#1153
Replaces #1194
### Description
Introduce universal structures that represent sync/full-scan requests/results.
### Notes to the reviewers
This is based on #1194 but is different in the following ways:
* The functionality to print scan/sync progress is not reduced.
* `SyncRequest` and `FullScanRequest` is simplified and fields are exposed for more flexibility.
### Changelog notice
* Add universal structures for initiating/receiving sync/full-scan requests/results for spk-based syncing.
* Updated `bdk_esplora` chain-source to make use of new universal sync/full-scan structures.
### Checklists
#### All Submissions:
* [x] I've signed all my commits
* [x] I followed the [contribution guidelines](https://github.com/bitcoindevkit/bdk/blob/master/CONTRIBUTING.md)
* [x] I ran `cargo fmt` and `cargo clippy` before committing
#### New Features:
* [x] I've added tests for the new feature
* [x] I've added docs for the new feature
ACKs for top commit:
notmandatory:
tACK c0374a0eeb155f1cb4f9d99223f6bc65d24f7df8
Tree-SHA512: c2ad66d972a6785079bca615dfd128edcedf6b7a02670651a0ab1ce5b5174dd96f54644680eedbf55e3f1955fe5c34f632eadbd3f71d7ffde658753c6c6d42be
96a9aa6e63474dbd93a2ef969eef5b07c79e6491 feat(chain): refactor `merge_chains` (志宇)
2f22987c9e924800f8682b2dcbdde60fd26b069a chore(chain): fix comment (志宇)
daf588f016ec3118c875db8ed6b55fa03683f0f6 feat(chain): optimize `merge_chains` (志宇)
77d35954c1f3a18f767267e9097f63ca11c709ec feat(chain)!: rm `local_chain::Update` (志宇)
1269b0610efb7bd86d92a909800f9330568c797a test(chain): fix incorrect test case (志宇)
72fe65b65f297ebb7160eee6859c46e29c2d9528 feat(esplora)!: simplify chain update logic (志宇)
eded1a7ea0c6a4b9664826df4f77b714cbad0bcc feat(chain): introduce `CheckPoint::insert` (志宇)
519cd75d23fbb72321b0b189dca12afbfd78c0c7 test(esplora): move esplora tests into src files (志宇)
a6e613e6b978b995abf6c92a16df0300b113aa2c test(esplora): add `test_finalize_chain_update` (志宇)
494d253493f1bc914adba16a28ccf1bc0a0f4ec8 feat(testenv): add `genesis_hash` method (志宇)
886d72e3d541d088320bbdad6804057f32aca684 chore(chain)!: rm `missing_heights` and `missing_heights_from` methods (志宇)
bd62aa0fe199d676710c9909617198d62f4897c0 feat(esplora)!: remove `EsploraExt::update_local_chain` (志宇)
1e997939837e9c1f0c087d6d28ac12e373c8c05f feat(testenv): add `make_checkpoint_tip` (志宇)
Pull request description:
Fixes#1354
### Description
Built on top of both #1369 and #1373, we simplify the `EsploraExt` API by removing the `update_local_chain` method and having `full_scan` and `sync` update the local chain in the same call. The `full_scan` and `sync` methods now takes in an additional input (`local_tip`) which provides us with the view of the `LocalChain` before the update. These methods now return structs `FullScanUpdate` and `SyncUpdate`.
The examples are updated to use this new API. `TxGraph::missing_heights` and `tx_graph::ChangeSet::missing_heights_from` are no longer needed, therefore they are removed.
Additionally, we used this opportunity to simplify the logic which updates `LocalChain`. We got rid of the `local_chain::Update` struct (which contained the update `CheckPoint` tip and a `bool` which signaled whether we want to introduce blocks below point of agreement). It turns out we can use something like `CheckPoint::insert` so the chain source can craft an update based on the old tip. This way, we can make better use of `merge_chains`' optimization that compares the `Arc` pointers of the local and update chain (before we were crafting the update chain NOT based on top of the previous local chain). With this, we no longer need the `Update::introduce_older_block` field since the logic will naturally break when we reach a matching `Arc` pointer.
### Notes to the reviewers
* Obtaining the `LocalChain`'s update now happens within `EsploraExt::full_scan` and `EsploraExt::sync`. Creating the `LocalChain` update is now split into two methods (`fetch_latest_blocks` and `chain_update`) that are called before and after fetching transactions and anchors.
* We need to duplicate code for `bdk_esplora`. One for blocking and one for async.
### Changelog notice
* Changed `EsploraExt` API so that sync only requires one round of fetching data. The `local_chain_update` method is removed and the `local_tip` parameter is added to the `full_scan` and `sync` methods.
* Removed `TxGraph::missing_heights` and `tx_graph::ChangeSet::missing_heights_from` methods.
* Introduced `CheckPoint::insert` which allows convenient checkpoint-insertion. This is intended for use by chain-sources when crafting an update.
* Refactored `merge_chains` to also return the resultant `CheckPoint` tip.
* Optimized the update `LocalChain` logic - use the update `CheckPoint` as the new `CheckPoint` tip when possible.
### Checklists
#### All Submissions:
* [x] I've signed all my commits
* [x] I followed the [contribution guidelines](https://github.com/bitcoindevkit/bdk/blob/master/CONTRIBUTING.md)
* [x] I ran `cargo fmt` and `cargo clippy` before committing
#### New Features:
* [x] I've added tests for the new feature
* [x] I've added docs for the new feature
ACKs for top commit:
LLFourn:
ACK 96a9aa6e63474dbd93a2ef969eef5b07c79e6491
Tree-SHA512: 3d4f2eab08a1fe94eb578c594126e99679f72e231680b2edd4bfb018ba1d998ca123b07acb2d19c644d5887fc36b8e42badba91cd09853df421ded04de45bf69
e51af49ffa951e14203ac2b465ade351dd90f6cd fix(wallet): remove generic from wallet (Rob N)
Pull request description:
### Description
The `PersistenceBackend` uses generics to describe errors returned while applying the change set to the persistence layer. This change removes generics wherever possible and introduces a new public error enum. Removing the generics from `PersistenceBackend` errors is the first step towards #1363
*Update*: I proceeded with removing the generics from `Wallet` by introducing a `Box<dyn PersistenceBackend>` .
### Notes to the reviewers
This one sort of blew up in the number of changes due to the use of generics for most of the `Wallet` error variants. The generics were only used for the persistence errors, so I removed the generics from higher level errors whenever possible. The error variants of `PersistenceBackend` may also be more expressive, but I will level that up for discussion and make any changes required.
### Changelog notice
- Changed `PersistenceBackend` errors to depend on the `anyhow` crate.
- Remove the generic `T` from `Wallet`
### Checklists
#### All Submissions:
* [x] I've signed all my commits
* [x] I followed the [contribution guidelines](https://github.com/bitcoindevkit/bdk/blob/master/CONTRIBUTING.md)
* [x] I ran `cargo fmt` and `cargo clippy` before committing
#### New Features:
* [ ] I've added tests for the new feature
* [x] I've added docs for the new feature
#### Bugfixes:
* [x] This pull request breaks the existing API
* [ ] I've added tests to reproduce the issue which are now passing
* [x] I'm linking the issue being fixed by this PR
ACKs for top commit:
evanlinjin:
ACK e51af49ffa951e14203ac2b465ade351dd90f6cd
Tree-SHA512: 8ce4f1c495310e16145555f4a6a29a0f42cf8944eda68004595c3532580767f64f779185022147a00d75001c40d69fdf8f8de2d348eb68484b170d2a181117ff
The intention is to remove the `Update::introduce_older_blocks`
parameter and update the local chain directly with `CheckPoint`.
This simplifies the API and there is a way to do this efficiently.
Previously, we would update the `TxGraph` and `KeychainTxOutIndex`
first, then create a second update for `LocalChain`. This required
locking the receiving structures 3 times (instead of twice, which
is optimal).
This PR eliminates this requirement by making use of the new `query`
method of `CheckPoint`.
Examples are also updated to use the new API.
Upgrade:
- bitcoin to v0.31.0
- miniscript to v11.0.0
Note: The bitcoin upgrade includes improvements to the
`Transaction::weight()` function, it appears those guys did good, we
no longer need to add the 2 additional weight units "just in case".
and `into_confirmation_time_tx_graph`, since now it makes
sense to use `TxGraph::update_last_seen_unconfirmed`.
Also, use `update_last_seen_unconfirmed` in examples for
electrum/esplora. We show how to update the last seen
time for transactions by calling `update_last_seen_unconfirmed`
on the graph update returned from a blockchain source, passing
in the current time, before applying it to another `TxGraph`.
