Other changes:
* The `async-https` feature of `bdk_esplora` is no longer default.
* Rename `ObservedAs` to `ChainPosition`.
* Set temporary MSRV to 1.60.0 to compile all workspace members will all
features.
* `ElectrumUpdate::missing_full_txs` now returns a `Vec<Txid>` so we
don't keep a reference to the passed-in `graph`.
* `ElectrumUpdate::finalize*` methods now takes in `missing` txids
instead of `full_txs`. `Client::batch_transaction_get` is called
within the methods.
Other changes:
* `wallet::ChangeSet` is now made public externally. This is required as
a wallet db should implement `PersistBackend<wallet::ChangeSet>`.
This corresponds to `keychain::KeychainChangeSet` but for the redesigned
structures with `LocalChain`.
This structure is now used in `Wallet` as well as the examples.
There are a number of improvements that can be done, but it is in a
decent state to be usable.
Possible improvements:
* Remove requirement to retry obtaining ALL data after reorg is
detected. Transactions can be anchored to a lower block (not block
tip), and an `assume_final_depth` value can be used.
* The logic to finalize an update with confirmation time can be improved
during reorgs to not require returning an error.
ed89de752c Improve txout filter/listing method docs for `TxGraph` (志宇)
fb75aa94a9 Clarify `TxGraph::try_filter_chain_unspents` logic (志宇)
96b1075132 Fix and improve `Persist::commit` method (志宇)
e01d17d59b Improve `txout` listing and balance APIs for redesigned structures (志宇)
Pull request description:
### Description
As noted in https://github.com/bitcoindevkit/bdk/issues/971#issuecomment-1542408941.
Instead of relying on a `OwnedIndexer` trait to filter for relevant txouts, we move the listing and balance methods from `IndexedTxGraph` to `TxGraph` and add an additional input (list of relevant outpoints) to these methods.
This provides a simpler implementation and a more flexible API.
#### Other Fixes
The `Persist::commit` method is fixed in 96b1075132.
Previously, regardless of whether writing to persistence backend is successful or not, the logic always cleared `self.staged`. This is changed to only clear `self.staged` after successful write.
Additionally, the written changeset (if any) is returned, and `PersistBackend::write_changes` will not be called if `self.staged` is empty.
### Notes to the reviewers
Yes, slightly more boilerplate to do the same things, but less code to maintain and a much more flexible API. Very worth it IMO.
### Changelog notice
### Checklists
#### All Submissions:
* [x] I've signed all my commits
* [x] I followed the [contribution guidelines](https://github.com/bitcoindevkit/bdk/blob/master/CONTRIBUTING.md)
* [x] I ran `cargo fmt` and `cargo clippy` before committing
#### New Features:
* [x] I've added tests for the new feature
* [x] I've added docs for the new feature
ACKs for top commit:
LLFourn:
ACK ed89de752c
Tree-SHA512: efae18c13ee74eff801febca61cb16bf4d8f3203ced96172b9ef555d8d2a749a382f87d024e8c92aacbab7eb4e50a8337de798e10524291ad80c6b35c4dc0161
Previously, regardless of whether writing to persistence backend is
successful or not, the logic always cleared `self.staged`. This is
changed to only clear `self.staged` after successful write.
Additionally, the written changeset (if any) is returned, and
`PersistBackend::write_changes` will not be called if `self.staged` is
empty.
Instead of relying on a `OwnedIndexer` trait to filter for relevant
txouts, we move the listing and balance methods from `IndexedTxGraph` to
`TxGraph` and add an additional input (list of relevant outpoints) to
these methods.
This provides a simpler implementation and a more flexible API.
4963240599 Add more `impl`s for `Append` and docs for file store `magic` (志宇)
2aa08a5898 [persist_redesign] Introduce redesigned `persist` types (志宇)
Pull request description:
### Description
This is part of #895 and #971
* Introduce a more generic version of the `keychain::persist::*` structures that only needs a single generic for the changeset type.
Additional changes:
* The `Append` trait has a new method `is_empty`.
* Introduce `Store` structure for `bdk_file_store` (which implements `PersistBackend`).
### Changelog notice
### Checklists
#### All Submissions:
* [x] I've signed all my commits
* [x] I followed the [contribution guidelines](https://github.com/bitcoindevkit/bdk/blob/master/CONTRIBUTING.md)
* [x] I ran `cargo fmt` and `cargo clippy` before committing
#### New Features:
* [x] I've added tests for the new feature
* [x] I've added docs for the new feature
Top commit has no ACKs.
Tree-SHA512: 0211fbe7d7e27805d3ed3a80b42f184cdff1cebb32fd559aa9838e4a7f7c7e47b6c366b6ef68e299f876bafed549b8d1d8b8cc0366bf5b61db079504a565b9b4
This is a more generic version of `keychain::persist::*` structures.
Additional changes:
* The `Append` trait has a new method `is_empty`.
* Introduce `Store` structure for `bdk_file_store`.
* Introduce `LocalChain::inner` method to get the inner map of block
height to hash.
* Replace `LocalChain::get_block` (which outputted `BlockId`, thus able
to return invalid representation) with `get_blockhash` that just
returns a `BlockHash`.
* Remove `TODO` comments that should be github tickets.
If `BlockId` implements `Anchor`, the meaning is ambiguous. We cannot
tell whether it means the tx is anchors at the block, or whether it also
means the tx is confirmed at that block.
Instead, `ConfirmationHeightAnchor` and `ConfirmationTimeAnchor` structs
are introduced as non-ambiguous `Anchor` implementations.
Additionally, `TxGraph::relevant_heights` is removed because it is also
ambiguous. What heights are deemed relevant? A simpler and more flexible
method `TxGraph::all_anchors` is introduced instead.
`TxGraph::try_get_chain_position` used to always exclude unconfirmed
transactions with last_seen value of 0. However, what is the point of
including a transaction in the graph if it cannot be part of the chain
history? Additionally, maybe sometimes we don't wish to use the
last_seen field at all.
The new behavior will consider unconfirmed transactions with last_seen
of 0.
* `Additions` now implements `Append` and uses `Append` to implement
`append()`.
* `append()` logic enforces that `last_seen` values should only
increase.
* Test written for `append()` with `last_seen` behaviour.
Instead of forcing all transactions inserted to use the same anchors, we
change the API to have unique anchors per transaction.
This allows for more flexibility in general. For example, use `Anchor`
implementations that contain the position in a block of a transaction.
The `insert_relevant_txs` test has also been changed to used
`KeychainTxOutIndex` so that index additions can be checked
(`SpkTxOutIndex` has no additions).
Additionally, generic bounds of some `IndexedTxGraph` list methods have
been fixed.
The `ConfirmationHeight` trait has been removed in favour of a second
method on the `Anchor` trait: `confirmation_height_upper_bound()`.
Methods `try_balance_at()` and `balance_at()` of `IndexedTxGraph` have
been removed as they do not provide additional functionality.
`IndexedTxGraph::insert_relevant_txs` now uses two loops, the first loop
indexes all transactions first. This is done as some indexes require
ancestor transactions to be indexed first and we cannot guarantee that
the input transactions are in topological order.
* Remove `chain_oracle::CacheBackend` for now as it is not used.
* `SparseChain` does not need to implement `ChainOracle`.
* Remove filter predicate for `list..` methods of `TxGraph` and
`IndexedTxGraph` as this is premature optimisation.
* `Append` can be implemented for all `BTreeMap`s and `BTreeSet`s,
instead of only `local_chain::ChangeSet`.
The `HashSet` was used for iterating without duplicate items. However,
since `anchors` is a `BTreeSet`, heights are in order. So a single
variable tracking last height will be sufficient.
This is mostly copying over the relevant tests from `SparseChain`.
Changes are made to `local_chain::ChangeSet` to re-add the ability to
remove blocks.
Previously, I have misunderstood the definition of anchor. If a tx is
anchored in a block, it does not necessarily mean it is confirmed in
that block. The tx can be confirmed in an ancestor block of the anchor
block.
With this new definition, we need a new trait `ConfirmationHeight` that
has one method `confirmation_height`. This trait can be used to extend
`Anchor` for those implementations that can give us the exact
conirmation height of a tx (which is useful in most cases).
Another change is to add another variant to the `ObservedAs` enum;
`ObservedAs::ConfirmedImplicit(A)`. If a tx does not have an anchor, but
another tx that spends it has an anchor that in in the best chain, we
can assume that tx is also in the best chain. The logic of
`TxGraph::try_get_chain_position` is also changed to reflect this.
Some methods from `IndexedTxGraph` have been moved to `TxGraph` as they
do not require the `Indexer`. Some `TxGraph` methods have been renamed
for clarity and consistency.
Also more docs are added.
The problem with the previous `ChainOracle` interface is that it had no
guarantee for consistency. For example, a block deemed to be part of the
"best chain" can be reorged out. So when `ChainOracle` is called
multiple times for an operation (such as getting the UTXO set), the
returned result may be inconsistent.
This PR changes `ChainOracle::is_block_in_chain` to take in another
input `static_block`, ensuring `block` is an ancestor of `static_block`.
Thus, if `static_block` is consistent across the operation, the result
will be consistent also.
`is_block_in_chain` now returns `Option<bool>`. The `None` case means
that the oracle implementation cannot determine whether block is an
ancestor of static block. `IndexedTxGraph::list_chain_txouts` handles
this case by checking child spends that are in chain, and if so, the
parent tx must be in chain too.
* Instead of implementing `ChainPosition` for `ObservedIn<BlockId>` to
use `FullTxOut` methods (`is_spendable_at` and `is_mature`), we create
alternative versions of those methods that require bounds with `Anchor`.
This removes all `ObservedIn<A>: ChainPosition` bounds for methods of
`IndexedTxGraph`.
* Various improvements to comments and names.
Introduce `chain_oracle::Cache` which is a cache for requests to the
chain oracle. `ChainOracle` has also been moved to the `chain_oracle`
module.
Introduce `get_tip_in_best_chain` method to the `ChainOracle` trait.
This allows for guaranteeing that chain state can be consistent across
operations with `IndexedTxGraph`.
Methods of old structures that return transaction(s) no longer return
`TxNode`, but `Transaction` as done previously.
`TxInGraph` is renamed to `TxNode`, while the internal `TxNode` is
renamed to `TxNodeInternal`.
Methods that list chain data have try and non-try versions. Both of
these versions now return an `Iterator`.
* Try versions return `Iterator<Item = Result>`.
* Non-try versions require the `ChainOracle` implementation to be
`ChainOracle<Error = Infallible>`.
The chain oracle keeps track of the best chain, while the transaction
index indexes transaction data in relation to script pubkeys.
This commit also includes initial work on `IndexedTxGraph`.
* Introduce `GraphedTx` struct to access transaction data of graphed
transactions.
* Ability to insert/access anchors and "seen at" values for graphed
transactions.
* `Additions` now records changes to anchors and last_seen_at.
Fixes include:
* Allow `bdk_esplora` to use async with tls
* Reorganize `bdk_esplora` crate to have separate files for
async vs blocking
* Use optional dependencies for `bdk_esplora` async
As mentioned by @LLFourn:
1. We have a "sparse chain" from which there is a subset of txids M that are missing from graph.
2. There is also another subset C that are in the graph but their positions have changed.
3. We used the Cow to avoid copying/duplicating in memory transactions in subset C and M
Instead in inflate_update we could remove transactions in subset M and just clone data in subset C (which is usually tiny).
`Wallet` stores the descriptors' checksum in the database for safety.
Previously, the checksum used was a checksum of a descriptor that
already had a checksum.
This PR allows for backward-compatibility of databases created with this
bug.
If `exclude_hash` is set, we split the input data, and if a checksum
already existed within the original data, we check the calculated
checksum against the original checksum.
Additionally, the implementation of `IntoWalletDescriptor` for `&str`
has been refactored for clarity.
This fixes a bug where `CoreTxIter` attempts to call `listtransactions`
immediately after a tx result is filtered (instead of being returned),
when in fact, the correct logic will be to pop another tx result.
The new logic also ensures that tx results are returned in chonological
order. The test `test_list_transactions` verifies this. We also now
ensure that `page_size` is between the range `[0 to 1000]` otherwise an
error is returned.
Some needless cloning is removed from `from_config` as well as logging
improvements.
These are as suggested by @danielabrozzoni and @afilini
Also introduced `RpcSyncParams::force_start_time` for users who
prioritise reliability above all else.
Also improved logging.
Before this commit, the rpc backend would not notice immature utxos
(`listunspent` does not return them), making the rpc balance different
to other blockchain implementations.
Co-authored-by: Daniela Brozzoni <danielabrozzoni@protonmail.com>
The new implementation fixes the following:
* We can track more than 100 scriptPubKeys
* We can obtain more than 1000 transactions per sync
* `TransactionDetails` for already-synced transactions are updated when
new scriptPubKeys are introduced (fixing the missing balance/coins
issue of supposedly tracked scriptPubKeys)
`RpcConfig` changes:
* Introduce `RpcSyncParams`.
* Remove `RpcConfig::skip_blocks` (this is replaced by
`RpcSyncParams::start_time`).
Previously, electrum-based blockchain implementations only synced for
`scriptPubKey`s that are already cached in `Database`.
This PR introduces a feedback mechanism, that uses `stop_gap` and the
difference between "current index" and "last active index" to determine
whether we need to cache more `scriptPubKeys`.
The `WalletSync::wallet_setup` trait now may return an
`Error::MissingCachedScripts` error which contains the number of extra
`scriptPubKey`s to cache, in order to satisfy `stop_gap` for the next call.
`Wallet::sync` now calls `WalletSync` in a loop, cacheing inbetween
subsequent calls (if needed).
`get_checksum_bytes` returns a descriptor checksum as `[u8; 8]` instead
of `String`, potentially improving performance and memory usage.
In addition to this, since descriptors only use charaters that fit
within a UTF-8 8-bit code unit, there is no need to use the `char` type
(which is 4 bytes). This can also potentially bring in some performance
and memory-usage benefits.
This bug seems to be Electrum-specific. The fix is to check the
proposed changes against the current state of the database. Ensure
newly suggested indexes are not smaller than indexes already in
database.
Changes:
* Check index updates before they are applied to database during
Electrum Blockchain sync (Thank you @rajarshimaitra for providing
an elegant solution).
Tests added:
* bdk_blockchain_tests!::test_sync_address_index_should_not_decrement
* bdk_blockchain_tests!::test_sync_address_index_should_increment
These tests ensure there will be no unexpected address reuse when
grabbing a new address via `Wallet::get_address` with `AddressIndex::New`.
Other changes:
* Tweak `rpc.rs` so that clippy is happy.
志宇
