Bump version to 0.19.0-dev

9ed36875f1 export: Rename `WalletExport` to `FullyNodedExport` (Alekos Filini)

Pull request description:

  ### Description

  The `WalletExport` type can still be used as an alias to `FullyNodedExport` but it has been marked as deprecated and will be removed in the next release.

  ### Checklists

  #### All Submissions:

  * [x] I've signed all my commits
  * [x] I followed the [contribution guidelines](https://github.com/bitcoindevkit/bdk/blob/master/CONTRIBUTING.md)
  * [x] I ran `cargo fmt` and `cargo clippy` before committing

  #### New Features:

  * [ ] I've added tests for the new feature
  * [x] I've added docs for the new feature
  * [x] I've updated `CHANGELOG.md`

ACKs for top commit:
  bucko13:
    Concept ACK 9ed3687
  notmandatory:
    ACK 9ed36875f1

Tree-SHA512: fe07307092d3040b30ae96901d5fc4afecee635412bca4a0d2e78d90f07990e7bbcd6b87ed923bf5804385356f419cacadbfefbaf97ebc9a622e34548438ee00

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,26 @@
+---
+name: Bug report
+about: Create a report to help us improve
+title: ''
+labels: 'bug'
+assignees: ''
+
+---
+
+**Describe the bug**  
+<!-- A clear and concise description of what the bug is. -->
+
+**To Reproduce**  
+<!-- Steps or code to reproduce the behavior. -->
+
+**Expected behavior**  
+<!-- A clear and concise description of what you expected to happen. -->
+
+**Build environment**  
+ - BDK tag/commit: <!-- e.g. v0.13.0, 3a07614 --> 
+ - OS+version: <!-- e.g. ubuntu 20.04.01, macOS 12.0.1, windows -->  
+ - Rust/Cargo version: <!-- e.g. 1.56.0 --> 
+ - Rust/Cargo target: <!-- e.g. x86_64-apple-darwin, x86_64-unknown-linux-gnu, etc. -->  
+
+**Additional context**  
+<!-- Add any other context about the problem here. --> 

.github/ISSUE_TEMPLATE/summer_project.md

@@ -0,0 +1,77 @@
+---
+name: Summer of Bitcoin Project
+about: Template to suggest a new https://www.summerofbitcoin.org/ project.
+title: ''
+labels: 'summer-of-bitcoin'
+assignees: ''
+
+---
+
+<!--
+## Overview  
+
+Project ideas are scoped for a university-level student with a basic background in CS and bitcoin 
+fundamentals - achievable over 12-weeks. Below are just a few types of ideas: 
+
+ - Low-hanging fruit: Relatively short projects with clear goals; requires basic technical knowledge 
+   and minimal familiarity with the codebase.
+ - Core development: These projects derive from the ongoing work from the core of your development 
+   team. The list of features and bugs is never-ending, and help is always welcome.
+ - Risky/Exploratory: These projects push the scope boundaries of your development effort. They 
+   might require expertise in an area not covered by your current development team. They might take 
+   advantage of a new technology. There is a reasonable chance that the project might be less 
+   successful, but the potential rewards make it worth the attempt.
+ - Infrastructure/Automation: These projects are the code that your organization uses to get its 
+   development work done; for example, projects that improve the automation of releases, regression 
+   tests and automated builds. This is a category where a Summer of Bitcoin student can be really 
+   helpful, doing work that the development team has been putting off while they focus on core 
+   development.
+ - Quality Assurance/Testing: Projects that work on and test your project's software development 
+   process. Additionally, projects that involve a thorough test and review of individual PRs.
+ - Fun/Peripheral: These projects might not be related to the current core development focus, but 
+   create new innovations and new perspectives for your project.
+-->
+
+**Description**
+<!-- Description: 3-7 sentences describing the project background and tasks to be done. -->  
+
+**Expected Outcomes**  
+<!-- Short bullet list describing what is to be accomplished -->   
+
+**Resources**  
+<!-- 2-3 reading materials for candidate to learn about the repo, project, scope etc -->  
+<!-- Recommended reading such as a developer/contributor guide -->  
+<!-- [Another example a paper citation](https://arxiv.org/pdf/1802.08091.pdf) -->  
+<!-- [Another example an existing issue](https://github.com/opencv/opencv/issues/11013) -->  
+<!-- [An existing related module](https://github.com/opencv/opencv_contrib/tree/master/modules/optflow) -->  
+
+**Skills Required**  
+<!-- 3-4 technical skills that the candidate should know -->  
+<!-- hands on experience with git -->  
+<!-- mastery plus experience coding in C++ -->  
+<!-- basic knowledge in matrix and tensor computations, college course work in cryptography -->  
+<!-- strong mathematical background -->
+<!-- Bonus - has experience with React Native. Best if you have also worked with OSSFuzz -->
+
+**Mentor(s)**  
+<!-- names of mentor(s) for this project go here -->
+
+**Difficulty**  
+<!-- Easy, Medium, Hard -->
+
+**Competency Test (optional)**  
+<!-- 2-3 technical tasks related to the project idea or repository you’d like a candidate to 
+    perform in order to demonstrate competency, good first bugs, warm-up exercises -->
+<!-- ex. Read the instructions here to get Bitcoin core running on your machine -->
+<!-- ex. pick an issue labeled as “newcomer” in the repository, and send a merge request to the 
+    repository. You can also suggest some other improvement that we did not think of yet, or 
+    something that you find interesting or useful -->
+<!-- ex. fixes for coding style are usually easy to do, and are good issues for first time 
+    contributions for those learning how to interact with the project. After you are done with the 
+    coding style issue, try making a different contribution. -->
+<!-- ex. setup a full Debian packaging development environment and learn the basics of Debian 
+    packaging. Then identify and package the missing dependencies to package Specter Desktop -->
+<!-- ex. write a pull parser for CSV files. You'll be judged by the decisions to store the parser 
+    state and how flexible it is to wrap this parser in other scenarios. -->
+<!-- ex. Stretch Goal: Implement some basic metaprogram/app to prove you're very familiar with BDK. 
+    Be prepared to make adjustments as we judge your solution. -->

.github/workflows/code_coverage.yml

@@ -31,7 +31,7 @@ jobs:
         uses: actions-rs/grcov@v0.1.5
 
       - name: Upload coverage to Codecov
-        uses: codecov/codecov-action@v1
+        uses: codecov/codecov-action@v2
         with:
           file: ${{ steps.coverage.outputs.report }}
           directory: ./coverage/reports/

.github/workflows/cont_integration.yml

@@ -10,8 +10,9 @@ jobs:
     strategy:
       matrix:
         rust:
-          - 1.53.0 # STABLE
+          - version: 1.56.0 # STABLE
-          - 1.46.0 # MSRV
+            clippy: true
+          - version: 1.46.0 # MSRV
         features:
           - default
           - minimal
@@ -27,11 +28,12 @@ jobs:
           - async-interface
           - use-esplora-reqwest
           - sqlite
+          - sqlite-bundled
     steps:
       - name: checkout
         uses: actions/checkout@v2
       - name: Generate cache key
-        run: echo "${{ matrix.rust }} ${{ matrix.features }}" | tee .cache_key
+        run: echo "${{ matrix.rust.version }} ${{ matrix.features }}" | tee .cache_key
       - name: cache
         uses: actions/cache@v2
         with:
@@ -41,16 +43,18 @@ jobs:
             target
           key: ${{ runner.os }}-cargo-${{ hashFiles('.cache_key') }}-${{ hashFiles('**/Cargo.toml','**/Cargo.lock') }}
       - name: Set default toolchain
-        run: rustup default ${{ matrix.rust }}
+        run: rustup default ${{ matrix.rust.version }}
       - name: Set profile
         run: rustup set profile minimal
       - name: Add clippy
+        if: ${{ matrix.rust.clippy }}
         run: rustup component add clippy
       - name: Update toolchain
         run: rustup update
       - name: Build
         run: cargo build --features ${{ matrix.features }} --no-default-features
       - name: Clippy
+        if: ${{ matrix.rust.clippy }}
         run: cargo clippy --all-targets --features ${{ matrix.features }} --no-default-features -- -D warnings
       - name: Test
         run: cargo test --features ${{ matrix.features }} --no-default-features
@@ -86,13 +90,13 @@ jobs:
       matrix:
         blockchain:
           - name: electrum
-            features: test-electrum
+            features: test-electrum,verify
           - name: rpc
             features: test-rpc
           - name: esplora
-            features: test-esplora,use-esplora-reqwest
+            features: test-esplora,use-esplora-reqwest,verify
           - name: esplora
-            features: test-esplora,use-esplora-ureq    
+            features: test-esplora,use-esplora-ureq,verify
     steps:
       - name: Checkout
         uses: actions/checkout@v2
@@ -111,7 +115,7 @@ jobs:
           override: true
       - name: Test
         run: cargo test --no-default-features --features ${{ matrix.blockchain.features }} ${{ matrix.blockchain.name }}::bdk_blockchain_tests
-  
+
   check-wasm:
     name: Check WASM
     runs-on: ubuntu-20.04
@@ -135,7 +139,7 @@ jobs:
       - run: sudo apt-get update || exit 1
       - run: sudo apt-get install -y libclang-common-10-dev clang-10 libc6-dev-i386 || exit 1
       - name: Set default toolchain
-        run: rustup default 1.53.0 # STABLE
+        run: rustup default 1.56.0 # STABLE
       - name: Set profile
         run: rustup set profile minimal
       - name: Add target wasm32

.github/workflows/nightly_docs.yml

@@ -18,7 +18,7 @@ jobs:
             target
           key: nightly-docs-${{ hashFiles('**/Cargo.toml','**/Cargo.lock') }}
       - name: Set default toolchain
-        run: rustup default nightly
+        run: rustup default nightly-2022-01-25
       - name: Set profile
         run: rustup set profile minimal
       - name: Update toolchain
@@ -44,18 +44,18 @@ jobs:
           repository: bitcoindevkit/bitcoindevkit.org
           ref: master
       - name: Create directories
-        run: mkdir -p ./static/docs-rs/bdk/nightly
+        run: mkdir -p ./docs/.vuepress/public/docs-rs/bdk/nightly
       - name: Remove old latest
-        run: rm -rf ./static/docs-rs/bdk/nightly/latest
+        run: rm -rf ./docs/.vuepress/public/docs-rs/bdk/nightly/latest
       - name: Download built docs
         uses: actions/download-artifact@v1
         with:
           name: built-docs
-          path: ./static/docs-rs/bdk/nightly/latest
+          path: ./docs/.vuepress/public/docs-rs/bdk/nightly/latest
       - name: Configure git
         run: git config user.email "github-actions@github.com" && git config user.name "github-actions"
       - name: Commit
         continue-on-error: true         # If there's nothing to commit this step fails, but it's fine
-        run: git add ./static && git commit -m "Publish autogenerated nightly docs"
+        run: git add ./docs/.vuepress/public/docs-rs && git commit -m "Publish autogenerated nightly docs"
       - name: Push
         run: git push origin master

CHANGELOG.md

@@ -7,6 +7,65 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ## [Unreleased]
 
 
+
+## [v0.18.0] - [v0.17.0]
+
+- Add `sqlite-bundled` feature for deployments that need a bundled version of sqlite, ie. for mobile platforms.
+- Added `Wallet::get_signers()`, `Wallet::descriptor_checksum()` and `Wallet::get_address_validators()`, exposed the `AsDerived` trait.
+- Deprecate `database::Database::flush()`, the function is only needed for the sled database on mobile, instead for mobile use the sqlite database.
+- Add `keychain: KeychainKind` to `wallet::AddressInfo`.
+- Improve key generation traits
+- Rename `WalletExport` to `FullyNodedExport`, deprecate the former.
+- Bump `miniscript` dependency version to `^6.1`. 
+
+## [v0.17.0] - [v0.16.1]
+
+- Removed default verification from `wallet::sync`. sync-time verification is added in `script_sync` and is activated by `verify` feature flag.
+- `verify` flag removed from `TransactionDetails`.
+- Add `get_internal_address` to allow you to get internal addresses just as you get external addresses.
+- added `ensure_addresses_cached` to `Wallet` to let offline wallets load and cache addresses in their database
+- Add `is_spent` field to `LocalUtxo`; when we notice that a utxo has been spent we set `is_spent` field to true instead of deleting it from the db.
+
+### Sync API change
+
+To decouple the `Wallet` from the `Blockchain` we've made major changes:
+
+- Removed `Blockchain` from Wallet.
+- Removed `Wallet::broadcast` (just use `Blockchain::broadcast`)
+- Deprecated `Wallet::new_offline` (all wallets are offline now)
+- Changed `Wallet::sync` to take a `Blockchain`.
+- Stop making a request for the block height when calling `Wallet:new`.
+- Added `SyncOptions` to capture extra (future) arguments to `Wallet::sync`.
+- Removed `max_addresses` sync parameter which determined how many addresses to cache before syncing since this can just be done with `ensure_addresses_cached`.
+
+## [v0.16.1] - [v0.16.0]
+
+- Pin tokio dependency version to ~1.14 to prevent errors due to their new MSRV 1.49.0
+
+## [v0.16.0] - [v0.15.0]
+
+- Disable `reqwest` default features.
+- Added `reqwest-default-tls` feature: Use this to restore the TLS defaults of reqwest if you don't want to add a dependency to it in your own manifest.
+- Use dust_value from rust-bitcoin
+- Fixed generating WIF in the correct network format.
+
+## [v0.15.0] - [v0.14.0]
+
+- Overhauled sync logic for electrum and esplora.
+- Unify ureq and reqwest esplora backends to have the same configuration parameters. This means reqwest now has a timeout parameter and ureq has a concurrency parameter.
+- Fixed esplora fee estimation.
+
+## [v0.14.0] - [v0.13.0]
+
+- BIP39 implementation dependency, in `keys::bip39` changed from tiny-bip39 to rust-bip39.
+- Add new method on the `TxBuilder` to embed data in the transaction via `OP_RETURN`. To allow that a fix to check the dust only on spendable output has been introduced.
+- Update the `Database` trait to store the last sync timestamp and block height
+- Rename `ConfirmationTime` to `BlockTime`
+
+## [v0.13.0] - [v0.12.0]
+
+- Exposed `get_tx()` method from `Database` to `Wallet`.
+
 ## [v0.12.0] - [v0.11.0]
 
 - Activate `miniscript/use-serde` feature to allow consumers of the library to access it via the re-exported `miniscript` crate.
@@ -371,7 +430,6 @@ final transaction is created by calling `finish` on the builder.
 - Use `MemoryDatabase` in the compiler example
 - Make the REPL return JSON
 
-[unreleased]: https://github.com/bitcoindevkit/bdk/compare/v0.11.0...HEAD
 [0.1.0-beta.1]: https://github.com/bitcoindevkit/bdk/compare/96c87ea5...0.1.0-beta.1
 [v0.2.0]: https://github.com/bitcoindevkit/bdk/compare/0.1.0-beta.1...v0.2.0
 [v0.3.0]: https://github.com/bitcoindevkit/bdk/compare/v0.2.0...v0.3.0
@@ -384,4 +442,12 @@ final transaction is created by calling `finish` on the builder.
 [v0.9.0]: https://github.com/bitcoindevkit/bdk/compare/v0.8.0...v0.9.0
 [v0.10.0]: https://github.com/bitcoindevkit/bdk/compare/v0.9.0...v0.10.0
 [v0.11.0]: https://github.com/bitcoindevkit/bdk/compare/v0.10.0...v0.11.0
-[v0.12.0]: https://github.com/bitcoindevkit/bdk/compare/v0.11.0...v0.12.0
+[v0.12.0]: https://github.com/bitcoindevkit/bdk/compare/v0.11.0...v0.12.0
+[v0.13.0]: https://github.com/bitcoindevkit/bdk/compare/v0.12.0...v0.13.0
+[v0.14.0]: https://github.com/bitcoindevkit/bdk/compare/v0.13.0...v0.14.0
+[v0.15.0]: https://github.com/bitcoindevkit/bdk/compare/v0.14.0...v0.15.0
+[v0.16.0]: https://github.com/bitcoindevkit/bdk/compare/v0.15.0...v0.16.0
+[v0.16.1]: https://github.com/bitcoindevkit/bdk/compare/v0.16.0...v0.16.1
+[v0.17.0]: https://github.com/bitcoindevkit/bdk/compare/v0.16.1...v0.17.0
+[v0.18.0]: https://github.com/bitcoindevkit/bdk/compare/v0.17.0...v0.18.0
+[unreleased]: https://github.com/bitcoindevkit/bdk/compare/v0.18.0...HEAD

Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "bdk"
-version = "0.12.1-dev"
+version = "0.19.0-dev"
 edition = "2018"
 authors = ["Alekos Filini <alekos.filini@gmail.com>", "Riccardo Casatta <riccardo@casatta.it>"]
 homepage = "https://bitcoindevkit.org"
@@ -14,7 +14,7 @@ license = "MIT OR Apache-2.0"
 [dependencies]
 bdk-macros = "^0.6"
 log = "^0.4"
-miniscript = { version = "^6.0", features = ["use-serde"] }
+miniscript = { version = "^6.1", features = ["use-serde"] }
 bitcoin = { version = "^0.27", features = ["use-serde", "base64"] }
 serde = { version = "^1.0", features = ["derive"] }
 serde_json = { version = "^1.0" }
@@ -24,8 +24,9 @@ rand = "^0.7"
 sled = { version = "0.34", optional = true }
 electrum-client = { version = "0.8", optional = true }
 rusqlite = { version = "0.25.3", optional = true }
-reqwest = { version = "0.11", optional = true, features = ["json"] }
+ahash = { version = "=0.7.4", optional = true }
-ureq = { version = "2.1", features = ["json"], optional = true }
+reqwest = { version = "0.11", optional = true, default-features = false, features = ["json"] }
+ureq = { version = "~2.2.0", features = ["json"], optional = true }
 futures = { version = "0.3", optional = true }
 async-trait = { version = "0.1", optional = true }
 rocksdb = { version = "0.14", default-features = false, features = ["snappy"], optional = true }
@@ -33,19 +34,15 @@ cc = { version = ">=1.0.64", optional = true }
 socks = { version = "0.3", optional = true }
 lazy_static = { version = "1.4", optional = true }
 
-# the latest 0.8 version of tiny-bip39 depends on zeroize_derive 1.2 which has MSRV 1.51 and our 
+bip39 = { version = "1.0.1", optional = true }
-# MSRV is 1.46, to fix this until we update our MSRV or replace the tiny-bip39 
-# dependency https://github.com/bitcoindevkit/bdk/issues/399 we can only use an older version
-tiny-bip39 = { version = "< 0.8", optional = true }
-
 bitcoinconsensus = { version = "0.19.0-3", optional = true }
 
 # Needed by bdk_blockchain_tests macro
-core-rpc = { version = "0.14", optional = true }
+bitcoincore-rpc = { version = "0.14", optional = true }
 
 # Platform-specific dependencies
 [target.'cfg(not(target_arch = "wasm32"))'.dependencies]
-tokio = { version = "1", features = ["rt"] }
+tokio = { version = "~1.14", features = ["rt"] }
 
 [target.'cfg(target_arch = "wasm32")'.dependencies]
 async-trait = "0.1"
@@ -57,12 +54,13 @@ minimal = []
 compiler = ["miniscript/compiler"]
 verify = ["bitcoinconsensus"]
 default = ["key-value-db", "electrum"]
-sqlite = ["rusqlite"]
+sqlite = ["rusqlite", "ahash"]
+sqlite-bundled = ["sqlite", "rusqlite/bundled"]
 compact_filters = ["rocksdb", "socks", "lazy_static", "cc"]
 key-value-db = ["sled"]
 all-keys = ["keys-bip39"]
-keys-bip39 = ["tiny-bip39"]
+keys-bip39 = ["bip39"]
-rpc = ["core-rpc"]
+rpc = ["bitcoincore-rpc"]
 
 # We currently provide mulitple implementations of `Blockchain`, all are
 # blocking except for the `EsploraBlockchain` which can be either async or
@@ -85,9 +83,11 @@ use-esplora-ureq = ["esplora", "ureq", "ureq/socks"]
 # Typical configurations will not need to use `esplora` feature directly.
 esplora = []
 
+# Use below feature with `use-esplora-reqwest` to enable reqwest default TLS support
+reqwest-default-tls = ["reqwest/default-tls"]
 
 # Debug/Test features
-test-blockchains = ["core-rpc", "electrum-client"]
+test-blockchains = ["bitcoincore-rpc", "electrum-client"]
 test-electrum = ["electrum", "electrsd/electrs_0_8_10", "test-blockchains"]
 test-rpc = ["rpc", "electrsd/electrs_0_8_10", "test-blockchains"]
 test-esplora = ["electrsd/legacy", "electrsd/esplora_a33e97e1", "test-blockchains"]
@@ -97,7 +97,7 @@ test-md-docs = ["electrum"]
 lazy_static = "1.4"
 env_logger = "0.7"
 clap = "2.33"
-electrsd = { version= "0.10", features = ["trigger", "bitcoind_0_21_1"] }
+electrsd = { version= "0.15", features = ["trigger", "bitcoind_22_0"] }
 
 [[example]]
 name = "address_validator"
@@ -110,6 +110,11 @@ name = "miniscriptc"
 path = "examples/compiler.rs"
 required-features = ["compiler"]
 
+[[example]]
+name = "rpcwallet"
+path = "examples/rpcwallet.rs"
+required-features = ["keys-bip39", "key-value-db", "rpc"]
+
 [workspace]
 members = ["macros"]
 [package.metadata.docs.rs]

README.md

@@ -1,7 +1,7 @@
 <div align="center">
   <h1>BDK</h1>
 
-  <img src="./static/bdk.svg" width="220" />
+  <img src="./static/bdk.png" width="220" />
 
   <p>
     <strong>A modern, lightweight, descriptor-based wallet library written in Rust!</strong>
@@ -41,21 +41,21 @@ The `bdk` library aims to be the core building block for Bitcoin wallets of any
 ```rust,no_run
 use bdk::Wallet;
 use bdk::database::MemoryDatabase;
-use bdk::blockchain::{noop_progress, ElectrumBlockchain};
+use bdk::blockchain::ElectrumBlockchain;
+use bdk::SyncOptions;
 
 use bdk::electrum_client::Client;
 
 fn main() -> Result<(), bdk::Error> {
-    let client = Client::new("ssl://electrum.blockstream.info:60002")?;
+    let blockchain = ElectrumBlockchain::from(Client::new("ssl://electrum.blockstream.info:60002")?);
     let wallet = Wallet::new(
         "wpkh([c258d2e4/84h/1h/0h]tpubDDYkZojQFQjht8Tm4jsS3iuEmKjTiEGjG6KnuFNKKJb5A6ZUCUZKdvLdSDWofKi4ToRCwb9poe1XdqfUnP4jaJjCB2Zwv11ZLgSbnZSNecE/0/*)",
         Some("wpkh([c258d2e4/84h/1h/0h]tpubDDYkZojQFQjht8Tm4jsS3iuEmKjTiEGjG6KnuFNKKJb5A6ZUCUZKdvLdSDWofKi4ToRCwb9poe1XdqfUnP4jaJjCB2Zwv11ZLgSbnZSNecE/1/*)"),
         bitcoin::Network::Testnet,
         MemoryDatabase::default(),
-        ElectrumBlockchain::from(client)
     )?;
 
-    wallet.sync(noop_progress(), None)?;
+    wallet.sync(&blockchain, SyncOptions::default())?;
 
     println!("Descriptor balance: {} SAT", wallet.get_balance()?);
 
@@ -70,7 +70,7 @@ use bdk::{Wallet, database::MemoryDatabase};
 use bdk::wallet::AddressIndex::New;
 
 fn main() -> Result<(), bdk::Error> {
-    let wallet = Wallet::new_offline(
+    let wallet = Wallet::new(
         "wpkh([c258d2e4/84h/1h/0h]tpubDDYkZojQFQjht8Tm4jsS3iuEmKjTiEGjG6KnuFNKKJb5A6ZUCUZKdvLdSDWofKi4ToRCwb9poe1XdqfUnP4jaJjCB2Zwv11ZLgSbnZSNecE/0/*)",
         Some("wpkh([c258d2e4/84h/1h/0h]tpubDDYkZojQFQjht8Tm4jsS3iuEmKjTiEGjG6KnuFNKKJb5A6ZUCUZKdvLdSDWofKi4ToRCwb9poe1XdqfUnP4jaJjCB2Zwv11ZLgSbnZSNecE/1/*)"),
         bitcoin::Network::Testnet,
@@ -88,9 +88,9 @@ fn main() -> Result<(), bdk::Error> {
 ### Create a transaction
 
 ```rust,no_run
-use bdk::{FeeRate, Wallet};
+use bdk::{FeeRate, Wallet, SyncOptions};
 use bdk::database::MemoryDatabase;
-use bdk::blockchain::{noop_progress, ElectrumBlockchain};
+use bdk::blockchain::ElectrumBlockchain;
 
 use bdk::electrum_client::Client;
 use bdk::wallet::AddressIndex::New;
@@ -98,16 +98,15 @@ use bdk::wallet::AddressIndex::New;
 use bitcoin::consensus::serialize;
 
 fn main() -> Result<(), bdk::Error> {
-    let client = Client::new("ssl://electrum.blockstream.info:60002")?;
+    let blockchain = ElectrumBlockchain::from(Client::new("ssl://electrum.blockstream.info:60002")?);
     let wallet = Wallet::new(
         "wpkh([c258d2e4/84h/1h/0h]tpubDDYkZojQFQjht8Tm4jsS3iuEmKjTiEGjG6KnuFNKKJb5A6ZUCUZKdvLdSDWofKi4ToRCwb9poe1XdqfUnP4jaJjCB2Zwv11ZLgSbnZSNecE/0/*)",
         Some("wpkh([c258d2e4/84h/1h/0h]tpubDDYkZojQFQjht8Tm4jsS3iuEmKjTiEGjG6KnuFNKKJb5A6ZUCUZKdvLdSDWofKi4ToRCwb9poe1XdqfUnP4jaJjCB2Zwv11ZLgSbnZSNecE/1/*)"),
         bitcoin::Network::Testnet,
         MemoryDatabase::default(),
-        ElectrumBlockchain::from(client)
     )?;
 
-    wallet.sync(noop_progress(), None)?;
+    wallet.sync(&blockchain, SyncOptions::default())?;
 
     let send_to = wallet.get_address(New)?;
     let (psbt, details) = {
@@ -135,7 +134,7 @@ use bdk::{Wallet, SignOptions, database::MemoryDatabase};
 use bitcoin::consensus::deserialize;
 
 fn main() -> Result<(), bdk::Error> {
-    let wallet = Wallet::new_offline(
+    let wallet = Wallet::new(
         "wpkh([c258d2e4/84h/1h/0h]tprv8griRPhA7342zfRyB6CqeKF8CJDXYu5pgnj1cjL1u2ngKcJha5jjTRimG82ABzJQ4MQe71CV54xfn25BbhCNfEGGJZnxvCDQCd6JkbvxW6h/0/*)",
         Some("wpkh([c258d2e4/84h/1h/0h]tprv8griRPhA7342zfRyB6CqeKF8CJDXYu5pgnj1cjL1u2ngKcJha5jjTRimG82ABzJQ4MQe71CV54xfn25BbhCNfEGGJZnxvCDQCd6JkbvxW6h/1/*)"),
         bitcoin::Network::Testnet,

examples/address_validator.rs

@@ -48,8 +48,7 @@ impl AddressValidator for DummyValidator {
 
 fn main() -> Result<(), bdk::Error> {
     let descriptor = "sh(and_v(v:pk(tpubDDpWvmUrPZrhSPmUzCMBHffvC3HyMAPnWDSAQNBTnj1iZeJa7BZQEttFiP4DS4GCcXQHezdXhn86Hj6LHX5EDstXPWrMaSneRWM8yUf6NFd/*),after(630000)))";
-    let mut wallet =
+    let mut wallet = Wallet::new(descriptor, None, Network::Regtest, MemoryDatabase::new())?;
-        Wallet::new_offline(descriptor, None, Network::Regtest, MemoryDatabase::new())?;
 
     wallet.add_address_validator(Arc::new(DummyValidator));
 

examples/compact_filters_balance.rs

@@ -10,7 +10,6 @@
 // licenses.
 
 use bdk::blockchain::compact_filters::*;
-use bdk::blockchain::noop_progress;
 use bdk::database::MemoryDatabase;
 use bdk::*;
 use bitcoin::*;
@@ -35,9 +34,8 @@ fn main() -> Result<(), CompactFiltersError> {
     let descriptor = "wpkh(tpubD6NzVbkrYhZ4X2yy78HWrr1M9NT8dKeWfzNiQqDdMqqa9UmmGztGGz6TaLFGsLfdft5iu32gxq1T4eMNxExNNWzVCpf9Y6JZi5TnqoC9wJq/*)";
 
     let database = MemoryDatabase::default();
-    let wallet =
+    let wallet = Arc::new(Wallet::new(descriptor, None, Network::Testnet, database).unwrap());
-        Arc::new(Wallet::new(descriptor, None, Network::Testnet, database, blockchain).unwrap());
+    wallet.sync(&blockchain, SyncOptions::default()).unwrap();
-    wallet.sync(noop_progress(), None).unwrap();
     info!("balance: {}", wallet.get_balance()?);
     Ok(())
 }

examples/compiler.rs

@@ -70,7 +70,7 @@ fn main() -> Result<(), Box<dyn Error>> {
     let policy_str = matches.value_of("POLICY").unwrap();
     info!("Compiling policy: {}", policy_str);
 
-    let policy = Concrete::<String>::from_str(&policy_str)?;
+    let policy = Concrete::<String>::from_str(policy_str)?;
 
     let descriptor = match matches.value_of("TYPE").unwrap() {
         "sh" => Descriptor::new_sh(policy.compile()?)?,
@@ -89,7 +89,7 @@ fn main() -> Result<(), Box<dyn Error>> {
         .transpose()
         .unwrap()
         .unwrap_or(Network::Testnet);
-    let wallet = Wallet::new_offline(&format!("{}", descriptor), None, network, database)?;
+    let wallet = Wallet::new(&format!("{}", descriptor), None, network, database)?;
 
     info!("... First address: {}", wallet.get_address(New)?);
 

examples/rpcwallet.rs

@@ -0,0 +1,229 @@
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
+//
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
+
+use bdk::bitcoin::secp256k1::Secp256k1;
+use bdk::bitcoin::Amount;
+use bdk::bitcoin::Network;
+use bdk::bitcoincore_rpc::RpcApi;
+
+use bdk::blockchain::rpc::{Auth, RpcBlockchain, RpcConfig};
+use bdk::blockchain::ConfigurableBlockchain;
+
+use bdk::keys::bip39::{Language, Mnemonic, WordCount};
+use bdk::keys::{DerivableKey, GeneratableKey, GeneratedKey};
+
+use bdk::miniscript::miniscript::Segwitv0;
+
+use bdk::sled;
+use bdk::template::Bip84;
+use bdk::wallet::{signer::SignOptions, wallet_name_from_descriptor, AddressIndex, SyncOptions};
+use bdk::KeychainKind;
+use bdk::Wallet;
+
+use bdk::blockchain::Blockchain;
+
+use electrsd;
+
+use std::error::Error;
+use std::path::PathBuf;
+use std::str::FromStr;
+
+/// This example demonstrates a typical way to create a wallet and work with bdk.
+///
+/// This example bdk wallet is connected to a bitcoin core rpc regtest node,
+/// and will attempt to receive, create and broadcast transactions.
+///
+/// To start a bitcoind regtest node programmatically, this example uses
+/// `electrsd` library, which is also a bdk dev-dependency.
+///
+/// But you can start your own bitcoind backend, and the rest of the example should work fine.
+
+fn main() -> Result<(), Box<dyn Error>> {
+    // -- Setting up background bitcoind process
+
+    println!(">> Setting up bitcoind");
+
+    // Start the bitcoind process
+    let bitcoind_conf = electrsd::bitcoind::Conf::default();
+
+    // electrsd will automatically download the bitcoin core binaries
+    let bitcoind_exe =
+        electrsd::bitcoind::downloaded_exe_path().expect("We should always have downloaded path");
+
+    // Launch bitcoind and gather authentication access
+    let bitcoind = electrsd::bitcoind::BitcoinD::with_conf(bitcoind_exe, &bitcoind_conf).unwrap();
+    let bitcoind_auth = Auth::Cookie {
+        file: bitcoind.params.cookie_file.clone(),
+    };
+
+    // Get a new core address
+    let core_address = bitcoind.client.get_new_address(None, None)?;
+
+    // Generate 101 blocks and use the above address as coinbase
+    bitcoind.client.generate_to_address(101, &core_address)?;
+
+    println!(">> bitcoind setup complete");
+    println!(
+        "Available coins in Core wallet : {}",
+        bitcoind.client.get_balance(None, None)?
+    );
+
+    // -- Setting up the Wallet
+
+    println!("\n>> Setting up BDK wallet");
+
+    // Get a random private key
+    let xprv = generate_random_ext_privkey()?;
+
+    // Use the derived descriptors from the privatekey to
+    // create unique wallet name.
+    // This is a special utility function exposed via `bdk::wallet_name_from_descriptor()`
+    let wallet_name = wallet_name_from_descriptor(
+        Bip84(xprv.clone(), KeychainKind::External),
+        Some(Bip84(xprv.clone(), KeychainKind::Internal)),
+        Network::Regtest,
+        &Secp256k1::new(),
+    )?;
+
+    // Create a database (using default sled type) to store wallet data
+    let mut datadir = PathBuf::from_str("/tmp/")?;
+    datadir.push(".bdk-example");
+    let database = sled::open(datadir)?;
+    let database = database.open_tree(wallet_name.clone())?;
+
+    // Create a RPC configuration of the running bitcoind backend we created in last step
+    // Note: If you are using custom regtest node, use the appropriate url and auth
+    let rpc_config = RpcConfig {
+        url: bitcoind.params.rpc_socket.to_string(),
+        auth: bitcoind_auth,
+        network: Network::Regtest,
+        wallet_name,
+        skip_blocks: None,
+    };
+
+    // Use the above configuration to create a RPC blockchain backend
+    let blockchain = RpcBlockchain::from_config(&rpc_config)?;
+
+    // Combine Database + Descriptor to create the final wallet
+    let wallet = Wallet::new(
+        Bip84(xprv.clone(), KeychainKind::External),
+        Some(Bip84(xprv.clone(), KeychainKind::Internal)),
+        Network::Regtest,
+        database,
+    )?;
+
+    // The `wallet` and the `blockchain` are independent structs.
+    // The wallet will be used to do all wallet level actions
+    // The blockchain can be used to do all blockchain level actions.
+    // For certain actions (like sync) the wallet will ask for a blockchain.
+
+    // Sync the wallet
+    // The first sync is important as this will instantiate the
+    // wallet files.
+    wallet.sync(&blockchain, SyncOptions::default())?;
+
+    println!(">> BDK wallet setup complete.");
+    println!(
+        "Available initial coins in BDK wallet : {} sats",
+        wallet.get_balance()?
+    );
+
+    // -- Wallet transaction demonstration
+
+    println!("\n>> Sending coins: Core --> BDK, 10 BTC");
+    // Get a new address to receive coins
+    let bdk_new_addr = wallet.get_address(AddressIndex::New)?.address;
+
+    // Send 10 BTC from core wallet to bdk wallet
+    bitcoind.client.send_to_address(
+        &bdk_new_addr,
+        Amount::from_btc(10.0)?,
+        None,
+        None,
+        None,
+        None,
+        None,
+        None,
+    )?;
+
+    // Confirm transaction by generating 1 block
+    bitcoind.client.generate_to_address(1, &core_address)?;
+
+    // Sync the BDK wallet
+    // This time the sync will fetch the new transaction and update it in
+    // wallet database
+    wallet.sync(&blockchain, SyncOptions::default())?;
+
+    println!(">> Received coins in BDK wallet");
+    println!(
+        "Available balance in BDK wallet: {} sats",
+        wallet.get_balance()?
+    );
+
+    println!("\n>> Sending coins: BDK --> Core, 5 BTC");
+    // Attempt to send back 5.0 BTC to core address by creating a transaction
+    //
+    // Transactions are created using a `TxBuilder`.
+    // This helps us to systematically build a transaction with all
+    // required customization.
+    // A full list of APIs offered by `TxBuilder` can be found at
+    // https://docs.rs/bdk/latest/bdk/wallet/tx_builder/struct.TxBuilder.html
+    let mut tx_builder = wallet.build_tx();
+
+    // For a regular transaction, just set the recipient and amount
+    tx_builder.set_recipients(vec![(core_address.script_pubkey(), 500000000)]);
+
+    // Finalize the transaction and extract the PSBT
+    let (mut psbt, _) = tx_builder.finish()?;
+
+    // Set signing option
+    let signopt = SignOptions {
+        assume_height: None,
+        ..Default::default()
+    };
+
+    // Sign the psbt
+    wallet.sign(&mut psbt, signopt)?;
+
+    // Extract the signed transaction
+    let tx = psbt.extract_tx();
+
+    // Broadcast the transaction
+    blockchain.broadcast(&tx)?;
+
+    // Confirm transaction by generating some blocks
+    bitcoind.client.generate_to_address(1, &core_address)?;
+
+    // Sync the BDK wallet
+    wallet.sync(&blockchain, SyncOptions::default())?;
+
+    println!(">> Coins sent to Core wallet");
+    println!(
+        "Remaining BDK wallet balance: {} sats",
+        wallet.get_balance()?
+    );
+    println!("\nCongrats!! you made your first test transaction with bdk and bitcoin core.");
+
+    Ok(())
+}
+
+// Helper function demonstrating privatekey extraction using bip39 mnemonic
+// The mnemonic can be shown to user to safekeeping and the same wallet
+// private descriptors can be recreated from it.
+fn generate_random_ext_privkey() -> Result<impl DerivableKey<Segwitv0> + Clone, Box<dyn Error>> {
+    // a Bip39 passphrase can be set optionally
+    let password = Some("random password".to_string());
+
+    // Generate a random mnemonic, and use that to create a "DerivableKey"
+    let mnemonic: GeneratedKey<_, _> = Mnemonic::generate((WordCount::Words12, Language::English))
+        .map_err(|e| e.expect("Unknown Error"))?;
+
+    // `Ok(mnemonic)` would also work if there's no passphrase and it would
+    // yield the same result as this construct with `password` = `None`.
+    Ok((mnemonic, password))
+}

src/blockchain/any.rs

@@ -16,61 +16,17 @@
 //!
 //! ## Example
 //!
-//! In this example both `wallet_electrum` and `wallet_esplora` have the same type of
+//! When paired with the use of [`ConfigurableBlockchain`], it allows creating any
-//! `Wallet<AnyBlockchain, MemoryDatabase>`. This means that they could both, for instance, be
-//! assigned to a struct member.
-//!
-//! ```no_run
-//! # use bitcoin::Network;
-//! # use bdk::blockchain::*;
-//! # use bdk::database::MemoryDatabase;
-//! # use bdk::Wallet;
-//! # #[cfg(feature = "electrum")]
-//! # {
-//! let electrum_blockchain = ElectrumBlockchain::from(electrum_client::Client::new("...")?);
-//! let wallet_electrum: Wallet<AnyBlockchain, _> = Wallet::new(
-//!     "...",
-//!     None,
-//!     Network::Testnet,
-//!     MemoryDatabase::default(),
-//!     electrum_blockchain.into(),
-//! )?;
-//! # }
-//!
-//! # #[cfg(all(feature = "esplora", feature = "ureq"))]
-//! # {
-//! let esplora_blockchain = EsploraBlockchain::new("...", 20);
-//! let wallet_esplora: Wallet<AnyBlockchain, _> = Wallet::new(
-//!     "...",
-//!     None,
-//!     Network::Testnet,
-//!     MemoryDatabase::default(),
-//!     esplora_blockchain.into(),
-//! )?;
-//! # }
-//!
-//! # Ok::<(), bdk::Error>(())
-//! ```
-//!
-//! When paired with the use of [`ConfigurableBlockchain`], it allows creating wallets with any
 //! blockchain type supported using a single line of code:
 //!
 //! ```no_run
 //! # use bitcoin::Network;
 //! # use bdk::blockchain::*;
-//! # use bdk::database::MemoryDatabase;
-//! # use bdk::Wallet;
 //! # #[cfg(all(feature = "esplora", feature = "ureq"))]
 //! # {
 //! let config = serde_json::from_str("...")?;
 //! let blockchain = AnyBlockchain::from_config(&config)?;
-//! let wallet = Wallet::new(
+//! let height = blockchain.get_height();
-//!     "...",
-//!     None,
-//!     Network::Testnet,
-//!     MemoryDatabase::default(),
-//!     blockchain,
-//! )?;
 //! # }
 //! # Ok::<(), bdk::Error>(())
 //! ```
@@ -133,33 +89,55 @@ impl Blockchain for AnyBlockchain {
         maybe_await!(impl_inner_method!(self, get_capabilities))
     }
 
-    fn setup<D: BatchDatabase, P: 'static + Progress>(
-        &self,
-        database: &mut D,
-        progress_update: P,
-    ) -> Result<(), Error> {
-        maybe_await!(impl_inner_method!(self, setup, database, progress_update))
-    }
-    fn sync<D: BatchDatabase, P: 'static + Progress>(
-        &self,
-        database: &mut D,
-        progress_update: P,
-    ) -> Result<(), Error> {
-        maybe_await!(impl_inner_method!(self, sync, database, progress_update))
-    }
-
-    fn get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error> {
-        maybe_await!(impl_inner_method!(self, get_tx, txid))
-    }
     fn broadcast(&self, tx: &Transaction) -> Result<(), Error> {
         maybe_await!(impl_inner_method!(self, broadcast, tx))
     }
 
+    fn estimate_fee(&self, target: usize) -> Result<FeeRate, Error> {
+        maybe_await!(impl_inner_method!(self, estimate_fee, target))
+    }
+}
+
+#[maybe_async]
+impl GetHeight for AnyBlockchain {
     fn get_height(&self) -> Result<u32, Error> {
         maybe_await!(impl_inner_method!(self, get_height))
     }
-    fn estimate_fee(&self, target: usize) -> Result<FeeRate, Error> {
+}
-        maybe_await!(impl_inner_method!(self, estimate_fee, target))
+
+#[maybe_async]
+impl GetTx for AnyBlockchain {
+    fn get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error> {
+        maybe_await!(impl_inner_method!(self, get_tx, txid))
+    }
+}
+
+#[maybe_async]
+impl WalletSync for AnyBlockchain {
+    fn wallet_sync<D: BatchDatabase>(
+        &self,
+        database: &mut D,
+        progress_update: Box<dyn Progress>,
+    ) -> Result<(), Error> {
+        maybe_await!(impl_inner_method!(
+            self,
+            wallet_sync,
+            database,
+            progress_update
+        ))
+    }
+
+    fn wallet_setup<D: BatchDatabase>(
+        &self,
+        database: &mut D,
+        progress_update: Box<dyn Progress>,
+    ) -> Result<(), Error> {
+        maybe_await!(impl_inner_method!(
+            self,
+            wallet_setup,
+            database,
+            progress_update
+        ))
     }
 }
 

src/blockchain/compact_filters/mod.rs

@@ -67,11 +67,11 @@ mod peer;
 mod store;
 mod sync;
 
-use super::{Blockchain, Capability, ConfigurableBlockchain, Progress};
+use crate::blockchain::*;
 use crate::database::{BatchDatabase, BatchOperations, DatabaseUtils};
 use crate::error::Error;
 use crate::types::{KeychainKind, LocalUtxo, TransactionDetails};
-use crate::{ConfirmationTime, FeeRate};
+use crate::{BlockTime, FeeRate};
 
 use peer::*;
 use store::*;
@@ -163,11 +163,19 @@ impl CompactFiltersBlockchain {
             if let Some(previous_output) = database.get_previous_output(&input.previous_output)? {
                 inputs_sum += previous_output.value;
 
-                if database.is_mine(&previous_output.script_pubkey)? {
+                // this output is ours, we have a path to derive it
+                if let Some((keychain, _)) =
+                    database.get_path_from_script_pubkey(&previous_output.script_pubkey)?
+                {
                     outgoing += previous_output.value;
 
-                    debug!("{} input #{} is mine, removing from utxo", tx.txid(), i);
+                    debug!("{} input #{} is mine, setting utxo as spent", tx.txid(), i);
-                    updates.del_utxo(&input.previous_output)?;
+                    updates.set_utxo(&LocalUtxo {
+                        outpoint: input.previous_output,
+                        txout: previous_output.clone(),
+                        keychain,
+                        is_spent: true,
+                    })?;
                 }
             }
         }
@@ -185,6 +193,7 @@ impl CompactFiltersBlockchain {
                     outpoint: OutPoint::new(tx.txid(), i as u32),
                     txout: output.clone(),
                     keychain,
+                    is_spent: false,
                 })?;
                 incoming += output.value;
 
@@ -206,8 +215,7 @@ impl CompactFiltersBlockchain {
                 transaction: Some(tx.clone()),
                 received: incoming,
                 sent: outgoing,
-                confirmation_time: ConfirmationTime::new(height, timestamp),
+                confirmation_time: BlockTime::new(height, timestamp),
-                verified: height.is_some(),
                 fee: Some(inputs_sum.saturating_sub(outputs_sum)),
             };
 
@@ -226,11 +234,38 @@ impl Blockchain for CompactFiltersBlockchain {
         vec![Capability::FullHistory].into_iter().collect()
     }
 
+    fn broadcast(&self, tx: &Transaction) -> Result<(), Error> {
+        self.peers[0].broadcast_tx(tx.clone())?;
+
+        Ok(())
+    }
+
+    fn estimate_fee(&self, _target: usize) -> Result<FeeRate, Error> {
+        // TODO
+        Ok(FeeRate::default())
+    }
+}
+
+impl GetHeight for CompactFiltersBlockchain {
+    fn get_height(&self) -> Result<u32, Error> {
+        Ok(self.headers.get_height()? as u32)
+    }
+}
+
+impl GetTx for CompactFiltersBlockchain {
+    fn get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error> {
+        Ok(self.peers[0]
+            .get_mempool()
+            .get_tx(&Inventory::Transaction(*txid)))
+    }
+}
+
+impl WalletSync for CompactFiltersBlockchain {
     #[allow(clippy::mutex_atomic)] // Mutex is easier to understand than a CAS loop.
-    fn setup<D: BatchDatabase, P: 'static + Progress>(
+    fn wallet_setup<D: BatchDatabase>(
         &self,
         database: &mut D,
-        progress_update: P,
+        progress_update: Box<dyn Progress>,
     ) -> Result<(), Error> {
         let first_peer = &self.peers[0];
 
@@ -254,7 +289,7 @@ impl Blockchain for CompactFiltersBlockchain {
         let total_cost = headers_cost + filters_cost + PROCESS_BLOCKS_COST;
 
         if let Some(snapshot) = sync::sync_headers(
-            Arc::clone(&first_peer),
+            Arc::clone(first_peer),
             Arc::clone(&self.headers),
             |new_height| {
                 let local_headers_cost =
@@ -275,7 +310,7 @@ impl Blockchain for CompactFiltersBlockchain {
         let buried_height = synced_height.saturating_sub(sync::BURIED_CONFIRMATIONS);
         info!("Synced headers to height: {}", synced_height);
 
-        cf_sync.prepare_sync(Arc::clone(&first_peer))?;
+        cf_sync.prepare_sync(Arc::clone(first_peer))?;
 
         let all_scripts = Arc::new(
             database
@@ -294,7 +329,7 @@ impl Blockchain for CompactFiltersBlockchain {
         let mut threads = Vec::with_capacity(self.peers.len());
         for peer in &self.peers {
             let cf_sync = Arc::clone(&cf_sync);
-            let peer = Arc::clone(&peer);
+            let peer = Arc::clone(peer);
             let headers = Arc::clone(&self.headers);
             let all_scripts = Arc::clone(&all_scripts);
             let last_synced_block = Arc::clone(&last_synced_block);
@@ -431,27 +466,6 @@ impl Blockchain for CompactFiltersBlockchain {
 
         Ok(())
     }
-
-    fn get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error> {
-        Ok(self.peers[0]
-            .get_mempool()
-            .get_tx(&Inventory::Transaction(*txid)))
-    }
-
-    fn broadcast(&self, tx: &Transaction) -> Result<(), Error> {
-        self.peers[0].broadcast_tx(tx.clone())?;
-
-        Ok(())
-    }
-
-    fn get_height(&self) -> Result<u32, Error> {
-        Ok(self.headers.get_height()? as u32)
-    }
-
-    fn estimate_fee(&self, _target: usize) -> Result<FeeRate, Error> {
-        // TODO
-        Ok(FeeRate::default())
-    }
 }
 
 /// Data to connect to a Bitcoin P2P peer
@@ -472,7 +486,7 @@ pub struct CompactFiltersBlockchainConfig {
     pub peers: Vec<BitcoinPeerConfig>,
     /// Network used
     pub network: Network,
-    /// Storage dir to save partially downloaded headers and full blocks
+    /// Storage dir to save partially downloaded headers and full blocks. Should be a separate directory per descriptor. Consider using [crate::wallet::wallet_name_from_descriptor] for this.
     pub storage_dir: String,
     /// Optionally skip initial `skip_blocks` blocks (default: 0)
     pub skip_blocks: Option<usize>,

src/blockchain/compact_filters/peer.rs

@@ -262,7 +262,7 @@ impl Peer {
         let message_resp = {
             let mut lock = responses.write().unwrap();
             let message_resp = lock.entry(wait_for).or_default();
-            Arc::clone(&message_resp)
+            Arc::clone(message_resp)
         };
 
         let (lock, cvar) = &*message_resp;
@@ -379,7 +379,7 @@ impl Peer {
             let message_resp = {
                 let mut lock = reader_thread_responses.write().unwrap();
                 let message_resp = lock.entry(in_message.cmd()).or_default();
-                Arc::clone(&message_resp)
+                Arc::clone(message_resp)
             };
 
             let (lock, cvar) = &*message_resp;

src/blockchain/compact_filters/store.rs

@@ -398,7 +398,7 @@ impl ChainStore<Full> {
             );
         }
 
-        // Delete full blocks overriden by snapshot
+        // Delete full blocks overridden by snapshot
         let from_key = StoreEntry::Block(Some(snaphost.min_height)).get_key();
         let to_key = StoreEntry::Block(Some(usize::MAX)).get_key();
         batch.delete_range(&from_key, &to_key);
@@ -760,7 +760,7 @@ impl CfStore {
         let cf_headers: Vec<FilterHeader> = filter_hashes
             .into_iter()
             .scan(checkpoint, |prev_header, filter_hash| {
-                let filter_header = filter_hash.filter_header(&prev_header);
+                let filter_header = filter_hash.filter_header(prev_header);
                 *prev_header = filter_header;
 
                 Some(filter_header)
@@ -801,7 +801,7 @@ impl CfStore {
             .zip(headers.into_iter())
             .scan(checkpoint, |prev_header, ((_, filter_content), header)| {
                 let filter = BlockFilter::new(&filter_content);
-                if header != filter.filter_header(&prev_header) {
+                if header != filter.filter_header(prev_header) {
                     return Some(Err(CompactFiltersError::InvalidFilter));
                 }
                 *prev_header = header;

src/blockchain/compact_filters/sync.rs

@@ -205,7 +205,7 @@ impl CfSync {
                     let block_hash = self.headers_store.get_block_hash(height)?.unwrap();
 
                     // TODO: also download random blocks?
-                    if process(&block_hash, &BlockFilter::new(&filter))? {
+                    if process(&block_hash, &BlockFilter::new(filter))? {
                         log::debug!("Downloading block {}", block_hash);
 
                         let block = peer

src/blockchain/electrum.rs

@@ -24,20 +24,20 @@
 //! # Ok::<(), bdk::Error>(())
 //! ```
 
-use std::collections::HashSet;
+use std::collections::{HashMap, HashSet};
 
 #[allow(unused_imports)]
 use log::{debug, error, info, trace};
 
-use bitcoin::{BlockHeader, Script, Transaction, Txid};
+use bitcoin::{Transaction, Txid};
 
 use electrum_client::{Client, ConfigBuilder, ElectrumApi, Socks5Config};
 
-use self::utils::{ElectrumLikeSync, ElsGetHistoryRes};
+use super::script_sync::Request;
 use super::*;
-use crate::database::BatchDatabase;
+use crate::database::{BatchDatabase, Database};
 use crate::error::Error;
-use crate::FeeRate;
+use crate::{BlockTime, FeeRate};
 
 /// Wrapper over an Electrum Client that implements the required blockchain traits
 ///
@@ -68,32 +68,10 @@ impl Blockchain for ElectrumBlockchain {
         .collect()
     }
 
-    fn setup<D: BatchDatabase, P: Progress>(
-        &self,
-        database: &mut D,
-        progress_update: P,
-    ) -> Result<(), Error> {
-        self.client
-            .electrum_like_setup(self.stop_gap, database, progress_update)
-    }
-
-    fn get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error> {
-        Ok(self.client.transaction_get(txid).map(Option::Some)?)
-    }
-
     fn broadcast(&self, tx: &Transaction) -> Result<(), Error> {
         Ok(self.client.transaction_broadcast(tx).map(|_| ())?)
     }
 
-    fn get_height(&self) -> Result<u32, Error> {
-        // TODO: unsubscribe when added to the client, or is there a better call to use here?
-
-        Ok(self
-            .client
-            .block_headers_subscribe()
-            .map(|data| data.height as u32)?)
-    }
-
     fn estimate_fee(&self, target: usize) -> Result<FeeRate, Error> {
         Ok(FeeRate::from_btc_per_kvb(
             self.client.estimate_fee(target)? as f32
@@ -101,43 +79,207 @@ impl Blockchain for ElectrumBlockchain {
     }
 }
 
-impl ElectrumLikeSync for Client {
+impl GetHeight for ElectrumBlockchain {
-    fn els_batch_script_get_history<'s, I: IntoIterator<Item = &'s Script> + Clone>(
+    fn get_height(&self) -> Result<u32, Error> {
+        // TODO: unsubscribe when added to the client, or is there a better call to use here?
+
+        Ok(self
+            .client
+            .block_headers_subscribe()
+            .map(|data| data.height as u32)?)
+    }
+}
+
+impl GetTx for ElectrumBlockchain {
+    fn get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error> {
+        Ok(self.client.transaction_get(txid).map(Option::Some)?)
+    }
+}
+
+impl WalletSync for ElectrumBlockchain {
+    fn wallet_setup<D: BatchDatabase>(
         &self,
-        scripts: I,
+        database: &mut D,
-    ) -> Result<Vec<Vec<ElsGetHistoryRes>>, Error> {
+        _progress_update: Box<dyn Progress>,
-        self.batch_script_get_history(scripts)
+    ) -> Result<(), Error> {
-            .map(|v| {
+        let mut request = script_sync::start(database, self.stop_gap)?;
-                v.into_iter()
+        let mut block_times = HashMap::<u32, u32>::new();
-                    .map(|v| {
+        let mut txid_to_height = HashMap::<Txid, u32>::new();
-                        v.into_iter()
+        let mut tx_cache = TxCache::new(database, &self.client);
-                            .map(
+        let chunk_size = self.stop_gap;
-                                |electrum_client::GetHistoryRes {
+        // The electrum server has been inconsistent somehow in its responses during sync. For
-                                     height, tx_hash, ..
+        // example, we do a batch request of transactions and the response contains less
-                                 }| ElsGetHistoryRes {
+        // tranascations than in the request. This should never happen but we don't want to panic.
-                                    height,
+        let electrum_goof = || Error::Generic("electrum server misbehaving".to_string());
-                                    tx_hash,
+
-                                },
+        let batch_update = loop {
-                            )
+            request = match request {
-                            .collect()
+                Request::Script(script_req) => {
-                    })
+                    let scripts = script_req.request().take(chunk_size);
-                    .collect()
+                    let txids_per_script: Vec<Vec<_>> = self
-            })
+                        .client
-            .map_err(Error::Electrum)
+                        .batch_script_get_history(scripts)
+                        .map_err(Error::Electrum)?
+                        .into_iter()
+                        .map(|txs| {
+                            txs.into_iter()
+                                .map(|tx| {
+                                    let tx_height = match tx.height {
+                                        none if none <= 0 => None,
+                                        height => {
+                                            txid_to_height.insert(tx.tx_hash, height as u32);
+                                            Some(height as u32)
+                                        }
+                                    };
+                                    (tx.tx_hash, tx_height)
+                                })
+                                .collect()
+                        })
+                        .collect();
+
+                    script_req.satisfy(txids_per_script)?
+                }
+
+                Request::Conftime(conftime_req) => {
+                    // collect up to chunk_size heights to fetch from electrum
+                    let needs_block_height = {
+                        let mut needs_block_height_iter = conftime_req
+                            .request()
+                            .filter_map(|txid| txid_to_height.get(txid).cloned())
+                            .filter(|height| block_times.get(height).is_none());
+                        let mut needs_block_height = HashSet::new();
+
+                        while needs_block_height.len() < chunk_size {
+                            match needs_block_height_iter.next() {
+                                Some(height) => needs_block_height.insert(height),
+                                None => break,
+                            };
+                        }
+                        needs_block_height
+                    };
+
+                    let new_block_headers = self
+                        .client
+                        .batch_block_header(needs_block_height.iter().cloned())?;
+
+                    for (height, header) in needs_block_height.into_iter().zip(new_block_headers) {
+                        block_times.insert(height, header.time);
+                    }
+
+                    let conftimes = conftime_req
+                        .request()
+                        .take(chunk_size)
+                        .map(|txid| {
+                            let confirmation_time = txid_to_height
+                                .get(txid)
+                                .map(|height| {
+                                    let timestamp =
+                                        *block_times.get(height).ok_or_else(electrum_goof)?;
+                                    Result::<_, Error>::Ok(BlockTime {
+                                        height: *height,
+                                        timestamp: timestamp.into(),
+                                    })
+                                })
+                                .transpose()?;
+                            Ok(confirmation_time)
+                        })
+                        .collect::<Result<_, Error>>()?;
+
+                    conftime_req.satisfy(conftimes)?
+                }
+                Request::Tx(tx_req) => {
+                    let needs_full = tx_req.request().take(chunk_size);
+                    tx_cache.save_txs(needs_full.clone())?;
+                    let full_transactions = needs_full
+                        .map(|txid| tx_cache.get(*txid).ok_or_else(electrum_goof))
+                        .collect::<Result<Vec<_>, _>>()?;
+                    let input_txs = full_transactions.iter().flat_map(|tx| {
+                        tx.input
+                            .iter()
+                            .filter(|input| !input.previous_output.is_null())
+                            .map(|input| &input.previous_output.txid)
+                    });
+                    tx_cache.save_txs(input_txs)?;
+
+                    let full_details = full_transactions
+                        .into_iter()
+                        .map(|tx| {
+                            let mut input_index = 0usize;
+                            let prev_outputs = tx
+                                .input
+                                .iter()
+                                .map(|input| {
+                                    if input.previous_output.is_null() {
+                                        return Ok(None);
+                                    }
+                                    let prev_tx = tx_cache
+                                        .get(input.previous_output.txid)
+                                        .ok_or_else(electrum_goof)?;
+                                    let txout = prev_tx
+                                        .output
+                                        .get(input.previous_output.vout as usize)
+                                        .ok_or_else(electrum_goof)?;
+                                    input_index += 1;
+                                    Ok(Some(txout.clone()))
+                                })
+                                .collect::<Result<Vec<_>, Error>>()?;
+                            Ok((prev_outputs, tx))
+                        })
+                        .collect::<Result<Vec<_>, Error>>()?;
+
+                    tx_req.satisfy(full_details)?
+                }
+                Request::Finish(batch_update) => break batch_update,
+            }
+        };
+
+        database.commit_batch(batch_update)?;
+        Ok(())
+    }
+}
+
+struct TxCache<'a, 'b, D> {
+    db: &'a D,
+    client: &'b Client,
+    cache: HashMap<Txid, Transaction>,
+}
+
+impl<'a, 'b, D: Database> TxCache<'a, 'b, D> {
+    fn new(db: &'a D, client: &'b Client) -> Self {
+        TxCache {
+            db,
+            client,
+            cache: HashMap::default(),
+        }
+    }
+    fn save_txs<'c>(&mut self, txids: impl Iterator<Item = &'c Txid>) -> Result<(), Error> {
+        let mut need_fetch = vec![];
+        for txid in txids {
+            if self.cache.get(txid).is_some() {
+                continue;
+            } else if let Some(transaction) = self.db.get_raw_tx(txid)? {
+                self.cache.insert(*txid, transaction);
+            } else {
+                need_fetch.push(txid);
+            }
+        }
+
+        if !need_fetch.is_empty() {
+            let txs = self
+                .client
+                .batch_transaction_get(need_fetch.clone())
+                .map_err(Error::Electrum)?;
+            for (tx, _txid) in txs.into_iter().zip(need_fetch) {
+                debug_assert_eq!(*_txid, tx.txid());
+                self.cache.insert(tx.txid(), tx);
+            }
+        }
+
+        Ok(())
     }
 
-    fn els_batch_transaction_get<'s, I: IntoIterator<Item = &'s Txid> + Clone>(
+    fn get(&self, txid: Txid) -> Option<Transaction> {
-        &self,
+        self.cache.get(&txid).map(Clone::clone)
-        txids: I,
-    ) -> Result<Vec<Transaction>, Error> {
-        self.batch_transaction_get(txids).map_err(Error::Electrum)
-    }
-
-    fn els_batch_block_header<I: IntoIterator<Item = u32> + Clone>(
-        &self,
-        heights: I,
-    ) -> Result<Vec<BlockHeader>, Error> {
-        self.batch_block_header(heights).map_err(Error::Electrum)
     }
 }
 

src/blockchain/esplora/api.rs

@@ -0,0 +1,117 @@
+//! structs from the esplora API
+//!
+//! see: <https://github.com/Blockstream/esplora/blob/master/API.md>
+use crate::BlockTime;
+use bitcoin::{OutPoint, Script, Transaction, TxIn, TxOut, Txid};
+
+#[derive(serde::Deserialize, Clone, Debug)]
+pub struct PrevOut {
+    pub value: u64,
+    pub scriptpubkey: Script,
+}
+
+#[derive(serde::Deserialize, Clone, Debug)]
+pub struct Vin {
+    pub txid: Txid,
+    pub vout: u32,
+    // None if coinbase
+    pub prevout: Option<PrevOut>,
+    pub scriptsig: Script,
+    #[serde(deserialize_with = "deserialize_witness", default)]
+    pub witness: Vec<Vec<u8>>,
+    pub sequence: u32,
+    pub is_coinbase: bool,
+}
+
+#[derive(serde::Deserialize, Clone, Debug)]
+pub struct Vout {
+    pub value: u64,
+    pub scriptpubkey: Script,
+}
+
+#[derive(serde::Deserialize, Clone, Debug)]
+pub struct TxStatus {
+    pub confirmed: bool,
+    pub block_height: Option<u32>,
+    pub block_time: Option<u64>,
+}
+
+#[derive(serde::Deserialize, Clone, Debug)]
+pub struct Tx {
+    pub txid: Txid,
+    pub version: i32,
+    pub locktime: u32,
+    pub vin: Vec<Vin>,
+    pub vout: Vec<Vout>,
+    pub status: TxStatus,
+    pub fee: u64,
+}
+
+impl Tx {
+    pub fn to_tx(&self) -> Transaction {
+        Transaction {
+            version: self.version,
+            lock_time: self.locktime,
+            input: self
+                .vin
+                .iter()
+                .cloned()
+                .map(|vin| TxIn {
+                    previous_output: OutPoint {
+                        txid: vin.txid,
+                        vout: vin.vout,
+                    },
+                    script_sig: vin.scriptsig,
+                    sequence: vin.sequence,
+                    witness: vin.witness,
+                })
+                .collect(),
+            output: self
+                .vout
+                .iter()
+                .cloned()
+                .map(|vout| TxOut {
+                    value: vout.value,
+                    script_pubkey: vout.scriptpubkey,
+                })
+                .collect(),
+        }
+    }
+
+    pub fn confirmation_time(&self) -> Option<BlockTime> {
+        match self.status {
+            TxStatus {
+                confirmed: true,
+                block_height: Some(height),
+                block_time: Some(timestamp),
+            } => Some(BlockTime { timestamp, height }),
+            _ => None,
+        }
+    }
+
+    pub fn previous_outputs(&self) -> Vec<Option<TxOut>> {
+        self.vin
+            .iter()
+            .cloned()
+            .map(|vin| {
+                vin.prevout.map(|po| TxOut {
+                    script_pubkey: po.scriptpubkey,
+                    value: po.value,
+                })
+            })
+            .collect()
+    }
+}
+
+fn deserialize_witness<'de, D>(d: D) -> Result<Vec<Vec<u8>>, D::Error>
+where
+    D: serde::de::Deserializer<'de>,
+{
+    use crate::serde::Deserialize;
+    use bitcoin::hashes::hex::FromHex;
+    let list = Vec::<String>::deserialize(d)?;
+    list.into_iter()
+        .map(|hex_str| Vec::<u8>::from_hex(&hex_str))
+        .collect::<Result<Vec<Vec<u8>>, _>>()
+        .map_err(serde::de::Error::custom)
+}

src/blockchain/esplora/mod.rs

@@ -21,8 +21,6 @@ use std::collections::HashMap;
 use std::fmt;
 use std::io;
 
-use serde::Deserialize;
-
 use bitcoin::consensus;
 use bitcoin::{BlockHash, Txid};
 
@@ -41,33 +39,24 @@ mod ureq;
 #[cfg(feature = "ureq")]
 pub use self::ureq::*;
 
+mod api;
+
 fn into_fee_rate(target: usize, estimates: HashMap<String, f64>) -> Result<FeeRate, Error> {
-    let fee_val = estimates
+    let fee_val = {
-        .into_iter()
+        let mut pairs = estimates
-        .map(|(k, v)| Ok::<_, std::num::ParseIntError>((k.parse::<usize>()?, v)))
+            .into_iter()
-        .collect::<Result<Vec<_>, _>>()
+            .filter_map(|(k, v)| Some((k.parse::<usize>().ok()?, v)))
-        .map_err(|e| Error::Generic(e.to_string()))?
+            .collect::<Vec<_>>();
-        .into_iter()
+        pairs.sort_unstable_by_key(|(k, _)| std::cmp::Reverse(*k));
-        .take_while(|(k, _)| k <= &target)
+        pairs
-        .map(|(_, v)| v)
+            .into_iter()
-        .last()
+            .find(|(k, _)| k <= &target)
-        .unwrap_or(1.0);
+            .map(|(_, v)| v)
-
+            .unwrap_or(1.0)
+    };
     Ok(FeeRate::from_sat_per_vb(fee_val as f32))
 }
 
-/// Data type used when fetching transaction history from Esplora.
-#[derive(Deserialize)]
-pub struct EsploraGetHistory {
-    txid: Txid,
-    status: EsploraGetHistoryStatus,
-}
-
-#[derive(Deserialize)]
-struct EsploraGetHistoryStatus {
-    block_height: Option<usize>,
-}
-
 /// Errors that can happen during a sync with [`EsploraBlockchain`]
 #[derive(Debug)]
 pub enum EsploraError {
@@ -107,10 +96,50 @@ impl fmt::Display for EsploraError {
     }
 }
 
+/// Configuration for an [`EsploraBlockchain`]
+#[derive(Debug, serde::Deserialize, serde::Serialize, Clone, PartialEq)]
+pub struct EsploraBlockchainConfig {
+    /// Base URL of the esplora service
+    ///
+    /// eg. `https://blockstream.info/api/`
+    pub base_url: String,
+    /// Optional URL of the proxy to use to make requests to the Esplora server
+    ///
+    /// The string should be formatted as: `<protocol>://<user>:<password>@host:<port>`.
+    ///
+    /// Note that the format of this value and the supported protocols change slightly between the
+    /// sync version of esplora (using `ureq`) and the async version (using `reqwest`). For more
+    /// details check with the documentation of the two crates. Both of them are compiled with
+    /// the `socks` feature enabled.
+    ///
+    /// The proxy is ignored when targeting `wasm32`.
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub proxy: Option<String>,
+    /// Number of parallel requests sent to the esplora service (default: 4)
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub concurrency: Option<u8>,
+    /// Stop searching addresses for transactions after finding an unused gap of this length.
+    pub stop_gap: usize,
+    /// Socket timeout.
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub timeout: Option<u64>,
+}
+
+impl EsploraBlockchainConfig {
+    /// create a config with default values given the base url and stop gap
+    pub fn new(base_url: String, stop_gap: usize) -> Self {
+        Self {
+            base_url,
+            proxy: None,
+            timeout: None,
+            stop_gap,
+            concurrency: None,
+        }
+    }
+}
+
 impl std::error::Error for EsploraError {}
 
-#[cfg(feature = "ureq")]
-impl_error!(::ureq::Error, Ureq, EsploraError);
 #[cfg(feature = "ureq")]
 impl_error!(::ureq::Transport, UreqTransport, EsploraError);
 #[cfg(feature = "reqwest")]
@@ -127,3 +156,57 @@ crate::bdk_blockchain_tests! {
         EsploraBlockchain::new(&format!("http://{}",test_client.electrsd.esplora_url.as_ref().unwrap()), 20)
     }
 }
+
+const DEFAULT_CONCURRENT_REQUESTS: u8 = 4;
+
+#[cfg(test)]
+mod test {
+    use super::*;
+
+    #[test]
+    fn feerate_parsing() {
+        let esplora_fees = serde_json::from_str::<HashMap<String, f64>>(
+            r#"{
+  "25": 1.015,
+  "5": 2.3280000000000003,
+  "12": 2.0109999999999997,
+  "15": 1.018,
+  "17": 1.018,
+  "11": 2.0109999999999997,
+  "3": 3.01,
+  "2": 4.9830000000000005,
+  "6": 2.2359999999999998,
+  "21": 1.018,
+  "13": 1.081,
+  "7": 2.2359999999999998,
+  "8": 2.2359999999999998,
+  "16": 1.018,
+  "20": 1.018,
+  "22": 1.017,
+  "23": 1.017,
+  "504": 1,
+  "9": 2.2359999999999998,
+  "14": 1.018,
+  "10": 2.0109999999999997,
+  "24": 1.017,
+  "1008": 1,
+  "1": 4.9830000000000005,
+  "4": 2.3280000000000003,
+  "19": 1.018,
+  "144": 1,
+  "18": 1.018
+}
+"#,
+        )
+        .unwrap();
+        assert_eq!(
+            into_fee_rate(6, esplora_fees.clone()).unwrap(),
+            FeeRate::from_sat_per_vb(2.236)
+        );
+        assert_eq!(
+            into_fee_rate(26, esplora_fees).unwrap(),
+            FeeRate::from_sat_per_vb(1.015),
+            "should inherit from value for 25"
+        );
+    }
+}

src/blockchain/esplora/reqwest.rs

@@ -21,20 +21,16 @@ use bitcoin::{BlockHeader, Script, Transaction, Txid};
 #[allow(unused_imports)]
 use log::{debug, error, info, trace};
 
-use futures::stream::{self, FuturesOrdered, StreamExt, TryStreamExt};
-
 use ::reqwest::{Client, StatusCode};
+use futures::stream::{FuturesOrdered, TryStreamExt};
 
-use crate::blockchain::esplora::{EsploraError, EsploraGetHistory};
+use super::api::Tx;
-use crate::blockchain::utils::{ElectrumLikeSync, ElsGetHistoryRes};
+use crate::blockchain::esplora::EsploraError;
 use crate::blockchain::*;
 use crate::database::BatchDatabase;
 use crate::error::Error;
-use crate::wallet::utils::ChunksIterator;
 use crate::FeeRate;
 
-const DEFAULT_CONCURRENT_REQUESTS: u8 = 4;
-
 #[derive(Debug)]
 struct UrlClient {
     url: String,
@@ -70,7 +66,7 @@ impl EsploraBlockchain {
             url_client: UrlClient {
                 url: base_url.to_string(),
                 client: Client::new(),
-                concurrency: DEFAULT_CONCURRENT_REQUESTS,
+                concurrency: super::DEFAULT_CONCURRENT_REQUESTS,
             },
             stop_gap,
         }
@@ -95,39 +91,124 @@ impl Blockchain for EsploraBlockchain {
         .collect()
     }
 
-    fn setup<D: BatchDatabase, P: Progress>(
-        &self,
-        database: &mut D,
-        progress_update: P,
-    ) -> Result<(), Error> {
-        maybe_await!(self
-            .url_client
-            .electrum_like_setup(self.stop_gap, database, progress_update))
-    }
-
-    fn get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error> {
-        Ok(await_or_block!(self.url_client._get_tx(txid))?)
-    }
-
     fn broadcast(&self, tx: &Transaction) -> Result<(), Error> {
         Ok(await_or_block!(self.url_client._broadcast(tx))?)
     }
 
-    fn get_height(&self) -> Result<u32, Error> {
-        Ok(await_or_block!(self.url_client._get_height())?)
-    }
-
     fn estimate_fee(&self, target: usize) -> Result<FeeRate, Error> {
         let estimates = await_or_block!(self.url_client._get_fee_estimates())?;
         super::into_fee_rate(target, estimates)
     }
 }
 
-impl UrlClient {
+#[maybe_async]
-    fn script_to_scripthash(script: &Script) -> String {
+impl GetHeight for EsploraBlockchain {
-        sha256::Hash::hash(script.as_bytes()).into_inner().to_hex()
+    fn get_height(&self) -> Result<u32, Error> {
+        Ok(await_or_block!(self.url_client._get_height())?)
     }
+}
 
+#[maybe_async]
+impl GetTx for EsploraBlockchain {
+    fn get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error> {
+        Ok(await_or_block!(self.url_client._get_tx(txid))?)
+    }
+}
+
+#[maybe_async]
+impl WalletSync for EsploraBlockchain {
+    fn wallet_setup<D: BatchDatabase>(
+        &self,
+        database: &mut D,
+        _progress_update: Box<dyn Progress>,
+    ) -> Result<(), Error> {
+        use crate::blockchain::script_sync::Request;
+        let mut request = script_sync::start(database, self.stop_gap)?;
+        let mut tx_index: HashMap<Txid, Tx> = HashMap::new();
+
+        let batch_update = loop {
+            request = match request {
+                Request::Script(script_req) => {
+                    let futures: FuturesOrdered<_> = script_req
+                        .request()
+                        .take(self.url_client.concurrency as usize)
+                        .map(|script| async move {
+                            let mut related_txs: Vec<Tx> =
+                                self.url_client._scripthash_txs(script, None).await?;
+
+                            let n_confirmed =
+                                related_txs.iter().filter(|tx| tx.status.confirmed).count();
+                            // esplora pages on 25 confirmed transactions. If there's 25 or more we
+                            // keep requesting to see if there's more.
+                            if n_confirmed >= 25 {
+                                loop {
+                                    let new_related_txs: Vec<Tx> = self
+                                        .url_client
+                                        ._scripthash_txs(
+                                            script,
+                                            Some(related_txs.last().unwrap().txid),
+                                        )
+                                        .await?;
+                                    let n = new_related_txs.len();
+                                    related_txs.extend(new_related_txs);
+                                    // we've reached the end
+                                    if n < 25 {
+                                        break;
+                                    }
+                                }
+                            }
+                            Result::<_, Error>::Ok(related_txs)
+                        })
+                        .collect();
+                    let txs_per_script: Vec<Vec<Tx>> = await_or_block!(futures.try_collect())?;
+                    let mut satisfaction = vec![];
+
+                    for txs in txs_per_script {
+                        satisfaction.push(
+                            txs.iter()
+                                .map(|tx| (tx.txid, tx.status.block_height))
+                                .collect(),
+                        );
+                        for tx in txs {
+                            tx_index.insert(tx.txid, tx);
+                        }
+                    }
+
+                    script_req.satisfy(satisfaction)?
+                }
+                Request::Conftime(conftime_req) => {
+                    let conftimes = conftime_req
+                        .request()
+                        .map(|txid| {
+                            tx_index
+                                .get(txid)
+                                .expect("must be in index")
+                                .confirmation_time()
+                        })
+                        .collect();
+                    conftime_req.satisfy(conftimes)?
+                }
+                Request::Tx(tx_req) => {
+                    let full_txs = tx_req
+                        .request()
+                        .map(|txid| {
+                            let tx = tx_index.get(txid).expect("must be in index");
+                            Ok((tx.previous_outputs(), tx.to_tx()))
+                        })
+                        .collect::<Result<_, Error>>()?;
+                    tx_req.satisfy(full_txs)?
+                }
+                Request::Finish(batch_update) => break batch_update,
+            }
+        };
+
+        database.commit_batch(batch_update)?;
+
+        Ok(())
+    }
+}
+
+impl UrlClient {
     async fn _get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, EsploraError> {
         let resp = self
             .client
@@ -196,71 +277,27 @@ impl UrlClient {
         Ok(req.error_for_status()?.text().await?.parse()?)
     }
 
-    async fn _script_get_history(
+    async fn _scripthash_txs(
         &self,
         script: &Script,
-    ) -> Result<Vec<ElsGetHistoryRes>, EsploraError> {
+        last_seen: Option<Txid>,
-        let mut result = Vec::new();
+    ) -> Result<Vec<Tx>, EsploraError> {
-        let scripthash = Self::script_to_scripthash(script);
+        let script_hash = sha256::Hash::hash(script.as_bytes()).into_inner().to_hex();
-
+        let url = match last_seen {
-        // Add the unconfirmed transactions first
+            Some(last_seen) => format!(
-        result.extend(
+                "{}/scripthash/{}/txs/chain/{}",
-            self.client
+                self.url, script_hash, last_seen
-                .get(&format!(
+            ),
-                    "{}/scripthash/{}/txs/mempool",
+            None => format!("{}/scripthash/{}/txs", self.url, script_hash),
-                    self.url, scripthash
+        };
-                ))
+        Ok(self
-                .send()
+            .client
-                .await?
+            .get(url)
-                .error_for_status()?
+            .send()
-                .json::<Vec<EsploraGetHistory>>()
+            .await?
-                .await?
+            .error_for_status()?
-                .into_iter()
+            .json::<Vec<Tx>>()
-                .map(|x| ElsGetHistoryRes {
+            .await?)
-                    tx_hash: x.txid,
-                    height: x.status.block_height.unwrap_or(0) as i32,
-                }),
-        );
-
-        debug!(
-            "Found {} mempool txs for {} - {:?}",
-            result.len(),
-            scripthash,
-            script
-        );
-
-        // Then go through all the pages of confirmed transactions
-        let mut last_txid = String::new();
-        loop {
-            let response = self
-                .client
-                .get(&format!(
-                    "{}/scripthash/{}/txs/chain/{}",
-                    self.url, scripthash, last_txid
-                ))
-                .send()
-                .await?
-                .error_for_status()?
-                .json::<Vec<EsploraGetHistory>>()
-                .await?;
-            let len = response.len();
-            if let Some(elem) = response.last() {
-                last_txid = elem.txid.to_hex();
-            }
-
-            debug!("... adding {} confirmed transactions", len);
-
-            result.extend(response.into_iter().map(|x| ElsGetHistoryRes {
-                tx_hash: x.txid,
-                height: x.status.block_height.unwrap_or(0) as i32,
-            }));
-
-            if len < 25 {
-                break;
-            }
-        }
-
-        Ok(result)
     }
 
     async fn _get_fee_estimates(&self) -> Result<HashMap<String, f64>, EsploraError> {
@@ -275,83 +312,8 @@ impl UrlClient {
     }
 }
 
-#[maybe_async]
-impl ElectrumLikeSync for UrlClient {
-    fn els_batch_script_get_history<'s, I: IntoIterator<Item = &'s Script>>(
-        &self,
-        scripts: I,
-    ) -> Result<Vec<Vec<ElsGetHistoryRes>>, Error> {
-        let mut results = vec![];
-        for chunk in ChunksIterator::new(scripts.into_iter(), self.concurrency as usize) {
-            let mut futs = FuturesOrdered::new();
-            for script in chunk {
-                futs.push(self._script_get_history(script));
-            }
-            let partial_results: Vec<Vec<ElsGetHistoryRes>> = await_or_block!(futs.try_collect())?;
-            results.extend(partial_results);
-        }
-        Ok(await_or_block!(stream::iter(results).collect()))
-    }
-
-    fn els_batch_transaction_get<'s, I: IntoIterator<Item = &'s Txid>>(
-        &self,
-        txids: I,
-    ) -> Result<Vec<Transaction>, Error> {
-        let mut results = vec![];
-        for chunk in ChunksIterator::new(txids.into_iter(), self.concurrency as usize) {
-            let mut futs = FuturesOrdered::new();
-            for txid in chunk {
-                futs.push(self._get_tx_no_opt(txid));
-            }
-            let partial_results: Vec<Transaction> = await_or_block!(futs.try_collect())?;
-            results.extend(partial_results);
-        }
-        Ok(await_or_block!(stream::iter(results).collect()))
-    }
-
-    fn els_batch_block_header<I: IntoIterator<Item = u32>>(
-        &self,
-        heights: I,
-    ) -> Result<Vec<BlockHeader>, Error> {
-        let mut results = vec![];
-        for chunk in ChunksIterator::new(heights.into_iter(), self.concurrency as usize) {
-            let mut futs = FuturesOrdered::new();
-            for height in chunk {
-                futs.push(self._get_header(height));
-            }
-            let partial_results: Vec<BlockHeader> = await_or_block!(futs.try_collect())?;
-            results.extend(partial_results);
-        }
-        Ok(await_or_block!(stream::iter(results).collect()))
-    }
-}
-
-/// Configuration for an [`EsploraBlockchain`]
-#[derive(Debug, serde::Deserialize, serde::Serialize, Clone, PartialEq)]
-pub struct EsploraBlockchainConfig {
-    /// Base URL of the esplora service
-    ///
-    /// eg. `https://blockstream.info/api/`
-    pub base_url: String,
-    /// Optional URL of the proxy to use to make requests to the Esplora server
-    ///
-    /// The string should be formatted as: `<protocol>://<user>:<password>@host:<port>`.
-    ///
-    /// Note that the format of this value and the supported protocols change slightly between the
-    /// sync version of esplora (using `ureq`) and the async version (using `reqwest`). For more
-    /// details check with the documentation of the two crates. Both of them are compiled with
-    /// the `socks` feature enabled.
-    ///
-    /// The proxy is ignored when targeting `wasm32`.
-    pub proxy: Option<String>,
-    /// Number of parallel requests sent to the esplora service (default: 4)
-    pub concurrency: Option<u8>,
-    /// Stop searching addresses for transactions after finding an unused gap of this length.
-    pub stop_gap: usize,
-}
-
 impl ConfigurableBlockchain for EsploraBlockchain {
-    type Config = EsploraBlockchainConfig;
+    type Config = super::EsploraBlockchainConfig;
 
     fn from_config(config: &Self::Config) -> Result<Self, Error> {
         let map_e = |e: reqwest::Error| Error::Esplora(Box::new(e.into()));
@@ -360,13 +322,19 @@ impl ConfigurableBlockchain for EsploraBlockchain {
         if let Some(concurrency) = config.concurrency {
             blockchain.url_client.concurrency = concurrency;
         }
+        let mut builder = Client::builder();
         #[cfg(not(target_arch = "wasm32"))]
         if let Some(proxy) = &config.proxy {
-            blockchain.url_client.client = Client::builder()
+            builder = builder.proxy(reqwest::Proxy::all(proxy).map_err(map_e)?);
-                .proxy(reqwest::Proxy::all(proxy).map_err(map_e)?)
-                .build()
-                .map_err(map_e)?;
         }
+
+        #[cfg(not(target_arch = "wasm32"))]
+        if let Some(timeout) = config.timeout {
+            builder = builder.timeout(core::time::Duration::from_secs(timeout));
+        }
+
+        blockchain.url_client.client = builder.build().map_err(map_e)?;
+
         Ok(blockchain)
     }
 }

src/blockchain/esplora/ureq.rs

@@ -26,14 +26,14 @@ use bitcoin::hashes::hex::{FromHex, ToHex};
 use bitcoin::hashes::{sha256, Hash};
 use bitcoin::{BlockHeader, Script, Transaction, Txid};
 
-use crate::blockchain::esplora::{EsploraError, EsploraGetHistory};
+use super::api::Tx;
-use crate::blockchain::utils::{ElectrumLikeSync, ElsGetHistoryRes};
+use crate::blockchain::esplora::EsploraError;
 use crate::blockchain::*;
 use crate::database::BatchDatabase;
 use crate::error::Error;
 use crate::FeeRate;
 
-#[derive(Debug)]
+#[derive(Debug, Clone)]
 struct UrlClient {
     url: String,
     agent: Agent,
@@ -47,15 +47,7 @@ struct UrlClient {
 pub struct EsploraBlockchain {
     url_client: UrlClient,
     stop_gap: usize,
-}
+    concurrency: u8,
-
-impl std::convert::From<UrlClient> for EsploraBlockchain {
-    fn from(url_client: UrlClient) -> Self {
-        EsploraBlockchain {
-            url_client,
-            stop_gap: 20,
-        }
-    }
 }
 
 impl EsploraBlockchain {
@@ -66,6 +58,7 @@ impl EsploraBlockchain {
                 url: base_url.to_string(),
                 agent: Agent::new(),
             },
+            concurrency: super::DEFAULT_CONCURRENT_REQUESTS,
             stop_gap,
         }
     }
@@ -75,6 +68,12 @@ impl EsploraBlockchain {
         self.url_client.agent = agent;
         self
     }
+
+    /// Set the number of parallel requests the client can make.
+    pub fn with_concurrency(mut self, concurrency: u8) -> Self {
+        self.concurrency = concurrency;
+        self
+    }
 }
 
 impl Blockchain for EsploraBlockchain {
@@ -88,39 +87,127 @@ impl Blockchain for EsploraBlockchain {
         .collect()
     }
 
-    fn setup<D: BatchDatabase, P: Progress>(
-        &self,
-        database: &mut D,
-        progress_update: P,
-    ) -> Result<(), Error> {
-        self.url_client
-            .electrum_like_setup(self.stop_gap, database, progress_update)
-    }
-
-    fn get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error> {
-        Ok(self.url_client._get_tx(txid)?)
-    }
-
     fn broadcast(&self, tx: &Transaction) -> Result<(), Error> {
         let _txid = self.url_client._broadcast(tx)?;
         Ok(())
     }
 
-    fn get_height(&self) -> Result<u32, Error> {
-        Ok(self.url_client._get_height()?)
-    }
-
     fn estimate_fee(&self, target: usize) -> Result<FeeRate, Error> {
         let estimates = self.url_client._get_fee_estimates()?;
         super::into_fee_rate(target, estimates)
     }
 }
 
-impl UrlClient {
+impl GetHeight for EsploraBlockchain {
-    fn script_to_scripthash(script: &Script) -> String {
+    fn get_height(&self) -> Result<u32, Error> {
-        sha256::Hash::hash(script.as_bytes()).into_inner().to_hex()
+        Ok(self.url_client._get_height()?)
     }
+}
 
+impl GetTx for EsploraBlockchain {
+    fn get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error> {
+        Ok(self.url_client._get_tx(txid)?)
+    }
+}
+
+impl WalletSync for EsploraBlockchain {
+    fn wallet_setup<D: BatchDatabase>(
+        &self,
+        database: &mut D,
+        _progress_update: Box<dyn Progress>,
+    ) -> Result<(), Error> {
+        use crate::blockchain::script_sync::Request;
+        let mut request = script_sync::start(database, self.stop_gap)?;
+        let mut tx_index: HashMap<Txid, Tx> = HashMap::new();
+        let batch_update = loop {
+            request = match request {
+                Request::Script(script_req) => {
+                    let scripts = script_req
+                        .request()
+                        .take(self.concurrency as usize)
+                        .cloned();
+
+                    let mut handles = vec![];
+                    for script in scripts {
+                        let client = self.url_client.clone();
+                        // make each request in its own thread.
+                        handles.push(std::thread::spawn(move || {
+                            let mut related_txs: Vec<Tx> = client._scripthash_txs(&script, None)?;
+
+                            let n_confirmed =
+                                related_txs.iter().filter(|tx| tx.status.confirmed).count();
+                            // esplora pages on 25 confirmed transactions. If there's 25 or more we
+                            // keep requesting to see if there's more.
+                            if n_confirmed >= 25 {
+                                loop {
+                                    let new_related_txs: Vec<Tx> = client._scripthash_txs(
+                                        &script,
+                                        Some(related_txs.last().unwrap().txid),
+                                    )?;
+                                    let n = new_related_txs.len();
+                                    related_txs.extend(new_related_txs);
+                                    // we've reached the end
+                                    if n < 25 {
+                                        break;
+                                    }
+                                }
+                            }
+                            Result::<_, Error>::Ok(related_txs)
+                        }));
+                    }
+
+                    let txs_per_script: Vec<Vec<Tx>> = handles
+                        .into_iter()
+                        .map(|handle| handle.join().unwrap())
+                        .collect::<Result<_, _>>()?;
+                    let mut satisfaction = vec![];
+
+                    for txs in txs_per_script {
+                        satisfaction.push(
+                            txs.iter()
+                                .map(|tx| (tx.txid, tx.status.block_height))
+                                .collect(),
+                        );
+                        for tx in txs {
+                            tx_index.insert(tx.txid, tx);
+                        }
+                    }
+
+                    script_req.satisfy(satisfaction)?
+                }
+                Request::Conftime(conftime_req) => {
+                    let conftimes = conftime_req
+                        .request()
+                        .map(|txid| {
+                            tx_index
+                                .get(txid)
+                                .expect("must be in index")
+                                .confirmation_time()
+                        })
+                        .collect();
+                    conftime_req.satisfy(conftimes)?
+                }
+                Request::Tx(tx_req) => {
+                    let full_txs = tx_req
+                        .request()
+                        .map(|txid| {
+                            let tx = tx_index.get(txid).expect("must be in index");
+                            Ok((tx.previous_outputs(), tx.to_tx()))
+                        })
+                        .collect::<Result<_, Error>>()?;
+                    tx_req.satisfy(full_txs)?
+                }
+                Request::Finish(batch_update) => break batch_update,
+            }
+        };
+
+        database.commit_batch(batch_update)?;
+
+        Ok(())
+    }
+}
+
+impl UrlClient {
     fn _get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, EsploraError> {
         let resp = self
             .agent
@@ -200,81 +287,6 @@ impl UrlClient {
         }
     }
 
-    fn _script_get_history(&self, script: &Script) -> Result<Vec<ElsGetHistoryRes>, EsploraError> {
-        let mut result = Vec::new();
-        let scripthash = Self::script_to_scripthash(script);
-
-        // Add the unconfirmed transactions first
-
-        let resp = self
-            .agent
-            .get(&format!(
-                "{}/scripthash/{}/txs/mempool",
-                self.url, scripthash
-            ))
-            .call();
-
-        let v = match resp {
-            Ok(resp) => {
-                let v: Vec<EsploraGetHistory> = resp.into_json()?;
-                Ok(v)
-            }
-            Err(ureq::Error::Status(code, _)) => Err(EsploraError::HttpResponse(code)),
-            Err(e) => Err(EsploraError::Ureq(e)),
-        }?;
-
-        result.extend(v.into_iter().map(|x| ElsGetHistoryRes {
-            tx_hash: x.txid,
-            height: x.status.block_height.unwrap_or(0) as i32,
-        }));
-
-        debug!(
-            "Found {} mempool txs for {} - {:?}",
-            result.len(),
-            scripthash,
-            script
-        );
-
-        // Then go through all the pages of confirmed transactions
-        let mut last_txid = String::new();
-        loop {
-            let resp = self
-                .agent
-                .get(&format!(
-                    "{}/scripthash/{}/txs/chain/{}",
-                    self.url, scripthash, last_txid
-                ))
-                .call();
-
-            let v = match resp {
-                Ok(resp) => {
-                    let v: Vec<EsploraGetHistory> = resp.into_json()?;
-                    Ok(v)
-                }
-                Err(ureq::Error::Status(code, _)) => Err(EsploraError::HttpResponse(code)),
-                Err(e) => Err(EsploraError::Ureq(e)),
-            }?;
-
-            let len = v.len();
-            if let Some(elem) = v.last() {
-                last_txid = elem.txid.to_hex();
-            }
-
-            debug!("... adding {} confirmed transactions", len);
-
-            result.extend(v.into_iter().map(|x| ElsGetHistoryRes {
-                tx_hash: x.txid,
-                height: x.status.block_height.unwrap_or(0) as i32,
-            }));
-
-            if len < 25 {
-                break;
-            }
-        }
-
-        Ok(result)
-    }
-
     fn _get_fee_estimates(&self) -> Result<HashMap<String, f64>, EsploraError> {
         let resp = self
             .agent
@@ -292,6 +304,22 @@ impl UrlClient {
 
         Ok(map)
     }
+
+    fn _scripthash_txs(
+        &self,
+        script: &Script,
+        last_seen: Option<Txid>,
+    ) -> Result<Vec<Tx>, EsploraError> {
+        let script_hash = sha256::Hash::hash(script.as_bytes()).into_inner().to_hex();
+        let url = match last_seen {
+            Some(last_seen) => format!(
+                "{}/scripthash/{}/txs/chain/{}",
+                self.url, script_hash, last_seen
+            ),
+            None => format!("{}/scripthash/{}/txs", self.url, script_hash),
+        };
+        Ok(self.agent.get(&url).call()?.into_json()?)
+    }
 }
 
 fn is_status_not_found(status: u16) -> bool {
@@ -315,84 +343,37 @@ fn into_bytes(resp: Response) -> Result<Vec<u8>, io::Error> {
     Ok(buf)
 }
 
-impl ElectrumLikeSync for UrlClient {
-    fn els_batch_script_get_history<'s, I: IntoIterator<Item = &'s Script>>(
-        &self,
-        scripts: I,
-    ) -> Result<Vec<Vec<ElsGetHistoryRes>>, Error> {
-        let mut results = vec![];
-        for script in scripts.into_iter() {
-            let v = self._script_get_history(script)?;
-            results.push(v);
-        }
-        Ok(results)
-    }
-
-    fn els_batch_transaction_get<'s, I: IntoIterator<Item = &'s Txid>>(
-        &self,
-        txids: I,
-    ) -> Result<Vec<Transaction>, Error> {
-        let mut results = vec![];
-        for txid in txids.into_iter() {
-            let tx = self._get_tx_no_opt(txid)?;
-            results.push(tx);
-        }
-        Ok(results)
-    }
-
-    fn els_batch_block_header<I: IntoIterator<Item = u32>>(
-        &self,
-        heights: I,
-    ) -> Result<Vec<BlockHeader>, Error> {
-        let mut results = vec![];
-        for height in heights.into_iter() {
-            let header = self._get_header(height)?;
-            results.push(header);
-        }
-        Ok(results)
-    }
-}
-
-/// Configuration for an [`EsploraBlockchain`]
-#[derive(Debug, serde::Deserialize, serde::Serialize, Clone, PartialEq)]
-pub struct EsploraBlockchainConfig {
-    /// Base URL of the esplora service eg. `https://blockstream.info/api/`
-    pub base_url: String,
-    /// Optional URL of the proxy to use to make requests to the Esplora server
-    ///
-    /// The string should be formatted as: `<protocol>://<user>:<password>@host:<port>`.
-    ///
-    /// Note that the format of this value and the supported protocols change slightly between the
-    /// sync version of esplora (using `ureq`) and the async version (using `reqwest`). For more
-    /// details check with the documentation of the two crates. Both of them are compiled with
-    /// the `socks` feature enabled.
-    ///
-    /// The proxy is ignored when targeting `wasm32`.
-    pub proxy: Option<String>,
-    /// Socket read timeout.
-    pub timeout_read: u64,
-    /// Socket write timeout.
-    pub timeout_write: u64,
-    /// Stop searching addresses for transactions after finding an unused gap of this length.
-    pub stop_gap: usize,
-}
-
 impl ConfigurableBlockchain for EsploraBlockchain {
-    type Config = EsploraBlockchainConfig;
+    type Config = super::EsploraBlockchainConfig;
 
     fn from_config(config: &Self::Config) -> Result<Self, Error> {
-        let mut agent_builder = ureq::AgentBuilder::new()
+        let mut agent_builder = ureq::AgentBuilder::new();
-            .timeout_read(Duration::from_secs(config.timeout_read))
+
-            .timeout_write(Duration::from_secs(config.timeout_write));
+        if let Some(timeout) = config.timeout {
+            agent_builder = agent_builder.timeout(Duration::from_secs(timeout));
+        }
 
         if let Some(proxy) = &config.proxy {
             agent_builder = agent_builder
                 .proxy(Proxy::new(proxy).map_err(|e| Error::Esplora(Box::new(e.into())))?);
         }
 
-        Ok(
+        let mut blockchain = EsploraBlockchain::new(config.base_url.as_str(), config.stop_gap)
-            EsploraBlockchain::new(config.base_url.as_str(), config.stop_gap)
+            .with_agent(agent_builder.build());
-                .with_agent(agent_builder.build()),
+
-        )
+        if let Some(concurrency) = config.concurrency {
+            blockchain = blockchain.with_concurrency(concurrency);
+        }
+
+        Ok(blockchain)
+    }
+}
+
+impl From<ureq::Error> for EsploraError {
+    fn from(e: ureq::Error) -> Self {
+        match e {
+            ureq::Error::Status(code, _) => EsploraError::HttpResponse(code),
+            e => EsploraError::Ureq(e),
+        }
     }
 }

src/blockchain/mod.rs

@@ -27,9 +27,6 @@ use crate::database::BatchDatabase;
 use crate::error::Error;
 use crate::FeeRate;
 
-#[cfg(any(feature = "electrum", feature = "esplora"))]
-pub(crate) mod utils;
-
 #[cfg(any(
     feature = "electrum",
     feature = "esplora",
@@ -37,6 +34,8 @@ pub(crate) mod utils;
     feature = "rpc"
 ))]
 pub mod any;
+mod script_sync;
+
 #[cfg(any(
     feature = "electrum",
     feature = "esplora",
@@ -87,28 +86,50 @@ pub enum Capability {
 
 /// Trait that defines the actions that must be supported by a blockchain backend
 #[maybe_async]
-pub trait Blockchain {
+pub trait Blockchain: WalletSync + GetHeight + GetTx {
     /// Return the set of [`Capability`] supported by this backend
     fn get_capabilities(&self) -> HashSet<Capability>;
+    /// Broadcast a transaction
+    fn broadcast(&self, tx: &Transaction) -> Result<(), Error>;
+    /// Estimate the fee rate required to confirm a transaction in a given `target` of blocks
+    fn estimate_fee(&self, target: usize) -> Result<FeeRate, Error>;
+}
 
+/// Trait for getting the current height of the blockchain.
+#[maybe_async]
+pub trait GetHeight {
+    /// Return the current height
+    fn get_height(&self) -> Result<u32, Error>;
+}
+
+#[maybe_async]
+/// Trait for getting a transaction by txid
+pub trait GetTx {
+    /// Fetch a transaction given its txid
+    fn get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error>;
+}
+
+/// Trait for blockchains that can sync by updating the database directly.
+#[maybe_async]
+pub trait WalletSync {
     /// Setup the backend and populate the internal database for the first time
     ///
-    /// This method is the equivalent of [`Blockchain::sync`], but it's guaranteed to only be
+    /// This method is the equivalent of [`Self::wallet_sync`], but it's guaranteed to only be
     /// called once, at the first [`Wallet::sync`](crate::wallet::Wallet::sync).
     ///
     /// The rationale behind the distinction between `sync` and `setup` is that some custom backends
     /// might need to perform specific actions only the first time they are synced.
     ///
     /// For types that do not have that distinction, only this method can be implemented, since
-    /// [`Blockchain::sync`] defaults to calling this internally if not overridden.
+    /// [`WalletSync::wallet_sync`] defaults to calling this internally if not overridden.
-    fn setup<D: BatchDatabase, P: 'static + Progress>(
+    /// Populate the internal database with transactions and UTXOs
+    fn wallet_setup<D: BatchDatabase>(
         &self,
         database: &mut D,
-        progress_update: P,
+        progress_update: Box<dyn Progress>,
     ) -> Result<(), Error>;
-    /// Populate the internal database with transactions and UTXOs
+
-    ///
+    /// If not overridden, it defaults to calling [`Self::wallet_setup`] internally.
-    /// If not overridden, it defaults to calling [`Blockchain::setup`] internally.
     ///
     /// This method should implement the logic required to iterate over the list of the wallet's
     /// script_pubkeys using [`Database::iter_script_pubkeys`] and look for relevant transactions
@@ -125,23 +146,13 @@ pub trait Blockchain {
     /// [`BatchOperations::set_tx`]: crate::database::BatchOperations::set_tx
     /// [`BatchOperations::set_utxo`]: crate::database::BatchOperations::set_utxo
     /// [`BatchOperations::del_utxo`]: crate::database::BatchOperations::del_utxo
-    fn sync<D: BatchDatabase, P: 'static + Progress>(
+    fn wallet_sync<D: BatchDatabase>(
         &self,
         database: &mut D,
-        progress_update: P,
+        progress_update: Box<dyn Progress>,
     ) -> Result<(), Error> {
-        maybe_await!(self.setup(database, progress_update))
+        maybe_await!(self.wallet_setup(database, progress_update))
     }
-
-    /// Fetch a transaction from the blockchain given its txid
-    fn get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error>;
-    /// Broadcast a transaction
-    fn broadcast(&self, tx: &Transaction) -> Result<(), Error>;
-
-    /// Return the current height
-    fn get_height(&self) -> Result<u32, Error>;
-    /// Estimate the fee rate required to confirm a transaction in a given `target` of blocks
-    fn estimate_fee(&self, target: usize) -> Result<FeeRate, Error>;
 }
 
 /// Trait for [`Blockchain`] types that can be created given a configuration
@@ -156,9 +167,9 @@ pub trait ConfigurableBlockchain: Blockchain + Sized {
 /// Data sent with a progress update over a [`channel`]
 pub type ProgressData = (f32, Option<String>);
 
-/// Trait for types that can receive and process progress updates during [`Blockchain::sync`] and
+/// Trait for types that can receive and process progress updates during [`WalletSync::wallet_sync`] and
-/// [`Blockchain::setup`]
+/// [`WalletSync::wallet_setup`]
-pub trait Progress: Send {
+pub trait Progress: Send + 'static + core::fmt::Debug {
     /// Send a new progress update
     ///
     /// The `progress` value should be in the range 0.0 - 100.0, and the `message` value is an
@@ -183,7 +194,7 @@ impl Progress for Sender<ProgressData> {
 }
 
 /// Type that implements [`Progress`] and drops every update received
-#[derive(Clone, Copy)]
+#[derive(Clone, Copy, Default, Debug)]
 pub struct NoopProgress;
 
 /// Create a new instance of [`NoopProgress`]
@@ -198,7 +209,7 @@ impl Progress for NoopProgress {
 }
 
 /// Type that implements [`Progress`] and logs at level `INFO` every update received
-#[derive(Clone, Copy)]
+#[derive(Clone, Copy, Default, Debug)]
 pub struct LogProgress;
 
 /// Create a new instance of [`LogProgress`]
@@ -224,33 +235,44 @@ impl<T: Blockchain> Blockchain for Arc<T> {
         maybe_await!(self.deref().get_capabilities())
     }
 
-    fn setup<D: BatchDatabase, P: 'static + Progress>(
-        &self,
-        database: &mut D,
-        progress_update: P,
-    ) -> Result<(), Error> {
-        maybe_await!(self.deref().setup(database, progress_update))
-    }
-
-    fn sync<D: BatchDatabase, P: 'static + Progress>(
-        &self,
-        database: &mut D,
-        progress_update: P,
-    ) -> Result<(), Error> {
-        maybe_await!(self.deref().sync(database, progress_update))
-    }
-
-    fn get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error> {
-        maybe_await!(self.deref().get_tx(txid))
-    }
     fn broadcast(&self, tx: &Transaction) -> Result<(), Error> {
         maybe_await!(self.deref().broadcast(tx))
     }
 
-    fn get_height(&self) -> Result<u32, Error> {
-        maybe_await!(self.deref().get_height())
-    }
     fn estimate_fee(&self, target: usize) -> Result<FeeRate, Error> {
         maybe_await!(self.deref().estimate_fee(target))
     }
 }
+
+#[maybe_async]
+impl<T: GetTx> GetTx for Arc<T> {
+    fn get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error> {
+        maybe_await!(self.deref().get_tx(txid))
+    }
+}
+
+#[maybe_async]
+impl<T: GetHeight> GetHeight for Arc<T> {
+    fn get_height(&self) -> Result<u32, Error> {
+        maybe_await!(self.deref().get_height())
+    }
+}
+
+#[maybe_async]
+impl<T: WalletSync> WalletSync for Arc<T> {
+    fn wallet_setup<D: BatchDatabase>(
+        &self,
+        database: &mut D,
+        progress_update: Box<dyn Progress>,
+    ) -> Result<(), Error> {
+        maybe_await!(self.deref().wallet_setup(database, progress_update))
+    }
+
+    fn wallet_sync<D: BatchDatabase>(
+        &self,
+        database: &mut D,
+        progress_update: Box<dyn Progress>,
+    ) -> Result<(), Error> {
+        maybe_await!(self.deref().wallet_sync(database, progress_update))
+    }
+}

src/blockchain/rpc.rs

@@ -33,18 +33,16 @@
 
 use crate::bitcoin::consensus::deserialize;
 use crate::bitcoin::{Address, Network, OutPoint, Transaction, TxOut, Txid};
-use crate::blockchain::{Blockchain, Capability, ConfigurableBlockchain, Progress};
+use crate::blockchain::*;
 use crate::database::{BatchDatabase, DatabaseUtils};
-use crate::descriptor::{get_checksum, IntoWalletDescriptor};
+use crate::{BlockTime, Error, FeeRate, KeychainKind, LocalUtxo, TransactionDetails};
-use crate::wallet::utils::SecpCtx;
+use bitcoincore_rpc::json::{
-use crate::{ConfirmationTime, Error, FeeRate, KeychainKind, LocalUtxo, TransactionDetails};
-use core_rpc::json::{
     GetAddressInfoResultLabel, ImportMultiOptions, ImportMultiRequest,
     ImportMultiRequestScriptPubkey, ImportMultiRescanSince,
 };
-use core_rpc::jsonrpc::serde_json::Value;
+use bitcoincore_rpc::jsonrpc::serde_json::Value;
-use core_rpc::Auth as RpcAuth;
+use bitcoincore_rpc::Auth as RpcAuth;
-use core_rpc::{Client, RpcApi};
+use bitcoincore_rpc::{Client, RpcApi};
 use log::debug;
 use serde::{Deserialize, Serialize};
 use std::collections::{HashMap, HashSet};
@@ -76,13 +74,13 @@ pub struct RpcConfig {
     pub auth: Auth,
     /// The network we are using (it will be checked the bitcoin node network matches this)
     pub network: Network,
-    /// The wallet name in the bitcoin node, consider using [wallet_name_from_descriptor] for this
+    /// The wallet name in the bitcoin node, consider using [crate::wallet::wallet_name_from_descriptor] for this
     pub wallet_name: String,
     /// Skip this many blocks of the blockchain at the first rescan, if None the rescan is done from the genesis block
     pub skip_blocks: Option<u32>,
 }
 
-/// This struct is equivalent to [core_rpc::Auth] but it implements [serde::Serialize]
+/// This struct is equivalent to [bitcoincore_rpc::Auth] but it implements [serde::Serialize]
 /// To be removed once upstream equivalent is implementing Serialize (json serialization format
 /// should be the same), see [rust-bitcoincore-rpc/pull/181](https://github.com/rust-bitcoin/rust-bitcoincore-rpc/pull/181)
 #[derive(Clone, Debug, Hash, Eq, PartialEq, Ord, PartialOrd, Serialize, Deserialize)]
@@ -141,10 +139,39 @@ impl Blockchain for RpcBlockchain {
         self.capabilities.clone()
     }
 
-    fn setup<D: BatchDatabase, P: 'static + Progress>(
+    fn broadcast(&self, tx: &Transaction) -> Result<(), Error> {
+        Ok(self.client.send_raw_transaction(tx).map(|_| ())?)
+    }
+
+    fn estimate_fee(&self, target: usize) -> Result<FeeRate, Error> {
+        let sat_per_kb = self
+            .client
+            .estimate_smart_fee(target as u16, None)?
+            .fee_rate
+            .ok_or(Error::FeeRateUnavailable)?
+            .as_sat() as f64;
+
+        Ok(FeeRate::from_sat_per_vb((sat_per_kb / 1000f64) as f32))
+    }
+}
+
+impl GetTx for RpcBlockchain {
+    fn get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error> {
+        Ok(Some(self.client.get_raw_transaction(txid, None)?))
+    }
+}
+
+impl GetHeight for RpcBlockchain {
+    fn get_height(&self) -> Result<u32, Error> {
+        Ok(self.client.get_blockchain_info().map(|i| i.blocks as u32)?)
+    }
+}
+
+impl WalletSync for RpcBlockchain {
+    fn wallet_setup<D: BatchDatabase>(
         &self,
         database: &mut D,
-        progress_update: P,
+        progress_update: Box<dyn Progress>,
     ) -> Result<(), Error> {
         let mut scripts_pubkeys = database.iter_script_pubkeys(Some(KeychainKind::External))?;
         scripts_pubkeys.extend(database.iter_script_pubkeys(Some(KeychainKind::Internal))?);
@@ -156,7 +183,7 @@ impl Blockchain for RpcBlockchain {
             .iter()
             .map(|s| ImportMultiRequest {
                 timestamp: ImportMultiRescanSince::Timestamp(0),
-                script_pubkey: Some(ImportMultiRequestScriptPubkey::Script(&s)),
+                script_pubkey: Some(ImportMultiRequestScriptPubkey::Script(s)),
                 watchonly: Some(true),
                 ..Default::default()
             })
@@ -189,13 +216,13 @@ impl Blockchain for RpcBlockchain {
             }
         }
 
-        self.sync(database, progress_update)
+        self.wallet_sync(database, progress_update)
     }
 
-    fn sync<D: BatchDatabase, P: 'static + Progress>(
+    fn wallet_sync<D: BatchDatabase>(
         &self,
         db: &mut D,
-        _progress_update: P,
+        _progress_update: Box<dyn Progress>,
     ) -> Result<(), Error> {
         let mut indexes = HashMap::new();
         for keykind in &[KeychainKind::External, KeychainKind::Internal] {
@@ -222,7 +249,7 @@ impl Blockchain for RpcBlockchain {
         let mut list_txs_ids = HashSet::new();
 
         for tx_result in list_txs.iter().filter(|t| {
-            // list_txs returns all conflicting tx we want to
+            // list_txs returns all conflicting txs, we want to
             // filter out replaced tx => unconfirmed and not in the mempool
             t.info.confirmations > 0 || self.client.get_mempool_entry(&t.info.txid).is_ok()
         }) {
@@ -230,7 +257,7 @@ impl Blockchain for RpcBlockchain {
             list_txs_ids.insert(txid);
             if let Some(mut known_tx) = known_txs.get_mut(&txid) {
                 let confirmation_time =
-                    ConfirmationTime::new(tx_result.info.blockheight, tx_result.info.blocktime);
+                    BlockTime::new(tx_result.info.blockheight, tx_result.info.blocktime);
                 if confirmation_time != known_tx.confirmation_time {
                     // reorg may change tx height
                     debug!(
@@ -238,7 +265,7 @@ impl Blockchain for RpcBlockchain {
                         txid, confirmation_time
                     );
                     known_tx.confirmation_time = confirmation_time;
-                    db.set_tx(&known_tx)?;
+                    db.set_tx(known_tx)?;
                 }
             } else {
                 //TODO check there is already the raw tx in db?
@@ -259,21 +286,22 @@ impl Blockchain for RpcBlockchain {
 
                 for input in tx.input.iter() {
                     if let Some(previous_output) = db.get_previous_output(&input.previous_output)? {
-                        sent += previous_output.value;
+                        if db.is_mine(&previous_output.script_pubkey)? {
+                            sent += previous_output.value;
+                        }
                     }
                 }
 
                 let td = TransactionDetails {
                     transaction: Some(tx),
                     txid: tx_result.info.txid,
-                    confirmation_time: ConfirmationTime::new(
+                    confirmation_time: BlockTime::new(
                         tx_result.info.blockheight,
                         tx_result.info.blocktime,
                     ),
                     received,
                     sent,
                     fee: tx_result.fee.map(|f| f.as_sat().abs() as u64),
-                    verified: true,
                 };
                 debug!(
                     "saving tx: {} tx_result.fee:{:?} td.fees:{:?}",
@@ -290,32 +318,37 @@ impl Blockchain for RpcBlockchain {
             }
         }
 
-        let current_utxos: HashSet<_> = current_utxo
+        // Filter out trasactions that are for script pubkeys that aren't in this wallet.
+        let current_utxos = current_utxo
             .into_iter()
-            .map(|u| {
+            .filter_map(
-                Ok(LocalUtxo {
+                |u| match db.get_path_from_script_pubkey(&u.script_pub_key) {
-                    outpoint: OutPoint::new(u.txid, u.vout),
+                    Err(e) => Some(Err(e)),
-                    keychain: db
+                    Ok(None) => None,
-                        .get_path_from_script_pubkey(&u.script_pub_key)?
+                    Ok(Some(path)) => Some(Ok(LocalUtxo {
-                        .ok_or(Error::TransactionNotFound)?
+                        outpoint: OutPoint::new(u.txid, u.vout),
-                        .0,
+                        keychain: path.0,
-                    txout: TxOut {
+                        txout: TxOut {
-                        value: u.amount.as_sat(),
+                            value: u.amount.as_sat(),
-                        script_pubkey: u.script_pub_key,
+                            script_pubkey: u.script_pub_key,
-                    },
+                        },
-                })
+                        is_spent: false,
-            })
+                    })),
-            .collect::<Result<_, Error>>()?;
+                },
+            )
+            .collect::<Result<HashSet<_>, Error>>()?;
 
         let spent: HashSet<_> = known_utxos.difference(&current_utxos).collect();
-        for s in spent {
+        for utxo in spent {
-            debug!("removing utxo: {:?}", s);
+            debug!("setting as spent utxo: {:?}", utxo);
-            db.del_utxo(&s.outpoint)?;
+            let mut spent_utxo = utxo.clone();
+            spent_utxo.is_spent = true;
+            db.set_utxo(&spent_utxo)?;
         }
         let received: HashSet<_> = current_utxos.difference(&known_utxos).collect();
-        for s in received {
+        for utxo in received {
-            debug!("adding utxo: {:?}", s);
+            debug!("adding utxo: {:?}", utxo);
-            db.set_utxo(s)?;
+            db.set_utxo(utxo)?;
         }
 
         for (keykind, index) in indexes {
@@ -325,29 +358,6 @@ impl Blockchain for RpcBlockchain {
 
         Ok(())
     }
-
-    fn get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error> {
-        Ok(Some(self.client.get_raw_transaction(txid, None)?))
-    }
-
-    fn broadcast(&self, tx: &Transaction) -> Result<(), Error> {
-        Ok(self.client.send_raw_transaction(tx).map(|_| ())?)
-    }
-
-    fn get_height(&self) -> Result<u32, Error> {
-        Ok(self.client.get_blockchain_info().map(|i| i.blocks as u32)?)
-    }
-
-    fn estimate_fee(&self, target: usize) -> Result<FeeRate, Error> {
-        let sat_per_kb = self
-            .client
-            .estimate_smart_fee(target as u16, None)?
-            .fee_rate
-            .ok_or(Error::FeeRateUnavailable)?
-            .as_sat() as f64;
-
-        Ok(FeeRate::from_sat_per_vb((sat_per_kb / 1000f64) as f32))
-    }
 }
 
 impl ConfigurableBlockchain for RpcBlockchain {
@@ -415,35 +425,6 @@ impl ConfigurableBlockchain for RpcBlockchain {
     }
 }
 
-/// Deterministically generate a unique name given the descriptors defining the wallet
-pub fn wallet_name_from_descriptor<T>(
-    descriptor: T,
-    change_descriptor: Option<T>,
-    network: Network,
-    secp: &SecpCtx,
-) -> Result<String, Error>
-where
-    T: IntoWalletDescriptor,
-{
-    //TODO check descriptors contains only public keys
-    let descriptor = descriptor
-        .into_wallet_descriptor(&secp, network)?
-        .0
-        .to_string();
-    let mut wallet_name = get_checksum(&descriptor[..descriptor.find('#').unwrap()])?;
-    if let Some(change_descriptor) = change_descriptor {
-        let change_descriptor = change_descriptor
-            .into_wallet_descriptor(&secp, network)?
-            .0
-            .to_string();
-        wallet_name.push_str(
-            get_checksum(&change_descriptor[..change_descriptor.find('#').unwrap()])?.as_str(),
-        );
-    }
-
-    Ok(wallet_name)
-}
-
 /// return the wallets available in default wallet directory
 //TODO use bitcoincore_rpc method when PR #179 lands
 fn list_wallet_dir(client: &Client) -> Result<Vec<String>, Error> {

src/blockchain/script_sync.rs

@@ -0,0 +1,415 @@
+/*!
+This models a how a sync happens where you have a server that you send your script pubkeys to and it
+returns associated transactions i.e. electrum.
+*/
+#![allow(dead_code)]
+use crate::{
+    database::{BatchDatabase, BatchOperations, DatabaseUtils},
+    wallet::time::Instant,
+    BlockTime, Error, KeychainKind, LocalUtxo, TransactionDetails,
+};
+use bitcoin::{OutPoint, Script, Transaction, TxOut, Txid};
+use log::*;
+use std::collections::{BTreeMap, BTreeSet, HashMap, HashSet, VecDeque};
+
+/// A request for on-chain information
+pub enum Request<'a, D: BatchDatabase> {
+    /// A request for transactions related to script pubkeys.
+    Script(ScriptReq<'a, D>),
+    /// A request for confirmation times for some transactions.
+    Conftime(ConftimeReq<'a, D>),
+    /// A request for full transaction details of some transactions.
+    Tx(TxReq<'a, D>),
+    /// Requests are finished here's a batch database update to reflect data gathered.
+    Finish(D::Batch),
+}
+
+/// starts a sync
+pub fn start<D: BatchDatabase>(db: &D, stop_gap: usize) -> Result<Request<'_, D>, Error> {
+    use rand::seq::SliceRandom;
+    let mut keychains = vec![KeychainKind::Internal, KeychainKind::External];
+    // shuffling improve privacy, the server doesn't know my first request is from my internal or external addresses
+    keychains.shuffle(&mut rand::thread_rng());
+    let keychain = keychains.pop().unwrap();
+    let scripts_needed = db
+        .iter_script_pubkeys(Some(keychain))?
+        .into_iter()
+        .collect();
+    let state = State::new(db);
+
+    Ok(Request::Script(ScriptReq {
+        state,
+        scripts_needed,
+        script_index: 0,
+        stop_gap,
+        keychain,
+        next_keychains: keychains,
+    }))
+}
+
+pub struct ScriptReq<'a, D: BatchDatabase> {
+    state: State<'a, D>,
+    script_index: usize,
+    scripts_needed: VecDeque<Script>,
+    stop_gap: usize,
+    keychain: KeychainKind,
+    next_keychains: Vec<KeychainKind>,
+}
+
+/// The sync starts by returning script pubkeys we are interested in.
+impl<'a, D: BatchDatabase> ScriptReq<'a, D> {
+    pub fn request(&self) -> impl Iterator<Item = &Script> + Clone {
+        self.scripts_needed.iter()
+    }
+
+    pub fn satisfy(
+        mut self,
+        // we want to know the txids assoiciated with the script and their height
+        txids: Vec<Vec<(Txid, Option<u32>)>>,
+    ) -> Result<Request<'a, D>, Error> {
+        for (txid_list, script) in txids.iter().zip(self.scripts_needed.iter()) {
+            debug!(
+                "found {} transactions for script pubkey {}",
+                txid_list.len(),
+                script
+            );
+            if !txid_list.is_empty() {
+                // the address is active
+                self.state
+                    .last_active_index
+                    .insert(self.keychain, self.script_index);
+            }
+
+            for (txid, height) in txid_list {
+                // have we seen this txid already?
+                match self.state.db.get_tx(txid, true)? {
+                    Some(mut details) => {
+                        let old_height = details.confirmation_time.as_ref().map(|x| x.height);
+                        match (old_height, height) {
+                            (None, Some(_)) => {
+                                // It looks like the tx has confirmed since we last saw it -- we
+                                // need to know the confirmation time.
+                                self.state.tx_missing_conftime.insert(*txid, details);
+                            }
+                            (Some(old_height), Some(new_height)) if old_height != *new_height => {
+                                // The height of the tx has changed !? -- It's a reorg get the new confirmation time.
+                                self.state.tx_missing_conftime.insert(*txid, details);
+                            }
+                            (Some(_), None) => {
+                                // A re-org where the tx is not in the chain anymore.
+                                details.confirmation_time = None;
+                                self.state.finished_txs.push(details);
+                            }
+                            _ => self.state.finished_txs.push(details),
+                        }
+                    }
+                    None => {
+                        // we've never seen it let's get the whole thing
+                        self.state.tx_needed.insert(*txid);
+                    }
+                };
+            }
+
+            self.script_index += 1;
+        }
+
+        for _ in txids {
+            self.scripts_needed.pop_front();
+        }
+
+        let last_active_index = self
+            .state
+            .last_active_index
+            .get(&self.keychain)
+            .map(|x| x + 1)
+            .unwrap_or(0); // so no addresses active maps to 0
+
+        Ok(
+            if self.script_index > last_active_index + self.stop_gap
+                || self.scripts_needed.is_empty()
+            {
+                debug!(
+                    "finished scanning for transactions for keychain {:?} at index {}",
+                    self.keychain, last_active_index
+                );
+                // we're done here -- check if we need to do the next keychain
+                if let Some(keychain) = self.next_keychains.pop() {
+                    self.keychain = keychain;
+                    self.script_index = 0;
+                    self.scripts_needed = self
+                        .state
+                        .db
+                        .iter_script_pubkeys(Some(keychain))?
+                        .into_iter()
+                        .collect();
+                    Request::Script(self)
+                } else {
+                    Request::Tx(TxReq { state: self.state })
+                }
+            } else {
+                Request::Script(self)
+            },
+        )
+    }
+}
+
+/// Then we get full transactions
+pub struct TxReq<'a, D> {
+    state: State<'a, D>,
+}
+
+impl<'a, D: BatchDatabase> TxReq<'a, D> {
+    pub fn request(&self) -> impl Iterator<Item = &Txid> + Clone {
+        self.state.tx_needed.iter()
+    }
+
+    pub fn satisfy(
+        mut self,
+        tx_details: Vec<(Vec<Option<TxOut>>, Transaction)>,
+    ) -> Result<Request<'a, D>, Error> {
+        let tx_details: Vec<TransactionDetails> = tx_details
+            .into_iter()
+            .zip(self.state.tx_needed.iter())
+            .map(|((vout, tx), txid)| {
+                debug!("found tx_details for {}", txid);
+                assert_eq!(tx.txid(), *txid);
+                let mut sent: u64 = 0;
+                let mut received: u64 = 0;
+                let mut inputs_sum: u64 = 0;
+                let mut outputs_sum: u64 = 0;
+
+                for (txout, (_input_index, input)) in
+                    vout.into_iter().zip(tx.input.iter().enumerate())
+                {
+                    let txout = match txout {
+                        Some(txout) => txout,
+                        None => {
+                            // skip coinbase inputs
+                            debug_assert!(
+                                input.previous_output.is_null(),
+                                "prevout should only be missing for coinbase"
+                            );
+                            continue;
+                        }
+                    };
+                    // Verify this input if requested via feature flag
+                    #[cfg(feature = "verify")]
+                    {
+                        use crate::wallet::verify::VerifyError;
+                        let serialized_tx = bitcoin::consensus::serialize(&tx);
+                        bitcoinconsensus::verify(
+                            txout.script_pubkey.to_bytes().as_ref(),
+                            txout.value,
+                            &serialized_tx,
+                            _input_index,
+                        )
+                        .map_err(VerifyError::from)?;
+                    }
+                    inputs_sum += txout.value;
+                    if self.state.db.is_mine(&txout.script_pubkey)? {
+                        sent += txout.value;
+                    }
+                }
+
+                for out in &tx.output {
+                    outputs_sum += out.value;
+                    if self.state.db.is_mine(&out.script_pubkey)? {
+                        received += out.value;
+                    }
+                }
+                // we need to saturating sub since we want coinbase txs to map to 0 fee and
+                // this subtraction will be negative for coinbase txs.
+                let fee = inputs_sum.saturating_sub(outputs_sum);
+                Result::<_, Error>::Ok(TransactionDetails {
+                    txid: *txid,
+                    transaction: Some(tx),
+                    received,
+                    sent,
+                    // we're going to fill this in later
+                    confirmation_time: None,
+                    fee: Some(fee),
+                })
+            })
+            .collect::<Result<Vec<_>, _>>()?;
+
+        for tx_detail in tx_details {
+            self.state.tx_needed.remove(&tx_detail.txid);
+            self.state
+                .tx_missing_conftime
+                .insert(tx_detail.txid, tx_detail);
+        }
+
+        if !self.state.tx_needed.is_empty() {
+            Ok(Request::Tx(self))
+        } else {
+            Ok(Request::Conftime(ConftimeReq { state: self.state }))
+        }
+    }
+}
+
+/// Final step is to get confirmation times
+pub struct ConftimeReq<'a, D> {
+    state: State<'a, D>,
+}
+
+impl<'a, D: BatchDatabase> ConftimeReq<'a, D> {
+    pub fn request(&self) -> impl Iterator<Item = &Txid> + Clone {
+        self.state.tx_missing_conftime.keys()
+    }
+
+    pub fn satisfy(
+        mut self,
+        confirmation_times: Vec<Option<BlockTime>>,
+    ) -> Result<Request<'a, D>, Error> {
+        let conftime_needed = self
+            .request()
+            .cloned()
+            .take(confirmation_times.len())
+            .collect::<Vec<_>>();
+        for (confirmation_time, txid) in confirmation_times.into_iter().zip(conftime_needed.iter())
+        {
+            debug!("confirmation time for {} was {:?}", txid, confirmation_time);
+            if let Some(mut tx_details) = self.state.tx_missing_conftime.remove(txid) {
+                tx_details.confirmation_time = confirmation_time;
+                self.state.finished_txs.push(tx_details);
+            }
+        }
+
+        if self.state.tx_missing_conftime.is_empty() {
+            Ok(Request::Finish(self.state.into_db_update()?))
+        } else {
+            Ok(Request::Conftime(self))
+        }
+    }
+}
+
+struct State<'a, D> {
+    db: &'a D,
+    last_active_index: HashMap<KeychainKind, usize>,
+    /// Transactions where we need to get the full details
+    tx_needed: BTreeSet<Txid>,
+    /// Transacitions that we know everything about
+    finished_txs: Vec<TransactionDetails>,
+    /// Transactions that discovered conftimes should be inserted into
+    tx_missing_conftime: BTreeMap<Txid, TransactionDetails>,
+    /// The start of the sync
+    start_time: Instant,
+}
+
+impl<'a, D: BatchDatabase> State<'a, D> {
+    fn new(db: &'a D) -> Self {
+        State {
+            db,
+            last_active_index: HashMap::default(),
+            finished_txs: vec![],
+            tx_needed: BTreeSet::default(),
+            tx_missing_conftime: BTreeMap::default(),
+            start_time: Instant::new(),
+        }
+    }
+    fn into_db_update(self) -> Result<D::Batch, Error> {
+        debug_assert!(self.tx_needed.is_empty() && self.tx_missing_conftime.is_empty());
+        let existing_txs = self.db.iter_txs(false)?;
+        let existing_txids: HashSet<Txid> = existing_txs.iter().map(|tx| tx.txid).collect();
+        let finished_txs = make_txs_consistent(&self.finished_txs);
+        let observed_txids: HashSet<Txid> = finished_txs.iter().map(|tx| tx.txid).collect();
+        let txids_to_delete = existing_txids.difference(&observed_txids);
+        let mut batch = self.db.begin_batch();
+
+        // Delete old txs that no longer exist
+        for txid in txids_to_delete {
+            if let Some(raw_tx) = self.db.get_raw_tx(txid)? {
+                for i in 0..raw_tx.output.len() {
+                    // Also delete any utxos from the txs that no longer exist.
+                    let _ = batch.del_utxo(&OutPoint {
+                        txid: *txid,
+                        vout: i as u32,
+                    })?;
+                }
+            } else {
+                unreachable!("we should always have the raw tx");
+            }
+            batch.del_tx(txid, true)?;
+        }
+
+        let mut spent_utxos = HashSet::new();
+
+        // track all the spent utxos
+        for finished_tx in &finished_txs {
+            let tx = finished_tx
+                .transaction
+                .as_ref()
+                .expect("transaction will always be present here");
+            for input in &tx.input {
+                spent_utxos.insert(&input.previous_output);
+            }
+        }
+
+        // set every utxo we observed, unless it's already spent
+        // we don't do this in the loop above as we want to know all the spent outputs before
+        // adding the non-spent to the batch in case there are new tranasactions
+        // that spend form each other.
+        for finished_tx in &finished_txs {
+            let tx = finished_tx
+                .transaction
+                .as_ref()
+                .expect("transaction will always be present here");
+            for (i, output) in tx.output.iter().enumerate() {
+                if let Some((keychain, _)) =
+                    self.db.get_path_from_script_pubkey(&output.script_pubkey)?
+                {
+                    // add utxos we own from the new transactions we've seen.
+                    let outpoint = OutPoint {
+                        txid: finished_tx.txid,
+                        vout: i as u32,
+                    };
+
+                    batch.set_utxo(&LocalUtxo {
+                        outpoint,
+                        txout: output.clone(),
+                        keychain,
+                        // Is this UTXO in the spent_utxos set?
+                        is_spent: spent_utxos.get(&outpoint).is_some(),
+                    })?;
+                }
+            }
+
+            batch.set_tx(finished_tx)?;
+        }
+
+        for (keychain, last_active_index) in self.last_active_index {
+            batch.set_last_index(keychain, last_active_index as u32)?;
+        }
+
+        info!(
+            "finished setup, elapsed {:?}ms",
+            self.start_time.elapsed().as_millis()
+        );
+        Ok(batch)
+    }
+}
+
+/// Remove conflicting transactions -- tie breaking them by fee.
+fn make_txs_consistent(txs: &[TransactionDetails]) -> Vec<&TransactionDetails> {
+    let mut utxo_index: HashMap<OutPoint, &TransactionDetails> = HashMap::default();
+    for tx in txs {
+        for input in &tx.transaction.as_ref().unwrap().input {
+            utxo_index
+                .entry(input.previous_output)
+                .and_modify(|existing| match (tx.fee, existing.fee) {
+                    (Some(fee), Some(existing_fee)) if fee > existing_fee => *existing = tx,
+                    (Some(_), None) => *existing = tx,
+                    _ => { /* leave it the same */ }
+                })
+                .or_insert(tx);
+        }
+    }
+
+    utxo_index
+        .into_iter()
+        .map(|(_, tx)| (tx.txid, tx))
+        .collect::<HashMap<_, _>>()
+        .into_iter()
+        .map(|(_, tx)| tx)
+        .collect()
+}

src/blockchain/utils.rs

@@ -1,388 +0,0 @@
-// Bitcoin Dev Kit
-// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
-//
-// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
-//
-// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
-// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
-// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
-// You may not use this file except in accordance with one or both of these
-// licenses.
-
-use std::collections::{HashMap, HashSet};
-
-#[allow(unused_imports)]
-use log::{debug, error, info, trace};
-use rand::seq::SliceRandom;
-use rand::thread_rng;
-
-use bitcoin::{BlockHeader, OutPoint, Script, Transaction, Txid};
-
-use super::*;
-use crate::database::{BatchDatabase, BatchOperations, DatabaseUtils};
-use crate::error::Error;
-use crate::types::{ConfirmationTime, KeychainKind, LocalUtxo, TransactionDetails};
-use crate::wallet::time::Instant;
-use crate::wallet::utils::ChunksIterator;
-
-#[derive(Debug)]
-pub struct ElsGetHistoryRes {
-    pub height: i32,
-    pub tx_hash: Txid,
-}
-
-/// Implements the synchronization logic for an Electrum-like client.
-#[maybe_async]
-pub trait ElectrumLikeSync {
-    fn els_batch_script_get_history<'s, I: IntoIterator<Item = &'s Script> + Clone>(
-        &self,
-        scripts: I,
-    ) -> Result<Vec<Vec<ElsGetHistoryRes>>, Error>;
-
-    fn els_batch_transaction_get<'s, I: IntoIterator<Item = &'s Txid> + Clone>(
-        &self,
-        txids: I,
-    ) -> Result<Vec<Transaction>, Error>;
-
-    fn els_batch_block_header<I: IntoIterator<Item = u32> + Clone>(
-        &self,
-        heights: I,
-    ) -> Result<Vec<BlockHeader>, Error>;
-
-    // Provided methods down here...
-
-    fn electrum_like_setup<D: BatchDatabase, P: Progress>(
-        &self,
-        stop_gap: usize,
-        db: &mut D,
-        _progress_update: P,
-    ) -> Result<(), Error> {
-        // TODO: progress
-        let start = Instant::new();
-        debug!("start setup");
-
-        let chunk_size = stop_gap;
-
-        let mut history_txs_id = HashSet::new();
-        let mut txid_height = HashMap::new();
-        let mut max_indexes = HashMap::new();
-
-        let mut wallet_chains = vec![KeychainKind::Internal, KeychainKind::External];
-        // shuffling improve privacy, the server doesn't know my first request is from my internal or external addresses
-        wallet_chains.shuffle(&mut thread_rng());
-        // download history of our internal and external script_pubkeys
-        for keychain in wallet_chains.iter() {
-            let script_iter = db.iter_script_pubkeys(Some(*keychain))?.into_iter();
-
-            for (i, chunk) in ChunksIterator::new(script_iter, stop_gap).enumerate() {
-                // TODO if i == last, should create another chunk of addresses in db
-                let call_result: Vec<Vec<ElsGetHistoryRes>> =
-                    maybe_await!(self.els_batch_script_get_history(chunk.iter()))?;
-                let max_index = call_result
-                    .iter()
-                    .enumerate()
-                    .filter_map(|(i, v)| v.first().map(|_| i as u32))
-                    .max();
-                if let Some(max) = max_index {
-                    max_indexes.insert(keychain, max + (i * chunk_size) as u32);
-                }
-                let flattened: Vec<ElsGetHistoryRes> = call_result.into_iter().flatten().collect();
-                debug!("#{} of {:?} results:{}", i, keychain, flattened.len());
-                if flattened.is_empty() {
-                    // Didn't find anything in the last `stop_gap` script_pubkeys, breaking
-                    break;
-                }
-
-                for el in flattened {
-                    // el.height = -1 means unconfirmed with unconfirmed parents
-                    // el.height =  0 means unconfirmed with confirmed parents
-                    // but we treat those tx the same
-                    if el.height <= 0 {
-                        txid_height.insert(el.tx_hash, None);
-                    } else {
-                        txid_height.insert(el.tx_hash, Some(el.height as u32));
-                    }
-                    history_txs_id.insert(el.tx_hash);
-                }
-            }
-        }
-
-        // saving max indexes
-        info!("max indexes are: {:?}", max_indexes);
-        for keychain in wallet_chains.iter() {
-            if let Some(index) = max_indexes.get(keychain) {
-                db.set_last_index(*keychain, *index)?;
-            }
-        }
-
-        // get db status
-        let txs_details_in_db: HashMap<Txid, TransactionDetails> = db
-            .iter_txs(false)?
-            .into_iter()
-            .map(|tx| (tx.txid, tx))
-            .collect();
-        let txs_raw_in_db: HashMap<Txid, Transaction> = db
-            .iter_raw_txs()?
-            .into_iter()
-            .map(|tx| (tx.txid(), tx))
-            .collect();
-        let utxos_deps = utxos_deps(db, &txs_raw_in_db)?;
-
-        // download new txs and headers
-        let new_txs = maybe_await!(self.download_and_save_needed_raw_txs(
-            &history_txs_id,
-            &txs_raw_in_db,
-            chunk_size,
-            db
-        ))?;
-        let new_timestamps = maybe_await!(self.download_needed_headers(
-            &txid_height,
-            &txs_details_in_db,
-            chunk_size
-        ))?;
-
-        let mut batch = db.begin_batch();
-
-        // save any tx details not in db but in history_txs_id or with different height/timestamp
-        for txid in history_txs_id.iter() {
-            let height = txid_height.get(txid).cloned().flatten();
-            let timestamp = new_timestamps.get(txid).cloned();
-            if let Some(tx_details) = txs_details_in_db.get(txid) {
-                // check if tx height matches, otherwise updates it. timestamp is not in the if clause
-                // because we are not asking headers for confirmed tx we know about
-                if tx_details.confirmation_time.as_ref().map(|c| c.height) != height {
-                    let confirmation_time = ConfirmationTime::new(height, timestamp);
-                    let mut new_tx_details = tx_details.clone();
-                    new_tx_details.confirmation_time = confirmation_time;
-                    batch.set_tx(&new_tx_details)?;
-                }
-            } else {
-                save_transaction_details_and_utxos(
-                    txid,
-                    db,
-                    timestamp,
-                    height,
-                    &mut batch,
-                    &utxos_deps,
-                )?;
-            }
-        }
-
-        // remove any tx details in db but not in history_txs_id
-        for txid in txs_details_in_db.keys() {
-            if !history_txs_id.contains(txid) {
-                batch.del_tx(txid, false)?;
-            }
-        }
-
-        // remove any spent utxo
-        for new_tx in new_txs.iter() {
-            for input in new_tx.input.iter() {
-                batch.del_utxo(&input.previous_output)?;
-            }
-        }
-
-        db.commit_batch(batch)?;
-        info!("finish setup, elapsed {:?}ms", start.elapsed().as_millis());
-
-        Ok(())
-    }
-
-    /// download txs identified by `history_txs_id` and theirs previous outputs if not already present in db
-    fn download_and_save_needed_raw_txs<D: BatchDatabase>(
-        &self,
-        history_txs_id: &HashSet<Txid>,
-        txs_raw_in_db: &HashMap<Txid, Transaction>,
-        chunk_size: usize,
-        db: &mut D,
-    ) -> Result<Vec<Transaction>, Error> {
-        let mut txs_downloaded = vec![];
-        let txids_raw_in_db: HashSet<Txid> = txs_raw_in_db.keys().cloned().collect();
-        let txids_to_download: Vec<&Txid> = history_txs_id.difference(&txids_raw_in_db).collect();
-        if !txids_to_download.is_empty() {
-            info!("got {} txs to download", txids_to_download.len());
-            txs_downloaded.extend(maybe_await!(self.download_and_save_in_chunks(
-                txids_to_download,
-                chunk_size,
-                db,
-            ))?);
-            let mut prev_txids = HashSet::new();
-            let mut txids_downloaded = HashSet::new();
-            for tx in txs_downloaded.iter() {
-                txids_downloaded.insert(tx.txid());
-                // add every previous input tx, but skip coinbase
-                for input in tx.input.iter().filter(|i| !i.previous_output.is_null()) {
-                    prev_txids.insert(input.previous_output.txid);
-                }
-            }
-            let already_present: HashSet<Txid> =
-                txids_downloaded.union(&txids_raw_in_db).cloned().collect();
-            let prev_txs_to_download: Vec<&Txid> =
-                prev_txids.difference(&already_present).collect();
-            info!("{} previous txs to download", prev_txs_to_download.len());
-            txs_downloaded.extend(maybe_await!(self.download_and_save_in_chunks(
-                prev_txs_to_download,
-                chunk_size,
-                db,
-            ))?);
-        }
-
-        Ok(txs_downloaded)
-    }
-
-    /// download headers at heights in `txid_height` if tx details not already present, returns a map Txid -> timestamp
-    fn download_needed_headers(
-        &self,
-        txid_height: &HashMap<Txid, Option<u32>>,
-        txs_details_in_db: &HashMap<Txid, TransactionDetails>,
-        chunk_size: usize,
-    ) -> Result<HashMap<Txid, u64>, Error> {
-        let mut txid_timestamp = HashMap::new();
-        let txid_in_db_with_conf: HashSet<_> = txs_details_in_db
-            .values()
-            .filter_map(|details| details.confirmation_time.as_ref().map(|_| details.txid))
-            .collect();
-        let needed_txid_height: HashMap<&Txid, u32> = txid_height
-            .iter()
-            .filter(|(t, _)| !txid_in_db_with_conf.contains(*t))
-            .filter_map(|(t, o)| o.map(|h| (t, h)))
-            .collect();
-        let needed_heights: HashSet<u32> = needed_txid_height.values().cloned().collect();
-        if !needed_heights.is_empty() {
-            info!("{} headers to download for timestamp", needed_heights.len());
-            let mut height_timestamp: HashMap<u32, u64> = HashMap::new();
-            for chunk in ChunksIterator::new(needed_heights.into_iter(), chunk_size) {
-                let call_result: Vec<BlockHeader> =
-                    maybe_await!(self.els_batch_block_header(chunk.clone()))?;
-                height_timestamp.extend(
-                    chunk
-                        .into_iter()
-                        .zip(call_result.iter().map(|h| h.time as u64)),
-                );
-            }
-            for (txid, height) in needed_txid_height {
-                let timestamp = height_timestamp
-                    .get(&height)
-                    .ok_or_else(|| Error::Generic("timestamp missing".to_string()))?;
-                txid_timestamp.insert(*txid, *timestamp);
-            }
-        }
-
-        Ok(txid_timestamp)
-    }
-
-    fn download_and_save_in_chunks<D: BatchDatabase>(
-        &self,
-        to_download: Vec<&Txid>,
-        chunk_size: usize,
-        db: &mut D,
-    ) -> Result<Vec<Transaction>, Error> {
-        let mut txs_downloaded = vec![];
-        for chunk in ChunksIterator::new(to_download.into_iter(), chunk_size) {
-            let call_result: Vec<Transaction> =
-                maybe_await!(self.els_batch_transaction_get(chunk))?;
-            let mut batch = db.begin_batch();
-            for new_tx in call_result.iter() {
-                batch.set_raw_tx(new_tx)?;
-            }
-            db.commit_batch(batch)?;
-            txs_downloaded.extend(call_result);
-        }
-
-        Ok(txs_downloaded)
-    }
-}
-
-fn save_transaction_details_and_utxos<D: BatchDatabase>(
-    txid: &Txid,
-    db: &mut D,
-    timestamp: Option<u64>,
-    height: Option<u32>,
-    updates: &mut dyn BatchOperations,
-    utxo_deps: &HashMap<OutPoint, OutPoint>,
-) -> Result<(), Error> {
-    let tx = db.get_raw_tx(txid)?.ok_or(Error::TransactionNotFound)?;
-
-    let mut incoming: u64 = 0;
-    let mut outgoing: u64 = 0;
-
-    let mut inputs_sum: u64 = 0;
-    let mut outputs_sum: u64 = 0;
-
-    // look for our own inputs
-    for input in tx.input.iter() {
-        // skip coinbase inputs
-        if input.previous_output.is_null() {
-            continue;
-        }
-
-        // We already downloaded all previous output txs in the previous step
-        if let Some(previous_output) = db.get_previous_output(&input.previous_output)? {
-            inputs_sum += previous_output.value;
-
-            if db.is_mine(&previous_output.script_pubkey)? {
-                outgoing += previous_output.value;
-            }
-        } else {
-            // The input is not ours, but we still need to count it for the fees
-            let tx = db
-                .get_raw_tx(&input.previous_output.txid)?
-                .ok_or(Error::TransactionNotFound)?;
-            inputs_sum += tx.output[input.previous_output.vout as usize].value;
-        }
-
-        // removes conflicting UTXO if any (generated from same inputs, like for example RBF)
-        if let Some(outpoint) = utxo_deps.get(&input.previous_output) {
-            updates.del_utxo(outpoint)?;
-        }
-    }
-
-    for (i, output) in tx.output.iter().enumerate() {
-        // to compute the fees later
-        outputs_sum += output.value;
-
-        // this output is ours, we have a path to derive it
-        if let Some((keychain, _child)) = db.get_path_from_script_pubkey(&output.script_pubkey)? {
-            debug!("{} output #{} is mine, adding utxo", txid, i);
-            updates.set_utxo(&LocalUtxo {
-                outpoint: OutPoint::new(tx.txid(), i as u32),
-                txout: output.clone(),
-                keychain,
-            })?;
-
-            incoming += output.value;
-        }
-    }
-
-    let tx_details = TransactionDetails {
-        txid: tx.txid(),
-        transaction: Some(tx),
-        received: incoming,
-        sent: outgoing,
-        confirmation_time: ConfirmationTime::new(height, timestamp),
-        fee: Some(inputs_sum.saturating_sub(outputs_sum)), /* if the tx is a coinbase, fees would be negative */
-        verified: height.is_some(),
-    };
-    updates.set_tx(&tx_details)?;
-
-    Ok(())
-}
-
-/// returns utxo dependency as the inputs needed for the utxo to exist
-/// `tx_raw_in_db` must contains utxo's generating txs or errors with [crate::Error::TransactionNotFound]
-fn utxos_deps<D: BatchDatabase>(
-    db: &mut D,
-    tx_raw_in_db: &HashMap<Txid, Transaction>,
-) -> Result<HashMap<OutPoint, OutPoint>, Error> {
-    let utxos = db.iter_utxos()?;
-    let mut utxos_deps = HashMap::new();
-    for utxo in utxos {
-        let from_tx = tx_raw_in_db
-            .get(&utxo.outpoint.txid)
-            .ok_or(Error::TransactionNotFound)?;
-        for input in from_tx.input.iter() {
-            utxos_deps.insert(input.previous_output, utxo.outpoint);
-        }
-    }
-    Ok(utxos_deps)
-}

src/database/any.rs

@@ -23,12 +23,12 @@
 //! # use bdk::database::{AnyDatabase, MemoryDatabase};
 //! # use bdk::{Wallet};
 //! let memory = MemoryDatabase::default();
-//! let wallet_memory = Wallet::new_offline("...", None, Network::Testnet, memory)?;
+//! let wallet_memory = Wallet::new("...", None, Network::Testnet, memory)?;
 //!
 //! # #[cfg(feature = "key-value-db")]
 //! # {
 //! let sled = sled::open("my-database")?.open_tree("default_tree")?;
-//! let wallet_sled = Wallet::new_offline("...", None, Network::Testnet, sled)?;
+//! let wallet_sled = Wallet::new("...", None, Network::Testnet, sled)?;
 //! # }
 //! # Ok::<(), bdk::Error>(())
 //! ```
@@ -42,7 +42,7 @@
 //! # use bdk::{Wallet};
 //! let config = serde_json::from_str("...")?;
 //! let database = AnyDatabase::from_config(&config)?;
-//! let wallet = Wallet::new_offline("...", None, Network::Testnet, database)?;
+//! let wallet = Wallet::new("...", None, Network::Testnet, database)?;
 //! # Ok::<(), bdk::Error>(())
 //! ```
 
@@ -61,6 +61,7 @@ macro_rules! impl_from {
 
 macro_rules! impl_inner_method {
     ( $enum_name:ident, $self:expr, $name:ident $(, $args:expr)* ) => {
+        #[allow(deprecated)]
         match $self {
             $enum_name::Memory(inner) => inner.$name( $($args, )* ),
             #[cfg(feature = "key-value-db")]
@@ -144,6 +145,9 @@ impl BatchOperations for AnyDatabase {
     fn set_last_index(&mut self, keychain: KeychainKind, value: u32) -> Result<(), Error> {
         impl_inner_method!(AnyDatabase, self, set_last_index, keychain, value)
     }
+    fn set_sync_time(&mut self, sync_time: SyncTime) -> Result<(), Error> {
+        impl_inner_method!(AnyDatabase, self, set_sync_time, sync_time)
+    }
 
     fn del_script_pubkey_from_path(
         &mut self,
@@ -180,6 +184,9 @@ impl BatchOperations for AnyDatabase {
     fn del_last_index(&mut self, keychain: KeychainKind) -> Result<Option<u32>, Error> {
         impl_inner_method!(AnyDatabase, self, del_last_index, keychain)
     }
+    fn del_sync_time(&mut self) -> Result<Option<SyncTime>, Error> {
+        impl_inner_method!(AnyDatabase, self, del_sync_time)
+    }
 }
 
 impl Database for AnyDatabase {
@@ -241,6 +248,9 @@ impl Database for AnyDatabase {
     fn get_last_index(&self, keychain: KeychainKind) -> Result<Option<u32>, Error> {
         impl_inner_method!(AnyDatabase, self, get_last_index, keychain)
     }
+    fn get_sync_time(&self) -> Result<Option<SyncTime>, Error> {
+        impl_inner_method!(AnyDatabase, self, get_sync_time)
+    }
 
     fn increment_last_index(&mut self, keychain: KeychainKind) -> Result<u32, Error> {
         impl_inner_method!(AnyDatabase, self, increment_last_index, keychain)
@@ -272,6 +282,9 @@ impl BatchOperations for AnyBatch {
     fn set_last_index(&mut self, keychain: KeychainKind, value: u32) -> Result<(), Error> {
         impl_inner_method!(AnyBatch, self, set_last_index, keychain, value)
     }
+    fn set_sync_time(&mut self, sync_time: SyncTime) -> Result<(), Error> {
+        impl_inner_method!(AnyBatch, self, set_sync_time, sync_time)
+    }
 
     fn del_script_pubkey_from_path(
         &mut self,
@@ -302,6 +315,9 @@ impl BatchOperations for AnyBatch {
     fn del_last_index(&mut self, keychain: KeychainKind) -> Result<Option<u32>, Error> {
         impl_inner_method!(AnyBatch, self, del_last_index, keychain)
     }
+    fn del_sync_time(&mut self) -> Result<Option<SyncTime>, Error> {
+        impl_inner_method!(AnyBatch, self, del_sync_time)
+    }
 }
 
 impl BatchDatabase for AnyDatabase {

src/database/keyvalue.rs

@@ -18,7 +18,7 @@ use bitcoin::hash_types::Txid;
 use bitcoin::{OutPoint, Script, Transaction};
 
 use crate::database::memory::MapKey;
-use crate::database::{BatchDatabase, BatchOperations, Database};
+use crate::database::{BatchDatabase, BatchOperations, Database, SyncTime};
 use crate::error::Error;
 use crate::types::*;
 
@@ -43,6 +43,7 @@ macro_rules! impl_batch_operations {
             let value = json!({
                 "t": utxo.txout,
                 "i": utxo.keychain,
+                "s": utxo.is_spent,
             });
             self.insert(key, serde_json::to_vec(&value)?)$($after_insert)*;
 
@@ -82,6 +83,13 @@ macro_rules! impl_batch_operations {
             Ok(())
         }
 
+        fn set_sync_time(&mut self, data: SyncTime) -> Result<(), Error> {
+            let key = MapKey::SyncTime.as_map_key();
+            self.insert(key, serde_json::to_vec(&data)?)$($after_insert)*;
+
+            Ok(())
+        }
+
         fn del_script_pubkey_from_path(&mut self, keychain: KeychainKind, path: u32) -> Result<Option<Script>, Error> {
             let key = MapKey::Path((Some(keychain), Some(path))).as_map_key();
             let res = self.remove(key);
@@ -118,8 +126,9 @@ macro_rules! impl_batch_operations {
                     let mut val: serde_json::Value = serde_json::from_slice(&b)?;
                     let txout = serde_json::from_value(val["t"].take())?;
                     let keychain = serde_json::from_value(val["i"].take())?;
+                    let is_spent = val.get_mut("s").and_then(|s| s.take().as_bool()).unwrap_or(false);
 
-                    Ok(Some(LocalUtxo { outpoint: outpoint.clone(), txout, keychain }))
+                    Ok(Some(LocalUtxo { outpoint: outpoint.clone(), txout, keychain, is_spent, }))
                 }
             }
         }
@@ -168,6 +177,14 @@ macro_rules! impl_batch_operations {
                 }
             }
         }
+
+        fn del_sync_time(&mut self) -> Result<Option<SyncTime>, Error> {
+            let key = MapKey::SyncTime.as_map_key();
+            let res = self.remove(key);
+            let res = $process_delete!(res);
+
+            Ok(res.map(|b| serde_json::from_slice(&b)).transpose()?)
+        }
     }
 }
 
@@ -231,11 +248,16 @@ impl Database for Tree {
                 let mut val: serde_json::Value = serde_json::from_slice(&v)?;
                 let txout = serde_json::from_value(val["t"].take())?;
                 let keychain = serde_json::from_value(val["i"].take())?;
+                let is_spent = val
+                    .get_mut("s")
+                    .and_then(|s| s.take().as_bool())
+                    .unwrap_or(false);
 
                 Ok(LocalUtxo {
                     outpoint,
                     txout,
                     keychain,
+                    is_spent,
                 })
             })
             .collect()
@@ -299,11 +321,16 @@ impl Database for Tree {
                 let mut val: serde_json::Value = serde_json::from_slice(&b)?;
                 let txout = serde_json::from_value(val["t"].take())?;
                 let keychain = serde_json::from_value(val["i"].take())?;
+                let is_spent = val
+                    .get_mut("s")
+                    .and_then(|s| s.take().as_bool())
+                    .unwrap_or(false);
 
                 Ok(LocalUtxo {
                     outpoint: *outpoint,
                     txout,
                     keychain,
+                    is_spent,
                 })
             })
             .transpose()
@@ -342,6 +369,14 @@ impl Database for Tree {
             .transpose()
     }
 
+    fn get_sync_time(&self) -> Result<Option<SyncTime>, Error> {
+        let key = MapKey::SyncTime.as_map_key();
+        Ok(self
+            .get(key)?
+            .map(|b| serde_json::from_slice(&b))
+            .transpose()?)
+    }
+
     // inserts 0 if not present
     fn increment_last_index(&mut self, keychain: KeychainKind) -> Result<u32, Error> {
         let key = MapKey::LastIndex(keychain).as_map_key();
@@ -470,4 +505,9 @@ mod test {
     fn test_last_index() {
         crate::database::test::test_last_index(get_tree());
     }
+
+    #[test]
+    fn test_sync_time() {
+        crate::database::test::test_sync_time(get_tree());
+    }
 }

src/database/memory.rs

@@ -14,6 +14,7 @@
 //! This module defines an in-memory database type called [`MemoryDatabase`] that is based on a
 //! [`BTreeMap`].
 
+use std::any::Any;
 use std::collections::BTreeMap;
 use std::ops::Bound::{Excluded, Included};
 
@@ -21,7 +22,7 @@ use bitcoin::consensus::encode::{deserialize, serialize};
 use bitcoin::hash_types::Txid;
 use bitcoin::{OutPoint, Script, Transaction};
 
-use crate::database::{BatchDatabase, BatchOperations, ConfigurableDatabase, Database};
+use crate::database::{BatchDatabase, BatchOperations, ConfigurableDatabase, Database, SyncTime};
 use crate::error::Error;
 use crate::types::*;
 
@@ -32,6 +33,7 @@ use crate::types::*;
 // transactions         t<txid> -> tx details
 // deriv indexes        c{i,e} -> u32
 // descriptor checksum  d{i,e} -> vec<u8>
+// last sync time       l -> { height, timestamp }
 
 pub(crate) enum MapKey<'a> {
     Path((Option<KeychainKind>, Option<u32>)),
@@ -40,6 +42,7 @@ pub(crate) enum MapKey<'a> {
     RawTx(Option<&'a Txid>),
     Transaction(Option<&'a Txid>),
     LastIndex(KeychainKind),
+    SyncTime,
     DescriptorChecksum(KeychainKind),
 }
 
@@ -58,6 +61,7 @@ impl MapKey<'_> {
             MapKey::RawTx(_) => b"r".to_vec(),
             MapKey::Transaction(_) => b"t".to_vec(),
             MapKey::LastIndex(st) => [b"c", st.as_ref()].concat(),
+            MapKey::SyncTime => b"l".to_vec(),
             MapKey::DescriptorChecksum(st) => [b"d", st.as_ref()].concat(),
         }
     }
@@ -105,12 +109,12 @@ fn after(key: &[u8]) -> Vec<u8> {
 /// Once it's dropped its content will be lost.
 ///
 /// If you are looking for a permanent storage solution, you can try with the default key-value
-/// database called [`sled`]. See the [`database`] module documentation for more defailts.
+/// database called [`sled`]. See the [`database`] module documentation for more details.
 ///
 /// [`database`]: crate::database
 #[derive(Debug, Default)]
 pub struct MemoryDatabase {
-    map: BTreeMap<Vec<u8>, Box<dyn std::any::Any>>,
+    map: BTreeMap<Vec<u8>, Box<dyn Any + Send + Sync>>,
     deleted_keys: Vec<Vec<u8>>,
 }
 
@@ -146,8 +150,10 @@ impl BatchOperations for MemoryDatabase {
 
     fn set_utxo(&mut self, utxo: &LocalUtxo) -> Result<(), Error> {
         let key = MapKey::Utxo(Some(&utxo.outpoint)).as_map_key();
-        self.map
+        self.map.insert(
-            .insert(key, Box::new((utxo.txout.clone(), utxo.keychain)));
+            key,
+            Box::new((utxo.txout.clone(), utxo.keychain, utxo.is_spent)),
+        );
 
         Ok(())
     }
@@ -179,6 +185,12 @@ impl BatchOperations for MemoryDatabase {
 
         Ok(())
     }
+    fn set_sync_time(&mut self, data: SyncTime) -> Result<(), Error> {
+        let key = MapKey::SyncTime.as_map_key();
+        self.map.insert(key, Box::new(data));
+
+        Ok(())
+    }
 
     fn del_script_pubkey_from_path(
         &mut self,
@@ -218,11 +230,12 @@ impl BatchOperations for MemoryDatabase {
         match res {
             None => Ok(None),
             Some(b) => {
-                let (txout, keychain) = b.downcast_ref().cloned().unwrap();
+                let (txout, keychain, is_spent) = b.downcast_ref().cloned().unwrap();
                 Ok(Some(LocalUtxo {
                     outpoint: *outpoint,
                     txout,
                     keychain,
+                    is_spent,
                 }))
             }
         }
@@ -269,6 +282,13 @@ impl BatchOperations for MemoryDatabase {
             Some(b) => Ok(Some(*b.downcast_ref().unwrap())),
         }
     }
+    fn del_sync_time(&mut self) -> Result<Option<SyncTime>, Error> {
+        let key = MapKey::SyncTime.as_map_key();
+        let res = self.map.remove(&key);
+        self.deleted_keys.push(key);
+
+        Ok(res.map(|b| b.downcast_ref().cloned().unwrap()))
+    }
 }
 
 impl Database for MemoryDatabase {
@@ -309,11 +329,12 @@ impl Database for MemoryDatabase {
             .range::<Vec<u8>, _>((Included(&key), Excluded(&after(&key))))
             .map(|(k, v)| {
                 let outpoint = deserialize(&k[1..]).unwrap();
-                let (txout, keychain) = v.downcast_ref().cloned().unwrap();
+                let (txout, keychain, is_spent) = v.downcast_ref().cloned().unwrap();
                 Ok(LocalUtxo {
                     outpoint,
                     txout,
                     keychain,
+                    is_spent,
                 })
             })
             .collect()
@@ -372,11 +393,12 @@ impl Database for MemoryDatabase {
     fn get_utxo(&self, outpoint: &OutPoint) -> Result<Option<LocalUtxo>, Error> {
         let key = MapKey::Utxo(Some(outpoint)).as_map_key();
         Ok(self.map.get(&key).map(|b| {
-            let (txout, keychain) = b.downcast_ref().cloned().unwrap();
+            let (txout, keychain, is_spent) = b.downcast_ref().cloned().unwrap();
             LocalUtxo {
                 outpoint: *outpoint,
                 txout,
                 keychain,
+                is_spent,
             }
         }))
     }
@@ -406,6 +428,14 @@ impl Database for MemoryDatabase {
         Ok(self.map.get(&key).map(|b| *b.downcast_ref().unwrap()))
     }
 
+    fn get_sync_time(&self) -> Result<Option<SyncTime>, Error> {
+        let key = MapKey::SyncTime.as_map_key();
+        Ok(self
+            .map
+            .get(&key)
+            .map(|b| b.downcast_ref().cloned().unwrap()))
+    }
+
     // inserts 0 if not present
     fn increment_last_index(&mut self, keychain: KeychainKind) -> Result<u32, Error> {
         let key = MapKey::LastIndex(keychain).as_map_key();
@@ -478,12 +508,10 @@ macro_rules! populate_test_db {
         };
 
         let txid = tx.txid();
-        let confirmation_time = tx_meta
+        let confirmation_time = tx_meta.min_confirmations.map(|conf| $crate::BlockTime {
-            .min_confirmations
+            height: current_height.unwrap().checked_sub(conf as u32).unwrap(),
-            .map(|conf| $crate::ConfirmationTime {
+            timestamp: 0,
-                height: current_height.unwrap().checked_sub(conf as u32).unwrap(),
+        });
-                timestamp: 0,
-            });
 
         let tx_details = $crate::TransactionDetails {
             transaction: Some(tx.clone()),
@@ -492,7 +520,6 @@ macro_rules! populate_test_db {
             received: 0,
             sent: 0,
             confirmation_time,
-            verified: current_height.is_some(),
         };
 
         db.set_tx(&tx_details).unwrap();
@@ -504,6 +531,7 @@ macro_rules! populate_test_db {
                     vout: vout as u32,
                 },
                 keychain: $crate::KeychainKind::External,
+                is_spent: false,
             })
             .unwrap();
         }
@@ -532,7 +560,7 @@ macro_rules! doctest_wallet {
             Some(100),
         );
 
-        $crate::Wallet::new_offline(
+        $crate::Wallet::new(
             &descriptors.0,
             descriptors.1.as_ref(),
             Network::Regtest,
@@ -589,4 +617,9 @@ mod test {
     fn test_last_index() {
         crate::database::test::test_last_index(get_tree());
     }
+
+    #[test]
+    fn test_sync_time() {
+        crate::database::test::test_sync_time(get_tree());
+    }
 }

src/database/mod.rs

@@ -24,6 +24,8 @@
 //!
 //! [`Wallet`]: crate::wallet::Wallet
 
+use serde::{Deserialize, Serialize};
+
 use bitcoin::hash_types::Txid;
 use bitcoin::{OutPoint, Script, Transaction, TxOut};
 
@@ -44,6 +46,15 @@ pub use sqlite::SqliteDatabase;
 pub mod memory;
 pub use memory::MemoryDatabase;
 
+/// Blockchain state at the time of syncing
+///
+/// Contains only the block time and height at the moment
+#[derive(Clone, Debug, Serialize, Deserialize)]
+pub struct SyncTime {
+    /// Block timestamp and height at the time of sync
+    pub block_time: BlockTime,
+}
+
 /// Trait for operations that can be batched
 ///
 /// This trait defines the list of operations that must be implemented on the [`Database`] type and
@@ -64,6 +75,8 @@ pub trait BatchOperations {
     fn set_tx(&mut self, transaction: &TransactionDetails) -> Result<(), Error>;
     /// Store the last derivation index for a given keychain.
     fn set_last_index(&mut self, keychain: KeychainKind, value: u32) -> Result<(), Error>;
+    /// Store the sync time
+    fn set_sync_time(&mut self, sync_time: SyncTime) -> Result<(), Error>;
 
     /// Delete a script_pubkey given the keychain and its child number.
     fn del_script_pubkey_from_path(
@@ -89,6 +102,10 @@ pub trait BatchOperations {
     ) -> Result<Option<TransactionDetails>, Error>;
     /// Delete the last derivation index for a keychain.
     fn del_last_index(&mut self, keychain: KeychainKind) -> Result<Option<u32>, Error>;
+    /// Reset the sync time to `None`
+    ///
+    /// Returns the removed value
+    fn del_sync_time(&mut self) -> Result<Option<SyncTime>, Error>;
 }
 
 /// Trait for reading data from a database
@@ -132,14 +149,20 @@ pub trait Database: BatchOperations {
     fn get_raw_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error>;
     /// Fetch the transaction metadata and optionally also the raw transaction
     fn get_tx(&self, txid: &Txid, include_raw: bool) -> Result<Option<TransactionDetails>, Error>;
-    /// Return the last defivation index for a keychain.
+    /// Return the last derivation index for a keychain.
     fn get_last_index(&self, keychain: KeychainKind) -> Result<Option<u32>, Error>;
+    /// Return the sync time, if present
+    fn get_sync_time(&self) -> Result<Option<SyncTime>, Error>;
 
     /// Increment the last derivation index for a keychain and return it
     ///
     /// It should insert and return `0` if not present in the database
     fn increment_last_index(&mut self, keychain: KeychainKind) -> Result<u32, Error>;
 
+    #[deprecated(
+        since = "0.18.0",
+        note = "The flush function is only needed for the sled database on mobile, instead for mobile use the sqlite database."
+    )]
     /// Force changes to be written to disk
     fn flush(&mut self) -> Result<(), Error>;
 }
@@ -297,6 +320,7 @@ pub mod test {
             txout,
             outpoint,
             keychain: KeychainKind::External,
+            is_spent: true,
         };
 
         tree.set_utxo(&utxo).unwrap();
@@ -325,11 +349,10 @@ pub mod test {
             received: 1337,
             sent: 420420,
             fee: Some(140),
-            confirmation_time: Some(ConfirmationTime {
+            confirmation_time: Some(BlockTime {
                 timestamp: 123456,
                 height: 1000,
             }),
-            verified: true,
         };
 
         tree.set_tx(&tx_details).unwrap();
@@ -377,5 +400,25 @@ pub mod test {
         );
     }
 
+    pub fn test_sync_time<D: Database>(mut tree: D) {
+        assert!(tree.get_sync_time().unwrap().is_none());
+
+        tree.set_sync_time(SyncTime {
+            block_time: BlockTime {
+                height: 100,
+                timestamp: 1000,
+            },
+        })
+        .unwrap();
+
+        let extracted = tree.get_sync_time().unwrap();
+        assert!(extracted.is_some());
+        assert_eq!(extracted.as_ref().unwrap().block_time.height, 100);
+        assert_eq!(extracted.as_ref().unwrap().block_time.timestamp, 1000);
+
+        tree.del_sync_time().unwrap();
+        assert!(tree.get_sync_time().unwrap().is_none());
+    }
+
     // TODO: more tests...
 }

src/database/sqlite.rs

@@ -13,7 +13,7 @@ use bitcoin::consensus::encode::{deserialize, serialize};
 use bitcoin::hash_types::Txid;
 use bitcoin::{OutPoint, Script, Transaction, TxOut};
 
-use crate::database::{BatchDatabase, BatchOperations, Database};
+use crate::database::{BatchDatabase, BatchOperations, Database, SyncTime};
 use crate::error::Error;
 use crate::types::*;
 
@@ -35,6 +35,12 @@ static MIGRATIONS: &[&str] = &[
     "CREATE UNIQUE INDEX idx_indices_keychain ON last_derivation_indices(keychain);",
     "CREATE TABLE checksums (keychain TEXT, checksum BLOB);",
     "CREATE INDEX idx_checksums_keychain ON checksums(keychain);",
+    "CREATE TABLE sync_time (id INTEGER PRIMARY KEY, height INTEGER, timestamp INTEGER);",
+    "ALTER TABLE transaction_details RENAME TO transaction_details_old;",
+    "CREATE TABLE transaction_details (txid BLOB, timestamp INTEGER, received INTEGER, sent INTEGER, fee INTEGER, height INTEGER);",
+    "INSERT INTO transaction_details SELECT txid, timestamp, received, sent, fee, height FROM transaction_details_old;",
+    "DROP TABLE transaction_details_old;",
+    "ALTER TABLE utxos ADD COLUMN is_spent;",
 ];
 
 /// Sqlite database stored on filesystem
@@ -78,14 +84,16 @@ impl SqliteDatabase {
         vout: u32,
         txid: &[u8],
         script: &[u8],
+        is_spent: bool,
     ) -> Result<i64, Error> {
-        let mut statement = self.connection.prepare_cached("INSERT INTO utxos (value, keychain, vout, txid, script) VALUES (:value, :keychain, :vout, :txid, :script)")?;
+        let mut statement = self.connection.prepare_cached("INSERT INTO utxos (value, keychain, vout, txid, script, is_spent) VALUES (:value, :keychain, :vout, :txid, :script, :is_spent)")?;
         statement.execute(named_params! {
             ":value": value,
             ":keychain": keychain,
             ":vout": vout,
             ":txid": txid,
-            ":script": script
+            ":script": script,
+            ":is_spent": is_spent,
         })?;
 
         Ok(self.connection.last_insert_rowid())
@@ -126,7 +134,7 @@ impl SqliteDatabase {
 
         let txid: &[u8] = &transaction.txid;
 
-        let mut statement = self.connection.prepare_cached("INSERT INTO transaction_details (txid, timestamp, received, sent, fee, height, verified) VALUES (:txid, :timestamp, :received, :sent, :fee, :height, :verified)")?;
+        let mut statement = self.connection.prepare_cached("INSERT INTO transaction_details (txid, timestamp, received, sent, fee, height) VALUES (:txid, :timestamp, :received, :sent, :fee, :height)")?;
 
         statement.execute(named_params! {
             ":txid": txid,
@@ -135,7 +143,6 @@ impl SqliteDatabase {
             ":sent": transaction.sent,
             ":fee": transaction.fee,
             ":height": height,
-            ":verified": transaction.verified
         })?;
 
         Ok(self.connection.last_insert_rowid())
@@ -152,7 +159,7 @@ impl SqliteDatabase {
 
         let txid: &[u8] = &transaction.txid;
 
-        let mut statement = self.connection.prepare_cached("UPDATE transaction_details SET timestamp=:timestamp, received=:received, sent=:sent, fee=:fee, height=:height, verified=:verified WHERE txid=:txid")?;
+        let mut statement = self.connection.prepare_cached("UPDATE transaction_details SET timestamp=:timestamp, received=:received, sent=:sent, fee=:fee, height=:height WHERE txid=:txid")?;
 
         statement.execute(named_params! {
             ":txid": txid,
@@ -161,7 +168,6 @@ impl SqliteDatabase {
             ":sent": transaction.sent,
             ":fee": transaction.fee,
             ":height": height,
-            ":verified": transaction.verified,
         })?;
 
         Ok(())
@@ -205,6 +211,19 @@ impl SqliteDatabase {
         Ok(())
     }
 
+    fn update_sync_time(&self, data: SyncTime) -> Result<i64, Error> {
+        let mut statement = self.connection.prepare_cached(
+            "INSERT INTO sync_time (id, height, timestamp) VALUES (0, :height, :timestamp) ON CONFLICT(id) DO UPDATE SET height=:height, timestamp=:timestamp WHERE id = 0",
+        )?;
+
+        statement.execute(named_params! {
+            ":height": data.block_time.height,
+            ":timestamp": data.block_time.timestamp,
+        })?;
+
+        Ok(self.connection.last_insert_rowid())
+    }
+
     fn select_script_pubkeys(&self) -> Result<Vec<Script>, Error> {
         let mut statement = self
             .connection
@@ -275,7 +294,7 @@ impl SqliteDatabase {
     fn select_utxos(&self) -> Result<Vec<LocalUtxo>, Error> {
         let mut statement = self
             .connection
-            .prepare_cached("SELECT value, keychain, vout, txid, script FROM utxos")?;
+            .prepare_cached("SELECT value, keychain, vout, txid, script, is_spent FROM utxos")?;
         let mut utxos: Vec<LocalUtxo> = vec![];
         let mut rows = statement.query([])?;
         while let Some(row) = rows.next()? {
@@ -284,6 +303,7 @@ impl SqliteDatabase {
             let vout = row.get(2)?;
             let txid: Vec<u8> = row.get(3)?;
             let script: Vec<u8> = row.get(4)?;
+            let is_spent: bool = row.get(5)?;
 
             let keychain: KeychainKind = serde_json::from_str(&keychain)?;
 
@@ -294,19 +314,16 @@ impl SqliteDatabase {
                     script_pubkey: script.into(),
                 },
                 keychain,
+                is_spent,
             })
         }
 
         Ok(utxos)
     }
 
-    fn select_utxo_by_outpoint(
+    fn select_utxo_by_outpoint(&self, txid: &[u8], vout: u32) -> Result<Option<LocalUtxo>, Error> {
-        &self,
-        txid: &[u8],
-        vout: u32,
-    ) -> Result<Option<(u64, KeychainKind, Script)>, Error> {
         let mut statement = self.connection.prepare_cached(
-            "SELECT value, keychain, script FROM utxos WHERE txid=:txid AND vout=:vout",
+            "SELECT value, keychain, script, is_spent FROM utxos WHERE txid=:txid AND vout=:vout",
         )?;
         let mut rows = statement.query(named_params! {":txid": txid,":vout": vout})?;
         match rows.next()? {
@@ -315,9 +332,18 @@ impl SqliteDatabase {
                 let keychain: String = row.get(1)?;
                 let keychain: KeychainKind = serde_json::from_str(&keychain)?;
                 let script: Vec<u8> = row.get(2)?;
-                let script: Script = script.into();
+                let script_pubkey: Script = script.into();
+                let is_spent: bool = row.get(3)?;
 
-                Ok(Some((value, keychain, script)))
+                Ok(Some(LocalUtxo {
+                    outpoint: OutPoint::new(deserialize(txid)?, vout),
+                    txout: TxOut {
+                        value,
+                        script_pubkey,
+                    },
+                    keychain,
+                    is_spent,
+                }))
             }
             None => Ok(None),
         }
@@ -353,7 +379,7 @@ impl SqliteDatabase {
     }
 
     fn select_transaction_details_with_raw(&self) -> Result<Vec<TransactionDetails>, Error> {
-        let mut statement = self.connection.prepare_cached("SELECT transaction_details.txid, transaction_details.timestamp, transaction_details.received, transaction_details.sent, transaction_details.fee, transaction_details.height, transaction_details.verified, transactions.raw_tx FROM transaction_details, transactions WHERE transaction_details.txid = transactions.txid")?;
+        let mut statement = self.connection.prepare_cached("SELECT transaction_details.txid, transaction_details.timestamp, transaction_details.received, transaction_details.sent, transaction_details.fee, transaction_details.height, transactions.raw_tx FROM transaction_details, transactions WHERE transaction_details.txid = transactions.txid")?;
         let mut transaction_details: Vec<TransactionDetails> = vec![];
         let mut rows = statement.query([])?;
         while let Some(row) = rows.next()? {
@@ -364,7 +390,6 @@ impl SqliteDatabase {
             let sent: u64 = row.get(3)?;
             let fee: Option<u64> = row.get(4)?;
             let height: Option<u32> = row.get(5)?;
-            let verified: bool = row.get(6)?;
             let raw_tx: Option<Vec<u8>> = row.get(7)?;
             let tx: Option<Transaction> = match raw_tx {
                 Some(raw_tx) => {
@@ -375,7 +400,7 @@ impl SqliteDatabase {
             };
 
             let confirmation_time = match (height, timestamp) {
-                (Some(height), Some(timestamp)) => Some(ConfirmationTime { height, timestamp }),
+                (Some(height), Some(timestamp)) => Some(BlockTime { height, timestamp }),
                 _ => None,
             };
 
@@ -386,7 +411,6 @@ impl SqliteDatabase {
                 sent,
                 fee,
                 confirmation_time,
-                verified,
             });
         }
         Ok(transaction_details)
@@ -394,7 +418,7 @@ impl SqliteDatabase {
 
     fn select_transaction_details(&self) -> Result<Vec<TransactionDetails>, Error> {
         let mut statement = self.connection.prepare_cached(
-            "SELECT txid, timestamp, received, sent, fee, height, verified FROM transaction_details",
+            "SELECT txid, timestamp, received, sent, fee, height FROM transaction_details",
         )?;
         let mut transaction_details: Vec<TransactionDetails> = vec![];
         let mut rows = statement.query([])?;
@@ -406,10 +430,9 @@ impl SqliteDatabase {
             let sent: u64 = row.get(3)?;
             let fee: Option<u64> = row.get(4)?;
             let height: Option<u32> = row.get(5)?;
-            let verified: bool = row.get(6)?;
 
             let confirmation_time = match (height, timestamp) {
-                (Some(height), Some(timestamp)) => Some(ConfirmationTime { height, timestamp }),
+                (Some(height), Some(timestamp)) => Some(BlockTime { height, timestamp }),
                 _ => None,
             };
 
@@ -420,7 +443,6 @@ impl SqliteDatabase {
                 sent,
                 fee,
                 confirmation_time,
-                verified,
             });
         }
         Ok(transaction_details)
@@ -430,7 +452,7 @@ impl SqliteDatabase {
         &self,
         txid: &[u8],
     ) -> Result<Option<TransactionDetails>, Error> {
-        let mut statement = self.connection.prepare_cached("SELECT transaction_details.timestamp, transaction_details.received, transaction_details.sent, transaction_details.fee, transaction_details.height, transaction_details.verified, transactions.raw_tx FROM transaction_details, transactions WHERE transaction_details.txid=transactions.txid AND transaction_details.txid=:txid")?;
+        let mut statement = self.connection.prepare_cached("SELECT transaction_details.timestamp, transaction_details.received, transaction_details.sent, transaction_details.fee, transaction_details.height, transactions.raw_tx FROM transaction_details, transactions WHERE transaction_details.txid=transactions.txid AND transaction_details.txid=:txid")?;
         let mut rows = statement.query(named_params! { ":txid": txid })?;
 
         match rows.next()? {
@@ -440,9 +462,8 @@ impl SqliteDatabase {
                 let sent: u64 = row.get(2)?;
                 let fee: Option<u64> = row.get(3)?;
                 let height: Option<u32> = row.get(4)?;
-                let verified: bool = row.get(5)?;
 
-                let raw_tx: Option<Vec<u8>> = row.get(6)?;
+                let raw_tx: Option<Vec<u8>> = row.get(5)?;
                 let tx: Option<Transaction> = match raw_tx {
                     Some(raw_tx) => {
                         let tx: Transaction = deserialize(&raw_tx)?;
@@ -452,7 +473,7 @@ impl SqliteDatabase {
                 };
 
                 let confirmation_time = match (height, timestamp) {
-                    (Some(height), Some(timestamp)) => Some(ConfirmationTime { height, timestamp }),
+                    (Some(height), Some(timestamp)) => Some(BlockTime { height, timestamp }),
                     _ => None,
                 };
 
@@ -463,7 +484,6 @@ impl SqliteDatabase {
                     sent,
                     fee,
                     confirmation_time,
-                    verified,
                 }))
             }
             None => Ok(None),
@@ -487,6 +507,24 @@ impl SqliteDatabase {
         }
     }
 
+    fn select_sync_time(&self) -> Result<Option<SyncTime>, Error> {
+        let mut statement = self
+            .connection
+            .prepare_cached("SELECT height, timestamp FROM sync_time WHERE id = 0")?;
+        let mut rows = statement.query([])?;
+
+        if let Some(row) = rows.next()? {
+            Ok(Some(SyncTime {
+                block_time: BlockTime {
+                    height: row.get(0)?,
+                    timestamp: row.get(1)?,
+                },
+            }))
+        } else {
+            Ok(None)
+        }
+    }
+
     fn select_checksum_by_keychain(&self, keychain: String) -> Result<Option<Vec<u8>>, Error> {
         let mut statement = self
             .connection
@@ -563,6 +601,14 @@ impl SqliteDatabase {
 
         Ok(())
     }
+
+    fn delete_sync_time(&self) -> Result<(), Error> {
+        let mut statement = self
+            .connection
+            .prepare_cached("DELETE FROM sync_time WHERE id = 0")?;
+        statement.execute([])?;
+        Ok(())
+    }
 }
 
 impl BatchOperations for SqliteDatabase {
@@ -584,6 +630,7 @@ impl BatchOperations for SqliteDatabase {
             utxo.outpoint.vout,
             &utxo.outpoint.txid,
             utxo.txout.script_pubkey.as_bytes(),
+            utxo.is_spent,
         )?;
         Ok(())
     }
@@ -622,6 +669,11 @@ impl BatchOperations for SqliteDatabase {
         Ok(())
     }
 
+    fn set_sync_time(&mut self, ct: SyncTime) -> Result<(), Error> {
+        self.update_sync_time(ct)?;
+        Ok(())
+    }
+
     fn del_script_pubkey_from_path(
         &mut self,
         keychain: KeychainKind,
@@ -653,16 +705,9 @@ impl BatchOperations for SqliteDatabase {
 
     fn del_utxo(&mut self, outpoint: &OutPoint) -> Result<Option<LocalUtxo>, Error> {
         match self.select_utxo_by_outpoint(&outpoint.txid, outpoint.vout)? {
-            Some((value, keychain, script_pubkey)) => {
+            Some(local_utxo) => {
                 self.delete_utxo_by_outpoint(&outpoint.txid, outpoint.vout)?;
-                Ok(Some(LocalUtxo {
+                Ok(Some(local_utxo))
-                    outpoint: *outpoint,
-                    txout: TxOut {
-                        value,
-                        script_pubkey,
-                    },
-                    keychain,
-                }))
             }
             None => Ok(None),
         }
@@ -707,6 +752,17 @@ impl BatchOperations for SqliteDatabase {
             None => Ok(None),
         }
     }
+
+    fn del_sync_time(&mut self) -> Result<Option<SyncTime>, Error> {
+        match self.select_sync_time()? {
+            Some(value) => {
+                self.delete_sync_time()?;
+
+                Ok(Some(value))
+            }
+            None => Ok(None),
+        }
+    }
 }
 
 impl Database for SqliteDatabase {
@@ -780,17 +836,7 @@ impl Database for SqliteDatabase {
     }
 
     fn get_utxo(&self, outpoint: &OutPoint) -> Result<Option<LocalUtxo>, Error> {
-        match self.select_utxo_by_outpoint(&outpoint.txid, outpoint.vout)? {
+        self.select_utxo_by_outpoint(&outpoint.txid, outpoint.vout)
-            Some((value, keychain, script_pubkey)) => Ok(Some(LocalUtxo {
-                outpoint: *outpoint,
-                txout: TxOut {
-                    value,
-                    script_pubkey,
-                },
-                keychain,
-            })),
-            None => Ok(None),
-        }
     }
 
     fn get_raw_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error> {
@@ -818,6 +864,10 @@ impl Database for SqliteDatabase {
         Ok(value)
     }
 
+    fn get_sync_time(&self) -> Result<Option<SyncTime>, Error> {
+        self.select_sync_time()
+    }
+
     fn increment_last_index(&mut self, keychain: KeychainKind) -> Result<u32, Error> {
         let keychain_string = serde_json::to_string(&keychain)?;
         match self.get_last_index(keychain)? {
@@ -965,4 +1015,9 @@ pub mod test {
     fn test_last_index() {
         crate::database::test::test_last_index(get_database());
     }
+
+    #[test]
+    fn test_sync_time() {
+        crate::database::test::test_sync_time(get_database());
+    }
 }

src/descriptor/derived.rs

@@ -10,6 +10,41 @@
 // licenses.
 
 //! Derived descriptor keys
+//!
+//! The [`DerivedDescriptorKey`] type is a wrapper over the standard [`DescriptorPublicKey`] which
+//! guarantees that all the extended keys have a fixed derivation path, i.e. all the wildcards have
+//! been replaced by actual derivation indexes.
+//!
+//! The [`AsDerived`] trait provides a quick way to derive descriptors to obtain a
+//! `Descriptor<DerivedDescriptorKey>` type. This, in turn, can be used to derive public
+//! keys for arbitrary derivation indexes.
+//!
+//! Combining this with [`Wallet::get_signers`], secret keys can also be derived.
+//!
+//! # Example
+//!
+//! ```
+//! # use std::str::FromStr;
+//! # use bitcoin::secp256k1::Secp256k1;
+//! use bdk::descriptor::{AsDerived, DescriptorPublicKey};
+//! use bdk::miniscript::{ToPublicKey, TranslatePk, MiniscriptKey};
+//!
+//! let secp = Secp256k1::gen_new();
+//!
+//! let key = DescriptorPublicKey::from_str("[aa600a45/84'/0'/0']tpubDCbDXFKoLTQp44wQuC12JgSn5g9CWGjZdpBHeTqyypZ4VvgYjTJmK9CkyR5bFvG9f4PutvwmvpYCLkFx2rpx25hiMs4sUgxJveW8ZzSAVAc/0/*")?;
+//! let (descriptor, _, _) = bdk::descriptor!(wpkh(key))?;
+//!
+//! // derived: wpkh([aa600a45/84'/0'/0']tpubDCbDXFKoLTQp44wQuC12JgSn5g9CWGjZdpBHeTqyypZ4VvgYjTJmK9CkyR5bFvG9f4PutvwmvpYCLkFx2rpx25hiMs4sUgxJveW8ZzSAVAc/0/42)#3ladd0t2
+//! let derived = descriptor.as_derived(42, &secp);
+//! println!("derived: {}", derived);
+//!
+//! // with_pks: wpkh(02373ecb54c5e83bd7e0d40adf78b65efaf12fafb13571f0261fc90364eee22e1e)#p4jjgvll
+//! let with_pks = derived.translate_pk_infallible(|pk| pk.to_public_key(), |pkh| pkh.to_public_key().to_pubkeyhash());
+//! println!("with_pks: {}", with_pks);
+//! # Ok::<(), Box<dyn std::error::Error>>(())
+//! ```
+//!
+//! [`Wallet::get_signers`]: crate::wallet::Wallet::get_signers
 
 use std::cmp::Ordering;
 use std::fmt;
@@ -19,10 +54,7 @@ use std::ops::Deref;
 use bitcoin::hashes::hash160;
 use bitcoin::PublicKey;
 
-pub use miniscript::{
+use miniscript::{descriptor::Wildcard, Descriptor, DescriptorPublicKey};
-    descriptor::KeyMap, descriptor::Wildcard, Descriptor, DescriptorPublicKey, Legacy, Miniscript,
-    ScriptContext, Segwitv0,
-};
 use miniscript::{MiniscriptKey, ToPublicKey, TranslatePk};
 
 use crate::wallet::utils::SecpCtx;
@@ -119,14 +151,19 @@ impl<'s> ToPublicKey for DerivedDescriptorKey<'s> {
     }
 }
 
-pub(crate) trait AsDerived {
+/// Utilities to derive descriptors
-    // Derive a descriptor and transform all of its keys to `DerivedDescriptorKey`
+///
+/// Check out the [module level] documentation for more.
+///
+/// [module level]: crate::descriptor::derived
+pub trait AsDerived {
+    /// Derive a descriptor and transform all of its keys to `DerivedDescriptorKey`
     fn as_derived<'s>(&self, index: u32, secp: &'s SecpCtx)
         -> Descriptor<DerivedDescriptorKey<'s>>;
 
-    // Transform the keys into `DerivedDescriptorKey`.
+    /// Transform the keys into `DerivedDescriptorKey`.
-    //
+    ///
-    // Panics if the descriptor is not "fixed", i.e. if it's derivable
+    /// Panics if the descriptor is not "fixed", i.e. if it's derivable
     fn as_derived_fixed<'s>(&self, secp: &'s SecpCtx) -> Descriptor<DerivedDescriptorKey<'s>>;
 }
 

src/descriptor/dsl.rs

@@ -84,7 +84,7 @@ macro_rules! impl_leaf_opcode {
         )
         .map_err($crate::descriptor::DescriptorError::Miniscript)
         .and_then(|minisc| {
-            minisc.check_minsicript()?;
+            minisc.check_miniscript()?;
             Ok(minisc)
         })
         .map(|minisc| {
@@ -108,7 +108,7 @@ macro_rules! impl_leaf_opcode_value {
         )
         .map_err($crate::descriptor::DescriptorError::Miniscript)
         .and_then(|minisc| {
-            minisc.check_minsicript()?;
+            minisc.check_miniscript()?;
             Ok(minisc)
         })
         .map(|minisc| {
@@ -132,7 +132,7 @@ macro_rules! impl_leaf_opcode_value_two {
         )
         .map_err($crate::descriptor::DescriptorError::Miniscript)
         .and_then(|minisc| {
-            minisc.check_minsicript()?;
+            minisc.check_miniscript()?;
             Ok(minisc)
         })
         .map(|minisc| {
@@ -165,7 +165,7 @@ macro_rules! impl_node_opcode_two {
                     std::sync::Arc::new(b_minisc),
                 ))?;
 
-                minisc.check_minsicript()?;
+                minisc.check_miniscript()?;
 
                 Ok((minisc, a_keymap, $crate::keys::merge_networks(&a_networks, &b_networks)))
             })
@@ -197,7 +197,7 @@ macro_rules! impl_node_opcode_three {
                     std::sync::Arc::new(c_minisc),
                 ))?;
 
-                minisc.check_minsicript()?;
+                minisc.check_miniscript()?;
 
                 Ok((minisc, a_keymap, networks))
             })
@@ -243,7 +243,7 @@ macro_rules! apply_modifier {
                     ),
                 )?;
 
-                minisc.check_minsicript()?;
+                minisc.check_miniscript()?;
 
                 Ok((minisc, keymap, networks))
             })
@@ -336,7 +336,7 @@ macro_rules! apply_modifier {
 /// syntax is more suitable for a fixed number of items known at compile time, while the other accepts a
 /// [`Vec`] of items, which makes it more suitable for writing dynamic descriptors.
 ///
-/// They both produce the descriptor: `wsh(thresh(2,pk(...),s:pk(...),sdv:older(...)))`
+/// They both produce the descriptor: `wsh(thresh(2,pk(...),s:pk(...),sndv:older(...)))`
 ///
 /// ```
 /// # use std::str::FromStr;
@@ -349,7 +349,7 @@ macro_rules! apply_modifier {
 ///
 /// let (descriptor_a, key_map_a, networks) = bdk::descriptor! {
 ///     wsh (
-///         thresh(2, pk(my_key_1), s:pk(my_key_2), s:d:v:older(my_timelock))
+///         thresh(2, pk(my_key_1), s:pk(my_key_2), s:n:d:v:older(my_timelock))
 ///     )
 /// }?;
 ///
@@ -357,7 +357,7 @@ macro_rules! apply_modifier {
 /// let b_items = vec![
 ///     bdk::fragment!(pk(my_key_1))?,
 ///     bdk::fragment!(s:pk(my_key_2))?,
-///     bdk::fragment!(s:d:v:older(my_timelock))?,
+///     bdk::fragment!(s:n:d:v:older(my_timelock))?,
 /// ];
 /// let (descriptor_b, mut key_map_b, networks) = bdk::descriptor!(wsh(thresh_vec(2, b_items)))?;
 ///
@@ -521,7 +521,7 @@ macro_rules! fragment_internal {
     // three operands it's (X, (X, (X, ()))), etc.
     //
     // To check that the right number of arguments has been passed we can "cast" those tuples to
-    // more convenient structures like `TupleTwo`. If the conversion succedes, the right number of
+    // more convenient structures like `TupleTwo`. If the conversion succeeds, the right number of
     // args was passed. Otherwise the compilation fails entirely.
     ( @t $op:ident ( $( $args:tt )* ) $( $tail:tt )* ) => ({
         ($crate::fragment!( $op ( $( $args )* ) ), $crate::fragment_internal!( @t $( $tail )* ))
@@ -1048,9 +1048,9 @@ mod test {
         let private_key =
             PrivateKey::from_wif("cSQPHDBwXGjVzWRqAHm6zfvQhaTuj1f2bFH58h55ghbjtFwvmeXR").unwrap();
         let (descriptor, _, _) =
-            descriptor!(wsh(thresh(2,d:v:older(1),s:pk(private_key),s:pk(private_key)))).unwrap();
+            descriptor!(wsh(thresh(2,n:d:v:older(1),s:pk(private_key),s:pk(private_key)))).unwrap();
 
-        assert_eq!(descriptor.to_string(), "wsh(thresh(2,dv:older(1),s:pk(02e96fe52ef0e22d2f131dd425ce1893073a3c6ad20e8cac36726393dfb4856a4c),s:pk(02e96fe52ef0e22d2f131dd425ce1893073a3c6ad20e8cac36726393dfb4856a4c)))#cfdcqs3s")
+        assert_eq!(descriptor.to_string(), "wsh(thresh(2,ndv:older(1),s:pk(02e96fe52ef0e22d2f131dd425ce1893073a3c6ad20e8cac36726393dfb4856a4c),s:pk(02e96fe52ef0e22d2f131dd425ce1893073a3c6ad20e8cac36726393dfb4856a4c)))#zzk3ux8g")
     }
 
     #[test]

src/descriptor/mod.rs

@@ -17,20 +17,21 @@
 use std::collections::{BTreeMap, HashMap, HashSet};
 use std::ops::Deref;
 
-use bitcoin::util::bip32::{
+use bitcoin::util::bip32::{ChildNumber, DerivationPath, ExtendedPubKey, Fingerprint, KeySource};
-    ChildNumber, DerivationPath, ExtendedPrivKey, ExtendedPubKey, Fingerprint, KeySource,
-};
 use bitcoin::util::psbt;
 use bitcoin::{Network, PublicKey, Script, TxOut};
 
-use miniscript::descriptor::{DescriptorPublicKey, DescriptorType, DescriptorXKey, Wildcard};
+use miniscript::descriptor::{DescriptorType, InnerXKey};
-pub use miniscript::{descriptor::KeyMap, Descriptor, Legacy, Miniscript, ScriptContext, Segwitv0};
+pub use miniscript::{
+    descriptor::DescriptorXKey, descriptor::KeyMap, descriptor::Wildcard, Descriptor,
+    DescriptorPublicKey, Legacy, Miniscript, ScriptContext, Segwitv0,
+};
 use miniscript::{DescriptorTrait, ForEachKey, TranslatePk};
 
 use crate::descriptor::policy::BuildSatisfaction;
 
 pub mod checksum;
-pub(crate) mod derived;
+pub mod derived;
 #[doc(hidden)]
 pub mod dsl;
 pub mod error;
@@ -38,8 +39,7 @@ pub mod policy;
 pub mod template;
 
 pub use self::checksum::get_checksum;
-use self::derived::AsDerived;
+pub use self::derived::{AsDerived, DerivedDescriptorKey};
-pub use self::derived::DerivedDescriptorKey;
 pub use self::error::Error as DescriptorError;
 pub use self::policy::Policy;
 use self::template::DescriptorTemplateOut;
@@ -238,13 +238,13 @@ pub(crate) fn into_wallet_descriptor_checked<T: IntoWalletDescriptor>(
 #[doc(hidden)]
 /// Used internally mainly by the `descriptor!()` and `fragment!()` macros
 pub trait CheckMiniscript<Ctx: miniscript::ScriptContext> {
-    fn check_minsicript(&self) -> Result<(), miniscript::Error>;
+    fn check_miniscript(&self) -> Result<(), miniscript::Error>;
 }
 
 impl<Ctx: miniscript::ScriptContext, Pk: miniscript::MiniscriptKey> CheckMiniscript<Ctx>
     for miniscript::Miniscript<Pk, Ctx>
 {
-    fn check_minsicript(&self) -> Result<(), miniscript::Error> {
+    fn check_miniscript(&self) -> Result<(), miniscript::Error> {
         Ctx::check_global_validity(self)?;
 
         Ok(())
@@ -267,41 +267,10 @@ pub(crate) trait XKeyUtils {
     fn root_fingerprint(&self, secp: &SecpCtx) -> Fingerprint;
 }
 
-// FIXME: `InnerXKey` was made private in rust-miniscript, so we have to implement this manually on
+impl<T> XKeyUtils for DescriptorXKey<T>
-// both `ExtendedPubKey` and `ExtendedPrivKey`.
+where
-//
+    T: InnerXKey,
-// Revert back to using the trait once https://github.com/rust-bitcoin/rust-miniscript/pull/230 is
+{
-// released
-impl XKeyUtils for DescriptorXKey<ExtendedPubKey> {
-    fn full_path(&self, append: &[ChildNumber]) -> DerivationPath {
-        let full_path = match self.origin {
-            Some((_, ref path)) => path
-                .into_iter()
-                .chain(self.derivation_path.into_iter())
-                .cloned()
-                .collect(),
-            None => self.derivation_path.clone(),
-        };
-
-        if self.wildcard != Wildcard::None {
-            full_path
-                .into_iter()
-                .chain(append.iter())
-                .cloned()
-                .collect()
-        } else {
-            full_path
-        }
-    }
-
-    fn root_fingerprint(&self, _: &SecpCtx) -> Fingerprint {
-        match self.origin {
-            Some((fingerprint, _)) => fingerprint,
-            None => self.xkey.fingerprint(),
-        }
-    }
-}
-impl XKeyUtils for DescriptorXKey<ExtendedPrivKey> {
     fn full_path(&self, append: &[ChildNumber]) -> DerivationPath {
         let full_path = match self.origin {
             Some((_, ref path)) => path
@@ -326,7 +295,7 @@ impl XKeyUtils for DescriptorXKey<ExtendedPrivKey> {
     fn root_fingerprint(&self, secp: &SecpCtx) -> Fingerprint {
         match self.origin {
             Some((fingerprint, _)) => fingerprint,
-            None => self.xkey.fingerprint(secp),
+            None => self.xkey.xkey_fingerprint(secp),
         }
     }
 }
@@ -667,7 +636,7 @@ mod test {
 
         // make a descriptor out of it
         let desc = crate::descriptor!(wpkh(key)).unwrap();
-        // this should conver the key that supports "any_network" to the right network (testnet)
+        // this should convert the key that supports "any_network" to the right network (testnet)
         let (wallet_desc, _) = desc
             .into_wallet_descriptor(&secp, Network::Testnet)
             .unwrap();

src/descriptor/policy.rs

@@ -354,9 +354,9 @@ impl Satisfaction {
                 indexes
                     .into_iter()
                     // .inspect(|x| println!("--- orig --- {:?}", x))
-                    // we map each of the combinations of elements into a tuple of ([choosen items], [conditions]). unfortunately, those items have potentially more than one
+                    // we map each of the combinations of elements into a tuple of ([chosen items], [conditions]). unfortunately, those items have potentially more than one
                     // condition (think about ORs), so we also use `mix` to expand those, i.e. [[0], [1, 2]] becomes [[0, 1], [0, 2]]. This is necessary to make sure that we
-                    // consider every possibile options and check whether or not they are compatible.
+                    // consider every possible options and check whether or not they are compatible.
                     .map(|i_vec| {
                         mix(i_vec
                             .iter()
@@ -1501,7 +1501,7 @@ mod test {
         let (prvkey_bob, _, _) = setup_keys(BOB_TPRV_STR, ALICE_BOB_PATH, &secp);
 
         let desc =
-            descriptor!(wsh(thresh(2,d:v:older(2),s:pk(prvkey_alice),s:pk(prvkey_bob)))).unwrap();
+            descriptor!(wsh(thresh(2,n:d:v:older(2),s:pk(prvkey_alice),s:pk(prvkey_bob)))).unwrap();
 
         let (wallet_desc, keymap) = desc
             .into_wallet_descriptor(&secp, Network::Testnet)
@@ -1513,7 +1513,7 @@ mod test {
             .address(Network::Testnet)
             .unwrap();
         assert_eq!(
-            "tb1qhpemaacpeu8ajlnh8k9v55ftg0px58r8630fz8t5mypxcwdk5d8sum522g",
+            "tb1qsydsey4hexagwkvercqsmes6yet0ndkyt6uzcphtqnygjd8hmzmsfxrv58",
             addr.to_string()
         );
 

src/descriptor/template.rs

@@ -79,7 +79,7 @@ impl<T: DescriptorTemplate> IntoWalletDescriptor for T {
 ///
 /// let key =
 ///     bitcoin::PrivateKey::from_wif("cTc4vURSzdx6QE6KVynWGomDbLaA75dNALMNyfjh3p8DRRar84Um")?;
-/// let wallet = Wallet::new_offline(
+/// let wallet = Wallet::new(
 ///     P2Pkh(key),
 ///     None,
 ///     Network::Testnet,
@@ -113,7 +113,7 @@ impl<K: IntoDescriptorKey<Legacy>> DescriptorTemplate for P2Pkh<K> {
 ///
 /// let key =
 ///     bitcoin::PrivateKey::from_wif("cTc4vURSzdx6QE6KVynWGomDbLaA75dNALMNyfjh3p8DRRar84Um")?;
-/// let wallet = Wallet::new_offline(
+/// let wallet = Wallet::new(
 ///     P2Wpkh_P2Sh(key),
 ///     None,
 ///     Network::Testnet,
@@ -148,7 +148,7 @@ impl<K: IntoDescriptorKey<Segwitv0>> DescriptorTemplate for P2Wpkh_P2Sh<K> {
 ///
 /// let key =
 ///     bitcoin::PrivateKey::from_wif("cTc4vURSzdx6QE6KVynWGomDbLaA75dNALMNyfjh3p8DRRar84Um")?;
-/// let wallet = Wallet::new_offline(
+/// let wallet = Wallet::new(
 ///     P2Wpkh(key),
 ///     None,
 ///     Network::Testnet,
@@ -186,7 +186,7 @@ impl<K: IntoDescriptorKey<Segwitv0>> DescriptorTemplate for P2Wpkh<K> {
 /// use bdk::template::Bip44;
 ///
 /// let key = bitcoin::util::bip32::ExtendedPrivKey::from_str("tprv8ZgxMBicQKsPeZRHk4rTG6orPS2CRNFX3njhUXx5vj9qGog5ZMH4uGReDWN5kCkY3jmWEtWause41CDvBRXD1shKknAMKxT99o9qUTRVC6m")?;
-/// let wallet = Wallet::new_offline(
+/// let wallet = Wallet::new(
 ///     Bip44(key.clone(), KeychainKind::External),
 ///     Some(Bip44(key, KeychainKind::Internal)),
 ///     Network::Testnet,
@@ -226,7 +226,7 @@ impl<K: DerivableKey<Legacy>> DescriptorTemplate for Bip44<K> {
 ///
 /// let key = bitcoin::util::bip32::ExtendedPubKey::from_str("tpubDDDzQ31JkZB7VxUr9bjvBivDdqoFLrDPyLWtLapArAi51ftfmCb2DPxwLQzX65iNcXz1DGaVvyvo6JQ6rTU73r2gqdEo8uov9QKRb7nKCSU")?;
 /// let fingerprint = bitcoin::util::bip32::Fingerprint::from_str("c55b303f")?;
-/// let wallet = Wallet::new_offline(
+/// let wallet = Wallet::new(
 ///     Bip44Public(key.clone(), fingerprint, KeychainKind::External),
 ///     Some(Bip44Public(key, fingerprint, KeychainKind::Internal)),
 ///     Network::Testnet,
@@ -262,7 +262,7 @@ impl<K: DerivableKey<Legacy>> DescriptorTemplate for Bip44Public<K> {
 /// use bdk::template::Bip49;
 ///
 /// let key = bitcoin::util::bip32::ExtendedPrivKey::from_str("tprv8ZgxMBicQKsPeZRHk4rTG6orPS2CRNFX3njhUXx5vj9qGog5ZMH4uGReDWN5kCkY3jmWEtWause41CDvBRXD1shKknAMKxT99o9qUTRVC6m")?;
-/// let wallet = Wallet::new_offline(
+/// let wallet = Wallet::new(
 ///     Bip49(key.clone(), KeychainKind::External),
 ///     Some(Bip49(key, KeychainKind::Internal)),
 ///     Network::Testnet,
@@ -302,7 +302,7 @@ impl<K: DerivableKey<Segwitv0>> DescriptorTemplate for Bip49<K> {
 ///
 /// let key = bitcoin::util::bip32::ExtendedPubKey::from_str("tpubDC49r947KGK52X5rBWS4BLs5m9SRY3pYHnvRrm7HcybZ3BfdEsGFyzCMzayi1u58eT82ZeyFZwH7DD6Q83E3fM9CpfMtmnTygnLfP59jL9L")?;
 /// let fingerprint = bitcoin::util::bip32::Fingerprint::from_str("c55b303f")?;
-/// let wallet = Wallet::new_offline(
+/// let wallet = Wallet::new(
 ///     Bip49Public(key.clone(), fingerprint, KeychainKind::External),
 ///     Some(Bip49Public(key, fingerprint, KeychainKind::Internal)),
 ///     Network::Testnet,
@@ -338,7 +338,7 @@ impl<K: DerivableKey<Segwitv0>> DescriptorTemplate for Bip49Public<K> {
 /// use bdk::template::Bip84;
 ///
 /// let key = bitcoin::util::bip32::ExtendedPrivKey::from_str("tprv8ZgxMBicQKsPeZRHk4rTG6orPS2CRNFX3njhUXx5vj9qGog5ZMH4uGReDWN5kCkY3jmWEtWause41CDvBRXD1shKknAMKxT99o9qUTRVC6m")?;
-/// let wallet = Wallet::new_offline(
+/// let wallet = Wallet::new(
 ///     Bip84(key.clone(), KeychainKind::External),
 ///     Some(Bip84(key, KeychainKind::Internal)),
 ///     Network::Testnet,
@@ -378,7 +378,7 @@ impl<K: DerivableKey<Segwitv0>> DescriptorTemplate for Bip84<K> {
 ///
 /// let key = bitcoin::util::bip32::ExtendedPubKey::from_str("tpubDC2Qwo2TFsaNC4ju8nrUJ9mqVT3eSgdmy1yPqhgkjwmke3PRXutNGRYAUo6RCHTcVQaDR3ohNU9we59brGHuEKPvH1ags2nevW5opEE9Z5Q")?;
 /// let fingerprint = bitcoin::util::bip32::Fingerprint::from_str("c55b303f")?;
-/// let wallet = Wallet::new_offline(
+/// let wallet = Wallet::new(
 ///     Bip84Public(key.clone(), fingerprint, KeychainKind::External),
 ///     Some(Bip84Public(key, fingerprint, KeychainKind::Internal)),
 ///     Network::Testnet,

src/error.rs

@@ -115,7 +115,7 @@ pub enum Error {
     Hex(bitcoin::hashes::hex::Error),
     /// Partially signed bitcoin transaction error
     Psbt(bitcoin::util::psbt::Error),
-    /// Partially signed bitcoin transaction parseerror
+    /// Partially signed bitcoin transaction parse error
     PsbtParse(bitcoin::util::psbt::PsbtParseError),
 
     //KeyMismatch(bitcoin::secp256k1::PublicKey, bitcoin::secp256k1::PublicKey),
@@ -139,7 +139,7 @@ pub enum Error {
     Sled(sled::Error),
     #[cfg(feature = "rpc")]
     /// Rpc client error
-    Rpc(core_rpc::Error),
+    Rpc(bitcoincore_rpc::Error),
     #[cfg(feature = "sqlite")]
     /// Rusqlite client error
     Rusqlite(rusqlite::Error),
@@ -196,7 +196,7 @@ impl_error!(electrum_client::Error, Electrum);
 #[cfg(feature = "key-value-db")]
 impl_error!(sled::Error, Sled);
 #[cfg(feature = "rpc")]
-impl_error!(core_rpc::Error, Rpc);
+impl_error!(bitcoincore_rpc::Error, Rpc);
 #[cfg(feature = "sqlite")]
 impl_error!(rusqlite::Error, Rusqlite);
 

src/keys/bip39.rs

@@ -19,7 +19,23 @@ use bitcoin::Network;
 
 use miniscript::ScriptContext;
 
-pub use bip39::{Language, Mnemonic, MnemonicType, Seed};
+pub use bip39::{Language, Mnemonic};
+
+type Seed = [u8; 64];
+
+/// Type describing entropy length (aka word count) in the mnemonic
+pub enum WordCount {
+    /// 12 words mnemonic (128 bits entropy)
+    Words12 = 128,
+    /// 15 words mnemonic (160 bits entropy)
+    Words15 = 160,
+    /// 18 words mnemonic (192 bits entropy)
+    Words18 = 192,
+    /// 21 words mnemonic (224 bits entropy)
+    Words21 = 224,
+    /// 24 words mnemonic (256 bits entropy)
+    Words24 = 256,
+}
 
 use super::{
     any_network, DerivableKey, DescriptorKey, ExtendedKey, GeneratableKey, GeneratedKey, KeyError,
@@ -40,7 +56,7 @@ pub type MnemonicWithPassphrase = (Mnemonic, Option<String>);
 #[cfg_attr(docsrs, doc(cfg(feature = "keys-bip39")))]
 impl<Ctx: ScriptContext> DerivableKey<Ctx> for Seed {
     fn into_extended_key(self) -> Result<ExtendedKey<Ctx>, KeyError> {
-        Ok(bip32::ExtendedPrivKey::new_master(Network::Bitcoin, &self.as_bytes())?.into())
+        Ok(bip32::ExtendedPrivKey::new_master(Network::Bitcoin, &self[..])?.into())
     }
 
     fn into_descriptor_key(
@@ -60,7 +76,7 @@ impl<Ctx: ScriptContext> DerivableKey<Ctx> for Seed {
 impl<Ctx: ScriptContext> DerivableKey<Ctx> for MnemonicWithPassphrase {
     fn into_extended_key(self) -> Result<ExtendedKey<Ctx>, KeyError> {
         let (mnemonic, passphrase) = self;
-        let seed = Seed::new(&mnemonic, passphrase.as_deref().unwrap_or(""));
+        let seed: Seed = mnemonic.to_seed(passphrase.as_deref().unwrap_or(""));
 
         seed.into_extended_key()
     }
@@ -78,6 +94,23 @@ impl<Ctx: ScriptContext> DerivableKey<Ctx> for MnemonicWithPassphrase {
     }
 }
 
+#[cfg_attr(docsrs, doc(cfg(feature = "keys-bip39")))]
+impl<Ctx: ScriptContext> DerivableKey<Ctx> for (GeneratedKey<Mnemonic, Ctx>, Option<String>) {
+    fn into_extended_key(self) -> Result<ExtendedKey<Ctx>, KeyError> {
+        let (mnemonic, passphrase) = self;
+        (mnemonic.into_key(), passphrase).into_extended_key()
+    }
+
+    fn into_descriptor_key(
+        self,
+        source: Option<bip32::KeySource>,
+        derivation_path: bip32::DerivationPath,
+    ) -> Result<DescriptorKey<Ctx>, KeyError> {
+        let (mnemonic, passphrase) = self;
+        (mnemonic.into_key(), passphrase).into_descriptor_key(source, derivation_path)
+    }
+}
+
 #[cfg_attr(docsrs, doc(cfg(feature = "keys-bip39")))]
 impl<Ctx: ScriptContext> DerivableKey<Ctx> for Mnemonic {
     fn into_extended_key(self) -> Result<ExtendedKey<Ctx>, KeyError> {
@@ -101,15 +134,15 @@ impl<Ctx: ScriptContext> DerivableKey<Ctx> for Mnemonic {
 impl<Ctx: ScriptContext> GeneratableKey<Ctx> for Mnemonic {
     type Entropy = [u8; 32];
 
-    type Options = (MnemonicType, Language);
+    type Options = (WordCount, Language);
-    type Error = Option<bip39::ErrorKind>;
+    type Error = Option<bip39::Error>;
 
     fn generate_with_entropy(
-        (mnemonic_type, language): Self::Options,
+        (word_count, language): Self::Options,
         entropy: Self::Entropy,
     ) -> Result<GeneratedKey<Self, Ctx>, Self::Error> {
-        let entropy = &entropy.as_ref()[..(mnemonic_type.entropy_bits() / 8)];
+        let entropy = &entropy.as_ref()[..(word_count as usize / 8)];
-        let mnemonic = Mnemonic::from_entropy(entropy, language).map_err(|e| e.downcast().ok())?;
+        let mnemonic = Mnemonic::from_entropy_in(language, entropy)?;
 
         Ok(GeneratedKey::new(mnemonic, any_network()))
     }
@@ -121,15 +154,17 @@ mod test {
 
     use bitcoin::util::bip32;
 
-    use bip39::{Language, Mnemonic, MnemonicType};
+    use bip39::{Language, Mnemonic};
 
     use crate::keys::{any_network, GeneratableKey, GeneratedKey};
 
+    use super::WordCount;
+
     #[test]
     fn test_keys_bip39_mnemonic() {
         let mnemonic =
             "aim bunker wash balance finish force paper analyst cabin spoon stable organ";
-        let mnemonic = Mnemonic::from_phrase(mnemonic, Language::English).unwrap();
+        let mnemonic = Mnemonic::parse_in(Language::English, mnemonic).unwrap();
         let path = bip32::DerivationPath::from_str("m/44'/0'/0'/0").unwrap();
 
         let key = (mnemonic, path);
@@ -143,7 +178,7 @@ mod test {
     fn test_keys_bip39_mnemonic_passphrase() {
         let mnemonic =
             "aim bunker wash balance finish force paper analyst cabin spoon stable organ";
-        let mnemonic = Mnemonic::from_phrase(mnemonic, Language::English).unwrap();
+        let mnemonic = Mnemonic::parse_in(Language::English, mnemonic).unwrap();
         let path = bip32::DerivationPath::from_str("m/44'/0'/0'/0").unwrap();
 
         let key = ((mnemonic, Some("passphrase".into())), path);
@@ -157,7 +192,7 @@ mod test {
     fn test_keys_generate_bip39() {
         let generated_mnemonic: GeneratedKey<_, miniscript::Segwitv0> =
             Mnemonic::generate_with_entropy(
-                (MnemonicType::Words12, Language::English),
+                (WordCount::Words12, Language::English),
                 crate::keys::test::TEST_ENTROPY,
             )
             .unwrap();
@@ -169,7 +204,7 @@ mod test {
 
         let generated_mnemonic: GeneratedKey<_, miniscript::Segwitv0> =
             Mnemonic::generate_with_entropy(
-                (MnemonicType::Words24, Language::English),
+                (WordCount::Words24, Language::English),
                 crate::keys::test::TEST_ENTROPY,
             )
             .unwrap();
@@ -180,11 +215,11 @@ mod test {
     #[test]
     fn test_keys_generate_bip39_random() {
         let generated_mnemonic: GeneratedKey<_, miniscript::Segwitv0> =
-            Mnemonic::generate((MnemonicType::Words12, Language::English)).unwrap();
+            Mnemonic::generate((WordCount::Words12, Language::English)).unwrap();
         assert_eq!(generated_mnemonic.valid_networks, any_network());
 
         let generated_mnemonic: GeneratedKey<_, miniscript::Segwitv0> =
-            Mnemonic::generate((MnemonicType::Words24, Language::English)).unwrap();
+            Mnemonic::generate((WordCount::Words24, Language::English)).unwrap();
         assert_eq!(generated_mnemonic.valid_networks, any_network());
     }
 }

src/keys/mod.rs

@@ -319,6 +319,7 @@ impl<Ctx: ScriptContext> ExtendedKey<Ctx> {
         match self {
             ExtendedKey::Private((mut xprv, _)) => {
                 xprv.network = network;
+                xprv.private_key.network = network;
                 Some(xprv)
             }
             ExtendedKey::Public(_) => None,
@@ -356,7 +357,7 @@ impl<Ctx: ScriptContext> From<bip32::ExtendedPrivKey> for ExtendedKey<Ctx> {
 
 /// Trait for keys that can be derived.
 ///
-/// When extra metadata are provided, a [`DerivableKey`] can be transofrmed into a
+/// When extra metadata are provided, a [`DerivableKey`] can be transformed into a
 /// [`DescriptorKey`]: the trait [`IntoDescriptorKey`] is automatically implemented
 /// for `(DerivableKey, DerivationPath)` and
 /// `(DerivableKey, KeySource, DerivationPath)` tuples.
@@ -460,9 +461,9 @@ use bdk::keys::bip39::{Mnemonic, Language};
 
 # fn main() -> Result<(), Box<dyn std::error::Error>> {
 let xkey: ExtendedKey =
-    Mnemonic::from_phrase(
+    Mnemonic::parse_in(
+        Language::English,
         "jelly crash boy whisper mouse ecology tuna soccer memory million news short",
-        Language::English
     )?
     .into_extended_key()?;
 let xprv = xkey.into_xprv(Network::Bitcoin).unwrap();
@@ -547,6 +548,16 @@ impl<K, Ctx: ScriptContext> Deref for GeneratedKey<K, Ctx> {
     }
 }
 
+impl<K: Clone, Ctx: ScriptContext> Clone for GeneratedKey<K, Ctx> {
+    fn clone(&self) -> GeneratedKey<K, Ctx> {
+        GeneratedKey {
+            key: self.key.clone(),
+            valid_networks: self.valid_networks.clone(),
+            phantom: self.phantom,
+        }
+    }
+}
+
 // Make generated "derivable" keys themselves "derivable". Also make sure they are assigned the
 // right `valid_networks`.
 impl<Ctx, K> DerivableKey<Ctx> for GeneratedKey<K, Ctx>
@@ -748,7 +759,7 @@ pub fn make_pk<Pk: IntoDescriptorKey<Ctx>, Ctx: ScriptContext>(
     let (key, key_map, valid_networks) = descriptor_key.into_descriptor_key()?.extract(secp)?;
     let minisc = Miniscript::from_ast(Terminal::PkK(key))?;
 
-    minisc.check_minsicript()?;
+    minisc.check_miniscript()?;
 
     Ok((minisc, key_map, valid_networks))
 }
@@ -762,7 +773,7 @@ pub fn make_pkh<Pk: IntoDescriptorKey<Ctx>, Ctx: ScriptContext>(
     let (key, key_map, valid_networks) = descriptor_key.into_descriptor_key()?.extract(secp)?;
     let minisc = Miniscript::from_ast(Terminal::PkH(key))?;
 
-    minisc.check_minsicript()?;
+    minisc.check_miniscript()?;
 
     Ok((minisc, key_map, valid_networks))
 }
@@ -777,7 +788,7 @@ pub fn make_multi<Pk: IntoDescriptorKey<Ctx>, Ctx: ScriptContext>(
     let (pks, key_map, valid_networks) = expand_multi_keys(pks, secp)?;
     let minisc = Miniscript::from_ast(Terminal::Multi(thresh, pks))?;
 
-    minisc.check_minsicript()?;
+    minisc.check_miniscript()?;
 
     Ok((minisc, key_map, valid_networks))
 }
@@ -931,4 +942,43 @@ pub mod test {
             "L2wTu6hQrnDMiFNWA5na6jB12ErGQqtXwqpSL7aWquJaZG8Ai3ch"
         );
     }
+
+    #[test]
+    fn test_keys_wif_network() {
+        // test mainnet wif
+        let generated_xprv: GeneratedKey<_, miniscript::Segwitv0> =
+            bip32::ExtendedPrivKey::generate_with_entropy_default(TEST_ENTROPY).unwrap();
+        let xkey = generated_xprv.into_extended_key().unwrap();
+
+        let network = Network::Bitcoin;
+        let xprv = xkey.into_xprv(network).unwrap();
+        let wif = PrivateKey::from_wif(&xprv.private_key.to_wif()).unwrap();
+        assert_eq!(wif.network, network);
+
+        // test testnet wif
+        let generated_xprv: GeneratedKey<_, miniscript::Segwitv0> =
+            bip32::ExtendedPrivKey::generate_with_entropy_default(TEST_ENTROPY).unwrap();
+        let xkey = generated_xprv.into_extended_key().unwrap();
+
+        let network = Network::Testnet;
+        let xprv = xkey.into_xprv(network).unwrap();
+        let wif = PrivateKey::from_wif(&xprv.private_key.to_wif()).unwrap();
+        assert_eq!(wif.network, network);
+    }
+
+    #[cfg(feature = "keys-bip39")]
+    #[test]
+    fn test_keys_wif_network_bip39() {
+        let xkey: ExtendedKey = bip39::Mnemonic::parse_in(
+            bip39::Language::English,
+            "jelly crash boy whisper mouse ecology tuna soccer memory million news short",
+        )
+        .unwrap()
+        .into_extended_key()
+        .unwrap();
+        let xprv = xkey.into_xprv(Network::Testnet).unwrap();
+        let wif = PrivateKey::from_wif(&xprv.private_key.to_wif()).unwrap();
+
+        assert_eq!(wif.network, Network::Testnet);
+    }
 }

src/lib.rs

@@ -14,6 +14,10 @@
 // only enables the `doc_cfg` feature when
 // the `docsrs` configuration attribute is defined
 #![cfg_attr(docsrs, feature(doc_cfg))]
+#![cfg_attr(
+    docsrs,
+    doc(html_logo_url = "https://github.com/bitcoindevkit/bdk/raw/master/static/bdk.png")
+)]
 
 //! A modern, lightweight, descriptor-based wallet library written in Rust.
 //!
@@ -40,31 +44,32 @@
 //! interact with the bitcoin P2P network.
 //!
 //! ```toml
-//! bdk = "0.12.0"
+//! bdk = "0.18.0"
 //! ```
+//!
+//! # Examples
 #![cfg_attr(
     feature = "electrum",
     doc = r##"
 ## Sync the balance of a descriptor
 
-### Example
 ```no_run
-use bdk::Wallet;
+use bdk::{Wallet, SyncOptions};
 use bdk::database::MemoryDatabase;
-use bdk::blockchain::{noop_progress, ElectrumBlockchain};
+use bdk::blockchain::ElectrumBlockchain;
 use bdk::electrum_client::Client;
 
 fn main() -> Result<(), bdk::Error> {
     let client = Client::new("ssl://electrum.blockstream.info:60002")?;
+    let blockchain = ElectrumBlockchain::from(client);
     let wallet = Wallet::new(
         "wpkh([c258d2e4/84h/1h/0h]tpubDDYkZojQFQjht8Tm4jsS3iuEmKjTiEGjG6KnuFNKKJb5A6ZUCUZKdvLdSDWofKi4ToRCwb9poe1XdqfUnP4jaJjCB2Zwv11ZLgSbnZSNecE/0/*)",
         Some("wpkh([c258d2e4/84h/1h/0h]tpubDDYkZojQFQjht8Tm4jsS3iuEmKjTiEGjG6KnuFNKKJb5A6ZUCUZKdvLdSDWofKi4ToRCwb9poe1XdqfUnP4jaJjCB2Zwv11ZLgSbnZSNecE/1/*)"),
         bitcoin::Network::Testnet,
         MemoryDatabase::default(),
-        ElectrumBlockchain::from(client)
     )?;
 
-    wallet.sync(noop_progress(), None)?;
+    wallet.sync(&blockchain, SyncOptions::default())?;
 
     println!("Descriptor balance: {} SAT", wallet.get_balance()?);
 
@@ -76,14 +81,13 @@ fn main() -> Result<(), bdk::Error> {
 //!
 //! ## Generate a few addresses
 //!
-//! ### Example
 //! ```
 //! use bdk::{Wallet};
 //! use bdk::database::MemoryDatabase;
 //! use bdk::wallet::AddressIndex::New;
 //!
 //! fn main() -> Result<(), bdk::Error> {
-//! let wallet = Wallet::new_offline(
+//! let wallet = Wallet::new(
 //!         "wpkh([c258d2e4/84h/1h/0h]tpubDDYkZojQFQjht8Tm4jsS3iuEmKjTiEGjG6KnuFNKKJb5A6ZUCUZKdvLdSDWofKi4ToRCwb9poe1XdqfUnP4jaJjCB2Zwv11ZLgSbnZSNecE/0/*)",
 //!         Some("wpkh([c258d2e4/84h/1h/0h]tpubDDYkZojQFQjht8Tm4jsS3iuEmKjTiEGjG6KnuFNKKJb5A6ZUCUZKdvLdSDWofKi4ToRCwb9poe1XdqfUnP4jaJjCB2Zwv11ZLgSbnZSNecE/1/*)"),
 //!         bitcoin::Network::Testnet,
@@ -102,11 +106,10 @@ fn main() -> Result<(), bdk::Error> {
     doc = r##"
 ## Create a transaction
 
-### Example
 ```no_run
-use bdk::{FeeRate, Wallet};
+use bdk::{FeeRate, Wallet, SyncOptions};
 use bdk::database::MemoryDatabase;
-use bdk::blockchain::{noop_progress, ElectrumBlockchain};
+use bdk::blockchain::ElectrumBlockchain;
 use bdk::electrum_client::Client;
 
 use bitcoin::consensus::serialize;
@@ -119,10 +122,10 @@ fn main() -> Result<(), bdk::Error> {
         Some("wpkh([c258d2e4/84h/1h/0h]tpubDDYkZojQFQjht8Tm4jsS3iuEmKjTiEGjG6KnuFNKKJb5A6ZUCUZKdvLdSDWofKi4ToRCwb9poe1XdqfUnP4jaJjCB2Zwv11ZLgSbnZSNecE/1/*)"),
         bitcoin::Network::Testnet,
         MemoryDatabase::default(),
-        ElectrumBlockchain::from(client)
     )?;
+    let blockchain = ElectrumBlockchain::from(client);
 
-    wallet.sync(noop_progress(), None)?;
+    wallet.sync(&blockchain, SyncOptions::default())?;
 
     let send_to = wallet.get_address(New)?;
     let (psbt, details) = {
@@ -146,7 +149,6 @@ fn main() -> Result<(), bdk::Error> {
 //!
 //! ## Sign a transaction
 //!
-//! ### Example
 //! ```no_run
 //! use std::str::FromStr;
 //!
@@ -156,7 +158,7 @@ fn main() -> Result<(), bdk::Error> {
 //! use bdk::database::MemoryDatabase;
 //!
 //! fn main() -> Result<(), bdk::Error> {
-//!     let wallet = Wallet::new_offline(
+//!     let wallet = Wallet::new(
 //!         "wpkh([c258d2e4/84h/1h/0h]tprv8griRPhA7342zfRyB6CqeKF8CJDXYu5pgnj1cjL1u2ngKcJha5jjTRimG82ABzJQ4MQe71CV54xfn25BbhCNfEGGJZnxvCDQCd6JkbvxW6h/0/*)",
 //!         Some("wpkh([c258d2e4/84h/1h/0h]tprv8griRPhA7342zfRyB6CqeKF8CJDXYu5pgnj1cjL1u2ngKcJha5jjTRimG82ABzJQ4MQe71CV54xfn25BbhCNfEGGJZnxvCDQCd6JkbvxW6h/1/*)"),
 //!         bitcoin::Network::Testnet,
@@ -188,7 +190,7 @@ fn main() -> Result<(), bdk::Error> {
 //! * `async-interface`: async functions in bdk traits
 //! * `keys-bip39`: [BIP-39](https://github.com/bitcoin/bips/blob/master/bip-0039.mediawiki) mnemonic codes for generating deterministic keys
 //!
-//! ## Internal features
+//! # Internal features
 //!
 //! These features do not expose any new API, but influence internal implementation aspects of
 //! BDK.
@@ -236,7 +238,7 @@ extern crate bdk_macros;
 extern crate lazy_static;
 
 #[cfg(feature = "rpc")]
-pub extern crate core_rpc;
+pub extern crate bitcoincore_rpc;
 
 #[cfg(feature = "electrum")]
 pub extern crate electrum_client;
@@ -268,6 +270,7 @@ pub use wallet::address_validator;
 pub use wallet::signer;
 pub use wallet::signer::SignOptions;
 pub use wallet::tx_builder::TxBuilder;
+pub use wallet::SyncOptions;
 pub use wallet::Wallet;
 
 /// Get the version of BDK at runtime
@@ -275,7 +278,7 @@ pub fn version() -> &'static str {
     env!("CARGO_PKG_VERSION", "unknown")
 }
 
-// We should consider putting this under a feature flag but we need the macro in doctets so we need
+// We should consider putting this under a feature flag but we need the macro in doctests so we need
 // to wait until https://github.com/rust-lang/rust/issues/67295 is fixed.
 //
 // Stuff in here is too rough to document atm

src/testutils/blockchain_tests.rs

@@ -3,11 +3,11 @@ use bitcoin::consensus::encode::{deserialize, serialize};
 use bitcoin::hashes::hex::{FromHex, ToHex};
 use bitcoin::hashes::sha256d;
 use bitcoin::{Address, Amount, Script, Transaction, Txid};
+pub use bitcoincore_rpc::bitcoincore_rpc_json::AddressType;
+pub use bitcoincore_rpc::{Auth, Client as RpcClient, RpcApi};
 use core::str::FromStr;
-pub use core_rpc::core_rpc_json::AddressType;
-pub use core_rpc::{Auth, Client as RpcClient, RpcApi};
 use electrsd::bitcoind::BitcoinD;
-use electrsd::{bitcoind, Conf, ElectrsD};
+use electrsd::{bitcoind, ElectrsD};
 pub use electrum_client::{Client as ElectrumClient, ElectrumApi};
 #[allow(unused_imports)]
 use log::{debug, error, info, log_enabled, trace, Level};
@@ -24,19 +24,15 @@ pub struct TestClient {
 impl TestClient {
     pub fn new(bitcoind_exe: String, electrs_exe: String) -> Self {
         debug!("launching {} and {}", &bitcoind_exe, &electrs_exe);
-        let conf = bitcoind::Conf {
+
-            view_stdout: log_enabled!(Level::Debug),
+        let mut conf = bitcoind::Conf::default();
-            ..Default::default()
+        conf.view_stdout = log_enabled!(Level::Debug);
-        };
         let bitcoind = BitcoinD::with_conf(bitcoind_exe, &conf).unwrap();
 
-        let http_enabled = cfg!(feature = "test-esplora");
+        let mut conf = electrsd::Conf::default();
+        conf.view_stderr = log_enabled!(Level::Debug);
+        conf.http_enabled = cfg!(feature = "test-esplora");
 
-        let conf = Conf {
-            http_enabled,
-            view_stderr: log_enabled!(Level::Debug),
-            ..Default::default()
-        };
         let electrsd = ElectrsD::with_conf(electrs_exe, &bitcoind, &conf).unwrap();
 
         let node_address = bitcoind.client.get_new_address(None, None).unwrap();
@@ -94,13 +90,19 @@ impl TestClient {
             map.insert(out.to_address.clone(), Amount::from_sat(out.value));
         }
 
+        let input: Vec<_> = meta_tx
+            .input
+            .into_iter()
+            .map(|x| x.into_raw_tx_input())
+            .collect();
+
         if self.get_balance(None, None).unwrap() < Amount::from_sat(required_balance) {
             panic!("Insufficient funds in bitcoind. Please generate a few blocks with: `bitcoin-cli generatetoaddress 10 {}`", self.get_new_address(None, None).unwrap());
         }
 
         // FIXME: core can't create a tx with two outputs to the same address
         let tx = self
-            .create_raw_transaction_hex(&[], &map, meta_tx.locktime, meta_tx.replaceable)
+            .create_raw_transaction_hex(&input, &map, meta_tx.locktime, meta_tx.replaceable)
             .unwrap();
         let tx = self.fund_raw_transaction(tx, None, None).unwrap();
         let mut tx: Transaction = deserialize(&tx.hex).unwrap();
@@ -149,9 +151,7 @@ impl TestClient {
 
         let bumped: serde_json::Value = self.call("bumpfee", &[txid.to_string().into()]).unwrap();
         let new_txid = Txid::from_str(&bumped["txid"].as_str().unwrap().to_string()).unwrap();
-
+        let monitor_script = Script::from_hex(&mut tx.vout[0].script_pub_key.hex.to_hex()).unwrap();
-        let monitor_script =
-            tx.vout[0].script_pub_key.addresses.as_ref().unwrap()[0].script_pubkey();
         self.wait_for_tx(new_txid, &monitor_script);
 
         debug!("Bumped {}, new txid {}", txid, new_txid);
@@ -320,7 +320,7 @@ impl Default for TestClient {
     fn default() -> Self {
         let bitcoind_exe = env::var("BITCOIND_EXE")
             .ok()
-            .or(bitcoind::downloaded_exe_path())
+            .or(bitcoind::downloaded_exe_path().ok())
             .expect(
                 "you should provide env var BITCOIND_EXE or specifiy a bitcoind version feature",
             );
@@ -359,12 +359,12 @@ macro_rules! bdk_blockchain_tests {
      fn $_fn_name:ident ( $( $test_client:ident : &TestClient )? $(,)? ) -> $blockchain:ty $block:block) => {
         #[cfg(test)]
         mod bdk_blockchain_tests {
-            use $crate::bitcoin::Network;
+            use $crate::bitcoin::{Transaction, Network};
             use $crate::testutils::blockchain_tests::TestClient;
-            use $crate::blockchain::noop_progress;
+            use $crate::blockchain::Blockchain;
             use $crate::database::MemoryDatabase;
             use $crate::types::KeychainKind;
-            use $crate::{Wallet, FeeRate};
+            use $crate::{Wallet, FeeRate, SyncOptions};
             use $crate::testutils;
 
             use super::*;
@@ -375,11 +375,11 @@ macro_rules! bdk_blockchain_tests {
                 $block
             }
 
-            fn get_wallet_from_descriptors(descriptors: &(String, Option<String>), test_client: &TestClient) -> Wallet<$blockchain, MemoryDatabase> {
+            fn get_wallet_from_descriptors(descriptors: &(String, Option<String>)) -> Wallet<MemoryDatabase> {
-                Wallet::new(&descriptors.0.to_string(), descriptors.1.as_ref(), Network::Regtest, MemoryDatabase::new(), get_blockchain(test_client)).unwrap()
+                Wallet::new(&descriptors.0.to_string(), descriptors.1.as_ref(), Network::Regtest, MemoryDatabase::new()).unwrap()
             }
 
-            fn init_single_sig() -> (Wallet<$blockchain, MemoryDatabase>, (String, Option<String>), TestClient) {
+            fn init_single_sig() -> (Wallet<MemoryDatabase>, $blockchain, (String, Option<String>), TestClient) {
                 let _ = env_logger::try_init();
 
                 let descriptors = testutils! {
@@ -387,18 +387,22 @@ macro_rules! bdk_blockchain_tests {
                 };
 
                 let test_client = TestClient::default();
-                let wallet = get_wallet_from_descriptors(&descriptors, &test_client);
+                let blockchain = get_blockchain(&test_client);
+                let wallet = get_wallet_from_descriptors(&descriptors);
 
                 // rpc need to call import_multi before receiving any tx, otherwise will not see tx in the mempool
                 #[cfg(feature = "test-rpc")]
-                wallet.sync(noop_progress(), None).unwrap();
+                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
 
-                (wallet, descriptors, test_client)
+                (wallet, blockchain, descriptors, test_client)
             }
 
             #[test]
             fn test_sync_simple() {
-                let (wallet, descriptors, mut test_client) = init_single_sig();
+                use std::ops::Deref;
+                use crate::database::Database;
+
+                let (wallet, blockchain, descriptors, mut test_client) = init_single_sig();
 
                 let tx = testutils! {
                     @tx ( (@external descriptors, 0) => 50_000 )
@@ -406,7 +410,13 @@ macro_rules! bdk_blockchain_tests {
                 println!("{:?}", tx);
                 let txid = test_client.receive(tx);
 
-                wallet.sync(noop_progress(), None).unwrap();
+                // the RPC blockchain needs to call `sync()` during initialization to import the
+                // addresses (see `init_single_sig()`), so we skip this assertion
+                #[cfg(not(feature = "test-rpc"))]
+                assert!(wallet.database().deref().get_sync_time().unwrap().is_none(), "initial sync_time not none");
+
+                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
+                assert!(wallet.database().deref().get_sync_time().unwrap().is_some(), "sync_time hasn't been updated");
 
                 assert_eq!(wallet.get_balance().unwrap(), 50_000, "incorrect balance");
                 assert_eq!(wallet.list_unspent().unwrap()[0].keychain, KeychainKind::External, "incorrect keychain kind");
@@ -420,7 +430,7 @@ macro_rules! bdk_blockchain_tests {
 
             #[test]
             fn test_sync_stop_gap_20() {
-                let (wallet, descriptors, mut test_client) = init_single_sig();
+                let (wallet, blockchain, descriptors, mut test_client) = init_single_sig();
 
                 test_client.receive(testutils! {
                     @tx ( (@external descriptors, 5) => 50_000 )
@@ -429,7 +439,7 @@ macro_rules! bdk_blockchain_tests {
                     @tx ( (@external descriptors, 25) => 50_000 )
                 });
 
-                wallet.sync(noop_progress(), None).unwrap();
+                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
 
                 assert_eq!(wallet.get_balance().unwrap(), 100_000, "incorrect balance");
                 assert_eq!(wallet.list_transactions(false).unwrap().len(), 2, "incorrect number of txs");
@@ -437,16 +447,16 @@ macro_rules! bdk_blockchain_tests {
 
             #[test]
             fn test_sync_before_and_after_receive() {
-                let (wallet, descriptors, mut test_client) = init_single_sig();
+                let (wallet, blockchain, descriptors, mut test_client) = init_single_sig();
 
-                wallet.sync(noop_progress(), None).unwrap();
+                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
                 assert_eq!(wallet.get_balance().unwrap(), 0);
 
                 test_client.receive(testutils! {
                     @tx ( (@external descriptors, 0) => 50_000 )
                 });
 
-                wallet.sync(noop_progress(), None).unwrap();
+                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
 
                 assert_eq!(wallet.get_balance().unwrap(), 50_000, "incorrect balance");
                 assert_eq!(wallet.list_transactions(false).unwrap().len(), 1, "incorrect number of txs");
@@ -454,13 +464,13 @@ macro_rules! bdk_blockchain_tests {
 
             #[test]
             fn test_sync_multiple_outputs_same_tx() {
-                let (wallet, descriptors, mut test_client) = init_single_sig();
+                let (wallet, blockchain, descriptors, mut test_client) = init_single_sig();
 
                 let txid = test_client.receive(testutils! {
                     @tx ( (@external descriptors, 0) => 50_000, (@external descriptors, 1) => 25_000, (@external descriptors, 5) => 30_000 )
                 });
 
-                wallet.sync(noop_progress(), None).unwrap();
+                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
 
                 assert_eq!(wallet.get_balance().unwrap(), 105_000, "incorrect balance");
                 assert_eq!(wallet.list_transactions(false).unwrap().len(), 1, "incorrect number of txs");
@@ -475,7 +485,7 @@ macro_rules! bdk_blockchain_tests {
 
             #[test]
             fn test_sync_receive_multi() {
-                let (wallet, descriptors, mut test_client) = init_single_sig();
+                let (wallet, blockchain, descriptors, mut test_client) = init_single_sig();
 
                 test_client.receive(testutils! {
                     @tx ( (@external descriptors, 0) => 50_000 )
@@ -484,7 +494,7 @@ macro_rules! bdk_blockchain_tests {
                     @tx ( (@external descriptors, 5) => 25_000 )
                 });
 
-                wallet.sync(noop_progress(), None).unwrap();
+                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
 
                 assert_eq!(wallet.get_balance().unwrap(), 75_000, "incorrect balance");
                 assert_eq!(wallet.list_transactions(false).unwrap().len(), 2, "incorrect number of txs");
@@ -493,32 +503,32 @@ macro_rules! bdk_blockchain_tests {
 
             #[test]
             fn test_sync_address_reuse() {
-                let (wallet, descriptors, mut test_client) = init_single_sig();
+                let (wallet, blockchain, descriptors, mut test_client) = init_single_sig();
 
                 test_client.receive(testutils! {
                     @tx ( (@external descriptors, 0) => 50_000 )
                 });
 
-                wallet.sync(noop_progress(), None).unwrap();
+                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
                 assert_eq!(wallet.get_balance().unwrap(), 50_000);
 
                 test_client.receive(testutils! {
                     @tx ( (@external descriptors, 0) => 25_000 )
                 });
 
-                wallet.sync(noop_progress(), None).unwrap();
+                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
                 assert_eq!(wallet.get_balance().unwrap(), 75_000, "incorrect balance");
             }
 
             #[test]
             fn test_sync_receive_rbf_replaced() {
-                let (wallet, descriptors, mut test_client) = init_single_sig();
+                let (wallet, blockchain, descriptors, mut test_client) = init_single_sig();
 
                 let txid = test_client.receive(testutils! {
                     @tx ( (@external descriptors, 0) => 50_000 ) ( @replaceable true )
                 });
 
-                wallet.sync(noop_progress(), None).unwrap();
+                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
 
                 assert_eq!(wallet.get_balance().unwrap(), 50_000, "incorrect balance");
                 assert_eq!(wallet.list_transactions(false).unwrap().len(), 1, "incorrect number of txs");
@@ -532,7 +542,7 @@ macro_rules! bdk_blockchain_tests {
 
                 let new_txid = test_client.bump_fee(&txid);
 
-                wallet.sync(noop_progress(), None).unwrap();
+                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
 
                 assert_eq!(wallet.get_balance().unwrap(), 50_000, "incorrect balance after bump");
                 assert_eq!(wallet.list_transactions(false).unwrap().len(), 1, "incorrect number of txs after bump");
@@ -550,13 +560,13 @@ macro_rules! bdk_blockchain_tests {
             #[cfg(not(feature = "esplora"))]
             #[test]
             fn test_sync_reorg_block() {
-                let (wallet, descriptors, mut test_client) = init_single_sig();
+                let (wallet, blockchain, descriptors, mut test_client) = init_single_sig();
 
                 let txid = test_client.receive(testutils! {
                     @tx ( (@external descriptors, 0) => 50_000 ) ( @confirmations 1 ) ( @replaceable true )
                 });
 
-                wallet.sync(noop_progress(), None).unwrap();
+                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
 
                 assert_eq!(wallet.get_balance().unwrap(), 50_000, "incorrect balance");
                 assert_eq!(wallet.list_transactions(false).unwrap().len(), 1, "incorrect number of txs");
@@ -569,7 +579,7 @@ macro_rules! bdk_blockchain_tests {
                 // Invalidate 1 block
                 test_client.invalidate(1);
 
-                wallet.sync(noop_progress(), None).unwrap();
+                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
 
                 assert_eq!(wallet.get_balance().unwrap(), 50_000, "incorrect balance after invalidate");
 
@@ -580,15 +590,15 @@ macro_rules! bdk_blockchain_tests {
 
             #[test]
             fn test_sync_after_send() {
-                let (wallet, descriptors, mut test_client) = init_single_sig();
+                let (wallet, blockchain, descriptors, mut test_client) = init_single_sig();
-                println!("{}", descriptors.0);
+                                println!("{}", descriptors.0);
                 let node_addr = test_client.get_node_address(None);
 
                 test_client.receive(testutils! {
                     @tx ( (@external descriptors, 0) => 50_000 )
                 });
 
-                wallet.sync(noop_progress(), None).unwrap();
+                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
                 assert_eq!(wallet.get_balance().unwrap(), 50_000, "incorrect balance");
 
                 let mut builder = wallet.build_tx();
@@ -598,25 +608,93 @@ macro_rules! bdk_blockchain_tests {
                 assert!(finalized, "Cannot finalize transaction");
                 let tx = psbt.extract_tx();
                 println!("{}", bitcoin::consensus::encode::serialize_hex(&tx));
-                wallet.broadcast(tx).unwrap();
+                blockchain.broadcast(&tx).unwrap();
-                wallet.sync(noop_progress(), None).unwrap();
+                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
                 assert_eq!(wallet.get_balance().unwrap(), details.received, "incorrect balance after send");
 
                 assert_eq!(wallet.list_transactions(false).unwrap().len(), 2, "incorrect number of txs");
                 assert_eq!(wallet.list_unspent().unwrap().len(), 1, "incorrect number of unspents");
             }
 
+            /// Send two conflicting transactions to the same address twice in a row.
+            /// The coins should only be received once!
+            #[test]
+            fn test_sync_double_receive() {
+                let (wallet, blockchain, descriptors, mut test_client) = init_single_sig();
+                let receiver_wallet = get_wallet_from_descriptors(&("wpkh(cVpPVruEDdmutPzisEsYvtST1usBR3ntr8pXSyt6D2YYqXRyPcFW)".to_string(), None));
+                // need to sync so rpc can start watching
+                receiver_wallet.sync(&blockchain, SyncOptions::default()).unwrap();
+
+                test_client.receive(testutils! {
+                    @tx ( (@external descriptors, 0) => 50_000, (@external descriptors, 1) => 25_000 ) (@confirmations 1)
+                });
+
+                wallet.sync(&blockchain, SyncOptions::default()).expect("sync");
+                assert_eq!(wallet.get_balance().unwrap(), 75_000, "incorrect balance");
+                let target_addr = receiver_wallet.get_address($crate::wallet::AddressIndex::New).unwrap().address;
+
+                let tx1 = {
+                    let mut builder = wallet.build_tx();
+                    builder.add_recipient(target_addr.script_pubkey(), 49_000).enable_rbf();
+                    let (mut psbt, _details) = builder.finish().expect("building first tx");
+                    let finalized = wallet.sign(&mut psbt, Default::default()).expect("signing first tx");
+                    assert!(finalized, "Cannot finalize transaction");
+                    psbt.extract_tx()
+                };
+
+                let tx2 = {
+                    let mut builder = wallet.build_tx();
+                    builder.add_recipient(target_addr.script_pubkey(), 49_000).enable_rbf().fee_rate(FeeRate::from_sat_per_vb(5.0));
+                    let (mut psbt, _details) = builder.finish().expect("building replacement tx");
+                    let finalized = wallet.sign(&mut psbt, Default::default()).expect("signing replacement tx");
+                    assert!(finalized, "Cannot finalize transaction");
+                    psbt.extract_tx()
+                };
+
+                blockchain.broadcast(&tx1).expect("broadcasting first");
+                blockchain.broadcast(&tx2).expect("broadcasting replacement");
+
+                receiver_wallet.sync(&blockchain, SyncOptions::default()).expect("syncing receiver");
+                assert_eq!(receiver_wallet.get_balance().expect("balance"), 49_000, "should have received coins once and only once");
+            }
+
+            #[test]
+            fn test_sync_many_sends_to_a_single_address() {
+                let (wallet, blockchain, descriptors, mut test_client) = init_single_sig();
+
+                for _ in 0..4 {
+                    // split this up into multiple blocks so rpc doesn't get angry
+                    for _ in 0..20 {
+                        test_client.receive(testutils! {
+                            @tx ( (@external descriptors, 0) => 1_000 )
+                        });
+                    }
+                    test_client.generate(1, None);
+                }
+
+                // add some to the mempool as well.
+                for _ in 0..20 {
+                    test_client.receive(testutils! {
+                        @tx ( (@external descriptors, 0) => 1_000 )
+                    });
+                }
+
+                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
+
+                assert_eq!(wallet.get_balance().unwrap(), 100_000);
+            }
+
             #[test]
             fn test_update_confirmation_time_after_generate() {
-                let (wallet, descriptors, mut test_client) = init_single_sig();
+                let (wallet, blockchain, descriptors, mut test_client) = init_single_sig();
-                println!("{}", descriptors.0);
+                                println!("{}", descriptors.0);
                 let node_addr = test_client.get_node_address(None);
 
                 let received_txid = test_client.receive(testutils! {
                     @tx ( (@external descriptors, 0) => 50_000 )
                 });
 
-                wallet.sync(noop_progress(), None).unwrap();
+                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
                 assert_eq!(wallet.get_balance().unwrap(), 50_000, "incorrect balance");
 
                 let tx_map = wallet.list_transactions(false).unwrap().into_iter().map(|tx| (tx.txid, tx)).collect::<std::collections::HashMap<_, _>>();
@@ -624,7 +702,7 @@ macro_rules! bdk_blockchain_tests {
                 assert!(details.confirmation_time.is_none());
 
                 test_client.generate(1, Some(node_addr));
-                wallet.sync(noop_progress(), None).unwrap();
+                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
 
                 let tx_map = wallet.list_transactions(false).unwrap().into_iter().map(|tx| (tx.txid, tx)).collect::<std::collections::HashMap<_, _>>();
                 let details = tx_map.get(&received_txid).unwrap();
@@ -634,13 +712,13 @@ macro_rules! bdk_blockchain_tests {
 
             #[test]
             fn test_sync_outgoing_from_scratch() {
-                let (wallet, descriptors, mut test_client) = init_single_sig();
+                let (wallet, blockchain, descriptors, mut test_client) = init_single_sig();
-                let node_addr = test_client.get_node_address(None);
+                                let node_addr = test_client.get_node_address(None);
                 let received_txid = test_client.receive(testutils! {
                     @tx ( (@external descriptors, 0) => 50_000 )
                 });
 
-                wallet.sync(noop_progress(), None).unwrap();
+                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
                 assert_eq!(wallet.get_balance().unwrap(), 50_000, "incorrect balance");
 
                 let mut builder = wallet.build_tx();
@@ -649,25 +727,26 @@ macro_rules! bdk_blockchain_tests {
 
                 let finalized = wallet.sign(&mut psbt, Default::default()).unwrap();
                 assert!(finalized, "Cannot finalize transaction");
-                let sent_txid = wallet.broadcast(psbt.extract_tx()).unwrap();
+                let sent_tx = psbt.extract_tx();
+                blockchain.broadcast(&sent_tx).unwrap();
 
-                wallet.sync(noop_progress(), None).unwrap();
+                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
                 assert_eq!(wallet.get_balance().unwrap(), details.received, "incorrect balance after receive");
 
                 // empty wallet
-                let wallet = get_wallet_from_descriptors(&descriptors, &test_client);
+                let wallet = get_wallet_from_descriptors(&descriptors);
 
                 #[cfg(feature = "rpc")]  // rpc cannot see mempool tx before importmulti
                 test_client.generate(1, Some(node_addr));
 
-                wallet.sync(noop_progress(), None).unwrap();
+                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
                 let tx_map = wallet.list_transactions(false).unwrap().into_iter().map(|tx| (tx.txid, tx)).collect::<std::collections::HashMap<_, _>>();
 
                 let received = tx_map.get(&received_txid).unwrap();
                 assert_eq!(received.received, 50_000, "incorrect received from receiver");
                 assert_eq!(received.sent, 0, "incorrect sent from receiver");
 
-                let sent = tx_map.get(&sent_txid).unwrap();
+                let sent = tx_map.get(&sent_tx.txid()).unwrap();
                 assert_eq!(sent.received, details.received, "incorrect received from sender");
                 assert_eq!(sent.sent, details.sent, "incorrect sent from sender");
                 assert_eq!(sent.fee.unwrap_or(0), details.fee.unwrap_or(0), "incorrect fees from sender");
@@ -675,14 +754,14 @@ macro_rules! bdk_blockchain_tests {
 
             #[test]
             fn test_sync_long_change_chain() {
-                let (wallet, descriptors, mut test_client) = init_single_sig();
+                let (wallet, blockchain, descriptors, mut test_client) = init_single_sig();
-                let node_addr = test_client.get_node_address(None);
+                                let node_addr = test_client.get_node_address(None);
 
                 test_client.receive(testutils! {
                     @tx ( (@external descriptors, 0) => 50_000 )
                 });
 
-                wallet.sync(noop_progress(), None).unwrap();
+                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
                 assert_eq!(wallet.get_balance().unwrap(), 50_000, "incorrect balance");
 
                 let mut total_sent = 0;
@@ -692,38 +771,38 @@ macro_rules! bdk_blockchain_tests {
                     let (mut psbt, details) = builder.finish().unwrap();
                     let finalized = wallet.sign(&mut psbt, Default::default()).unwrap();
                     assert!(finalized, "Cannot finalize transaction");
-                    wallet.broadcast(psbt.extract_tx()).unwrap();
+                    blockchain.broadcast(&psbt.extract_tx()).unwrap();
 
-                    wallet.sync(noop_progress(), None).unwrap();
+                    wallet.sync(&blockchain, SyncOptions::default()).unwrap();
 
                     total_sent += 5_000 + details.fee.unwrap_or(0);
                 }
 
-                wallet.sync(noop_progress(), None).unwrap();
+                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
                 assert_eq!(wallet.get_balance().unwrap(), 50_000 - total_sent, "incorrect balance after chain");
 
                 // empty wallet
 
-                let wallet = get_wallet_from_descriptors(&descriptors, &test_client);
+                let wallet = get_wallet_from_descriptors(&descriptors);
 
                 #[cfg(feature = "rpc")]  // rpc cannot see mempool tx before importmulti
                 test_client.generate(1, Some(node_addr));
 
-                wallet.sync(noop_progress(), None).unwrap();
+                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
                 assert_eq!(wallet.get_balance().unwrap(), 50_000 - total_sent, "incorrect balance empty wallet");
 
             }
 
             #[test]
             fn test_sync_bump_fee_basic() {
-                let (wallet, descriptors, mut test_client) = init_single_sig();
+                let (wallet, blockchain, descriptors, mut test_client) = init_single_sig();
-                let node_addr = test_client.get_node_address(None);
+                                let node_addr = test_client.get_node_address(None);
 
                 test_client.receive(testutils! {
                     @tx ( (@external descriptors, 0) => 50_000 ) (@confirmations 1)
                 });
 
-                wallet.sync(noop_progress(), None).unwrap();
+                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
                 assert_eq!(wallet.get_balance().unwrap(), 50_000, "incorrect balance");
 
                 let mut builder = wallet.build_tx();
@@ -731,18 +810,18 @@ macro_rules! bdk_blockchain_tests {
                 let (mut psbt, details) = builder.finish().unwrap();
                 let finalized = wallet.sign(&mut psbt, Default::default()).unwrap();
                 assert!(finalized, "Cannot finalize transaction");
-                wallet.broadcast(psbt.extract_tx()).unwrap();
+                blockchain.broadcast(&psbt.extract_tx()).unwrap();
-                wallet.sync(noop_progress(), None).unwrap();
+                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
                 assert_eq!(wallet.get_balance().unwrap(), 50_000 - details.fee.unwrap_or(0) - 5_000, "incorrect balance from fees");
                 assert_eq!(wallet.get_balance().unwrap(), details.received, "incorrect balance from received");
 
                 let mut builder = wallet.build_fee_bump(details.txid).unwrap();
                 builder.fee_rate(FeeRate::from_sat_per_vb(2.1));
-                let (mut new_psbt, new_details) = builder.finish().unwrap();
+                let (mut new_psbt, new_details) = builder.finish().expect("fee bump tx");
                 let finalized = wallet.sign(&mut new_psbt, Default::default()).unwrap();
                 assert!(finalized, "Cannot finalize transaction");
-                wallet.broadcast(new_psbt.extract_tx()).unwrap();
+                blockchain.broadcast(&new_psbt.extract_tx()).unwrap();
-                wallet.sync(noop_progress(), None).unwrap();
+                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
                 assert_eq!(wallet.get_balance().unwrap(), 50_000 - new_details.fee.unwrap_or(0) - 5_000, "incorrect balance from fees after bump");
                 assert_eq!(wallet.get_balance().unwrap(), new_details.received, "incorrect balance from received after bump");
 
@@ -751,14 +830,14 @@ macro_rules! bdk_blockchain_tests {
 
             #[test]
             fn test_sync_bump_fee_remove_change() {
-                let (wallet, descriptors, mut test_client) = init_single_sig();
+                let (wallet, blockchain, descriptors, mut test_client) = init_single_sig();
-                let node_addr = test_client.get_node_address(None);
+                                let node_addr = test_client.get_node_address(None);
 
                 test_client.receive(testutils! {
                     @tx ( (@external descriptors, 0) => 50_000 ) (@confirmations 1)
                 });
 
-                wallet.sync(noop_progress(), None).unwrap();
+                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
                 assert_eq!(wallet.get_balance().unwrap(), 50_000, "incorrect balance");
 
                 let mut builder = wallet.build_tx();
@@ -766,18 +845,18 @@ macro_rules! bdk_blockchain_tests {
                 let (mut psbt, details) = builder.finish().unwrap();
                 let finalized = wallet.sign(&mut psbt, Default::default()).unwrap();
                 assert!(finalized, "Cannot finalize transaction");
-                wallet.broadcast(psbt.extract_tx()).unwrap();
+                blockchain.broadcast(&psbt.extract_tx()).unwrap();
-                wallet.sync(noop_progress(), None).unwrap();
+                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
                 assert_eq!(wallet.get_balance().unwrap(), 1_000 - details.fee.unwrap_or(0), "incorrect balance after send");
                 assert_eq!(wallet.get_balance().unwrap(), details.received, "incorrect received after send");
 
                 let mut builder = wallet.build_fee_bump(details.txid).unwrap();
-                builder.fee_rate(FeeRate::from_sat_per_vb(5.0));
+                builder.fee_rate(FeeRate::from_sat_per_vb(5.1));
                 let (mut new_psbt, new_details) = builder.finish().unwrap();
                 let finalized = wallet.sign(&mut new_psbt, Default::default()).unwrap();
                 assert!(finalized, "Cannot finalize transaction");
-                wallet.broadcast(new_psbt.extract_tx()).unwrap();
+                blockchain.broadcast(&new_psbt.extract_tx()).unwrap();
-                wallet.sync(noop_progress(), None).unwrap();
+                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
                 assert_eq!(wallet.get_balance().unwrap(), 0, "incorrect balance after change removal");
                 assert_eq!(new_details.received, 0, "incorrect received after change removal");
 
@@ -786,14 +865,14 @@ macro_rules! bdk_blockchain_tests {
 
             #[test]
             fn test_sync_bump_fee_add_input_simple() {
-                let (wallet, descriptors, mut test_client) = init_single_sig();
+                let (wallet, blockchain, descriptors, mut test_client) = init_single_sig();
-                let node_addr = test_client.get_node_address(None);
+                                let node_addr = test_client.get_node_address(None);
 
                 test_client.receive(testutils! {
                     @tx ( (@external descriptors, 0) => 50_000, (@external descriptors, 1) => 25_000 ) (@confirmations 1)
                 });
 
-                wallet.sync(noop_progress(), None).unwrap();
+                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
                 assert_eq!(wallet.get_balance().unwrap(), 75_000, "incorrect balance");
 
                 let mut builder = wallet.build_tx();
@@ -801,8 +880,8 @@ macro_rules! bdk_blockchain_tests {
                 let (mut psbt, details) = builder.finish().unwrap();
                 let finalized = wallet.sign(&mut psbt, Default::default()).unwrap();
                 assert!(finalized, "Cannot finalize transaction");
-                wallet.broadcast(psbt.extract_tx()).unwrap();
+                blockchain.broadcast(&psbt.extract_tx()).unwrap();
-                wallet.sync(noop_progress(), None).unwrap();
+                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
                 assert_eq!(wallet.get_balance().unwrap(), 26_000 - details.fee.unwrap_or(0), "incorrect balance after send");
                 assert_eq!(details.received, 1_000 - details.fee.unwrap_or(0), "incorrect received after send");
 
@@ -811,22 +890,22 @@ macro_rules! bdk_blockchain_tests {
                 let (mut new_psbt, new_details) = builder.finish().unwrap();
                 let finalized = wallet.sign(&mut new_psbt, Default::default()).unwrap();
                 assert!(finalized, "Cannot finalize transaction");
-                wallet.broadcast(new_psbt.extract_tx()).unwrap();
+                blockchain.broadcast(&new_psbt.extract_tx()).unwrap();
-                wallet.sync(noop_progress(), None).unwrap();
+                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
                 assert_eq!(new_details.sent, 75_000, "incorrect sent");
                 assert_eq!(wallet.get_balance().unwrap(), new_details.received, "incorrect balance after add input");
             }
 
             #[test]
             fn test_sync_bump_fee_add_input_no_change() {
-                let (wallet, descriptors, mut test_client) = init_single_sig();
+                let (wallet, blockchain, descriptors, mut test_client) = init_single_sig();
-                let node_addr = test_client.get_node_address(None);
+                                let node_addr = test_client.get_node_address(None);
 
                 test_client.receive(testutils! {
                     @tx ( (@external descriptors, 0) => 50_000, (@external descriptors, 1) => 25_000 ) (@confirmations 1)
                 });
 
-                wallet.sync(noop_progress(), None).unwrap();
+                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
                 assert_eq!(wallet.get_balance().unwrap(), 75_000, "incorrect balance");
 
                 let mut builder = wallet.build_tx();
@@ -834,8 +913,8 @@ macro_rules! bdk_blockchain_tests {
                 let (mut psbt, details) = builder.finish().unwrap();
                 let finalized = wallet.sign(&mut psbt, Default::default()).unwrap();
                 assert!(finalized, "Cannot finalize transaction");
-                wallet.broadcast(psbt.extract_tx()).unwrap();
+                blockchain.broadcast(&psbt.extract_tx()).unwrap();
-                wallet.sync(noop_progress(), None).unwrap();
+                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
                 assert_eq!(wallet.get_balance().unwrap(), 26_000 - details.fee.unwrap_or(0), "incorrect balance after send");
                 assert_eq!(details.received, 1_000 - details.fee.unwrap_or(0), "incorrect received after send");
 
@@ -846,20 +925,51 @@ macro_rules! bdk_blockchain_tests {
 
                 let finalized = wallet.sign(&mut new_psbt, Default::default()).unwrap();
                 assert!(finalized, "Cannot finalize transaction");
-                wallet.broadcast(new_psbt.extract_tx()).unwrap();
+                blockchain.broadcast(&new_psbt.extract_tx()).unwrap();
-                wallet.sync(noop_progress(), None).unwrap();
+                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
                 assert_eq!(new_details.sent, 75_000, "incorrect sent");
                 assert_eq!(wallet.get_balance().unwrap(), 0, "incorrect balance after add input");
                 assert_eq!(new_details.received, 0, "incorrect received after add input");
             }
 
+
+            #[test]
+            fn test_add_data() {
+                let (wallet, blockchain, descriptors, mut test_client) = init_single_sig();
+                                let node_addr = test_client.get_node_address(None);
+                let _ = test_client.receive(testutils! {
+                    @tx ( (@external descriptors, 0) => 50_000 )
+                });
+
+                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
+                assert_eq!(wallet.get_balance().unwrap(), 50_000, "incorrect balance");
+
+                let mut builder = wallet.build_tx();
+                let data = [42u8;80];
+                builder.add_data(&data);
+                let (mut psbt, details) = builder.finish().unwrap();
+
+                let finalized = wallet.sign(&mut psbt, Default::default()).unwrap();
+                assert!(finalized, "Cannot finalize transaction");
+                let tx = psbt.extract_tx();
+                let serialized_tx = bitcoin::consensus::encode::serialize(&tx);
+                assert!(serialized_tx.windows(data.len()).any(|e| e==data), "cannot find op_return data in transaction");
+                blockchain.broadcast(&tx).unwrap();
+                test_client.generate(1, Some(node_addr));
+                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
+                assert_eq!(wallet.get_balance().unwrap(), 50_000 - details.fee.unwrap_or(0), "incorrect balance after send");
+
+                let tx_map = wallet.list_transactions(false).unwrap().into_iter().map(|tx| (tx.txid, tx)).collect::<std::collections::HashMap<_, _>>();
+                let _ = tx_map.get(&tx.txid()).unwrap();
+            }
+
             #[test]
             fn test_sync_receive_coinbase() {
-                let (wallet, _, mut test_client) = init_single_sig();
+                let (wallet, blockchain, _, mut test_client) = init_single_sig();
 
                 let wallet_addr = wallet.get_address($crate::wallet::AddressIndex::New).unwrap().address;
 
-                wallet.sync(noop_progress(), None).unwrap();
+                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
                 assert_eq!(wallet.get_balance().unwrap(), 0, "incorrect balance");
 
                 test_client.generate(1, Some(wallet_addr));
@@ -872,9 +982,219 @@ macro_rules! bdk_blockchain_tests {
                 }
 
 
-                wallet.sync(noop_progress(), None).unwrap();
+                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
                 assert!(wallet.get_balance().unwrap() > 0, "incorrect balance after receiving coinbase");
             }
+
+            #[test]
+            fn test_send_to_bech32m_addr() {
+                use std::str::FromStr;
+                use serde;
+                use serde_json;
+                use serde::Serialize;
+                use bitcoincore_rpc::jsonrpc::serde_json::Value;
+                use bitcoincore_rpc::{Auth, Client, RpcApi};
+
+                let (wallet, blockchain, descriptors, mut test_client) = init_single_sig();
+
+                // TODO remove once rust-bitcoincore-rpc with PR 199 released
+                // https://github.com/rust-bitcoin/rust-bitcoincore-rpc/pull/199
+                /// Import Descriptor Request
+                #[derive(Serialize, Clone, PartialEq, Eq, Debug)]
+                pub struct ImportDescriptorRequest {
+                    pub active: bool,
+                    #[serde(rename = "desc")]
+                    pub descriptor: String,
+                    pub range: [i64; 2],
+                    pub next_index: i64,
+                    pub timestamp: String,
+                    pub internal: bool,
+                }
+
+                // TODO remove once rust-bitcoincore-rpc with PR 199 released
+                impl ImportDescriptorRequest {
+                    /// Create a new Import Descriptor request providing just the descriptor and internal flags
+                    pub fn new(descriptor: &str, internal: bool) -> Self {
+                        ImportDescriptorRequest {
+                            descriptor: descriptor.to_string(),
+                            internal,
+                            active: true,
+                            range: [0, 100],
+                            next_index: 0,
+                            timestamp: "now".to_string(),
+                        }
+                    }
+                }
+
+                // 1. Create and add descriptors to a test bitcoind node taproot wallet
+
+                // TODO replace once rust-bitcoincore-rpc with PR 174 released
+                // https://github.com/rust-bitcoin/rust-bitcoincore-rpc/pull/174
+                let _createwallet_result: Value = test_client.bitcoind.client.call("createwallet", &["taproot_wallet".into(),false.into(),true.into(),serde_json::to_value("").unwrap(), false.into(), true.into()]).unwrap();
+
+                // TODO replace once bitcoind released with support for rust-bitcoincore-rpc PR 174
+                let taproot_wallet_client = Client::new(&test_client.bitcoind.rpc_url_with_wallet("taproot_wallet"), Auth::CookieFile(test_client.bitcoind.params.cookie_file.clone())).unwrap();
+
+                let wallet_descriptor = "tr(tprv8ZgxMBicQKsPdBtxmEMPnNq58KGusNAimQirKFHqX2yk2D8q1v6pNLiKYVAdzDHy2w3vF4chuGfMvNtzsbTTLVXBcdkCA1rje1JG6oksWv8/86h/1h/0h/0/*)#y283ssmn";
+                let change_descriptor = "tr(tprv8ZgxMBicQKsPdBtxmEMPnNq58KGusNAimQirKFHqX2yk2D8q1v6pNLiKYVAdzDHy2w3vF4chuGfMvNtzsbTTLVXBcdkCA1rje1JG6oksWv8/86h/1h/0h/1/*)#47zsd9tt";
+
+                let tr_descriptors = vec![
+                            ImportDescriptorRequest::new(wallet_descriptor, false),
+                            ImportDescriptorRequest::new(change_descriptor, false),
+                        ];
+
+                // TODO replace once rust-bitcoincore-rpc with PR 199 released
+                let _import_result: Value = taproot_wallet_client.call("importdescriptors", &[serde_json::to_value(tr_descriptors).unwrap()]).unwrap();
+
+                // 2. Get a new bech32m address from test bitcoind node taproot wallet
+
+                // TODO replace once rust-bitcoincore-rpc with PR 199 released
+                let node_addr: bitcoin::Address = taproot_wallet_client.call("getnewaddress", &["test address".into(), "bech32m".into()]).unwrap();
+                assert_eq!(node_addr, bitcoin::Address::from_str("bcrt1pj5y3f0fu4y7g98k4v63j9n0xvj3lmln0cpwhsjzknm6nt0hr0q7qnzwsy9").unwrap());
+
+                // 3. Send 50_000 sats from test bitcoind node to test BDK wallet
+
+                test_client.receive(testutils! {
+                    @tx ( (@external descriptors, 0) => 50_000 )
+                });
+
+                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
+                assert_eq!(wallet.get_balance().unwrap(), 50_000, "wallet has incorrect balance");
+
+                // 4. Send 25_000 sats from test BDK wallet to test bitcoind node taproot wallet
+
+                let mut builder = wallet.build_tx();
+                builder.add_recipient(node_addr.script_pubkey(), 25_000);
+                let (mut psbt, details) = builder.finish().unwrap();
+                let finalized = wallet.sign(&mut psbt, Default::default()).unwrap();
+                assert!(finalized, "wallet cannot finalize transaction");
+                let tx = psbt.extract_tx();
+                blockchain.broadcast(&tx).unwrap();
+                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
+                assert_eq!(wallet.get_balance().unwrap(), details.received, "wallet has incorrect balance after send");
+                assert_eq!(wallet.list_transactions(false).unwrap().len(), 2, "wallet has incorrect number of txs");
+                assert_eq!(wallet.list_unspent().unwrap().len(), 1, "wallet has incorrect number of unspents");
+                test_client.generate(1, None);
+
+                // 5. Verify 25_000 sats are received by test bitcoind node taproot wallet
+
+                let taproot_balance = taproot_wallet_client.get_balance(None, None).unwrap();
+                assert_eq!(taproot_balance.as_sat(), 25_000, "node has incorrect taproot wallet balance");
+            }
+
+            #[test]
+            fn test_tx_chain() {
+                use bitcoincore_rpc::RpcApi;
+                use bitcoin::consensus::encode::deserialize;
+                use $crate::wallet::AddressIndex;
+
+                // Here we want to test that we set correctly the send and receive
+                // fields in the transaction object. For doing so, we create two
+                // different txs, the second one spending from the first:
+                // 1.
+                // Core (#1) -> Core (#2)
+                //           -> Us   (#3)
+                // 2.
+                // Core (#2) -> Us   (#4)
+
+                let (wallet, blockchain, _, mut test_client) = init_single_sig();
+                let bdk_address = wallet.get_address(AddressIndex::New).unwrap().address;
+                let core_address = test_client.get_new_address(None, None).unwrap();
+                let tx = testutils! {
+                    @tx ( (@addr bdk_address.clone()) => 50_000, (@addr core_address.clone()) => 40_000 )
+                };
+
+                // Tx one: from Core #1 to Core #2 and Us #3.
+                let txid_1 = test_client.receive(tx);
+                let tx_1: Transaction = deserialize(&test_client.get_transaction(&txid_1, None).unwrap().hex).unwrap();
+                let vout_1 = tx_1.output.into_iter().position(|o| o.script_pubkey == core_address.script_pubkey()).unwrap() as u32;
+                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
+                let tx_1 = wallet.list_transactions(false).unwrap().into_iter().find(|tx| tx.txid == txid_1).unwrap();
+                assert_eq!(tx_1.received, 50_000);
+                assert_eq!(tx_1.sent, 0);
+
+                // Tx two: from Core #2 to Us #4.
+                let tx = testutils! {
+                    @tx ( (@addr bdk_address) => 10_000 ) ( @inputs (txid_1,vout_1))
+                };
+                let txid_2 = test_client.receive(tx);
+
+                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
+                let tx_2 = wallet.list_transactions(false).unwrap().into_iter().find(|tx| tx.txid == txid_2).unwrap();
+                assert_eq!(tx_2.received, 10_000);
+                assert_eq!(tx_2.sent, 0);
+            }
+
+            #[test]
+            fn test_double_spend() {
+                // We create a tx and then we try to double spend it; BDK will always allow
+                // us to do so, as it never forgets about spent UTXOs
+                let (wallet, blockchain, descriptors, mut test_client) = init_single_sig();
+                let node_addr = test_client.get_node_address(None);
+                let _ = test_client.receive(testutils! {
+                    @tx ( (@external descriptors, 0) => 50_000 )
+                });
+
+                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
+                let mut builder = wallet.build_tx();
+                builder.add_recipient(node_addr.script_pubkey(), 25_000);
+                let (mut psbt, _details) = builder.finish().unwrap();
+                let finalized = wallet.sign(&mut psbt, Default::default()).unwrap();
+                assert!(finalized, "Cannot finalize transaction");
+                let initial_tx = psbt.extract_tx();
+                let _sent_txid = blockchain.broadcast(&initial_tx).unwrap();
+                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
+                for utxo in wallet.list_unspent().unwrap() {
+                    // Making sure the TXO we just spent is not returned by list_unspent
+                    assert!(utxo.outpoint != initial_tx.input[0].previous_output, "wallet displays spent txo in unspents");
+                }
+                // We can still create a transaction double spending `initial_tx`
+                let mut builder = wallet.build_tx();
+                builder
+                    .add_utxo(initial_tx.input[0].previous_output)
+                    .expect("Can't manually add an UTXO spent");
+                test_client.generate(1, Some(node_addr));
+                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
+                // Even after confirmation, we can still create a tx double spend it
+                let mut builder = wallet.build_tx();
+                builder
+                    .add_utxo(initial_tx.input[0].previous_output)
+                    .expect("Can't manually add an UTXO spent");
+                for utxo in wallet.list_unspent().unwrap() {
+                    // Making sure the TXO we just spent is not returned by list_unspent
+                    assert!(utxo.outpoint != initial_tx.input[0].previous_output, "wallet displays spent txo in unspents");
+                }
+            }
+
+            #[test]
+            fn test_send_receive_pkh() {
+                let descriptors = ("pkh(cVpPVruEDdmutPzisEsYvtST1usBR3ntr8pXSyt6D2YYqXRyPcFW)".to_string(), None);
+                let mut test_client = TestClient::default();
+                let blockchain = get_blockchain(&test_client);
+
+                let wallet = get_wallet_from_descriptors(&descriptors);
+                #[cfg(feature = "test-rpc")]
+                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
+
+                let _ = test_client.receive(testutils! {
+                    @tx ( (@external descriptors, 0)   => 50_000 )
+                });
+
+                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
+
+                assert_eq!(wallet.get_balance().unwrap(), 50_000);
+
+                let tx = {
+                    let mut builder = wallet.build_tx();
+                    builder.add_recipient(test_client.get_node_address(None).script_pubkey(), 25_000);
+                    let (mut psbt, _details) = builder.finish().unwrap();
+                    wallet.sign(&mut psbt, Default::default()).unwrap();
+                    psbt.extract_tx()
+                };
+                blockchain.broadcast(&tx).unwrap();
+
+                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
+            }
         }
     };
 

src/testutils/mod.rs

@@ -15,11 +15,37 @@
 pub mod blockchain_tests;
 
 use bitcoin::secp256k1::{Secp256k1, Verification};
-use bitcoin::{Address, PublicKey};
+use bitcoin::{Address, PublicKey, Txid};
 
 use miniscript::descriptor::DescriptorPublicKey;
 use miniscript::{Descriptor, MiniscriptKey, TranslatePk};
 
+#[derive(Clone, Debug)]
+pub struct TestIncomingInput {
+    pub txid: Txid,
+    pub vout: u32,
+    pub sequence: Option<u32>,
+}
+
+impl TestIncomingInput {
+    pub fn new(txid: Txid, vout: u32, sequence: Option<u32>) -> Self {
+        Self {
+            txid,
+            vout,
+            sequence,
+        }
+    }
+
+    #[cfg(feature = "test-blockchains")]
+    pub fn into_raw_tx_input(self) -> bitcoincore_rpc::json::CreateRawTransactionInput {
+        bitcoincore_rpc::json::CreateRawTransactionInput {
+            txid: self.txid,
+            vout: self.vout,
+            sequence: self.sequence,
+        }
+    }
+}
+
 #[derive(Clone, Debug)]
 pub struct TestIncomingOutput {
     pub value: u64,
@@ -37,6 +63,7 @@ impl TestIncomingOutput {
 
 #[derive(Clone, Debug)]
 pub struct TestIncomingTx {
+    pub input: Vec<TestIncomingInput>,
     pub output: Vec<TestIncomingOutput>,
     pub min_confirmations: Option<u64>,
     pub locktime: Option<i64>,
@@ -45,12 +72,14 @@ pub struct TestIncomingTx {
 
 impl TestIncomingTx {
     pub fn new(
+        input: Vec<TestIncomingInput>,
         output: Vec<TestIncomingOutput>,
         min_confirmations: Option<u64>,
         locktime: Option<i64>,
         replaceable: Option<bool>,
     ) -> Self {
         Self {
+            input,
             output,
             min_confirmations,
             locktime,
@@ -58,6 +87,10 @@ impl TestIncomingTx {
         }
     }
 
+    pub fn add_input(&mut self, input: TestIncomingInput) {
+        self.input.push(input);
+    }
+
     pub fn add_output(&mut self, output: TestIncomingOutput) {
         self.output.push(output);
     }
@@ -123,16 +156,21 @@ macro_rules! testutils {
     });
     ( @e $descriptors:expr, $child:expr ) => ({ testutils!(@external $descriptors, $child) });
     ( @i $descriptors:expr, $child:expr ) => ({ testutils!(@internal $descriptors, $child) });
+    ( @addr $addr:expr ) => ({ $addr });
 
-    ( @tx ( $( ( $( $addr:tt )* ) => $amount:expr ),+ ) $( ( @locktime $locktime:expr ) )? $( ( @confirmations $confirmations:expr ) )? $( ( @replaceable $replaceable:expr ) )? ) => ({
+    ( @tx ( $( ( $( $addr:tt )* ) => $amount:expr ),+ ) $( ( @inputs $( ($txid:expr, $vout:expr) ),+ ) )? $( ( @locktime $locktime:expr ) )? $( ( @confirmations $confirmations:expr ) )? $( ( @replaceable $replaceable:expr ) )? ) => ({
         let outs = vec![$( $crate::testutils::TestIncomingOutput::new($amount, testutils!( $($addr)* ))),+];
+        let _ins: Vec<$crate::testutils::TestIncomingInput> = vec![];
+        $(
+            let _ins = vec![$( $crate::testutils::TestIncomingInput { txid: $txid, vout: $vout, sequence: None }),+];
+        )?
 
         let locktime = None::<i64>$(.or(Some($locktime)))?;
 
         let min_confirmations = None::<u64>$(.or(Some($confirmations)))?;
         let replaceable = None::<bool>$(.or(Some($replaceable)))?;
 
-        $crate::testutils::TestIncomingTx::new(outs, min_confirmations, locktime, replaceable)
+        $crate::testutils::TestIncomingTx::new(_ins, outs, min_confirmations, locktime, replaceable)
     });
 
     ( @literal $key:expr ) => ({

src/types.rs

@@ -131,6 +131,8 @@ pub struct LocalUtxo {
     pub txout: TxOut,
     /// Type of keychain
     pub keychain: KeychainKind,
+    /// Whether this UTXO is spent or not
+    pub is_spent: bool,
 }
 
 /// A [`Utxo`] with its `satisfaction_weight`.
@@ -210,32 +212,29 @@ pub struct TransactionDetails {
     pub fee: Option<u64>,
     /// If the transaction is confirmed, contains height and timestamp of the block containing the
     /// transaction, unconfirmed transaction contains `None`.
-    pub confirmation_time: Option<ConfirmationTime>,
+    pub confirmation_time: Option<BlockTime>,
-    /// Whether the tx has been verified against the consensus rules
-    ///
-    /// Confirmed txs are considered "verified" by default, while unconfirmed txs are checked to
-    /// ensure an unstrusted [`Blockchain`](crate::blockchain::Blockchain) backend can't trick the
-    /// wallet into using an invalid tx as an RBF template.
-    ///
-    /// The check is only perfomed when the `verify` feature is enabled.
-    #[serde(default = "bool::default")] // default to `false` if not specified
-    pub verified: bool,
 }
 
-/// Block height and timestamp of the block containing the confirmed transaction
+/// Block height and timestamp of a block
 #[derive(Serialize, Deserialize, Debug, Clone, PartialEq, Eq, Default)]
-pub struct ConfirmationTime {
+pub struct BlockTime {
     /// confirmation block height
     pub height: u32,
     /// confirmation block timestamp
     pub timestamp: u64,
 }
 
-impl ConfirmationTime {
+/// **DEPRECATED**: Confirmation time of a transaction
-    /// Returns `Some` `ConfirmationTime` if both `height` and `timestamp` are `Some`
+///
+/// The structure has been renamed to `BlockTime`
+#[deprecated(note = "This structure has been renamed to `BlockTime`")]
+pub type ConfirmationTime = BlockTime;
+
+impl BlockTime {
+    /// Returns `Some` `BlockTime` if both `height` and `timestamp` are `Some`
     pub fn new(height: Option<u32>, timestamp: Option<u64>) -> Option<Self> {
         match (height, timestamp) {
-            (Some(height), Some(timestamp)) => Some(ConfirmationTime { height, timestamp }),
+            (Some(height), Some(timestamp)) => Some(BlockTime { height, timestamp }),
             _ => None,
         }
     }

src/wallet/address_validator.rs

@@ -55,7 +55,7 @@
 //! }
 //!
 //! let descriptor = "wpkh(tpubD6NzVbkrYhZ4Xferm7Pz4VnjdcDPFyjVu5K4iZXQ4pVN8Cks4pHVowTBXBKRhX64pkRyJZJN5xAKj4UDNnLPb5p2sSKXhewoYx5GbTdUFWq/*)";
-//! let mut wallet = Wallet::new_offline(descriptor, None, Network::Testnet, MemoryDatabase::default())?;
+//! let mut wallet = Wallet::new(descriptor, None, Network::Testnet, MemoryDatabase::default())?;
 //! wallet.add_address_validator(Arc::new(PrintAddressAndContinue));
 //!
 //! let address = wallet.get_address(New)?;

src/wallet/coin_selection.rs

@@ -372,7 +372,7 @@ impl<D: Database> CoinSelectionAlgorithm<D> for BranchAndBoundCoinSelection {
 
 impl BranchAndBoundCoinSelection {
     // TODO: make this more Rust-onic :)
-    // (And perhpaps refactor with less arguments?)
+    // (And perhaps refactor with less arguments?)
     #[allow(clippy::too_many_arguments)]
     fn bnb(
         &self,
@@ -569,6 +569,7 @@ mod test {
                     script_pubkey: Script::new(),
                 },
                 keychain: KeychainKind::External,
+                is_spent: false,
             }),
         }
     }
@@ -596,6 +597,7 @@ mod test {
                         script_pubkey: Script::new(),
                     },
                     keychain: KeychainKind::External,
+                    is_spent: false,
                 }),
             });
         }
@@ -615,6 +617,7 @@ mod test {
                     script_pubkey: Script::new(),
                 },
                 keychain: KeychainKind::External,
+                is_spent: false,
             }),
         };
         vec![utxo; utxos_number]

src/wallet/export.rs

@@ -29,8 +29,8 @@
 //!     "label":"testnet"
 //! }"#;
 //!
-//! let import = WalletExport::from_str(import)?;
+//! let import = FullyNodedExport::from_str(import)?;
-//! let wallet = Wallet::new_offline(
+//! let wallet = Wallet::new(
 //!     &import.descriptor(),
 //!     import.change_descriptor().as_ref(),
 //!     Network::Testnet,
@@ -45,13 +45,13 @@
 //! # use bdk::database::*;
 //! # use bdk::wallet::export::*;
 //! # use bdk::*;
-//! let wallet = Wallet::new_offline(
+//! let wallet = Wallet::new(
 //!     "wpkh([c258d2e4/84h/1h/0h]tpubDD3ynpHgJQW8VvWRzQ5WFDCrs4jqVFGHB3vLC3r49XHJSqP8bHKdK4AriuUKLccK68zfzowx7YhmDN8SiSkgCDENUFx9qVw65YyqM78vyVe/0/*)",
 //!     Some("wpkh([c258d2e4/84h/1h/0h]tpubDD3ynpHgJQW8VvWRzQ5WFDCrs4jqVFGHB3vLC3r49XHJSqP8bHKdK4AriuUKLccK68zfzowx7YhmDN8SiSkgCDENUFx9qVw65YyqM78vyVe/1/*)"),
 //!     Network::Testnet,
 //!     MemoryDatabase::default()
 //! )?;
-//! let export = WalletExport::export_wallet(&wallet, "exported wallet", true)
+//! let export = FullyNodedExport::export_wallet(&wallet, "exported wallet", true)
 //!     .map_err(ToString::to_string)
 //!     .map_err(bdk::Error::Generic)?;
 //!
@@ -64,16 +64,21 @@ use std::str::FromStr;
 use serde::{Deserialize, Serialize};
 
 use miniscript::descriptor::{ShInner, WshInner};
-use miniscript::{Descriptor, DescriptorPublicKey, ScriptContext, Terminal};
+use miniscript::{Descriptor, ScriptContext, Terminal};
 
 use crate::database::BatchDatabase;
+use crate::types::KeychainKind;
 use crate::wallet::Wallet;
 
+/// Alias for [`FullyNodedExport`]
+#[deprecated(since = "0.18.0", note = "Please use [`FullyNodedExport`] instead")]
+pub type WalletExport = FullyNodedExport;
+
 /// Structure that contains the export of a wallet
 ///
 /// For a usage example see [this module](crate::wallet::export)'s documentation.
 #[derive(Debug, Serialize, Deserialize)]
-pub struct WalletExport {
+pub struct FullyNodedExport {
     descriptor: String,
     /// Earliest block to rescan when looking for the wallet's transactions
     pub blockheight: u32,
@@ -81,13 +86,13 @@ pub struct WalletExport {
     pub label: String,
 }
 
-impl ToString for WalletExport {
+impl ToString for FullyNodedExport {
     fn to_string(&self) -> String {
         serde_json::to_string(self).unwrap()
     }
 }
 
-impl FromStr for WalletExport {
+impl FromStr for FullyNodedExport {
     type Err = serde_json::Error;
 
     fn from_str(s: &str) -> Result<Self, Self::Err> {
@@ -99,7 +104,7 @@ fn remove_checksum(s: String) -> String {
     s.splitn(2, '#').next().map(String::from).unwrap()
 }
 
-impl WalletExport {
+impl FullyNodedExport {
     /// Export a wallet
     ///
     /// This function returns an error if it determines that the `wallet`'s descriptor(s) are not
@@ -111,14 +116,18 @@ impl WalletExport {
     ///
     /// If the database is empty or `include_blockheight` is false, the `blockheight` field
     /// returned will be `0`.
-    pub fn export_wallet<B, D: BatchDatabase>(
+    pub fn export_wallet<D: BatchDatabase>(
-        wallet: &Wallet<B, D>,
+        wallet: &Wallet<D>,
         label: &str,
         include_blockheight: bool,
     ) -> Result<Self, &'static str> {
         let descriptor = wallet
-            .descriptor
+            .get_descriptor_for_keychain(KeychainKind::External)
-            .to_string_with_secret(&wallet.signers.as_key_map(wallet.secp_ctx()));
+            .to_string_with_secret(
+                &wallet
+                    .get_signers(KeychainKind::External)
+                    .as_key_map(wallet.secp_ctx()),
+            );
         let descriptor = remove_checksum(descriptor);
         Self::is_compatible_with_core(&descriptor)?;
 
@@ -136,18 +145,30 @@ impl WalletExport {
             }
         };
 
-        let export = WalletExport {
+        let export = FullyNodedExport {
             descriptor,
             label: label.into(),
             blockheight,
         };
 
-        let desc_to_string = |d: &Descriptor<DescriptorPublicKey>| {
+        let change_descriptor = match wallet
-            let descriptor =
+            .public_descriptor(KeychainKind::Internal)
-                d.to_string_with_secret(&wallet.change_signers.as_key_map(wallet.secp_ctx()));
+            .map_err(|_| "Invalid change descriptor")?
-            remove_checksum(descriptor)
+            .is_some()
+        {
+            false => None,
+            true => {
+                let descriptor = wallet
+                    .get_descriptor_for_keychain(KeychainKind::Internal)
+                    .to_string_with_secret(
+                        &wallet
+                            .get_signers(KeychainKind::Internal)
+                            .as_key_map(wallet.secp_ctx()),
+                    );
+                Some(remove_checksum(descriptor))
+            }
         };
-        if export.change_descriptor() != wallet.change_descriptor.as_ref().map(desc_to_string) {
+        if export.change_descriptor() != change_descriptor {
             return Err("Incompatible change descriptor");
         }
 
@@ -212,7 +233,7 @@ mod test {
     use crate::database::{memory::MemoryDatabase, BatchOperations};
     use crate::types::TransactionDetails;
     use crate::wallet::Wallet;
-    use crate::ConfirmationTime;
+    use crate::BlockTime;
 
     fn get_test_db() -> MemoryDatabase {
         let mut db = MemoryDatabase::new();
@@ -226,11 +247,10 @@ mod test {
             received: 100_000,
             sent: 0,
             fee: Some(500),
-            confirmation_time: Some(ConfirmationTime {
+            confirmation_time: Some(BlockTime {
                 timestamp: 12345678,
                 height: 5000,
             }),
-            verified: true,
         })
         .unwrap();
 
@@ -242,14 +262,14 @@ mod test {
         let descriptor = "wpkh(xprv9s21ZrQH143K4CTb63EaMxja1YiTnSEWKMbn23uoEnAzxjdUJRQkazCAtzxGm4LSoTSVTptoV9RbchnKPW9HxKtZumdyxyikZFDLhogJ5Uj/44'/0'/0'/0/*)";
         let change_descriptor = "wpkh(xprv9s21ZrQH143K4CTb63EaMxja1YiTnSEWKMbn23uoEnAzxjdUJRQkazCAtzxGm4LSoTSVTptoV9RbchnKPW9HxKtZumdyxyikZFDLhogJ5Uj/44'/0'/0'/1/*)";
 
-        let wallet = Wallet::new_offline(
+        let wallet = Wallet::new(
             descriptor,
             Some(change_descriptor),
             Network::Bitcoin,
             get_test_db(),
         )
         .unwrap();
-        let export = WalletExport::export_wallet(&wallet, "Test Label", true).unwrap();
+        let export = FullyNodedExport::export_wallet(&wallet, "Test Label", true).unwrap();
 
         assert_eq!(export.descriptor(), descriptor);
         assert_eq!(export.change_descriptor(), Some(change_descriptor.into()));
@@ -266,9 +286,8 @@ mod test {
 
         let descriptor = "wpkh(xprv9s21ZrQH143K4CTb63EaMxja1YiTnSEWKMbn23uoEnAzxjdUJRQkazCAtzxGm4LSoTSVTptoV9RbchnKPW9HxKtZumdyxyikZFDLhogJ5Uj/44'/0'/0'/0/*)";
 
-        let wallet =
+        let wallet = Wallet::new(descriptor, None, Network::Bitcoin, get_test_db()).unwrap();
-            Wallet::new_offline(descriptor, None, Network::Bitcoin, get_test_db()).unwrap();
+        FullyNodedExport::export_wallet(&wallet, "Test Label", true).unwrap();
-        WalletExport::export_wallet(&wallet, "Test Label", true).unwrap();
     }
 
     #[test]
@@ -280,14 +299,14 @@ mod test {
         let descriptor = "wpkh(xprv9s21ZrQH143K4CTb63EaMxja1YiTnSEWKMbn23uoEnAzxjdUJRQkazCAtzxGm4LSoTSVTptoV9RbchnKPW9HxKtZumdyxyikZFDLhogJ5Uj/44'/0'/0'/0/*)";
         let change_descriptor = "wpkh(xprv9s21ZrQH143K4CTb63EaMxja1YiTnSEWKMbn23uoEnAzxjdUJRQkazCAtzxGm4LSoTSVTptoV9RbchnKPW9HxKtZumdyxyikZFDLhogJ5Uj/50'/0'/1/*)";
 
-        let wallet = Wallet::new_offline(
+        let wallet = Wallet::new(
             descriptor,
             Some(change_descriptor),
             Network::Bitcoin,
             get_test_db(),
         )
         .unwrap();
-        WalletExport::export_wallet(&wallet, "Test Label", true).unwrap();
+        FullyNodedExport::export_wallet(&wallet, "Test Label", true).unwrap();
     }
 
     #[test]
@@ -303,14 +322,14 @@ mod test {
                                        [c98b1535/48'/0'/0'/2']tpubDCDi5W4sP6zSnzJeowy8rQDVhBdRARaPhK1axABi8V1661wEPeanpEXj4ZLAUEoikVtoWcyK26TKKJSecSfeKxwHCcRrge9k1ybuiL71z4a/1/*\
                                  ))";
 
-        let wallet = Wallet::new_offline(
+        let wallet = Wallet::new(
             descriptor,
             Some(change_descriptor),
             Network::Testnet,
             get_test_db(),
         )
         .unwrap();
-        let export = WalletExport::export_wallet(&wallet, "Test Label", true).unwrap();
+        let export = FullyNodedExport::export_wallet(&wallet, "Test Label", true).unwrap();
 
         assert_eq!(export.descriptor(), descriptor);
         assert_eq!(export.change_descriptor(), Some(change_descriptor.into()));
@@ -323,14 +342,14 @@ mod test {
         let descriptor = "wpkh(xprv9s21ZrQH143K4CTb63EaMxja1YiTnSEWKMbn23uoEnAzxjdUJRQkazCAtzxGm4LSoTSVTptoV9RbchnKPW9HxKtZumdyxyikZFDLhogJ5Uj/44'/0'/0'/0/*)";
         let change_descriptor = "wpkh(xprv9s21ZrQH143K4CTb63EaMxja1YiTnSEWKMbn23uoEnAzxjdUJRQkazCAtzxGm4LSoTSVTptoV9RbchnKPW9HxKtZumdyxyikZFDLhogJ5Uj/44'/0'/0'/1/*)";
 
-        let wallet = Wallet::new_offline(
+        let wallet = Wallet::new(
             descriptor,
             Some(change_descriptor),
             Network::Bitcoin,
             get_test_db(),
         )
         .unwrap();
-        let export = WalletExport::export_wallet(&wallet, "Test Label", true).unwrap();
+        let export = FullyNodedExport::export_wallet(&wallet, "Test Label", true).unwrap();
 
         assert_eq!(export.to_string(), "{\"descriptor\":\"wpkh(xprv9s21ZrQH143K4CTb63EaMxja1YiTnSEWKMbn23uoEnAzxjdUJRQkazCAtzxGm4LSoTSVTptoV9RbchnKPW9HxKtZumdyxyikZFDLhogJ5Uj/44\'/0\'/0\'/0/*)\",\"blockheight\":5000,\"label\":\"Test Label\"}");
     }
@@ -341,7 +360,7 @@ mod test {
         let change_descriptor = "wpkh(xprv9s21ZrQH143K4CTb63EaMxja1YiTnSEWKMbn23uoEnAzxjdUJRQkazCAtzxGm4LSoTSVTptoV9RbchnKPW9HxKtZumdyxyikZFDLhogJ5Uj/44'/0'/0'/1/*)";
 
         let import_str = "{\"descriptor\":\"wpkh(xprv9s21ZrQH143K4CTb63EaMxja1YiTnSEWKMbn23uoEnAzxjdUJRQkazCAtzxGm4LSoTSVTptoV9RbchnKPW9HxKtZumdyxyikZFDLhogJ5Uj/44\'/0\'/0\'/0/*)\",\"blockheight\":5000,\"label\":\"Test Label\"}";
-        let export = WalletExport::from_str(import_str).unwrap();
+        let export = FullyNodedExport::from_str(import_str).unwrap();
 
         assert_eq!(export.descriptor(), descriptor);
         assert_eq!(export.change_descriptor(), Some(change_descriptor.into()));

src/wallet/mod.rs

@@ -53,11 +53,11 @@ use address_validator::AddressValidator;
 use coin_selection::DefaultCoinSelectionAlgorithm;
 use signer::{SignOptions, Signer, SignerOrdering, SignersContainer};
 use tx_builder::{BumpFee, CreateTx, FeePolicy, TxBuilder, TxParams};
-use utils::{check_nlocktime, check_nsequence_rbf, After, Older, SecpCtx, DUST_LIMIT_SATOSHI};
+use utils::{check_nlocktime, check_nsequence_rbf, After, Older, SecpCtx};
 
-use crate::blockchain::{Blockchain, Progress};
+use crate::blockchain::{GetHeight, NoopProgress, Progress, WalletSync};
 use crate::database::memory::MemoryDatabase;
-use crate::database::{BatchDatabase, BatchOperations, DatabaseUtils};
+use crate::database::{BatchDatabase, BatchOperations, DatabaseUtils, SyncTime};
 use crate::descriptor::derived::AsDerived;
 use crate::descriptor::policy::BuildSatisfaction;
 use crate::descriptor::{
@@ -75,16 +75,17 @@ const CACHE_ADDR_BATCH_SIZE: u32 = 100;
 
 /// A Bitcoin wallet
 ///
-/// A wallet takes descriptors, a [`database`](trait@crate::database::Database) and a
+/// The `Wallet` struct acts as a way of coherently interfacing with output descriptors and related transactions.
-/// [`blockchain`](trait@crate::blockchain::Blockchain) and implements the basic functions that a Bitcoin wallets
+/// Its main components are:
-/// needs to operate, like [generating addresses](Wallet::get_address), [returning the balance](Wallet::get_balance),
-/// [creating transactions](Wallet::build_tx), etc.
 ///
-/// A wallet can be either "online" if the [`blockchain`](crate::blockchain) type provided
+/// 1. output *descriptors* from which it can derive addresses.
-/// implements [`Blockchain`], or "offline" if it is the unit type `()`. Offline wallets only expose
+/// 2. A [`Database`] where it tracks transactions and utxos related to the descriptors.
-/// methods that don't need any interaction with the blockchain to work.
+/// 3. [`Signer`]s that can contribute signatures to addresses instantiated from the descriptors.
+///
+/// [`Database`]: crate::database::Database
+/// [`Signer`]: crate::signer::Signer
 #[derive(Debug)]
-pub struct Wallet<B, D> {
+pub struct Wallet<D> {
     descriptor: ExtendedDescriptor,
     change_descriptor: Option<ExtendedDescriptor>,
 
@@ -95,88 +96,11 @@ pub struct Wallet<B, D> {
 
     network: Network,
 
-    current_height: Option<u32>,
-
-    client: B,
     database: RefCell<D>,
 
     secp: SecpCtx,
 }
 
-impl<D> Wallet<(), D>
-where
-    D: BatchDatabase,
-{
-    /// Create a new "offline" wallet
-    pub fn new_offline<E: IntoWalletDescriptor>(
-        descriptor: E,
-        change_descriptor: Option<E>,
-        network: Network,
-        database: D,
-    ) -> Result<Self, Error> {
-        Self::_new(descriptor, change_descriptor, network, database, (), None)
-    }
-}
-
-impl<B, D> Wallet<B, D>
-where
-    D: BatchDatabase,
-{
-    fn _new<E: IntoWalletDescriptor>(
-        descriptor: E,
-        change_descriptor: Option<E>,
-        network: Network,
-        mut database: D,
-        client: B,
-        current_height: Option<u32>,
-    ) -> Result<Self, Error> {
-        let secp = Secp256k1::new();
-
-        let (descriptor, keymap) = into_wallet_descriptor_checked(descriptor, &secp, network)?;
-        database.check_descriptor_checksum(
-            KeychainKind::External,
-            get_checksum(&descriptor.to_string())?.as_bytes(),
-        )?;
-        let signers = Arc::new(SignersContainer::from(keymap));
-        let (change_descriptor, change_signers) = match change_descriptor {
-            Some(desc) => {
-                let (change_descriptor, change_keymap) =
-                    into_wallet_descriptor_checked(desc, &secp, network)?;
-                database.check_descriptor_checksum(
-                    KeychainKind::Internal,
-                    get_checksum(&change_descriptor.to_string())?.as_bytes(),
-                )?;
-
-                let change_signers = Arc::new(SignersContainer::from(change_keymap));
-                // if !parsed.same_structure(descriptor.as_ref()) {
-                //     return Err(Error::DifferentDescriptorStructure);
-                // }
-
-                (Some(change_descriptor), change_signers)
-            }
-            None => (None, Arc::new(SignersContainer::new())),
-        };
-
-        Ok(Wallet {
-            descriptor,
-            change_descriptor,
-            signers,
-            change_signers,
-            address_validators: Vec::new(),
-            network,
-            current_height,
-            client,
-            database: RefCell::new(database),
-            secp,
-        })
-    }
-
-    /// Get the Bitcoin network the wallet is using.
-    pub fn network(&self) -> Network {
-        self.network
-    }
-}
-
 /// The address index selection strategy to use to derived an address from the wallet's external
 /// descriptor. See [`Wallet::get_address`]. If you're unsure which one to use use `WalletIndex::New`.
 #[derive(Debug)]
@@ -216,6 +140,8 @@ pub struct AddressInfo {
     pub index: u32,
     /// Address
     pub address: Address,
+    /// Type of keychain
+    pub keychain: KeychainKind,
 }
 
 impl Deref for AddressInfo {
@@ -232,17 +158,89 @@ impl fmt::Display for AddressInfo {
     }
 }
 
-// offline actions, always available
+#[derive(Debug, Default)]
-impl<B, D> Wallet<B, D>
+/// Options to a [`sync`].
+///
+/// [`sync`]: Wallet::sync
+pub struct SyncOptions {
+    /// The progress tracker which may be informed when progress is made.
+    pub progress: Option<Box<dyn Progress>>,
+}
+
+impl<D> Wallet<D>
 where
     D: BatchDatabase,
 {
-    // Return a newly derived address using the external descriptor
+    #[deprecated = "Just use Wallet::new -- all wallets are offline now!"]
-    fn get_new_address(&self) -> Result<AddressInfo, Error> {
+    /// Create a new "offline" wallet
-        let incremented_index = self.fetch_and_increment_index(KeychainKind::External)?;
+    pub fn new_offline<E: IntoWalletDescriptor>(
+        descriptor: E,
+        change_descriptor: Option<E>,
+        network: Network,
+        database: D,
+    ) -> Result<Self, Error> {
+        Self::new(descriptor, change_descriptor, network, database)
+    }
+
+    /// Create a wallet.
+    ///
+    /// The only way this can fail is if the descriptors passed in do not match the checksums in `database`.
+    pub fn new<E: IntoWalletDescriptor>(
+        descriptor: E,
+        change_descriptor: Option<E>,
+        network: Network,
+        mut database: D,
+    ) -> Result<Self, Error> {
+        let secp = Secp256k1::new();
+
+        let (descriptor, keymap) = into_wallet_descriptor_checked(descriptor, &secp, network)?;
+        database.check_descriptor_checksum(
+            KeychainKind::External,
+            get_checksum(&descriptor.to_string())?.as_bytes(),
+        )?;
+        let signers = Arc::new(SignersContainer::from(keymap));
+        let (change_descriptor, change_signers) = match change_descriptor {
+            Some(desc) => {
+                let (change_descriptor, change_keymap) =
+                    into_wallet_descriptor_checked(desc, &secp, network)?;
+                database.check_descriptor_checksum(
+                    KeychainKind::Internal,
+                    get_checksum(&change_descriptor.to_string())?.as_bytes(),
+                )?;
+
+                let change_signers = Arc::new(SignersContainer::from(change_keymap));
+                // if !parsed.same_structure(descriptor.as_ref()) {
+                //     return Err(Error::DifferentDescriptorStructure);
+                // }
+
+                (Some(change_descriptor), change_signers)
+            }
+            None => (None, Arc::new(SignersContainer::new())),
+        };
+
+        Ok(Wallet {
+            descriptor,
+            change_descriptor,
+            signers,
+            change_signers,
+            address_validators: Vec::new(),
+            network,
+            database: RefCell::new(database),
+            secp,
+        })
+    }
+
+    /// Get the Bitcoin network the wallet is using.
+    pub fn network(&self) -> Network {
+        self.network
+    }
+
+    // Return a newly derived address for the specified `keychain`.
+    fn get_new_address(&self, keychain: KeychainKind) -> Result<AddressInfo, Error> {
+        let incremented_index = self.fetch_and_increment_index(keychain)?;
 
         let address_result = self
-            .descriptor
+            .get_descriptor_for_keychain(keychain)
             .as_derived(incremented_index, &self.secp)
             .address(self.network);
 
@@ -250,16 +248,19 @@ where
             .map(|address| AddressInfo {
                 address,
                 index: incremented_index,
+                keychain,
             })
             .map_err(|_| Error::ScriptDoesntHaveAddressForm)
     }
 
-    // Return the the last previously derived address if it has not been used in a received
+    // Return the the last previously derived address for `keychain` if it has not been used in a
-    // transaction. Otherwise return a new address using [`Wallet::get_new_address`].
+    // received transaction. Otherwise return a new address using [`Wallet::get_new_address`].
-    fn get_unused_address(&self) -> Result<AddressInfo, Error> {
+    fn get_unused_address(&self, keychain: KeychainKind) -> Result<AddressInfo, Error> {
-        let current_index = self.fetch_index(KeychainKind::External)?;
+        let current_index = self.fetch_index(keychain)?;
 
-        let derived_key = self.descriptor.as_derived(current_index, &self.secp);
+        let derived_key = self
+            .get_descriptor_for_keychain(keychain)
+            .as_derived(current_index, &self.secp);
 
         let script_pubkey = derived_key.script_pubkey();
 
@@ -271,36 +272,45 @@ where
             .any(|o| o.script_pubkey == script_pubkey);
 
         if found_used {
-            self.get_new_address()
+            self.get_new_address(keychain)
         } else {
             derived_key
                 .address(self.network)
                 .map(|address| AddressInfo {
                     address,
                     index: current_index,
+                    keychain,
                 })
                 .map_err(|_| Error::ScriptDoesntHaveAddressForm)
         }
     }
 
-    // Return derived address for the external descriptor at a specific index
+    // Return derived address for the descriptor of given [`KeychainKind`] at a specific index
-    fn peek_address(&self, index: u32) -> Result<AddressInfo, Error> {
+    fn peek_address(&self, index: u32, keychain: KeychainKind) -> Result<AddressInfo, Error> {
-        self.descriptor
+        self.get_descriptor_for_keychain(keychain)
             .as_derived(index, &self.secp)
             .address(self.network)
-            .map(|address| AddressInfo { index, address })
+            .map(|address| AddressInfo {
+                index,
+                address,
+                keychain,
+            })
             .map_err(|_| Error::ScriptDoesntHaveAddressForm)
     }
 
-    // Return derived address for the external descriptor at a specific index and reset current
+    // Return derived address for `keychain` at a specific index and reset current
     // address index
-    fn reset_address(&self, index: u32) -> Result<AddressInfo, Error> {
+    fn reset_address(&self, index: u32, keychain: KeychainKind) -> Result<AddressInfo, Error> {
-        self.set_index(KeychainKind::External, index)?;
+        self.set_index(keychain, index)?;
 
-        self.descriptor
+        self.get_descriptor_for_keychain(keychain)
             .as_derived(index, &self.secp)
             .address(self.network)
-            .map(|address| AddressInfo { index, address })
+            .map(|address| AddressInfo {
+                index,
+                address,
+                keychain,
+            })
             .map_err(|_| Error::ScriptDoesntHaveAddressForm)
     }
 
@@ -308,14 +318,77 @@ where
     /// available address index selection strategies. If none of the keys in the descriptor are derivable
     /// (ie. does not end with /*) then the same address will always be returned for any [`AddressIndex`].
     pub fn get_address(&self, address_index: AddressIndex) -> Result<AddressInfo, Error> {
+        self._get_address(address_index, KeychainKind::External)
+    }
+
+    /// Return a derived address using the internal (change) descriptor.
+    ///
+    /// If the wallet doesn't have an internal descriptor it will use the external descriptor.
+    ///
+    /// see [`AddressIndex`] for available address index selection strategies. If none of the keys
+    /// in the descriptor are derivable (ie. does not end with /*) then the same address will always
+    /// be returned for any [`AddressIndex`].
+    pub fn get_internal_address(&self, address_index: AddressIndex) -> Result<AddressInfo, Error> {
+        self._get_address(address_index, KeychainKind::Internal)
+    }
+
+    fn _get_address(
+        &self,
+        address_index: AddressIndex,
+        keychain: KeychainKind,
+    ) -> Result<AddressInfo, Error> {
         match address_index {
-            AddressIndex::New => self.get_new_address(),
+            AddressIndex::New => self.get_new_address(keychain),
-            AddressIndex::LastUnused => self.get_unused_address(),
+            AddressIndex::LastUnused => self.get_unused_address(keychain),
-            AddressIndex::Peek(index) => self.peek_address(index),
+            AddressIndex::Peek(index) => self.peek_address(index, keychain),
-            AddressIndex::Reset(index) => self.reset_address(index),
+            AddressIndex::Reset(index) => self.reset_address(index, keychain),
         }
     }
 
+    /// Ensures that there are at least `max_addresses` addresses cached in the database if the
+    /// descriptor is derivable, or 1 address if it is not.
+    /// Will return `Ok(true)` if there are new addresses generated (either external or internal),
+    /// and `Ok(false)` if all the required addresses are already cached. This function is useful to
+    /// explicitly cache addresses in a wallet to do things like check [`Wallet::is_mine`] on
+    /// transaction output scripts.
+    pub fn ensure_addresses_cached(&self, max_addresses: u32) -> Result<bool, Error> {
+        let mut new_addresses_cached = false;
+        let max_address = match self.descriptor.is_deriveable() {
+            false => 0,
+            true => max_addresses,
+        };
+        debug!("max_address {}", max_address);
+        if self
+            .database
+            .borrow()
+            .get_script_pubkey_from_path(KeychainKind::External, max_address.saturating_sub(1))?
+            .is_none()
+        {
+            debug!("caching external addresses");
+            new_addresses_cached = true;
+            self.cache_addresses(KeychainKind::External, 0, max_address)?;
+        }
+
+        if let Some(change_descriptor) = &self.change_descriptor {
+            let max_address = match change_descriptor.is_deriveable() {
+                false => 0,
+                true => max_addresses,
+            };
+
+            if self
+                .database
+                .borrow()
+                .get_script_pubkey_from_path(KeychainKind::Internal, max_address.saturating_sub(1))?
+                .is_none()
+            {
+                debug!("caching internal addresses");
+                new_addresses_cached = true;
+                self.cache_addresses(KeychainKind::Internal, 0, max_address)?;
+            }
+        }
+        Ok(new_addresses_cached)
+    }
+
     /// Return whether or not a `script` is part of this wallet (either internal or external)
     pub fn is_mine(&self, script: &Script) -> Result<bool, Error> {
         self.database.borrow().is_mine(script)
@@ -323,10 +396,16 @@ where
 
     /// Return the list of unspent outputs of this wallet
     ///
-    /// Note that this methods only operate on the internal database, which first needs to be
+    /// Note that this method only operates on the internal database, which first needs to be
     /// [`Wallet::sync`] manually.
     pub fn list_unspent(&self) -> Result<Vec<LocalUtxo>, Error> {
-        self.database.borrow().iter_utxos()
+        Ok(self
+            .database
+            .borrow()
+            .iter_utxos()?
+            .into_iter()
+            .filter(|l| !l.is_spent)
+            .collect())
     }
 
     /// Returns the `UTXO` owned by this wallet corresponding to `outpoint` if it exists in the
@@ -335,6 +414,21 @@ where
         self.database.borrow().get_utxo(&outpoint)
     }
 
+    /// Return a single transactions made and received by the wallet
+    ///
+    /// Optionally fill the [`TransactionDetails::transaction`] field with the raw transaction if
+    /// `include_raw` is `true`.
+    ///
+    /// Note that this method only operates on the internal database, which first needs to be
+    /// [`Wallet::sync`] manually.
+    pub fn get_tx(
+        &self,
+        txid: &Txid,
+        include_raw: bool,
+    ) -> Result<Option<TransactionDetails>, Error> {
+        self.database.borrow().get_tx(txid, include_raw)
+    }
+
     /// Return the list of transactions made and received by the wallet
     ///
     /// Optionally fill the [`TransactionDetails::transaction`] field with the raw transaction if
@@ -374,6 +468,29 @@ where
         signers.add_external(signer.id(&self.secp), ordering, signer);
     }
 
+    /// Get the signers
+    ///
+    /// ## Example
+    ///
+    /// ```
+    /// # use bdk::{Wallet, KeychainKind};
+    /// # use bdk::bitcoin::Network;
+    /// # use bdk::database::MemoryDatabase;
+    /// let wallet = Wallet::new("wpkh(tprv8ZgxMBicQKsPe73PBRSmNbTfbcsZnwWhz5eVmhHpi31HW29Z7mc9B4cWGRQzopNUzZUT391DeDJxL2PefNunWyLgqCKRMDkU1s2s8bAfoSk/84'/0'/0'/0/*)", None, Network::Testnet, MemoryDatabase::new())?;
+    /// for secret_key in wallet.get_signers(KeychainKind::External).signers().iter().filter_map(|s| s.descriptor_secret_key()) {
+    ///     // secret_key: tprv8ZgxMBicQKsPe73PBRSmNbTfbcsZnwWhz5eVmhHpi31HW29Z7mc9B4cWGRQzopNUzZUT391DeDJxL2PefNunWyLgqCKRMDkU1s2s8bAfoSk/84'/0'/0'/0/*
+    ///     println!("secret_key: {}", secret_key);
+    /// }
+    ///
+    /// Ok::<(), Box<dyn std::error::Error>>(())
+    /// ```
+    pub fn get_signers(&self, keychain: KeychainKind) -> Arc<SignersContainer> {
+        match keychain {
+            KeychainKind::External => Arc::clone(&self.signers),
+            KeychainKind::Internal => Arc::clone(&self.change_signers),
+        }
+    }
+
     /// Add an address validator
     ///
     /// See [the `address_validator` module](address_validator) for an example.
@@ -381,6 +498,11 @@ where
         self.address_validators.push(validator);
     }
 
+    /// Get the address validators
+    pub fn get_address_validators(&self) -> &[Arc<dyn AddressValidator>] {
+        &self.address_validators
+    }
+
     /// Start building a transaction.
     ///
     /// This returns a blank [`TxBuilder`] from which you can specify the parameters for the transaction.
@@ -407,7 +529,7 @@ where
     /// ```
     ///
     /// [`TxBuilder`]: crate::TxBuilder
-    pub fn build_tx(&self) -> TxBuilder<'_, B, D, DefaultCoinSelectionAlgorithm, CreateTx> {
+    pub fn build_tx(&self) -> TxBuilder<'_, D, DefaultCoinSelectionAlgorithm, CreateTx> {
         TxBuilder {
             wallet: self,
             params: TxParams::default(),
@@ -586,7 +708,7 @@ where
         let recipients = params.recipients.iter().map(|(r, v)| (r, *v));
 
         for (index, (script_pubkey, value)) in recipients.enumerate() {
-            if value.is_dust() {
+            if value.is_dust(script_pubkey) && !script_pubkey.is_provably_unspendable() {
                 return Err(Error::OutputBelowDustLimit(index));
             }
 
@@ -647,7 +769,10 @@ where
         let mut drain_output = {
             let script_pubkey = match params.drain_to {
                 Some(ref drain_recipient) => drain_recipient.clone(),
-                None => self.get_change_address()?,
+                None => self
+                    .get_internal_address(AddressIndex::New)?
+                    .address
+                    .script_pubkey(),
             };
 
             TxOut {
@@ -662,9 +787,9 @@ where
 
         if tx.output.is_empty() {
             if params.drain_to.is_some() {
-                if drain_val.is_dust() {
+                if drain_val.is_dust(&drain_output.script_pubkey) {
                     return Err(Error::InsufficientFunds {
-                        needed: DUST_LIMIT_SATOSHI,
+                        needed: drain_output.script_pubkey.dust_value().as_sat(),
                         available: drain_val,
                     });
                 }
@@ -673,7 +798,7 @@ where
             }
         }
 
-        if drain_val.is_dust() {
+        if drain_val.is_dust(&drain_output.script_pubkey) {
             fee_amount += drain_val;
         } else {
             drain_output.value = drain_val;
@@ -697,7 +822,6 @@ where
             received,
             sent,
             fee: Some(fee_amount),
-            verified: true,
         };
 
         Ok((psbt, transaction_details))
@@ -706,7 +830,7 @@ where
     /// Bump the fee of a transaction previously created with this wallet.
     ///
     /// Returns an error if the transaction is already confirmed or doesn't explicitly signal
-    /// *repalce by fee* (RBF). If the transaction can be fee bumped then it returns a [`TxBuilder`]
+    /// *replace by fee* (RBF). If the transaction can be fee bumped then it returns a [`TxBuilder`]
     /// pre-populated with the inputs and outputs of the original transaction.
     ///
     /// ## Example
@@ -748,7 +872,7 @@ where
     pub fn build_fee_bump(
         &self,
         txid: Txid,
-    ) -> Result<TxBuilder<'_, B, D, DefaultCoinSelectionAlgorithm, BumpFee>, Error> {
+    ) -> Result<TxBuilder<'_, D, DefaultCoinSelectionAlgorithm, BumpFee>, Error> {
         let mut details = match self.database.borrow().get_tx(&txid, true)? {
             None => return Err(Error::TransactionNotFound),
             Some(tx) if tx.transaction.is_none() => return Err(Error::TransactionNotFound),
@@ -801,6 +925,7 @@ where
                     outpoint: txin.previous_output,
                     txout,
                     keychain,
+                    is_spent: true,
                 };
 
                 Ok(WeightedUtxo {
@@ -979,7 +1104,11 @@ where
                 .borrow()
                 .get_tx(&input.previous_output.txid, false)?
                 .map(|tx| tx.confirmation_time.map(|c| c.height).unwrap_or(u32::MAX));
-            let current_height = sign_options.assume_height.or(self.current_height);
+            let last_sync_height = self
+                .database()
+                .get_sync_time()?
+                .map(|sync_time| sync_time.block_time.height);
+            let current_height = sign_options.assume_height.or(last_sync_height);
 
             debug!(
                 "Input #{} - {}, using `create_height` = {:?}, `current_height` = {:?}",
@@ -1044,7 +1173,7 @@ where
         &self.secp
     }
 
-    /// Returns the descriptor used to create adddresses for a particular `keychain`.
+    /// Returns the descriptor used to create addresses for a particular `keychain`.
     pub fn get_descriptor_for_keychain(&self, keychain: KeychainKind) -> &ExtendedDescriptor {
         let (descriptor, _) = self._get_descriptor_for_keychain(keychain);
         descriptor
@@ -1077,13 +1206,6 @@ where
             .map(|(desc, child)| desc.as_derived(child, &self.secp)))
     }
 
-    fn get_change_address(&self) -> Result<Script, Error> {
-        let (desc, keychain) = self._get_descriptor_for_keychain(KeychainKind::Internal);
-        let index = self.fetch_and_increment_index(keychain)?;
-
-        Ok(desc.as_derived(index, &self.secp).script_pubkey())
-    }
-
     fn fetch_and_increment_index(&self, keychain: KeychainKind) -> Result<u32, Error> {
         let (descriptor, keychain) = self._get_descriptor_for_keychain(keychain);
         let index = match descriptor.is_deriveable() {
@@ -1432,122 +1554,59 @@ where
 
         Ok(())
     }
-}
 
-impl<B, D> Wallet<B, D>
+    /// Return an immutable reference to the internal database
-where
+    pub fn database(&self) -> impl std::ops::Deref<Target = D> + '_ {
-    B: Blockchain,
+        self.database.borrow()
-    D: BatchDatabase,
-{
-    /// Create a new "online" wallet
-    #[maybe_async]
-    pub fn new<E: IntoWalletDescriptor>(
-        descriptor: E,
-        change_descriptor: Option<E>,
-        network: Network,
-        database: D,
-        client: B,
-    ) -> Result<Self, Error> {
-        let current_height = Some(maybe_await!(client.get_height())? as u32);
-        Self::_new(
-            descriptor,
-            change_descriptor,
-            network,
-            database,
-            client,
-            current_height,
-        )
     }
 
     /// Sync the internal database with the blockchain
     #[maybe_async]
-    pub fn sync<P: 'static + Progress>(
+    pub fn sync<B: WalletSync + GetHeight>(
         &self,
-        progress_update: P,
+        blockchain: &B,
-        max_address_param: Option<u32>,
+        sync_opts: SyncOptions,
     ) -> Result<(), Error> {
         debug!("Begin sync...");
 
-        let mut run_setup = false;
+        let SyncOptions { progress } = sync_opts;
+        let progress = progress.unwrap_or_else(|| Box::new(NoopProgress));
 
-        let max_address = match self.descriptor.is_deriveable() {
+        let run_setup = self.ensure_addresses_cached(CACHE_ADDR_BATCH_SIZE)?;
-            false => 0,
-            true => max_address_param.unwrap_or(CACHE_ADDR_BATCH_SIZE),
-        };
-        debug!("max_address {}", max_address);
-        if self
-            .database
-            .borrow()
-            .get_script_pubkey_from_path(KeychainKind::External, max_address.saturating_sub(1))?
-            .is_none()
-        {
-            debug!("caching external addresses");
-            run_setup = true;
-            self.cache_addresses(KeychainKind::External, 0, max_address)?;
-        }
-
-        if let Some(change_descriptor) = &self.change_descriptor {
-            let max_address = match change_descriptor.is_deriveable() {
-                false => 0,
-                true => max_address_param.unwrap_or(CACHE_ADDR_BATCH_SIZE),
-            };
-
-            if self
-                .database
-                .borrow()
-                .get_script_pubkey_from_path(KeychainKind::Internal, max_address.saturating_sub(1))?
-                .is_none()
-            {
-                debug!("caching internal addresses");
-                run_setup = true;
-                self.cache_addresses(KeychainKind::Internal, 0, max_address)?;
-            }
-        }
 
         debug!("run_setup: {}", run_setup);
         // TODO: what if i generate an address first and cache some addresses?
         // TODO: we should sync if generating an address triggers a new batch to be stored
         if run_setup {
-            maybe_await!(self
+            maybe_await!(
-                .client
+                blockchain.wallet_setup(self.database.borrow_mut().deref_mut(), progress,)
-                .setup(self.database.borrow_mut().deref_mut(), progress_update,))?;
+            )?;
         } else {
-            maybe_await!(self
+            maybe_await!(blockchain.wallet_sync(self.database.borrow_mut().deref_mut(), progress,))?;
-                .client
-                .sync(self.database.borrow_mut().deref_mut(), progress_update,))?;
         }
 
-        #[cfg(feature = "verify")]
+        let sync_time = SyncTime {
-        {
+            block_time: BlockTime {
-            debug!("Verifying transactions...");
+                height: maybe_await!(blockchain.get_height())?,
-            for mut tx in self.database.borrow().iter_txs(true)? {
+                timestamp: time::get_timestamp(),
-                if !tx.verified {
+            },
-                    verify::verify_tx(
+        };
-                        tx.transaction.as_ref().ok_or(Error::TransactionNotFound)?,
+        debug!("Saving `sync_time` = {:?}", sync_time);
-                        self.database.borrow().deref(),
+        self.database.borrow_mut().set_sync_time(sync_time)?;
-                        &self.client,
-                    )?;
-
-                    tx.verified = true;
-                    self.database.borrow_mut().set_tx(&tx)?;
-                }
-            }
-        }
 
         Ok(())
     }
 
-    /// Return a reference to the internal blockchain client
+    /// Return the checksum of the public descriptor associated to `keychain`
-    pub fn client(&self) -> &B {
+    ///
-        &self.client
+    /// Internally calls [`Self::get_descriptor_for_keychain`] to fetch the right descriptor
-    }
+    pub fn descriptor_checksum(&self, keychain: KeychainKind) -> String {
-
+        self.get_descriptor_for_keychain(keychain)
-    /// Broadcast a transaction to the network
+            .to_string()
-    #[maybe_async]
+            .splitn(2, '#')
-    pub fn broadcast(&self, tx: Transaction) -> Result<Txid, Error> {
+            .next()
-        maybe_await!(self.client.broadcast(&tx))?;
+            .unwrap()
-
+            .to_string()
-        Ok(tx.txid())
     }
 }
 
@@ -1555,12 +1614,12 @@ where
 pub fn get_funded_wallet(
     descriptor: &str,
 ) -> (
-    Wallet<(), MemoryDatabase>,
+    Wallet<MemoryDatabase>,
     (String, Option<String>),
     bitcoin::Txid,
 ) {
     let descriptors = testutils!(@descriptors (descriptor));
-    let wallet = Wallet::new_offline(
+    let wallet = Wallet::new(
         &descriptors.0,
         None,
         Network::Regtest,
@@ -1610,7 +1669,7 @@ pub(crate) mod test {
     #[test]
     fn test_cache_addresses_fixed() {
         let db = MemoryDatabase::new();
-        let wallet = Wallet::new_offline(
+        let wallet = Wallet::new(
             "wpkh(L5EZftvrYaSudiozVRzTqLcHLNDoVn7H5HSfM9BAN6tMJX8oTWz6)",
             None,
             Network::Testnet,
@@ -1644,7 +1703,7 @@ pub(crate) mod test {
     #[test]
     fn test_cache_addresses() {
         let db = MemoryDatabase::new();
-        let wallet = Wallet::new_offline("wpkh(tpubEBr4i6yk5nf5DAaJpsi9N2pPYBeJ7fZ5Z9rmN4977iYLCGco1VyjB9tvvuvYtfZzjD5A8igzgw3HeWeeKFmanHYqksqZXYXGsw5zjnj7KM9/*)", None, Network::Testnet, db).unwrap();
+        let wallet = Wallet::new("wpkh(tpubEBr4i6yk5nf5DAaJpsi9N2pPYBeJ7fZ5Z9rmN4977iYLCGco1VyjB9tvvuvYtfZzjD5A8igzgw3HeWeeKFmanHYqksqZXYXGsw5zjnj7KM9/*)", None, Network::Testnet, db).unwrap();
 
         assert_eq!(
             wallet.get_address(New).unwrap().to_string(),
@@ -1672,7 +1731,7 @@ pub(crate) mod test {
     #[test]
     fn test_cache_addresses_refill() {
         let db = MemoryDatabase::new();
-        let wallet = Wallet::new_offline("wpkh(tpubEBr4i6yk5nf5DAaJpsi9N2pPYBeJ7fZ5Z9rmN4977iYLCGco1VyjB9tvvuvYtfZzjD5A8igzgw3HeWeeKFmanHYqksqZXYXGsw5zjnj7KM9/*)", None, Network::Testnet, db).unwrap();
+        let wallet = Wallet::new("wpkh(tpubEBr4i6yk5nf5DAaJpsi9N2pPYBeJ7fZ5Z9rmN4977iYLCGco1VyjB9tvvuvYtfZzjD5A8igzgw3HeWeeKFmanHYqksqZXYXGsw5zjnj7KM9/*)", None, Network::Testnet, db).unwrap();
 
         assert_eq!(
             wallet.get_address(New).unwrap().to_string(),
@@ -2763,7 +2822,7 @@ pub(crate) mod test {
         let txid = tx.txid();
         // skip saving the utxos, we know they can't be used anyways
         details.transaction = Some(tx);
-        details.confirmation_time = Some(ConfirmationTime {
+        details.confirmation_time = Some(BlockTime {
             timestamp: 12345678,
             height: 42,
         });
@@ -3395,7 +3454,7 @@ pub(crate) mod test {
             .unwrap();
 
         let mut builder = wallet.build_fee_bump(txid).unwrap();
-        builder.fee_rate(FeeRate::from_sat_per_vb(140.0));
+        builder.fee_rate(FeeRate::from_sat_per_vb(141.0));
         let (psbt, details) = builder.finish().unwrap();
 
         assert_eq!(
@@ -3770,7 +3829,7 @@ pub(crate) mod test {
     #[test]
     fn test_unused_address() {
         let db = MemoryDatabase::new();
-        let wallet = Wallet::new_offline("wpkh(tpubEBr4i6yk5nf5DAaJpsi9N2pPYBeJ7fZ5Z9rmN4977iYLCGco1VyjB9tvvuvYtfZzjD5A8igzgw3HeWeeKFmanHYqksqZXYXGsw5zjnj7KM9/*)",
+        let wallet = Wallet::new("wpkh(tpubEBr4i6yk5nf5DAaJpsi9N2pPYBeJ7fZ5Z9rmN4977iYLCGco1VyjB9tvvuvYtfZzjD5A8igzgw3HeWeeKFmanHYqksqZXYXGsw5zjnj7KM9/*)",
                                          None, Network::Testnet, db).unwrap();
 
         assert_eq!(
@@ -3787,7 +3846,7 @@ pub(crate) mod test {
     fn test_next_unused_address() {
         let descriptor = "wpkh(tpubEBr4i6yk5nf5DAaJpsi9N2pPYBeJ7fZ5Z9rmN4977iYLCGco1VyjB9tvvuvYtfZzjD5A8igzgw3HeWeeKFmanHYqksqZXYXGsw5zjnj7KM9/*)";
         let descriptors = testutils!(@descriptors (descriptor));
-        let wallet = Wallet::new_offline(
+        let wallet = Wallet::new(
             &descriptors.0,
             None,
             Network::Testnet,
@@ -3816,7 +3875,7 @@ pub(crate) mod test {
     #[test]
     fn test_peek_address_at_index() {
         let db = MemoryDatabase::new();
-        let wallet = Wallet::new_offline("wpkh(tpubEBr4i6yk5nf5DAaJpsi9N2pPYBeJ7fZ5Z9rmN4977iYLCGco1VyjB9tvvuvYtfZzjD5A8igzgw3HeWeeKFmanHYqksqZXYXGsw5zjnj7KM9/*)",
+        let wallet = Wallet::new("wpkh(tpubEBr4i6yk5nf5DAaJpsi9N2pPYBeJ7fZ5Z9rmN4977iYLCGco1VyjB9tvvuvYtfZzjD5A8igzgw3HeWeeKFmanHYqksqZXYXGsw5zjnj7KM9/*)",
                                          None, Network::Testnet, db).unwrap();
 
         assert_eq!(
@@ -3849,7 +3908,7 @@ pub(crate) mod test {
     #[test]
     fn test_peek_address_at_index_not_derivable() {
         let db = MemoryDatabase::new();
-        let wallet = Wallet::new_offline("wpkh(tpubEBr4i6yk5nf5DAaJpsi9N2pPYBeJ7fZ5Z9rmN4977iYLCGco1VyjB9tvvuvYtfZzjD5A8igzgw3HeWeeKFmanHYqksqZXYXGsw5zjnj7KM9/1)",
+        let wallet = Wallet::new("wpkh(tpubEBr4i6yk5nf5DAaJpsi9N2pPYBeJ7fZ5Z9rmN4977iYLCGco1VyjB9tvvuvYtfZzjD5A8igzgw3HeWeeKFmanHYqksqZXYXGsw5zjnj7KM9/1)",
                                          None, Network::Testnet, db).unwrap();
 
         assert_eq!(
@@ -3871,7 +3930,7 @@ pub(crate) mod test {
     #[test]
     fn test_reset_address_index() {
         let db = MemoryDatabase::new();
-        let wallet = Wallet::new_offline("wpkh(tpubEBr4i6yk5nf5DAaJpsi9N2pPYBeJ7fZ5Z9rmN4977iYLCGco1VyjB9tvvuvYtfZzjD5A8igzgw3HeWeeKFmanHYqksqZXYXGsw5zjnj7KM9/*)",
+        let wallet = Wallet::new("wpkh(tpubEBr4i6yk5nf5DAaJpsi9N2pPYBeJ7fZ5Z9rmN4977iYLCGco1VyjB9tvvuvYtfZzjD5A8igzgw3HeWeeKFmanHYqksqZXYXGsw5zjnj7KM9/*)",
                                          None, Network::Testnet, db).unwrap();
 
         // new index 0
@@ -3908,7 +3967,7 @@ pub(crate) mod test {
     #[test]
     fn test_returns_index_and_address() {
         let db = MemoryDatabase::new();
-        let wallet = Wallet::new_offline("wpkh(tpubEBr4i6yk5nf5DAaJpsi9N2pPYBeJ7fZ5Z9rmN4977iYLCGco1VyjB9tvvuvYtfZzjD5A8igzgw3HeWeeKFmanHYqksqZXYXGsw5zjnj7KM9/*)",
+        let wallet = Wallet::new("wpkh(tpubEBr4i6yk5nf5DAaJpsi9N2pPYBeJ7fZ5Z9rmN4977iYLCGco1VyjB9tvvuvYtfZzjD5A8igzgw3HeWeeKFmanHYqksqZXYXGsw5zjnj7KM9/*)",
                                          None, Network::Testnet, db).unwrap();
 
         // new index 0
@@ -3917,6 +3976,7 @@ pub(crate) mod test {
             AddressInfo {
                 index: 0,
                 address: Address::from_str("tb1q6yn66vajcctph75pvylgkksgpp6nq04ppwct9a").unwrap(),
+                keychain: KeychainKind::External,
             }
         );
 
@@ -3925,7 +3985,8 @@ pub(crate) mod test {
             wallet.get_address(New).unwrap(),
             AddressInfo {
                 index: 1,
-                address: Address::from_str("tb1q4er7kxx6sssz3q7qp7zsqsdx4erceahhax77d7").unwrap()
+                address: Address::from_str("tb1q4er7kxx6sssz3q7qp7zsqsdx4erceahhax77d7").unwrap(),
+                keychain: KeychainKind::External,
             }
         );
 
@@ -3934,7 +3995,8 @@ pub(crate) mod test {
             wallet.get_address(Peek(25)).unwrap(),
             AddressInfo {
                 index: 25,
-                address: Address::from_str("tb1qsp7qu0knx3sl6536dzs0703u2w2ag6ppl9d0c2").unwrap()
+                address: Address::from_str("tb1qsp7qu0knx3sl6536dzs0703u2w2ag6ppl9d0c2").unwrap(),
+                keychain: KeychainKind::External,
             }
         );
 
@@ -3943,7 +4005,8 @@ pub(crate) mod test {
             wallet.get_address(New).unwrap(),
             AddressInfo {
                 index: 2,
-                address: Address::from_str("tb1qzntf2mqex4ehwkjlfdyy3ewdlk08qkvkvrz7x2").unwrap()
+                address: Address::from_str("tb1qzntf2mqex4ehwkjlfdyy3ewdlk08qkvkvrz7x2").unwrap(),
+                keychain: KeychainKind::External,
             }
         );
 
@@ -3952,7 +4015,8 @@ pub(crate) mod test {
             wallet.get_address(Reset(1)).unwrap(),
             AddressInfo {
                 index: 1,
-                address: Address::from_str("tb1q4er7kxx6sssz3q7qp7zsqsdx4erceahhax77d7").unwrap()
+                address: Address::from_str("tb1q4er7kxx6sssz3q7qp7zsqsdx4erceahhax77d7").unwrap(),
+                keychain: KeychainKind::External,
             }
         );
 
@@ -3961,8 +4025,98 @@ pub(crate) mod test {
             wallet.get_address(New).unwrap(),
             AddressInfo {
                 index: 2,
-                address: Address::from_str("tb1qzntf2mqex4ehwkjlfdyy3ewdlk08qkvkvrz7x2").unwrap()
+                address: Address::from_str("tb1qzntf2mqex4ehwkjlfdyy3ewdlk08qkvkvrz7x2").unwrap(),
+                keychain: KeychainKind::External,
             }
         );
     }
+
+    #[test]
+    fn test_sending_to_bip350_bech32m_address() {
+        let (wallet, _, _) = get_funded_wallet(get_test_wpkh());
+        let addr =
+            Address::from_str("tb1pqqqqp399et2xygdj5xreqhjjvcmzhxw4aywxecjdzew6hylgvsesf3hn0c")
+                .unwrap();
+        let mut builder = wallet.build_tx();
+        builder.add_recipient(addr.script_pubkey(), 45_000);
+        builder.finish().unwrap();
+    }
+
+    #[test]
+    fn test_get_address() {
+        use crate::descriptor::template::Bip84;
+        let key = bitcoin::util::bip32::ExtendedPrivKey::from_str("tprv8ZgxMBicQKsPcx5nBGsR63Pe8KnRUqmbJNENAfGftF3yuXoMMoVJJcYeUw5eVkm9WBPjWYt6HMWYJNesB5HaNVBaFc1M6dRjWSYnmewUMYy").unwrap();
+        let wallet = Wallet::new(
+            Bip84(key, KeychainKind::External),
+            Some(Bip84(key, KeychainKind::Internal)),
+            Network::Regtest,
+            MemoryDatabase::default(),
+        )
+        .unwrap();
+
+        assert_eq!(
+            wallet.get_address(AddressIndex::New).unwrap(),
+            AddressInfo {
+                index: 0,
+                address: Address::from_str("bcrt1qkmvk2nadgplmd57ztld8nf8v2yxkzmdvwtjf8s").unwrap(),
+                keychain: KeychainKind::External,
+            }
+        );
+
+        assert_eq!(
+            wallet.get_internal_address(AddressIndex::New).unwrap(),
+            AddressInfo {
+                index: 0,
+                address: Address::from_str("bcrt1qtrwtz00wxl69e5xex7amy4xzlxkaefg3gfdkxa").unwrap(),
+                keychain: KeychainKind::Internal,
+            }
+        );
+
+        let wallet = Wallet::new(
+            Bip84(key, KeychainKind::External),
+            None,
+            Network::Regtest,
+            MemoryDatabase::default(),
+        )
+        .unwrap();
+
+        assert_eq!(
+            wallet.get_internal_address(AddressIndex::New).unwrap(),
+            AddressInfo {
+                index: 0,
+                address: Address::from_str("bcrt1qkmvk2nadgplmd57ztld8nf8v2yxkzmdvwtjf8s").unwrap(),
+                keychain: KeychainKind::Internal,
+            },
+            "when there's no internal descriptor it should just use external"
+        );
+    }
+}
+
+/// Deterministically generate a unique name given the descriptors defining the wallet
+pub fn wallet_name_from_descriptor<T>(
+    descriptor: T,
+    change_descriptor: Option<T>,
+    network: Network,
+    secp: &SecpCtx,
+) -> Result<String, Error>
+where
+    T: IntoWalletDescriptor,
+{
+    //TODO check descriptors contains only public keys
+    let descriptor = descriptor
+        .into_wallet_descriptor(secp, network)?
+        .0
+        .to_string();
+    let mut wallet_name = get_checksum(&descriptor[..descriptor.find('#').unwrap()])?;
+    if let Some(change_descriptor) = change_descriptor {
+        let change_descriptor = change_descriptor
+            .into_wallet_descriptor(secp, network)?
+            .0
+            .to_string();
+        wallet_name.push_str(
+            get_checksum(&change_descriptor[..change_descriptor.find('#').unwrap()])?.as_str(),
+        );
+    }
+
+    Ok(wallet_name)
 }

src/wallet/signer.rs

@@ -72,7 +72,7 @@
 //! let custom_signer = CustomSigner::connect();
 //!
 //! let descriptor = "wpkh(tpubD6NzVbkrYhZ4Xferm7Pz4VnjdcDPFyjVu5K4iZXQ4pVN8Cks4pHVowTBXBKRhX64pkRyJZJN5xAKj4UDNnLPb5p2sSKXhewoYx5GbTdUFWq/*)";
-//! let mut wallet = Wallet::new_offline(descriptor, None, Network::Testnet, MemoryDatabase::default())?;
+//! let mut wallet = Wallet::new(descriptor, None, Network::Testnet, MemoryDatabase::default())?;
 //! wallet.add_signer(
 //!     KeychainKind::External,
 //!     SignerOrdering(200),
@@ -143,7 +143,7 @@ pub enum SignerError {
     InvalidNonWitnessUtxo,
     /// The `witness_utxo` field of the transaction is required to sign this input
     MissingWitnessUtxo,
-    /// The `witness_script` field of the transaction is requied to sign this input
+    /// The `witness_script` field of the transaction is required to sign this input
     MissingWitnessScript,
     /// The fingerprint and derivation path are missing from the psbt input
     MissingHdKeypath,
@@ -289,7 +289,7 @@ impl Signer for PrivateKey {
         }
 
         // FIXME: use the presence of `witness_utxo` as an indication that we should make a bip143
-        // sig. Does this make sense? Should we add an extra argument to explicitly swith between
+        // sig. Does this make sense? Should we add an extra argument to explicitly switch between
         // these? The original idea was to declare sign() as sign<Ctx: ScriptContex>() and use Ctx,
         // but that violates the rules for trait-objects, so we can't do it.
         let (hash, sighash) = match psbt.inputs[input_index].witness_utxo {

src/wallet/tx_builder.rs

@@ -120,8 +120,8 @@ impl TxBuilderContext for BumpFee {}
 /// [`finish`]: Self::finish
 /// [`coin_selection`]: Self::coin_selection
 #[derive(Debug)]
-pub struct TxBuilder<'a, B, D, Cs, Ctx> {
+pub struct TxBuilder<'a, D, Cs, Ctx> {
-    pub(crate) wallet: &'a Wallet<B, D>,
+    pub(crate) wallet: &'a Wallet<D>,
     pub(crate) params: TxParams,
     pub(crate) coin_selection: Cs,
     pub(crate) phantom: PhantomData<Ctx>,
@@ -170,7 +170,7 @@ impl std::default::Default for FeePolicy {
     }
 }
 
-impl<'a, Cs: Clone, Ctx, B, D> Clone for TxBuilder<'a, B, D, Cs, Ctx> {
+impl<'a, Cs: Clone, Ctx, D> Clone for TxBuilder<'a, D, Cs, Ctx> {
     fn clone(&self) -> Self {
         TxBuilder {
             wallet: self.wallet,
@@ -182,8 +182,8 @@ impl<'a, Cs: Clone, Ctx, B, D> Clone for TxBuilder<'a, B, D, Cs, Ctx> {
 }
 
 // methods supported by both contexts, for any CoinSelectionAlgorithm
-impl<'a, B, D: BatchDatabase, Cs: CoinSelectionAlgorithm<D>, Ctx: TxBuilderContext>
+impl<'a, D: BatchDatabase, Cs: CoinSelectionAlgorithm<D>, Ctx: TxBuilderContext>
-    TxBuilder<'a, B, D, Cs, Ctx>
+    TxBuilder<'a, D, Cs, Ctx>
 {
     /// Set a custom fee rate
     pub fn fee_rate(&mut self, fee_rate: FeeRate) -> &mut Self {
@@ -310,7 +310,7 @@ impl<'a, B, D: BatchDatabase, Cs: CoinSelectionAlgorithm<D>, Ctx: TxBuilderConte
     /// 2. `psbt_input`: To know the value.
     /// 3. `satisfaction_weight`: To know how much weight/vbytes the input will add to the transaction for fee calculation.
     ///
-    /// There are several security concerns about adding foregin UTXOs that application
+    /// There are several security concerns about adding foreign UTXOs that application
     /// developers should consider. First, how do you know the value of the input is correct? If a
     /// `non_witness_utxo` is provided in the `psbt_input` then this method implicitly verifies the
     /// value by checking it against the transaction. If only a `witness_utxo` is provided then this
@@ -508,7 +508,7 @@ impl<'a, B, D: BatchDatabase, Cs: CoinSelectionAlgorithm<D>, Ctx: TxBuilderConte
     pub fn coin_selection<P: CoinSelectionAlgorithm<D>>(
         self,
         coin_selection: P,
-    ) -> TxBuilder<'a, B, D, P, Ctx> {
+    ) -> TxBuilder<'a, D, P, Ctx> {
         TxBuilder {
             wallet: self.wallet,
             params: self.params,
@@ -547,7 +547,7 @@ impl<'a, B, D: BatchDatabase, Cs: CoinSelectionAlgorithm<D>, Ctx: TxBuilderConte
     }
 }
 
-impl<'a, B, D: BatchDatabase, Cs: CoinSelectionAlgorithm<D>> TxBuilder<'a, B, D, Cs, CreateTx> {
+impl<'a, D: BatchDatabase, Cs: CoinSelectionAlgorithm<D>> TxBuilder<'a, D, Cs, CreateTx> {
     /// Replace the recipients already added with a new list
     pub fn set_recipients(&mut self, recipients: Vec<(Script, u64)>) -> &mut Self {
         self.params.recipients = recipients;
@@ -560,6 +560,13 @@ impl<'a, B, D: BatchDatabase, Cs: CoinSelectionAlgorithm<D>> TxBuilder<'a, B, D,
         self
     }
 
+    /// Add data as an output, using OP_RETURN
+    pub fn add_data(&mut self, data: &[u8]) -> &mut Self {
+        let script = Script::new_op_return(data);
+        self.add_recipient(script, 0u64);
+        self
+    }
+
     /// Sets the address to *drain* excess coins to.
     ///
     /// Usually, when there are excess coins they are sent to a change address generated by the
@@ -607,7 +614,7 @@ impl<'a, B, D: BatchDatabase, Cs: CoinSelectionAlgorithm<D>> TxBuilder<'a, B, D,
 }
 
 // methods supported only by bump_fee
-impl<'a, B, D: BatchDatabase> TxBuilder<'a, B, D, DefaultCoinSelectionAlgorithm, BumpFee> {
+impl<'a, D: BatchDatabase> TxBuilder<'a, D, DefaultCoinSelectionAlgorithm, BumpFee> {
     /// Explicitly tells the wallet that it is allowed to reduce the fee of the output matching this
     /// `script_pubkey` in order to bump the transaction fee. Without specifying this the wallet
     /// will attempt to find a change output to shrink instead.
@@ -831,6 +838,7 @@ mod test {
                 },
                 txout: Default::default(),
                 keychain: KeychainKind::External,
+                is_spent: false,
             },
             LocalUtxo {
                 outpoint: OutPoint {
@@ -839,6 +847,7 @@ mod test {
                 },
                 txout: Default::default(),
                 keychain: KeychainKind::Internal,
+                is_spent: false,
             },
         ]
     }

src/wallet/utils.rs

@@ -9,13 +9,11 @@
 // You may not use this file except in accordance with one or both of these
 // licenses.
 
+use bitcoin::blockdata::script::Script;
 use bitcoin::secp256k1::{All, Secp256k1};
 
 use miniscript::{MiniscriptKey, Satisfier, ToPublicKey};
 
-// De-facto standard "dust limit" (even though it should change based on the output type)
-pub const DUST_LIMIT_SATOSHI: u64 = 546;
-
 // MSB of the nSequence. If set there's no consensus-constraint, so it must be disabled when
 // spending using CSV in order to enforce CSV rules
 pub(crate) const SEQUENCE_LOCKTIME_DISABLE_FLAG: u32 = 1 << 31;
@@ -28,18 +26,19 @@ pub(crate) const SEQUENCE_LOCKTIME_MASK: u32 = 0x0000FFFF;
 // Threshold for nLockTime to be considered a block-height-based timelock rather than time-based
 pub(crate) const BLOCKS_TIMELOCK_THRESHOLD: u32 = 500000000;
 
-/// Trait to check if a value is below the dust limit
+/// Trait to check if a value is below the dust limit.
+/// We are performing dust value calculation for a given script public key using rust-bitcoin to
+/// keep it compatible with network dust rate
 // we implement this trait to make sure we don't mess up the comparison with off-by-one like a <
-// instead of a <= etc. The constant value for the dust limit is not public on purpose, to
+// instead of a <= etc.
-// encourage the usage of this trait.
 pub trait IsDust {
     /// Check whether or not a value is below dust limit
-    fn is_dust(&self) -> bool;
+    fn is_dust(&self, script: &Script) -> bool;
 }
 
 impl IsDust for u64 {
-    fn is_dust(&self) -> bool {
+    fn is_dust(&self, script: &Script) -> bool {
-        *self <= DUST_LIMIT_SATOSHI
+        *self < script.dust_value().as_sat()
     }
 }
 
@@ -138,47 +137,32 @@ impl<Pk: MiniscriptKey + ToPublicKey> Satisfier<Pk> for Older {
 
 pub(crate) type SecpCtx = Secp256k1<All>;
 
-pub struct ChunksIterator<I: Iterator> {
-    iter: I,
-    size: usize,
-}
-
-#[cfg(any(feature = "electrum", feature = "esplora"))]
-impl<I: Iterator> ChunksIterator<I> {
-    pub fn new(iter: I, size: usize) -> Self {
-        ChunksIterator { iter, size }
-    }
-}
-
-impl<I: Iterator> Iterator for ChunksIterator<I> {
-    type Item = Vec<<I as std::iter::Iterator>::Item>;
-
-    fn next(&mut self) -> Option<Self::Item> {
-        let mut v = Vec::new();
-        for _ in 0..self.size {
-            let e = self.iter.next();
-
-            match e {
-                None => break,
-                Some(val) => v.push(val),
-            }
-        }
-
-        if v.is_empty() {
-            return None;
-        }
-
-        Some(v)
-    }
-}
-
 #[cfg(test)]
 mod test {
     use super::{
-        check_nlocktime, check_nsequence_rbf, BLOCKS_TIMELOCK_THRESHOLD,
+        check_nlocktime, check_nsequence_rbf, IsDust, BLOCKS_TIMELOCK_THRESHOLD,
         SEQUENCE_LOCKTIME_TYPE_FLAG,
     };
+    use crate::bitcoin::Address;
     use crate::types::FeeRate;
+    use std::str::FromStr;
+
+    #[test]
+    fn test_is_dust() {
+        let script_p2pkh = Address::from_str("1GNgwA8JfG7Kc8akJ8opdNWJUihqUztfPe")
+            .unwrap()
+            .script_pubkey();
+        assert!(script_p2pkh.is_p2pkh());
+        assert!(545.is_dust(&script_p2pkh));
+        assert!(!546.is_dust(&script_p2pkh));
+
+        let script_p2wpkh = Address::from_str("bc1qxlh2mnc0yqwas76gqq665qkggee5m98t8yskd8")
+            .unwrap()
+            .script_pubkey();
+        assert!(script_p2wpkh.is_v0_p2wpkh());
+        assert!(293.is_dust(&script_p2wpkh));
+        assert!(!294.is_dust(&script_p2wpkh));
+    }
 
     #[test]
     fn test_fee_from_btc_per_kb() {

src/wallet/verify.rs

@@ -17,7 +17,7 @@ use std::fmt;
 use bitcoin::consensus::serialize;
 use bitcoin::{OutPoint, Transaction, Txid};
 
-use crate::blockchain::Blockchain;
+use crate::blockchain::GetTx;
 use crate::database::Database;
 use crate::error::Error;
 
@@ -29,7 +29,7 @@ use crate::error::Error;
 /// Depending on the [capabilities](crate::blockchain::Blockchain::get_capabilities) of the
 /// [`Blockchain`] backend, the method could fail when called with old "historical" transactions or
 /// with unconfirmed transactions that have been evicted from the backend's memory.
-pub fn verify_tx<D: Database, B: Blockchain>(
+pub fn verify_tx<D: Database, B: GetTx>(
     tx: &Transaction,
     database: &D,
     blockchain: &B,
@@ -104,43 +104,18 @@ impl_error!(bitcoinconsensus::Error, Consensus, VerifyError);
 
 #[cfg(test)]
 mod test {
-    use std::collections::HashSet;
+    use super::*;
-
+    use crate::database::{BatchOperations, MemoryDatabase};
     use bitcoin::consensus::encode::deserialize;
     use bitcoin::hashes::hex::FromHex;
     use bitcoin::{Transaction, Txid};
 
-    use crate::blockchain::{Blockchain, Capability, Progress};
-    use crate::database::{BatchDatabase, BatchOperations, MemoryDatabase};
-    use crate::FeeRate;
-
-    use super::*;
-
     struct DummyBlockchain;
 
-    impl Blockchain for DummyBlockchain {
+    impl GetTx for DummyBlockchain {
-        fn get_capabilities(&self) -> HashSet<Capability> {
-            Default::default()
-        }
-        fn setup<D: BatchDatabase, P: 'static + Progress>(
-            &self,
-            _database: &mut D,
-            _progress_update: P,
-        ) -> Result<(), Error> {
-            Ok(())
-        }
         fn get_tx(&self, _txid: &Txid) -> Result<Option<Transaction>, Error> {
             Ok(None)
         }
-        fn broadcast(&self, _tx: &Transaction) -> Result<(), Error> {
-            Ok(())
-        }
-        fn get_height(&self) -> Result<u32, Error> {
-            Ok(42)
-        }
-        fn estimate_fee(&self, _target: usize) -> Result<FeeRate, Error> {
-            Ok(FeeRate::default_min_relay_fee())
-        }
     }
 
     #[test]