Steve Myers b2ac4a0dfd
Merge bitcoindevkit/bdk#461: Restructure electrum/esplora sync logic
9c5770831de58760066dbbb26c0785632dde4991 Make stop_gap a parameter to EsploraBlockchainConfig::new (LLFourn)
0f0a01a742448fa9db67752722fed7d1c028117b s/vin/vout/ (LLFourn)
1a64fd9c9595566366cfe96cc20d8ba2f4da1bd3 Delete src/blockchain/utils.rs (LLFourn)
d3779fac73321f41b82897e6ab69708c957212b2 Fix comments (LLFourn)
d39401162fb72a9a02999908ac0347685d8861a5 Less intermediary data states in sync (LLFourn)
dfb63d389b2804cb55da60de0d8fcad985d50e5c s/observed_txs/finished_txs/g (LLFourn)
188d9a4a8b502b3f681f4c83fd9933b9a952e42c Make variable names consistent (LLFourn)
5eadf5ccf9722aa331008304221e981d01bca1c1 Add some logging to script_sync (LLFourn)
aaad560a91872318890208c4b3d5cb73a63029a8 Always get up to chunk_size heights to request headers for (LLFourn)
e7c13575c88be4fce1f407eae3710cb06c5db039 Don't request conftime during tx request (LLFourn)
808d7d8463c56cbcda0bd55a5eba422cf5080ce7 Update changelog (LLFourn)
732166fcb689ad666eab02a4bb28c33018f7b819 Fix feerate calculation for esplora (LLFourn)
3f5cb6997f8bf0c889747badb9498a402ecf96a4 Invert dependencies in electrum sync (LLFourn)

Pull request description:

  ## Description

  This PR does dependency inversion on the previous sync logic for electrum and esplora captured in the trait `ElectrumLikeSync`. This means that the sync logic does not reference the blockchain at all. Instead the blockchain asks the sync logic (in `script_sync.rs`) what it needs to continue the sync and tries to retrieve it.

  The initial purpose of doing this is to remove invocations of `maybe_await` in the abstract sync logic in preparation for completely removing `maybe_await` in the future. The other major benefit is it gives a lot more freedom for the esplora logic to use the rich data from the responses to complete the sync with less HTTP requests than it did previously.

  ## List of changes

  - sync logic moved to `script_sync.rs` and `ElectrumLikeSync` is gone.
  - esplora makes one http request per sync address. This means it makes half the number of http requests for a fully synced wallet and N*M less requests for a wallet which has N new transactions with M unique input transactions.
  - electrum and esplora save less raw transactions in the database. Electrum still requests input transactions for each of its transactions to calculate the fee but it does not save them to the database anymore.
  - The ureq and reqwest blockchain configuration is now unified into the same struct. This is the only API change. `read_timeout` and `write_timeout` have been removed in favor of a single `timeout` option which is set in both ureq and reqwest.
  - ureq now does concurrent (parallel) requests using threads.
  - An previously unnoticed bug has been fixed where by sending a lot of double spending transactions to the same address you could trick a bdk Esplora wallet into thinking it had a lot of unconfirmed coins. This is because esplora doesn't delete double spent transactions from its indexes immediately (not sure if this is a bug or a feature). A blockchain test is added for this.
  - BONUS: The second commit in this PR fixes the feerate calculation for esplora and adds a test (the previous algorithm didn't work at all). I could have made a separate PR but since I was touching this file a lot I decided to fix it here.

  ## Notes to the reviewers

  - The most important thing to review is the the logic in `script_sync.rs` is sound.
  - Look at the two commits separately.
  - I think CI is failing because of MSRV problems again!
  - It would be cool to measure how much sync time is improved for your existing wallets/projects. For `gun` the speed improvements for modest but it is at least hammering the esplora server much less.
  - I noticed the performance of reqwest in blocking is much worse in this patch than previously. This is because somehow reqwest is not re-using the connection for each request in this new code. I have no idea why. The plan is to get rid of the blocking reqwest implementation in a follow up PR.

  ### Checklists

  #### All Submissions:

  * [x] I've signed all my commits

ACKs for top commit:
  rajarshimaitra:
    Retested ACK a630685a0a

Tree-SHA512: de74981e9d1f80758a9f20a3314ed7381c6b7c635f7ede80b177651fe2f9e9468064fae26bf80d4254098accfacfe50326ae0968e915186e13313f05bf77990b
2021-11-24 19:51:09 -08:00
2021-11-02 15:21:37 +05:30
2021-09-30 11:24:01 -07:00
2020-09-14 15:13:43 -07:00
2020-11-16 12:09:14 +01:00
2021-06-29 11:35:02 +02:00

BDK

A modern, lightweight, descriptor-based wallet library written in Rust!

Crate Info MIT or Apache-2.0 Licensed CI Status API Docs Rustc Version 1.46+ Chat on Discord

Project Homepage | Documentation

About

The bdk library aims to be the core building block for Bitcoin wallets of any kind.

  • It uses Miniscript to support descriptors with generalized conditions. This exact same library can be used to build single-sig wallets, multisigs, timelocked contracts and more.
  • It supports multiple blockchain backends and databases, allowing developers to choose exactly what's right for their projects.
  • It's built to be cross-platform: the core logic works on desktop, mobile, and even WebAssembly.
  • It's very easy to extend: developers can implement customized logic for blockchain backends, databases, signers, coin selection, and more, without having to fork and modify this library.

Examples

Sync the balance of a descriptor

use bdk::Wallet;
use bdk::database::MemoryDatabase;
use bdk::blockchain::{noop_progress, ElectrumBlockchain};

use bdk::electrum_client::Client;

fn main() -> Result<(), bdk::Error> {
    let client = Client::new("ssl://electrum.blockstream.info:60002")?;
    let wallet = Wallet::new(
        "wpkh([c258d2e4/84h/1h/0h]tpubDDYkZojQFQjht8Tm4jsS3iuEmKjTiEGjG6KnuFNKKJb5A6ZUCUZKdvLdSDWofKi4ToRCwb9poe1XdqfUnP4jaJjCB2Zwv11ZLgSbnZSNecE/0/*)",
        Some("wpkh([c258d2e4/84h/1h/0h]tpubDDYkZojQFQjht8Tm4jsS3iuEmKjTiEGjG6KnuFNKKJb5A6ZUCUZKdvLdSDWofKi4ToRCwb9poe1XdqfUnP4jaJjCB2Zwv11ZLgSbnZSNecE/1/*)"),
        bitcoin::Network::Testnet,
        MemoryDatabase::default(),
        ElectrumBlockchain::from(client)
    )?;

    wallet.sync(noop_progress(), None)?;

    println!("Descriptor balance: {} SAT", wallet.get_balance()?);

    Ok(())
}

Generate a few addresses

use bdk::{Wallet, database::MemoryDatabase};
use bdk::wallet::AddressIndex::New;

fn main() -> Result<(), bdk::Error> {
    let wallet = Wallet::new_offline(
        "wpkh([c258d2e4/84h/1h/0h]tpubDDYkZojQFQjht8Tm4jsS3iuEmKjTiEGjG6KnuFNKKJb5A6ZUCUZKdvLdSDWofKi4ToRCwb9poe1XdqfUnP4jaJjCB2Zwv11ZLgSbnZSNecE/0/*)",
        Some("wpkh([c258d2e4/84h/1h/0h]tpubDDYkZojQFQjht8Tm4jsS3iuEmKjTiEGjG6KnuFNKKJb5A6ZUCUZKdvLdSDWofKi4ToRCwb9poe1XdqfUnP4jaJjCB2Zwv11ZLgSbnZSNecE/1/*)"),
        bitcoin::Network::Testnet,
        MemoryDatabase::default(),
    )?;

    println!("Address #0: {}", wallet.get_address(New)?);
    println!("Address #1: {}", wallet.get_address(New)?);
    println!("Address #2: {}", wallet.get_address(New)?);

    Ok(())
}

Create a transaction

use bdk::{FeeRate, Wallet};
use bdk::database::MemoryDatabase;
use bdk::blockchain::{noop_progress, ElectrumBlockchain};

use bdk::electrum_client::Client;
use bdk::wallet::AddressIndex::New;

use bitcoin::consensus::serialize;

fn main() -> Result<(), bdk::Error> {
    let client = Client::new("ssl://electrum.blockstream.info:60002")?;
    let wallet = Wallet::new(
        "wpkh([c258d2e4/84h/1h/0h]tpubDDYkZojQFQjht8Tm4jsS3iuEmKjTiEGjG6KnuFNKKJb5A6ZUCUZKdvLdSDWofKi4ToRCwb9poe1XdqfUnP4jaJjCB2Zwv11ZLgSbnZSNecE/0/*)",
        Some("wpkh([c258d2e4/84h/1h/0h]tpubDDYkZojQFQjht8Tm4jsS3iuEmKjTiEGjG6KnuFNKKJb5A6ZUCUZKdvLdSDWofKi4ToRCwb9poe1XdqfUnP4jaJjCB2Zwv11ZLgSbnZSNecE/1/*)"),
        bitcoin::Network::Testnet,
        MemoryDatabase::default(),
        ElectrumBlockchain::from(client)
    )?;

    wallet.sync(noop_progress(), None)?;

    let send_to = wallet.get_address(New)?;
    let (psbt, details) = {
        let mut builder = wallet.build_tx();
        builder
            .add_recipient(send_to.script_pubkey(), 50_000)
            .enable_rbf()
            .do_not_spend_change()
            .fee_rate(FeeRate::from_sat_per_vb(5.0));
        builder.finish()?
    };

    println!("Transaction details: {:#?}", details);
    println!("Unsigned PSBT: {}", base64::encode(&serialize(&psbt)));

    Ok(())
}

Sign a transaction

use bdk::{Wallet, SignOptions, database::MemoryDatabase};

use bitcoin::consensus::deserialize;

fn main() -> Result<(), bdk::Error> {
    let wallet = Wallet::new_offline(
        "wpkh([c258d2e4/84h/1h/0h]tprv8griRPhA7342zfRyB6CqeKF8CJDXYu5pgnj1cjL1u2ngKcJha5jjTRimG82ABzJQ4MQe71CV54xfn25BbhCNfEGGJZnxvCDQCd6JkbvxW6h/0/*)",
        Some("wpkh([c258d2e4/84h/1h/0h]tprv8griRPhA7342zfRyB6CqeKF8CJDXYu5pgnj1cjL1u2ngKcJha5jjTRimG82ABzJQ4MQe71CV54xfn25BbhCNfEGGJZnxvCDQCd6JkbvxW6h/1/*)"),
        bitcoin::Network::Testnet,
        MemoryDatabase::default(),
    )?;

    let psbt = "...";
    let mut psbt = deserialize(&base64::decode(psbt).unwrap())?;

    let finalized = wallet.sign(&mut psbt, SignOptions::default())?;

    Ok(())
}

Testing

Unit testing

cargo test

Integration testing

Integration testing require testing features, for example:

cargo test --features test-electrum

The other options are test-esplora or test-rpc. Note that electrs and bitcoind binaries are automatically downloaded (on mac and linux), to specify you already have installed binaries you must use --no-default-features and provide BITCOIND_EXE and ELECTRS_EXE as environment variables.

License

Licensed under either of

at your option.

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

Languages
Rust 99.9%