* first pass compilation of croaring update
* cargo.lock
* add roaring arch flag into CI build scripts
* revert CI to use windows 2019 image
* add more debug
* more debug info
* update range arguments to bitmap remove_range function calls
* remove unnecessary cast
* [PIBD_IMPL] Introduce PIBD state into sync workflow (#3685)
* experimental addition of pibd download state for testnet only
* fixes to bitmap number of segments calculation + conversion of bitmap accumulator to bitmap
* attempt to call a test message
* add p2p methods for receiving bitmap segment and applying to desegmenter associated with chain
* fixes to state sync
* add pibd receive messages to network, and basic calls to desegmenter from each (#3686)
* [PIBD_IMPL] PIBD Desegmenter State (#3688)
* add functions to desegmenter to report next desired segments, begin to add state to determine which segments have been requested
* add segmentidentifier type to id requested segments uniquely
* make a call on where to keep track of which PIBD segments have been requested
* move segmenttype definition, add functions to manipulate peer segment list
* remove desegmenter state enum
* change chain desegmenter function to provide rwlock
* trace, warning cleanup
* udpate to test compliation
* [PIBD_IMPL] Bitmap accumulator reconstruction + TxHashset set reconstruction (#3689)
* application of received bitmap segments to local accumulator
* add all required elements to send/receive output segment requests and responses
* testing of output sync
* add special cases to pmmr segment request
* [PIBD_IMPL] PMMR Reassembly from Segments (#3690)
* update pibd copy test to use new desgmenter structure
* begin reconstruction of output pmmr
* clean up hash/leaf insertion logic
* push pruned subtree appears to be working, now also calculates left hand hashes correctly
* factor out ordering of segment/hash order array
* refactor for pmmr application code
* test of chain copy appears to be working
* add rangeproof functions to desegmenter
* add kernel functions, attempt refactor
* small test cleanup, reconstruction of live chain working in manual copy test
* [PIBD_IMPL] PIBD tree sync via network and kill/resume functionality (#3691)
* add functions to determing latest verifiable block height for the given pibd state
* attempting to allow for pibd to resume after killing process
* fix to ensure prune list is properly flushed during pibd sync
* removal of unneeded code
* ignore test for now (fix before full merge)
* [PIBD_IMPL] Finalize PIBD download and move state to chain validation (#3692)
* investigations as to why a slight rewind is needed on startup during PIBD
* move validation code into desegmenter validation thread (for now)
* ensure genesis entries in pmmrs are removed if they're removed in the first segment
* validation all working except for verifying kernel sums
* remove unneeded pmmr rollbacks on resume now root cause was found
* updates to remove unpruned leaves from leaf set when rebuilding pmmr
* remove + 1 to segment traversal iter length
* [PIBD_IMPL] PIBD Stats + Retry on validation errors (#3694)
* start to add stats and reset chain state after errors detected
* add functions to reset prune list when resetting chain pibd state
* debug statement
* remove test function
* [PIBD_IMPL] Update number of simultaneous peer requests for segments (#3696)
* cleanup of segment request list
* allow for more simultaneous requests during state sync
* up number of simultaneous peer requests for segments
* [PIBD_IMPL] Thread simplification + More TUI Updates + Stop State Propagation (#3698)
* change pibd stat display to show progress as a percentage of downloaded leaves
* attempt some inline rp validation
* propagate shutdown state through kernel validation
* change validation loop timing
* simplify validator threading
* add more detailed tracking of kernel history validation to tui, allow stop state during
* adding more stop state + tui progress indication
* remove progressive validate
* test fix
* revert to previous method of applying segments (#3699)
* fix for deadlock issue (#3700)
* update Cargo.lock for next release
* [PIBD_IMPL] Catch-Up functionality + Fixes based on testing (#3702)
* ensure desegmenter attempts to apply correct block after a resumte
* ensure txhashset's committed implementation takes into account output bitmap for summing purposes
* remove check to de-apply outputs during segment application
* return removal of spent outputs during pibd
* remove unneeded status
* remove uneeded change to rewind function
* documentation updates + todo fixes (#3703)
* add pibd abort timeout case (#3704)
* [PIBD_IMPL] BitmapAccumulator Serialization Fix (#3705)
* fix for writing / calculating incorrect length for negative indices
* update capabilities with new version of PIBD hist
* remove incorrect comment
* fix capabilities flag, trace output
* test fix
* Merge DNSSeed scope changes into pibd impl branch (#3708)
* update Cargo.lock for next release
* visibility scope tweaks to aid seed test utilities (#3707)
* move all PIBD-related constants into pibd_params modules (#3711)
* remove potential double read lock during compaction
* WIP remove failure from all `Cargo.toml`
* WIP remove `extern crate failure_derive`
* Use `thiserror` to fix all errors
* StoreErr is still a tuple
* Remove another set of unnecessary `.into()`s
* update fuzz tests
* update pool/fuzz dependencies in cargo.lock
* small changes based on feedback
Co-authored-by: trevyn <trevyn-git@protonmail.com>
* replace bitvec with more efficient bitpack algorithm
* optimise proof_unpack_len
* move proof pack length calculation
* small refactor
* first pass attempt at not deserializing proof nonces in difficulty iter
* another 10 seconds gained by not deserialising the proof from the difficulty iterator
* add new deser parameters to tests where needed
* add skip_proof variants to store
* remove hash from difficulty iterator struct, rename HeaderInfo to HeaderDifficultyInfo
* replace bitvec with more efficient bitpack algorithm
* optimise proof_unpack_len
* move proof pack length calculation
* small refactor
* first pass attempt at not deserializing proof nonces in difficulty iter
* another 10 seconds gained by not deserialising the proof from the difficulty iterator
* add new deser parameters to tests where needed
* add skip_proof variants to store
* remove hash from difficulty iterator struct, rename HeaderInfo to HeaderDifficultyInfo
* use 0-based positions in methods pmmr_leaf_to_insertion_index and bintree_postorder_height; add round_up_to_leaf_pos method
* use 0-based positions in method insertion_to_pmmr_index
* use 0-based positions in method is_leaf
* use 0-based positions in method family()
* use 0-based positions in method is_left_sibling
* use 0-based positions in method family_branch
* use 0-based positions in methods bintree_{left,right}most
* use 0-based positions in method bintree_pos_iter
* use 0-based positions in method bintree_range
* use 0-based positions in method bintree_leaf_pos_iter
* rename last_pos in MMR related structs to size
* use 0-based positions in method prune
* use 0-based positions in method push and apply_output return value
* use 0-based position argument of method merkle_proof
* use 0-based outputs in method pmmr::peaks
* fix peaks() code comments
* refix peaks() code comments
* use 0-based positions in method get_peak_from_file
* use 0-based positions in methods get_data_from_file
* use 0-based positions in methods get_from_file
* use 0-based positions in methods get_data
* use 0-based positions in methods get_hash
* use 0-based positions in method peak_path
* use 0-based positions in method bag_the_rhs
* use 0-based positions in method Backend::remove
* use 0-based positions in method leaf_pos_iter
* use 0-based positions in method self.LeafSet::includes
* use 0-based positions in methods self.LeafSet::{add,remove}
* use 0-based positions in methods is_pruned,is_pruned_root,is_compacted
* use 0-based positions in methods PruneList::append
* use 0-based positions in methods append_pruned_subtree
* use 0-based positions in method calculate_next_leaf_shift
* use 0-based positions in method append_single
* use 0-based positions in method calculate_next_shift
* use 0-based positions in method segment_pos_range
* use 0-based positions in method reconstruct_root
* use 0-based positions in method validate_with
* use 0-based positions in method validate
* rename size (formerly last_pos) to mmr_size
* use 0-based positions in Segment's hash_pos and leaf_pos
* minimize use of saturating_sub(1) and rename some pos/idx to size
* use 0-based positions in methods get_output_pos
* use 0-based positions in method get_unspent_output_at
* use 0-based positions in method get_header_hash
* use 0-based positions in methods MerkleProof::verify{,_consume}
* use 0-based positions in method cleanup_subtree
* don't allow 0 in prunelist bitmap
* use 0-based positions in methods get_{,leaf_}shift
* rename some 1-based pos to pos1; identify TODO
* Address yeastplume's PR review comments
* refactor prune_list with aim of allowing pruned subtree appending
* add test coverage around pmmr::is_leaf() and pmmr::bintree_leaf_pos_iter()
* comments
* cleanup
* implement append pruned subtree for prune_list
* commit
* we can now append to prune_list
* fix our prune_list corruption...
* rework how we rewrite the prune list during compaction
* test coverage for improved prune list api
* continuing to merge
* finish merge, tests passing again
* add function pmmr_leaf_to_insertion_index, and modify bintree_lef_pos_iter to use it. Note there's still an unwrap that needs to be dealt with sanely
* change pmmr_leaf_to_insertion_index to simpler version + handle conversion between 1 and 0 based in bintree_leaf_pos_iter
Co-authored-by: antiochp <30642645+antiochp@users.noreply.github.com>
* wip
* use range beneath subtree for efficient is_pruned check
* various iterators over the prune list
* improved prune list iter and subtree handling
* use take_while so unpruned iterators are not infinite
* Introduce CommitOnly variant of Inputs.
Introduce CommitWrapper so we can sort commit only inputs correctly.
* rememebr to resort if converting
* write inputs based on variant and protocol version
* read and write protocol version specific inputs
* store full blocks in local db in v3
convert to v2 when relaying to v2 peers
* add debug version_str for inputs
* no assumptions about spent index sort order
* add additional version debug logs
* fix ser/deser tests for proto v3
* cleanup coinbase maturity
* rework pool to better handle v2 conversion robustly
* cleanup txpool add_to_pool
* fix nrd kernel test
* move init conversion earlier
* cleanup
* cleanup based on PR feedback
* variants for output_pos linked list entries (head/tail/middle/unique)
next and prev and Vec<u8> lmdb keys
* get_pos on enum
* break list and list entries out into separate enums
* track output features in the new output_pos index, so we can determine coinbase maturity
* push entry impl for none and unique
* some test coverage for output_pos_list
* commit
* wip - FooListEntry
* use instance of the index
* linked list of output_pos and commit_pos both now supported
* linked_list
* cleanup and rename
* rename
* peek_pos
* push some, peek some, pop some
* cleanup
* commit pos
cleanup
* split list and entry out into separate db prefixes
* cleanup and add placeholder for pop_back
* pop_pos_back (for popping off the back of the linked list)
test coverage for pop_pos_back
* wip
* placeholder for prune via a trait
pos must always increase in the index
* rewind kernel_pos_idx when calling rewind_single_block
* RewindableListIndex with rewind support.
* test coverage for rewindable list index
* test coverage for rewind back to 0
* rewind past end of list
* add tests for kernel_pos_idx with multiple commits
* commit
* cleanup
* hook NRD relative lock height validation into block processing and tx validation
* cleanup
* set local chain type for kernel_idx tests
* add test coverage for NRD rules in block processing
* NRD test coverage and cleanup
* NRD relative height 1 test
* test coverage for NRD kernels in block processing
* cleanup
* start of test coverage for txpool NRD kernel rules
* wip
* rework pool tests to use real chain (was mock chain) to better reflect reality (tx/block validation rules etc.)
* cleanup
* cleanup pruneable trait for kernel pos index
* add clear() to kernel_pos idx and test coverage
* hook kernel_pos rebuild into node startup, compaction and fast sync
* verify full NRD history on fast sync
* return early if nrd disabled
* fix header sync issue
We have a pattern in the code - return Vec, turn it into an iterator, filter and collect to another Vec. The idea was to return iterator where possible to avoid allocating intermediate vecs.
We have a method get which returns a Vec, it's used in cases when we can't implement FromLmdbBytes or Readable on a desired type, eg when both traits and type are external for the current crate. This approach requires an extra allocation of a Vec, this PR introduces a method which accepts desirialzer as closure, which allows to desrialize in place without an intermidiate heap allocation.
It's still possible to get a Vec if needed, just pass a Vec constructor to get_with.
We have to make an extra allocation per db get request because key generation function to_key takes Vec. Taking byte slice (AsRef<[u8]> to be precise) also simplifes the code a bit.
Currently Writable accepts trait Write as a type parameter but Readable
takes Read as a trait object, which is not symmetrical and also less performant. This PR changes Readable trait and all places where it's used
* no need for AsFixedBytes we can just use AsRef<[u8]>
* cleanup FixedLength trait
* revert this change for now
* fix store tests
* cleanup and fix tests after rebase
* fix tests
* update based on PR review
less hard-coded values now
* cleanup
* add bitmap accumulator
refactor vec backend so we can use it outside of tests
introduce a "hash only" vec backend for the accumulator
* get core tests passing
* initial test coverage for bitmap_accumulator
* better test coverage for bitmap accumulator and cleanup code
* refactor txhashset roots, call validate() on roots during block validation
* fix store tests
* log the "merged" root when validating roots
* cleanup, revise based on feedback
* cleanup
* rework it to pass explicit size into bitmap accumulator when applying
* fix: split state validation status into kernel and rproof updates. And fix sync status for these two states
* fix: show correct number of leaves for pruned MMR as well as unpruned
* docs: better docs for kernel/range proof validation
* fix: ordering of kernel and rproofs validation in TUI
* fix: typo in rangeproofs api and comments
To convert option to error we generate an error message. In some places
it contains header or block hash code or other data which is costly to
produce. So during the initial header sync we spend 12% of all time on
generating those messages (in 99% cases we don't use it). This PR
introduces a lazy generation of error messages which completely
eliminates CPU load during the header sync.
* store both mmr index and block height into database for output
* rustfmt
* fix: mmr position is 1-based instead of 0-based
* (Hash, u64, u64) deserves a type
* maintain header_head as distinctly separate from head
* cleanup corrupted storage log msg
* simplify process_header and check_header_known
* remember to commit the batch when successfully processing a header...
* rework sync_block_headers for consistency with process_block_header
* cleanup unrelated code
* fix pool tests
* cleanup chain tests
* cleanup chain tests (reuse helpers more)
* cleanup - head not header on an extension
shortcircuit "rewind and apply fork" for headers if next header
* introduce protocol version to deserialize and read
* thread protocol version through our reader
* cleanup
* cleanup
* streaming_reader cleanup
* Pass protocol version into BinWriter to allow for version specific serialization rules.
* rustfmt
* read and write now protocol version specific
