To convert option to error we generate an error message. In some places
it contains header or block hash code or other data which is costly to
produce. So during the initial header sync we spend 12% of all time on
generating those messages (in 99% cases we don't use it). This PR
introduces a lazy generation of error messages which completely
eliminates CPU load during the header sync.
* wip
* sync sort of works now
* get rid of the deadlock during compaction
there is *always* a deadlock in there when we make changes like this...
* cleanup how we rebuild the sync MMR on init
* cleanup rewind logic
* roll the "fix invalid root" changes into this PR
* move rebuild_height_pos_index into txhashset
and pass in header_pmmr for header lookups
* cleanup and remember to setup sync head on init
* cleanup unnecessary ref muts
* rebuild_height_pos_index when writing txhashset
* use head (not header_head) when rewinding to find fork point for full blocks
* get rid of shortcut when nothing to rewind - always something to rewind
* schnorr signature batch verification
* Split the fee and lock_height verification out from the batch signature verification
* Remove the new added fee_height_verify to comply with #2859
* fix: the last n could not be leaf?
* Kernel lookup api
* Min and max height parameters, scan backwards
* Return null instead of 404 if kernel not found
* Return TxKernel instead of TxKernelPrintable
* Update description of KernelHandler
* store both mmr index and block height into database for output
* rustfmt
* fix: mmr position is 1-based instead of 0-based
* (Hash, u64, u64) deserves a type
* Ban on cannot get block header in txhashset_write
* Rusfmt
* Fix typo
* Missing error handling
* Rustfmt
* Only accept txhashset from corresponding peer
* Switch to AtomicBool instead of RwLock<bool>
* Rustfmt
* maintain header_head as distinctly separate from head
* cleanup corrupted storage log msg
* simplify process_header and check_header_known
* remember to commit the batch when successfully processing a header...
* rework sync_block_headers for consistency with process_block_header
* cleanup unrelated code
* fix pool tests
* cleanup chain tests
* cleanup chain tests (reuse helpers more)
* cleanup - head not header on an extension
shortcircuit "rewind and apply fork" for headers if next header
* simplify how we extract files from the txhashset zip archive
* bump version of zip-rs
cleanup create_zip (was compress)
use explicit list of files when creating zip archive
* rework zip tests for our simplified impl
* periodic debug during txhashset download
* rework list_files to return something usable
* introduce protocol version to deserialize and read
* thread protocol version through our reader
* cleanup
* cleanup
* streaming_reader cleanup
* Pass protocol version into BinWriter to allow for version specific serialization rules.
* rustfmt
* read and write now protocol version specific
* introduce protocol version to deserialize and read
* thread protocol version through our reader
* example protocol version access in kernel read
* fix our StreamingReader impl (WouldBlock woes)
* debug log progress of txhashset download
* Calculate reorg depth in BlockStatus::Reorg enum member
* rustfmt
* Fix reorg height calculation and implement reorg test
* rustfmt
* Report reorg depth in webhook payload
* Add optional depth field to the block webhook JSON reply
* fix: try to fix issue #2585 by adding block cleanup from db directly.
Signed-off-by: Mike Tang <daogangtang@gmail.com>
* use another effective algorithm to do old block and short-lived block cleaup.
Signed-off-by: Mike Tang <daogangtang@gmail.com>
* 1. rename iter_lived_blocks to blocks_iter;
2. comments and iterator calling optimiztions.
Signed-off-by: Mike Tang <daogangtang@gmail.com>
* Fix locking bug when calling is_on_current_chain() in batch.blocks_iter just by removing it.
Because "we want to delete block older (i.e. lower height) than tail.height".
Signed-off-by: Mike Tang <daogangtang@gmail.com>
* introduce HeaderVersion (was u16) for type safety
cleanup pow ser/deser (version unused)
* fixup tests for HeaderVersion
* validate header version during header deserialization
* use a grin specific tmp dir instead of a system tmp dir
* rustfmt
* using create_new instead of create
* switch the txhashset sandbox from system tmp dir to grin specific tmp dir
* rustfmt
* adapter get_tmp_dir delegate to chain implementation
* simply the logic of protolc for get_tmpfile_pathname
* rustfmt
* fix the merge mistake
* Introduce optional size_file.
Support fixed size data file via an optional elmt_size.
Support variable size data file via optional size_file.
* remember to release the size_file
* fix scoping for windows support
* cleanup legacy "3 dot" check (#2625)
* Allow to peers behind NAT to get up to preferred_max connections (#2543)
Allow to peers behind NAT to get up to preffered_max connections
If peer has only outbound connections it's mot likely behind NAT and we should not stop it from getting more outbound connections
* Reduce usage of unwrap in p2p crate (#2627)
Also change store crate a bit
* Simplify (and fix) output_pos cleanup during chain compaction (#2609)
* expose leaf pos iterator
use it for various things in txhashset when iterating over outputs
* fix
* cleanup
* rebuild output_pos index (and clear it out first) when compacting the chain
* fixup tests
* refactor to match on (output, proof) tuple
* add comments to compact() to explain what is going on.
* get rid of some boxing around the leaf_set iterator
* cleanup
* [docs] Add switch commitment documentation (#2526)
* remove references to no-longer existing switch commitment hash
(as switch commitments were removed in ca8447f3bd
and moved into the blinding factor of the Pedersen Commitment)
* some rewording (points vs curves) and fix of small formatting issues
* Add switch commitment documentation
* [docs] Documents in grin repo had translated in Korean. (#2604)
* Start to M/W intro translate in Korean
* translate in Korean
* add korean translation on intro
* table_of_content.md translate in Korean.
* table_of_content_KR.md finish translate in Korean, start to translate State_KR.md
* add state_KR.md & commit some translation in State_KR.md
* WIP stat_KR.md translation
* add build_KR.md && stratum_KR.md
* finish translate stratum_KR.md & table_of_content_KR.md
* rename intro.KR.md to intro_KR.md
* add intro_KR.md file path each language's intro.md
* add Korean translation file path to stratum.md & table_of_contents.md
* fix difference with grin/master
* Fix TxHashSet file filter for Windows. (#2641)
* Fix TxHashSet file filter for Windows.
* rustfmt
* Updating regexp
* Adding in test case
* Display the current download rate rather than the average when syncing the chain (#2633)
* When syncing the chain, calculate the displayed download speed using the current rate from the most recent iteration, rather than the average download speed from the entire syncing process.
* Replace the explicitly ignored variables in the pattern with an implicit ignore
* remove root = true from editorconfig (#2655)
* Add Medium post to intro (#2654)
Spoke to @yeastplume who agreed it makes sense to add the "Grin Transactions Explained, Step-by-Step" Medium post to intro.md
Open for suggestions on a better location.
* add a new configure item for log_max_files (#2601)
* add a new configure item for log_max_files
* rustfmt
* use a constant instead of multiple 32
* rustfmt
* Fix the build warning of deprecated trim_right_matches (#2662)
* [DOC] state.md, build.md and chain directory documents translate in Korean. (#2649)
* add md files for translation.
* start to translation fast-sync, code_structure. add file build_KR.md, states_KR.md
* add dandelion_KR.md && simulation_KR.md for Korean translation.
* add md files for translation.
* start to translation fast-sync, code_structure. add file build_KR.md, states_KR.md
* add dandelion_KR.md && simulation_KR.md for Korean translation.
* remove some useless md files for translation. this is rearrange set up translation order.
* add dot end of sentence & translate build.md in korean
* remove fast-sync_KR.md
* finish build_KR.md translation
* finish build_KR.md translation
* finish translation state_KR.md & add phrase in state.md to move other language md file
* translate blocks_and_headers.md && chain_sync.md in Korean
* add . in chain_sync.md , translation finished in doc/chain dir.
* fix some miss typos
* Api documentation fixes (#2646)
* Fix the API documentation for Chain Validate (v1/chain/validate). It was documented as a POST, but it is actually a GET request, which can be seen in its handler ChainValidationHandler
* Update the API V1 route list response to include the headers and merkleproof routes. Also clarify that for the chain/outputs route you must specify either byids or byheight to select outputs.
* refactor(ci): reorganize CI related code (#2658)
Break-down the CI related code into smaller more maintainable pieces.
* Specify grin or nanogrins in API docs where applicable (#2642)
* Set Content-Type in API client (#2680)
* Reduce number of unwraps in chain crate (#2679)
* fix: the restart of state sync doesn't work sometimes (#2687)
* let check_txhashset_needed return true on abnormal case (#2684)
* Reduce number of unwwaps in api crate (#2681)
* Reduce number of unwwaps in api crate
* Format use section
* Small QoL improvements for wallet developers (#2651)
* Small changes for wallet devs
* Move create_nonce into Keychain trait
* Replace match by map_err
* Add flag to Slate to skip fee check
* Fix secp dependency
* Remove check_fee flag in Slate
* Add Japanese edition of build.md (#2697)
* catch the panic to avoid peer thread quit early (#2686)
* catch the panic to avoid peer thread quit before taking the chance to ban
* move catch wrapper logic down into the util crate
* log the panic info
* keep txhashset.rs untouched
* remove a warning
* [DOC] dandelion.md, simulation.md ,fast-sync.md and pruning.md documents translate in Korean. (#2678)
* Show response code in API client error message (#2683)
It's hard to investigate what happens when an API client error is
printed out
* Add some better logging for get_outputs_by_id failure states (#2705)
* Switch commitment doc fixes (#2645)
Fix some typos and remove the use of parentheses in a
couple of places to make the reading flow a bit better.
* docs: update/add new README.md badges (#2708)
Replace existing badges with SVG counterparts and add a bunch of new ones.
* Update intro.md (#2702)
Add mention of censoring attack prevented by range proofs
* use sandbox folder for txhashset validation on state sync (#2685)
* use sandbox folder for txhashset validation on state sync
* rustfmt
* use temp directory as the sandbox instead actual db_root txhashset dir
* rustfmt
* move txhashset overwrite to the end of full validation
* fix travis-ci test
* rustfmt
* fix: hashset have 2 folders including txhashset and header
* rustfmt
*
(1)switch to rebuild_header_mmr instead of copy the sandbox header mmr
(2)lock txhashset when overwriting and opening and rebuild
* minor improve on sandbox_dir
* add Japanese edition of state.md (#2703)
* Attempt to fix broken TUI locale (#2713)
Can confirm that on the same machine 1.0.2 TUI looks great and is broken on
the current master. Bump of `cursive` version fixed it for me.
Fixes#2676
* clean the header folder in sandbox (#2716)
* forgot to clean the header folder in sandbox in #2685
* Reduce number of unwraps in servers crate (#2707)
It doesn't include stratum server which is sufficiently changed in 1.1
branch and adapters, which is big enough for a separate PR.
* rustfmt
* change version to beta
* use sandbox folder for txhashset validation on state sync
* rustfmt
* use temp directory as the sandbox instead actual db_root txhashset dir
* rustfmt
* move txhashset overwrite to the end of full validation
* fix travis-ci test
* rustfmt
* fix: hashset have 2 folders including txhashset and header
* rustfmt
*
(1)switch to rebuild_header_mmr instead of copy the sandbox header mmr
(2)lock txhashset when overwriting and opening and rebuild
* minor improve on sandbox_dir
* expose leaf pos iterator
use it for various things in txhashset when iterating over outputs
* fix
* cleanup
* rebuild output_pos index (and clear it out first) when compacting the chain
* fixup tests
* refactor to match on (output, proof) tuple
* add comments to compact() to explain what is going on.
* get rid of some boxing around the leaf_set iterator
* cleanup
* dynamically resize lmdb
* rustfmt
* explicitly close db before resizing
* rustfmt
* test fix
* rustfmt
* pool tests
* chain fix
* merge
* move RwLock into Store, ensure resize gives a min threshold
* rustfmt
* move locks based on testing
* rustfmt
* clean-up the txhashset zip file and temp folder
* adapt test_txhashset with the new zip_read function
* fix: the wrong folder when cleaning up old zips
* adding optional mmr index to key ids
* rustfmt
* update index
* add mmr index to output display
* change restore to match on commit instead of ID, add extensive restore/check tests for multiple wallets using same seed
* rustfmt
* ensure check restores unknown accounts as well
* rustfmt
* remove storage of commit from wallet
* Fix secondary scaling bugs; rename is_testnet -> is_floonet (#2215)
* add global::is_mainnet()
* use it to change pre-genesis pow type
* rename is_testnet -> is_floonet
* Support multiple chain configurations (#2217)
* Support multiple chain configurations
Supports generating the proper configuration for each chain type
(mainnet, floonet, usernet). Will run them by default under
their respective root directory (~/.grin/main, ~/.grin/floo, etc).
Assigned default ports for mainnet, overriding them to keep Floonet
ports unchanged.
For now, starting on mainnet will abort.
* Fixed usernet command line help message. Fixes#2217
* Differing magic numbers for each chain type (#2208)
* stick to e=H(R|P|m) when use schnorr signature (#2200)
* stick to e=H(R|P|m) when use schnorr signature
* (1)add verify_slate_messages for wallet receive (2)log the message content
* remove debug log on verify_slate_messages
* verify the sender's message signature when receive_tx in wallet listen
* Revert "remove debug log on verify_slate_messages"
This reverts commit 65ea32a407.
* Revert "rustfmt"
This reverts commit c380ab9185.
* Revert "(1)add verify_slate_messages for wallet receive (2)log the message content"
This reverts commit 9584ca7a89.
* [re-floonet] Keychain Floonet BIP32 version/network option (#2235)
* add 'is_floonet' property to keychain
* fix hex encoding and tests
* Fix couple floonet loose ends (#2230)
* Fix couple floonet loose ends. Fixes#2216
* Doc fix for sig message
* Refuse unkown kernel features (#2244)
* Minor: magic number change for re-floonet
* Set pre genesis is_secondary to true (#2247)
* Minor: tx validation error display underlying
* New floonet genesis
* genesis rustfmt
* Use chain-specific config for wallet toml gen
* Fix default wallet_listener_url
* New more reasonable genesis block, bumped version
* genesis rustfmt
* Couple minor fixes to genesis generation script
* add feature for height locked kernels
* add function to compute kernel features appropriate for lock height, and use it
* only sign kernel-features relevant fields; refactor Features
* simplify invalid kernel logic
* remove unused height arg to reward::output and run some rustfmt
* replace nested if/else by match
* Chain init now handles genesis body properly, related unit test creating the genesis with reward
* Avoid making block body public by adding a with_reward method
* apply_block in all genesis cases works
* start wallet command refactoring
* another re-structuring attempt
* rustfmt
* begin splitting up wallet commands
* rustfmt
* clean up wallet arg checking
* rustfmt
* macro for arg parsing
* rustfmt
* factor out init commands
* rustfmt
* move recover to new format
* rustfmt
* add listen command to new format
* rustfmt
* Finish moving commands to new format
* rustfmt
* rustfmt
* propogate errors more cleanly
* rustfmt
* error handling cleanup
* replace header_by_height index with reads into the header MMR
* rustfmt
* cleanup
* cleanup chain tests
* fix locate_headers to stop on our max header
* fix the deadlock in comact_blocks_db...
* cleanup and docs/comments
* Cuckatoo size shift upgrade schedule
* Move the schedule into graph_weight instead of messing with min edge bits
* Cleanup and fixes now that we have an agreed upon schedule
* refactor header MMR to use regular backend
this lets us store header hashes in the mmr data file
the hashes in the hash file are "hash with index"
* rustfmt
* get_header_hash(pos) on header extension
* rustfmt
* cleanup
* cleanup
* comments
* cleanup
* fixup testelem so tests work
* PoW context is now properly picked depending on the chain type,
edge bits and block height. Height const for T4 hard fork leaving
a couple weeks to have miners in place. Removed now unused Cuckoo context.
* Simplified block siphash
* Fix servers crate compilation
* Tiny bit cleaner block siphash. Maybe.
* Cuckatoo min edge bits update for T4 and mainnet
* Fix header size tests, Cuckatoo31 default means one more bit per edge
* Remove redundant param from verify_size
* block_accepted via adapter is now reorg aware
we skip reconciling the txpool is block is not most work
we skip reconciling the reorg_cache is not a reorg
* rustfmt
* logging tweaks
* rework block_accepted interface
* rustfmt
* rework reorg determination
* introduce BlockStatus to represent next vs reorg vs fork
* rustfmt
* cleanup logging
* log from adapter, even during sync
* split horizon into two explicit values for cut through and txhashset request
* let node which has 2-7 days of history be able to handle forks larger than 2 days
* add test simulate_long_fork
* add pause/resume feature on p2p for tests
* refactor the state_sync
* ignore the test case simulate_long_fork for normal Travis-CI
* refactor function check_txhashset_needed to be shared with body_sync
* fix: state TxHashsetDone should allow header sync
* cleanup pmmrable and len()
introduce FixedLength trait with a const LEN
make Hash impl FixedLength for consistency
* rustfmt
* store tests cleanup
* rustfmt
* whats going on with those comments and rustfmt?
* be careful resetting sync_head
make sure we have the header corresponding to header_head
and that nothing is corrupted from an earlier shutdown
* add logging when resetting heads due to missing header
* Update secp dependency to use crates.io version
* Add more details to various Cargo.toml
* Remove use of env variables that aren't reliably provided by cargo
* Replace logging backend to flexi-logger and add log rotation
* Changed flexi_logger to log4rs
* Disable logging level filtering in Root logger
* Support different logging levels for file and stdout
* Don't log messages from modules other than Grin-related
* Fix formatting
* Place backed up compressed log copies into log file directory
* Increase default log file size to 16 MiB
* Add comment to config file on log_max_size option
* header MMR in use within txhashset itself
works with fast sync
not yet in place for initial header sync
* add the (currently unused) sync_head mmr
* use sync MMR during fast sync
rebuild header MMR after we validate full txhashset after download
* support missing header MMR (rebuild as necessary) for legacy nodes
* rename to HashOnly
* cleanup backend.append()
* simplify vec_backend to match simpler append api
* header MMR in use within txhashset itself
works with fast sync
not yet in place for initial header sync
* add the (currently unused) sync_head mmr
* use sync MMR during fast sync
rebuild header MMR after we validate full txhashset after download
* support missing header MMR (rebuild as necessary) for legacy nodes
* rustfmt
* comments/docs
* rustfmt
* cleanup DBBackend
* cleanup DBBackend
* cleanup
* rename to HashOnly
* rustfmt
* cleanup backend.append()
* simply pmmr append api
no need to pass position when appending
* cleanup
* simplify vec_backend to match simpler append api
* rustfmt
* docs/comments
* rustfmt
* cleanup
* First pass at secondary proof of work difficulty adjustments
* Core and chain test fixes
* Next difficulty calc now needs a height. Scaling calculation fixes. Setting scaling on mined block.
* Change factor to u32 instead of u64.
* Cleanup structs used by next_difficulty
* Fix header size calc with u32 scaling
* change keychain to use bip32 paths
* convert keychain to use bip32
* change identifier to be serialisation of 4-level bip32 path
* wallet changes compiling, pass parent key into all wallet functions
* rustfmt
* fix tests in chain
* rustfmt
* core tests passing
* rustfmt
* pool tests
* rustfmt
* fixing wallet tests
* rustfmt
* remove file wallet
* wallet tests compiling
* rustfmt
* remove db_migrate
* successful tx exchange test using BIP32 paths
* rustfmt
* fix wallet derivation paths to m/0/0/0
* wallet test fixed again, working with default path
* rustfmt
* fix server tests
* rustfmt
* make parent_id a trait on walletbackend
* rustfmt
* add ability for wallet to switch between multiple named accounts, and tests (not complete)
* rustfmt
* account switching tests in place and passing
* rustfmt
* compile and test with latest libsecp changes
* added public key sum to calculated e for aggsig
* rustfmt
* Update secp to 26
* bulletproof bip32 path integration
* rustfmt
* wallet restore updated with bip32 paths, also restores accounts
* rustfmt
* rustfmt
* remove old extkey
* remove old extkey
* rustfmt
* add wallet account commands
* rustfmt
* update wallet documentation
* rustfmt
* merge from master
* update libsecp tag
* merge from upstream and fix server test
* rustfmt
* rustfmt
* merge from master
* update latest libsecp merge
* fix commitment to zero value generation
When processing a new block, attemps to make the scope of the
corresponding `txhashet` lock as small as possible. The idea is
to avoid bad interactions with orphan locks or block post
processing (i.e. adapter events, etc)
* Go back to db for head and header_head
* Use batch consistently for db access
* Pass active txhashset and batch in the ctx
* Only update head in db if total work increases
* Only updated header_head if total work (on header chain) increases
* cuck placeholder
* rustfmt
* cuckatoo, early days
* rustfmt
* data structures are in place, siphash key creation is consistent with @tromp
* solver in place, (not yet working)
* cuckatoo test solver working with test nonce
* rustfmt
* update solver to remove adjacency list removals
* verifier functioning
* rustfmt
* Proper error handing in Cuckatoo module, couple of tests
* modify cuckoo/cuckatoo solvers and verifiers to function identically, in advance of trait refactoring
* rustfmt
* refactor PoW context into trait, default to using cuckoo context
* rustfmt
* create macros for integer casting/unwraps
* don't instantiate structs when just verifying, add test validation vector for cuckatoo 29
* rustfmt
* don't init cuckoo structs if just validating
* test fix
* ensure BH hashing for POW is only done within miner/validators
* introduce rewindable_kernel_view
cleanup header in extension
* cleanup and docs/comments
* txhashset does not need to be mutable here
* pull validate_kernel_history out into fn
* block_sums and full kernel sum verification
* rustfmt
* add docs/comments
* docs
* rustfmt
* comment on fact total_kernel_sum is redundant now
* make sure we setup block_sums correctly on a fork
* rustfmt
* replace those asserts with errors
* rustfmt
* Move Proof and Difficulty types to pow module. Difficulty scaling calculation.
* Diffculty scaling and size shift for 2nd PoW
* Backport e7eb26ee
* Test compilation fixes
* Scaling only need to go one way, as @tromp pointed out
