Currently we pass a Vec. This requires an extra allocation and copy of all elements if a caller doesn't have a Vec already, which is at least 95% of cases.
Another, a smaller issue, we have a function util::to_hex and some structs implement to_hex() on top of it, so we have a mix of it in the code. This PR introduces a trait and a blanket impl for AsRef<[u8]> which brings a uniform API (obj.to_hex()). One unfortunate case is arrays of size bigger than 32 - Rust doesn't implement AsRef for them so it requires an ugly hack (&array[..]).to_hex().
* no need to rehash with index to compare output with input spending it
* compare output identifier when checking is_unspent()
* output identifier from cleanup
* no need for AsFixedBytes we can just use AsRef<[u8]>
* cleanup FixedLength trait
* revert this change for now
* fix store tests
* cleanup and fix tests after rebase
* fix tests
* update based on PR review
less hard-coded values now
* cleanup
* Implement validation and generation of block header by proof and pre pow
* Optimize
* handle missed ser result errors and test invalid pre_pow
* switch to specific error mapping
* clean import
* header version now explicit, no "default" version
"empty" block can safely have verion 0 as it needs to be set later
* default block header uses version 1
* HeaderVersion(1)
* add 2nd HF and cuckaroom
* add cuckaroom and hardfork tests
* remove all traces of later phaseouts
* refactor header_version from valid_header_version
* add bitmap accumulator
refactor vec backend so we can use it outside of tests
introduce a "hash only" vec backend for the accumulator
* get core tests passing
* initial test coverage for bitmap_accumulator
* better test coverage for bitmap accumulator and cleanup code
* refactor txhashset roots, call validate() on roots during block validation
* fix store tests
* log the "merged" root when validating roots
* cleanup, revise based on feedback
* cleanup
* rework it to pass explicit size into bitmap accumulator when applying
* fix: split state validation status into kernel and rproof updates. And fix sync status for these two states
* fix: show correct number of leaves for pruned MMR as well as unpruned
* docs: better docs for kernel/range proof validation
* fix: ordering of kernel and rproofs validation in TUI
* fix: typo in rangeproofs api and comments
We use crate renaming to remove grin prefix, as result imports for util and keychain crates start with crate::
which looks like a part of the current crate. This PR fixes it.
Also some small improvements were made.
If this approach works I'll replicate it to other crates.
* add function to retrieve a set of pmmr indices between a given block height range
* typo
* change APU to just return required indices
* change pmmr index retrieval, change new function to only return pmmr indices between blocks
* tx combinators now take operate on Result to allow for more robust errors handling
replace with_fee() and with_lock_height() with a more flexible with_features()
* pass kernel features in as arg to build::transaction()
* fix chain tests
* fix pool tests
* do not pass kernel around in the combinators
just set it once on the tx when building
* build::partial_transaction now takes a existing tx to build on
* Verify headers and blocks only when needed
Curretnly we have some lightweigt validation implemented as part of
entity deserialization, which is safer and allows us to not parse the
entire object if some part is invalid. At the same time this logic
always applies when we read an entity, eg when reading from DB.
This PR introduces UntrustedHeader/Block which is used when we read from
the network. It does partial validation during read, then it is supposed
to be converted into regular header/block which doesn't validate itself.
Also this PR adds "lightweight" validation to block header read like we have
for block body, so we don't parse block body if the header is invalid.
Fixes#1642
* Move version validation to untrusted header
* update fuzz tests
* wip
* exhaustive match
* write with fixed v1 strategy when writing for hashing
* local protocol version is 2
* cleanup "size" tests that exercise v1 vs v2 vs default protocol versions
* add proto version to Connected! log msg
* cleanup docs
* negotiate protocol version min(local, peer) when doing hand/shake
* Optimize POW read
This functionality is used to deserialize header (from network or from
DB), it was taking up to 40% cpu time during initial header sync. This
PR brings it down to 3-4%. Function read_number would look better as
closure, unfortunately the compliler doesn't inline it in this case, so
it would be 2x slower.
* Remove unused code
* wip
* sync sort of works now
* get rid of the deadlock during compaction
there is *always* a deadlock in there when we make changes like this...
* cleanup how we rebuild the sync MMR on init
* cleanup rewind logic
* roll the "fix invalid root" changes into this PR
* move rebuild_height_pos_index into txhashset
and pass in header_pmmr for header lookups
* cleanup and remember to setup sync head on init
* cleanup unnecessary ref muts
* rebuild_height_pos_index when writing txhashset
* schnorr signature batch verification
* Split the fee and lock_height verification out from the batch signature verification
* Remove the new added fee_height_verify to comply with #2859
* fix: the last n could not be leaf?
* fee and lock_height now maintained in kernel features enum variants
* sum fees via explicit enum variants
* cleanup semantics around with_fee and with_lock_height for tx builders
* document the kernel feature variants
* serialize kernel features correctly for api json
* cleanup
* commit
* bump to unstick azure
* introduce protocol version to deserialize and read
* thread protocol version through our reader
* cleanup
* cleanup
* streaming_reader cleanup
* Pass protocol version into BinWriter to allow for version specific serialization rules.
* rustfmt
* read and write now protocol version specific
* introduce protocol version to deserialize and read
* thread protocol version through our reader
* example protocol version access in kernel read
* fix our StreamingReader impl (WouldBlock woes)
* debug log progress of txhashset download
* create 2.0.0 branch
* fix humansize version
* update grin.yml version
* PoW HardFork (#2866)
* allow version 2 blocks for next 6 months
* add cuckarood.rs with working tests
* switch cuckaroo to cuckarood at right heights
* reorder to reduce conditions
* remove _ prefix on used args; fix typo
* Make Valid Header Version dependant on ChainType
* Rustfmt
* Add tests, uncomment header v2
* Rustfmt
* Add FLOONET_FIRST_HARD_FORK height and simplify logic
* assume floonet stays closer to avg 60s block time
* move floonet hf forward by half a day
* update version in new block when previous no longer valid
* my next commit:-)
* micro optimization
* Support new Bulletproof rewind scheme (#2848)
* Update keychain with new rewind scheme
* Refactor: proof builder trait
* Update tests, cleanup
* rustfmt
* Move conversion of SwitchCommitmentType
* Add proof build trait to tx builders
* Cache hashes in proof builders
* Proof builder tests
* Add ViewKey struct
* Fix some warnings
* Zeroize proof builder secrets on drop
* Modify mine_block to use wallet V2 API (#2892)
* update mine_block to use V2 wallet API
* rustfmt
* Add version endpoint to node API, rename pool/push (#2897)
* add node version API, tweak pool/push parameter
* rustfmt
* Upate version api call (#2899)
* Update version number for next (potential) release
* zeroize: Upgrade to v0.9 (#2914)
* zeroize: Upgrade to v0.9
* missed Cargo.lock
* [PENDING APPROVAL] put phase outs of C32 and beyond on hold (#2714)
* put phase outs of C32 and beyond on hold
* update tests for phaseouts on hold
* Don't wait for p2p-server thread (#2917)
Currently p2p.stop() stops and wait for all peers to exit, that's
basically all we need. However we also run a TCP listener in this thread
which is blocked on `accept` most of the time. We do an attempt to stop
it but it would work only if we get an incoming connection during the
shutdown, which is a week guarantee.
This fix remove joining to p2p-server thread, it stops all peers and
makes an attempt to stop the listener.
Fixes [#2906]
* rustfmt
* allow version 2 blocks for next 6 months
* add cuckarood.rs with working tests
* switch cuckaroo to cuckarood at right heights
* reorder to reduce conditions
* remove _ prefix on used args; fix typo
* Make Valid Header Version dependant on ChainType
* Rustfmt
* Add tests, uncomment header v2
* Rustfmt
* Add FLOONET_FIRST_HARD_FORK height and simplify logic
* assume floonet stays closer to avg 60s block time
* move floonet hf forward by half a day
* update version in new block when previous no longer valid
* my next commit:-)
* micro optimization
* generate txhashset archives on 250 block intervals.
* moved txhashset_archive_interval to global and added a simple test.
* cleaning up the tests and adding license.
* increasing cleanup duration to 24 hours to prevent premature deletion of the current txhashset archive
* bug fixes and changing request_state to request height using archive_interval.
* removing stopstate from chain_test_helper to fix compile issue
* Implement simple zeroing of BlindingFactor in Drop
* rustfmt
* Make Debug implementation for BlindingFactor empty
* Implement BlindingFactor zeroing unit test
* mnemonic.rs: fix deprecated warning in test_bip39_random test
* Use zeroize crate to clear BlindingFactor
* Fix comment and implement dummy Debug trait for BlindingFactor
* Fix formatter
* introduce HeaderVersion (was u16) for type safety
cleanup pow ser/deser (version unused)
* fixup tests for HeaderVersion
* validate header version during header deserialization
* bucket_transactions is now fee_to_weight aware
bucket_transactions now returns the underlying txs
* cleanup
* simplify pool bucket sorting, no need for depth sort key
* Introduce optional size_file.
Support fixed size data file via an optional elmt_size.
Support variable size data file via optional size_file.
* remember to release the size_file
* fix scoping for windows support
* cleanup legacy "3 dot" check (#2625)
* Allow to peers behind NAT to get up to preferred_max connections (#2543)
Allow to peers behind NAT to get up to preffered_max connections
If peer has only outbound connections it's mot likely behind NAT and we should not stop it from getting more outbound connections
* Reduce usage of unwrap in p2p crate (#2627)
Also change store crate a bit
* Simplify (and fix) output_pos cleanup during chain compaction (#2609)
* expose leaf pos iterator
use it for various things in txhashset when iterating over outputs
* fix
* cleanup
* rebuild output_pos index (and clear it out first) when compacting the chain
* fixup tests
* refactor to match on (output, proof) tuple
* add comments to compact() to explain what is going on.
* get rid of some boxing around the leaf_set iterator
* cleanup
* [docs] Add switch commitment documentation (#2526)
* remove references to no-longer existing switch commitment hash
(as switch commitments were removed in ca8447f3bd
and moved into the blinding factor of the Pedersen Commitment)
* some rewording (points vs curves) and fix of small formatting issues
* Add switch commitment documentation
* [docs] Documents in grin repo had translated in Korean. (#2604)
* Start to M/W intro translate in Korean
* translate in Korean
* add korean translation on intro
* table_of_content.md translate in Korean.
* table_of_content_KR.md finish translate in Korean, start to translate State_KR.md
* add state_KR.md & commit some translation in State_KR.md
* WIP stat_KR.md translation
* add build_KR.md && stratum_KR.md
* finish translate stratum_KR.md & table_of_content_KR.md
* rename intro.KR.md to intro_KR.md
* add intro_KR.md file path each language's intro.md
* add Korean translation file path to stratum.md & table_of_contents.md
* fix difference with grin/master
* Fix TxHashSet file filter for Windows. (#2641)
* Fix TxHashSet file filter for Windows.
* rustfmt
* Updating regexp
* Adding in test case
* Display the current download rate rather than the average when syncing the chain (#2633)
* When syncing the chain, calculate the displayed download speed using the current rate from the most recent iteration, rather than the average download speed from the entire syncing process.
* Replace the explicitly ignored variables in the pattern with an implicit ignore
* remove root = true from editorconfig (#2655)
* Add Medium post to intro (#2654)
Spoke to @yeastplume who agreed it makes sense to add the "Grin Transactions Explained, Step-by-Step" Medium post to intro.md
Open for suggestions on a better location.
* add a new configure item for log_max_files (#2601)
* add a new configure item for log_max_files
* rustfmt
* use a constant instead of multiple 32
* rustfmt
* Fix the build warning of deprecated trim_right_matches (#2662)
* [DOC] state.md, build.md and chain directory documents translate in Korean. (#2649)
* add md files for translation.
* start to translation fast-sync, code_structure. add file build_KR.md, states_KR.md
* add dandelion_KR.md && simulation_KR.md for Korean translation.
* add md files for translation.
* start to translation fast-sync, code_structure. add file build_KR.md, states_KR.md
* add dandelion_KR.md && simulation_KR.md for Korean translation.
* remove some useless md files for translation. this is rearrange set up translation order.
* add dot end of sentence & translate build.md in korean
* remove fast-sync_KR.md
* finish build_KR.md translation
* finish build_KR.md translation
* finish translation state_KR.md & add phrase in state.md to move other language md file
* translate blocks_and_headers.md && chain_sync.md in Korean
* add . in chain_sync.md , translation finished in doc/chain dir.
* fix some miss typos
* Api documentation fixes (#2646)
* Fix the API documentation for Chain Validate (v1/chain/validate). It was documented as a POST, but it is actually a GET request, which can be seen in its handler ChainValidationHandler
* Update the API V1 route list response to include the headers and merkleproof routes. Also clarify that for the chain/outputs route you must specify either byids or byheight to select outputs.
* refactor(ci): reorganize CI related code (#2658)
Break-down the CI related code into smaller more maintainable pieces.
* Specify grin or nanogrins in API docs where applicable (#2642)
* Set Content-Type in API client (#2680)
* Reduce number of unwraps in chain crate (#2679)
* fix: the restart of state sync doesn't work sometimes (#2687)
* let check_txhashset_needed return true on abnormal case (#2684)
* Reduce number of unwwaps in api crate (#2681)
* Reduce number of unwwaps in api crate
* Format use section
* Small QoL improvements for wallet developers (#2651)
* Small changes for wallet devs
* Move create_nonce into Keychain trait
* Replace match by map_err
* Add flag to Slate to skip fee check
* Fix secp dependency
* Remove check_fee flag in Slate
* Add Japanese edition of build.md (#2697)
* catch the panic to avoid peer thread quit early (#2686)
* catch the panic to avoid peer thread quit before taking the chance to ban
* move catch wrapper logic down into the util crate
* log the panic info
* keep txhashset.rs untouched
* remove a warning
* [DOC] dandelion.md, simulation.md ,fast-sync.md and pruning.md documents translate in Korean. (#2678)
* Show response code in API client error message (#2683)
It's hard to investigate what happens when an API client error is
printed out
* Add some better logging for get_outputs_by_id failure states (#2705)
* Switch commitment doc fixes (#2645)
Fix some typos and remove the use of parentheses in a
couple of places to make the reading flow a bit better.
* docs: update/add new README.md badges (#2708)
Replace existing badges with SVG counterparts and add a bunch of new ones.
* Update intro.md (#2702)
Add mention of censoring attack prevented by range proofs
* use sandbox folder for txhashset validation on state sync (#2685)
* use sandbox folder for txhashset validation on state sync
* rustfmt
* use temp directory as the sandbox instead actual db_root txhashset dir
* rustfmt
* move txhashset overwrite to the end of full validation
* fix travis-ci test
* rustfmt
* fix: hashset have 2 folders including txhashset and header
* rustfmt
*
(1)switch to rebuild_header_mmr instead of copy the sandbox header mmr
(2)lock txhashset when overwriting and opening and rebuild
* minor improve on sandbox_dir
* add Japanese edition of state.md (#2703)
* Attempt to fix broken TUI locale (#2713)
Can confirm that on the same machine 1.0.2 TUI looks great and is broken on
the current master. Bump of `cursive` version fixed it for me.
Fixes#2676
* clean the header folder in sandbox (#2716)
* forgot to clean the header folder in sandbox in #2685
* Reduce number of unwraps in servers crate (#2707)
It doesn't include stratum server which is sufficiently changed in 1.1
branch and adapters, which is big enough for a separate PR.
* rustfmt
* change version to beta
* Add hooks for network and chain events. Move logging to an EventLogger
* implement webhooks
* fix failing test
* remove unnecessary 'pub'
* add some metadata to the json payload
* avoid unecessary init
* resolve conflicts
* chore: replace trim_right with $ sed -i'' 's/trim_right/trim_end/' **/*.rs
* docs: individually document macros to avoid warning, add TODO to make to_edge hygenic
* docs: document impl_array_newtype macros, refactor: move all impl_array_newtype macro traits into impl_array_netype_index
* Add a slate version field
* Move current slate version to constant
* Change slate version to uint
* Add default 0 incase it's missing (pre-versioning)
* rework bucket txs
* introduce max_block_weight to global (based on chain_type)
not yet in use anywhere...
* now using global::max_block_weight() everywhere
* testing max_block_weight needs to be big enough to cover existing chain tests...
* add some test coverage for prepare mineable transactions at the block weight limit
introduce AsLimitedTx{max_weight} so we can build artifically small blocks (per mining config)
* cleanup
* default_mineable_max_weight is just the default max_block_weight
we do not need to account for coinbase reward here (tx vs block)
* 75 change outputs in a test is not valid now that we have a low block weight limit...
* Save and display optional slate messages
* rustfmt
* fixes and test updates
* rustfmt
* output json
* rustfmt
* add better serialisation of slate message, rename to secp_ser, add tests for serialization functions
* rustfmt
* max length for message enforced by API
* rustfmt
