* added mining subview, changed main menu selection
* collecting difficulty stats from miner
* add diff calc view, separate server stats
* rustfmt
* block difficulty data output
* rustfmt
* ensure diff data is always shown
* don't write to stderr when tui running
Fixes#783.
* 2^60 still fails on OSX, experimentally found safe max 2^55
* Handle negative values
I tried to use .abs(), unfortuantely fuzz test kills it with
`panicked at 'attempt to negate with overflow', /Users/travis/build/rust-lang/rust/src/libcore/num/mod.rs:1146:17`
Fuzz test demonstrated that the current implemenation panics on reading from the network when serialized tx. Contains for example huge number of input or outputs. It allows a mailicious sender
to kill the server.
* hash_with_index on non-leaf nodes
rework the Merkle proof to include the pos of each sibling in the path
* rustfmt
* cleanup
* cleanup
* use get_from_file in validate (children may have been "removed")
* rustfmt
* fixup store tests
* remove difficulty field from block_header (derive from total_difficulty)
* address feedback on PR and general cleanup
* rustfmt
* rework pow/difficulty validation in pipe::validate_header
now that we only have total_difficulty available
* cleanup various todos
* rustfmt
* rework DifficultyIterator to track header and prev_header state
* rustfmt caught some garbage syntax
* cleanup
* move from storing per block kernel_offset in the block header
to storing the total accumulated kernel offset on each block header
* rustfmt
* cleanup
* wip
* failing test for being too eager when pruning a sibling
* commit
* rustfmt
* [WIP] modified get_shift and get_leaf_shift to account for leaving "pruned but not compacted" leaves in place
Note: this currently breaks check_compact as nothing else is aware of the modified behavior
* rustfmt
* commit
* rustfmt
* basic prune/compact/shift working
* rustfmt
* commit
* rustfmt
* next_pruned_idx working (I think)
* commit
* horizon test uncovered some subtle issues - wip
* rustfmt
* cleanup
* rustfmt
* commit
* cleanup
* cleanup
* commit
* rustfmt
* contains -> binary_search
* rustfmt
* no need for height==0 special case
* wip - works for single compact, 2nd one breaks the mmr hashes
* commit
* rustfmt
* fixed it (needs a lot of cleanup)
we were not traversing all the way up to the peak if we pruned an entire tree
so rm_log and prune list were inconsistent
* multiple compact steps are working
data file not being copmacted currently (still to investigate)
* cleanup store tests
* cleanup
* cleanup up debug
* rustfmt
* take kernel offsets into account when summing kernels and outputs for full txhashset validation
validate chain state pre and post compaction
* rustfmt
* fix wallet refresh (we need block height to be refreshed on non-coinbase outputs)
otherwise we cannot spend them...
* rustfmt
* Implementation of compaction for the chain. Single entry point on the chain triggers compaction of all MMRs as well as the cleanup of the positional index and full blocks.
* API endpoint, additional tests and more fixes for compaction
* Also prune PMMR metadata, minor bug fix
* PMMR store tests fix
* family_branch() to recursively call family() up the branch
todo
- we hit a peak, then we need to get to the root somehow
- actually get the hashes to build the proof
* wip
* some additional testing around merkle tree branches
* track left/right branch for each sibling as we build the merkle path up
* MerkleProof and basic (incomplete) verify fn
* I think a MerkleProof verifies correctly now
need to test on test case with multiple peaks
* basic pmmr merkle proof working
* MerkleProof now serializable/deserializable
* coinbase maturity via merkle proof basically working
* ser/deser merkle proof into hex in api and wallet.dat
* cleanup
* wip - temporarily saving merkle proofs to the commit index
* assert merkle proof in store matches the rewound version
there are cases where it does not...
* commit
* commit
* can successfully rewind the output PMMR and generate a Merkle proof
need to fix the tests up now
and cleanup the code
and add docs for functions etc.
* core tests passing
* fixup chain tests using merkle proofs
* pool tests working with merkle proofs
* api tests working with merkle proof
* fix the broken comapct block hashing behavior
made nonce for short_ids explicit to help with this
* cleanup and comment as necessary
* cleanup variety of TODOs
* Initial version
* store failure parameters inside ErrorKind variants
* continue failure transformation
* 4 errors left
* still two errors
* return old code back
* finally compiling
* Fix compilation and test errors after merge
* beginning to add bullet proof messages
* Updated core transaction creation to embed the output's value and switch commit hash as part of the rangeproof message
* formatting issue
* more formatting issues
* Removing conditional feature compliation.. just bulletproofs from now on
* ensure MAX_PROOF_SIZE uses bulletproof sizing instead of earlier version
* updated with switch commit committed to in extra data
* accidentally commented out bullet-proof-size feature
* beginning to remove sum
* continuing to remove sumtree sums
* finished removing sums from pmmr core
* renamed sumtree files, and completed changes+test updates in core and store
* updating grin/chain to include removelogs
* integration of flatfile structure, changes to chain/sumtree to start using them
* tests on chain, core and store passing
* cleaning up api and tests
* formatting
* flatfiles stored as part of PMMR backend instead
* all compiling and tests running
* documentation
* added remove + pruning to flatfiles
* remove unneeded enum
* adding sumtree root struct
* wip - basic tx pool lookup for kernel short ids (compact block hydration)
* use the nonce in the compact_block to correctly generate short_ids for lookup
* query the tx pool based on kernel short_ids
* tests passing
* cleanup some logging
* cleanup logging
* WIP - split the key in final tx step
store "offset" on transaction itself
* rebase
* commit
* tx with offset
* got a test tx validating successfully using a sig from a split key and the appropriate offset
* sum up the offset for the block_header
* fix size tests for blocks and compact blocks (header now includes offset)
* use txs with offsets in most of the core tests
some tests now failing
* build kernel from k1G (k2 stored on tx, sum stored on header)
* commit
* tx now has vec of kernels
rework tx and kernel validation
* add test for tx cut_through
* wip - working on splitting in aggsig
* split the key when creating the initial sender aggsig context
* cleanup
* cleanup
* code needs claning up but split keys working for sender/receiver aggsig flow
* cleanup debug logging
* fix tests
* fix merge and basic cleanup
* fix keychain tests to use new tx_id
* Util to zip and unzip directories
* First pass at sumtree request/response. Add message types, implement the exchange in the protocol, zip up the sumtree directory and stream the file over, with necessary adapter hooks.
* Implement the sumtree archive receive logicGets the sumtree archive data stream from the network and write it to a file. Unzip the file, place it at the right spot and reconstruct the sumtree data structure, rewinding where to the right spot.
* Sumtree hash structure validation
* Simplify sumtree backend buffering logic. The backend for a sumtree has to implement some in-memory buffering logic to provide a commit/rollback interface. The backend itself is an aggregate of 3 underlying storages (an append only file, a remove log and a skip list). The buffering was previously implemented both by the backend and some of the underlying storages. Now pushing back all buffering logic to the storages to keep the backend simpler.
* Add kernel append only store file to sumtrees. The chain sumtrees structure now also saves all kernels to a dedicated file. As that storage is implemented by the append only file wrapper, it's also rewind-aware.
* Full state validation. Checks that:
- MMRs are sane (hash and sum each node)
- Tree roots match the corresponding header
- Kernel signatures are valid
- Sum of all kernel excesses equals the sum of UTXO commitments
minus the supply
* Fast sync handoff to body sync. Once the fast-sync state is fully setup, get bacj in body sync
mode to get the full bodies of the last blocks we're missing.
* First fully working fast sync
* Facility in p2p conn to deal with attachments (raw binary after message).
* Re-introduced sumtree send and receive message handling using the above.
* Fixed test and finished updating all required db state after sumtree validation.
* Massaged a little bit the pipeline orphan check to still work after the new sumtrees have been setup.
* Various cleanup. Consolidated fast sync and full sync into a single function as they're very similar. Proper conditions to trigger a sumtree request and some checks on receiving it.
* we can now hydrate empty compact blocks and process them
* add some tests to verify size of
various serialized blocks and compact blocks
* add_test_hydrate_empty_block
* fix broken test
* First tests for API handlers
* Test chain UTXO handler
* Add test sumtrees handlers
* Fix typo
* Removed unused logger
* Update bitflags to ^1.0
* Missing test for bitflags update
* Moved test to grin directory
* Remove tests dependencies
* Add API P2P tests
* Fix hex string commitment
* Fix conflicting port
* Fix directory conflict and server port
* Wait for at least one block is mined on Travis
* large updates for mining, fix async mode, will list changes in PR
* reset config and build defaults
* change to difficulty calculations
* tweaking mining params and tests
* tweaking to tests
* including pre-genesis data for difficulty adjustments, adding adjustment scenario tests
* further clarifying next_difficulty function
* moving tests out of consensus.rs
* pow test fix
* changing pre-genesis generation
* [wip] short_id implementation (first attempt)
todo - make this more reusable (a trait?) so we can use it for inputs/outputs/kernels easily
* factor short_id support out into ShortIdentifiable trait
* block can now be converted to compact_block
rename existing block.compact() -> block.cut_through()
* expose compact block representation via block api endpoint
optional with ?compact query param
* experiment with lock_heights on outputs
* playing around with lock_height as part of the switch commitment hash
* cleanup
* include features in the switch commit hash key
* commit
* rebase off master
* commit
* cleanup
* missing docs
* rework coinbase maturity test to build valid tx
* pool and chain tests passing (inputs have switch commitments)
* commit
* cleanup
* check inputs spending coinbase outputs have valid lock_heights
* wip - got it building (tests still failing)
* use zero key for non coinbase switch commit hash
* fees and height wrong order...
* send output lock_height over to wallet via api
* no more header by height index
workaround this for wallet refresh and wallet restore
* refresh heights for unspent wallet outputs where missing
* TODO - might be slow?
* simplify - do not pass around lock_height for non coinbase outputs
* commit
* fix tests after merge
* build input vs coinbase_input
switch commit hash key encodes lock_height
cleanup output by commit index (currently broken...)
* is_unspent and get_unspent cleanup - we have no outputs, only switch_commit_hashes
* separate concept of utxo vs output in the api
utxos come from the sumtrees (and only the sumtrees, limited info)
outputs come from blocks (and we need to look them up via block height)
* cleanup
* better api support for block outputs with range proofs
* basic wallet operations appear to work
restore is not working fully
refresh refreshes heights correctly (at least appears to)
* wallet refresh and wallet restore appear to be working now
* fix core tests
* fix some mine_simple_chain tests
* fixup chain tests
* rework so pool tests pass
* wallet restore now safely habndles duplicate commitments (reused wallet keys)
for coinbase outputs where lock_height is _very_ important
* wip
* validate_coinbase_maturity
got things building
tests are failing
* lite vs full versions of is_unspent
* builds and working locally
zero-conf - what to do here?
* handle zero-conf edge case (use latest block)
* introduce OutputIdentifier, avoid leaking SumCommit everywhere
* fix the bad merge
* pool verifies coinbase maturity via is_matured
this uses sumtree in a consistent way
* cleanup
* add docs, cleanup build warnings
* fix core tests
* fix chain tests
* fix pool tests
* cleanup debug logging that we no longer need
* make out_block optional on an input (only care about it for spending coinbase outputs)
* cleanup
* bump the build
* Make total_difficulty the sum of network difficulty, not whatever the miner happened to mine.
- Only for Testnet2 / Mainnet (hardforks Testnet1)
- update chain::pipe validate_header to validate according to Testnet2 rules for cumulative difference
Fixes#280
* tests that should ignore (network) difficulty
* fn new_block is explained as "utility to create a block without worrying about the key or previous header" so it gets network difficulty := 0 too
* update tx pool tests (going with "minimum" for network difficulty for now)
* add ERR outputs about bannable offences
(#406 should know about these)
* whitespace fix
* mine_simple_chain: Probably DON'T overwrite difficulty (?)
* core/mod tests "reward_empty_block" and "reward_with_tx_block" tests set to use lowest network difficulty possible
* First steps converting transaction workflow to be aggsig-enable
* integrating updated version of aggsig, which gives greater control over the contents of e
* added wallet transaction test to testing framework to enable testing the whole thing, completed interaction as far as inital response from recipient
* more aggsig work, final signature is produced now
* Construction of aggsig transaction now working to the point of the signature being built
* aggsig transactions working end-to-end in the nominal case
* refactor aggsig verify from commit and fix some tests
* more cleanup and test fixing
* cleaning up automated tests
* test+formatting fix
* added debug logging to see what we're doing in save_pos_index
* add counts to logging
* make sure inputs/outputs in tx are sorted
tweak debug logging
* DRY up sorting of inputs/outputs/kernels via a simple sort() in ser/deser
* add explicit validation fo sort order during tx and block validation
* sort order consensus rule for inputs/outputs/kernels in blocks
* we want a max of 10*num_peers blocks during sync...
not min
* fix test imports
* Very quick peer banning endpoint, helps with #406
* Ping heights (#407)
* add height to ping/ping
* reformat output
* fix p2p test
* Fix orphan handling, not related to current head. Fixes#412
* Check before borrow, fixes#267
* Not finding an output commit in pos is an AlreadySpent
* Fix race condition, sending before conn is ready
* Explicit error for unknown pos of a forked block
* Remove config outdated tests. Fix#333
* Check ref and try before borrow, fix#400
* We do not want to sync with old peers anyway
* Hide cargo compiler warning for unused NoopAdapter and unused test code. Add TODOs
* Limit number of inputs in a block, fix#261
* MAX_BLOCK_INPUTS = 300_000 // IO-time bound DoS protection. 5MB/s disk random reads gives ~150k UTXO lookups/s, so set limit at 300k UTXO inputs
* beginning to add wallet restore... api endpoints and basic restore
* basic restore working, still missing features
* rustfmt
* large speed up to output search, should be more or less working
* properly mark coinbase status
Renamed mining parameter mode to chain type, with existing types
of CI testing, user testing, testnet1 or mainnet. The public
chains (testnet1 and mainnet) come with their fully pre-defined
genesis block.
Still need to set the nonce and cycle for testnet1 genesis.
* allow selecting a commit while providing a key index
* misnamed variable
* added static reference to libsecp that can be called throughout
* don't serialise rangeproof to json if it's not desired
* forgotten new file
* amounts input and displayed in wallet are now in full grins, with optional decimal place
* rustfmt
* merge branch
* better acknowledgement of transaction being sent
* allow selecting a commit while providing a key index
* added static reference to libsecp that can be called throughout
* don't serialise rangeproof to json if it's not desired
Moved the HTTP APIs away from the REST endpoint abstraction and
to simpler Hyper handlers. Re-established all routes as v1.
Changed wallet receiver port to 13415 to avoid a gap in port
numbers.
Finally, rustfmt seems to have ignored specific files arguments,
running on everything.
* adding more useful handlers
* added method to return last n leaf nodes inserted into the sum tree
* endpoints in place for getting last n sumtree nodes
* Fix for the chain pipeline partly relying on an outdated head,
leading to not properly recognizing a fork and inconsistent sum
tree state.
* Do not drop block requests during sync that don't get satisfied,
retry enough time to get them and avoid stall.
* Always validate header, even in sync where we may have validated
it already. We don't want a block coming from a peer that could
squeeze through with an invalid header.
* When syncing, do not mark blocks that were errored by the chain
as received (typical case: orphan). Keep retrying.
* Improved chain state dump for debugging.
* Do not add to orphans blocks too far in the future.
* Better error reporting on db errors.
* Related sync test fixes.
TODO figure out why syncing peers timeout so often, very useful
to test but not that great for a fast sync experience.
* Adding switch commit to grin outputs
* logging output fix
* adding switch commitment hash to sum tree node
* added hash_with to Hashed trait, to allow for hashing to include another writeable element
* adding hash_with as method in hashed trait
* fail deserialization if input or outputs are not sorted for txs, fail deserialization if inputs or outputs or kernels are not sorted for blocks, make sure we sort them before serialization
* extract consensus rule into verify_sort_order on VerifySortOrder trait
* rework the sort and write into WriteableSorted trait
* fix wallet sending bug (fees related) so we can test
* refactored read_and_verify_ordered out into ser, used in transaction and block
* rename pubkey -> key_id, these are key identifiers, not public keys
* reduce identifier seize to 10 bytes, get rid of fingerprints (identifiers are now small enough to use as-is)
* IDENTIFIER_SIZE const
* add FeeDispute error for when sender and recipient disagre on fee calculation (should never happen)
* does not need to be mut
* cleaned up some warnings
* add lock_height to Transaction and TxKernel, sign msg including both fee and lock_height in kernel
* make the order of the fields in tx and kernel more consistent
* rename to kernel_sig_msg
* add test to cover adding timelocked tx to pool, where tx is invalid based on current height of the blockchain
* add tests for adding timelocked txs to blocks (valid and otherwise)
* store wallet output data in hashmap
* cleanup up commented out code
* pass pubkey/identifier and not derivation around to miner etc.
* fix failing tests
