* Refactor hash cache handling to add properly. Only add to cache if the block is definitevely accepted or rejected.
* Dedup of header processing now looks like premature optimization. Very fast anyway. Removing the header hash cache.
* Remove now unnecessary txhashset write lock
* Ring buffer of hashes the chain has already processed
* Specifically report too old blocks as peer should be banned
* Move sync check for block relay, clean TODO
* No use processing transactions when syncing
* Ignore blocks older than horizon in pruning nodes
Only rewind kernels to avoid requiring proper bitmap extraction. Also avoids maintaining bitmap data for kernels by introducing a "prunable" flag on PMMR backend.
* Added sizeshift to proof of work and encoding as VLQ
* Fix some loose ends to get things to compile
* Rename sizehift to min_sizeshift
* Pipeline checks for PoW Cuckoo Cycle size shift and adjustment
factor before difficulty comparison.
* Working delta+VLQ encoding of u64 PoW, unfortunately still a
little larger than u32, at least for Cuckoo30.
* Changed binary encoding of proof of work to a sequence of exact
compact bit representation of each nonce. Somewhat simpler,
shorter and matches the data to be hashed.
* Few fixes based on @tromp feedback:
* Max nonce is 2^(N-1)
* Need a separate constant for reference sizeshift
* Cuckoo implementation now conserves sizeshift. Test providing a
block at higher sizeshift.
* Last small overflow protection
* Migrate main node store to LMDB
In preparation to using LMDB as a wallet database, migrate the
node db. There's no point in having 2 key-value stores.
In addition LMDB provides a few advantages as a node db, namely a
much faster build (compared to RocksDb), lesser dependencies and
transactions.
* Migrated p2p store to lmdb, stuff compiles
* More fixes, chain tests starting to pass
* Fixed txhashset rollback messing with block save and general batch delimitation. Chain tests passing.
* rustfmt
* LMDB max map size of 10MB isn't really workable. Half TB seems reasonable.
* Fix wallet tests
* Rather crucial commit was missing
* rustfmt
* Fixing new merged tests following lmdb changes
* rustfmt
* * Make txhashset validation read-only on fast sync to avoid having
a really long open transaction.
* Fix deadlock in new block processing, batch should always be
created within a txhashset lock (when they interact).
* Comment about batch and txhashset interlacing
* Fix store tests to use batch
* Externalize wallet config and seed
* Converted direct read access to file outputs map to an iterator
* Cleaned up and simplified wallet Backend trait:
* No more direct mutable access to internal structures (HashMap)
* Batch interface for all writes
* Remove unneeded read wrapper (read_wallet)
* rustfmt
* First (incomplete) pass at wallet LMDB backend
* Progressing on lmdb backent iml
* Added batch impl for LMDB wallet backend. Pretty much done with it, but not sure how to deal with commit (owned).
* rustfmt
* Wrapping LMDB batch around a refcell to work around borrow rules
* Compilation up to grin chain
* Add kernel commitments sum and kernel and output MMR sizes to block header
* Sum a block without including previous sums, cleanup. Blocks are now summed and validated based on their own totals and not the totals since genesis. This allows to get rid of BlockSum and simplified the setting of a new block's roots, kernel sum and MMR sizes. Fixes#116
* Additional kernel MMR validation to check all prior header roots successively
* Wallet tests fix
* only store leaves in rm_log
* cleanup
* commit
* add failing test to cover case where we compact
an already compacted data file
* fix the logic for pruning the data file
* rm_log only contains leaves
prunelist only contains non-leaf subtree roots
* cleanup
* commit
* bitmap impl running in parallel
* rustfmt
* this is better - rewind unremoves outpu pos spent by rewound inputs
* commit
* commit
* pass bitmap around when rewinding
* store and retrieve input bitmap per block from the db
* Replace the RemoveLog with the UTXO set.
* utxo set starting to pass tests
* stuff works...
* split store types out
added some tests to compare the performance of the rm_log and the proposed utxo_set
* pull prune_list out into standalone file
* cleanup, get rid of unused height param
* cleanup and add doc comments
* add support for migrating rm_log to utxo_set
* take snapshot of utxo file during fast sync
implement migration of rm_log -> utxo_set
* rename rewound_copy to snapshot
* fixup pmmr tests to reflect cutoff_pos
* cleanup unused import
* check_compact needs to rewind the utxo_set as appropriate
* fix pool tests
* fixup core tests
* cache block_input_bitmaps via LruCache in store
* cache block header on initial write to db
* rename utxo_set -> leaf_set
and remove references to "spent" in grin_store
* better document the rewind behavior
Small refactoring of one folder, if it makes sense I could extend the scope.
* Remove some cloning (real and just verbosity in the code)
* Naming conventions like to/into*
* Some Clippy's suggestions
I found that we don't use field init shorthand syntax, so I didn't touch this part, was it discussed before?
* verify a tx like we verify a block (experimental)
* first minimal_pool test up and running but not testing what we need to
* rework tx_pool validation to use txhashset extension
* minimal tx pool wired up but rough
* works locally (rough statew though)
delete "legacy" pool and graph code
* rework the new pool into TransactionPool and Pool impls
* rework pool to store pool entries
with associated timer and source etc.
* all_transactions
* extra_txs so we can validate stempool against existing txpool
* rework reconcile_block
* txhashset apply_raw_tx can now rewind to a checkpoint (prev raw tx)
* wip - txhashset tx tests
* more flexible rewind on MMRs
* add tests to cover apply_raw_txs on txhashset extension
* add_to_stempool and add_to_txpool
* deaggregate multi kernel tx when adding to txpoool
* handle freshness in stempool
handle propagation of stempool txs via dandelion monitor
* patience timer and fluff if we cannot propagate
to next relay
* aggregate and fluff stempool is we have no relay
* refactor coinbase maturity
* rewrote basic tx pool tests to use a real txhashset via chain adapter
* rework dandelion monitor to reflect recent discussion
works locally but needs a cleanup
* refactor dandelion_monitor - split out phases
* more pool test coverage
* remove old test code from pool (still wip)
* block_building and block_reconciliation tests
* tracked down chain test failure...
* fix test_coinbase_maturity
* dandelion_monitor now runs...
* refactor dandelion config, shared across p2p and pool components
* fix pool tests with new config
* fix p2p tests
* rework tx pool to deal with duplicate commitments (testnet2 limitation)
* cleanup and address some PR feedback
* add big comment about pre_tx...
* block sums and reworked block validation
read and write block_sums
refactor validate on both block and txhashset
write block_sum on fast sync
we store the kernel_sum (need to account for the offset)
* block_sums
* rustfmt
* cleanup
* wip BlockMarker struct, get rid of PMMRMetadata
* use rewind to init the txhashet correctly on startup, we do not need to track pos via metadata (we have block markers), we do not need to open the txhashset with specific pos (we have rewind)
* better logging on init
* keep rewinding and validating on init, to find a good block
* use validate_roots on chain init
* Beginning to remove in-process miner
* rustfmt
* rustfmt
* rustfmt
* rustfmt
* remove pow crate and put remnants into core
* rustfmt
* fix test compilation in core and chain
* rustfmt
* Updating server tests to use test miner
* rustfmt
* rustfmt
* remove pow from test matrix
* adding basic stratum stats to TUI
* run stratum server at all times, and halt messages while syncing
* fix core tests
* add ability to run internal test miner for cuckoo 16 testing
* modify build instructions
* add hint for gcc-5 for building CUDA plugin
* add N_BLOCks comment for >=6GB GPU cards
* minor logging cleanup
* mark to be kept + reformat as "@ height [hash]"
* show CuckooNN + better wording
* move some debug! to trace!
* more informative debugs
* standardising on always showing chain tips as "cumulative difficulty @ height [hash]"
* make 2 debug outputs into a single
* "no peers" as warning (not info) to let it stand out more clearly
* move fn param (used only in this one debug line)
* clarify difficulty "units"
* remove difficulty field from block_header (derive from total_difficulty)
* address feedback on PR and general cleanup
* rustfmt
* rework pow/difficulty validation in pipe::validate_header
now that we only have total_difficulty available
* cleanup various todos
* rustfmt
* rework DifficultyIterator to track header and prev_header state
* rustfmt caught some garbage syntax
* cleanup
* move from storing per block kernel_offset in the block header
to storing the total accumulated kernel offset on each block header
* rustfmt
* cleanup
* Cleanup MMRs positional indexes. Get rid of the kernel position index.
Introduced a new block marker index that keeps, for each block, the
respective positions in the output and kernel MMRs. This is now sufficient
for rewind.
* Block marker compaction
* adding file position index data accessable to the chain, and allowing for storage of such within db
* missing file
* restart files at last recorded position in stored file metadata
* just use tip to store last pmmr index information
* error handling
* test fix
* beginning to remove sum
* continuing to remove sumtree sums
* finished removing sums from pmmr core
* renamed sumtree files, and completed changes+test updates in core and store
* updating grin/chain to include removelogs
* integration of flatfile structure, changes to chain/sumtree to start using them
* tests on chain, core and store passing
* cleaning up api and tests
* formatting
* flatfiles stored as part of PMMR backend instead
* all compiling and tests running
* documentation
* added remove + pruning to flatfiles
* remove unneeded enum
* adding sumtree root struct
* Util to zip and unzip directories
* First pass at sumtree request/response. Add message types, implement the exchange in the protocol, zip up the sumtree directory and stream the file over, with necessary adapter hooks.
* Implement the sumtree archive receive logicGets the sumtree archive data stream from the network and write it to a file. Unzip the file, place it at the right spot and reconstruct the sumtree data structure, rewinding where to the right spot.
* Sumtree hash structure validation
* Simplify sumtree backend buffering logic. The backend for a sumtree has to implement some in-memory buffering logic to provide a commit/rollback interface. The backend itself is an aggregate of 3 underlying storages (an append only file, a remove log and a skip list). The buffering was previously implemented both by the backend and some of the underlying storages. Now pushing back all buffering logic to the storages to keep the backend simpler.
* Add kernel append only store file to sumtrees. The chain sumtrees structure now also saves all kernels to a dedicated file. As that storage is implemented by the append only file wrapper, it's also rewind-aware.
* Full state validation. Checks that:
- MMRs are sane (hash and sum each node)
- Tree roots match the corresponding header
- Kernel signatures are valid
- Sum of all kernel excesses equals the sum of UTXO commitments
minus the supply
* Fast sync handoff to body sync. Once the fast-sync state is fully setup, get bacj in body sync
mode to get the full bodies of the last blocks we're missing.
* First fully working fast sync
* Facility in p2p conn to deal with attachments (raw binary after message).
* Re-introduced sumtree send and receive message handling using the above.
* Fixed test and finished updating all required db state after sumtree validation.
* Massaged a little bit the pipeline orphan check to still work after the new sumtrees have been setup.
* Various cleanup. Consolidated fast sync and full sync into a single function as they're very similar. Proper conditions to trigger a sumtree request and some checks on receiving it.
* first pass at allow/deny lists for hard-coded peers (not just seeds)
* commit
* add peers_allow and peers_deny examples and comments to grin.toml
* always ask for compact block
always fail to hydrate
always fallback to requesting full block
* decide to send full (empty) block over compact (empty) block
* add some randomness to the decision around broadcasting an empty block
as a block or as a compact block (so we can exercise more code paths easily)
* [wip] header first propagation
successfully propagating headers (unless we mined the block itself)
not yet asking for the block if we receive a header
* call request_block after successful processing header
* cleanup and skip asking for block if header is an orphan
* comments around error handling in receive_header
* large updates for mining, fix async mode, will list changes in PR
* reset config and build defaults
* change to difficulty calculations
* tweaking mining params and tests
* tweaking to tests
* including pre-genesis data for difficulty adjustments, adding adjustment scenario tests
* further clarifying next_difficulty function
* moving tests out of consensus.rs
* pow test fix
* changing pre-genesis generation
* Making initial sync easier on the eyes:
- display "chain pointers" as: cumulative @ height [hash]
- clarify and line up to make the "pointers" easy to compare
- make every 100th block show info on debug level info, else as before
* experiment with lock_heights on outputs
* playing around with lock_height as part of the switch commitment hash
* cleanup
* include features in the switch commit hash key
* commit
* rebase off master
* commit
* cleanup
* missing docs
* rework coinbase maturity test to build valid tx
* pool and chain tests passing (inputs have switch commitments)
* commit
* cleanup
* check inputs spending coinbase outputs have valid lock_heights
* wip - got it building (tests still failing)
* use zero key for non coinbase switch commit hash
* fees and height wrong order...
* send output lock_height over to wallet via api
* no more header by height index
workaround this for wallet refresh and wallet restore
* refresh heights for unspent wallet outputs where missing
* TODO - might be slow?
* simplify - do not pass around lock_height for non coinbase outputs
* commit
* fix tests after merge
* build input vs coinbase_input
switch commit hash key encodes lock_height
cleanup output by commit index (currently broken...)
* is_unspent and get_unspent cleanup - we have no outputs, only switch_commit_hashes
* separate concept of utxo vs output in the api
utxos come from the sumtrees (and only the sumtrees, limited info)
outputs come from blocks (and we need to look them up via block height)
* cleanup
* better api support for block outputs with range proofs
* basic wallet operations appear to work
restore is not working fully
refresh refreshes heights correctly (at least appears to)
* wallet refresh and wallet restore appear to be working now
* fix core tests
* fix some mine_simple_chain tests
* fixup chain tests
* rework so pool tests pass
* wallet restore now safely habndles duplicate commitments (reused wallet keys)
for coinbase outputs where lock_height is _very_ important
* wip
* validate_coinbase_maturity
got things building
tests are failing
* lite vs full versions of is_unspent
* builds and working locally
zero-conf - what to do here?
* handle zero-conf edge case (use latest block)
* introduce OutputIdentifier, avoid leaking SumCommit everywhere
* fix the bad merge
* pool verifies coinbase maturity via is_matured
this uses sumtree in a consistent way
* cleanup
* add docs, cleanup build warnings
* fix core tests
* fix chain tests
* fix pool tests
* cleanup debug logging that we no longer need
* make out_block optional on an input (only care about it for spending coinbase outputs)
* cleanup
* bump the build
* Make total_difficulty the sum of network difficulty, not whatever the miner happened to mine.
- Only for Testnet2 / Mainnet (hardforks Testnet1)
- update chain::pipe validate_header to validate according to Testnet2 rules for cumulative difference
Fixes#280
* tests that should ignore (network) difficulty
* fn new_block is explained as "utility to create a block without worrying about the key or previous header" so it gets network difficulty := 0 too
* update tx pool tests (going with "minimum" for network difficulty for now)
* add ERR outputs about bannable offences
(#406 should know about these)
* whitespace fix
* mine_simple_chain: Probably DON'T overwrite difficulty (?)
* core/mod tests "reward_empty_block" and "reward_with_tx_block" tests set to use lowest network difficulty possible
Due to the construction of PMMRs the last element, when its a leaf,
can never be pruned as it has no parent yet and it will be needed
to calculate that hash. To work around this, we now insert coinbase
outputs first to add at least one output of padding.
Also changed the `set_sumtree_root` function on chain a bit to allow
setting the roots on a fork. Mostly useful for tests.
Added new test case to handle both the issue above and spending
transactions within a fork.
* distinguish select (among futures) from select coins. Regex search in project for select\b shows we hardly use select, but maybe could use it to add timeouts more cleanly. ("Want to add a timeout to any future? Just do a select of that future and a timeout future!" from https://aturon.github.io/blog/2016/08/11/futures/)
* remove a trailing space
* FAQ.md - fix typo
* wallet: display problematic tx
* update FAQ build troubleshooting to cover #443
* stdout_log_level = Info
file_log_level = Debug
* sync: show total diff @ height when syncronization is completed
* better wallet send dest format error
* move INFO "Client conn ... lost" and "Connected to peer" down to Debug
* move some level=Info to Debug, add 1000-block outputs
* Very quick peer banning endpoint, helps with #406
* Ping heights (#407)
* add height to ping/ping
* reformat output
* fix p2p test
* Fix orphan handling, not related to current head. Fixes#412
* Check before borrow, fixes#267
* Not finding an output commit in pos is an AlreadySpent
* Fix race condition, sending before conn is ready
* Explicit error for unknown pos of a forked block
* Remove config outdated tests. Fix#333
* Check ref and try before borrow, fix#400
* We do not want to sync with old peers anyway
* Hide cargo compiler warning for unused NoopAdapter and unused test code. Add TODOs
* port simple_sync across from testnet1, sync head & improved fork handling
* introduce new sync_head for tracking header chain during sync
* add debug logging in body_sync for state of various heads
* update header_head during sync if header is now the one with most work, pass in sync and head contexts to pipe
* port across sync_head changes from master, put the 1s sleep back in...
* let sync run even if the full 512 headers are all known, give sync a chance to sync against a fork that forked a long time ago
* handle height jumping during a reorg
block validaton should not check for height+1 based on head
no assumptions should be made about height indices
* quick(er) check for previous block in process_block (we cannot check height against ctx head)
* make body_sync a _lot_ faster by finding the forked block more efficiently...
* fix monitoring peers log msg
* fix chain tests
* fix grin tests - we were using the wrong genesis hash (wrong chain type)
* apparently needs setting in both places...
* body -> header -> sync (#414)
* port over body -> header -> sync changes from testnet1
* port across "simple_sync" changes from testnet1 branch
Cleanup direct refs to peer map or peer store
P2P server acts as a facade, handling the list of connected peers
and the storage of their information. Everything else goes through
the p2p server instead of having a peer map reference or going
straight to the store.
Fix p2p tests
* fix "monitoring peers" log msg (use connected_peers)
Fixes#281 by doing:
- "Starting validation pipeline for " -> "Processing "
- stdout logging default = Warning
- ERR -> INFO for "Transaction rejected: Already in pool" -- only for netadapter transaction_received, which (hopefully!) only gets tx from peers, and lots of those will be duplicates, that's good and not an error.
* Downgrade ERR even down to DEBUG.
Also let Transaction rejected show tx hash,
so users can check if such a rejection is for the
payment they expected, or just any random peer
sending in some old and already known tx.
* bad case of trying to rewind to block header height 0
* rewind_to_genesis appears to work
* do not assume genesis block at height 0 is empty, pass full block in to rewind, check for last output and kernel, use index 0 if block is empty
Moved the HTTP APIs away from the REST endpoint abstraction and
to simpler Hyper handlers. Re-established all routes as v1.
Changed wallet receiver port to 13415 to avoid a gap in port
numbers.
Finally, rustfmt seems to have ignored specific files arguments,
running on everything.
* Fix for the chain pipeline partly relying on an outdated head,
leading to not properly recognizing a fork and inconsistent sum
tree state.
* Do not drop block requests during sync that don't get satisfied,
retry enough time to get them and avoid stall.
* Always validate header, even in sync where we may have validated
it already. We don't want a block coming from a peer that could
squeeze through with an invalid header.
* When syncing, do not mark blocks that were errored by the chain
as received (typical case: orphan). Keep retrying.
* Improved chain state dump for debugging.
* Do not add to orphans blocks too far in the future.
* Better error reporting on db errors.
* Related sync test fixes.
TODO figure out why syncing peers timeout so often, very useful
to test but not that great for a fast sync experience.
Addresses a corner case of sync. If we're still in sync mode but
just caught up to the head, a block could be added through normal
gossip. So we can't short-circuit some of the header handling
even though during sync the header should have already been
validated and saved, because we can still get a block from gossip.
* Adding switch commit to grin outputs
* logging output fix
* adding switch commitment hash to sum tree node
* added hash_with to Hashed trait, to allow for hashing to include another writeable element
* adding hash_with as method in hashed trait
* added global slog instance, changed all logging macro formats to include logger instance
* adding configuration to logging, allowing for multiple log outputs
* updates to test, changes to build docs
* rustfmt
* moving logging functions into util crate
* add lock_height to Transaction and TxKernel, sign msg including both fee and lock_height in kernel
* make the order of the fields in tx and kernel more consistent
* rename to kernel_sig_msg
* add test to cover adding timelocked tx to pool, where tx is invalid based on current height of the blockchain
* add tests for adding timelocked txs to blocks (valid and otherwise)
* Integrate PMMR and its persistent backend with the Chain
* Chain can set tree roots; PMMR backend discard
* Check spent and prune for each input in new block
* Handling of forks by rewinding the state
* More PMMR tests and fixes, mostly around rewind
* Rewrite get_unspent to use the sumtrees, fix remaining compilation issues
* use head_header in add_to_memory_pool
* add COINBASE_MATURITY const to consensus
* add coinbase maturity (wip) validaton rule to validate_block
* add coinbase maturity check to validate_block
* map errors in adapters - specific errors still wip
* reworked so adapter translates chain errors to pool errors (core errors not required)
* add test for spending immature coinbase in memory pool
* wip - add test to cover spending coinbase output in chain.process_block
* added test coverage around process_block - we have a problem with coinbase output commitments
* add a comment on the failing test
* process_block will now fail validation if we attempt to spend coinbase that has not yet matured
(remember to use a new reward_key for every block). test coverage in place to verify this
* Adding output stats from plugin, when available
* adding grin config
* moving pow crate from core/pow to it's own crate
* moving POW/mining stuff into pow module, and removing mining dependency from chain
* refactored most of mining into pow module...miner.rs still in server
* update grin tests
* updating genesis block mining to use configured miner, if available
* chain tests back into chain, done with pow refactor
* use tag of cuckoo-miner for pr
* minor cleanup - unused imports
* cleanup build warnings - unused vars
* make structs pub to get rid of the private_in_public lint warning
* missing docs on RangeProof
* add missing docs to store delete function
* cleaned up deprecation warning -
tokio_core -> tokio_io
complete() -> send()
* playing around with changing cuckoo sizes on the fly
* modifying tests to use global cuckoo parameters, and checking results
* check for pow size
* Changing global function names, and removing length from POW serialization
When a new block is rejected by the validation pipeline as orphan,
the chain now keeps it in a ring buffer of capped size. Anytime
a new block is accepted, all orphans in the ring buffer are
retried. This is helpful to handle all the cases where we receive
blocks out of order for networking/timing reasons.
Introduce a new Chain struct that maintains the current head,
acts as facade to the store and the block processing pipeline
and handles all appropriate locking.
All higher level components, like the overall server, REST APIs
and miner have been update to only depend on Chain. Cleaned up
some duplication and tests now that there's a single entry point
to all blockchain operations.
Integrate the new difficulty calculation into the block chain
validation, the miner and tests. As the difficulty calculation
doesn't use varying Cuckoo sizes anymore and we narrowed down
reasonable final Cuckoo Cycle parameters, removed all Cuckoo
Cycle sizes from block headers.
Formalized easier Cuckoo Cycle sizes for testing (and possibly
testnet) by introducing a test mode in configuration. Updated
all tests.
Fixes a few loose ends in the full cycle of sending coins,
accepting them, pushing that transaction to the pool and having it
mined. More specifically:
* The API output endpoint needs to be a UTXO endpoint, as the
server can't make any guarantee about having a spent output.
* Bubbling up HTTP not found errors.
* Wallet output status checker now handles spent outputs.
* Transaction pool validates the transaction before accepting it.
* Fixed the operation API routes.
* Fixed too greedy wallet coin selection loop.
Adds checks to ignore a block that's already known (can happen
if the block was saved but the chain update failed). Also saves
saves the chain slightly differently during bootstrap as the
header chain should only be updated when a header is received
(and not when we got a full block). Finally, do not broadcast
during bootstrap.
Block, Output and Kernel now have bitmasks to hold supported
features and eventually versioning. Will make adding features and
updates easier and open the possibility of soft forks.
First added feature for Output and Kernel is the marking of coinbase
related ones. Allows the validation of the coinbase part of a block.
