* refactor txhashset to only take a single header
no need to pass in current head_header each time
* rustfmt
* fix bug introduced in rewind with LMDB
* rustfmt
* add some comments
* Sort pool tx by fee over weight for mining. Fixes#1105
* Bucketing dependent transactions before weighing. Minor tx weight fixes.
* Limit length of tx chain, cleanup and test fixes
* Cleanup all mining references to a hardcoded tx count
* Small test improvement, cleanup
* Include commitments non-duplicate checks in aggregate
* Remove said check from the pool
* Block building now uses tx aggregation to reduce duplication
* improve: time crate flagged as deprecated, switch to actively maintained chrono crate
* improve: complete the switching from deprecated time crate to chrono crate, for all the tests part
* improve: complete switching to chrono crate, for the left tests in 'chain' and 'wallet'
* Make grin.toml config optional. Mirror exisiting config parameters in grin.toml to source code, so binary can run without a config file. Add test for it.
* fixup! Make grin.toml config optional
* we do not need Merkle proofs to spend coinbase outputs
we only need the output_mmr_size from the block header
* tests working with no Merkle proofs in inputs
* Migrate main node store to LMDB
In preparation to using LMDB as a wallet database, migrate the
node db. There's no point in having 2 key-value stores.
In addition LMDB provides a few advantages as a node db, namely a
much faster build (compared to RocksDb), lesser dependencies and
transactions.
* Migrated p2p store to lmdb, stuff compiles
* More fixes, chain tests starting to pass
* Fixed txhashset rollback messing with block save and general batch delimitation. Chain tests passing.
* rustfmt
* LMDB max map size of 10MB isn't really workable. Half TB seems reasonable.
* Fix wallet tests
* Rather crucial commit was missing
* rustfmt
* Fixing new merged tests following lmdb changes
* rustfmt
* * Make txhashset validation read-only on fast sync to avoid having
a really long open transaction.
* Fix deadlock in new block processing, batch should always be
created within a txhashset lock (when they interact).
* Comment about batch and txhashset interlacing
* Fix store tests to use batch
* Externalize wallet config and seed
* Converted direct read access to file outputs map to an iterator
* Cleaned up and simplified wallet Backend trait:
* No more direct mutable access to internal structures (HashMap)
* Batch interface for all writes
* Remove unneeded read wrapper (read_wallet)
* rustfmt
* First (incomplete) pass at wallet LMDB backend
* Progressing on lmdb backent iml
* Added batch impl for LMDB wallet backend. Pretty much done with it, but not sure how to deal with commit (owned).
* rustfmt
* Wrapping LMDB batch around a refcell to work around borrow rules
* Compilation up to grin chain
* Add kernel commitments sum and kernel and output MMR sizes to block header
* Sum a block without including previous sums, cleanup. Blocks are now summed and validated based on their own totals and not the totals since genesis. This allows to get rid of BlockSum and simplified the setting of a new block's roots, kernel sum and MMR sizes. Fixes#116
* Additional kernel MMR validation to check all prior header roots successively
* Wallet tests fix
* only store leaves in rm_log
* cleanup
* commit
* add failing test to cover case where we compact
an already compacted data file
* fix the logic for pruning the data file
* rm_log only contains leaves
prunelist only contains non-leaf subtree roots
* cleanup
* commit
* bitmap impl running in parallel
* rustfmt
* this is better - rewind unremoves outpu pos spent by rewound inputs
* commit
* commit
* pass bitmap around when rewinding
* store and retrieve input bitmap per block from the db
* Replace the RemoveLog with the UTXO set.
* utxo set starting to pass tests
* stuff works...
* split store types out
added some tests to compare the performance of the rm_log and the proposed utxo_set
* pull prune_list out into standalone file
* cleanup, get rid of unused height param
* cleanup and add doc comments
* add support for migrating rm_log to utxo_set
* take snapshot of utxo file during fast sync
implement migration of rm_log -> utxo_set
* rename rewound_copy to snapshot
* fixup pmmr tests to reflect cutoff_pos
* cleanup unused import
* check_compact needs to rewind the utxo_set as appropriate
* fix pool tests
* fixup core tests
* cache block_input_bitmaps via LruCache in store
* cache block header on initial write to db
* rename utxo_set -> leaf_set
and remove references to "spent" in grin_store
* better document the rewind behavior
* First pass at restructuring the keychain crate and introducing a Keychain trait
* Parameterized everything that had to. Stuff compiles.
* More stuff compiles, fix most tests
* Big merge, pushing down opening the keychain forced adding factory methods on trait
* Test fixes for pool and servers crate
* verify a tx like we verify a block (experimental)
* first minimal_pool test up and running but not testing what we need to
* rework tx_pool validation to use txhashset extension
* minimal tx pool wired up but rough
* works locally (rough statew though)
delete "legacy" pool and graph code
* rework the new pool into TransactionPool and Pool impls
* rework pool to store pool entries
with associated timer and source etc.
* all_transactions
* extra_txs so we can validate stempool against existing txpool
* rework reconcile_block
* txhashset apply_raw_tx can now rewind to a checkpoint (prev raw tx)
* wip - txhashset tx tests
* more flexible rewind on MMRs
* add tests to cover apply_raw_txs on txhashset extension
* add_to_stempool and add_to_txpool
* deaggregate multi kernel tx when adding to txpoool
* handle freshness in stempool
handle propagation of stempool txs via dandelion monitor
* patience timer and fluff if we cannot propagate
to next relay
* aggregate and fluff stempool is we have no relay
* refactor coinbase maturity
* rewrote basic tx pool tests to use a real txhashset via chain adapter
* rework dandelion monitor to reflect recent discussion
works locally but needs a cleanup
* refactor dandelion_monitor - split out phases
* more pool test coverage
* remove old test code from pool (still wip)
* block_building and block_reconciliation tests
* tracked down chain test failure...
* fix test_coinbase_maturity
* dandelion_monitor now runs...
* refactor dandelion config, shared across p2p and pool components
* fix pool tests with new config
* fix p2p tests
* rework tx pool to deal with duplicate commitments (testnet2 limitation)
* cleanup and address some PR feedback
* add big comment about pre_tx...
* beginning to refactor keychain into wallet lib
* rustfmt
* more refactor of aggsig lib, simplify aggsig context manager, hold instance statically for now
* clean some warnings
* clean some warnings
* fix wallet send test a bit
* fix core tests, move wallet dependent tests into integration tests
* repair chain tests
* refactor/fix pool tests
* fix wallet tests, moved from keychain
* add wallet tests
* Test multi kernel deaggregation
* Add aggregate without cut_through and deaggregate function
* Add deaggregate function in pool and test
* Rustfmt
* Add deaggregate_and_add_to_memory_pool
* Deaggregate regular multi kernel transaction by default
* Rustfmt
* Add error type faileddeaggregation
* Add find candidates function
* Rustfmt
* Use intersection of sets instead of for comparisons
* Rustfmt
* Removed unnecessary if
* Stricter verification with is_subset
* Rustfmt
This PR adresses #166
Error handling in wallet was ported to failure in https://github.com/mimblewimble/grin/pull/713
Using the same error model makes wallet code simpler and may simplify migration to Hyper.
* Bump up crates versions
* Finally add a Cargo.lock to avoid dependency breakages
* Build doc update for testnet2
* Fix test framework not really using its mining config
* Testnet2 genesis, best so far at 128 difficulty (a nice number)
* Minor build doc update
* adding appropriate message to bulletproofs to allow for restore
* rustfmt
* should work, now test
* rustfmt
* fix to wallet restore, works now
* fix pool tests
* fix pool tests
* rustfmt
* Initial Dandelion Commit
* Changed stem_tx_pool to tx_stempool
* Introduction of stem memory pool and stem pool config
* Pool push now send to stem memory pool
* Add stem transaction functions
* Add stem transaction pool
* Drastically simplified code structure
* Add monitor transactions
* Add Dandelion monitor and remove transactions from stempool
* Add peer relay monitor
* Reconcile block with stempool
* Fix total size bug
* Add fluff option for pool push
* Added details on dandelion monitor
* Fix issue with missing parent
* Child transaction with stempool parent are now forced stem
* Update Dandelion Relay from outgoing peers
* Fix missing pool reconciliation
* Add the ability to fluff a transaction directly
* Fix tests for Dandelion
* Missing send_stem_transaction method...
* Add fluff handler for wallet
* Add logger when successfully updated Dandelion relay
* Launch transaction monitor last
* Fix dandelion relay misplaced
* Add logging and updating for stempool
* Additionnal check for stem transaction
* Fix 2 Locks in a row
* family_branch() to recursively call family() up the branch
todo
- we hit a peak, then we need to get to the root somehow
- actually get the hashes to build the proof
* wip
* some additional testing around merkle tree branches
* track left/right branch for each sibling as we build the merkle path up
* MerkleProof and basic (incomplete) verify fn
* I think a MerkleProof verifies correctly now
need to test on test case with multiple peaks
* basic pmmr merkle proof working
* MerkleProof now serializable/deserializable
* coinbase maturity via merkle proof basically working
* ser/deser merkle proof into hex in api and wallet.dat
* cleanup
* wip - temporarily saving merkle proofs to the commit index
* assert merkle proof in store matches the rewound version
there are cases where it does not...
* commit
* commit
* can successfully rewind the output PMMR and generate a Merkle proof
need to fix the tests up now
and cleanup the code
and add docs for functions etc.
* core tests passing
* fixup chain tests using merkle proofs
* pool tests working with merkle proofs
* api tests working with merkle proof
* fix the broken comapct block hashing behavior
made nonce for short_ids explicit to help with this
* cleanup and comment as necessary
* cleanup variety of TODOs
* beginning to add bullet proof messages
* Updated core transaction creation to embed the output's value and switch commit hash as part of the rangeproof message
* formatting issue
* more formatting issues
* Removing conditional feature compliation.. just bulletproofs from now on
* ensure MAX_PROOF_SIZE uses bulletproof sizing instead of earlier version
* updated with switch commit committed to in extra data
* accidentally commented out bullet-proof-size feature
* wip - basic tx pool lookup for kernel short ids (compact block hydration)
* use the nonce in the compact_block to correctly generate short_ids for lookup
* query the tx pool based on kernel short_ids
* tests passing
* cleanup some logging
* cleanup logging
* WIP - split the key in final tx step
store "offset" on transaction itself
* rebase
* commit
* tx with offset
* got a test tx validating successfully using a sig from a split key and the appropriate offset
* sum up the offset for the block_header
* fix size tests for blocks and compact blocks (header now includes offset)
* use txs with offsets in most of the core tests
some tests now failing
* build kernel from k1G (k2 stored on tx, sum stored on header)
* commit
* tx now has vec of kernels
rework tx and kernel validation
* add test for tx cut_through
* wip - working on splitting in aggsig
* split the key when creating the initial sender aggsig context
* cleanup
* cleanup
* code needs claning up but split keys working for sender/receiver aggsig flow
* cleanup debug logging
* fix tests
* fix merge and basic cleanup
* fix keychain tests to use new tx_id
* Util to zip and unzip directories
* First pass at sumtree request/response. Add message types, implement the exchange in the protocol, zip up the sumtree directory and stream the file over, with necessary adapter hooks.
* Implement the sumtree archive receive logicGets the sumtree archive data stream from the network and write it to a file. Unzip the file, place it at the right spot and reconstruct the sumtree data structure, rewinding where to the right spot.
* Sumtree hash structure validation
* Simplify sumtree backend buffering logic. The backend for a sumtree has to implement some in-memory buffering logic to provide a commit/rollback interface. The backend itself is an aggregate of 3 underlying storages (an append only file, a remove log and a skip list). The buffering was previously implemented both by the backend and some of the underlying storages. Now pushing back all buffering logic to the storages to keep the backend simpler.
* Add kernel append only store file to sumtrees. The chain sumtrees structure now also saves all kernels to a dedicated file. As that storage is implemented by the append only file wrapper, it's also rewind-aware.
* Full state validation. Checks that:
- MMRs are sane (hash and sum each node)
- Tree roots match the corresponding header
- Kernel signatures are valid
- Sum of all kernel excesses equals the sum of UTXO commitments
minus the supply
* Fast sync handoff to body sync. Once the fast-sync state is fully setup, get bacj in body sync
mode to get the full bodies of the last blocks we're missing.
* First fully working fast sync
* Facility in p2p conn to deal with attachments (raw binary after message).
* Re-introduced sumtree send and receive message handling using the above.
* Fixed test and finished updating all required db state after sumtree validation.
* Massaged a little bit the pipeline orphan check to still work after the new sumtrees have been setup.
* Various cleanup. Consolidated fast sync and full sync into a single function as they're very similar. Proper conditions to trigger a sumtree request and some checks on receiving it.
* Tried but failed to fix `cargo build` complaint about unused #[macro use]. See also 7a803a8dc1
* Compiler complaints be-gone
* Give sumtree tests method-tagged folder names so they don't overwrite each others' files
Fixes#658
* large updates for mining, fix async mode, will list changes in PR
* reset config and build defaults
* change to difficulty calculations
* tweaking mining params and tests
* tweaking to tests
* including pre-genesis data for difficulty adjustments, adding adjustment scenario tests
* further clarifying next_difficulty function
* moving tests out of consensus.rs
* pow test fix
* changing pre-genesis generation
* experiment with lock_heights on outputs
* playing around with lock_height as part of the switch commitment hash
* cleanup
* include features in the switch commit hash key
* commit
* rebase off master
* commit
* cleanup
* missing docs
* rework coinbase maturity test to build valid tx
* pool and chain tests passing (inputs have switch commitments)
* commit
* cleanup
* check inputs spending coinbase outputs have valid lock_heights
* wip - got it building (tests still failing)
* use zero key for non coinbase switch commit hash
* fees and height wrong order...
* send output lock_height over to wallet via api
* no more header by height index
workaround this for wallet refresh and wallet restore
* refresh heights for unspent wallet outputs where missing
* TODO - might be slow?
* simplify - do not pass around lock_height for non coinbase outputs
* commit
* fix tests after merge
* build input vs coinbase_input
switch commit hash key encodes lock_height
cleanup output by commit index (currently broken...)
* is_unspent and get_unspent cleanup - we have no outputs, only switch_commit_hashes
* separate concept of utxo vs output in the api
utxos come from the sumtrees (and only the sumtrees, limited info)
outputs come from blocks (and we need to look them up via block height)
* cleanup
* better api support for block outputs with range proofs
* basic wallet operations appear to work
restore is not working fully
refresh refreshes heights correctly (at least appears to)
* wallet refresh and wallet restore appear to be working now
* fix core tests
* fix some mine_simple_chain tests
* fixup chain tests
* rework so pool tests pass
* wallet restore now safely habndles duplicate commitments (reused wallet keys)
for coinbase outputs where lock_height is _very_ important
* wip
* validate_coinbase_maturity
got things building
tests are failing
* lite vs full versions of is_unspent
* builds and working locally
zero-conf - what to do here?
* handle zero-conf edge case (use latest block)
* introduce OutputIdentifier, avoid leaking SumCommit everywhere
* fix the bad merge
* pool verifies coinbase maturity via is_matured
this uses sumtree in a consistent way
* cleanup
* add docs, cleanup build warnings
* fix core tests
* fix chain tests
* fix pool tests
* cleanup debug logging that we no longer need
* make out_block optional on an input (only care about it for spending coinbase outputs)
* cleanup
* bump the build
* Make total_difficulty the sum of network difficulty, not whatever the miner happened to mine.
- Only for Testnet2 / Mainnet (hardforks Testnet1)
- update chain::pipe validate_header to validate according to Testnet2 rules for cumulative difference
Fixes#280
* tests that should ignore (network) difficulty
* fn new_block is explained as "utility to create a block without worrying about the key or previous header" so it gets network difficulty := 0 too
* update tx pool tests (going with "minimum" for network difficulty for now)
* add ERR outputs about bannable offences
(#406 should know about these)
* whitespace fix
* mine_simple_chain: Probably DON'T overwrite difficulty (?)
* core/mod tests "reward_empty_block" and "reward_with_tx_block" tests set to use lowest network difficulty possible
* First steps converting transaction workflow to be aggsig-enable
* integrating updated version of aggsig, which gives greater control over the contents of e
* added wallet transaction test to testing framework to enable testing the whole thing, completed interaction as far as inital response from recipient
* more aggsig work, final signature is produced now
* Construction of aggsig transaction now working to the point of the signature being built
* aggsig transactions working end-to-end in the nominal case
* refactor aggsig verify from commit and fix some tests
* more cleanup and test fixing
* cleaning up automated tests
* test+formatting fix
* Very quick peer banning endpoint, helps with #406
* Ping heights (#407)
* add height to ping/ping
* reformat output
* fix p2p test
* Fix orphan handling, not related to current head. Fixes#412
* Check before borrow, fixes#267
* Not finding an output commit in pos is an AlreadySpent
* Fix race condition, sending before conn is ready
* Explicit error for unknown pos of a forked block
* Remove config outdated tests. Fix#333
* Check ref and try before borrow, fix#400
* We do not want to sync with old peers anyway
* Hide cargo compiler warning for unused NoopAdapter and unused test code. Add TODOs
Renamed mining parameter mode to chain type, with existing types
of CI testing, user testing, testnet1 or mainnet. The public
chains (testnet1 and mainnet) come with their fully pre-defined
genesis block.
Still need to set the nonce and cycle for testnet1 genesis.
Moved the HTTP APIs away from the REST endpoint abstraction and
to simpler Hyper handlers. Re-established all routes as v1.
Changed wallet receiver port to 13415 to avoid a gap in port
numbers.
Finally, rustfmt seems to have ignored specific files arguments,
running on everything.
* failing test case that exercises
prepare_mineable_transactions and reconcile_block
and highlights what appears to be unexpected behavior
* adjust the failing test - the failure state is where we have
a tx in the pool *but* it is not in the list of roots in the pool
* zero confirmation txs are now working
introduce update_roots to graph to ensure pool is consistent
after calling remove_pool_transaction
* move update_roots to sweep_transactions so we only call it once
rework update_roots to be more efficient
* use HashSet in reconcile_block to avoid marking txs multiple times
* use HashSet and not HashMap
return from mark_transaction early if already seen tx
* Adding switch commit to grin outputs
* logging output fix
* adding switch commitment hash to sum tree node
* added hash_with to Hashed trait, to allow for hashing to include another writeable element
* adding hash_with as method in hashed trait
* rename pubkey -> key_id, these are key identifiers, not public keys
* reduce identifier seize to 10 bytes, get rid of fingerprints (identifiers are now small enough to use as-is)
* IDENTIFIER_SIZE const
* add FeeDispute error for when sender and recipient disagre on fee calculation (should never happen)
* does not need to be mut
* cleaned up some warnings
* added global slog instance, changed all logging macro formats to include logger instance
* adding configuration to logging, allowing for multiple log outputs
* updates to test, changes to build docs
* rustfmt
* moving logging functions into util crate
* add lock_height to Transaction and TxKernel, sign msg including both fee and lock_height in kernel
* make the order of the fields in tx and kernel more consistent
* rename to kernel_sig_msg
* add test to cover adding timelocked tx to pool, where tx is invalid based on current height of the blockchain
* add tests for adding timelocked txs to blocks (valid and otherwise)
Configuration for a minum accept fee base for the transaction
pool. The base is multipled by a weight computed from the
transaction number of inputs, outputs and kernels. The transaction
fee is required to always be larger than the weight times the
base.
min_fee = base * (-1*input_len + 4*output_len + kernel_len)
The weight is set to never be less than one.
Also added a configurable (and fairly naive for now) max pool
capacity in number of transactions.
* store wallet output data in hashmap
* cleanup up commented out code
* pass pubkey/identifier and not derivation around to miner etc.
* fix failing tests
* introduce grin_keychain, encapsulate derivation of secret_keys
* core compiles against keychain, tests don't run yet
* core tests are now passing against keychain
* wip - getting wallet working with keychain
* add util and keychain to travis test matrix
* basic test around key derivation
* Integrate PMMR and its persistent backend with the Chain
* Chain can set tree roots; PMMR backend discard
* Check spent and prune for each input in new block
* Handling of forks by rewinding the state
* More PMMR tests and fixes, mostly around rewind
* Rewrite get_unspent to use the sumtrees, fix remaining compilation issues
* workaround the "commitment is not a public key" problem by using to_two_keys() from rust-secp256k1-zkp
* remove secp256k1zkp, update the various Cargo.toml to use rust-secp256k1-zkp from external git repo, update code to not rely on to_pubkey (and use verify_from_commit)
* secp256k1zkp from git
* cleanup secp256k1 dependencies
* gitignore node1 and node2 dirs
* use head_header in add_to_memory_pool
* add COINBASE_MATURITY const to consensus
* add coinbase maturity (wip) validaton rule to validate_block
* add coinbase maturity check to validate_block
* map errors in adapters - specific errors still wip
* reworked so adapter translates chain errors to pool errors (core errors not required)
* add test for spending immature coinbase in memory pool
* wip - add test to cover spending coinbase output in chain.process_block
* added test coverage around process_block - we have a problem with coinbase output commitments
* add a comment on the failing test
* process_block will now fail validation if we attempt to spend coinbase that has not yet matured
(remember to use a new reward_key for every block). test coverage in place to verify this
* minor cleanup - unused imports
* cleanup build warnings - unused vars
* make structs pub to get rid of the private_in_public lint warning
* missing docs on RangeProof
* add missing docs to store delete function
* cleaned up deprecation warning -
tokio_core -> tokio_io
complete() -> send()
Operations on the wallet data file are now fenced by a lock to
avoid potentially messy concurrent modifications by multiple
processes (i.e. the wallet receiver and a send command). The lock
is done using a create-only lock file, which is an atomic
operation.
Fixes a few loose ends in the full cycle of sending coins,
accepting them, pushing that transaction to the pool and having it
mined. More specifically:
* The API output endpoint needs to be a UTXO endpoint, as the
server can't make any guarantee about having a spent output.
* Bubbling up HTTP not found errors.
* Wallet output status checker now handles spent outputs.
* Transaction pool validates the transaction before accepting it.
* Fixed the operation API routes.
* Fixed too greedy wallet coin selection loop.
* Transactions coming from the network are now pushed to the pool
through the net adapter.
* New blocks accepted by the chain are sent to the pool for
eviction.
* The miner requests transactions from the pool to build its
blocks.
* The push API adds to the pool, removing the mock.
* Implementation of the adapter to the chain required by the pool
to get consistent UTXOs. Grossly unoptimized until we have the UTXO
MMR ready.
Introduced new non-dummy trait for the blockchain as seen from the
pool that just produces a UTXO. Made to pool parametric on that
trait to get rid of the Box wrapper and still allow the test
implementation.
* Beginning work on pool design doc
* Refining data structures; adding connect capability
* Fleshing out the connectivity paths for the tx pool
* Bringing tx pool and orphan set add logic up into parent TransactionPool
* Use output's commitment as identifier in graph structures
* Breaking a bunch of stuff to start migration to output commitment as id instead of hash
* Wrapping up updates to pool using commitment keys, dummy blockchain. Contains lots of cleanup on the internal flow.
* Beginning work on new block reconciliation
* WIP: Replacing monolithic pool cleanup with mark-and-sweep, which greatly simplifies the logic.
* Laying the groundwork for pool tests; test tx generator
* WIP: More elaborate test helpers; starting work on more elaborate block acceptance test.
* Need DummyUtxoSet to actually apply blocks now
* Using search_for_best_output to validate output status in test_basic_pool_add
* Enable modification of chain while under shared pool ownership. Cleanup pending
* WIP: Begining to untangle the TransactionPool impl from Pool and Orphans data structures
* Finishing refactoring of pool block reconciliaition; getting tests working again
* Add metrics for graph sizes; prereq to pool size throttling
* Remove redundant search_for_available_output from pool graph container
* Minimum viable block builder: return all fully rooted txs
* Tests for block building procedure
* Delegate duplicate output checking to check_duplicate_outputs
* Delegate orphan reference resolution to resolve_orphan_refs
