* Migrate main node store to LMDB
In preparation to using LMDB as a wallet database, migrate the
node db. There's no point in having 2 key-value stores.
In addition LMDB provides a few advantages as a node db, namely a
much faster build (compared to RocksDb), lesser dependencies and
transactions.
* Migrated p2p store to lmdb, stuff compiles
* More fixes, chain tests starting to pass
* Fixed txhashset rollback messing with block save and general batch delimitation. Chain tests passing.
* rustfmt
* LMDB max map size of 10MB isn't really workable. Half TB seems reasonable.
* Fix wallet tests
* Rather crucial commit was missing
* rustfmt
* Fixing new merged tests following lmdb changes
* rustfmt
* * Make txhashset validation read-only on fast sync to avoid having
a really long open transaction.
* Fix deadlock in new block processing, batch should always be
created within a txhashset lock (when they interact).
* Comment about batch and txhashset interlacing
* Fix store tests to use batch
* Externalize wallet config and seed
* Converted direct read access to file outputs map to an iterator
* Cleaned up and simplified wallet Backend trait:
* No more direct mutable access to internal structures (HashMap)
* Batch interface for all writes
* Remove unneeded read wrapper (read_wallet)
* rustfmt
* First (incomplete) pass at wallet LMDB backend
* Progressing on lmdb backent iml
* Added batch impl for LMDB wallet backend. Pretty much done with it, but not sure how to deal with commit (owned).
* rustfmt
* Wrapping LMDB batch around a refcell to work around borrow rules
* Compilation up to grin chain
* only store leaves in rm_log
* cleanup
* commit
* add failing test to cover case where we compact
an already compacted data file
* fix the logic for pruning the data file
* rm_log only contains leaves
prunelist only contains non-leaf subtree roots
* cleanup
* commit
* bitmap impl running in parallel
* rustfmt
* this is better - rewind unremoves outpu pos spent by rewound inputs
* commit
* commit
* pass bitmap around when rewinding
* store and retrieve input bitmap per block from the db
* Replace the RemoveLog with the UTXO set.
* utxo set starting to pass tests
* stuff works...
* split store types out
added some tests to compare the performance of the rm_log and the proposed utxo_set
* pull prune_list out into standalone file
* cleanup, get rid of unused height param
* cleanup and add doc comments
* add support for migrating rm_log to utxo_set
* take snapshot of utxo file during fast sync
implement migration of rm_log -> utxo_set
* rename rewound_copy to snapshot
* fixup pmmr tests to reflect cutoff_pos
* cleanup unused import
* check_compact needs to rewind the utxo_set as appropriate
* fix pool tests
* fixup core tests
* cache block_input_bitmaps via LruCache in store
* cache block header on initial write to db
* rename utxo_set -> leaf_set
and remove references to "spent" in grin_store
* better document the rewind behavior
* Bump up crates versions
* Finally add a Cargo.lock to avoid dependency breakages
* Build doc update for testnet2
* Fix test framework not really using its mining config
* Testnet2 genesis, best so far at 128 difficulty (a nice number)
* Minor build doc update
* adding file position index data accessable to the chain, and allowing for storage of such within db
* missing file
* restart files at last recorded position in stored file metadata
* just use tip to store last pmmr index information
* error handling
* test fix
* added global slog instance, changed all logging macro formats to include logger instance
* adding configuration to logging, allowing for multiple log outputs
* updates to test, changes to build docs
* rustfmt
* moving logging functions into util crate
* Integrate PMMR and its persistent backend with the Chain
* Chain can set tree roots; PMMR backend discard
* Check spent and prune for each input in new block
* Handling of forks by rewinding the state
* More PMMR tests and fixes, mostly around rewind
* Rewrite get_unspent to use the sumtrees, fix remaining compilation issues
* Base MMR storage structures
Implementations of the MMR append-only file structure and its
remove log. The append-only file is backed by a mmap for read
access. The remove log is stored in memory for quick checking
and backed by a simple file to persist it.
* Add PMMR backend buffer, make PMMR Backend mutable
* The Backend trait now has &mut self methods, and an &mut
reference in PMMR. This simplifies the implementation of all
backends by not forcing them to be interior mutable. Slight
drawback is that a backend can't be used directly as long as it's
used by a PMMR instance.
* Introduced a buffer in the PMMR persistent backend to allow
reads before the underlying files are fully flushed. Implemented
with a temporary VecBackend.
* Implement a prune list to use with dense backends
The PruneList is useful when implementing compact backends for a PMMR (for
example a single large byte array or a file). As nodes get pruned and
removed from the backend to free space, the backend will get more compact
but positions of a node within the PMMR will not match positions in the
backend storage anymore. The PruneList accounts for that mismatch and does
the position translation.
* PMMR store compaction
Implement actual pruning of the underlying PMMR storage by
flushing the remove log. This triggers a rewrite of the PMMR nodes
data (hashes and sums), removing pruned nodes. The information of
what has been removed is kept in a prune list and the remove log
is truncated.
* PMMR store pruning tests and fixes
* Adding ability to serialise parts of the header, pre-nonce and post-nonce
* Some test integration of queueing functions in cuckoo-miner
* more cuckoo-miner async mode integration, now more or less working
* integrating async miner workflow
* rocksdb update
* u64 internal difficulty representation, and integration of latest Cuckoo-miner API
* change to cuckoo-miner notify function
* Issue in testing, and if use_async value is None in grin.toml
* making async mode explicit in tests - 2
* fiddle with port numbers for CI
* update tag to ensure cuckoo-miner build doesn't fail on windows
* change the order in which tests are run
* Starting to refactor test, adding http post to wallet sender
* Implemented ability to run servers on different ports (mostly for testing), and implemented ability to post http requests directly to receiving wallets
* Adding detailed instructions on running multiple servers on the same machine
* Changes to build.doc to outline server running process
* Removed unwanted debug statements
* WIP Local server testing framework evolution
* More refactoring of server pool, checked in because there's a problem
* Added server reference structure, and ability to return server references from tests cleanly
* Added simulate_parallel_mining test, which puts some artificial slowdown into test mining loops, and the difficulty can currently be watched in the log
* Removing the ServerRef structure placed in earlier and replaced with a ServerStats structure, that just returns relevant info about the Server state without exposing it to the world
Builds codecs to encode and decode blocks, block headers, transactions, kernels, etc. Will be used by the store and peer-to-peer layer for serialization, but also to compute hashes. Separates out serialization from core.
