* Migrate main node store to LMDB
In preparation to using LMDB as a wallet database, migrate the
node db. There's no point in having 2 key-value stores.
In addition LMDB provides a few advantages as a node db, namely a
much faster build (compared to RocksDb), lesser dependencies and
transactions.
* Migrated p2p store to lmdb, stuff compiles
* More fixes, chain tests starting to pass
* Fixed txhashset rollback messing with block save and general batch delimitation. Chain tests passing.
* rustfmt
* LMDB max map size of 10MB isn't really workable. Half TB seems reasonable.
* Fix wallet tests
* Rather crucial commit was missing
* rustfmt
* Fixing new merged tests following lmdb changes
* rustfmt
* * Make txhashset validation read-only on fast sync to avoid having
a really long open transaction.
* Fix deadlock in new block processing, batch should always be
created within a txhashset lock (when they interact).
* Comment about batch and txhashset interlacing
* Fix store tests to use batch
* Externalize wallet config and seed
* Converted direct read access to file outputs map to an iterator
* Cleaned up and simplified wallet Backend trait:
* No more direct mutable access to internal structures (HashMap)
* Batch interface for all writes
* Remove unneeded read wrapper (read_wallet)
* rustfmt
* First (incomplete) pass at wallet LMDB backend
* Progressing on lmdb backent iml
* Added batch impl for LMDB wallet backend. Pretty much done with it, but not sure how to deal with commit (owned).
* rustfmt
* Wrapping LMDB batch around a refcell to work around borrow rules
* Compilation up to grin chain
* Add kernel commitments sum and kernel and output MMR sizes to block header
* Sum a block without including previous sums, cleanup. Blocks are now summed and validated based on their own totals and not the totals since genesis. This allows to get rid of BlockSum and simplified the setting of a new block's roots, kernel sum and MMR sizes. Fixes#116
* Additional kernel MMR validation to check all prior header roots successively
* Wallet tests fix
* First pass at restructuring the keychain crate and introducing a Keychain trait
* Parameterized everything that had to. Stuff compiles.
* More stuff compiles, fix most tests
* Big merge, pushing down opening the keychain forced adding factory methods on trait
* Test fixes for pool and servers crate
* move http calls out from libwallet internal
* rustfmt
* start to think about wallet communication traits
* rustfmt
* start of factoring out wallet client trait
* rustfmt
* move node_url trait fn into walletclient
* rustfmt
* comms factored out (with exception of wallet restore)
* rustfmt
* fix test
* rustfmt
* further test fix
Small refactoring of one folder, if it makes sense I could extend the scope.
* Remove some cloning (real and just verbosity in the code)
* Naming conventions like to/into*
* Some Clippy's suggestions
I found that we don't use field init shorthand syntax, so I didn't touch this part, was it discussed before?
* move checker and rename to updater
* rustfmt
* complete checker/updater move
* rustfmt
* move libwallet error into separate file
* rustfmt
* starting to sort our error types
* updating errors in libtx and libwallet
* rustfmt
* factor out error type
* rustfmt
* compiling, errors split into libwallet and wallet errors
* rustfmt
* changing user error reporting to new format
* rustfmt
* clean up error types
* clean up error types
* move restore into libwallet
* rustfmt
* verify a tx like we verify a block (experimental)
* first minimal_pool test up and running but not testing what we need to
* rework tx_pool validation to use txhashset extension
* minimal tx pool wired up but rough
* works locally (rough statew though)
delete "legacy" pool and graph code
* rework the new pool into TransactionPool and Pool impls
* rework pool to store pool entries
with associated timer and source etc.
* all_transactions
* extra_txs so we can validate stempool against existing txpool
* rework reconcile_block
* txhashset apply_raw_tx can now rewind to a checkpoint (prev raw tx)
* wip - txhashset tx tests
* more flexible rewind on MMRs
* add tests to cover apply_raw_txs on txhashset extension
* add_to_stempool and add_to_txpool
* deaggregate multi kernel tx when adding to txpoool
* handle freshness in stempool
handle propagation of stempool txs via dandelion monitor
* patience timer and fluff if we cannot propagate
to next relay
* aggregate and fluff stempool is we have no relay
* refactor coinbase maturity
* rewrote basic tx pool tests to use a real txhashset via chain adapter
* rework dandelion monitor to reflect recent discussion
works locally but needs a cleanup
* refactor dandelion_monitor - split out phases
* more pool test coverage
* remove old test code from pool (still wip)
* block_building and block_reconciliation tests
* tracked down chain test failure...
* fix test_coinbase_maturity
* dandelion_monitor now runs...
* refactor dandelion config, shared across p2p and pool components
* fix pool tests with new config
* fix p2p tests
* rework tx pool to deal with duplicate commitments (testnet2 limitation)
* cleanup and address some PR feedback
* add big comment about pre_tx...
* remove context object from aggsig and transaction libs
* fix to aggsig, and remove unnecessary warnings
* put tx_fee function into libwallet::transaction
* Error cleanup, and creating libwallet error type
* remove some unwraps
* checker bug
* ensure transaction tests checks sender's wallet
* Beginning to rework aggsig library workflow
* more refactoring of transaction api
* whoever does round 1 first creates offset
* slate finalisation now context-free, so anyone can do it
* remove concept of transaction phase
* remove slate phase enum
* update actual send/receive code with new transaction lib workflow
* refactoring transaction building code
* serialise return transaction
* move shared functions into transactions, ensure wallet manipulation is only done outside of aggsig transaction lib
* remove unneeded wallet config from fn
* adding test functions to facilitate libwallet transaction testing
* rustfmt
* refactoring checker somewhat, adding ability to create and transactions against local copy of chain for simpler testing
* finish transaction testing functionality which verifies transactions work properly
* Remove wallet output manipulation from transaction building lib
* ensure sender expects full transaction back on last phase
* ensure sender expects full transaction back on last phase
* beginning to refactor keychain into wallet lib
* rustfmt
* more refactor of aggsig lib, simplify aggsig context manager, hold instance statically for now
* clean some warnings
* clean some warnings
* fix wallet send test a bit
* fix core tests, move wallet dependent tests into integration tests
* repair chain tests
* refactor/fix pool tests
* fix wallet tests, moved from keychain
* add wallet tests
* block sums and reworked block validation
read and write block_sums
refactor validate on both block and txhashset
write block_sum on fast sync
we store the kernel_sum (need to account for the offset)
* block_sums
* rustfmt
* cleanup
* mods to speed up restore a bit
* mods to speed up restore a bit
* performance improvements to wallet restore and split large server queries into multiple
It looks like lockfile implementation did still suffer from a race
condition. Only when creating the file with O_EXCL, file creation fails
if the file does already exist. One could use O_EXCL, but Windows
might use another flag to achieve the same. Moreover, O_EXCL doesn't
work as expected if the files are accessed over NFS.
In contrast, mkdir() is atomic in every of the mentioned cases.
Aside from using a lockdirectory, this patch also backups the current
wallet.dat contents to wallet.bck in case writing a possibly hefty
amount of JSON to the new wallet.dat fails. It is hoped that at least
one of the .bck and .dat files has usable contents in case failure.
This PR adresses #166
Error handling in wallet was ported to failure in https://github.com/mimblewimble/grin/pull/713
Using the same error model makes wallet code simpler and may simplify migration to Hyper.
* update pmmr to get batch of elements by insertion position
* update pmmr to get batch of elements by insertion position
* add api + chain calls to get traversed outputs back out
* add api + chain calls to get traversed outputs back out
* first pass getting wallet restore to work again with updated utxo-walking api
* documentation and spelling
* unused values in tests; just .unwrap()
* unneeded macro_use (router)
* unneeded macro_use and use-imports
* unused value -> .unwrap
* unused variables
* clarify that " is required (see 107ea3c4dd)
* first steps to dig into #933 test_p2p issues
* less simulnet build warnings
* Bump up crates versions
* Finally add a Cargo.lock to avoid dependency breakages
* Build doc update for testnet2
* Fix test framework not really using its mining config
* Testnet2 genesis, best so far at 128 difficulty (a nice number)
* Minor build doc update
* move some debug! to trace!
* more informative debugs
* standardising on always showing chain tips as "cumulative difficulty @ height [hash]"
* make 2 debug outputs into a single
* "no peers" as warning (not info) to let it stand out more clearly
* move fn param (used only in this one debug line)
* clarify difficulty "units"
* adding appropriate message to bulletproofs to allow for restore
* rustfmt
* should work, now test
* rustfmt
* fix to wallet restore, works now
* fix pool tests
* fix pool tests
* rustfmt
* Initial Dandelion Commit
* Changed stem_tx_pool to tx_stempool
* Introduction of stem memory pool and stem pool config
* Pool push now send to stem memory pool
* Add stem transaction functions
* Add stem transaction pool
* Drastically simplified code structure
* Add monitor transactions
* Add Dandelion monitor and remove transactions from stempool
* Add peer relay monitor
* Reconcile block with stempool
* Fix total size bug
* Add fluff option for pool push
* Added details on dandelion monitor
* Fix issue with missing parent
* Child transaction with stempool parent are now forced stem
* Update Dandelion Relay from outgoing peers
* Fix missing pool reconciliation
* Add the ability to fluff a transaction directly
* Fix tests for Dandelion
* Missing send_stem_transaction method...
* Add fluff handler for wallet
* Add logger when successfully updated Dandelion relay
* Launch transaction monitor last
* Fix dandelion relay misplaced
* Add logging and updating for stempool
* Additionnal check for stem transaction
* Fix 2 Locks in a row
* wip
* failing test for being too eager when pruning a sibling
* commit
* rustfmt
* [WIP] modified get_shift and get_leaf_shift to account for leaving "pruned but not compacted" leaves in place
Note: this currently breaks check_compact as nothing else is aware of the modified behavior
* rustfmt
* commit
* rustfmt
* basic prune/compact/shift working
* rustfmt
* commit
* rustfmt
* next_pruned_idx working (I think)
* commit
* horizon test uncovered some subtle issues - wip
* rustfmt
* cleanup
* rustfmt
* commit
* cleanup
* cleanup
* commit
* rustfmt
* contains -> binary_search
* rustfmt
* no need for height==0 special case
* wip - works for single compact, 2nd one breaks the mmr hashes
* commit
* rustfmt
* fixed it (needs a lot of cleanup)
we were not traversing all the way up to the peak if we pruned an entire tree
so rm_log and prune list were inconsistent
* multiple compact steps are working
data file not being copmacted currently (still to investigate)
* cleanup store tests
* cleanup
* cleanup up debug
* rustfmt
* take kernel offsets into account when summing kernels and outputs for full txhashset validation
validate chain state pre and post compaction
* rustfmt
* fix wallet refresh (we need block height to be refreshed on non-coinbase outputs)
otherwise we cannot spend them...
* rustfmt
* family_branch() to recursively call family() up the branch
todo
- we hit a peak, then we need to get to the root somehow
- actually get the hashes to build the proof
* wip
* some additional testing around merkle tree branches
* track left/right branch for each sibling as we build the merkle path up
* MerkleProof and basic (incomplete) verify fn
* I think a MerkleProof verifies correctly now
need to test on test case with multiple peaks
* basic pmmr merkle proof working
* MerkleProof now serializable/deserializable
* coinbase maturity via merkle proof basically working
* ser/deser merkle proof into hex in api and wallet.dat
* cleanup
* wip - temporarily saving merkle proofs to the commit index
* assert merkle proof in store matches the rewound version
there are cases where it does not...
* commit
* commit
* can successfully rewind the output PMMR and generate a Merkle proof
need to fix the tests up now
and cleanup the code
and add docs for functions etc.
* core tests passing
* fixup chain tests using merkle proofs
* pool tests working with merkle proofs
* api tests working with merkle proof
* fix the broken comapct block hashing behavior
made nonce for short_ids explicit to help with this
* cleanup and comment as necessary
* cleanup variety of TODOs
* Initial version
* store failure parameters inside ErrorKind variants
* continue failure transformation
* 4 errors left
* still two errors
* return old code back
* finally compiling
* Fix compilation and test errors after merge
* WIP - split the key in final tx step
store "offset" on transaction itself
* rebase
* commit
* tx with offset
* got a test tx validating successfully using a sig from a split key and the appropriate offset
* sum up the offset for the block_header
* fix size tests for blocks and compact blocks (header now includes offset)
* use txs with offsets in most of the core tests
some tests now failing
* build kernel from k1G (k2 stored on tx, sum stored on header)
* commit
* tx now has vec of kernels
rework tx and kernel validation
* add test for tx cut_through
* wip - working on splitting in aggsig
* split the key when creating the initial sender aggsig context
* cleanup
* cleanup
* code needs claning up but split keys working for sender/receiver aggsig flow
* cleanup debug logging
* fix tests
* fix merge and basic cleanup
* fix keychain tests to use new tx_id
* update mean cuda miner to latest trompcode, and added tweakable parameters to grin configuration file
* Added UUID for transactions, and store aggsig contexts indexed by transaction ID
* updating test framework to allow checking of wallet contents during test
* experiment with lock_heights on outputs
* playing around with lock_height as part of the switch commitment hash
* cleanup
* include features in the switch commit hash key
* commit
* rebase off master
* commit
* cleanup
* missing docs
* rework coinbase maturity test to build valid tx
* pool and chain tests passing (inputs have switch commitments)
* commit
* cleanup
* check inputs spending coinbase outputs have valid lock_heights
* wip - got it building (tests still failing)
* use zero key for non coinbase switch commit hash
* fees and height wrong order...
* send output lock_height over to wallet via api
* no more header by height index
workaround this for wallet refresh and wallet restore
* refresh heights for unspent wallet outputs where missing
* TODO - might be slow?
* simplify - do not pass around lock_height for non coinbase outputs
* commit
* fix tests after merge
* build input vs coinbase_input
switch commit hash key encodes lock_height
cleanup output by commit index (currently broken...)
* is_unspent and get_unspent cleanup - we have no outputs, only switch_commit_hashes
* separate concept of utxo vs output in the api
utxos come from the sumtrees (and only the sumtrees, limited info)
outputs come from blocks (and we need to look them up via block height)
* cleanup
* better api support for block outputs with range proofs
* basic wallet operations appear to work
restore is not working fully
refresh refreshes heights correctly (at least appears to)
* wallet refresh and wallet restore appear to be working now
* fix core tests
* fix some mine_simple_chain tests
* fixup chain tests
* rework so pool tests pass
* wallet restore now safely habndles duplicate commitments (reused wallet keys)
for coinbase outputs where lock_height is _very_ important
* wip
* validate_coinbase_maturity
got things building
tests are failing
* lite vs full versions of is_unspent
* builds and working locally
zero-conf - what to do here?
* handle zero-conf edge case (use latest block)
* introduce OutputIdentifier, avoid leaking SumCommit everywhere
* fix the bad merge
* pool verifies coinbase maturity via is_matured
this uses sumtree in a consistent way
* cleanup
* add docs, cleanup build warnings
* fix core tests
* fix chain tests
* fix pool tests
* cleanup debug logging that we no longer need
* make out_block optional on an input (only care about it for spending coinbase outputs)
* cleanup
* bump the build
* better error messages when `wallet restore` fails
* wallet info: show height, and where we got the height number from
* make "grin wallet listen" show some more lifesigns, so users can know it's running fine
* First steps converting transaction workflow to be aggsig-enable
* integrating updated version of aggsig, which gives greater control over the contents of e
* added wallet transaction test to testing framework to enable testing the whole thing, completed interaction as far as inital response from recipient
* more aggsig work, final signature is produced now
* Construction of aggsig transaction now working to the point of the signature being built
* aggsig transactions working end-to-end in the nominal case
* refactor aggsig verify from commit and fix some tests
* more cleanup and test fixing
* cleaning up automated tests
* test+formatting fix
* distinguish select (among futures) from select coins. Regex search in project for select\b shows we hardly use select, but maybe could use it to add timeouts more cleanly. ("Want to add a timeout to any future? Just do a select of that future and a timeout future!" from https://aturon.github.io/blog/2016/08/11/futures/)
* remove a trailing space
* FAQ.md - fix typo
* wallet: display problematic tx
* update FAQ build troubleshooting to cover #443
* stdout_log_level = Info
file_log_level = Debug
* sync: show total diff @ height when syncronization is completed
* better wallet send dest format error
* move INFO "Client conn ... lost" and "Connected to peer" down to Debug
* move some level=Info to Debug, add 1000-block outputs
* Make Get coinbase via wallet API error more self-help friendly. Also show the wallet API URL that failed to respond.
* spelling and rustfmt nit
* more informational output from `grin server stop`
* newcomer friendly error when wallet.seed isn't found
* grin wallet (info|outputs): better error message
* beginning to add wallet restore... api endpoints and basic restore
* basic restore working, still missing features
* rustfmt
* large speed up to output search, should be more or less working
* properly mark coinbase status
* allow selecting a commit while providing a key index
* misnamed variable
* added static reference to libsecp that can be called throughout
* don't serialise rangeproof to json if it's not desired
* forgotten new file
* amounts input and displayed in wallet are now in full grins, with optional decimal place
* rustfmt
* merge branch
* better acknowledgement of transaction being sent
* allow selecting a commit while providing a key index
* added static reference to libsecp that can be called throughout
* don't serialise rangeproof to json if it's not desired
* default wallet receive to listen on 127.0.0.1, optional flag on wallet command to listen on 0.0.0.0
* fix simulnet for new wallet port/interface config
Moved the HTTP APIs away from the REST endpoint abstraction and
to simpler Hyper handlers. Re-established all routes as v1.
Changed wallet receiver port to 13415 to avoid a gap in port
numbers.
Finally, rustfmt seems to have ignored specific files arguments,
running on everything.
* mount v2 router for flexibility, wallet checker now refreshes multiple outputs via single api call
* fix the api router
* wallet api handlers, miner uses wallet_client
* retry logic via tokio_retry, miner creates new coinbase output via wallet API (retries several times)
* move wallet client into wallet crateand rework the lock acquisition logic to use tokio_retry
* mount v2 router for flexibility, wallet checker now refreshes multiple outputs via single api call
* add read_wallet so we can read without acquiring the lock
* fix the api router
* read wallet without acquiring or holding lock, only acquire the write lock for wallet when updating or adding outputs
* "wallet init" to create the initial wallet.seed file
* cleanup and better error msgs
* add some basic wallet docs
* cleanup and make passphrase optional
* refactor burn key into key_overrides on keychain
* introduce UnconfirmedChange output status, we can potentially spend these with zero confirmations
* pass in burn_key_id for the burn enabled keychain, spend *all* coins when spending from a wallet, spend UnconfirmedChange coins also
* add comment about simplifying wallet_data.select logic
* replace UnconfirmedChange output status with a more flexible zero_ok, flag on the output data
* rename pubkey -> key_id, these are key identifiers, not public keys
* reduce identifier seize to 10 bytes, get rid of fingerprints (identifiers are now small enough to use as-is)
* IDENTIFIER_SIZE const
* add FeeDispute error for when sender and recipient disagre on fee calculation (should never happen)
* does not need to be mut
* cleaned up some warnings
* added global slog instance, changed all logging macro formats to include logger instance
* adding configuration to logging, allowing for multiple log outputs
* updates to test, changes to build docs
* rustfmt
* moving logging functions into util crate
* add lock_height to Transaction and TxKernel, sign msg including both fee and lock_height in kernel
* make the order of the fields in tx and kernel more consistent
* rename to kernel_sig_msg
* add test to cover adding timelocked tx to pool, where tx is invalid based on current height of the blockchain
* add tests for adding timelocked txs to blocks (valid and otherwise)
* store wallet output data in hashmap
* cleanup up commented out code
* pass pubkey/identifier and not derivation around to miner etc.
* fix failing tests
Update coinbase building and block summation to account for half of
the fees going to the coinbase. Forcing fees to be even as a
consequence. Now that we can't build the coinbase independently
from the block (because fees), had to update the miner to keep the
key derivation so a new derivation isn't made any time a new block
gets worked on.
Minor doc and warning cleanups.
* introduce grin_keychain, encapsulate derivation of secret_keys
* core compiles against keychain, tests don't run yet
* core tests are now passing against keychain
* wip - getting wallet working with keychain
* add util and keychain to travis test matrix
* basic test around key derivation
* wallet key fingerprint as hex string
* use lowercase hex in util::to_hex for consistency with various crypto libs, add some tests for util::to_hex and util::from_hex
* filter wallet info based on ext_key fingerprint
* cleanup format for fingerprints and output status
* workaround the "commitment is not a public key" problem by using to_two_keys() from rust-secp256k1-zkp
* remove secp256k1zkp, update the various Cargo.toml to use rust-secp256k1-zkp from external git repo, update code to not rely on to_pubkey (and use verify_from_commit)
* secp256k1zkp from git
* cleanup secp256k1 dependencies
* gitignore node1 and node2 dirs
* minor cleanup - unused imports
* cleanup build warnings - unused vars
* make structs pub to get rid of the private_in_public lint warning
* missing docs on RangeProof
* add missing docs to store delete function
* cleaned up deprecation warning -
tokio_core -> tokio_io
complete() -> send()
* addition of configuration file to grin executable, instructions on how it's picked up are in grin.toml
* Starting to add a configuration file, and change some elements of configuration structures to make them more consistent
* Starting to refactor test, adding http post to wallet sender
* Implemented ability to run servers on different ports (mostly for testing), and implemented ability to post http requests directly to receiving wallets
* Adding detailed instructions on running multiple servers on the same machine
* Changes to build.doc to outline server running process
* Removed unwanted debug statements
* WIP Local server testing framework evolution
* More refactoring of server pool, checked in because there's a problem
* Added server reference structure, and ability to return server references from tests cleanly
* Added simulate_parallel_mining test, which puts some artificial slowdown into test mining loops, and the difficulty can currently be watched in the log
* Removing the ServerRef structure placed in earlier and replaced with a ServerStats structure, that just returns relevant info about the Server state without exposing it to the world
* Beginning to refactor server tests
* Extended test framework to include running wallet servers and miners. Also some changes to the wallet reciever api to include a wallet config and set the data directly for the wallet file explicitly rather than assuming it's always in the working dir.
* Adding wallet config to main executable
* Adding ability to select wallet server while mining
* Configurable wallet receiver url
Operations on the wallet data file are now fenced by a lock to
avoid potentially messy concurrent modifications by multiple
processes (i.e. the wallet receiver and a send command). The lock
is done using a create-only lock file, which is an atomic
operation.
Fixes a few loose ends in the full cycle of sending coins,
accepting them, pushing that transaction to the pool and having it
mined. More specifically:
* The API output endpoint needs to be a UTXO endpoint, as the
server can't make any guarantee about having a spent output.
* Bubbling up HTTP not found errors.
* Wallet output status checker now handles spent outputs.
* Transaction pool validates the transaction before accepting it.
* Fixed the operation API routes.
* Fixed too greedy wallet coin selection loop.
* Transactions coming from the network are now pushed to the pool
through the net adapter.
* New blocks accepted by the chain are sent to the pool for
eviction.
* The miner requests transactions from the pool to build its
blocks.
* The push API adds to the pool, removing the mock.
* Implementation of the adapter to the chain required by the pool
to get consistent UTXOs. Grossly unoptimized until we have the UTXO
MMR ready.
Once the wallet transaction is fully built, serializing it and
sending the push request to a node. Implemented the push node
API, mocked up for now (until the tx pool is integrated).
Parse the partial transaction encoded in JSON, adding a new output
and finalizing. Should push the final tx to a grin node for
broadcast once that's fleshed out. Should also add an endpoint for
the http receiver.
Most of the logic to build a transaction that sends coin to
another party. Still requires more debugging and clean up.
Main changes and additions are:
* Update to serde 1.0
* API endpoint to retrieve an Output
* Output is now Serialize and Deserialize
* Wallet configuration
* Command line for the send operation
* Wallet data checker to update created outputs into confirmed
* Wallet-specific configuration
With the coinbase receiver daemon in place, when starting a Grin
server in mining mode, the miner will now ask the receiver for a
coinbase output. The output is then used to insert in a block when
successfully mined.
Beginning of a first pass at simple wallet functionalities so
Grin can be used to author transactions. We introduce a
receiving server, to be at least able to build coinbase outputs
that can be used by the mining daemon.
Present:
* Coinbase receiving API.
* Command to start the receiving server.
* Beginning of a transaction sending command.
* Improvements to the REST API abstractions to support the above.
Still to do:
* Change to the miner daemon to use the receiving server.
* A command line sender.
* API to receive any transaction (not just coinbase).
* A command line receiver.
Beyond that, HD derivation and seed generation are very simple
so far and almost certainly insecure. Just for testing for now.
The wallet library will be used to manage the keys of the user. Currenty it only contains a minimal implementation of the extended keys. The format of the extended keys is subject to change
