It was introduced in #2418, it turns out std read_line always appends to
the string, not writes from the beginning, so when we have multiple
workers to read from all messages are concatenated
* outbound connection count
* rustfmt
* display outbound peer count
* rustfmt
* wip - allow connections to exceed max when trying to create new outcound connections
any in excess of total max will be cleaned up subsequently
* fix
* we care about connected outbound peer count when deciding to connect to new peers
Currently we allocate a string in the heap per each worker check (not even
read) and clone message in write. This pr introduces a buffer which is
used for a round of workers check and remove clone in write_message.
Read optimization reduces number of allocations by factor of N (where N
is a number of workers), if N = 1 number of allocations doesn't change.
* fix: avoid duplicate connection
* ignore the duplicate connecting to same peer within 10 seconds
* refactor: use hashmap instead of vector for connecting history
* remove the double checking for already connected peer on connect
* add previous connecting time to the log
* fix a mistake on shrink
* move the now() into the inter loop for accurate time
* change the minimum allowed inverval time from 10s to 30s
* Only update outputs with unconfirmed transactions outstanding
* add further check for Unspent output that have been spent to check_repair
* rename to
* make updating all outputs optional API parameter. do full update before a wallet check, remove unspent->spent check from check process
* rustfmt
* typo
* add recover from phrase option to init
* rustfmt
* don't panic if recover is cancelled, grin-wallet.toml is created but wallet_data dir doesn't yet exits
* rustfmt
* Only update outputs with unconfirmed transactions outstanding
* add further check for Unspent output that have been spent to check_repair
* rename to
* make updating all outputs optional API parameter. do full update before a wallet check, remove unspent->spent check from check process
* rustfmt
* typo
* shutdown the tcpstream when peer connect or accept fail
* remove the unnecessary 3 times retry on peer connecting
* connect/accept are actually handshakings instead of tcpstream connect/accept
* Fix secondary scaling bugs; rename is_testnet -> is_floonet (#2215)
* add global::is_mainnet()
* use it to change pre-genesis pow type
* rename is_testnet -> is_floonet
* Support multiple chain configurations (#2217)
* Support multiple chain configurations
Supports generating the proper configuration for each chain type
(mainnet, floonet, usernet). Will run them by default under
their respective root directory (~/.grin/main, ~/.grin/floo, etc).
Assigned default ports for mainnet, overriding them to keep Floonet
ports unchanged.
For now, starting on mainnet will abort.
* Fixed usernet command line help message. Fixes#2217
* Differing magic numbers for each chain type (#2208)
* stick to e=H(R|P|m) when use schnorr signature (#2200)
* stick to e=H(R|P|m) when use schnorr signature
* (1)add verify_slate_messages for wallet receive (2)log the message content
* remove debug log on verify_slate_messages
* verify the sender's message signature when receive_tx in wallet listen
* Revert "remove debug log on verify_slate_messages"
This reverts commit 65ea32a407.
* Revert "rustfmt"
This reverts commit c380ab9185.
* Revert "(1)add verify_slate_messages for wallet receive (2)log the message content"
This reverts commit 9584ca7a89.
* [re-floonet] Keychain Floonet BIP32 version/network option (#2235)
* add 'is_floonet' property to keychain
* fix hex encoding and tests
* Fix couple floonet loose ends (#2230)
* Fix couple floonet loose ends. Fixes#2216
* Doc fix for sig message
* Refuse unkown kernel features (#2244)
* Minor: magic number change for re-floonet
* Set pre genesis is_secondary to true (#2247)
* Minor: tx validation error display underlying
* New floonet genesis
* genesis rustfmt
* Use chain-specific config for wallet toml gen
* Fix default wallet_listener_url
* New more reasonable genesis block, bumped version
* genesis rustfmt
* Couple minor fixes to genesis generation script
* ban a peer with highest height but can't send us the proper headers
* reset self.stalling_ts if syncing goes well
* fix: switch to peers.ban_peer() instead of peer.set_banned()
* Get last bitcon block hash, setup genesis header without PoW (for now)
* More a few properties to mainnet genesis. Don't get too excited, several are placeholders.
* Mine a valid Cuckaroo solution for genesis block
* Use miner as library to get a solution for genesis. Replace final values in genesis.rs before committing it.
* Complete genesis replacement
* Fixed various replacements to obtain a compilable, well-formed genesis
* Check plugin errors, uncomment PoW validation
* Fixes to nonce handling in genesis mining
* Also produce full block hashes
* Fix genesis hash test
* Switch commitments (#2157)
* [Floonet] Use switch commits for all blinding factors (#2178)
* move wallet mods back into dirs
* use switched keys for blinding factor in all cases
* re-implement flag to turn off switch commit derivation
* rename tx log entry field tx_hex -> stored_tx (#2181)
* [Floonet] add feature for height locked kernels (#2168)
* add feature for height locked kernels
* add function to compute kernel features appropriate for lock height, and use it
* only sign kernel-features relevant fields; refactor Features
* simplify invalid kernel logic
* remove unused height arg to reward::output and run some rustfmt
* replace nested if/else by match
* Floonet chain type and genesis, testnets cleanup (#2182)
* [Floonet] Encrypt private slate data upon storage in DB (#2189)
* xor encrypt stored nonce and blind sum in transaction data
* stop doc tests splatting wallet files throughout
* Remove bzip2 dependency
* Changed magic number and seeds for Floonet (#2188)
* Genesis generator now loads a local wallet seed to build coinbase.
* Floonet genesis block
* Add floonet to generated grin-server.toml comments
* Test with final Floonet genesis hashes
* Fix get_header_for_output for genesis (#2192)
* start search with min height 0 (#2195)
* add feature for height locked kernels
* add function to compute kernel features appropriate for lock height, and use it
* only sign kernel-features relevant fields; refactor Features
* simplify invalid kernel logic
* remove unused height arg to reward::output and run some rustfmt
* replace nested if/else by match
* move wallet mods back into dirs
* move wallet mod files
* use switched keys for blinding factor in all cases
* rustfmt
* test fix
* test fix
* test fix
* rustfmt
* re-implement flag to turn off switch commit derivation
* rustfmt
* add block hash to tui mining diff screen
include current header in there also
* rework the mining screen on the tui
show block hash as well as block height
fix the "off by one" error
* Chain init now handles genesis body properly, related unit test creating the genesis with reward
* Avoid making block body public by adding a with_reward method
* apply_block in all genesis cases works
* start wallet command refactoring
* another re-structuring attempt
* rustfmt
* begin splitting up wallet commands
* rustfmt
* clean up wallet arg checking
* rustfmt
* macro for arg parsing
* rustfmt
* factor out init commands
* rustfmt
* move recover to new format
* rustfmt
* add listen command to new format
* rustfmt
* Finish moving commands to new format
* rustfmt
* rustfmt
* propogate errors more cleanly
* rustfmt
* error handling cleanup
* replace header_by_height index with reads into the header MMR
* rustfmt
* cleanup
* cleanup chain tests
* fix locate_headers to stop on our max header
* fix the deadlock in comact_blocks_db...
* cleanup and docs/comments
* Cuckatoo size shift upgrade schedule
* Move the schedule into graph_weight instead of messing with min edge bits
* Cleanup and fixes now that we have an agreed upon schedule
* PoW context is now properly picked depending on the chain type,
edge bits and block height. Height const for T4 hard fork leaving
a couple weeks to have miners in place. Removed now unused Cuckoo context.
* Simplified block siphash
* Fix servers crate compilation
* Tiny bit cleaner block siphash. Maybe.
* Cuckatoo min edge bits update for T4 and mainnet
* Fix header size tests, Cuckatoo31 default means one more bit per edge
* Remove redundant param from verify_size
* block_accepted via adapter is now reorg aware
we skip reconciling the txpool is block is not most work
we skip reconciling the reorg_cache is not a reorg
* rustfmt
* logging tweaks
* rework block_accepted interface
* rustfmt
* rework reorg determination
* introduce BlockStatus to represent next vs reorg vs fork
* rustfmt
* cleanup logging
* log from adapter, even during sync
* default to minimum_confirmation = 10
pass in minimum_confirmations to grin wallet info
* rustfmt
* rebase against master
* fixup wallet tests
* fixup server tests
* remove receive_coinbase fn from wallet to wallet client
* start moving clients into separate mod
* rustfmt
* move wallet client + start listener into wallet 'clients'
* rustfmt
* refactor API to make it more modular and completely decouple sending
* rustfmt
* further decouple API from sending methods
* rustfmt
* remove wallet to wallet client trait
* rustfmt
* rename / refactor client + adapters
* rustfmt
* add adapter concept
* add midding node_clients dir
* add file and null adapters
* rustfmt
* remove receive from owner api
* factor out receiving slates into trait
* rustfmt
* adding listen trait
* rustfmt
* change listener to use trait
* rustfmt
* add test for file-based exchange
* replace http api send command
* rustfmt
* move controller out of libwallet and into top-level wallet dir
* rustfmt
* add moved controller
* split horizon into two explicit values for cut through and txhashset request
* let node which has 2-7 days of history be able to handle forks larger than 2 days
* add test simulate_long_fork
* add pause/resume feature on p2p for tests
* refactor the state_sync
* ignore the test case simulate_long_fork for normal Travis-CI
* refactor function check_txhashset_needed to be shared with body_sync
* fix: state TxHashsetDone should allow header sync
* Cleanup syncer and sync header (locator)
* Simplify body sync
* Remove duplicate head in locator, add greater case in close_enough
* Various sync small fixes and tuning after testing
* More close_enough tests and related minor fixes
* Update secp dependency to use crates.io version
* Add more details to various Cargo.toml
* Remove use of env variables that aren't reliably provided by cargo
* Add a struct to encapsulate common references and avoid passing
them around on every function.
* Consolidate `skip_sync_wait` and `awaiting_peers` in an
additional sync status.
* New awaiting peer status is initial too
* Initial expired peers removal
* Stop expired peers
* Simplify peer removal and remove only Defunct peers
* Make seed to check for expired peers every hour
* Get rid of unused vector of peers to remove
* Make peer deletion predicate closure immutable
* Replace logging backend to flexi-logger and add log rotation
* Changed flexi_logger to log4rs
* Disable logging level filtering in Root logger
* Support different logging levels for file and stdout
* Don't log messages from modules other than Grin-related
* Fix formatting
* Place backed up compressed log copies into log file directory
* Increase default log file size to 16 MiB
* Add comment to config file on log_max_size option
* Add peers used bandwidth calculation and display in TUI
* Fix formatting
* Change Mutex to RwLock from peer's used bandwidth statistics in Tracker
* Make used bandwidth column in TUI peers list sort by sum of bytes
We select a peer to ask a block randomly. Peer's send channel has
capacity 10. If we need too many blocks we limit number of blocks to asks as a number of peers
* 10, which means that there is some probability (pretty high) that we
will overflow send buffer capacity.
This fix freezes a peer list (which gives also some performance boost)
and create a cycle iteraror to equally distribute requests among the
peers.
There is a risk that a peer may be disconnected while we are sending a
request to the chanel, but stricltly speaking it was possible in the old
code too, perhaps with a lower probability.
Fixes#1748
* header MMR in use within txhashset itself
works with fast sync
not yet in place for initial header sync
* add the (currently unused) sync_head mmr
* use sync MMR during fast sync
rebuild header MMR after we validate full txhashset after download
* support missing header MMR (rebuild as necessary) for legacy nodes
* rename to HashOnly
* cleanup backend.append()
* simplify vec_backend to match simpler append api
* header MMR in use within txhashset itself
works with fast sync
not yet in place for initial header sync
* add the (currently unused) sync_head mmr
* use sync MMR during fast sync
rebuild header MMR after we validate full txhashset after download
* support missing header MMR (rebuild as necessary) for legacy nodes
* rustfmt
* comments/docs
* rustfmt
* cleanup DBBackend
* cleanup DBBackend
* cleanup
* rename to HashOnly
* rustfmt
* cleanup backend.append()
* simply pmmr append api
no need to pass position when appending
* cleanup
* simplify vec_backend to match simpler append api
* rustfmt
* docs/comments
* rustfmt
* cleanup
* First pass at secondary proof of work difficulty adjustments
* Core and chain test fixes
* Next difficulty calc now needs a height. Scaling calculation fixes. Setting scaling on mined block.
* Change factor to u32 instead of u64.
* Cleanup structs used by next_difficulty
* Fix header size calc with u32 scaling
* fix: avoid a confusing log when fastsync start (#1720)
(cherry picked from commit 4f462cdfdc)
* small speed optimization for header sync (#1719)
(cherry picked from commit d3589d1bf5)
* change keychain to use bip32 paths
* convert keychain to use bip32
* change identifier to be serialisation of 4-level bip32 path
* wallet changes compiling, pass parent key into all wallet functions
* rustfmt
* fix tests in chain
* rustfmt
* core tests passing
* rustfmt
* pool tests
* rustfmt
* fixing wallet tests
* rustfmt
* remove file wallet
* wallet tests compiling
* rustfmt
* remove db_migrate
* successful tx exchange test using BIP32 paths
* rustfmt
* fix wallet derivation paths to m/0/0/0
* wallet test fixed again, working with default path
* rustfmt
* fix server tests
* rustfmt
* make parent_id a trait on walletbackend
* rustfmt
* add ability for wallet to switch between multiple named accounts, and tests (not complete)
* rustfmt
* account switching tests in place and passing
* rustfmt
* compile and test with latest libsecp changes
* added public key sum to calculated e for aggsig
* rustfmt
* Update secp to 26
* bulletproof bip32 path integration
* rustfmt
* wallet restore updated with bip32 paths, also restores accounts
* rustfmt
* rustfmt
* remove old extkey
* remove old extkey
* rustfmt
* add wallet account commands
* rustfmt
* update wallet documentation
* rustfmt
* merge from master
* update libsecp tag
* merge from upstream and fix server test
* rustfmt
* rustfmt
* merge from master
* update latest libsecp merge
* fix commitment to zero value generation
* get rid of RwLock around peer instances
* rustfmt
* rename stuff to live_info
* rustfmt
* serialize into PeerInfoDisplay
limit live_info write lock to peer_info.update()
rename to PeerLiveInfo
* simplify broadcast logic a bit more
connected peers are connected by definition
* return true/false on broadcast so we can send to more peers
* rustfmt
* fix p2p and server tests for PeerInfoDisplay
* commit
* Go back to db for head and header_head
* Use batch consistently for db access
* Pass active txhashset and batch in the ctx
* Only update head in db if total work increases
* Only updated header_head if total work (on header chain) increases
Picks a random defunct peer and marks it healthy again anytime
we're querying peers to find more. Over enough time, any peer will
see another as defunct, leading to peer list atrophy. This
allows us random retries.
In addition, we need to cleanup peers after a while, we quickly
accumulate hundreds of dead peers over time. This should be
tracked by a different issue however.
Related: #1632
* Allow TLS for Wallet APIs
This PR adds an optional support of TLS for wallet APIs. Only PKCS12 format is supported, will address .pem support in next PR and provide some documentation.
Address #1425
* cuck placeholder
* rustfmt
* cuckatoo, early days
* rustfmt
* data structures are in place, siphash key creation is consistent with @tromp
* solver in place, (not yet working)
* cuckatoo test solver working with test nonce
* rustfmt
* update solver to remove adjacency list removals
* verifier functioning
* rustfmt
* Proper error handing in Cuckatoo module, couple of tests
* modify cuckoo/cuckatoo solvers and verifiers to function identically, in advance of trait refactoring
* rustfmt
* refactor PoW context into trait, default to using cuckoo context
* rustfmt
* create macros for integer casting/unwraps
* don't instantiate structs when just verifying, add test validation vector for cuckatoo 29
* rustfmt
* don't init cuckoo structs if just validating
* test fix
* ensure BH hashing for POW is only done within miner/validators
* Add api_secret
* Add to base64 method
* Add basic auth in API
* Add Basic Auth to owner API
* Add flag to enable disable basic auth
* Add .api_secret file
* Move Proof and Difficulty types to pow module. Difficulty scaling calculation.
* Diffculty scaling and size shift for 2nd PoW
* Backport e7eb26ee
* Test compilation fixes
* Scaling only need to go one way, as @tromp pointed out
Currently on startup we wait for 30 secs if we don't have at least 4 peers with more work. If a node was quickly restarted it is already fully synced so thereare no peers with more work at all. This pr assumes that if we've done some work already and we have enough peers but still 0 with more work we are fully synced and good to go.
* fix for unstable travis-ci test on servers module
* skip the PeerWithSelf connection request on sending side
* logs and comments change; and remove an unnecessary unwrap() in test
* improve: HeaderSync optimization (#1372)
* remove get_locator() optimization, which should be an independent pr for security review
* refactoring: move 'headers_streaming_body()' from Message to Protocol
* move 2 headers utils functions out of Protocol, and remove 'pub'
* support reading variable size of BlockHeader, from Cuckoo30 to Cuckoo36
* fix: use global::min_sizeshift() instead of hardcoded 30, because Cuckoo10 will be used for AutomatedTesting chain
* fix: should use global::proofsize() instead of hardcoded 42 when calculate serialized_size_of_header
* replace another 42 with global::proofsize()
* config file can now be generated by executable
* rustfmt
* remove now-unnecessary config defaults test
* set up paths and config file creation in user's home directory
* rustfmt
* remove default grin.toml
* add grin configuration command to spit out config file
* Split configuration into wallet and server
* rustfmt
* Restore logging to wallet configurations
* rustfmt
* simplify tx validation and aggregation
we *only* need to account for reward when building a block from txs
* rustfmt
* cleanup and tests passing
* rustfmt
* better comments in with_reward()
* fix wallet tests
It produces 1000+ otputs on a fast machine (?) and generates invalid URL as result (too many paramaters). We don't care about sending all outputs in this test, we just need something.
* Local test container used different folder than regular server, so test could not find txhashset files
* Error stacktrace was printed in cases when error was expected, which created some noise and confusion
* refactoring get_locator(), to ensure the gap between neighbours >= 2^n
* security enhancement for get_locator
* in case of sync head and the header head are different, clear history locators
Header sync used to run when complete (right count of blocks
received) or every 10 sec regardless. Sometimes 10 secs is not
enough, especially when a big batch of blocks bodies is arriving
at the same time.
This tweaks the time-based heuristic just a bit to only ask for
headers when stalled, meaning that the header head has not
*progressed* for 10 seconds.
* Fix bad header sync timeout logic
* Improve stability of fast sync test
Adds regular pings so that the 2 test servers know each others'
heights (necessary to find most work peer). Stop mining on the
first server once the 2nd kicks in. Avoids chasing a fast moving
target on Cuckoo10.
* Sort pool tx by fee over weight for mining. Fixes#1105
* Bucketing dependent transactions before weighing. Minor tx weight fixes.
* Limit length of tx chain, cleanup and test fixes
* Cleanup all mining references to a hardcoded tx count
* Small test improvement, cleanup
* first addition of static wallet file server
* rustfmt
* add custom build functions to download and untar particular release of the web-wallet
* rustfmt
* remove comments
* Include commitments non-duplicate checks in aggregate
* Remove said check from the pool
* Block building now uses tx aggregation to reduce duplication
* fix: in case txhashset validation fail, 'TxHashsetDownload' should retry (#1293)
And add 2 more fixes:
1. handle sending request failure of 'send_txhashset_request';
2. use a 10-mins timeout to handle noresponse failure, in case a peer ignore our txhashset request, or in case a peer fail to send a file for some reason.
* Remove Iron dependecy and update hyper to version 0.12 #876
* REMOVE ME
* Revert "REMOVE ME"
This reverts commit e9a976eee98a2d5a4dfae5d9e1e4f5ed640c05d3.
* Rebase and start updating libwallet
Libwallet doesn't compile yet.
* Wallet compiles
* Grin compiles
* No compilation errors in tests
* All tests pass
* Reeturn future from handler
* Refactoring
* Fix lifetime issue one more time
I have to force push to rollback all the work done in last 2 days
* Fix wallet send issue
* Clean up
* improve: time crate flagged as deprecated, switch to actively maintained chrono crate
* improve: complete the switching from deprecated time crate to chrono crate, for all the tests part
* improve: complete switching to chrono crate, for the left tests in 'chain' and 'wallet'
* Make grin.toml config optional. Mirror exisiting config parameters in grin.toml to source code, so binary can run without a config file. Add test for it.
* fixup! Make grin.toml config optional
* Remove now unnecessary txhashset write lock
* Ring buffer of hashes the chain has already processed
* Specifically report too old blocks as peer should be banned
* Move sync check for block relay, clean TODO
* No use processing transactions when syncing
* Ignore blocks older than horizon in pruning nodes
* beginnings to testclient implementation for more complete wallet API testing
* rework TestWalletClient to be message-based proxy
* test fix
* wallet tests now exercising the API directly as much as possible, capable of instantiating multiple wallets
* test in place to run both file and db wallets
* ensure all wallet api functions lock wallet as needed. Split up transaction creation from posting to chain
* fix misbehaving simulnet fastsync test
cleanup redundant cutoff vs bitmap params is rewind and check_compact
* make sure we do not verify full kernel history on a writeable txhashset extension
rework simulnet simulate_fast_sync test to be more robust
* fixup store tests
* sleep for a bit longer to give nodes time to update
their sync_state correctly
* tweak timing of simulate_block_propagation
* A new sync status enum encapsulated in a state struct allows tracking of where sync is at. Leveraging it in the TUI to provide more helpful messages.
* Percentage progression for most sync steps
* Added sizeshift to proof of work and encoding as VLQ
* Fix some loose ends to get things to compile
* Rename sizehift to min_sizeshift
* Pipeline checks for PoW Cuckoo Cycle size shift and adjustment
factor before difficulty comparison.
* Working delta+VLQ encoding of u64 PoW, unfortunately still a
little larger than u32, at least for Cuckoo30.
* Changed binary encoding of proof of work to a sequence of exact
compact bit representation of each nonce. Somewhat simpler,
shorter and matches the data to be hashed.
* Few fixes based on @tromp feedback:
* Max nonce is 2^(N-1)
* Need a separate constant for reference sizeshift
* Cuckoo implementation now conserves sizeshift. Test providing a
block at higher sizeshift.
* Last small overflow protection
