* improve: HeaderSync optimization (#1372)
* remove get_locator() optimization, which should be an independent pr for security review
* refactoring: move 'headers_streaming_body()' from Message to Protocol
* move 2 headers utils functions out of Protocol, and remove 'pub'
* support reading variable size of BlockHeader, from Cuckoo30 to Cuckoo36
* fix: use global::min_sizeshift() instead of hardcoded 30, because Cuckoo10 will be used for AutomatedTesting chain
* fix: should use global::proofsize() instead of hardcoded 42 when calculate serialized_size_of_header
* replace another 42 with global::proofsize()
* Remove Iron dependecy and update hyper to version 0.12 #876
* REMOVE ME
* Revert "REMOVE ME"
This reverts commit e9a976eee98a2d5a4dfae5d9e1e4f5ed640c05d3.
* Rebase and start updating libwallet
Libwallet doesn't compile yet.
* Wallet compiles
* Grin compiles
* No compilation errors in tests
* All tests pass
* Reeturn future from handler
* Refactoring
* Fix lifetime issue one more time
I have to force push to rollback all the work done in last 2 days
* Fix wallet send issue
* Clean up
* move some debug! to trace!
* more informative debugs
* standardising on always showing chain tips as "cumulative difficulty @ height [hash]"
* make 2 debug outputs into a single
* "no peers" as warning (not info) to let it stand out more clearly
* move fn param (used only in this one debug line)
* clarify difficulty "units"
* Initial Dandelion Commit
* Changed stem_tx_pool to tx_stempool
* Introduction of stem memory pool and stem pool config
* Pool push now send to stem memory pool
* Add stem transaction functions
* Add stem transaction pool
* Drastically simplified code structure
* Add monitor transactions
* Add Dandelion monitor and remove transactions from stempool
* Add peer relay monitor
* Reconcile block with stempool
* Fix total size bug
* Add fluff option for pool push
* Added details on dandelion monitor
* Fix issue with missing parent
* Child transaction with stempool parent are now forced stem
* Update Dandelion Relay from outgoing peers
* Fix missing pool reconciliation
* Add the ability to fluff a transaction directly
* Fix tests for Dandelion
* Missing send_stem_transaction method...
* Add fluff handler for wallet
* Add logger when successfully updated Dandelion relay
* Launch transaction monitor last
* Fix dandelion relay misplaced
* Add logging and updating for stempool
* Additionnal check for stem transaction
* Fix 2 Locks in a row
* Util to zip and unzip directories
* First pass at sumtree request/response. Add message types, implement the exchange in the protocol, zip up the sumtree directory and stream the file over, with necessary adapter hooks.
* Implement the sumtree archive receive logicGets the sumtree archive data stream from the network and write it to a file. Unzip the file, place it at the right spot and reconstruct the sumtree data structure, rewinding where to the right spot.
* Sumtree hash structure validation
* Simplify sumtree backend buffering logic. The backend for a sumtree has to implement some in-memory buffering logic to provide a commit/rollback interface. The backend itself is an aggregate of 3 underlying storages (an append only file, a remove log and a skip list). The buffering was previously implemented both by the backend and some of the underlying storages. Now pushing back all buffering logic to the storages to keep the backend simpler.
* Add kernel append only store file to sumtrees. The chain sumtrees structure now also saves all kernels to a dedicated file. As that storage is implemented by the append only file wrapper, it's also rewind-aware.
* Full state validation. Checks that:
- MMRs are sane (hash and sum each node)
- Tree roots match the corresponding header
- Kernel signatures are valid
- Sum of all kernel excesses equals the sum of UTXO commitments
minus the supply
* Fast sync handoff to body sync. Once the fast-sync state is fully setup, get bacj in body sync
mode to get the full bodies of the last blocks we're missing.
* First fully working fast sync
* Facility in p2p conn to deal with attachments (raw binary after message).
* Re-introduced sumtree send and receive message handling using the above.
* Fixed test and finished updating all required db state after sumtree validation.
* Massaged a little bit the pipeline orphan check to still work after the new sumtrees have been setup.
* Various cleanup. Consolidated fast sync and full sync into a single function as they're very similar. Proper conditions to trigger a sumtree request and some checks on receiving it.
* first pass at allow/deny lists for hard-coded peers (not just seeds)
* commit
* add peers_allow and peers_deny examples and comments to grin.toml
* always ask for compact block
always fail to hydrate
always fallback to requesting full block
* decide to send full (empty) block over compact (empty) block
* add some randomness to the decision around broadcasting an empty block
as a block or as a compact block (so we can exercise more code paths easily)
* [wip] header first propagation
successfully propagating headers (unless we mined the block itself)
not yet asking for the block if we receive a header
* call request_block after successful processing header
* cleanup and skip asking for block if header is an orphan
* comments around error handling in receive_header
* Very quick peer banning endpoint, helps with #406
* Ping heights (#407)
* add height to ping/ping
* reformat output
* fix p2p test
* Fix orphan handling, not related to current head. Fixes#412
* Check before borrow, fixes#267
* Not finding an output commit in pos is an AlreadySpent
* Fix race condition, sending before conn is ready
* Explicit error for unknown pos of a forked block
* Remove config outdated tests. Fix#333
* Check ref and try before borrow, fix#400
* We do not want to sync with old peers anyway
* Hide cargo compiler warning for unused NoopAdapter and unused test code. Add TODOs
* introduce cpu_pool to read_msg
* cleanup and less unwrap() in handler
* expose cpu_pool as part of net_adapter (reuse across all peer connections and handlers)
* fix DummyAdaptor for test usage
* add new() to DummyAdapter for convenience
* Alleviate slog-async channel overflow by shortening while clarifying some log lines.
* add debug logging to GetBlock handling
* make another log line less rendudantly chatty
Added support for peer banning on the p2p server. The peer status
is changed and the peer is disconnected. A banned peer won't be
able to reconnect as well.
Tracking of chain errors due to a block that's intrinsically bad
and banning of the peer that sent it. If we're syncing, resetting
the header chain to the same as the main chain to force
backtracking.
* port across "simple_sync" changes from testnet1 branch
Cleanup direct refs to peer map or peer store
P2P server acts as a facade, handling the list of connected peers
and the storage of their information. Everything else goes through
the p2p server instead of having a peer map reference or going
straight to the store.
Fix p2p tests
* fix "monitoring peers" log msg (use connected_peers)
* add total_diff to ping/pong msgs
debug log for total_diff on each ping/pong
* expose peer addr to the handle_payload fn
so we know where it came from
* fix p2p tests for ping
* default to 0 if we cannot read total_difficulty
* updating a connected peer in place
* actually update peer info diff
* fixup p2p tests
Moved the HTTP APIs away from the REST endpoint abstraction and
to simpler Hyper handlers. Re-established all routes as v1.
Changed wallet receiver port to 13415 to avoid a gap in port
numbers.
Finally, rustfmt seems to have ignored specific files arguments,
running on everything.
* Do not send txs and blocks to senders. Keeps a ring buffer of transaction and blocks hashes that a peer has received. Do not send what we've already received.
* Test fix and fmt
* added global slog instance, changed all logging macro formats to include logger instance
* adding configuration to logging, allowing for multiple log outputs
* updates to test, changes to build docs
* rustfmt
* moving logging functions into util crate
* minor cleanup - unused imports
* cleanup build warnings - unused vars
* make structs pub to get rid of the private_in_public lint warning
* missing docs on RangeProof
* add missing docs to store delete function
* cleaned up deprecation warning -
tokio_core -> tokio_io
complete() -> send()
* Replace AsFixedBytes with Sized + AsRef<[u8]>
* Add AsRef<u8> to impl_array_newtype!
* Include AsFixedBytes as marker trait
* Related fixes
* Remove Deref
