Commit graph

162 commits

Author SHA1 Message Date
hashmap
923adf0fe5 Fix header_by_height (#922)
Currently for fast sync index header_by_height is created only for headers above the horizon
2018-04-02 23:10:30 +00:00
Ignotus Peverell
5ba0dbf38d
Fix and cleanup of fast sync triggering logic (#916)
* Fix and cleanup of fast sync triggering logic
* New txhashset on fast sync has to be applied, not rolled back
* Do not block if peer send buffer is full, fixes #912
2018-03-30 06:02:40 +00:00
Simon B
3e3fe6cae6 Testnet2 small fixes (#899)
* add hint for gcc-5 for building CUDA plugin
* add N_BLOCks comment for >=6GB GPU cards
* minor logging cleanup
* mark to be kept + reformat as "@ height [hash]"
* show CuckooNN + better wording
2018-03-29 15:56:46 +00:00
Antioch Peverell
696097370e
do not save header during header first sync (#891)
just use the header as a hint to go request the block
2018-03-27 16:48:09 -04:00
Antioch Peverell
5bd0c251dc
log some more fork related info when rewinding and applying a fork (#884) 2018-03-27 11:11:21 -04:00
Ignotus Peverell
16beeca8c9
Correct chain initialization at genesis 2018-03-26 17:38:27 +01:00
Yeastplume
6f264b2f7e
Testnet2 test fixes (#870)
* use dev genesis for automated tests

* never test block time on automated tests

* automated genesis block creation
2018-03-26 12:07:04 +01:00
John Tromp
a327427178 0-based positions, bag peaks right-to-left with mmr size as index (#863)
* 0-based positions, bag peaks right-to-left with mmr size as index
* fix remaining instances of hash_with_index
* fix pmmr test
2018-03-26 02:26:11 +00:00
Simon B
a8fc82d8c1 Console debug fixes (#861)
* move some debug! to trace!

* more informative debugs

* standardising on always showing chain tips as "cumulative difficulty @ height [hash]"

* make 2 debug outputs into a single

* "no peers" as warning (not info) to let it stand out more clearly

* move fn param (used only in this one debug line)

* clarify difficulty "units"
2018-03-25 12:41:12 -04:00
Antioch Peverell
7a8d6147f4 Split pmmr.get() into get_hash() and get_data() (#855) 2018-03-23 23:33:59 +00:00
Antioch Peverell
90a7f3d0f6 Validating an empty chain no longer panics (#850) 2018-03-23 01:48:05 +00:00
Antioch Peverell
ce2fb6ecb7 Rewind force rollback (#847)
* rewind() vs rewind_readonly()
* cleanup comments
2018-03-22 16:57:37 +00:00
Ignotus Peverell
ca8447f3bd
Removed all switch commitment usages, including restore (#841)
* Removed all switch commitment usages, including restore
* Fixed pool tests
* Fix keychain tests
* Get rid of the switch key in keychain
2018-03-22 00:10:11 +00:00
Antioch Peverell
c595a4b35c
GET /v1/chain/validate (#832)
chain.validate() now takes skip_rproof=true|false
2018-03-21 08:28:05 -04:00
Antioch Peverell
b56f21a9ac
split out verify_rangeproofs from sum_outputs (#823)
* split out verify_rangeproofs from sum_outputs
added some timing to the debug msgs
force_rollback at start of chain.validate()

* cleanup
2018-03-20 20:34:19 -04:00
Antioch Peverell
1d7a7457fc
force rollback on txhashset extension when validating (#824) 2018-03-20 17:46:29 -04:00
Antioch Peverell
7816f35238
rewind to header as part of txhashset validation (#808)
* rewind to header as part of txhashset validation
otherwise we risk including a new block and the roots do not match

* fix bug in rm_log rewind (wants to be inclusive of provided index)

* put block marker in the index so we can rewind correctly
during validation of the new txhashset

* rustfmt
2018-03-19 21:31:57 -04:00
Ignotus Peverell
154d52e548
Unecessary block cloning in chain 2018-03-19 21:02:02 +00:00
Antioch Peverell
2552b34118
remove difficulty field from block_header (derive from total_difficulty) (#777)
* remove difficulty field from block_header (derive from total_difficulty)

* address feedback on PR and general cleanup

* rustfmt

* rework pow/difficulty validation in pipe::validate_header
now that we only have total_difficulty available

* cleanup various todos

* rustfmt

* rework DifficultyIterator to track header and prev_header state

* rustfmt caught some garbage syntax

* cleanup
2018-03-15 15:16:34 -04:00
Antioch Peverell
00bfc4ec38
add dump_stats so we can see pmmr sizes after compaction (#785)
* add dump_stats so we can see pmmr sizes after compaction

* rustfmt

* fix tests with dump_stats()
2018-03-15 12:53:40 -04:00
Antioch Peverell
6a1a45e21f
store "total kernel offset" on block header (#769)
* move from storing per block kernel_offset in the block header
to storing the total accumulated kernel offset on each block header

* rustfmt

* cleanup
2018-03-14 11:22:09 -04:00
Antioch Peverell
65633c7611
check_compact retains leaves and roots until parents are pruned (#753)
* wip

* failing test for being too eager when pruning a sibling

* commit

* rustfmt

* [WIP] modified get_shift and get_leaf_shift to account for leaving "pruned but not compacted" leaves in place
Note: this currently breaks check_compact as nothing else is aware of the modified behavior

* rustfmt

* commit

* rustfmt

* basic prune/compact/shift working

* rustfmt

* commit

* rustfmt

* next_pruned_idx working (I think)

* commit

* horizon test uncovered some subtle issues - wip

* rustfmt

* cleanup

* rustfmt

* commit

* cleanup

* cleanup

* commit

* rustfmt

* contains -> binary_search

* rustfmt

* no need for height==0 special case

* wip - works for single compact, 2nd one breaks the mmr hashes

* commit

* rustfmt

* fixed it (needs a lot of cleanup)
we were not traversing all the way up to the peak if we pruned an entire tree
so rm_log and prune list were inconsistent

* multiple compact steps are working
data file not being copmacted currently (still to investigate)

* cleanup store tests

* cleanup

* cleanup up debug

* rustfmt

* take kernel offsets into account when summing kernels and outputs for full txhashset validation
validate chain state pre and post compaction

* rustfmt

* fix wallet refresh (we need block height to be refreshed on non-coinbase outputs)
otherwise we cannot spend them...

* rustfmt
2018-03-13 14:22:34 -04:00
Ignotus Peverell
65c0a4b7b0
Fee burning no more. Fixes #767 2018-03-13 02:39:22 +00:00
Æde
bbbbdc455d Use header hash for by height index (#761)
* Use header hash for by height index
* Update semantics of save_header_height
2018-03-09 18:59:15 +00:00
Yeastplume
5cf780ab98
Rm log rewind on startup (#760)
* add rewind to block height to rm_log on startup

* add rewind to block height to rm_log on startup

* flush remove log
2018-03-09 17:17:48 +00:00
Ignotus Peverell
ab4b2a19e3
Cleanup positional indexes for rewind, introducing block markers (#759)
* Cleanup MMRs positional indexes. Get rid of the kernel position index.
Introduced a new block marker index that keeps, for each block, the
respective positions in the output and kernel MMRs. This is now sufficient
for rewind.
* Block marker compaction
2018-03-08 21:36:51 +00:00
Ignotus Peverell
a9d1b76414
Compaction of pruned data for chain data structures (#746)
* Implementation of compaction for the chain. Single entry point on the chain triggers compaction of all MMRs as well as the cleanup of the positional index and full blocks.
* API endpoint, additional tests and more fixes for compaction
* Also prune PMMR metadata, minor bug fix
* PMMR store tests fix
2018-03-06 17:58:33 +00:00
Yeastplume
1143d84238
Remove Sumtree References and disambiguate some naming (#747)
* start of renamathon

* api renaming

* Rename UTXO-Output to lessen ambiguity

* compile warning

* compile error

* readme fix

* remove file commit in error
2018-03-05 19:33:44 +00:00
Yeastplume
90f844d382
Include index as part of pmmr element hash (#748)
* storing index as part of pmmr element hash

* factor out hash_with_index into generic trait impl

* tighten up trait bounds for PMMRable
2018-03-05 15:05:42 +00:00
Ignotus Peverell
be8d9633e4
rustfmt all the things 2018-03-04 00:19:54 +00:00
Yeastplume
4c34c9ab52
PMMR Block to File positions metadata storage (#739)
* adding file position index data accessable to the chain, and allowing for storage of such within db

* missing file

* restart files at last recorded position in stored file metadata

* just use tip to store last pmmr index information

* error handling

* test fix
2018-03-03 09:08:36 +00:00
Antioch Peverell
cc12798d7a
Merkle Proofs (#716)
* family_branch() to recursively call family() up the branch
todo
  - we hit a peak, then we need to get to the root somehow
  - actually get the hashes to build the proof

* wip

* some additional testing around merkle tree branches

* track left/right branch for each sibling as we build the merkle path up

* MerkleProof and basic (incomplete) verify fn

* I think a MerkleProof verifies correctly now
need to test on test case with multiple peaks

* basic pmmr merkle proof working

* MerkleProof now serializable/deserializable

* coinbase maturity via merkle proof basically working

* ser/deser merkle proof into hex in api and wallet.dat

* cleanup

* wip - temporarily saving merkle proofs to the commit index

* assert merkle proof in store matches the rewound version
there are cases where it does not...

* commit

* commit

* can successfully rewind the output PMMR and generate a Merkle proof
need to fix the tests up now
and cleanup the code
and add docs for functions etc.

* core tests passing

* fixup chain tests using merkle proofs

* pool tests working with merkle proofs

* api tests working with merkle proof

* fix the broken comapct block hashing behavior
made nonce for short_ids explicit to help with this

* cleanup and comment as necessary

* cleanup variety of TODOs
2018-03-02 15:47:27 -05:00
Ignotus Peverell
7f478d79f6
Proper error when trying to rewind to empty genesis 2018-03-01 03:09:10 +00:00
Ignotus Peverell
07df881a72
Initialize sumtree if genesis isn't empty 2018-03-01 03:08:35 +00:00
Yeastplume
5d1f1af892
Bulletproof messages (#730)
* beginning to add bullet proof messages

* Updated core transaction creation to embed the output's value and switch commit hash as part of the rangeproof message

* formatting issue

* more formatting issues

* Removing conditional feature compliation.. just bulletproofs from now on

* ensure MAX_PROOF_SIZE uses bulletproof sizing instead of earlier version

* updated with switch commit committed to in extra data

* accidentally commented out bullet-proof-size feature
2018-02-27 21:11:55 +00:00
Ignotus Peverell
341a3c9f4d
Minor cleanup, range proof pmmr validation and tests 2018-02-24 23:40:13 +00:00
Antioch Peverell
29044b5a32 last output in a block is the last regular output, failing that the last coinbase output (#727)
apply_block() processes regular outputs last so we need to be consistent with this
2018-02-24 19:40:23 +00:00
Alexey Miroshkin
452996a421 Attempt to recover from isolated orphaning (#724)
Due to occasional bad network conditions, a node can miss a block.
When the next one is received and detected orphaned, we should at least try a request for the
previous block, assuming it's not already an orphan in itself and we're not syncing already.

No additinal tests were implemented, test like `simulate_full_sync` test this functionality because
sync starts later. It seems to be an issue with sync test coverage. As fix I'd suggest to start sync
as soon as we get a peer connection.

[https://github.com/mimblewimble/grin/issues/705]
2018-02-24 02:48:02 +00:00
Yeastplume
05d1c6c817
Take the 'Sum' out of 'Sumtree' (#702)
* beginning to remove sum

* continuing to remove sumtree sums

* finished removing sums from pmmr core

* renamed sumtree files, and completed changes+test updates in core and store

* updating grin/chain to include removelogs

* integration of flatfile structure, changes to chain/sumtree to start using them

* tests on chain, core and store passing

* cleaning up api and tests

* formatting

* flatfiles stored as part of PMMR backend instead

* all compiling and tests running

* documentation

* added remove + pruning to flatfiles

* remove unneeded enum

* adding sumtree root struct
2018-02-22 13:45:13 +00:00
Ignotus Peverell
866ec977da
Factoring in mmr leaf count function 2018-02-19 23:20:32 +00:00
Antioch Peverell
4022b82817
cleanup various build warnings (#714)
* cleanup various build warnings

* use serde_json in tests only
2018-02-17 12:56:22 -05:00
Antioch Peverell
5572fa075e
tx pool lookup for kernel short ids (compact block hydration) (#710)
* wip - basic tx pool lookup for kernel short ids (compact block hydration)

* use the nonce in the compact_block to correctly generate short_ids for lookup

* query the tx pool based on kernel short_ids

* tests passing

* cleanup some logging

* cleanup logging
2018-02-16 10:42:27 -05:00
Ignotus Peverell
b82fa0ea1d
Block deletion, to be used for pruning 2018-02-14 23:25:12 +00:00
Ignotus Peverell
22c521eec8
[WIP] Abridged sync (#440)
* Util to zip and unzip directories
* First pass at sumtree request/response. Add message types, implement the exchange in the protocol, zip up the sumtree directory and stream the file over, with necessary adapter hooks.
* Implement the sumtree archive receive logicGets the sumtree archive data stream from the network and write it to a file. Unzip the file, place it at the right spot and reconstruct the sumtree data structure, rewinding where to the right spot.
* Sumtree hash structure validation
* Simplify sumtree backend buffering logic. The backend for a sumtree has to implement some in-memory buffering logic to provide a commit/rollback interface. The backend itself is an aggregate of 3 underlying storages (an append only file, a remove log and a skip list). The buffering was previously implemented both by the backend and some of the underlying storages. Now pushing back all buffering logic to the storages to keep the backend simpler.
* Add kernel append only store file to sumtrees. The chain sumtrees structure now also saves all kernels to a dedicated file. As that storage is implemented by the append only file wrapper, it's also rewind-aware.
* Full state validation. Checks that:
    - MMRs are sane (hash and sum each node)
    - Tree roots match the corresponding header
    - Kernel signatures are valid
    - Sum of all kernel excesses equals the sum of UTXO commitments
minus the supply
* Fast sync handoff to body sync. Once the fast-sync state is fully setup, get bacj in body sync
mode to get the full bodies of the last blocks we're missing.
* First fully working fast sync
* Facility in p2p conn to deal with attachments (raw binary after message).
* Re-introduced sumtree send and receive message handling using the above.
* Fixed test and finished updating all required db state after sumtree validation.
* Massaged a little bit the pipeline orphan check to still work after the new sumtrees have been setup.
* Various cleanup. Consolidated fast sync and full sync into a single function as they're very similar. Proper conditions to trigger a sumtree request and some checks on receiving it.
2018-02-09 22:32:16 +00:00
Quentin Le Sceller
8a7eb94759 Update bitflags to ^0.1 (#682)
* Removed unused crates
* Add listconnectedpeers in grin client
* Update bitflags to ^0.1 globally
2018-02-05 19:43:54 +00:00
Antioch Peverell
3f15f7f2f9
Request compact blocks (#667)
* first pass at allow/deny lists for hard-coded peers (not just seeds)

* commit

* add peers_allow and peers_deny examples and comments to grin.toml

* always ask for compact block
always fail to hydrate
always fallback to requesting full block

* decide to send full (empty) block over compact (empty) block

* add some randomness to the decision around broadcasting an empty block
as a block or as a compact block (so we can exercise more code paths easily)
2018-01-31 15:39:55 -05:00
Antioch Peverell
6647823177
header first propagation (#654)
* [wip] header first propagation
successfully propagating headers (unless we mined the block itself)
not yet asking for the block if we receive a header

* call request_block after successful processing header

* cleanup and skip asking for block if header is an orphan

* comments around error handling in receive_header
2018-01-30 09:42:04 -05:00
Yeastplume
783e4c250d
[WIP] Difficulty Adjustment Updates (#651)
* large updates for mining, fix async mode, will list changes in PR

* reset config and build defaults

* change to difficulty calculations

* tweaking mining params and tests

* tweaking to tests

* including pre-genesis data for difficulty adjustments, adding adjustment scenario tests

* further clarifying next_difficulty function

* moving tests out of consensus.rs

* pow test fix

* changing pre-genesis generation
2018-01-27 07:48:53 +00:00
Yeastplume
2def215553 Master port #605 (#606)
* port of stack overflow fix from testnet to master
* update process_block to encapsulate orphan check workflow
2018-01-24 18:20:34 +00:00
Simon B
885c2d73ea Making initial sync easier on the eyes [master] (#623)
* Making initial sync easier on the eyes:
- display "chain pointers" as: cumulative @ height [hash]
- clarify and line up to make the "pointers" easy to compare
- make every 100th block show info on debug level info, else as before
2018-01-18 22:47:42 +00:00