Commit graph

20 commits

Author SHA1 Message Date
Antioch Peverell
fffee377dd
Fix 2nd compact (#1143)
* add failing test to cover case where we compact
an already compacted data file

* fix the logic for pruning the data file
2018-06-06 22:46:21 +01:00
Antioch Peverell
4fda7a6899
Minimal Transaction Pool (#1067)
* verify a tx like we verify a block (experimental)

* first minimal_pool test up and running but not testing what we need to

* rework tx_pool validation to use txhashset extension

* minimal tx pool wired up but rough

* works locally (rough statew though)
delete "legacy" pool and graph code

* rework the new pool into TransactionPool and Pool impls

* rework pool to store pool entries
with associated timer and source etc.

* all_transactions

* extra_txs so we can validate stempool against existing txpool

* rework reconcile_block

* txhashset apply_raw_tx can now rewind to a checkpoint (prev raw tx)

* wip - txhashset tx tests

* more flexible rewind on MMRs

* add tests to cover apply_raw_txs on txhashset extension

* add_to_stempool and add_to_txpool

* deaggregate multi kernel tx when adding to txpoool

* handle freshness in stempool
handle propagation of stempool txs via dandelion monitor

* patience timer and fluff if we cannot propagate
to next relay

* aggregate and fluff stempool is we have no relay

* refactor coinbase maturity

* rewrote basic tx pool tests to use a real txhashset via chain adapter

* rework dandelion monitor to reflect recent discussion
works locally but needs a cleanup

* refactor dandelion_monitor - split out phases

* more pool test coverage

* remove old test code from pool (still wip)

* block_building and block_reconciliation tests

* tracked down chain test failure...

* fix test_coinbase_maturity

* dandelion_monitor now runs...

* refactor dandelion config, shared across p2p and pool components

* fix pool tests with new config

* fix p2p tests

* rework tx pool to deal with duplicate commitments (testnet2 limitation)

* cleanup and address some PR feedback

* add big comment about pre_tx...
2018-05-30 16:57:13 -04:00
Antioch Peverell
e22d025dc8 Simplify block marker metadata (#985)
* wip BlockMarker struct, get rid of PMMRMetadata
* use rewind to init the txhashet correctly on startup, we do not need to track pos via metadata (we have block markers), we do not need to open the txhashset with specific pos (we have rewind)
* better logging on init
* keep rewinding and validating on init, to find a good block
* use validate_roots on chain init
2018-04-24 19:53:01 +00:00
Antioch Peverell
ced6e647b8
still do not fully understand what PhantomData is *for* exactly (#991)
but standardized naming and usage to match https://doc.rust-lang.org/beta/nomicon/phantom-data.html
so at least it looks similar to the official docs
2018-04-21 17:03:45 -04:00
Antioch Peverell
f0cf903adc
fix various issues with rewinding data file (#983)
* fix issue with rewinding data file
also fix fragility around an empty MMR (and the memory map)

* cleanup
2018-04-19 19:52:46 +01:00
Ignotus Peverell
eefb4319f8
Fix buffer index in compact
Indexes used to be positions in the full file, not in the
intermediary buffer.
2018-04-02 21:49:35 +01:00
Antioch Peverell
7a8d6147f4 Split pmmr.get() into get_hash() and get_data() (#855) 2018-03-23 23:33:59 +00:00
Antioch Peverell
7816f35238
rewind to header as part of txhashset validation (#808)
* rewind to header as part of txhashset validation
otherwise we risk including a new block and the roots do not match

* fix bug in rm_log rewind (wants to be inclusive of provided index)

* put block marker in the index so we can rewind correctly
during validation of the new txhashset

* rustfmt
2018-03-19 21:31:57 -04:00
Antioch Peverell
cb71386097
hash with the pos for non-leaf nodes (#788)
* hash_with_index on non-leaf nodes
rework the Merkle proof to include the pos of each sibling in the path

* rustfmt

* cleanup

* cleanup

* use get_from_file in validate (children may have been "removed")

* rustfmt

* fixup store tests
2018-03-16 10:45:58 -04:00
Yeastplume
244fc8d32e
PMMR data compaction (#784)
* fix and re-introduce data file compaction

* rustfmt

* remove commented line

* test cleanup

* remove println

* fix to core test
2018-03-15 17:12:30 +00:00
Antioch Peverell
00bfc4ec38
add dump_stats so we can see pmmr sizes after compaction (#785)
* add dump_stats so we can see pmmr sizes after compaction

* rustfmt

* fix tests with dump_stats()
2018-03-15 12:53:40 -04:00
Antioch Peverell
65633c7611
check_compact retains leaves and roots until parents are pruned (#753)
* wip

* failing test for being too eager when pruning a sibling

* commit

* rustfmt

* [WIP] modified get_shift and get_leaf_shift to account for leaving "pruned but not compacted" leaves in place
Note: this currently breaks check_compact as nothing else is aware of the modified behavior

* rustfmt

* commit

* rustfmt

* basic prune/compact/shift working

* rustfmt

* commit

* rustfmt

* next_pruned_idx working (I think)

* commit

* horizon test uncovered some subtle issues - wip

* rustfmt

* cleanup

* rustfmt

* commit

* cleanup

* cleanup

* commit

* rustfmt

* contains -> binary_search

* rustfmt

* no need for height==0 special case

* wip - works for single compact, 2nd one breaks the mmr hashes

* commit

* rustfmt

* fixed it (needs a lot of cleanup)
we were not traversing all the way up to the peak if we pruned an entire tree
so rm_log and prune list were inconsistent

* multiple compact steps are working
data file not being copmacted currently (still to investigate)

* cleanup store tests

* cleanup

* cleanup up debug

* rustfmt

* take kernel offsets into account when summing kernels and outputs for full txhashset validation
validate chain state pre and post compaction

* rustfmt

* fix wallet refresh (we need block height to be refreshed on non-coinbase outputs)
otherwise we cannot spend them...

* rustfmt
2018-03-13 14:22:34 -04:00
Yeastplume
5cf780ab98
Rm log rewind on startup (#760)
* add rewind to block height to rm_log on startup

* add rewind to block height to rm_log on startup

* flush remove log
2018-03-09 17:17:48 +00:00
Ignotus Peverell
a9d1b76414
Compaction of pruned data for chain data structures (#746)
* Implementation of compaction for the chain. Single entry point on the chain triggers compaction of all MMRs as well as the cleanup of the positional index and full blocks.
* API endpoint, additional tests and more fixes for compaction
* Also prune PMMR metadata, minor bug fix
* PMMR store tests fix
2018-03-06 17:58:33 +00:00
Yeastplume
1143d84238
Remove Sumtree References and disambiguate some naming (#747)
* start of renamathon

* api renaming

* Rename UTXO-Output to lessen ambiguity

* compile warning

* compile error

* readme fix

* remove file commit in error
2018-03-05 19:33:44 +00:00
Ignotus Peverell
be8d9633e4
rustfmt all the things 2018-03-04 00:19:54 +00:00
Yeastplume
4c34c9ab52
PMMR Block to File positions metadata storage (#739)
* adding file position index data accessable to the chain, and allowing for storage of such within db

* missing file

* restart files at last recorded position in stored file metadata

* just use tip to store last pmmr index information

* error handling

* test fix
2018-03-03 09:08:36 +00:00
Antioch Peverell
cc12798d7a
Merkle Proofs (#716)
* family_branch() to recursively call family() up the branch
todo
  - we hit a peak, then we need to get to the root somehow
  - actually get the hashes to build the proof

* wip

* some additional testing around merkle tree branches

* track left/right branch for each sibling as we build the merkle path up

* MerkleProof and basic (incomplete) verify fn

* I think a MerkleProof verifies correctly now
need to test on test case with multiple peaks

* basic pmmr merkle proof working

* MerkleProof now serializable/deserializable

* coinbase maturity via merkle proof basically working

* ser/deser merkle proof into hex in api and wallet.dat

* cleanup

* wip - temporarily saving merkle proofs to the commit index

* assert merkle proof in store matches the rewound version
there are cases where it does not...

* commit

* commit

* can successfully rewind the output PMMR and generate a Merkle proof
need to fix the tests up now
and cleanup the code
and add docs for functions etc.

* core tests passing

* fixup chain tests using merkle proofs

* pool tests working with merkle proofs

* api tests working with merkle proof

* fix the broken comapct block hashing behavior
made nonce for short_ids explicit to help with this

* cleanup and comment as necessary

* cleanup variety of TODOs
2018-03-02 15:47:27 -05:00
Ignotus Peverell
9fa344383c
Fix MMR compaction to take a horizon. Additional tests. 2018-02-28 21:20:54 +00:00
Yeastplume
05d1c6c817
Take the 'Sum' out of 'Sumtree' (#702)
* beginning to remove sum

* continuing to remove sumtree sums

* finished removing sums from pmmr core

* renamed sumtree files, and completed changes+test updates in core and store

* updating grin/chain to include removelogs

* integration of flatfile structure, changes to chain/sumtree to start using them

* tests on chain, core and store passing

* cleaning up api and tests

* formatting

* flatfiles stored as part of PMMR backend instead

* all compiling and tests running

* documentation

* added remove + pruning to flatfiles

* remove unneeded enum

* adding sumtree root struct
2018-02-22 13:45:13 +00:00