Commit graph

130 commits

Author SHA1 Message Date
Antioch Peverell
f042f67fcd
Reintroduce block sums, verify full kernel sums per block (#1559)
* block_sums and full kernel sum verification

* rustfmt

* add docs/comments

* docs

* rustfmt

* comment on fact total_kernel_sum is redundant now

* make sure we setup block_sums correctly on a fork

* rustfmt

* replace those asserts with errors

* rustfmt
2018-09-20 09:19:32 +01:00
Gary Yu
e1c8dc5a3a
fix: reset sync_head to header_head on initial transition to HeaderSync (#1531)
* fix for bug #1524, reset sync_head to header_head on initial transition to HeaderSync.
* using @antiochp fix solution in #1539
2018-09-20 07:29:24 +08:00
Ignotus Peverell
9cc7aed713
Dual proof of work, first step (#1534)
* Move Proof and Difficulty types to pow module. Difficulty scaling calculation.
* Diffculty scaling and size shift for 2nd PoW
* Backport e7eb26ee
* Test compilation fixes
* Scaling only need to go one way, as @tromp pointed out
2018-09-18 15:12:57 -07:00
Gary Yu
e760ea2ce2 a little bit speed-up for full node syncing (#1463)
* simplified: track the head and orphans instead of every single block hash that sync requests
* change 1st block timeout from 1s to 5s
2018-09-10 12:32:20 -07:00
Antioch Peverell
dbe53f63c8
disable chain compaction on archive nodes (#1498)
* disable chain compaction when in archive mode

* rustfmt
2018-09-10 15:59:42 +01:00
Gary Yu
f971e8de77 infinite loops in check_orphans, if block in orphans has a forked parent (#1452)
* fix: if block in orphans queue has a forked parent, cause infinite loop in check_orphans()
* process all orphans at a given height before go to next height
2018-09-01 15:09:38 -04:00
Antioch Peverell
939d391e0e
cache rangeproof and kernel signature verification results (#1419)
* wip - cache rangeproof verification

* rustfmt

* rustfmt

* rename to ok_verifier

* rustfmt

* wire in caching verifier in more places
skip rangeproof and kernel signature verification during deserialization

* rustfmt

* actually cache verification results via ok verifier

* rustfmt

* cleanup

* pass a cache around, not a verifier

* rustfmt

* cleanup trait shenanigans

* rustfmt

* core tests passing

* rustfmt

* rustfmt

* tests passing

* rustfmt

* verifier cache now takes vecs of data in and out

* rustfmt

* logging + rustfmt

* add docs and comments
2018-08-30 15:44:34 +01:00
Antioch Peverell
a6bd81fdb3 Cleanup generics (where not needed) (#1429)
BlockChain does not need to be generic
2018-08-27 17:22:48 -04:00
Antioch Peverell
7ca4d45bf5
Cleanup build warnings (#1391)
* cleanup some unused vars etc.

* rustfmt
2018-08-20 21:56:32 +01:00
Antioch Peverell
bb7bc2c4f2
Refactor rewind single header (#1390)
* refactor txhashset to only take a single header
no need to pass in current head_header each time

* rustfmt

* fix bug introduced in rewind with LMDB

* rustfmt

* add some comments
2018-08-20 20:34:12 +01:00
Yeastplume
8440aad7ea
Retrieve header by output commit (#1388)
* Add get_header_for_output function to chain

* add api call to retrieve associated header for a given output

* rustfmt
2018-08-20 19:02:44 +01:00
Antioch Peverell
82a467ac3c
Fix txpool race condition (#1385)
* fix txpool race condition

* rustfmt

* fixup pool tests

* rustfmt

* rebase against master, pool tests passing
2018-08-20 14:48:05 +01:00
hashmap
99a66c1960
Consolidate validation in Block and Transaction (#1354)
* Consolidate validation in Block and Transaction

Introduce TransactionBody which is included into block and tx.
Fixes #1333
2018-08-15 23:14:48 +02:00
antiochp
bcebe7b5b3
run rustfmt against chain crate
suspect it has not been running successfully for a while...
2018-08-10 14:56:35 +01:00
Quentin Le Sceller
3df050cc93 Check content before zip/unzip the txhashset (#1174)
* Check txhashset content before zip/unzip
* Add header in txhashset verification
* Add copy function and test
* Add file util
* Now check and remove unexpected files instead of just crashing
2018-08-02 19:16:16 -07:00
Gary Yu
724b19e648 process_block check_known() check orphans also, to avoid double-processing (#1287) 2018-07-23 21:29:31 -07:00
Ignotus Peverell
f5b03a6d5a
Fix sync hash cache (#1285)
* Refactor hash cache handling to add properly. Only add to cache if the block is definitevely accepted or rejected.
* Dedup of header processing now looks like premature optimization. Very fast anyway. Removing the header hash cache.
2018-07-22 12:43:55 -07:00
Gary Yu
75b72e48ff Fix bug on block sync (#1271) (#1281)
Do not add orphans to the "already seen" cache.
2018-07-22 12:25:56 -07:00
Ignotus Peverell
3d89e86906
Various improvements and cleanup in chain crate (#1265)
* Remove now unnecessary txhashset write lock
* Ring buffer of hashes the chain has already processed
* Specifically report too old blocks as peer should be banned
* Move sync check for block relay, clean TODO
* No use processing transactions when syncing
* Ignore blocks older than horizon in pruning nodes
2018-07-16 21:58:56 +01:00
Antioch Peverell
f33c39cb66
call validate_kernel_history directly from txhashset_write (#1234)
no need for thr skip_kernel_hist bool now
2018-07-09 19:04:17 +01:00
Ignotus Peverell
a7cde3fe2b
No need to double-validate 2018-07-08 18:28:56 +01:00
Antioch Peverell
980378eb65
fix misbehaving simulnet fastsync test (#1227)
* fix misbehaving simulnet fastsync test
cleanup redundant cutoff vs bitmap params is rewind and check_compact

* make sure we do not verify full kernel history on a writeable txhashset extension
rework simulnet simulate_fast_sync test to be more robust

* fixup store tests

* sleep for a bit longer to give nodes time to update
their sync_state correctly

* tweak timing of simulate_block_propagation
2018-07-08 17:37:09 +01:00
Ignotus Peverell
0ab3cf29b8
Re-enable kernel history validation, re-add lost fix to validate the txhashset in a read only extension. 2018-07-06 23:48:41 +01:00
Ignotus Peverell
26d41fefd0
Fix kernel history verification (#1218)
Only rewind kernels to avoid requiring proper bitmap extraction. Also avoids maintaining bitmap data for kernels by introducing a "prunable" flag on PMMR backend.
2018-07-05 03:31:08 +01:00
Ignotus Peverell
1a410f115c
Cleanup a bunch of build warnings 2018-07-02 00:25:20 +01:00
Ignotus Peverell
2d4236f75b
Introduce state to track sync stages (#1210) (#1211)
* A new sync status enum encapsulated in a state struct allows tracking of where sync is at. Leveraging it in the TUI to provide more helpful messages.
* Percentage progression for most sync steps
2018-07-02 00:08:39 +01:00
Yeastplume
d2a84b7600
Error handling improvements (particularly in chain) (#1208)
* update error handling in chain and other modules to use error/errorkind

* sizeshift errorkind
2018-06-30 23:36:38 +01:00
Ignotus Peverell
dc827ebe93
Remove block markers, use header mmr pos instead (#1188)
Also fixes `spend_in_fork_and_compact` chain test.
2018-06-23 00:36:10 +01:00
yeastplume
836d5441e6 fix wallet tests, chain test output 2018-06-22 14:04:51 +01:00
yeastplume
8dfc7f3b2f rework block input bitmap logic to avoid batch locks 2018-06-22 11:44:50 +01:00
yeastplume
e7c380836b rustfmt 2018-06-22 09:44:38 +01:00
Ignotus Peverell
d189770080 Wallet LMDB backend (#1151)
* Migrate main node store to LMDB

In preparation to using LMDB as a wallet database, migrate the
node db. There's no point in having 2 key-value stores.
In addition LMDB provides a few advantages as a node db, namely a
much faster build (compared to RocksDb), lesser dependencies and
transactions.

* Migrated p2p store to lmdb, stuff compiles

* More fixes, chain tests starting to pass

* Fixed txhashset rollback messing with block save and general batch delimitation. Chain tests passing.

* rustfmt

* LMDB max map size of 10MB isn't really workable. Half TB seems reasonable.

* Fix wallet tests

* Rather crucial commit was missing

* rustfmt

* Fixing new merged tests following lmdb changes

* rustfmt

* * Make txhashset validation read-only on fast sync to avoid having
a really long open transaction.
* Fix deadlock in new block processing, batch should always be
created within a txhashset lock (when they interact).

* Comment about batch and txhashset interlacing

* Fix store tests to use batch

* Externalize wallet config and seed

* Converted direct read access to file outputs map to an iterator

* Cleaned up and simplified wallet Backend trait:

* No more direct mutable access to internal structures (HashMap)
* Batch interface for all writes
* Remove unneeded read wrapper (read_wallet)

* rustfmt

* First (incomplete) pass at wallet LMDB backend

* Progressing on lmdb backent iml

* Added batch impl for LMDB wallet backend. Pretty much done with it, but not sure how to deal with commit (owned).

* rustfmt

* Wrapping LMDB batch around a refcell to work around borrow rules

* Compilation up to grin chain
2018-06-22 09:08:06 +01:00
Ignotus Peverell
0967a5302b
Kernel sum and MMR sizes in block header (#1163)
* Add kernel commitments sum and kernel and output MMR sizes to block header
* Sum a block without including previous sums, cleanup. Blocks are now summed and validated based on their own totals and not the totals since genesis. This allows to get rid of BlockSum and simplified the setting of a new block's roots, kernel sum and MMR sizes. Fixes #116
* Additional kernel MMR validation to check all prior header roots successively
* Wallet tests fix
2018-06-21 02:30:22 +01:00
Antioch Peverell
0ff6763ee6
[consensus breaking] New Improved Merkle Proofs (#1119)
* new improved Merkle proofs

* fix pool and chain tests

* fixup core tests after the merge
2018-06-20 15:18:52 -04:00
Antioch Peverell
028b14d9d9
Replace rm_log with leaf_set (compact bitmap) (#1147)
* only store leaves in rm_log

* cleanup

* commit

* add failing test to cover case where we compact
an already compacted data file

* fix the logic for pruning the data file

* rm_log only contains leaves
prunelist only contains non-leaf subtree roots

* cleanup

* commit

* bitmap impl running in parallel

* rustfmt

* this is better - rewind unremoves outpu pos spent by rewound inputs

* commit

* commit

* pass bitmap around when rewinding

* store and retrieve input bitmap per block from the db

* Replace the RemoveLog with the UTXO set.

* utxo set starting to pass tests

* stuff works...

* split store types out
added some tests to compare the performance of the rm_log and the proposed utxo_set

* pull prune_list out into standalone file

* cleanup, get rid of unused height param

* cleanup and add doc comments

* add support for migrating rm_log to utxo_set

* take snapshot of utxo file during fast sync
implement migration of rm_log -> utxo_set

* rename rewound_copy to snapshot

* fixup pmmr tests to reflect cutoff_pos

* cleanup unused import

* check_compact needs to rewind the utxo_set as appropriate

* fix pool tests

* fixup core tests

* cache block_input_bitmaps via LruCache in store

* cache block header on initial write to db

* rename utxo_set -> leaf_set
and remove references to "spent" in grin_store

* better document the rewind behavior
2018-06-18 11:18:38 -04:00
Quentin Le Sceller
a30ee88236
Cleanup imports (#1161)
Reorganize imports
2018-06-14 08:16:14 -04:00
Yeastplume
88616fd341
[WIP] Updates to support web-wallet (#1160)
* updates to support web wallet workflow

* rustfmt

* functions to support wallet, error handling

* rustfmt

* rebase rustfmt

* test fix
2018-06-13 21:58:45 +01:00
Quentin Le Sceller
8f4dbfa540 Many typo fixes (#1158)
* Typo fix
* Parameterise
2018-06-13 17:03:34 +01:00
Ignotus Peverell
af178f82f8
Refactor the Keychain to be based on a trait (#1146)
* First pass at restructuring the keychain crate and introducing a Keychain trait
* Parameterized everything that had to. Stuff compiles.
* More stuff compiles, fix most tests
* Big merge, pushing down opening the keychain forced adding factory methods on trait
* Test fixes for pool and servers crate
2018-06-08 06:21:54 +01:00
Antioch Peverell
0ecadd3486
move verify_kernel_sums into committed trait (#1131) 2018-06-02 19:00:44 +01:00
Antioch Peverell
0a80023527
pass in next height when applying raw txs (#1129)
rewind correctly to the previous height as necessary
2018-06-02 12:29:18 +01:00
hashmap
2fa32d15ce Rustify core/src/core (#1122)
Small refactoring of one folder, if it makes sense I could extend the scope.
* Remove some cloning (real and just verbosity in the code)
* Naming conventions like to/into*
* Some Clippy's suggestions

I found that we don't use field init shorthand syntax, so I didn't touch this part, was it discussed before?
2018-06-01 20:41:26 +01:00
Antioch Peverell
4fda7a6899
Minimal Transaction Pool (#1067)
* verify a tx like we verify a block (experimental)

* first minimal_pool test up and running but not testing what we need to

* rework tx_pool validation to use txhashset extension

* minimal tx pool wired up but rough

* works locally (rough statew though)
delete "legacy" pool and graph code

* rework the new pool into TransactionPool and Pool impls

* rework pool to store pool entries
with associated timer and source etc.

* all_transactions

* extra_txs so we can validate stempool against existing txpool

* rework reconcile_block

* txhashset apply_raw_tx can now rewind to a checkpoint (prev raw tx)

* wip - txhashset tx tests

* more flexible rewind on MMRs

* add tests to cover apply_raw_txs on txhashset extension

* add_to_stempool and add_to_txpool

* deaggregate multi kernel tx when adding to txpoool

* handle freshness in stempool
handle propagation of stempool txs via dandelion monitor

* patience timer and fluff if we cannot propagate
to next relay

* aggregate and fluff stempool is we have no relay

* refactor coinbase maturity

* rewrote basic tx pool tests to use a real txhashset via chain adapter

* rework dandelion monitor to reflect recent discussion
works locally but needs a cleanup

* refactor dandelion_monitor - split out phases

* more pool test coverage

* remove old test code from pool (still wip)

* block_building and block_reconciliation tests

* tracked down chain test failure...

* fix test_coinbase_maturity

* dandelion_monitor now runs...

* refactor dandelion config, shared across p2p and pool components

* fix pool tests with new config

* fix p2p tests

* rework tx pool to deal with duplicate commitments (testnet2 limitation)

* cleanup and address some PR feedback

* add big comment about pre_tx...
2018-05-30 16:57:13 -04:00
Ignotus Peverell
d3a33c790e
Automatically trigger compaction every 2000 blocks (#1054)
Does so in a separate thread as it can take some time. Also
remove validation pre-compaction as it's hopefully unecessary
now.
2018-05-11 17:58:52 +01:00
Antioch Peverell
4dd94ff39e
[testnet2] Store output sum (#1043)
* block sums and reworked block validation
read and write block_sums
refactor validate on both block and txhashset
write block_sum on fast sync
we store the kernel_sum (need to account for the offset)

* block_sums

* rustfmt

* cleanup
2018-05-07 09:21:41 -04:00
Yeastplume
59664181e4
update wallet restore, generate new merkle proof for coinbase outputs (#1008) 2018-04-26 14:01:01 +01:00
Antioch Peverell
e22d025dc8 Simplify block marker metadata (#985)
* wip BlockMarker struct, get rid of PMMRMetadata
* use rewind to init the txhashet correctly on startup, we do not need to track pos via metadata (we have block markers), we do not need to open the txhashset with specific pos (we have rewind)
* better logging on init
* keep rewinding and validating on init, to find a good block
* use validate_roots on chain init
2018-04-24 19:53:01 +00:00
Yeastplume
4c6a193e01
[WIP] Miner removal (#979)
* Beginning to remove in-process miner

* rustfmt

* rustfmt

* rustfmt

* rustfmt

* remove pow crate and put remnants into core

* rustfmt

* fix test compilation in core and chain

* rustfmt

* Updating server tests to use test miner

* rustfmt

* rustfmt

* remove pow from test matrix

* adding basic stratum stats to TUI

* run stratum server at all times, and halt messages while syncing

* fix core tests

* add ability to run internal test miner for cuckoo 16 testing

* modify build instructions
2018-04-24 09:18:24 +01:00
Antioch Peverell
2503811e89
init_head and not reset_head on startup (#971)
so we do not reset the sync_head if we restart during a sync
2018-04-18 01:41:51 +01:00
hashmap
8e26d4dbac Fix Bus Error (core dumped) when validating fast sync txhashset (#956)
This PR fixes #953 by introducing a lock for txhashet_write. It's not enough
to synchronize access to in memory data, files also needs to be protected, so
a general txhashset lock was introduced.
2018-04-12 10:10:11 +01:00