Commit graph

161 commits

Author SHA1 Message Date
Dobby the House Elf
b8124676c7 Move libtx to core 2018-12-05 09:41:26 -08:00
Antioch Peverell
8e3a3e1a40
Replace "header by height" index with reads into the header MMR (#2030)
* replace header_by_height index with reads into the header MMR

* rustfmt

* cleanup

* cleanup chain tests

* fix locate_headers to stop on our max header

* fix the deadlock in comact_blocks_db...

* cleanup and docs/comments
2018-11-29 09:26:24 +00:00
Gary Yu
1b264d3554
protection of parallel requests of TxHashSetRequest (#1923)
* protection of parallel requests of TxHashSetRequest

* fix: adapt the test of test_unexpected_zip
2018-11-05 22:56:45 +08:00
Antioch Peverell
12be191ecd
validate root of header MMR when processing headers (during sync and full blocks) (#1836)
validate root of header MMR when processing headers (during sync and full blocks) (#1836)
2018-11-01 09:51:32 +00:00
Ignotus Peverell
a42d66efff
Merge branch 'master' into unitdiff 2018-10-25 14:20:41 -07:00
Gary Yu
0d06561a91
replace stdlib RwLock and Mutex with parking_lot (#1793)
* replace all stdlib RwLock with parking_lot RwLock

* replace stdlib Mutex with parking_lot Mutex

* rustfmt
2018-10-20 08:13:07 +08:00
John Tromp
e1b5d7f3c2 revert secondary scaling default in HeaderInfo; rename scaling_difficulty; refactor difficulty_data_to_vector 2018-10-19 21:39:54 +02:00
John Tromp
846b38308c replace all Difficulty::one by ::min 2018-10-18 21:18:16 +02:00
John Tromp
41293e9ef7 improve minimum difficulty handling 2018-10-18 20:37:33 +02:00
yeastplume
6980278b95 merge T4 into master - rustfmt 2018-10-18 11:23:22 +01:00
John Tromp
8540e4f723 [T4] tweaks and fixes (#1766)
* refactor consensus.rs, tweaking some values
* move scale() there
* fix set_header_nonce bug
* remove maturity soft-fork code
* increase diff target precision
* fix weight comments and try resolve PR conflict
2018-10-16 16:14:22 -07:00
John Tromp
34646ddf51 [T4] Rename all shiftsize / cuckoo_size to edge_bits and change value for T4 (#1752)
* replace all size_shift / cuckoo_size by edge_bits and change some constants for T4
* replace remaining occurrences of sizeshift
2018-10-15 16:14:23 -07:00
Ignotus Peverell
3c6b5a0a9c
Fix chain tests missing difficulty scaling 2018-10-15 22:24:13 +00:00
Ignotus Peverell
43f4f92730
[T4] Secondary proof of work difficulty adjustments (#1709)
* First pass at secondary proof of work difficulty adjustments
* Core and chain test fixes
* Next difficulty calc now needs a height. Scaling calculation fixes. Setting scaling on mined block.
* Change factor to u32 instead of u64.
* Cleanup structs used by next_difficulty
* Fix header size calc with u32 scaling
2018-10-13 13:57:01 -07:00
Yeastplume
6c8c483172
[T4 ONLY] BIP32 Wallet Compliance - Aggsig Updates - Bulletproof Updates (#1501)
* change keychain to use bip32 paths

* convert keychain to use bip32

* change identifier to be serialisation of 4-level bip32 path

* wallet changes compiling, pass parent key into all wallet functions

* rustfmt

* fix tests in chain

* rustfmt

* core tests passing

* rustfmt

* pool tests

* rustfmt

* fixing wallet tests

* rustfmt

* remove file wallet

* wallet tests compiling

* rustfmt

* remove db_migrate

* successful tx exchange test using BIP32 paths

* rustfmt

* fix wallet derivation paths to m/0/0/0

* wallet test fixed again, working with default path

* rustfmt

* fix server tests

* rustfmt

* make parent_id a trait on walletbackend

* rustfmt

* add ability for wallet to switch between multiple named accounts, and tests (not complete)

* rustfmt

* account switching tests in place and passing

* rustfmt

* compile and test with latest libsecp changes

* added public key sum to calculated e for aggsig

* rustfmt

* Update secp to 26

* bulletproof bip32 path integration

* rustfmt

* wallet restore updated with bip32 paths, also restores accounts

* rustfmt

* rustfmt

* remove old extkey

* remove old extkey

* rustfmt

* add wallet account commands

* rustfmt

* update wallet documentation

* rustfmt

* merge from master

* update libsecp tag

* merge from upstream and fix server test

* rustfmt

* rustfmt

* merge from master

* update latest libsecp merge

* fix commitment to zero value generation
2018-10-10 10:11:01 +01:00
Antioch Peverell
4d70968e70
More robust block pruning (#1637)
* more robust block deletion

* rustfmt
2018-10-02 16:17:15 +01:00
Antioch Peverell
e72d8b58e4
[WIP] txpool (tx validation) using block sums for full validation (#1567)
tx validation (txpool) rework/simplify
2018-09-24 09:24:10 +01:00
Ignotus Peverell
9cc7aed713
Dual proof of work, first step (#1534)
* Move Proof and Difficulty types to pow module. Difficulty scaling calculation.
* Diffculty scaling and size shift for 2nd PoW
* Backport e7eb26ee
* Test compilation fixes
* Scaling only need to go one way, as @tromp pointed out
2018-09-18 15:12:57 -07:00
Antioch Peverell
dca0d52dcd
Cleanup transaction with offset (#1514)
* cleanup build::transaction_with_offset

* rustfmt
2018-09-12 12:17:36 +01:00
Ignotus Peverell
ecf20602d5
Hard fork tryout: better PoW encapsulation in block header (#1478)
* Improve encapsulation with ProofOfWork struct
* Add dual pow scaling factor, fix test
* Fix pre_pow serialization, chain tests
* Adjust header serialized size calc
* Hard fork handling, version-based serialization
2018-09-10 15:36:57 -07:00
Antioch Peverell
dbe53f63c8
disable chain compaction on archive nodes (#1498)
* disable chain compaction when in archive mode

* rustfmt
2018-09-10 15:59:42 +01:00
John Tromp
c359c8aaf2 one day coinbase maturity (#1457)
* one day coinbase maturity

* make one day maturity a soft fork
2018-09-04 10:59:55 +01:00
Antioch Peverell
939d391e0e
cache rangeproof and kernel signature verification results (#1419)
* wip - cache rangeproof verification

* rustfmt

* rustfmt

* rename to ok_verifier

* rustfmt

* wire in caching verifier in more places
skip rangeproof and kernel signature verification during deserialization

* rustfmt

* actually cache verification results via ok verifier

* rustfmt

* cleanup

* pass a cache around, not a verifier

* rustfmt

* cleanup trait shenanigans

* rustfmt

* core tests passing

* rustfmt

* rustfmt

* tests passing

* rustfmt

* verifier cache now takes vecs of data in and out

* rustfmt

* logging + rustfmt

* add docs and comments
2018-08-30 15:44:34 +01:00
Antioch Peverell
7dfca6077c
Refactor compact block body (#1394)
* introduce CompactBlockBody

* rustfmt

* implement From<Block> for CompactBlock

* rustfmt

* remove debug logging

* wip

* rustfmt
2018-08-22 20:19:37 +01:00
Antioch Peverell
bb7bc2c4f2
Refactor rewind single header (#1390)
* refactor txhashset to only take a single header
no need to pass in current head_header each time

* rustfmt

* fix bug introduced in rewind with LMDB

* rustfmt

* add some comments
2018-08-20 20:34:12 +01:00
Yeastplume
8440aad7ea
Retrieve header by output commit (#1388)
* Add get_header_for_output function to chain

* add api call to retrieve associated header for a given output

* rustfmt
2018-08-20 19:02:44 +01:00
hashmap
99a66c1960
Consolidate validation in Block and Transaction (#1354)
* Consolidate validation in Block and Transaction

Introduce TransactionBody which is included into block and tx.
Fixes #1333
2018-08-15 23:14:48 +02:00
Quentin Le Sceller
3df050cc93 Check content before zip/unzip the txhashset (#1174)
* Check txhashset content before zip/unzip
* Add header in txhashset verification
* Add copy function and test
* Add file util
* Now check and remove unexpected files instead of just crashing
2018-08-02 19:16:16 -07:00
Gary Yu
76f4915bf9 improve: time crate flagged as deprecated, switch to actively maintained chrono crate (#1297)
* improve: time crate flagged as deprecated, switch to actively maintained chrono crate

* improve: complete the switching from deprecated time crate to chrono crate, for all the tests part

* improve: complete switching to chrono crate, for the left tests in 'chain' and 'wallet'
2018-07-30 09:33:28 +01:00
Yeastplume
d2a84b7600
Error handling improvements (particularly in chain) (#1208)
* update error handling in chain and other modules to use error/errorkind

* sizeshift errorkind
2018-06-30 23:36:38 +01:00
Ignotus Peverell
1398e0bf85
Accept multiple Cuckoo graph sizes (#1193)
* Added sizeshift to proof of work and encoding as VLQ
* Fix some loose ends to get things to compile
* Rename sizehift to min_sizeshift
  * Pipeline checks for PoW Cuckoo Cycle size shift and adjustment
factor before difficulty comparison.
  * Working delta+VLQ encoding of u64 PoW, unfortunately still a
little larger than u32, at least for Cuckoo30.
* Changed binary encoding of proof of work to a sequence of exact
compact bit representation of each nonce. Somewhat simpler,
shorter and matches the data to be hashed.
* Few fixes based on @tromp feedback:
* Max nonce is 2^(N-1)
* Need a separate constant for reference sizeshift
* Cuckoo implementation now conserves sizeshift. Test providing a
block at higher sizeshift.
* Last small overflow protection
2018-06-29 18:41:28 +01:00
Antioch Peverell
1df409fa69
verify coinbase maturity via output_mmr_size (#1203)
* we do not need Merkle proofs to spend coinbase outputs
we only need the output_mmr_size from the block header

* tests working with no Merkle proofs in inputs
2018-06-28 21:56:07 -04:00
Ignotus Peverell
dc827ebe93
Remove block markers, use header mmr pos instead (#1188)
Also fixes `spend_in_fork_and_compact` chain test.
2018-06-23 00:36:10 +01:00
yeastplume
836d5441e6 fix wallet tests, chain test output 2018-06-22 14:04:51 +01:00
yeastplume
f058b14aea Merge: Compiling at least 2018-06-22 09:33:42 +01:00
Ignotus Peverell
d189770080 Wallet LMDB backend (#1151)
* Migrate main node store to LMDB

In preparation to using LMDB as a wallet database, migrate the
node db. There's no point in having 2 key-value stores.
In addition LMDB provides a few advantages as a node db, namely a
much faster build (compared to RocksDb), lesser dependencies and
transactions.

* Migrated p2p store to lmdb, stuff compiles

* More fixes, chain tests starting to pass

* Fixed txhashset rollback messing with block save and general batch delimitation. Chain tests passing.

* rustfmt

* LMDB max map size of 10MB isn't really workable. Half TB seems reasonable.

* Fix wallet tests

* Rather crucial commit was missing

* rustfmt

* Fixing new merged tests following lmdb changes

* rustfmt

* * Make txhashset validation read-only on fast sync to avoid having
a really long open transaction.
* Fix deadlock in new block processing, batch should always be
created within a txhashset lock (when they interact).

* Comment about batch and txhashset interlacing

* Fix store tests to use batch

* Externalize wallet config and seed

* Converted direct read access to file outputs map to an iterator

* Cleaned up and simplified wallet Backend trait:

* No more direct mutable access to internal structures (HashMap)
* Batch interface for all writes
* Remove unneeded read wrapper (read_wallet)

* rustfmt

* First (incomplete) pass at wallet LMDB backend

* Progressing on lmdb backent iml

* Added batch impl for LMDB wallet backend. Pretty much done with it, but not sure how to deal with commit (owned).

* rustfmt

* Wrapping LMDB batch around a refcell to work around borrow rules

* Compilation up to grin chain
2018-06-22 09:08:06 +01:00
Ignotus Peverell
0967a5302b
Kernel sum and MMR sizes in block header (#1163)
* Add kernel commitments sum and kernel and output MMR sizes to block header
* Sum a block without including previous sums, cleanup. Blocks are now summed and validated based on their own totals and not the totals since genesis. This allows to get rid of BlockSum and simplified the setting of a new block's roots, kernel sum and MMR sizes. Fixes #116
* Additional kernel MMR validation to check all prior header roots successively
* Wallet tests fix
2018-06-21 02:30:22 +01:00
Antioch Peverell
0ff6763ee6
[consensus breaking] New Improved Merkle Proofs (#1119)
* new improved Merkle proofs

* fix pool and chain tests

* fixup core tests after the merge
2018-06-20 15:18:52 -04:00
Antioch Peverell
028b14d9d9
Replace rm_log with leaf_set (compact bitmap) (#1147)
* only store leaves in rm_log

* cleanup

* commit

* add failing test to cover case where we compact
an already compacted data file

* fix the logic for pruning the data file

* rm_log only contains leaves
prunelist only contains non-leaf subtree roots

* cleanup

* commit

* bitmap impl running in parallel

* rustfmt

* this is better - rewind unremoves outpu pos spent by rewound inputs

* commit

* commit

* pass bitmap around when rewinding

* store and retrieve input bitmap per block from the db

* Replace the RemoveLog with the UTXO set.

* utxo set starting to pass tests

* stuff works...

* split store types out
added some tests to compare the performance of the rm_log and the proposed utxo_set

* pull prune_list out into standalone file

* cleanup, get rid of unused height param

* cleanup and add doc comments

* add support for migrating rm_log to utxo_set

* take snapshot of utxo file during fast sync
implement migration of rm_log -> utxo_set

* rename rewound_copy to snapshot

* fixup pmmr tests to reflect cutoff_pos

* cleanup unused import

* check_compact needs to rewind the utxo_set as appropriate

* fix pool tests

* fixup core tests

* cache block_input_bitmaps via LruCache in store

* cache block header on initial write to db

* rename utxo_set -> leaf_set
and remove references to "spent" in grin_store

* better document the rewind behavior
2018-06-18 11:18:38 -04:00
Quentin Le Sceller
d3a2ecd489
Update env_logger to 0.5.1 (#1169)
* Update env_logger to 0.5.1
2018-06-14 18:59:53 -04:00
Quentin Le Sceller
a30ee88236
Cleanup imports (#1161)
Reorganize imports
2018-06-14 08:16:14 -04:00
Ignotus Peverell
af178f82f8
Refactor the Keychain to be based on a trait (#1146)
* First pass at restructuring the keychain crate and introducing a Keychain trait
* Parameterized everything that had to. Stuff compiles.
* More stuff compiles, fix most tests
* Big merge, pushing down opening the keychain forced adding factory methods on trait
* Test fixes for pool and servers crate
2018-06-08 06:21:54 +01:00
hashmap
2fa32d15ce Rustify core/src/core (#1122)
Small refactoring of one folder, if it makes sense I could extend the scope.
* Remove some cloning (real and just verbosity in the code)
* Naming conventions like to/into*
* Some Clippy's suggestions

I found that we don't use field init shorthand syntax, so I didn't touch this part, was it discussed before?
2018-06-01 20:41:26 +01:00
Antioch Peverell
4fda7a6899
Minimal Transaction Pool (#1067)
* verify a tx like we verify a block (experimental)

* first minimal_pool test up and running but not testing what we need to

* rework tx_pool validation to use txhashset extension

* minimal tx pool wired up but rough

* works locally (rough statew though)
delete "legacy" pool and graph code

* rework the new pool into TransactionPool and Pool impls

* rework pool to store pool entries
with associated timer and source etc.

* all_transactions

* extra_txs so we can validate stempool against existing txpool

* rework reconcile_block

* txhashset apply_raw_tx can now rewind to a checkpoint (prev raw tx)

* wip - txhashset tx tests

* more flexible rewind on MMRs

* add tests to cover apply_raw_txs on txhashset extension

* add_to_stempool and add_to_txpool

* deaggregate multi kernel tx when adding to txpoool

* handle freshness in stempool
handle propagation of stempool txs via dandelion monitor

* patience timer and fluff if we cannot propagate
to next relay

* aggregate and fluff stempool is we have no relay

* refactor coinbase maturity

* rewrote basic tx pool tests to use a real txhashset via chain adapter

* rework dandelion monitor to reflect recent discussion
works locally but needs a cleanup

* refactor dandelion_monitor - split out phases

* more pool test coverage

* remove old test code from pool (still wip)

* block_building and block_reconciliation tests

* tracked down chain test failure...

* fix test_coinbase_maturity

* dandelion_monitor now runs...

* refactor dandelion config, shared across p2p and pool components

* fix pool tests with new config

* fix p2p tests

* rework tx pool to deal with duplicate commitments (testnet2 limitation)

* cleanup and address some PR feedback

* add big comment about pre_tx...
2018-05-30 16:57:13 -04:00
Yeastplume
82ed280625
[WIP] Factoring out wallet traits and continued wallet library work (#1096)
* rename wallet libs

* rename transaction.rs to slate.rs

* rename transaction.rs to slate.rs

* move some types into libwallet/types

* rustfmt

* rename libtransaction libtx

* rename libtransaction libtx

* change types.rs to file_wallet

* rustfmt

* rename WalletData to FileWallet

* refactoring WalletBackend types out

* rustfmt

* All compiling, at least

* rustfmt

* fix FileWallet paths to persist

* rustfmt

* re-ignore wallet integration test
2018-05-30 17:48:32 +01:00
Yeastplume
4121ea1240
Wallet+Keychain refactoring (#1035)
* beginning to refactor keychain into wallet lib

* rustfmt

* more refactor of aggsig lib, simplify aggsig context manager, hold instance statically for now

* clean some warnings

* clean some warnings

* fix wallet send test a bit

* fix core tests, move wallet dependent tests into integration tests

* repair chain tests

* refactor/fix pool tests

* fix wallet tests, moved from keychain

* add wallet tests
2018-05-09 10:15:58 +01:00
Antioch Peverell
4dd94ff39e
[testnet2] Store output sum (#1043)
* block sums and reworked block validation
read and write block_sums
refactor validate on both block and txhashset
write block_sum on fast sync
we store the kernel_sum (need to account for the offset)

* block_sums

* rustfmt

* cleanup
2018-05-07 09:21:41 -04:00
Antioch Peverell
e22d025dc8 Simplify block marker metadata (#985)
* wip BlockMarker struct, get rid of PMMRMetadata
* use rewind to init the txhashet correctly on startup, we do not need to track pos via metadata (we have block markers), we do not need to open the txhashset with specific pos (we have rewind)
* better logging on init
* keep rewinding and validating on init, to find a good block
* use validate_roots on chain init
2018-04-24 19:53:01 +00:00
Yeastplume
4c6a193e01
[WIP] Miner removal (#979)
* Beginning to remove in-process miner

* rustfmt

* rustfmt

* rustfmt

* rustfmt

* remove pow crate and put remnants into core

* rustfmt

* fix test compilation in core and chain

* rustfmt

* Updating server tests to use test miner

* rustfmt

* rustfmt

* remove pow from test matrix

* adding basic stratum stats to TUI

* run stratum server at all times, and halt messages while syncing

* fix core tests

* add ability to run internal test miner for cuckoo 16 testing

* modify build instructions
2018-04-24 09:18:24 +01:00
Simon B
98efaf88df mixed small stuff (#934)
* documentation and spelling
* unused values in tests; just .unwrap()
* unneeded macro_use (router)
* unneeded macro_use and use-imports
* unused value -> .unwrap
* unused variables
* clarify that " is required (see 107ea3c4dd)
* first steps to dig into #933 test_p2p issues
* less simulnet build warnings
2018-04-05 23:31:34 +00:00
Yeastplume
6a6e600e48
tests for difficulty adjustment scenarios (#887)
* tests for difficulty adjustment scenarios

* ignore test
2018-03-27 19:15:42 +01:00
Antioch Peverell
c595a4b35c
GET /v1/chain/validate (#832)
chain.validate() now takes skip_rproof=true|false
2018-03-21 08:28:05 -04:00
Yeastplume
b53b84b0c6 Move nonce/pow to end of block header, change post-pow hash (#794)
* move nonce/pow to end of block header, update call to cuckoo-miner
* change header hash to be pow only
* fix genesis block miner and tests
2018-03-16 19:04:31 +00:00
Antioch Peverell
2552b34118
remove difficulty field from block_header (derive from total_difficulty) (#777)
* remove difficulty field from block_header (derive from total_difficulty)

* address feedback on PR and general cleanup

* rustfmt

* rework pow/difficulty validation in pipe::validate_header
now that we only have total_difficulty available

* cleanup various todos

* rustfmt

* rework DifficultyIterator to track header and prev_header state

* rustfmt caught some garbage syntax

* cleanup
2018-03-15 15:16:34 -04:00
Antioch Peverell
e75eef927b
Fix test dirs to isolate the tests (#786)
* 2 tests treading on each others dirs

* rustfmt

* actually use a different dir in each test...

* conflicting dirs across different test files also
2018-03-15 11:37:48 -04:00
Æde
bbbbdc455d Use header hash for by height index (#761)
* Use header hash for by height index
* Update semantics of save_header_height
2018-03-09 18:59:15 +00:00
Yeastplume
23ac36a834
Basic TUI Integration (#756)
* first explorations at attempting to integrate more user-friendly status screen/ui

* rustfmt

* adding some logo and color for visual interest

* formatting

* better integration with stdout, cleaner looking startup and shutdown

* rustfmt

* update to framework, and first collection of stats from server

* rustfmt

* commit of basic stat screen, think it's in a good enough state to share

* rustfmt

* fix to automated tests

* fix to automated tests

* grin.toml setting

* grin.toml setting

* grin.toml setting
2018-03-09 17:16:31 +00:00
Ignotus Peverell
ab4b2a19e3
Cleanup positional indexes for rewind, introducing block markers (#759)
* Cleanup MMRs positional indexes. Get rid of the kernel position index.
Introduced a new block marker index that keeps, for each block, the
respective positions in the output and kernel MMRs. This is now sufficient
for rewind.
* Block marker compaction
2018-03-08 21:36:51 +00:00
Ignotus Peverell
a9d1b76414
Compaction of pruned data for chain data structures (#746)
* Implementation of compaction for the chain. Single entry point on the chain triggers compaction of all MMRs as well as the cleanup of the positional index and full blocks.
* API endpoint, additional tests and more fixes for compaction
* Also prune PMMR metadata, minor bug fix
* PMMR store tests fix
2018-03-06 17:58:33 +00:00
Yeastplume
1143d84238
Remove Sumtree References and disambiguate some naming (#747)
* start of renamathon

* api renaming

* Rename UTXO-Output to lessen ambiguity

* compile warning

* compile error

* readme fix

* remove file commit in error
2018-03-05 19:33:44 +00:00
Ignotus Peverell
be8d9633e4
rustfmt all the things 2018-03-04 00:19:54 +00:00
Yeastplume
4c34c9ab52
PMMR Block to File positions metadata storage (#739)
* adding file position index data accessable to the chain, and allowing for storage of such within db

* missing file

* restart files at last recorded position in stored file metadata

* just use tip to store last pmmr index information

* error handling

* test fix
2018-03-03 09:08:36 +00:00
Antioch Peverell
cc12798d7a
Merkle Proofs (#716)
* family_branch() to recursively call family() up the branch
todo
  - we hit a peak, then we need to get to the root somehow
  - actually get the hashes to build the proof

* wip

* some additional testing around merkle tree branches

* track left/right branch for each sibling as we build the merkle path up

* MerkleProof and basic (incomplete) verify fn

* I think a MerkleProof verifies correctly now
need to test on test case with multiple peaks

* basic pmmr merkle proof working

* MerkleProof now serializable/deserializable

* coinbase maturity via merkle proof basically working

* ser/deser merkle proof into hex in api and wallet.dat

* cleanup

* wip - temporarily saving merkle proofs to the commit index

* assert merkle proof in store matches the rewound version
there are cases where it does not...

* commit

* commit

* can successfully rewind the output PMMR and generate a Merkle proof
need to fix the tests up now
and cleanup the code
and add docs for functions etc.

* core tests passing

* fixup chain tests using merkle proofs

* pool tests working with merkle proofs

* api tests working with merkle proof

* fix the broken comapct block hashing behavior
made nonce for short_ids explicit to help with this

* cleanup and comment as necessary

* cleanup variety of TODOs
2018-03-02 15:47:27 -05:00
Ignotus Peverell
341a3c9f4d
Minor cleanup, range proof pmmr validation and tests 2018-02-24 23:40:13 +00:00
Yeastplume
05d1c6c817
Take the 'Sum' out of 'Sumtree' (#702)
* beginning to remove sum

* continuing to remove sumtree sums

* finished removing sums from pmmr core

* renamed sumtree files, and completed changes+test updates in core and store

* updating grin/chain to include removelogs

* integration of flatfile structure, changes to chain/sumtree to start using them

* tests on chain, core and store passing

* cleaning up api and tests

* formatting

* flatfiles stored as part of PMMR backend instead

* all compiling and tests running

* documentation

* added remove + pruning to flatfiles

* remove unneeded enum

* adding sumtree root struct
2018-02-22 13:45:13 +00:00
Antioch Peverell
ebd801f14e
Transaction contain kernels. Transactions and blocks maintain a kernel offset (split key). (#681)
* WIP - split the key in final tx step
store "offset" on transaction itself

* rebase

* commit

* tx with offset

* got a test tx validating successfully using a sig from a split key and the appropriate offset

* sum up the offset for the block_header

* fix size tests for blocks and compact blocks (header now includes offset)

* use txs with offsets in most of the core tests
some tests now failing

* build kernel from k1G (k2 stored on tx, sum stored on header)

* commit

* tx now has vec of kernels
rework tx and kernel validation

* add test for tx cut_through

* wip - working on splitting in aggsig

* split the key when creating the initial sender aggsig context

* cleanup

* cleanup

* code needs claning up but split keys working for sender/receiver aggsig flow

* cleanup debug logging

* fix tests

* fix merge and basic cleanup

* fix keychain tests to use new tx_id
2018-02-13 10:35:30 -05:00
Ignotus Peverell
22c521eec8
[WIP] Abridged sync (#440)
* Util to zip and unzip directories
* First pass at sumtree request/response. Add message types, implement the exchange in the protocol, zip up the sumtree directory and stream the file over, with necessary adapter hooks.
* Implement the sumtree archive receive logicGets the sumtree archive data stream from the network and write it to a file. Unzip the file, place it at the right spot and reconstruct the sumtree data structure, rewinding where to the right spot.
* Sumtree hash structure validation
* Simplify sumtree backend buffering logic. The backend for a sumtree has to implement some in-memory buffering logic to provide a commit/rollback interface. The backend itself is an aggregate of 3 underlying storages (an append only file, a remove log and a skip list). The buffering was previously implemented both by the backend and some of the underlying storages. Now pushing back all buffering logic to the storages to keep the backend simpler.
* Add kernel append only store file to sumtrees. The chain sumtrees structure now also saves all kernels to a dedicated file. As that storage is implemented by the append only file wrapper, it's also rewind-aware.
* Full state validation. Checks that:
    - MMRs are sane (hash and sum each node)
    - Tree roots match the corresponding header
    - Kernel signatures are valid
    - Sum of all kernel excesses equals the sum of UTXO commitments
minus the supply
* Fast sync handoff to body sync. Once the fast-sync state is fully setup, get bacj in body sync
mode to get the full bodies of the last blocks we're missing.
* First fully working fast sync
* Facility in p2p conn to deal with attachments (raw binary after message).
* Re-introduced sumtree send and receive message handling using the above.
* Fixed test and finished updating all required db state after sumtree validation.
* Massaged a little bit the pipeline orphan check to still work after the new sumtrees have been setup.
* Various cleanup. Consolidated fast sync and full sync into a single function as they're very similar. Proper conditions to trigger a sumtree request and some checks on receiving it.
2018-02-09 22:32:16 +00:00
Quentin Le Sceller
8a7eb94759 Update bitflags to ^0.1 (#682)
* Removed unused crates
* Add listconnectedpeers in grin client
* Update bitflags to ^0.1 globally
2018-02-05 19:43:54 +00:00
Antioch Peverell
6647823177
header first propagation (#654)
* [wip] header first propagation
successfully propagating headers (unless we mined the block itself)
not yet asking for the block if we receive a header

* call request_block after successful processing header

* cleanup and skip asking for block if header is an orphan

* comments around error handling in receive_header
2018-01-30 09:42:04 -05:00
Yeastplume
783e4c250d
[WIP] Difficulty Adjustment Updates (#651)
* large updates for mining, fix async mode, will list changes in PR

* reset config and build defaults

* change to difficulty calculations

* tweaking mining params and tests

* tweaking to tests

* including pre-genesis data for difficulty adjustments, adding adjustment scenario tests

* further clarifying next_difficulty function

* moving tests out of consensus.rs

* pow test fix

* changing pre-genesis generation
2018-01-27 07:48:53 +00:00
AntiochP
cbd3b2ff87
hash (features|commitment) in output mmr (#615)
* experiment with lock_heights on outputs

* playing around with lock_height as part of the switch commitment hash

* cleanup

* include features in the switch commit hash key

* commit

* rebase off master

* commit

* cleanup

* missing docs

* rework coinbase maturity test to build valid tx

* pool and chain tests passing (inputs have switch commitments)

* commit

* cleanup

* check inputs spending coinbase outputs have valid lock_heights

* wip - got it building (tests still failing)

* use zero key for non coinbase switch commit hash

* fees and height wrong order...

* send output lock_height over to wallet via api

* no more header by height index
workaround this for wallet refresh and wallet restore

* refresh heights for unspent wallet outputs where missing

* TODO - might be slow?

* simplify - do not pass around lock_height for non coinbase outputs

* commit

* fix tests after merge

* build input vs coinbase_input
switch commit hash key encodes lock_height
cleanup output by commit index (currently broken...)

* is_unspent and get_unspent cleanup - we have no outputs, only switch_commit_hashes

* separate concept of utxo vs output in the api
utxos come from the sumtrees (and only the sumtrees, limited info)
outputs come from blocks (and we need to look them up via block height)

* cleanup

* better api support for block outputs with range proofs

* basic wallet operations appear to work
restore is not working fully
refresh refreshes heights correctly (at least appears to)

* wallet refresh and wallet restore appear to be working now

* fix core tests

* fix some mine_simple_chain tests

* fixup chain tests

* rework so pool tests pass

* wallet restore now safely habndles duplicate commitments (reused wallet keys)
for coinbase outputs where lock_height is _very_ important

* wip

* validate_coinbase_maturity
got things building
tests are failing

* lite vs full versions of is_unspent

* builds and working locally
zero-conf - what to do here?

* handle zero-conf edge case (use latest block)

* introduce OutputIdentifier, avoid leaking SumCommit everywhere

* fix the bad merge

* pool verifies coinbase maturity via is_matured
this uses sumtree in a consistent way

* cleanup

* add docs, cleanup build warnings

* fix core tests

* fix chain tests

* fix pool tests

* cleanup debug logging that we no longer need

* make out_block optional on an input (only care about it for spending coinbase outputs)

* cleanup

* bump the build
2018-01-16 22:03:40 -05:00
Simon B
8b2f9503c9 Make total_difficulty the sum of network difficulty [testnet2] (#523)
* Make total_difficulty the sum of network difficulty, not whatever the miner happened to mine.
  - Only for Testnet2 / Mainnet (hardforks Testnet1)
  - update chain::pipe validate_header to validate according to Testnet2 rules for cumulative difference

Fixes #280

* tests that should ignore (network) difficulty

* fn new_block is explained as "utility to create a block without worrying about the key or previous header" so it gets network difficulty := 0 too

* update tx pool tests (going with "minimum" for network difficulty for now)

* add ERR outputs about bannable offences
(#406 should know about these)

* whitespace fix

* mine_simple_chain: Probably DON'T overwrite difficulty (?)

* core/mod tests "reward_empty_block" and "reward_with_tx_block" tests set to use lowest network difficulty possible
2018-01-12 18:35:37 +00:00
Yeastplume
1199ed2cc1
[WIP] Aggsig Transactions (#530)
* First steps converting transaction workflow to be aggsig-enable

* integrating updated version of aggsig, which gives greater control over the contents of e

* added wallet transaction test to testing framework to enable testing the whole thing, completed interaction as far as inital response from recipient

* more aggsig work, final signature is produced now

* Construction of aggsig transaction now working to the point of the signature being built

* aggsig transactions working end-to-end in the nominal case

* refactor aggsig verify from commit and fix some tests

* more cleanup and test fixing

* cleaning up automated tests

* test+formatting fix
2018-01-10 19:36:27 +00:00
Ignotus Peverell
6a9a584c43
Fix pruning of last PMMR leaf, additional tests
Due to the construction of PMMRs the last element, when its a leaf,
can never be pruned as it has no parent yet and it will be needed
to calculate that hash. To work around this, we now insert coinbase
outputs first to add at least one output of padding.

Also changed the `set_sumtree_root` function on chain a bit to allow
setting the roots on a fork. Mostly useful for tests.

Added new test case to handle both the issue above and spending
transactions within a fork.
2018-01-08 01:23:23 +00:00
Ignotus Peverell
515aacc73c
Remove future block heights from store when backtracking
Fixes #499
2017-12-16 00:30:55 +00:00
AntiochP
cc9ec53390 sync_head port from testnet1 (#425)
* port simple_sync across from testnet1, sync head & improved fork handling
* introduce new sync_head for tracking header chain during sync
* add debug logging in body_sync for state of various heads
* update header_head during sync if header is now the one with most work, pass in sync and head contexts to pipe
* port across sync_head changes from master, put the 1s sleep back in...
* let sync run even if the full 512 headers are all known, give sync a chance to sync against a fork that forked a long time ago
* handle height jumping during a reorg
block validaton should not check for height+1 based on head
no assumptions should be made about height indices
* quick(er) check for previous block in process_block (we cannot check height against ctx head)
* make body_sync a _lot_ faster by finding the forked block more efficiently...
* fix monitoring peers log msg
* fix chain tests
* fix grin tests - we were using the wrong genesis hash (wrong chain type)
* apparently needs setting in both places...
* body -> header -> sync (#414)
* port over body -> header -> sync changes from testnet1
2017-12-04 19:16:57 +00:00
AntiochP
00d82f2c04 compare genesis during peering handshake (#327)
* wip - send genesis in handshake
* error if genesis mismatch on handshake
* fix the tests
* preserve order of existing fields in hand/shake
2017-11-20 12:35:52 -05:00
Ignotus Peverell
fec8858ead
Ported acbe983 and 58d7dc7 from testnet1 branch 2017-11-16 18:17:56 -05:00
Ignotus Peverell
91fdaa8320
Genesis block for known chains, mine dev chains (#269)
Renamed mining parameter mode to chain type, with existing types
of CI testing, user testing, testnet1 or mainnet. The public
chains (testnet1 and mainnet) come with their fully pre-defined
genesis block.

Still need to set the nonce and cycle for testnet1 genesis.
2017-11-15 16:49:15 -05:00
Ignotus Peverell
e4ebb7c7cb
Cleanup HTTP APIs, update ports to avoid gap, rustfmt
Moved the HTTP APIs away from the REST endpoint abstraction and
to simpler Hyper handlers. Re-established all routes as v1.
Changed wallet receiver port to 13415 to avoid a gap in port
numbers.

Finally, rustfmt seems to have ignored specific files arguments,
running on everything.
2017-10-31 19:42:56 -04:00
AntiochP
9a230ca7f6 fail deserialization if inputs or outputs are not sorted for txs (#173)
* fail deserialization if input or outputs are not sorted for txs, fail deserialization if inputs or outputs or kernels are not sorted for blocks, make sure we sort them before serialization
* extract consensus rule into verify_sort_order on VerifySortOrder trait
* rework the sort and write into WriteableSorted trait
* fix wallet sending bug (fees related) so we can test
* refactored read_and_verify_ordered out into ser, used in transaction and block
2017-10-13 17:23:18 +00:00
AntiochP
86420d4bca Shorter identifiers and call them key_id (not pubkey) (#175)
* rename pubkey -> key_id, these are key identifiers, not public keys
* reduce identifier seize to 10 bytes, get rid of fingerprints (identifiers are now small enough to use as-is)
* IDENTIFIER_SIZE const
* add FeeDispute error for when sender and recipient disagre on fee calculation (should never happen)
* does not need to be mut
* cleaned up some warnings
2017-10-13 04:45:07 +00:00
AntiochP
3dd1dde00b Pass pubkey for coinbase (#159)
* store wallet output data in hashmap
* cleanup up commented out code
* pass pubkey/identifier and not derivation around to miner etc.
* fix failing tests
2017-10-07 17:38:41 +00:00
Ignotus Peverell
3c137c0d1c
Fix odd fee in coinbase maturity test 2017-10-05 09:56:39 +00:00
AntiochP
677d0a3a95 keychain crate (no more secretkeys in core) (#146)
* introduce grin_keychain, encapsulate derivation of secret_keys
* core compiles against keychain, tests don't run yet
* core tests are now passing against keychain
* wip - getting wallet working with keychain
* add util and keychain to travis test matrix
* basic test around key derivation
2017-10-03 00:02:31 +00:00
Ignotus Peverell
8504efb796
Cargo fmt all the things 2017-09-29 18:44:25 +00:00
Ignotus Peverell
36bcd3cc39 Integrate sum trees with the rest of the system (#116)
* Integrate PMMR and its persistent backend with the Chain
* Chain can set tree roots; PMMR backend discard
* Check spent and prune for each input in new block
* Handling of forks by rewinding the state
* More PMMR tests and fixes, mostly around rewind
* Rewrite get_unspent to use the sumtrees, fix remaining compilation issues
2017-09-27 23:46:32 +00:00
AntiochP
95a92eefc9 Cannot spend coinbase for N blocks (#111)
* use head_header in add_to_memory_pool
* add COINBASE_MATURITY const to consensus
* add coinbase maturity (wip) validaton rule to validate_block
* add coinbase maturity check to validate_block
* map errors in adapters - specific errors still wip
* reworked so adapter translates chain errors to pool errors (core errors not required)
* add test for spending immature coinbase in memory pool
* wip - add test to cover spending coinbase output in chain.process_block
* added test coverage around process_block - we have a problem with coinbase output commitments
* add a comment on the failing test
* process_block will now fail validation if we attempt to spend coinbase that has not yet matured
(remember to use a new reward_key for every block). test coverage in place to verify this
2017-09-12 17:24:24 +00:00
AntiochP
7b12746a1f Store block header hash by input and output commitment (#104)
* initial pass at indexing block header hashes by commitments
* check block height matches currently indexed block height
when getting block header for an output commit
* add some test coverage
* document get_block_header_by_output_commit
2017-08-29 16:32:45 +00:00
Yeastplume
045f5bb4da POW refactor (#100)
* Adding output stats from plugin, when available
* adding grin config
* moving pow crate from core/pow to it's own crate
* moving POW/mining stuff into pow module, and removing mining dependency from chain
* refactored most of mining into pow module...miner.rs still in server
* update grin tests
* updating genesis block mining to use configured miner, if available
* chain tests back into chain, done with pow refactor
* use tag of cuckoo-miner for pr
2017-08-22 18:23:54 +00:00
AntiochP
be6b26e8f1 clean up .grin and .grin2 dirs before running the respective tests (#93) 2017-08-11 21:04:47 +00:00
AntiochP
3b4a48b2fd Cleanup build warnings (#87)
* minor cleanup - unused imports
* cleanup build warnings - unused vars
* make structs pub to get rid of the private_in_public lint warning
* missing docs on RangeProof
* add missing docs to store delete function
* cleaned up deprecation warning -
tokio_core -> tokio_io
complete() -> send()
2017-08-10 00:54:10 +00:00
Yeastplume
131ea2f799 Ci mode fixes (#86)
* playing around with changing cuckoo sizes on the fly
* modifying tests to use global cuckoo parameters, and checking results
* check for pow size
* Changing global function names, and removing length from POW serialization
2017-08-09 16:40:23 +00:00
Ignotus Peverell
22bff54f66
rustfmt (new version) 2017-07-27 21:13:34 +00:00
Ignotus Peverell
3c3b12b13a
Chain handles orphaned blocks and retries them
When a new block is rejected by the validation pipeline as orphan,
the chain now keeps it in a ring buffer of capped size. Anytime
a new block is accepted, all orphans in the ring buffer are
retried. This is helpful to handle all the cases where we receive
blocks out of order for networking/timing reasons.
2017-07-27 19:08:48 +00:00
Yeastplume
42376e68bc Cuckoo-miner better merged into Grin (#80)
* Adding cuckoo-miner into grin as a proper dependency now
* Defaulting to using cuckoo-miner
* Updates to tests to use cuckoo_miner by default, (using cuckoo16)
2017-07-18 20:57:09 +00:00
Ignotus Peverell
15ea8da34a
New Chain struct acting as facade for all chain operations
Introduce a new Chain struct that maintains the current head,
acts as facade to the store and the block processing pipeline
and handles all appropriate locking.

All higher level components, like the overall server, REST APIs
and miner have been update to only depend on Chain. Cleaned up
some duplication and tests now that there's a single entry point
to all blockchain operations.
2017-07-04 09:17:11 -07:00
Ignotus Peverell
e8a6b61100
Apply new difficulty algo, remove configurable cuckoo size
Integrate the new difficulty calculation into the block chain
validation, the miner and tests. As the difficulty calculation
doesn't use varying Cuckoo sizes anymore and we narrowed down
reasonable final Cuckoo Cycle parameters, removed all Cuckoo
Cycle sizes from block headers.

Formalized easier Cuckoo Cycle sizes for testing (and possibly
testnet) by introducing a test mode in configuration. Updated
all tests.
2017-06-19 08:59:56 -07:00
Ignotus Peverell
1c2b70f78f
Lower cuckoo graph size for test. 2017-02-07 15:12:35 -08:00
Ignotus Peverell
f6114231ae
Added support for a header chain, in addition to the main chain. Main chain must strictly follow header chain but header chain is allowed to be ahead (for sync). 2017-02-07 13:50:01 -08:00