Commit graph

278 commits

Author SHA1 Message Date
Antioch Peverell
028b14d9d9
Replace rm_log with leaf_set (compact bitmap) (#1147)
* only store leaves in rm_log

* cleanup

* commit

* add failing test to cover case where we compact
an already compacted data file

* fix the logic for pruning the data file

* rm_log only contains leaves
prunelist only contains non-leaf subtree roots

* cleanup

* commit

* bitmap impl running in parallel

* rustfmt

* this is better - rewind unremoves outpu pos spent by rewound inputs

* commit

* commit

* pass bitmap around when rewinding

* store and retrieve input bitmap per block from the db

* Replace the RemoveLog with the UTXO set.

* utxo set starting to pass tests

* stuff works...

* split store types out
added some tests to compare the performance of the rm_log and the proposed utxo_set

* pull prune_list out into standalone file

* cleanup, get rid of unused height param

* cleanup and add doc comments

* add support for migrating rm_log to utxo_set

* take snapshot of utxo file during fast sync
implement migration of rm_log -> utxo_set

* rename rewound_copy to snapshot

* fixup pmmr tests to reflect cutoff_pos

* cleanup unused import

* check_compact needs to rewind the utxo_set as appropriate

* fix pool tests

* fixup core tests

* cache block_input_bitmaps via LruCache in store

* cache block header on initial write to db

* rename utxo_set -> leaf_set
and remove references to "spent" in grin_store

* better document the rewind behavior
2018-06-18 11:18:38 -04:00
Quentin Le Sceller
a30ee88236
Cleanup imports (#1161)
Reorganize imports
2018-06-14 08:16:14 -04:00
Yeastplume
88616fd341
[WIP] Updates to support web-wallet (#1160)
* updates to support web wallet workflow

* rustfmt

* functions to support wallet, error handling

* rustfmt

* rebase rustfmt

* test fix
2018-06-13 21:58:45 +01:00
Quentin Le Sceller
8f4dbfa540 Many typo fixes (#1158)
* Typo fix
* Parameterise
2018-06-13 17:03:34 +01:00
Quentin Le Sceller
169ffee607 Externalize PMMR test (#1154)
Externalize PMMR tests into their own file under `tests/`
2018-06-11 22:01:30 +01:00
Antioch Peverell
edc01bb05a Add verify_cut_through to both transaction and block (#1130)
We have no code that will produce a non-cut-through version of either
so this should be safe to add in testnet2
2018-06-10 20:59:58 +01:00
Ignotus Peverell
af178f82f8
Refactor the Keychain to be based on a trait (#1146)
* First pass at restructuring the keychain crate and introducing a Keychain trait
* Parameterized everything that had to. Stuff compiles.
* More stuff compiles, fix most tests
* Big merge, pushing down opening the keychain forced adding factory methods on trait
* Test fixes for pool and servers crate
2018-06-08 06:21:54 +01:00
Quentin Le Sceller
1255ea2224
Upgrade and uniformize dependencies (#1138)
* Remove unused DandelionConfig

* Cargo upgrade and cleanup
2018-06-05 13:26:32 -04:00
Ignotus Peverell
057eb2dcb8
Minor typo fix 2018-06-05 06:46:07 +01:00
Quentin Le Sceller
663733e72b
Replace try! for ? (#1134)
* Replace try for ?
2018-06-04 12:05:49 -04:00
Antioch Peverell
0ecadd3486
move verify_kernel_sums into committed trait (#1131) 2018-06-02 19:00:44 +01:00
Antioch Peverell
6fd2afccb4
block::new() now sets a random pow on the header (#1128) 2018-06-02 15:17:07 +01:00
hashmap
2fa32d15ce Rustify core/src/core (#1122)
Small refactoring of one folder, if it makes sense I could extend the scope.
* Remove some cloning (real and just verbosity in the code)
* Naming conventions like to/into*
* Some Clippy's suggestions

I found that we don't use field init shorthand syntax, so I didn't touch this part, was it discussed before?
2018-06-01 20:41:26 +01:00
Yeastplume
7812a02233
Libwallet refactoring - Library functions + ErrorTypes (#1113)
* move checker and rename to updater

* rustfmt

* complete checker/updater move

* rustfmt

* move libwallet error into separate file

* rustfmt

* starting to sort our error types

* updating errors in libtx and libwallet

* rustfmt

* factor out error type

* rustfmt

* compiling, errors split into libwallet and wallet errors

* rustfmt

* changing user error reporting to new format

* rustfmt

* clean up error types

* clean up error types

* move restore into libwallet

* rustfmt
2018-06-01 15:06:59 +01:00
Antioch Peverell
4fda7a6899
Minimal Transaction Pool (#1067)
* verify a tx like we verify a block (experimental)

* first minimal_pool test up and running but not testing what we need to

* rework tx_pool validation to use txhashset extension

* minimal tx pool wired up but rough

* works locally (rough statew though)
delete "legacy" pool and graph code

* rework the new pool into TransactionPool and Pool impls

* rework pool to store pool entries
with associated timer and source etc.

* all_transactions

* extra_txs so we can validate stempool against existing txpool

* rework reconcile_block

* txhashset apply_raw_tx can now rewind to a checkpoint (prev raw tx)

* wip - txhashset tx tests

* more flexible rewind on MMRs

* add tests to cover apply_raw_txs on txhashset extension

* add_to_stempool and add_to_txpool

* deaggregate multi kernel tx when adding to txpoool

* handle freshness in stempool
handle propagation of stempool txs via dandelion monitor

* patience timer and fluff if we cannot propagate
to next relay

* aggregate and fluff stempool is we have no relay

* refactor coinbase maturity

* rewrote basic tx pool tests to use a real txhashset via chain adapter

* rework dandelion monitor to reflect recent discussion
works locally but needs a cleanup

* refactor dandelion_monitor - split out phases

* more pool test coverage

* remove old test code from pool (still wip)

* block_building and block_reconciliation tests

* tracked down chain test failure...

* fix test_coinbase_maturity

* dandelion_monitor now runs...

* refactor dandelion config, shared across p2p and pool components

* fix pool tests with new config

* fix p2p tests

* rework tx pool to deal with duplicate commitments (testnet2 limitation)

* cleanup and address some PR feedback

* add big comment about pre_tx...
2018-05-30 16:57:13 -04:00
Yeastplume
82ed280625
[WIP] Factoring out wallet traits and continued wallet library work (#1096)
* rename wallet libs

* rename transaction.rs to slate.rs

* rename transaction.rs to slate.rs

* move some types into libwallet/types

* rustfmt

* rename libtransaction libtx

* rename libtransaction libtx

* change types.rs to file_wallet

* rustfmt

* rename WalletData to FileWallet

* refactoring WalletBackend types out

* rustfmt

* All compiling, at least

* rustfmt

* fix FileWallet paths to persist

* rustfmt

* re-ignore wallet integration test
2018-05-30 17:48:32 +01:00
Quentin Le Sceller
ec3c2e89a4 Store and communicate ban reason (#1085)
* Add read_i32 and write_i32
* Store and send reason for ban
* Client corruption ban reason
* Forgotten max message size
2018-05-29 03:45:31 +01:00
Yeastplume
85285473bd
[WIP] Wallet refactor - part 3 (#1072)
* Beginning to rework aggsig library workflow

* more refactoring of transaction api

* whoever does round 1 first creates offset

* slate finalisation now context-free, so anyone can do it

* remove concept of transaction phase

* remove slate phase enum

* update actual send/receive code with new transaction lib workflow
2018-05-21 16:28:11 +01:00
Yeastplume
4bbaa8d05f
Move aggsig transaction building functions into separate lib (#1061)
* refactoring transaction building code

* serialise return transaction

* move shared functions into transactions, ensure wallet manipulation is only done outside of aggsig transaction lib

* remove unneeded wallet config from fn

* adding test functions to facilitate libwallet transaction testing

* rustfmt

* refactoring checker somewhat, adding ability to create and transactions against local copy of chain for simpler testing

* finish transaction testing functionality which verifies transactions work properly

* Remove wallet output manipulation from transaction building lib

* ensure sender expects full transaction back on last phase

* ensure sender expects full transaction back on last phase
2018-05-16 13:18:09 +01:00
Antioch Peverell
0d9242e5dc
refactor on master, pass in offset as blinding_factor (#1064) 2018-05-13 13:59:27 -04:00
Yeastplume
4121ea1240
Wallet+Keychain refactoring (#1035)
* beginning to refactor keychain into wallet lib

* rustfmt

* more refactor of aggsig lib, simplify aggsig context manager, hold instance statically for now

* clean some warnings

* clean some warnings

* fix wallet send test a bit

* fix core tests, move wallet dependent tests into integration tests

* repair chain tests

* refactor/fix pool tests

* fix wallet tests, moved from keychain

* add wallet tests
2018-05-09 10:15:58 +01:00
Antioch Peverell
f90506d869
txhashset extension now implements Committed (#1049)
* txhashset now implements committed for consistency

* rustfmt and cleanup
2018-05-08 10:23:33 -04:00
Antioch Peverell
4dd94ff39e
[testnet2] Store output sum (#1043)
* block sums and reworked block validation
read and write block_sums
refactor validate on both block and txhashset
write block_sum on fast sync
we store the kernel_sum (need to account for the offset)

* block_sums

* rustfmt

* cleanup
2018-05-07 09:21:41 -04:00
Antioch Peverell
e22d025dc8 Simplify block marker metadata (#985)
* wip BlockMarker struct, get rid of PMMRMetadata
* use rewind to init the txhashet correctly on startup, we do not need to track pos via metadata (we have block markers), we do not need to open the txhashset with specific pos (we have rewind)
* better logging on init
* keep rewinding and validating on init, to find a good block
* use validate_roots on chain init
2018-04-24 19:53:01 +00:00
Quentin Le Sceller
55f6e3e63f
Anti-aggregation mechanism for multi-kernel transaction (#984)
* Test multi kernel deaggregation

* Add aggregate without cut_through and deaggregate function

* Add deaggregate function in pool and test

* Rustfmt

* Add deaggregate_and_add_to_memory_pool

* Deaggregate regular multi kernel transaction by default

* Rustfmt

* Add error type faileddeaggregation

* Add find candidates function

* Rustfmt

* Use intersection of sets instead of for comparisons

* Rustfmt

* Removed unnecessary if

* Stricter verification with is_subset

* Rustfmt
2018-04-24 15:47:13 -04:00
Yeastplume
4c6a193e01
[WIP] Miner removal (#979)
* Beginning to remove in-process miner

* rustfmt

* rustfmt

* rustfmt

* rustfmt

* remove pow crate and put remnants into core

* rustfmt

* fix test compilation in core and chain

* rustfmt

* Updating server tests to use test miner

* rustfmt

* rustfmt

* remove pow from test matrix

* adding basic stratum stats to TUI

* run stratum server at all times, and halt messages while syncing

* fix core tests

* add ability to run internal test miner for cuckoo 16 testing

* modify build instructions
2018-04-24 09:18:24 +01:00
Antioch Peverell
ef304ee9c1
add block header cache to store (#996)
* WIP - add blocks header cache to store
The DifficultyIter is expensive to use when validating block headers.

* lru_cache

* rustfmt

* cleanup
2018-04-23 14:55:25 -04:00
Antioch Peverell
ced6e647b8
still do not fully understand what PhantomData is *for* exactly (#991)
but standardized naming and usage to match https://doc.rust-lang.org/beta/nomicon/phantom-data.html
so at least it looks similar to the official docs
2018-04-21 17:03:45 -04:00
Antioch Peverell
f0cf903adc
fix various issues with rewinding data file (#983)
* fix issue with rewinding data file
also fix fragility around an empty MMR (and the memory map)

* cleanup
2018-04-19 19:52:46 +01:00
Ziyuan Liu
a8cc58bb89 Remove oddfee error references + validation check (#975) 2018-04-18 12:21:21 +01:00
Quentin Le Sceller
bd64c6099f Add aggregate function for multi-kernel transaction (#966)
* Add aggregate function

* Rustfmt

* Add multikernel tx test

* Rustfmt

* Add test bad multi-kernel transaction

* Add sorting

* Modified multikernel test
2018-04-17 21:40:51 +01:00
hashmap
7bad33d249 Merkle proofs fee (#898)
* Refactoring before new functionality

* Simplify and use just count of proofs
2018-04-17 18:09:29 +01:00
Yeastplume
dcdf654bc9
[WIP] Core PMMR and API updates to support wallet restore (#950)
* update pmmr to get batch of elements by insertion position

* update pmmr to get batch of elements by insertion position

* add api + chain calls to get traversed outputs back out

* add api + chain calls to get traversed outputs back out

* first pass getting wallet restore to work again with updated utxo-walking api
2018-04-11 10:02:07 +01:00
Antioch Peverell
6ceed79c03 Use hashable_ord on short_ids (#932) 2018-04-05 15:48:30 +00:00
Riccardo Casatta
40212942da use std lib function in pmmr for all_ones() and most_significant_pos() (#924)
* use std lib function in pmmr for all_ones() and most_significant_pos()

* fix all_ones() edge case with 0 and add comparison tests with old functions
2018-04-04 13:25:03 +01:00
John Tromp
c3939107a5 Avoid repeated truncation in difficulty adjust + minor refactoring (#915)
* avoid repeated truncation in difficulty adjust + minor refactoring
* reinstate needed low-diffculty exception on dampening
* leave consensus breaking change for later one line change; obsolete lower testnet1 max target
2018-03-30 18:21:06 +00:00
Simon B
3e3fe6cae6 Testnet2 small fixes (#899)
* add hint for gcc-5 for building CUDA plugin
* add N_BLOCks comment for >=6GB GPU cards
* minor logging cleanup
* mark to be kept + reformat as "@ height [hash]"
* show CuckooNN + better wording
2018-03-29 15:56:46 +00:00
Yeastplume
19bedff664
Difficulty scenario fix (#892)
* fix difficulty scenario to match testnet2

* fix difficulty scenario to match testnet2
2018-03-27 22:11:52 +01:00
Yeastplume
6a6e600e48
tests for difficulty adjustment scenarios (#887)
* tests for difficulty adjustment scenarios

* ignore test
2018-03-27 19:15:42 +01:00
Ignotus Peverell
16beeca8c9
Correct chain initialization at genesis 2018-03-26 17:38:27 +01:00
Ignotus Peverell
74674c2307
Hopefully last genesis update... 2018-03-26 17:36:54 +01:00
Ignotus Peverell
aa0e488e07
Genesis update with correct nonce... 2018-03-26 17:32:10 +01:00
Ignotus Peverell
e0594d81fc
Genesis update with correct month 2018-03-26 17:31:13 +01:00
Ignotus Peverell
f3fc26c5c9
Genesis update with 172 difficulty 2018-03-26 16:12:18 +01:00
Ignotus Peverell
327293d28e Genesis, version changes and doc for testnet2 (#865)
* Bump up crates versions

* Finally add a Cargo.lock to avoid dependency breakages

* Build doc update for testnet2

* Fix test framework not really using its mining config

* Testnet2 genesis, best so far at 128 difficulty (a nice number)

* Minor build doc update
2018-03-26 10:48:46 +01:00
John Tromp
a327427178 0-based positions, bag peaks right-to-left with mmr size as index (#863)
* 0-based positions, bag peaks right-to-left with mmr size as index
* fix remaining instances of hash_with_index
* fix pmmr test
2018-03-26 02:26:11 +00:00
Simon B
9285de2c24 Here is a case where ~ is needed, to allow cargo update to only touch the patch level version (third number). Anything else than ~ means that both first numbers can be raised, so "2.1" updaates to "2.2" (#866) 2018-03-25 20:08:38 +00:00
Ignotus Peverell
2bf656646c
Force slog version to 2.1 (2.2 broke some traits) 2018-03-25 19:54:41 +01:00
Simon B
f9b87164e7 Idiomatic cargo toml (#862)
* Specify versions SemVer-style
 - slog and serde
* Comments for not update-able pre-1.0 crates:
 - urlencoded at "0.5" since 0.6+ lacks trait `plugin::Plugin<iron::Request<'_, '_>>`
* alpha-sort deps
* Specify versions SemVer-style
 - backtrace, bitflags, blake2-rfc, bodyparser, built, byteorder
 - chrono, clap, demonize, enum_primitive, env_logger
 - failure, failure_derive, futures
 - hyper, iron, itertools, lazy_static, libc
 - memmap, mount, net2, num_bigint, prettytable-rs
 - rand, regex, rocksdb, router, siphasher
 - term, time, tokio-core, tokio-retry, urlencoded, uuid, walkdir, zip
2018-03-25 17:44:27 +00:00
Alexey Miroshkin
bb134758e7 Rename Hash::zero() to Hash::default() (#858)
It is more idiomatic, also `zero` is misleading in context of `+` operator.
2018-03-23 23:35:10 +00:00