Commit graph

249 commits

Author SHA1 Message Date
Quentin Le Sceller
be4779c923
Less cloning and pattern simplifications (#3305)
* Cleanup 
* Pattern simplification
2020-04-24 11:19:34 +02:00
Antioch Peverell
bab9f41343
go back to storing header_head in the db (#3268)
* go back to storing header_head in the db

* simplify and cleanup
2020-04-10 16:12:53 +01:00
Antioch Peverell
e49eecae5c
Cleanup path handling with AsRef<Path> (#3284)
* AsRef<Path>

* bump
2020-03-30 11:35:21 +01:00
Antioch Peverell
3bbf60ee92
simplify when block_sums and spent_index are added to the db (#3253)
* simplify when block_sums and spent_index are added to the db

* fix pool tests

* cleanup
2020-03-10 14:58:32 +00:00
jaspervdm
d5b523248b
API: don't error on missing output (#3256)
* Node API: don't error on missing output

* Propagate errors from get_output*

* Rename is_unspent and small refactor

* Forgot to rename function in tests

* Change Batch get_output_pos_height type signature
2020-03-04 23:42:10 +01:00
Antioch Peverell
cb2b909090
Block input bitmap rework (#3236)
* first pass at rewind_single_block
and reworking rewind to simply iterate over blocks, rewinding each incrementally

* commit

* commit

* cleanup

* add test coverage for output_pos index transactional semantics during rewind

* commit

* do not store commitments in spent_index
just use the order of the inputs in the block

* compare key with commitment when cleaning output_pos index

* remove unused OutputPos struct
2020-02-24 09:48:51 +00:00
Antioch Peverell
2d4a2c30ce
Cleanup output pos idx (#3227)
* cleanup legacy output pos index
cleanup unused migration code
consolidate use of output_pos index

* refactor to cleanup duplicate code
2020-02-14 20:42:07 +00:00
Antioch Peverell
3c040b96e1
account for the utxo rewind when attempting to fix a bad chain init (#3229) 2020-02-14 15:03:45 +00:00
Antioch Peverell
4c081b8f73
use backend file when compacting the output pos index (#3226) 2020-02-13 10:26:56 +00:00
Quentin Le Sceller
04a0123752
Less cloning and additional pattern simplifications (#3223)
* API Cleanup

* Chain Cleanup

* Core Cleanup

* Keychain Cleanup

* P2P Cleanup

* Pool Cleanup

* Store Cleanup

* Util Cleanup

* Cleanup clone_from_slice

* Address jasper comments
2020-02-12 19:35:33 +01:00
Quentin Le Sceller
c4e69717ab
Less cloning and pattern simplifications (#3216)
* Less cloning and pattern simplifications

* Revert inclusive range and remove unecessary Error:From
2020-02-05 11:02:07 -05:00
Antioch Peverell
616dad43fd
Cleanup redundant AsFixedBytes and FixedLength traits (#3131)
* no need for AsFixedBytes we can just use AsRef<[u8]>

* cleanup FixedLength trait

* revert this change for now

* fix store tests

* cleanup and fix tests after rebase

* fix tests

* update based on PR review
less hard-coded values now

* cleanup
2020-01-29 13:41:50 +00:00
Antioch Peverell
9ec9d04457 Refactor to pass batch around explicitly (#3176)
* Refactor to pass batch around explicitly rather than keeping it in the extension.
We would like to be able to pass an extension around and potentially clone it.

* cleanup, pass header around where it makes sense
2020-01-28 12:23:11 -05:00
Quentin Le Sceller
6e5afe496b Update License to 2020 (#3196) 2020-01-20 11:40:58 +00:00
Quentin Le Sceller
cdb2d6c72c Node API v2 (#3094)
* Node API v2

* Update gitignore

* Add get_pmmr_indices method

* Add Foreign and Owner API each one with specific secret

* Fix failing tests

* Revert to 'Option<u64>'
2019-12-06 10:57:53 +00:00
Antioch Peverell
0b21ee607a
height (header version) specific output PMMR root rules (#3147)
* height (header version) specific output PMMR root rules

* cleanup

* cleanup based on PR feedback

* address feedback
2019-12-05 11:55:10 +00:00
Antioch Peverell
3a333ae00f
deadlock risk in check_orphans called from txhashset_write (#3156) 2019-12-05 10:32:01 +00:00
Antioch Peverell
e2795b1593
call zip_read fro within the txhashet lock for a consistent view on the files (#3142) 2019-11-26 21:17:28 +00:00
Antioch Peverell
11ac7d827a
Enable faster sync (#3108)
* add bitmap accumulator
refactor vec backend so we can use it outside of tests
introduce a "hash only" vec backend for the accumulator

* get core tests passing

* initial test coverage for bitmap_accumulator

* better test coverage for bitmap accumulator and cleanup code

* refactor txhashset roots, call validate() on roots during block validation

* fix store tests

* log the "merged" root when validating roots

* cleanup, revise based on feedback

* cleanup

* rework it to pass explicit size into bitmap accumulator when applying
2019-11-26 20:21:49 +00:00
Antioch Peverell
78220febed
use try_read_for() with timeout for tui status updates (header and pool stats) (#3129)
* use try_read_for with timeout for tui status updates (header and pool stats)

* rework based on feedback
2019-11-22 12:46:07 +00:00
Joseph Goulden
29b871841a fix: for TUI status try to acquire read lock on pmmr_header but if no… (#3119)
fix: for TUI status try to acquire read lock on pmmr_header but if not available just leave the header stats alone
2019-11-15 08:21:42 +01:00
Yeastplume
38e6497919
Retrieve outputs within a block height range (#3103)
* add function to retrieve a set of pmmr indices between a given block height range

* typo

* change APU to just return required indices

* change pmmr index retrieval, change new function to only return pmmr indices between blocks
2019-11-04 15:04:21 +00:00
Antioch Peverell
a362888ab9
read header_head and sync_head from the header and sync MMR respectively (#3045)
no need to maintain header_head and sync_head in the db explicitly
2019-10-29 16:47:08 +00:00
Quentin Le Sceller
6be6391225 Change 2018 to 2019 in copyright files (#3072) 2019-10-02 09:40:20 +01:00
Antioch Peverell
973a03c826
support for v2 txhashset (but only using v1 in mainnent and floonet) (#3051) 2019-09-24 09:57:34 +01:00
Antioch Peverell
1c072f535c
DB v1->v2 migration (#3044)
* wip

* exhaustive match

* wip

* cleanup docs and move migration fns to chain itself
2019-09-19 21:00:14 +01:00
Joseph Goulden
02cee80229 fix: Add some more stats to basic status page of TUI (disk usage, chain timestamp, tx pool size) (#3046)
* fix: Add some more stats to basic status page of TUI (disk usage, chain timestamp, tx pool size)

* chore: add latest header timestamp to TUI

* fix: calculate total disk usage of database to show in TUI
2019-09-19 20:57:26 +01:00
Gary Yu
7a07541c26
Clean the unused build_index (#3026) 2019-09-10 07:55:24 +08:00
Gary Yu
fa0fed9de5
fix: find common header between the header chain and the current body chain (#3033)
* fix: find common header between the header chain and the current body chain

* add some code comments
2019-09-10 07:54:42 +08:00
Antioch Peverell
383985292c Split header MMR (and sync MMR) out from txhashset (#3004)
* wip

* sync sort of works now

* get rid of the deadlock during compaction
there is *always* a deadlock in there when we make changes like this...

* cleanup how we rebuild the sync MMR on init

* cleanup rewind logic

* roll the "fix invalid root" changes into this PR

* move rebuild_height_pos_index into txhashset
and pass in header_pmmr for header lookups

* cleanup and remember to setup sync head on init

* cleanup unnecessary ref muts

* rebuild_height_pos_index when writing txhashset
2019-09-07 07:28:26 +08:00
Gary Yu
d55ebe8a2b fix: the race condition in compact (#3015) 2019-09-04 19:51:28 +01:00
Antioch Peverell
b291467b96 use head (not header_head) when rewinding full blocks (#3017)
* use head (not header_head) when rewinding to find fork point for full blocks

* get rid of shortcut when nothing to rewind - always something to rewind
2019-09-04 08:02:45 +08:00
jaspervdm
30156cdcf9 Kernel lookup api method (#3000)
* Kernel lookup api

* Min and max height parameters, scan backwards

* Return null instead of 404 if kernel not found

* Return TxKernel instead of TxKernelPrintable

* Update description of KernelHandler
2019-08-30 10:03:12 +01:00
Gary Yu
d36a0b29ef store both mmr index and block height into database for output (#2903)
* store both mmr index and block height into database for output

* rustfmt

* fix: mmr position is 1-based instead of 0-based

* (Hash, u64, u64) deserves a type
2019-08-29 17:14:39 +01:00
Antioch Peverell
dcd405e263
cleanup deprecation warnings about missing dyn with trait objects (#2997) 2019-08-26 21:17:47 +01:00
j01tz
d220410571 Improve error handling when computing PMMR roots (#2988)
* Improve error handling for invalid PMMR roots

* Update tests that rely on pmmr root

* Fix pmmr store tests
2019-08-09 15:10:54 +01:00
Quentin Le Sceller
f79d05ba53 TxHashSet Download Improvement (#2984)
* Ban on cannot get block header in txhashset_write

* Rusfmt

* Fix typo

* Missing error handling

* Rustfmt

* Only accept txhashset from corresponding peer

* Switch to AtomicBool instead of RwLock<bool>

* Rustfmt
2019-08-01 17:46:06 +01:00
Antioch Peverell
515fa54614
Rewind head and header_head consistently. (#2918)
* maintain header_head as distinctly separate from head

* cleanup corrupted storage log msg

* simplify process_header and check_header_known

* remember to commit the batch when successfully processing a header...

* rework sync_block_headers for consistency with process_block_header

* cleanup unrelated code

* fix pool tests

* cleanup chain tests

* cleanup chain tests (reuse helpers more)

* cleanup - head not header on an extension
shortcircuit "rewind and apply fork" for headers if next header
2019-07-26 08:36:24 +01:00
Quentin Le Sceller
4966dc04f7
[2.x.x] Generate txhashset archives on 720 block intervals. (#2951)
Generate txhashset archives on 720 block intervals.
2019-07-23 10:46:29 +02:00
hashmap
d3dbafa80b
[2.x.x] Use blocking IO in P2P to reduce CPU load (#2855)
* Use blocking IO in P2P to reduce CPU load
2019-07-12 23:17:38 +02:00
Antioch Peverell
2348214719
fix our check_known check for blocks/headers (#2943) 2019-07-11 17:08:36 +01:00
Antioch Peverell
d284d8f6de
[2.x.x] Writeable protocol version aware (#2856)
* introduce protocol version to deserialize and read

* thread protocol version through our reader

* cleanup

* cleanup

* streaming_reader cleanup

* Pass protocol version into BinWriter to allow for version specific serialization rules.

* rustfmt

* read and write now protocol version specific
2019-07-06 15:51:03 +01:00
Antioch Peverell
5aaf2d058d
[2.x.x] deserialization now protocol version aware (#2824)
* introduce protocol version to deserialize and read

* thread protocol version through our reader

* example protocol version access in kernel read

* fix our StreamingReader impl (WouldBlock woes)

* debug log progress of txhashset download
2019-06-27 17:19:41 +01:00
eupn
f9c5505e9f Report reorg depth in block status (#2839)
* Calculate reorg depth in BlockStatus::Reorg enum member

* rustfmt

* Fix reorg height calculation and implement reorg test

* rustfmt

* Report reorg depth in webhook payload

* Add optional depth field to the block webhook JSON reply
2019-05-24 21:44:28 +02:00
hashmap
9ab23f6eef Implement graceful shutdown (#2812)
* Remove stop status mutex
* remove some deadlocks
* Rewrite stop channel handling
* fix deadlock in peers object
* add missing test fixes
2019-05-15 08:51:35 -07:00
Antioch Peverell
e56cd55980
initial support for kernel data download (#2765)
* initial support for kernel data download

* fix vec backend for tests

* cleanup after rebase
2019-05-14 17:17:38 +01:00
Daogang Tang
9ac4143bc9 fix: try to fix issue #2585 by adding block cleanup from db directly. (#2815)
* fix: try to fix issue #2585 by adding block cleanup from db directly.

Signed-off-by: Mike Tang <daogangtang@gmail.com>

* use another effective algorithm to do old block and short-lived block cleaup.

Signed-off-by: Mike Tang <daogangtang@gmail.com>

* 1. rename iter_lived_blocks to blocks_iter;
2. comments and iterator calling optimiztions.

Signed-off-by: Mike Tang <daogangtang@gmail.com>

* Fix locking bug when calling is_on_current_chain() in batch.blocks_iter just by removing it.
Because "we want to delete block older (i.e. lower height) than tail.height".

Signed-off-by: Mike Tang <daogangtang@gmail.com>
2019-05-13 16:10:31 +01:00
Gary Yu
97e96e4fcf use a grin specific tmp dir instead of a system tmp dir (#2753) (#2772)
* use a grin specific tmp dir instead of a system tmp dir

* rustfmt

* using create_new instead of create

* switch the txhashset sandbox from system tmp dir to grin specific tmp dir

* rustfmt

* adapter get_tmp_dir delegate to chain implementation

* simply the logic of protolc for get_tmpfile_pathname

* rustfmt

* fix the merge mistake
2019-04-22 16:54:36 -07:00
Gary Yu
b61b773cee Get back the lost code of #2575 (#2771) 2019-04-22 16:53:24 -07:00
Yeastplume
12fe928112
merge master into 1.1.0 2019-04-16 11:24:43 +01:00