grin/store/src/lib.rs

91 lines
2.2 KiB
Rust
Raw Normal View History

// Copyright 2018 The Grin Developers
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
2016-10-21 03:06:12 +03:00
//! Storage of core types using RocksDB.
#![deny(non_upper_case_globals)]
#![deny(non_camel_case_types)]
#![deny(non_snake_case)]
#![deny(unused_mut)]
#![warn(missing_docs)]
extern crate byteorder;
extern crate croaring;
extern crate env_logger;
extern crate libc;
Wallet LMDB backend (#1151) * Migrate main node store to LMDB In preparation to using LMDB as a wallet database, migrate the node db. There's no point in having 2 key-value stores. In addition LMDB provides a few advantages as a node db, namely a much faster build (compared to RocksDb), lesser dependencies and transactions. * Migrated p2p store to lmdb, stuff compiles * More fixes, chain tests starting to pass * Fixed txhashset rollback messing with block save and general batch delimitation. Chain tests passing. * rustfmt * LMDB max map size of 10MB isn't really workable. Half TB seems reasonable. * Fix wallet tests * Rather crucial commit was missing * rustfmt * Fixing new merged tests following lmdb changes * rustfmt * * Make txhashset validation read-only on fast sync to avoid having a really long open transaction. * Fix deadlock in new block processing, batch should always be created within a txhashset lock (when they interact). * Comment about batch and txhashset interlacing * Fix store tests to use batch * Externalize wallet config and seed * Converted direct read access to file outputs map to an iterator * Cleaned up and simplified wallet Backend trait: * No more direct mutable access to internal structures (HashMap) * Batch interface for all writes * Remove unneeded read wrapper (read_wallet) * rustfmt * First (incomplete) pass at wallet LMDB backend * Progressing on lmdb backent iml * Added batch impl for LMDB wallet backend. Pretty much done with it, but not sure how to deal with commit (owned). * rustfmt * Wrapping LMDB batch around a refcell to work around borrow rules * Compilation up to grin chain
2018-06-22 11:08:06 +03:00
extern crate lmdb_zero;
Prunable MMR storage (#112) * Base MMR storage structures Implementations of the MMR append-only file structure and its remove log. The append-only file is backed by a mmap for read access. The remove log is stored in memory for quick checking and backed by a simple file to persist it. * Add PMMR backend buffer, make PMMR Backend mutable * The Backend trait now has &mut self methods, and an &mut reference in PMMR. This simplifies the implementation of all backends by not forcing them to be interior mutable. Slight drawback is that a backend can't be used directly as long as it's used by a PMMR instance. * Introduced a buffer in the PMMR persistent backend to allow reads before the underlying files are fully flushed. Implemented with a temporary VecBackend. * Implement a prune list to use with dense backends The PruneList is useful when implementing compact backends for a PMMR (for example a single large byte array or a file). As nodes get pruned and removed from the backend to free space, the backend will get more compact but positions of a node within the PMMR will not match positions in the backend storage anymore. The PruneList accounts for that mismatch and does the position translation. * PMMR store compaction Implement actual pruning of the underlying PMMR storage by flushing the remove log. This triggers a rewrite of the PMMR nodes data (hashes and sums), removing pruned nodes. The information of what has been removed is kept in a prune list and the remove log is truncated. * PMMR store pruning tests and fixes
2017-09-05 08:50:25 +03:00
extern crate memmap;
extern crate serde;
#[macro_use]
extern crate slog;
2016-10-21 03:06:12 +03:00
Wallet LMDB backend (#1151) * Migrate main node store to LMDB In preparation to using LMDB as a wallet database, migrate the node db. There's no point in having 2 key-value stores. In addition LMDB provides a few advantages as a node db, namely a much faster build (compared to RocksDb), lesser dependencies and transactions. * Migrated p2p store to lmdb, stuff compiles * More fixes, chain tests starting to pass * Fixed txhashset rollback messing with block save and general batch delimitation. Chain tests passing. * rustfmt * LMDB max map size of 10MB isn't really workable. Half TB seems reasonable. * Fix wallet tests * Rather crucial commit was missing * rustfmt * Fixing new merged tests following lmdb changes * rustfmt * * Make txhashset validation read-only on fast sync to avoid having a really long open transaction. * Fix deadlock in new block processing, batch should always be created within a txhashset lock (when they interact). * Comment about batch and txhashset interlacing * Fix store tests to use batch * Externalize wallet config and seed * Converted direct read access to file outputs map to an iterator * Cleaned up and simplified wallet Backend trait: * No more direct mutable access to internal structures (HashMap) * Batch interface for all writes * Remove unneeded read wrapper (read_wallet) * rustfmt * First (incomplete) pass at wallet LMDB backend * Progressing on lmdb backent iml * Added batch impl for LMDB wallet backend. Pretty much done with it, but not sure how to deal with commit (owned). * rustfmt * Wrapping LMDB batch around a refcell to work around borrow rules * Compilation up to grin chain
2018-06-22 11:08:06 +03:00
#[macro_use]
extern crate grin_core as core;
extern crate grin_util as util;
pub mod leaf_set;
2018-06-22 11:44:38 +03:00
mod lmdb;
pub mod pmmr;
pub mod rm_log;
pub mod types;
Prunable MMR storage (#112) * Base MMR storage structures Implementations of the MMR append-only file structure and its remove log. The append-only file is backed by a mmap for read access. The remove log is stored in memory for quick checking and backed by a simple file to persist it. * Add PMMR backend buffer, make PMMR Backend mutable * The Backend trait now has &mut self methods, and an &mut reference in PMMR. This simplifies the implementation of all backends by not forcing them to be interior mutable. Slight drawback is that a backend can't be used directly as long as it's used by a PMMR instance. * Introduced a buffer in the PMMR persistent backend to allow reads before the underlying files are fully flushed. Implemented with a temporary VecBackend. * Implement a prune list to use with dense backends The PruneList is useful when implementing compact backends for a PMMR (for example a single large byte array or a file). As nodes get pruned and removed from the backend to free space, the backend will get more compact but positions of a node within the PMMR will not match positions in the backend storage anymore. The PruneList accounts for that mismatch and does the position translation. * PMMR store compaction Implement actual pruning of the underlying PMMR storage by flushing the remove log. This triggers a rewrite of the PMMR nodes data (hashes and sums), removing pruned nodes. The information of what has been removed is kept in a prune list and the remove log is truncated. * PMMR store pruning tests and fixes
2017-09-05 08:50:25 +03:00
const SEP: u8 = ':' as u8;
use byteorder::{BigEndian, WriteBytesExt};
2016-10-21 03:06:12 +03:00
Wallet LMDB backend (#1151) * Migrate main node store to LMDB In preparation to using LMDB as a wallet database, migrate the node db. There's no point in having 2 key-value stores. In addition LMDB provides a few advantages as a node db, namely a much faster build (compared to RocksDb), lesser dependencies and transactions. * Migrated p2p store to lmdb, stuff compiles * More fixes, chain tests starting to pass * Fixed txhashset rollback messing with block save and general batch delimitation. Chain tests passing. * rustfmt * LMDB max map size of 10MB isn't really workable. Half TB seems reasonable. * Fix wallet tests * Rather crucial commit was missing * rustfmt * Fixing new merged tests following lmdb changes * rustfmt * * Make txhashset validation read-only on fast sync to avoid having a really long open transaction. * Fix deadlock in new block processing, batch should always be created within a txhashset lock (when they interact). * Comment about batch and txhashset interlacing * Fix store tests to use batch * Externalize wallet config and seed * Converted direct read access to file outputs map to an iterator * Cleaned up and simplified wallet Backend trait: * No more direct mutable access to internal structures (HashMap) * Batch interface for all writes * Remove unneeded read wrapper (read_wallet) * rustfmt * First (incomplete) pass at wallet LMDB backend * Progressing on lmdb backent iml * Added batch impl for LMDB wallet backend. Pretty much done with it, but not sure how to deal with commit (owned). * rustfmt * Wrapping LMDB batch around a refcell to work around borrow rules * Compilation up to grin chain
2018-06-22 11:08:06 +03:00
pub use lmdb::*;
/// An iterator thad produces Readable instances back. Wraps the lower level
/// DBIterator and deserializes the returned values.
Wallet LMDB backend (#1151) * Migrate main node store to LMDB In preparation to using LMDB as a wallet database, migrate the node db. There's no point in having 2 key-value stores. In addition LMDB provides a few advantages as a node db, namely a much faster build (compared to RocksDb), lesser dependencies and transactions. * Migrated p2p store to lmdb, stuff compiles * More fixes, chain tests starting to pass * Fixed txhashset rollback messing with block save and general batch delimitation. Chain tests passing. * rustfmt * LMDB max map size of 10MB isn't really workable. Half TB seems reasonable. * Fix wallet tests * Rather crucial commit was missing * rustfmt * Fixing new merged tests following lmdb changes * rustfmt * * Make txhashset validation read-only on fast sync to avoid having a really long open transaction. * Fix deadlock in new block processing, batch should always be created within a txhashset lock (when they interact). * Comment about batch and txhashset interlacing * Fix store tests to use batch * Externalize wallet config and seed * Converted direct read access to file outputs map to an iterator * Cleaned up and simplified wallet Backend trait: * No more direct mutable access to internal structures (HashMap) * Batch interface for all writes * Remove unneeded read wrapper (read_wallet) * rustfmt * First (incomplete) pass at wallet LMDB backend * Progressing on lmdb backent iml * Added batch impl for LMDB wallet backend. Pretty much done with it, but not sure how to deal with commit (owned). * rustfmt * Wrapping LMDB batch around a refcell to work around borrow rules * Compilation up to grin chain
2018-06-22 11:08:06 +03:00
// pub struct SerIterator<T>
// where
// T: ser::Readable,
// {
// iter: DBIterator,
// _marker: marker::PhantomData<T>,
// }
//
// impl<T> Iterator for SerIterator<T>
// where
// T: ser::Readable,
// {
// type Item = T;
//
// fn next(&mut self) -> Option<T> {
// let next = self.iter.next();
// next.and_then(|r| {
// let (_, v) = r;
// ser::deserialize(&mut &v[..]).ok()
// })
// }
// }
/// Build a db key from a prefix and a byte vector identifier.
pub fn to_key(prefix: u8, k: &mut Vec<u8>) -> Vec<u8> {
let mut res = Vec::with_capacity(k.len() + 2);
res.push(prefix);
res.push(SEP);
res.append(k);
res
}
/// Build a db key from a prefix and a numeric identifier.
pub fn u64_to_key<'a>(prefix: u8, val: u64) -> Vec<u8> {
let mut u64_vec = vec![];
u64_vec.write_u64::<BigEndian>(val).unwrap();
u64_vec.insert(0, SEP);
u64_vec.insert(0, prefix);
u64_vec
}