Commit Graph

30 Commits

Author SHA1 Message Date
ledgerwatch
623545a47c
Fix index chunks (#478)
* First stab at fixing index chunks

* Fix compilation

* Fix tests

* Fix walkAsOf for storage

* Fix

* Fix for walkAsOf accounts

* Fix linter
2020-04-21 09:15:40 +01:00
b00ris
a9f71298aa
Index chunking (#473)
* index generation

* save state

* save state

* save state

* save state

* save state

* save state

* save state

* save state

* fix

* save state

* fix
2020-04-20 11:35:33 +01:00
Igor Mandrigin
acbf8bcbdf
Enable thin history by default (#456)
* Enable thin history by default

* remove thin history flags

* remove from tests

* fix test compile error

* test fixups

* fix more stuff
2020-04-15 12:33:22 +03:00
ledgerwatch
ad64ae2085
[THIN_HISTORY] Decouple changesets and history from ethdb.Database (mutation/bolt_db/badger_db) (#432)
* Prepare for refactoring

* Decoupling

* Further cleanup of mutation

* Further cleanup of mutation

* Remove some tests (temporarily)

* Fix linter

* Fix lint

* Fix lint

* Fix lint

* Fix lint

* Fix lint

* Fix lint

* Recover mutation_test cases

* Fix lint

* Fix WalkAsOf test

* Fix lint

* Fix TODO, linter

* Fix lint

* Fix lint

* Fix tests for THIN_HISTORY

* Fix lint

* Fix lint

* Reduce visibility of DecodeStorage

* Revert "Reduce visibility of DecodeStorage"

This reverts commit 2e11e16ea095b75a62cdddb77b9477eabc259e67.
2020-04-09 18:23:29 +01:00
Alex Sharov
9324d83cb2
ethdb.KV interface - pase 2 (#420)
* - handle cursor.Prefix on server
- move state reports to KV interface

* add CmdCursorSeekKey

* tests for abstract_kv

* avoid reading configs of databases

* avoid reading configs of databases

* make linter happy

* make linter happy

* cleanup

* port badger features from original implementation

* try to fix test

* try to fix test

* .Close() don't return error anymore - defer friendly

* try to enable badger now

* try to enable badger now

* badger can't run on CI yet

* badger can't run on CI yet

* re-run ci

* skip ctx cancelation for badger
2020-04-04 08:18:10 +01:00
Alex Sharov
b490192e67
Use KV Abstraction in RestAPI (#400)
* Introduce NoValuesCursor. From() method is useless because can be replaced by Seek().`
* implement NoValueCursor interface
* use abstract db in restapi
* cleanup .md
2020-03-24 09:12:55 +07:00
Alex Sharov
965d1101f4
Intermediate hash phase 4 (#396)
* another way to check if account has storage

* cleanup

* v0 of walk by db version

* save progress, to switch to another task. Put tombstone is still not correct.

* place tombstone only if exists something to hide

* db-based implementation

* db-based implementation

* db-based implementation

* fix prop check

* improve prop check logic

* Need custom logic to skip subtree for account and storage buckets because storage bucket has incarnation in key

* rebase to master

* remove all tombstones when account deleted

* remove all tombstones when account deleted

* added db integrity check

* don't rely on account.Root because it valid only about last incarnation

* remove all tombstones when account deleted

* dial with incarnation in MultiWalk2

* dial with incarnation in MultiWalk2

* when fixedbytes=40 resolver did compare cacheKey with storageKey without removing incarnation

* rebase to master

* rebase to master

Co-authored-by: alex.sharov <alex.sharov@lazada.com>
2020-03-20 11:30:14 +00:00
Alex Sharov
84ae2eb1f8
Intermediate Hashes UI (#395)
* implement NoValueCursor interface

* cleanup

* fix tests

* add more stats data to ui

* can't display error

* re-open DB low-level net interface when changing db

* re-open DB low-level net interface when changing db

* fix problem with displaying errors

* run ci

* improve prop check logic

* storage tombstones integrity checks UI

* storage page

* make DB configurable

Co-authored-by: alex.sharov <alex.sharov@lazada.com>
2020-03-20 10:06:14 +00:00
Andrew Ashikhmin
8585516071
Serve GetNodeData for nodes in memory (Issue #300) 2020-02-06 11:53:09 +01:00
Alex Sharov
ff23980dc6
Intermediate hash phase 2 (#341)
* add env INTERMEDIATE_TRIE_CACHE
* try to use assert.New() pattern
* Fix "maligned" linter warnings to reduce space consumption of structs:

core/types/accounts/account.go:18:14: struct of size 136 bytes could be of size 128 bytes (maligned)
type Account struct {
--
trie/node.go:44:10: struct of size 80 bytes could be of size 72 bytes (maligned)
	duoNode struct {
--
trie/resolve_set.go:28:17: struct of size 56 bytes could be of size 48 bytes (maligned)
type ResolveSet struct {
--
trie/resolver.go:34:15: struct of size 88 bytes could be of size 72 bytes (maligned)
type Resolver struct {
--
trie/visual.go:32:17: struct of size 104 bytes could be of size 96 bytes (maligned)
type VisualOpts struct {
2020-01-31 14:31:50 +07:00
b00ris
2ece0e2277
remove gets (#297) 2020-01-07 13:41:33 +03:00
Evgeny Danilenko
5b4f352acb
Restore Ethash mining (#231)
* initial

* mining

* remove debug

* debug

* restore random seed in the mining tests

* green tests

* fix blockchain tests

* fix lint

* init miner only if asked

* linters

* do not store trie as singlton

* fmt

* new trieDbState constructor
2019-12-10 16:12:21 +03:00
Andrew Ashikhmin
84285a5ab1
Roughly finish with adding BadgerDB (first cut) (#203)
* Bumping GOMAXPROCS for Badger

* fixes related to database size

* Schedule GC for Badger

* pacify linter

* Don't start GC for ephemeral Badger

* Don't log "Value log GC attempt didn't result in any cleanup"

* Start GC in backround

* Bump GC period and IdealBatchSize for Badger

* BadgerDatabase RewindData

* Boolean badger flag -> string database flag

* cosmetic change
2019-11-27 13:15:25 +00:00
Andrew Ashikhmin
1c0cf9be72
BadgerDB: MultiWalk & some other APIs, command line flag, scaffolding for testing (#183)
* BadgerDatabase MemCopy

* fix err shadowing

* BadgerDatabase MultiPut & NewBatch

* Remove goOn from MultiWalk & MultiWalkAsOf

* BadgerDatabase MultiWalk. Scaffolding for testing Badger in blockchain_test

* Badger Flag

* fix error logging

* Split IdealBatchSize between BoltDB and BadgerDB

* NewEphemeralBadger
2019-11-21 16:12:38 +01:00
andrew
0eb9d3915b Add method stubs to BadgerDatabase so it formally implements the Database interface 2019-11-14 13:00:38 +01:00
andrew
62d36ada39 BadgerDatabase Walk 2019-11-13 17:43:26 +01:00
andrew
109217a9da BadgerDatabase Has 2019-11-12 11:40:46 +01:00
andrew
6352fc024c Don't expose Bolt DB in the interface 2019-11-11 20:28:27 +01:00
andrew
a6d0e795c0 BadgerDatabase GetAsOf 2019-11-11 17:02:37 +01:00
andrew
93ff66f900 Merge branch 'master' into badger2 2019-11-07 15:55:57 +01:00
andrew
26cb9bd33b BadgerDatabase GetS 2019-11-07 15:55:21 +01:00
andrew
069343d5a8 BadgerDatabase DeleteTimestamp 2019-11-06 11:24:48 +01:00
andrew
9be775c8bd Port my work from badger after the master rebase 2019-11-05 13:28:36 +01:00
Péter Szilágyi
054412e335
all: clean up and proerly abstract database access 2019-03-06 13:35:03 +02:00
gary rong
a4a2343cdc ethdb, core: implement delete for db batch (#17101) 2018-07-02 11:16:30 +03:00
Martin Holst Swende
017b9f7eac core, ethdb: reuse database batches (#15989)
* leveldb: Update leveldb to 211f780 (poolfix)

* core, ethdb: reuse database batches
2018-01-30 19:03:31 +02:00
Felix Lange
10181b57a9 core, eth/downloader: commit block data using batches (#15115)
* ethdb: add Putter interface and Has method

* ethdb: improve docs and add IdealBatchSize

* ethdb: remove memory batch lock

Batches are not safe for concurrent use.

* core: use ethdb.Putter for Write* functions

This covers the easy cases.

* core/state: simplify StateSync

* trie: optimize local node check

* ethdb: add ValueSize to Batch

* core: optimize HasHeader check

This avoids one random database read get the block number. For many uses
of HasHeader, the expectation is that it's actually there. Using Has
avoids a load + decode of the value.

* core: write fast sync block data in batches

Collect writes into batches up to the ideal size instead of issuing many
small, concurrent writes.

* eth/downloader: commit larger state batches

Collect nodes into a batch up to the ideal size instead of committing
whenever a node is received.

* core: optimize HasBlock check

This avoids a random database read to get the number.

* core: use numberCache in HasHeader

numberCache has higher capacity, increasing the odds of finding the
header without a database lookup.

* core: write imported block data using a batch

Restore batch writes of state and add blocks, tx entries, receipts to
the same batch. The change also simplifies the miner.

This commit also removes posting of logs when a forked block is imported.

* core: fix DB write error handling

* ethdb: use RLock for Has

* core: fix HasBlock comment
2017-09-09 19:03:07 +03:00
Felix Lange
b252589960 ethdb: remove Flush 2015-09-14 23:36:30 +02:00
Felix Lange
8b32f10f16 ethdb: add NewBatch 2015-09-14 23:36:30 +02:00
Felix Lange
8c4dab77ba all: move common.Database to package ethdb 2015-09-14 23:36:30 +02:00