Commit Graph

80 Commits

Author SHA1 Message Date
Alex Sharov
61fbe34024
Better cbor support (#1163) 2020-10-02 10:56:13 +07:00
Alex Sharov
48cf5fb43e
History-style sharding for bitmap indices (#1159) 2020-10-02 10:54:11 +07:00
ledgerwatch
7bb9cb4274
Remove BloomStatus from the API (#1155)
* Remove BloomStatus from the API

* Trying to find missing space

* Fix lint
2020-09-29 15:09:29 +01:00
Alex Sharov
e02d6acc7d
bitmap indices for logs (#1124)
* save progress

* try now

* don't create bloom inside rlpDecode

* don't create bloom inside ApplyTransaction

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* rename method

* print timings

* print timings

* print timings

* sort before flush

* fix err lint

* clean

* move tests to transactions

* compressed version

* up bound

* up bound

* more tests

* more tests

* more tests

* more tests

* better removal

* clean

* better performance of get/put methods

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* optimize rpcdaemon

* fix test

* fix rpcdaemon

* fix test

* simplify

* simplify

* fix nil pointer

* clean

* revert some changes

* add some logs

* clean

* try without optimize

* clean

* clean

* clean

* clean

* try

* move log_index to own stage

* move log_index to own stage

* integration add log_index stage

* integration add log_index stage

* clean

* clean

* print timing

* remove duplicates at unwind

* extract truncateBitmaps func

* try detect

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* add blackList of topics

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* sharding 1

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 3

* sharded 3

* sharded 3

* speedup things by putCurrent and putReserve

* clean

* optimize trim

* clean

* remove blacklist

* add more info to err

* ?

* clean

* clean

* clean

* clean

* clean

* working version

* switch to cgo version of roaring bitmaps

* clean

* clean

* clean

* clean

* more docs

* clean

* clean

* fix logs bloom field

* Fix debug_getModifiedAccountsByNumber

* Try to fix crash

* fix problem with "absent block"

* fix problem with "absent block"

* remove optimize method call

* remove roaring iterator

* fix problem with rebuild indicess

* remove debug prints

* tests for eth_getLogs involving topics

* add tests for new stage, speparate topics into 2 buckets

* version up

* remove debug logs

* remove debug logs

* remove bloom filter implementation

* Optimisation

* Optimisatin not required, make rpctest lenient to geth errors

* Lenient to geth failures

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-09-28 18:18:36 +01:00
ledgerwatch
0f24a01677
Close the ticker (#1104) 2020-09-11 21:44:24 +01:00
Giulio rebuffo
23d89c0941
Added TLS handshake to RPCDaemon (#1089)
* added eth_getStorageAt

* used uint32

* now its 256

* incarnation

* added TLS handshake

* code minified

* lint

* minified client

* Update flags.go

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-09-11 21:17:37 +01:00
Alex Sharov
fd9c7c3b2d
kill bolt (#1085)
* kill bolt

* kill bolt

* fix test

* clean
2020-09-09 11:09:55 +01:00
Alex Sharov
afe808c88e
time based reopen tx (#1069) 2020-09-07 12:47:08 +07:00
Evgeny Danilenko
e4f495fa44
Get logs (#1028)
* it compiles

* after recent master

* fix linters warnings

* grpcV7

* go mod tidy

* unmarshall adresses or adress

* fix linters

* after cr

* after cr

* after cr

* after cr

* fix tests

* remove dev version

* it compiles

* mod tidy

* fix bin deps

* use stable version of grpc

* switch back to master constructor

* switch back to master constructor

* add a bit docs

* add a bit docs

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
2020-09-03 08:51:19 +01:00
Alex Sharov
07ca5c643a
eth_syncing (#991)
* eth_syncing

* linters

* fix readme

* cleanup cli params

* go mod tidy

* remove memsize ui dep

* remove bloomfilter

* implement net_peerCount

* remove legacy flags
2020-08-29 08:24:50 +01:00
Evgeny Danilenko
3b40819444
grpc regenerate (#936)
* grpc regenerate

* use string type for id in net_version
2020-08-18 20:22:49 +03:00
Alex Sharov
081d02920d
Remove "Bucket" class from KV abstraction (#928)
* drop Bucket object from KV abstraction

* add some non-public methods

* rebase master

* rebase master
2020-08-17 08:24:59 +01:00
Alex Sharov
b2e35eb63f
rename "remote2" to "remote" (#932) 2020-08-17 07:47:06 +01:00
Alex Sharov
e04c4ebe58
KV: move bucket.Get and bucket.Cursor methods to Tx object (#914) 2020-08-14 13:41:18 +07:00
Evgeny Danilenko
5d39fe8873
RPCDeamon coinbase, net_version (#911)
* use eth.backend

* net_version

* add coinbase to the console test
2020-08-12 14:47:59 +01:00
Evgeny Danilenko
3980fa7d45
Grps eth_sendTransaction (#882)
* implementation

* tidy gomod

* linters

* fix cmd test

* fix

* fix lint
2020-08-11 22:09:30 +01:00
Alex Sharov
e89ad5d6aa
sub tx support (#895) 2020-08-11 17:35:59 +07:00
Alex Sharov
d9d9e14f45
change bucket type to string (#894) 2020-08-11 06:55:32 +07:00
Alex Sharov
7a1b892022
remove .SeekTo() method (#901) 2020-08-11 06:41:15 +07:00
b00ris
a84230d45c
fix streams leak (#847) 2020-08-01 16:18:31 +01:00
Alex Sharov
b6399a2256
better error messages (#824) 2020-07-30 13:55:40 +07:00
Alex Sharov
0a367001d1
implement db stats page (#803) 2020-07-29 05:31:46 +01:00
Alex Sharov
c4a1ffa4f0
Grpc-based remoteDb (#788) 2020-07-27 19:15:48 +07:00
Alex Sharov
a7e5c4b453
Remove remote_chain package, fix rpc methods: eth_blockNumber and eth_getBlockByNumber (#769) 2020-07-21 15:19:04 +07:00
ledgerwatch
49181a298b
Special bucket for transaction senders (#707)
* Senders in the separate bucket

* Pre-create txSenders bucket

* Fix compile error

* Fix compile error

* Fix compile error, remove block transformations

* Fix formatting

* Fix test
2020-07-03 08:23:49 +01:00
Alex Sharov
a7f5b1ca08
KV: defer friendly rollback (#696)
* AbstractKV: defer friendly rollback

* AbstractKV: defer friendly rollback
2020-06-30 04:48:46 +01:00
Alex Sharov
38f63f9af2
make lmdb default db (#675)
* make lmdb default db

* switch nodes db to lmdb

* log errors in nodedb

* fix miss typing

* fix default test target in makefile

* report error of .Get method

* add debug info

* run tests with debug

* run tests with debug

* remove default db from TEST_DB env

* cleanup

* wait for goroutines finish

* wait for goroutines finish before close db

* wait for goroutines finish before close db

* wait for goroutines finish before close db

* wait for goroutines finish before close db
2020-06-18 19:13:58 +01:00
Alex Sharov
57358730a4
Minor lmdb related improvements (#667)
* don't call initCursor on happy path

* don't call initCursor on happy path

* don't run stale reads goroutine for inMem mode

* don't call initCursor on happy path

* remove buffers from cursor object - they are useful only in Badger implementation

* commit kv benchmark

* remove buffers from cursor object - they are useful only in Badger implementation

* remove buffers from cursor object - they are useful only in Badger implementation

* cancel server before return pipe to pool

* try  to fix test

* set field db in managed tx
2020-06-15 13:30:54 +01:00
Alex Sharov
15096f273d
Remove ctx from Open. Stop goroutines on Close. (#650)
* remove ctx from MustOpen

* remove ctx from Open. Stop goroutines on Close.

* remove ctx from Open. Stop goroutines on Close.

* remove ctx from remote open (we have DialTimeout field to manage connection timeouts)

* enable RawReads and add native implementation of Get/Has methods
2020-06-12 10:31:21 +01:00
Alex Sharov
f0bc2b2146
Run tests on lmdb and badger (#624)
* lmdb tests

* trigger ci

* fix tests

* disable parallelism

* disable parallelism

* cleanup resources

* cleanup resources

* reduce concurency

* try run tests on bolt

* try run tests on bolt

* fix downloader test

* run bolt tests

* rely on interface instead of exact instance

* Rename AbstractKV to KV

* don't use separator for badger

* don't initialize badger cursor - because it not used here

* fix linter

* try reduce badger compactors

* compat with master

* try lmdb

* try lmdb

* try lmdb

* reduce badger's MaxTableSize, reduce badger's minGoMaxProc for inMem option

* allow to close closed db

* release

* release

* ideal batch size for badger

* ideal batch size for badger
2020-06-05 10:25:33 +01:00
Alex Sharov
7aa050ca79
[Breaking change in db!] UI for Buckets stat (#607)
* UI for buckets size
2020-06-04 16:35:42 +07:00
Alex Sharov
916a1f8b83
[wip] Lmdb: AbstractKV and DB interfaces (#589)
* resetIH from scratch if needed

* lmdb

* add AbstractKV to loader, added new Object accessor around AbstractKV

* add lmdb cli flag

* add requirement of k!=nil on error in docs

* add Size method for compatibility

* read after put tests

* fix multiput nils

* simplify loops

* increase mmap size

* better error messages

* better error messages

* fix tests

* better error messages

* cleanup

* avoid bolt usage in test

* move hardcoded bucket name to dbutils

* register more buckets

* register more buckets

* fix test
2020-05-30 09:12:21 +01:00
b00ris
d0af8a2139
Blocks compression (#510)
* block compression

* rerun lint

* fix lint
2020-05-04 06:55:51 +01:00
Alex Sharov
339316deb2
Metrics server without pprof (#451)
* metrics_server_without_pprof

* re-run ci

* re-run ci
2020-04-14 13:49:49 +01:00
Alex Sharov
9324d83cb2
ethdb.KV interface - pase 2 (#420)
* - handle cursor.Prefix on server
- move state reports to KV interface

* add CmdCursorSeekKey

* tests for abstract_kv

* avoid reading configs of databases

* avoid reading configs of databases

* make linter happy

* make linter happy

* cleanup

* port badger features from original implementation

* try to fix test

* try to fix test

* .Close() don't return error anymore - defer friendly

* try to enable badger now

* try to enable badger now

* badger can't run on CI yet

* badger can't run on CI yet

* re-run ci

* skip ctx cancelation for badger
2020-04-04 08:18:10 +01:00
Alex Sharov
b490192e67
Use KV Abstraction in RestAPI (#400)
* Introduce NoValuesCursor. From() method is useless because can be replaced by Seek().`
* implement NoValueCursor interface
* use abstract db in restapi
* cleanup .md
2020-03-24 09:12:55 +07:00
Alex Sharov
965d1101f4
Intermediate hash phase 4 (#396)
* another way to check if account has storage

* cleanup

* v0 of walk by db version

* save progress, to switch to another task. Put tombstone is still not correct.

* place tombstone only if exists something to hide

* db-based implementation

* db-based implementation

* db-based implementation

* fix prop check

* improve prop check logic

* Need custom logic to skip subtree for account and storage buckets because storage bucket has incarnation in key

* rebase to master

* remove all tombstones when account deleted

* remove all tombstones when account deleted

* added db integrity check

* don't rely on account.Root because it valid only about last incarnation

* remove all tombstones when account deleted

* dial with incarnation in MultiWalk2

* dial with incarnation in MultiWalk2

* when fixedbytes=40 resolver did compare cacheKey with storageKey without removing incarnation

* rebase to master

* rebase to master

Co-authored-by: alex.sharov <alex.sharov@lazada.com>
2020-03-20 11:30:14 +00:00
Alex Sharov
84ae2eb1f8
Intermediate Hashes UI (#395)
* implement NoValueCursor interface

* cleanup

* fix tests

* add more stats data to ui

* can't display error

* re-open DB low-level net interface when changing db

* re-open DB low-level net interface when changing db

* fix problem with displaying errors

* run ci

* improve prop check logic

* storage tombstones integrity checks UI

* storage page

* make DB configurable

Co-authored-by: alex.sharov <alex.sharov@lazada.com>
2020-03-20 10:06:14 +00:00
Alex Sharov
5f8b1b75d5
Storage Tombstones UI (#394)
* implement NoValueCursor interface

* cleanup

* fix tests

Co-authored-by: alex.sharov <alex.sharov@lazada.com>
2020-03-16 22:00:48 +00:00
Alex Sharov
c5ffc971c5
[WIP] Badger v2 (#378)
* badger v2 investigation

* buf pool - use native New method and avoid double checks

* db.Open prototype

* db.Tx/Bucket/Cursor prototypes

* Chained config

* Item concept added

* save changes to test on master

* make hack resumable

* Design document v0

* Cursor concept

* less brackets syntax of cursor builder

* benchmarks

* cleanup fs

* test for context cancelations

* test for context cancelations

* test for cursor.Prefix option

* add ForEachKey method

* add ForEachKey method

* add naming explanation

* experiment of non-pointers cursor/bucket

* .Bucket() and .Cursor() doesn't returns error

* .Bucket() and .Cursor() doesn't returns error

* .Bucket() and .Cursor() doesn't returns error

* remove CursorOpts concept

* more test-cases

* simplify open api

* Tx, Bucket, Cursor - now are interfaces

* Tx, Bucket, Cursor - now are interfaces

* switch to interfaces

* rebase master

Co-authored-by: alex.sharov <alex.sharov@lazada.com>
2020-03-11 11:02:37 +00:00
Alex Sharov
fdbba5202b
Trie: store self-destructed accounts (#355)
* squash commits

* enable storage cache

* make linter happy

* fix subtree prefix len check

* save cahnges to test master

* remove restriction on prefix len

* fix comparison of last bits

* fix wrong alignment

* remove debug prints

* commit current state

* commit current state

* avoid changing state of resolver from multiwalk

* remove debug code

* remove debug code

* remove debug code

* remove unnecessary copy

* make code more readable

* reduce rebuildHashes initial resolution

* fix test after rebase to master

* make code more readable

* improve pruner

* pruner add IntermediateCache bucket

* fix panic in Walk on short keys

* reduce allocations for storage keys decompression by increasing default buffer size

* re-run CI

* fix iterator behaviour

* rename cache to hash for unification

* re-run ci

* avoid using underlying DB

* hash all subtree nodes before unload

* fix getNode method

* need to check node type, not parent - before put to hashBucket

* return back parent type check, doesn't work without it.

* don't recalculate hash again

* move unloadFunc from trie to pruner

* rename bucket to shorter name

* rename bucket to shorter name

* clean

* rebase to master
2020-02-12 13:52:59 +00:00
Alex Sharov
ff23980dc6
Intermediate hash phase 2 (#341)
* add env INTERMEDIATE_TRIE_CACHE
* try to use assert.New() pattern
* Fix "maligned" linter warnings to reduce space consumption of structs:

core/types/accounts/account.go:18:14: struct of size 136 bytes could be of size 128 bytes (maligned)
type Account struct {
--
trie/node.go:44:10: struct of size 80 bytes could be of size 72 bytes (maligned)
	duoNode struct {
--
trie/resolve_set.go:28:17: struct of size 56 bytes could be of size 48 bytes (maligned)
type ResolveSet struct {
--
trie/resolver.go:34:15: struct of size 88 bytes could be of size 72 bytes (maligned)
type Resolver struct {
--
trie/visual.go:32:17: struct of size 104 bytes could be of size 96 bytes (maligned)
type VisualOpts struct {
2020-01-31 14:31:50 +07:00
Alex Sharov
fc7291ec34
Use local db in analytics (#308)
* Use local db in analytics (#308)
* pool of encoders
* incr and decr funcs
2020-01-15 19:47:13 +07:00
Alex Sharov
d2e1a1279b
reconnect after server gone (#296)
* reconnect after server gone
2020-01-09 09:46:29 +07:00
Alex Sharov
438476d1ba
Serialization bench (#288)
* bench shows that need to pass pointer to []byte when use encoder.Encode()
2020-01-08 13:09:07 +07:00
b00ris
2ece0e2277
remove gets (#297) 2020-01-07 13:41:33 +03:00
Alex Sharov
33308dd5c2
State analytic less memory and checkpoints (#299)
* added prefix tree to analyses to reduce memory usage
* make new partition every day
* merge concepts of reporter and snapshot
* tests for .FirstKey() and .NextKey()
2020-01-07 09:27:19 +07:00
Alex Sharov
8874f64445
remote_bolt.CmdGetAsOf (#289)
* remote_bolt.CmdGetAsOf
2020-01-06 21:33:25 +07:00
Alex Sharov
85009dabe6
return error from .First() and .Next() methods of RemoteDb (#292)
* return error from .First() and .Next() methods of RemoteDb
* re-run CI
2020-01-06 18:38:21 +07:00
Alex Sharov
f3d3e861dd
fix reusing pointer to memory inside boltdb pages (#286)
* fix reusing pointer to memory inside boltdb pages

* fix shadow variables and unhandled errors
2019-12-24 17:37:51 +06:00