Commit Graph

33 Commits

Author SHA1 Message Date
Alex Sharov
ce96cf75b2
Intermediate hash phase 3 (#377)
* #remove debug prints

* remove storage-mode="i"

* minnet re-execute hack with checkpoints

* minnet re-execute hack with checkpoints

* rollback to master setup

* mainnet re-exec hack

* rollback some changes

* v0 of "push down" functionality

* move all logic to own functions

* handle case when re-created account already has some storage

* clear path for storage

* try to rely on tree structure (but maybe need to rely on DB because can be intra-block re-creations of account)

* fix some bugs with indexes, moving to tests

* tests added

* make linter happy

* make linter happy

* simplify logic

* adjust comparison of keys with and without incarnation

* test for keyIsBefore

* test for keyIsBefore

* better nibbles alignment

* better nibbles alignment

* cleanup

* continue work on tests

* simplify test

* check tombstone existence before pushing it down.

* put tombstone only when account deleted, not created

* put tombstone only when account has storage

* make linter happy

* test for storage resolver

* make fixedbytes work without incarnation

* fix panic on short keys

* use special comparison only when working with keys from cache

* add blockNr for better tracing

* fix: incorrect tombstone check

* fix: incorrect tombstone check

* trigger ci

* hack for problem block

* more test-cases

* add test case for too long keys

* speedup cached resolver by removing bucket creation transaction

* remove parent type check in pruning, remove unused copy from mutation.put

* dump resolving info on fail

* dump resolving info on fail

* set tombstone everytime for now to check if it will help

* on unload: check parent type, not type of node

* fix wrong order of checking node type

* fix wrong order of checking node type

* rebase to new master

* make linter happy

* rebase to new master

* place tombstone only if acc has storage

* rebase master

* rebase master

* rebase master

* rebase master

Co-authored-by: alex.sharov <alex.sharov@lazada.com>
2020-03-11 10:31:49 +00:00
ledgerwatch
80f06b279a
Fix --download-only mode (#374)
* Fix download only

* Fix lint

* Reset references

* Only reset on error

* Potential fixes

* no NPE

* no NPE

* Not use multi-put

* Reduce ideal batch size for download only

* Handle tds == nil

* remove nested mutation

* Return multiput

* Better reporting

* Reduce batch size for download only

* Avoid extra copying

* Avoid extra copying

* IdealBatchSize

* Not write tx lookup entries

* Larger batches

* Go back to normal batch size

* Fix lint

* Gen tx lookup

* print progress

* Add filling up the lookup array

* Show tx count

* Introduce second round

* Add generating tx lookup

* Fix lint

* properly stop at specified block

* measure the duration of the last phase

* not to fail if the bucket is not found

* Fix lint

* Alternative tx generation

* Fix out of memory

* Fix out of memory

* Split in parts to conserve memory

* Copy keys

* Fix lint

* Fix lint
2020-03-01 09:00:14 +00:00
b00ris
83e65fdd81 migrations
fix build

add storage mode check

add intermitiate migration state&snapshot migrations
2020-02-26 00:25:30 +03:00
Alex Sharov
fdbba5202b
Trie: store self-destructed accounts (#355)
* squash commits

* enable storage cache

* make linter happy

* fix subtree prefix len check

* save cahnges to test master

* remove restriction on prefix len

* fix comparison of last bits

* fix wrong alignment

* remove debug prints

* commit current state

* commit current state

* avoid changing state of resolver from multiwalk

* remove debug code

* remove debug code

* remove debug code

* remove unnecessary copy

* make code more readable

* reduce rebuildHashes initial resolution

* fix test after rebase to master

* make code more readable

* improve pruner

* pruner add IntermediateCache bucket

* fix panic in Walk on short keys

* reduce allocations for storage keys decompression by increasing default buffer size

* re-run CI

* fix iterator behaviour

* rename cache to hash for unification

* re-run ci

* avoid using underlying DB

* hash all subtree nodes before unload

* fix getNode method

* need to check node type, not parent - before put to hashBucket

* return back parent type check, doesn't work without it.

* don't recalculate hash again

* move unloadFunc from trie to pruner

* rename bucket to shorter name

* rename bucket to shorter name

* clean

* rebase to master
2020-02-12 13:52:59 +00:00
ledgerwatch
767a374c2e
Tx pool use triedbstate, fixes to GetNodeData and move it to experimental (#364)
* Use TrieDbState for tx pool

* Not initialise tx pool until state is loaded

* Add preimage

* Fix account

* Print codehash

* Print correct code hash

* Print incarnatin

* Print incarnatin

* Use proper incarnation

* Print dbValue

* Actually fix

* Actually fix

* Fix verifySnapshot

* readAccount to get code hash

* Next incarnation

* Print addrHashes with 0 incarnations

* Print storage history

* Print storage history

* Print storage history

* Print storage history

* Print all storage history

* print change set keys

* print change set keys

* print change set keys

* print change set keys

* Not print codebucket info

* Fixes

* Fix for incarnation

* Fix for storage history bucket

* Try to fix the leak

* Try to fix the leak

* Try to fix the leak

* Try to fix the leak

* Try to fix the leak

* Try to fix the leak

* Try to fix the leak

* Fix embedded nodes

* Hasher

* Fix

* Test fixes

* Add experimental debug flag

* Fix tx_pool_test

* Disable GetNodeData test unless in experiment

* Fix more tests

* Fix lint and revert some changes

* Fix lint

* Fix lint
2020-02-10 17:05:32 +00:00
Igor Mandrigin
f4d0747949
stateless: flush the partial witnesses DB for every saved witness (#354)
* stateless: flush the partial witnesses DB for every saved witness.

it seems to fix to the out of memory error I had on blocks over 2.5M;

* fix linters

* simplify
2020-02-06 13:58:58 +03:00
ledgerwatch
dca36e8b29
Restore the functionality CalcTrieRoots (compute trie root without modifying the trie) (#327)
* Trace first block

* Fixes for CalcTrieRoots

* Timings of the CalcTrieRoot

* Fix lint

* Add memory profiling

* Reduce allocations in StreamHash

* Fix

* Fix

* Fix

* Optimise allocations

* Reuse streams

* Fixes

* Fix

* Unit test fix

* Fix lint

* Reuse hashbuilder

* No loop

* Reuse resolver

* Fixed tests

* Fix test

* Fix test

* Fix test

* Fix witness threshold

* Optimise allocations in RLP transform

* Optimise allocations in RLP transform

* Optimise branchHash

* 100 times again

* Replace ToStream with Iterator

* StreamMergeIterator

* No streams

* Amplification

* Minimise the use of hashOnly

* 100 times

* Reduce stack operations

* Reduce appends

* More optimisations

* More optimisations

* More optimisations

* local hex

* Small fix

* Remove test

* Fix lint

* Fix lint

* Fix lint

* Add test for empty

* Fix lint

* More tests

* Fix lint

* Add measurement of stateless exec
2020-01-30 13:16:12 +00:00
Igor Mandrigin
1115b23832
Stateless Resolver: store and use witnesses for resolved subtries (#326) 2020-01-24 11:58:01 +01:00
ledgerwatch
4f5c858f00
Collect data about Keccak256 invocations for evaluation of STARK proof sizes and performance (#315)
* Gathering start data on witnesses

* Fix number parsing

* Proper filename, actually outputting

* Correct block

* Visualise states

* Code and value fix

* Visualise code hash and storage hash

* Current block number

* Print bucket

* Fix lint

* Fix lint

* Fix lint

* Fix lint

* Fix typo

* Fixes after merging master

* Fix lint

Co-authored-by: b00ris <b00ris@mail.ru>
2020-01-15 17:33:36 +00:00
b00ris
df8798b94b
Thin history for storage (#324)
* save state

* save state

* fmt

* add check to test

* move incarnation length to common

* remove line

* fix lint
2020-01-15 17:55:43 +03:00
Igor Mandrigin
b7710ff7fa
New Witness Format (and remove the rest of the tapes) (#317) 2020-01-15 14:56:50 +01:00
Alex Sharov
fc7291ec34
Use local db in analytics (#308)
* Use local db in analytics (#308)
* pool of encoders
* incr and decr funcs
2020-01-15 19:47:13 +07:00
Alex Sharov
438476d1ba
Serialization bench (#288)
* bench shows that need to pass pointer to []byte when use encoder.Encode()
2020-01-08 13:09:07 +07:00
Alex Sharov
33308dd5c2
State analytic less memory and checkpoints (#299)
* added prefix tree to analyses to reduce memory usage
* make new partition every day
* merge concepts of reporter and snapshot
* tests for .FirstKey() and .NextKey()
2020-01-07 09:27:19 +07:00
Alex Sharov
85009dabe6
return error from .First() and .Next() methods of RemoteDb (#292)
* return error from .First() and .Next() methods of RemoteDb
* re-run CI
2020-01-06 18:38:21 +07:00
Alex Sharov
805f9aa501
benchmark rpc daemon
* tracing, write bench1() result into files

* add context to computeTxEnv, remote_db cursor lazy cache allocation,

* more results

* added

* added

* added

* more info

* make linter happy

* exit from tracing gorutines

* remove unused param

* re-run CI

* re-run CI

* remove concept txHandle

* batch cursor in reports

* remove debug output

* fix shadow variables and unhandled errors
2019-12-22 21:10:46 +06:00
alex.sharov
e6b33b376f make ci happy 2019-12-16 20:55:10 +06:00
alex.sharov
88b05bd162 make linter happy 2019-12-16 20:54:30 +06:00
alex.sharov
71dc098968 include to report` only blocks on chain with highest difficulty 2019-12-09 21:34:47 +07:00
alex.sharov
04a6517c5b added gasLimits to report object 2019-12-09 21:34:47 +07:00
alex.sharov
7a4023b4eb added gasLimits to report object 2019-12-09 21:34:47 +07:00
alex.sharov
005d715545 move gasLimits to remote db 2019-12-09 21:34:47 +07:00
alex.sharov
bb2f7080ce Moved connection to .View method. Not very optimal, but it allow cancel connection from client side or configure timeout, also in future it will allow us make connection pool there (maybe don't need). 2019-12-06 08:12:02 +07:00
alex.sharov
8b69c54936 move context to from .NewDb to .View method 2019-12-06 08:12:02 +07:00
alex.sharov
344b8029cc handle interruption signals, added context for client and server. Only problem that it's context to DB, but not to each server call - it means you can't use HTTP Request context here (or create new connection on each request). 2019-12-06 08:12:02 +07:00
alex.sharov
49a614eae8 Bug fixes:
- rollback must catch local variable, if you write value to lastError variable
- then you can't do "return" - because it closing connection and client can't send CmdLastError
- somewhere error values was ommited in loging
2019-12-06 08:12:02 +07:00
alex.sharov
75e0b1bc85 revert changed bucket name 2019-12-06 08:12:02 +07:00
alex.sharov
1ffbb97752 removed unnecessary allocations, add tcp context 2019-12-06 08:12:02 +07:00
alex.sharov
d8f93ae187 Batch cursor.First and cursor.Next for remote db. 2019-12-06 08:12:02 +07:00
Igor Mandrigin
28d9b7c812
Use binary tries for the stateless action as an option. (#218) 2019-12-04 13:48:38 +01:00
Igor Mandrigin
ebae2fe281
Add --noverify flag to /cmd/state to skip roots check. (#205) 2019-11-27 14:52:22 +01:00
Igor Mandrigin
4fb54a7b82
Write proper stats to the CSV file; restore charting capabilities. (#201) 2019-11-26 14:29:41 +01:00
Igor Mandrigin
ac085db07d
Use 'cobra' package to make a more modular CLI for stateless ethereum prototype. (#200) 2019-11-25 14:46:36 +01:00