Commit Graph

207 Commits

Author SHA1 Message Date
Martin Holst Swende
859a345680 trie: separate hashes and committer, collapse on commit
* trie:  make db insert use size instead of full data

* core/state: minor optimization in state onleaf allocation

* trie: implement dedicated committer and hasher

* trie: use dedicated committer/hasher

* trie: linter nitpicks

* core/state, trie: avoid unnecessary storage trie load+commit

* trie: review feedback, mainly docs + minor changes

* trie: start deprecating old hasher

* trie: fix misspell+lint

* trie: deprecate hasher.go, make proof framework use new hasher

* trie: rename pure_committer/hasher to committer/hasher

* trie, core/state: fix review concerns

* trie: more review concerns

* trie: make commit collapse into hashnode, don't touch dirtyness

* trie: goimports fixes

* trie: remove panics
2020-02-27 17:20:36 +03:00
Alex Sharov
21ce7b4c0d
check in trie.hook was transformed incorrectly (#372) 2020-02-17 13:24:07 +07:00
Andrew Ashikhmin
b541bff1eb
Enable GetNodeData by default (#368)
* Add a missing IsGetNodeData check

* Enable GetNodeData by default
2020-02-16 16:50:24 +00:00
Alex Sharov
fdbba5202b
Trie: store self-destructed accounts (#355)
* squash commits

* enable storage cache

* make linter happy

* fix subtree prefix len check

* save cahnges to test master

* remove restriction on prefix len

* fix comparison of last bits

* fix wrong alignment

* remove debug prints

* commit current state

* commit current state

* avoid changing state of resolver from multiwalk

* remove debug code

* remove debug code

* remove debug code

* remove unnecessary copy

* make code more readable

* reduce rebuildHashes initial resolution

* fix test after rebase to master

* make code more readable

* improve pruner

* pruner add IntermediateCache bucket

* fix panic in Walk on short keys

* reduce allocations for storage keys decompression by increasing default buffer size

* re-run CI

* fix iterator behaviour

* rename cache to hash for unification

* re-run ci

* avoid using underlying DB

* hash all subtree nodes before unload

* fix getNode method

* need to check node type, not parent - before put to hashBucket

* return back parent type check, doesn't work without it.

* don't recalculate hash again

* move unloadFunc from trie to pruner

* rename bucket to shorter name

* rename bucket to shorter name

* clean

* rebase to master
2020-02-12 13:52:59 +00:00
Andrew Ashikhmin
05a3483cb1
Fix hash map leaks (#367)
* Enable testGetNodeData by default

* Crude TestHashMapLeak

* nodeFlag ->nodeRef

* linter

* accountNode shouln't be in hashMap since only branch and short nodes are the standart ones

* Finalize hash eviction logic

* small changes to TestHashMapLeak

* Fix for the incorrect hash

Co-authored-by: ledgerwatch <akhounov@gmail.com>
2020-02-12 11:34:44 +00:00
ledgerwatch
767a374c2e
Tx pool use triedbstate, fixes to GetNodeData and move it to experimental (#364)
* Use TrieDbState for tx pool

* Not initialise tx pool until state is loaded

* Add preimage

* Fix account

* Print codehash

* Print correct code hash

* Print incarnatin

* Print incarnatin

* Use proper incarnation

* Print dbValue

* Actually fix

* Actually fix

* Fix verifySnapshot

* readAccount to get code hash

* Next incarnation

* Print addrHashes with 0 incarnations

* Print storage history

* Print storage history

* Print storage history

* Print storage history

* Print all storage history

* print change set keys

* print change set keys

* print change set keys

* print change set keys

* Not print codebucket info

* Fixes

* Fix for incarnation

* Fix for storage history bucket

* Try to fix the leak

* Try to fix the leak

* Try to fix the leak

* Try to fix the leak

* Try to fix the leak

* Try to fix the leak

* Try to fix the leak

* Fix embedded nodes

* Hasher

* Fix

* Test fixes

* Add experimental debug flag

* Fix tx_pool_test

* Disable GetNodeData test unless in experiment

* Fix more tests

* Fix lint and revert some changes

* Fix lint

* Fix lint
2020-02-10 17:05:32 +00:00
Andrew Ashikhmin
08e651c779 Add subtrie's nodes to the hash map when it's hooked (#360)
* log trie's hashMap size

* Add subtrie's nodes to the hash map when it's hooked
2020-02-08 22:18:20 +00:00
Andrew Ashikhmin
8585516071
Serve GetNodeData for nodes in memory (Issue #300) 2020-02-06 11:53:09 +01:00
Evgeny Danilenko
17a4a56634
Refactor mining. Remove a few mining goroutines (#338)
* add context

* extract chain events

* run commit in goroutines

* mine only on canonical

* typo

* linters

* fmt

* mark unused methods

* restore stress test

* test single miner

* remove unsafe Trie storage

* remove locks from miner

* restore interrupt

* remove result goroutine

* remove unconfirmedBlocks

* cherry-pick 04a1d475ff1a36ad8f92fec80385df18c52bdc1f

* extract uncles

* one miner succeeded

* restore context cancel

* cleanup

* skip an unstable test

* remove pending state

* use context instead of interrupt func

* calculate sealHash only once

* comment out unstable test

* after merge

* fix after merge

Co-authored-by: ledgerwatch <akhounov@gmail.com>
2020-02-03 15:02:26 +03:00
Alex Sharov
ff23980dc6
Intermediate hash phase 2 (#341)
* add env INTERMEDIATE_TRIE_CACHE
* try to use assert.New() pattern
* Fix "maligned" linter warnings to reduce space consumption of structs:

core/types/accounts/account.go:18:14: struct of size 136 bytes could be of size 128 bytes (maligned)
type Account struct {
--
trie/node.go:44:10: struct of size 80 bytes could be of size 72 bytes (maligned)
	duoNode struct {
--
trie/resolve_set.go:28:17: struct of size 56 bytes could be of size 48 bytes (maligned)
type ResolveSet struct {
--
trie/resolver.go:34:15: struct of size 88 bytes could be of size 72 bytes (maligned)
type Resolver struct {
--
trie/visual.go:32:17: struct of size 104 bytes could be of size 96 bytes (maligned)
type VisualOpts struct {
2020-01-31 14:31:50 +07:00
Alex Sharov
0dd4386a37
Intermediate trie hashes phase 1 (#334)
* add intermediate trie hashes bucket
* update docs
* use version of CompressNibbles with buf pool (less GC) but without io.ByteWriter (slow)
2020-01-31 11:11:20 +07:00
ledgerwatch
dca36e8b29
Restore the functionality CalcTrieRoots (compute trie root without modifying the trie) (#327)
* Trace first block

* Fixes for CalcTrieRoots

* Timings of the CalcTrieRoot

* Fix lint

* Add memory profiling

* Reduce allocations in StreamHash

* Fix

* Fix

* Fix

* Optimise allocations

* Reuse streams

* Fixes

* Fix

* Unit test fix

* Fix lint

* Reuse hashbuilder

* No loop

* Reuse resolver

* Fixed tests

* Fix test

* Fix test

* Fix test

* Fix witness threshold

* Optimise allocations in RLP transform

* Optimise allocations in RLP transform

* Optimise branchHash

* 100 times again

* Replace ToStream with Iterator

* StreamMergeIterator

* No streams

* Amplification

* Minimise the use of hashOnly

* 100 times

* Reduce stack operations

* Reduce appends

* More optimisations

* More optimisations

* More optimisations

* local hex

* Small fix

* Remove test

* Fix lint

* Fix lint

* Fix lint

* Add test for empty

* Fix lint

* More tests

* Fix lint

* Add measurement of stateless exec
2020-01-30 13:16:12 +00:00
Igor Mandrigin
785d0eafd8 post-rebase fixups v1.9.10 2020-01-30 13:36:30 +02:00
Martin Holst Swende
3bda450baa * trie: utilize callbacks instead of amassing lists in ref/unref (#20529)
* trie/tests: add benchmarks and update trie tests

* trie: update benchmark tests

* trie: utilize callbacks instead of amassing lists of hashes in database ref/unref

* trie: replace remaining non-callback based accesses
2020-01-30 13:36:30 +02:00
me020523
08487ecc6b add node.go unit test file node_test.go (#20028)
* add node.go unit test file node_test.go

* add node_test.go file license and rollback trie_test.go

* fix unused variable v

* trie: fix license year

Co-authored-by: Péter Szilágyi <peterke@gmail.com>
2020-01-30 13:36:30 +02:00
Igor Mandrigin
1115b23832
Stateless Resolver: store and use witnesses for resolved subtries (#326) 2020-01-24 11:58:01 +01:00
ledgerwatch
4f5c858f00
Collect data about Keccak256 invocations for evaluation of STARK proof sizes and performance (#315)
* Gathering start data on witnesses

* Fix number parsing

* Proper filename, actually outputting

* Correct block

* Visualise states

* Code and value fix

* Visualise code hash and storage hash

* Current block number

* Print bucket

* Fix lint

* Fix lint

* Fix lint

* Fix lint

* Fix typo

* Fixes after merging master

* Fix lint

Co-authored-by: b00ris <b00ris@mail.ru>
2020-01-15 17:33:36 +00:00
Igor Mandrigin
2053d5c64f
add incarnation to HashBuilder (#318) 2020-01-15 15:12:33 +01:00
Igor Mandrigin
b7710ff7fa
New Witness Format (and remove the rest of the tapes) (#317) 2020-01-15 14:56:50 +01:00
b00ris
29145a8b7d
Incorrect incarnation fix (#298)
* small fixes

* fmt

* change default incarnation for forEachStorage

* fix test

* apply alex changes

* small fixes

* rerun test
2020-01-08 12:55:56 +03:00
Giulio rebuffo
bcf04bbbcf Change "for storage" encoding of Account from RLP to bitmask + fields encoding (#287)
* migration tool + CBOR encoding for storage of account bucket

* documentation updated

* fixed

* removed noncontract in EncodeForStorage

* code adjustments

* updated tests

* switched to custom encoding

* documentation updated
2020-01-07 09:02:44 +00:00
b00ris
bcee4845dc
Thin history (#272)
* save state

* remove repair

* save state

* remove emptydb check

* save state

* add walkAsOf test

* add WalkAsOf and MultiWalkAsOf tests

* deployed contracts counter

* reference counter for contract code

* drop storage root&contract hash for changesets

* start incarnation is 1(save state)

* fix ReorgOverSelfDestruct test

* hack fix TestReorgOverSelfDestruct

* test benchmark

* cleanup

* remove useless debug

* remove print trie

* return remove subtrie call  to updateTrieRoot

* save state

* add mutation test

* remove useless test

* fix

* added mutation commit test

* rename experiment to thin history

* thin history mutation commit test

* fix ethdb tests

* getAsOf test

* add test&fix history index

* fix test

* make test for index search

* compute trie root incarnation fix

* tests fixes

* done job in case of panic

* fix lint

* fix&test for bad incarnation

* fix initial incarnation for genesis

* fix lint

* fix changeset test

* fix storage ranges test

* fix lint

* move set incarnation to create contract

* add comment

Co-authored-by: ledgerwatch <akhounov@gmail.com>
Co-authored-by: Evgeny Danilenko <6655321@bk.ru>
2019-12-20 15:25:40 +03:00
Igor Mandrigin
0ab530a614
Remote tapes from GenStructStep (#260) 2019-12-12 18:17:13 +01:00
Igor Mandrigin
422b154fb1
Remove tapes from HashBuilder (#238) 2019-12-09 15:59:30 +01:00
Igor Mandrigin
cbfc2fa8d1
Merge pull request #248 from ledgerwatch/geth-1-9-9
Geth 1.9.9
2019-12-09 13:32:32 +01:00
Péter Szilágyi
6f8dba02ef trie: track dirty cache metrics, track clean writes on commit 2019-12-09 12:56:07 +01:00
gary rong
7b664b8c51 core, miner: remove PostChainEvents (#19396)
This change:

- removes the PostChainEvents method on core.BlockChain.
- sorts 'removed log' events by block number.
- fire the NewChainHead event if we inject a canonical block into the chain
  even if the entire insertion is not successful.
- guarantees correct event ordering in all cases.
2019-12-09 12:17:48 +01:00
gary rong
14bb7db725 trie: remove dead code (#20405) 2019-12-09 12:12:12 +01:00
ledgerwatch
8a632e764b
Continue comparison of genesis block with geth, expand long values (#223) 2019-12-06 12:03:12 +00:00
Igor Mandrigin
28d9b7c812
Use binary tries for the stateless action as an option. (#218) 2019-12-04 13:48:38 +01:00
Igor Mandrigin
ec6350fa15 apply post-rebase fixups 2019-12-03 16:27:57 +01:00
gary rong
2494457f6e trie: replace bigcache with fastcache (#19971) 2019-12-03 14:00:04 +01:00
Felix Lange
9438438a0a trie: remove unused code (#20366) 2019-12-03 13:54:39 +01:00
Andrew Ashikhmin
84285a5ab1
Roughly finish with adding BadgerDB (first cut) (#203)
* Bumping GOMAXPROCS for Badger

* fixes related to database size

* Schedule GC for Badger

* pacify linter

* Don't start GC for ephemeral Badger

* Don't log "Value log GC attempt didn't result in any cleanup"

* Start GC in backround

* Bump GC period and IdealBatchSize for Badger

* BadgerDatabase RewindData

* Boolean badger flag -> string database flag

* cosmetic change
2019-11-27 13:15:25 +00:00
Igor Mandrigin
4fb54a7b82
Write proper stats to the CSV file; restore charting capabilities. (#201) 2019-11-26 14:29:41 +01:00
ledgerwatch
5f667d3225
Further fix to calculation of state root without modification of the trie (#192)
* Fix

* Minimize changes
2019-11-25 13:37:34 +00:00
ledgerwatch
b2ca635d86
Debug tool for recursively comparing state (in DB or in a file) with geth archive node (#191)
* Fetching results of eth_getProof

* Dump 5 levels of the trie in a file for repeated runs

* Drill down to 6 levels of the trie

* Fix lint

* Fix lint

* Fix lint

* verifySnapshot to check accounts with emptyRoot

* Descend into short nodes

* Latest tool fixes

* Fix lint

* Fix state properly working
2019-11-25 13:36:21 +00:00
ledgerwatch
244d70fb9c
Further fixes for the no-mod-root (#186)
* Further fixes

* Repace 1000 with a symbol
2019-11-21 15:56:39 +00:00
Andrew Ashikhmin
1c0cf9be72
BadgerDB: MultiWalk & some other APIs, command line flag, scaffolding for testing (#183)
* BadgerDatabase MemCopy

* fix err shadowing

* BadgerDatabase MultiPut & NewBatch

* Remove goOn from MultiWalk & MultiWalkAsOf

* BadgerDatabase MultiWalk. Scaffolding for testing Badger in blockchain_test

* Badger Flag

* fix error logging

* Split IdealBatchSize between BoltDB and BadgerDB

* NewEphemeralBadger
2019-11-21 16:12:38 +01:00
Igor Mandrigin
aa71b298cc
Write tape stats to a csv file. (#185) 2019-11-21 15:09:21 +01:00
Igor Mandrigin
f4cda8ba4c
Fix snapshot saving and add an interval setting. (#181) 2019-11-21 14:36:24 +01:00
Giulio rebuffo
21c981eb84 HexToQuad fixed in trie/visual.go (#173) 2019-11-19 07:48:08 +00:00
ledgerwatch
c187d80152
Preparations for computing root without modifying the trie (#172)
Preparations for computing root without modifying the trie
2019-11-15 22:48:49 +00:00
Igor Mandrigin
7fd4b0431b
refactor trie package a bit (#171)
* Refactor `structural*` files.

* some refactor

* dry hashbuilder a bit

* rename constants back to the magic rlp values
2019-11-15 17:50:16 +01:00
Igor Mandrigin
00074e0ed7 Don't store RLPs in valueNodes (except tests). 2019-11-13 20:08:20 +01:00
Igor Mandrigin
02af800e66 Use GenStructStep from structural_2 in DeriveSha. 2019-11-07 16:14:33 +01:00
Andrew Ashikhmin
e14b02e1c9 Merge pull request #135 from ledgerwatch/badger 2019-11-04 14:15:26 +01:00
ledgerwatch
f760001638 Merge pull request #112 from ledgerwatch/block_witness 2019-11-04 14:15:26 +01:00
Andrew Ashikhmin
6992db3180 Merge pull request #121 from ledgerwatch/red-queen 2019-11-04 14:15:26 +01:00
ledgerwatch
d1c8416270 Merge pull request #114 from ledgerwatch/fix_sync_crash
Fix the sync crash due to the mis-accounting of prunable entries
2019-11-04 14:15:26 +01:00