Commit Graph

61 Commits

Author SHA1 Message Date
ledgerwatch
b2ca635d86
Debug tool for recursively comparing state (in DB or in a file) with geth archive node (#191)
* Fetching results of eth_getProof

* Dump 5 levels of the trie in a file for repeated runs

* Drill down to 6 levels of the trie

* Fix lint

* Fix lint

* Fix lint

* verifySnapshot to check accounts with emptyRoot

* Descend into short nodes

* Latest tool fixes

* Fix lint

* Fix state properly working
2019-11-25 13:36:21 +00:00
ledgerwatch
c187d80152
Preparations for computing root without modifying the trie (#172)
Preparations for computing root without modifying the trie
2019-11-15 22:48:49 +00:00
Igor Mandrigin
00074e0ed7 Don't store RLPs in valueNodes (except tests). 2019-11-13 20:08:20 +01:00
ledgerwatch
f760001638 Merge pull request #112 from ledgerwatch/block_witness 2019-11-04 14:15:26 +01:00
Andrew Ashikhmin
6992db3180 Merge pull request #121 from ledgerwatch/red-queen 2019-11-04 14:15:26 +01:00
ledgerwatch
d1c8416270 Merge pull request #114 from ledgerwatch/fix_sync_crash
Fix the sync crash due to the mis-accounting of prunable entries
2019-11-04 14:15:26 +01:00
Alexey Akhunov
fe01bccbb8 Apply Turbo-Geth modifications to go-ethereum codebase 2019-11-01 21:52:03 +01:00
Péter Szilágyi
91eec1251c
cmd, core, eth, trie: get rid of trie cache generations (#19262)
* cmd, core, eth, trie: get rid of trie cache generations

* core, trie: get rid of remainder of cache gen boilerplate
2019-03-14 15:25:12 +02:00
Péter Szilágyi
53634f1e04 trie: remove unused originalRoot field (#17862) 2018-10-08 13:16:16 +02:00
Oleg Kovalov
cf05ef9106 p2p, swarm, trie: avoid copying slices in loops (#17265) 2018-08-07 13:56:40 +03:00
Péter Szilágyi
d926bf2c7e trie: cache collapsed tries node, not rlp blobs (#16876)
The current trie memory database/cache that we do pruning on stores
trie nodes as binary rlp encoded blobs, and also stores the node
relationships/references for GC purposes. However, most of the trie
nodes (everything apart from a value node) is in essence just a
collection of references.

This PR switches out the RLP encoded trie blobs with the
collapsed-but-not-serialized trie nodes. This permits most of the
references to be recovered from within the node data structure,
avoiding the need to track them a second time (expensive memory wise).
2018-06-21 11:28:05 +02:00
Wenbiao Zheng
d6ed2f67a8 eth, node, trie: fix minor typos (#16802) 2018-05-24 15:55:20 +03:00
Anton Evangelatov
ae9f97221a metrics: pull library and introduce ResettingTimer and InfluxDB reporter (#15910)
* go-metrics: fork library and introduce ResettingTimer and InfluxDB reporter.

* vendor: change nonsense/go-metrics to ethersphere/go-metrics

* go-metrics: add tests. move ResettingTimer logic from reporter to type.

* all, metrics: pull in metrics package in go-ethereum

* metrics/test: make sure metrics are enabled for tests

* metrics: apply gosimple rules

* metrics/exp, internal/debug: init expvar endpoint when starting pprof server

* internal/debug: tiny comment formatting fix
2018-02-23 11:56:08 +02:00
Péter Szilágyi
55599ee95d core, trie: intermediate mempool between trie and database (#15857)
This commit reduces database I/O by not writing every state trie to disk.
2018-02-05 17:40:32 +01:00
Péter Szilágyi
216e584899
Revert "trie: make fullnode children hash calculation concurrently (#15131)" (#15889)
This reverts commit 0f7fbb85d6.
2018-01-15 15:32:14 +02:00
Furkan KAMACI
b8caba9709 various: remove redundant parentheses (#15793) 2018-01-03 14:14:47 +02:00
gary rong
0f7fbb85d6 trie: make fullnode children hash calculation concurrently (#15131)
* trie: make fullnode children hash calculation concurrently

* trie: thread out only on topmost fullnode

* trie: clean up full node children hash calculation

* trie: minor code fixups
2017-11-27 13:34:17 +02:00
Felix Lange
10181b57a9 core, eth/downloader: commit block data using batches (#15115)
* ethdb: add Putter interface and Has method

* ethdb: improve docs and add IdealBatchSize

* ethdb: remove memory batch lock

Batches are not safe for concurrent use.

* core: use ethdb.Putter for Write* functions

This covers the easy cases.

* core/state: simplify StateSync

* trie: optimize local node check

* ethdb: add ValueSize to Batch

* core: optimize HasHeader check

This avoids one random database read get the block number. For many uses
of HasHeader, the expectation is that it's actually there. Using Has
avoids a load + decode of the value.

* core: write fast sync block data in batches

Collect writes into batches up to the ideal size instead of issuing many
small, concurrent writes.

* eth/downloader: commit larger state batches

Collect nodes into a batch up to the ideal size instead of committing
whenever a node is received.

* core: optimize HasBlock check

This avoids a random database read to get the number.

* core: use numberCache in HasHeader

numberCache has higher capacity, increasing the odds of finding the
header without a database lookup.

* core: write imported block data using a batch

Restore batch writes of state and add blocks, tx entries, receipts to
the same batch. The change also simplifies the miner.

This commit also removes posting of logs when a forked block is imported.

* core: fix DB write error handling

* ethdb: use RLock for Has

* core: fix HasBlock comment
2017-09-09 19:03:07 +03:00
Felix Lange
693d9ccbfb trie: more node iterator improvements (#14615)
* ethdb: remove Set

Set deadlocks immediately and isn't part of the Database interface.

* trie: add Err to Iterator

This is useful for testing because the underlying NodeIterator doesn't
need to be kept in a separate variable just to get the error.

* trie: add LeafKey to iterator, panic when not at leaf

LeafKey is useful for callers that can't interpret Path.

* trie: retry failed seek/peek in iterator Next

Instead of failing iteration irrecoverably, make it so Next retries the
pending seek or peek every time.

Smaller changes in this commit make this easier to test:

* The iterator previously returned from Next on encountering a hash
  node. This caused it to visit the same path twice.
* Path returned nibbles with terminator symbol for valueNode attached
  to fullNode, but removed it for valueNode attached to shortNode. Now
  the terminator is always present. This makes Path unique to each node
  and simplifies Leaf.

* trie: add Path to MissingNodeError

The light client trie iterator needs to know the path of the node that's
missing so it can retrieve a proof for it. NodeIterator.Path is not
sufficient because it is updated when the node is resolved and actually
visited by the iterator.

Also remove unused fields. They were added a long time ago before we
knew which fields would be needed for the light client.
2017-06-20 18:26:09 +02:00
S. Matthew English
061889d4ea rlp, trie, contracts, compression, consensus: improve comments (#14580) 2017-06-12 14:45:17 +02:00
Felix Lange
4047ccad2f trie: add start key to NodeIterator constructors
The 'step' method is split into two parts, 'peek' and 'push'. peek
returns the next state but doesn't make it current.

The end of iteration was previously tracked by setting 'trie' to nil.
End of iteration is now tracked using the 'iteratorEnd' error, which is
slightly cleaner and requires less code.
2017-04-25 02:14:31 +02:00
Felix Lange
a13e920af0 trie: clean up iterator constructors
Make it so each iterator has exactly one public constructor:

- NodeIterators can be created through a method.
- Iterators can be created through NewIterator on any NodeIterator.
2017-04-25 02:14:31 +02:00
Felix Lange
f958d7d482 trie: rework and document key encoding
'encode' and 'decode' are meaningless because the code deals with three
encodings. Document the encodings and give a name to each one.
2017-04-25 02:14:31 +02:00
Felix Lange
a31d268b76 trie: remove Key in MissingNodeError
The key was constructed from nibbles, which isn't possible for all
nodes. Remove the only use of Key in LightTrie by always retrying with
the original key that was looked up.
2017-04-18 14:52:11 +02:00
Péter Szilágyi
d4fd06c3dc
all: blidly swap out glog to our log15, logs need rework 2017-02-23 12:16:44 +02:00
Felix Lange
d3b751e4d9 trie: remove dependency on ethdb
This removes the core/types -> leveldb dependency.
2017-01-06 14:15:22 +01:00
Péter Szilágyi
88a593d559
cmd/geth, trie: report on trie cache unloads, also add debug log 2016-10-19 17:31:19 +03:00
Felix Lange
5d9bb0a050 trie: add metric to track cache unloading (#3175) 2016-10-19 16:07:19 +02:00
Felix Lange
d48e6ae66f trie: make the cache miss counter a real metric 2016-10-19 15:22:37 +02:00
Jeffrey Wilcke
25ac04a444 Merge pull request #3153 from fjl/trie-unload-fix
trie: improve cache unloading mechanism
2016-10-19 13:35:49 +02:00
Péter Szilágyi
2bb5ec1e41
cmd/geth, trie: track and report trie cache misses 2016-10-19 14:12:35 +03:00
Felix Lange
177cab5fe7 trie: ensure resolved nodes stay loaded
Commit 40cdcf1183 broke the optimisation which kept nodes resolved
during Get in the trie. The decoder assigned cache generation 0
unconditionally, causing resolved nodes to get flushed on Commit.

This commit fixes it and adds two tests.
2016-10-18 04:57:47 +02:00
Felix Lange
187d6a66a5 trie: avoid loading the root node twice
New checks whether the root node is present by loading it from the
database. Keep the node around instead of discarding it.
2016-10-18 04:57:47 +02:00
Felix Lange
40cdcf1183 trie, core/state: improve memory usage and performance (#3135)
* trie: store nodes as pointers

This avoids memory copies when unwrapping node interface values.

name      old time/op  new time/op  delta
Get        388ns ± 8%   215ns ± 2%  -44.56%  (p=0.000 n=15+15)
GetDB      363ns ± 3%   202ns ± 2%  -44.21%  (p=0.000 n=15+15)
UpdateBE  1.57µs ± 2%  1.29µs ± 3%  -17.80%  (p=0.000 n=13+15)
UpdateLE  1.92µs ± 2%  1.61µs ± 2%  -16.25%  (p=0.000 n=14+14)
HashBE    2.16µs ± 6%  2.18µs ± 6%     ~     (p=0.436 n=15+15)
HashLE    7.43µs ± 3%  7.21µs ± 3%   -2.96%  (p=0.000 n=15+13)

* trie: close temporary databases in GetDB benchmark

* trie: don't keep []byte from DB load around

Nodes decoded from a DB load kept hashes and values as sub-slices of
the DB value. This can be a problem because loading from leveldb often
returns []byte with a cap that's larger than necessary, increasing
memory usage.

* trie: unload old cached nodes

* trie, core/state: use cache unloading for account trie

* trie: use explicit private flags (fixes Go 1.5 reflection issue).

* trie: fixup cachegen overflow at request of nick

* core/state: rename journal size constant
2016-10-14 19:04:33 +03:00
Felix Lange
c3a77d6268 trie: fix delete bug for values contained in fullNode
Delete crashed if a fullNode contained a valueNode directly. This bug is
very unlikely to occur with SecureTrie, but can happen with regular
tries. This commit also introduces a randomised test which triggers all
trie operations, which should prevent such bugs in the future.

Credit for finding this bug goes to Github user @rjl493456442.
2016-09-29 22:53:59 +02:00
Felix Lange
ba8c4c6b1a trie: remove dependency on package crypto
Package crypto needs cgo, which is inconvenient for some build
configurations.
2016-09-29 22:23:24 +02:00
Felix Lange
cd791bd855 core, trie: replace state caches with trie journal 2016-09-28 11:27:28 +03:00
Péter Szilágyi
748d1c171d core, core/state, trie: enterprise hand-tuned multi-level caching 2016-05-26 16:33:09 +03:00
Ricardo Catalinas Jiménez
436fc8d76a all: Rename crypto.Sha3{,Hash}() to crypto.Keccak256{,Hash}()
As we aren't really using the standarized SHA-3
2016-02-21 22:34:34 +00:00
zsfelfoldi
ef422ee1e1 light: implemented odr-capable trie and state structures 2015-12-17 16:07:54 +01:00
zsfelfoldi
52904ae32f trie: added error handling
Created alternate versions of Trie and SecureTrie functions that can return a MissingNodeError (used by ODR services)
2015-12-01 01:38:32 +01:00
Péter Szilágyi
b97e34a8e4 eth/downloader: concurrent receipt and state processing 2015-10-19 10:03:10 +03:00
Felix Lange
565d9f2306 core, trie: new trie 2015-09-22 22:53:49 +02:00
Jeffrey Wilcke
b8ca0a830e eth, trie: removed key prefixing from state entries & merge db fix
Fixed database merge strategy to use the correct database. Due to a copy
paste fail when doing type evaluation the same database was being
iterated (chain), all others were ignored.

Removed state prefixing because {H(code): code} is stored in the same
database as the rest of the state.
2015-08-13 20:44:03 +02:00
Jeffrey Wilcke
a23478c0be core, eth, trie, xeth: merged state, chain, extra databases in one 2015-08-07 22:29:02 +02:00
Ethan Buchman
c1d516546d faster hex-prefix codec and string -> []byte 2015-08-06 03:17:59 -04:00
Felix Lange
bfbcfbe4a9 all: fix license headers one more time
I forgot to update one instance of "go-ethereum" in commit 3f047be5a.
2015-07-23 18:35:11 +02:00
Felix Lange
3f047be5aa all: update license headers to distiguish GPL/LGPL
All code outside of cmd/ is licensed as LGPL. The headers
now reflect this by calling the whole work "the go-ethereum library".
2015-07-22 18:51:45 +02:00
Felix Lange
bdae4fd573 all: add some godoc synopsis comments 2015-07-07 14:12:45 +02:00
Felix Lange
ea54283b30 all: update license information 2015-07-07 14:12:44 +02:00