* Revert "remove tombstones (#426)"
This reverts commit aa6bab40e8.
* tombstones don't hide storage or account anymore
* auto-format code by prettier (similar to gofmt)
* wow, it works.....
* small simplification, but need make it more clear
* rebase to master
* rebase to master
* rebase to master
* re-run ci
* clean test files
* Debugging the cached resolver
* make logic of comparison more explicit
* fix stuck on block 696817
* Remove tracing
* Fix CI
* Search for root hash
Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
* CheckChangeSets and thin history
* small code de-duplication
* small code clean-up
* Fix an error in mutation.getNoLock
* CheckChangeSets: truly make historyfile = chaindata by default
* use NoValues cursor where possible
* add ctx
* fix broken logs
* rebase master
* rebase master
* simplify generators
* hack to measure space distribution
* naive epoch and chunking implementation
* make stateless loop cancelable
* make stateless loop cancelable
* remove one rlp layer
* eth64 protocol support - add forkId to status message
* remove allocations related to "remove incarnation" actions
* invalidate startKeyNoInc when startKey changed
* dbutils.RemoveIncarnationFromKey - doesn't do allocation
* Introduce NoValuesCursor. From() method is useless because can be replaced by Seek().`
* implement NoValueCursor interface
* use abstract db in restapi
* cleanup .md
* move .Put out of .View and remove deletion of all tombstones when delete acc
* move .Put out of .View and remove deletion of all tombstones when delete acc
* introduce code node
* replace codeMap with code touches
* fix a comment
* fixups to tests
* fix compile error
* fix getnodedata tests
* add tests and test stubs
* add more test stubs
* add test method bodies
* add and fix more tests on trie for new codenode
* add test change code between blocks
* fix crash in stateless
* remove unneded files
* remove comment
* fix deleted account code
* fix resolve set builder for code nodes
* another way to check if account has storage
* cleanup
* v0 of walk by db version
* save progress, to switch to another task. Put tombstone is still not correct.
* place tombstone only if exists something to hide
* db-based implementation
* db-based implementation
* db-based implementation
* fix prop check
* improve prop check logic
* Need custom logic to skip subtree for account and storage buckets because storage bucket has incarnation in key
* rebase to master
* remove all tombstones when account deleted
* remove all tombstones when account deleted
* added db integrity check
* don't rely on account.Root because it valid only about last incarnation
* remove all tombstones when account deleted
* dial with incarnation in MultiWalk2
* dial with incarnation in MultiWalk2
* when fixedbytes=40 resolver did compare cacheKey with storageKey without removing incarnation
* rebase to master
* rebase to master
Co-authored-by: alex.sharov <alex.sharov@lazada.com>
* root was unused in BlockChain.StateAt
* TestDoubleAccountRemoval
* Preserve the original when a contract is self-destructed and then its address is touched in the same block (e.g. #
156634)
* Add revive and phoenix
* store enode address to file, then read it from tester
* store enode address to file, then read it from tester
* rebase master
* fix miss-type
* dbg p2p-sub-protocol, add self-destruct test case
* re-create blockFetcher
* exit syncer loop and start new one
* rebase to master
* use core.GenerateChain
* root miss-match
* introduce reduceComplexity flag
* fix transfer to 0 account
* cleanup
* test-case for intermediate cache
* clean
* clean
* clean
* fix handler panic
Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
Co-authored-by: alex.sharov <alex.sharov@lazada.com>
* #remove debug prints
* remove storage-mode="i"
* minnet re-execute hack with checkpoints
* minnet re-execute hack with checkpoints
* rollback to master setup
* mainnet re-exec hack
* rollback some changes
* v0 of "push down" functionality
* move all logic to own functions
* handle case when re-created account already has some storage
* clear path for storage
* try to rely on tree structure (but maybe need to rely on DB because can be intra-block re-creations of account)
* fix some bugs with indexes, moving to tests
* tests added
* make linter happy
* make linter happy
* simplify logic
* adjust comparison of keys with and without incarnation
* test for keyIsBefore
* test for keyIsBefore
* better nibbles alignment
* better nibbles alignment
* cleanup
* continue work on tests
* simplify test
* check tombstone existence before pushing it down.
* put tombstone only when account deleted, not created
* put tombstone only when account has storage
* make linter happy
* test for storage resolver
* make fixedbytes work without incarnation
* fix panic on short keys
* use special comparison only when working with keys from cache
* add blockNr for better tracing
* fix: incorrect tombstone check
* fix: incorrect tombstone check
* trigger ci
* hack for problem block
* more test-cases
* add test case for too long keys
* speedup cached resolver by removing bucket creation transaction
* remove parent type check in pruning, remove unused copy from mutation.put
* dump resolving info on fail
* dump resolving info on fail
* set tombstone everytime for now to check if it will help
* on unload: check parent type, not type of node
* fix wrong order of checking node type
* fix wrong order of checking node type
* rebase to new master
* make linter happy
* rebase to new master
* place tombstone only if acc has storage
* rebase master
* rebase master
* rebase master
* rebase master
Co-authored-by: alex.sharov <alex.sharov@lazada.com>
* Fix download only
* Fix lint
* Reset references
* Only reset on error
* Potential fixes
* no NPE
* no NPE
* Not use multi-put
* Reduce ideal batch size for download only
* Handle tds == nil
* remove nested mutation
* Return multiput
* Better reporting
* Reduce batch size for download only
* Avoid extra copying
* Avoid extra copying
* IdealBatchSize
* Not write tx lookup entries
* Larger batches
* Go back to normal batch size
* Fix lint
* Gen tx lookup
* print progress
* Add filling up the lookup array
* Show tx count
* Introduce second round
* Add generating tx lookup
* Fix lint
* properly stop at specified block
* measure the duration of the last phase
* not to fail if the bucket is not found
* Fix lint
* Alternative tx generation
* Fix out of memory
* Fix out of memory
* Split in parts to conserve memory
* Copy keys
* Fix lint
* Fix lint
* core/vm/runtime: add test for blockhash
* core/evm: less iteration in blockhash
* core/vm/runtime: nitpickfix
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
* squash commits
* enable storage cache
* make linter happy
* fix subtree prefix len check
* save cahnges to test master
* remove restriction on prefix len
* fix comparison of last bits
* fix wrong alignment
* remove debug prints
* commit current state
* commit current state
* avoid changing state of resolver from multiwalk
* remove debug code
* remove debug code
* remove debug code
* remove unnecessary copy
* make code more readable
* reduce rebuildHashes initial resolution
* fix test after rebase to master
* make code more readable
* improve pruner
* pruner add IntermediateCache bucket
* fix panic in Walk on short keys
* reduce allocations for storage keys decompression by increasing default buffer size
* re-run CI
* fix iterator behaviour
* rename cache to hash for unification
* re-run ci
* avoid using underlying DB
* hash all subtree nodes before unload
* fix getNode method
* need to check node type, not parent - before put to hashBucket
* return back parent type check, doesn't work without it.
* don't recalculate hash again
* move unloadFunc from trie to pruner
* rename bucket to shorter name
* rename bucket to shorter name
* clean
* rebase to master
* Use TrieDbState for tx pool
* Not initialise tx pool until state is loaded
* Add preimage
* Fix account
* Print codehash
* Print correct code hash
* Print incarnatin
* Print incarnatin
* Use proper incarnation
* Print dbValue
* Actually fix
* Actually fix
* Fix verifySnapshot
* readAccount to get code hash
* Next incarnation
* Print addrHashes with 0 incarnations
* Print storage history
* Print storage history
* Print storage history
* Print storage history
* Print all storage history
* print change set keys
* print change set keys
* print change set keys
* print change set keys
* Not print codebucket info
* Fixes
* Fix for incarnation
* Fix for storage history bucket
* Try to fix the leak
* Try to fix the leak
* Try to fix the leak
* Try to fix the leak
* Try to fix the leak
* Try to fix the leak
* Try to fix the leak
* Fix embedded nodes
* Hasher
* Fix
* Test fixes
* Add experimental debug flag
* Fix tx_pool_test
* Disable GetNodeData test unless in experiment
* Fix more tests
* Fix lint and revert some changes
* Fix lint
* Fix lint
* add context
* extract chain events
* run commit in goroutines
* mine only on canonical
* typo
* linters
* fmt
* mark unused methods
* restore stress test
* test single miner
* remove unsafe Trie storage
* remove locks from miner
* restore interrupt
* remove result goroutine
* remove unconfirmedBlocks
* cherry-pick 04a1d475ff1a36ad8f92fec80385df18c52bdc1f
* extract uncles
* one miner succeeded
* restore context cancel
* cleanup
* skip an unstable test
* remove pending state
* use context instead of interrupt func
* calculate sealHash only once
* comment out unstable test
* after merge
* fix after merge
Co-authored-by: ledgerwatch <akhounov@gmail.com>
* add env INTERMEDIATE_TRIE_CACHE
* try to use assert.New() pattern
* Fix "maligned" linter warnings to reduce space consumption of structs:
core/types/accounts/account.go:18:14: struct of size 136 bytes could be of size 128 bytes (maligned)
type Account struct {
--
trie/node.go:44:10: struct of size 80 bytes could be of size 72 bytes (maligned)
duoNode struct {
--
trie/resolve_set.go:28:17: struct of size 56 bytes could be of size 48 bytes (maligned)
type ResolveSet struct {
--
trie/resolver.go:34:15: struct of size 88 bytes could be of size 72 bytes (maligned)
type Resolver struct {
--
trie/visual.go:32:17: struct of size 104 bytes could be of size 96 bytes (maligned)
type VisualOpts struct {
* save state
* remove repair
* save state
* remove emptydb check
* save state
* add walkAsOf test
* add WalkAsOf and MultiWalkAsOf tests
* deployed contracts counter
* reference counter for contract code
* drop storage root&contract hash for changesets
* start incarnation is 1(save state)
* fix ReorgOverSelfDestruct test
* hack fix TestReorgOverSelfDestruct
* test benchmark
* cleanup
* remove useless debug
* remove print trie
* return remove subtrie call to updateTrieRoot
* save state
* add mutation test
* remove useless test
* fix
* added mutation commit test
* rename experiment to thin history
* thin history mutation commit test
* fix ethdb tests
* getAsOf test
* add test&fix history index
* fix test
* make test for index search
* compute trie root incarnation fix
* tests fixes
* done job in case of panic
* fix lint
* fix&test for bad incarnation
* fix initial incarnation for genesis
* fix lint
* fix changeset test
* fix storage ranges test
* fix lint
* move set incarnation to create contract
* add comment
Co-authored-by: ledgerwatch <akhounov@gmail.com>
Co-authored-by: Evgeny Danilenko <6655321@bk.ru>
* initial
* mining
* remove debug
* debug
* restore random seed in the mining tests
* green tests
* fix blockchain tests
* fix lint
* init miner only if asked
* linters
* do not store trie as singlton
* fmt
* new trieDbState constructor
This change:
- removes the PostChainEvents method on core.BlockChain.
- sorts 'removed log' events by block number.
- fire the NewChainHead event if we inject a canonical block into the chain
even if the entire insertion is not successful.
- guarantees correct event ordering in all cases.
* core: s/isEIP155/isEIP2/ (fix)
This signature variable name reflects a spec'd change
in gas cost for creating contracts as documented in EIP2 (Homestead HF).
https://github.com/ethereum/EIPs/blob/master/EIPS/eip-2.md#specification
* core: s/isEIP2/sIsHomestead/g
Use isHomestead since Homestead is what the caller
and rest of the code uses.
* build: use golangci-lint
This changes build/ci.go to download and run golangci-lint instead
of gometalinter.
* core/state: fix unnecessary conversion
* p2p/simulations: fix lock copying (found by go vet)
* signer/core: fix unnecessary conversions
* crypto/ecies: remove unused function cmpPublic
* core/rawdb: remove unused function print
* core/state: remove unused function xTestFuzzCutter
* core/vm: disable TestWriteExpectedValues in a different way
* core/forkid: remove unused function checksum
* les: remove unused type proofsData
* cmd/utils: remove unused functions prefixedNames, prefixFor
* crypto/bn256: run goimports
* p2p/nat: fix goimports lint issue
* cmd/clef: avoid using unkeyed struct fields
* les: cancel context in testRequest
* rlp: delete unreachable code
* core: gofmt
* internal/build: simplify DownloadFile for Go 1.11 compatibility
* build: remove go test --short flag
* .travis.yml: disable build cache
* whisper/whisperv6: fix ineffectual assignment in TestWhisperIdentityManagement
* .golangci.yml: enable goconst and ineffassign linters
* build: print message when there are no lint issues
* internal/build: refactor download a bit
* core/evm, contracts: avoid copying memory for input in calls + make ecrecover not modify input buffer
* core/vm: optimize mstore a bit
* core/vm: change Get -> GetCopy in vm memory access
* core/asm: Fix encoding of pushed labels
EVM uses big-endian byte-order, so to pad a label value to 4 bytes,
zeros must be added to the front, not the end.
* core/asm: Fix PC calculations when a label is pushed
Incrementing PC by 5 is only correct if the label appears after a jump,
in which case there is an implicit push. When it appears after an explicit
push, PC should only be incremented by 4.
* core/asm: Allow JUMP with no argument
This way, a label can be pushed explicitly, or loaded from memory to
implement a jump table.
In Geth, we have two sources for configuration:
(1) Config file
(2) Command line flag
Basically geth will first resolve config file and then overwrite
configs with command line flags.
This issue is: geth should only overwrite configs if flags are truly
set. So before we apply any flag to configs, `GlobalIsSet` check
is necessary.
* Bumping GOMAXPROCS for Badger
* fixes related to database size
* Schedule GC for Badger
* pacify linter
* Don't start GC for ephemeral Badger
* Don't log "Value log GC attempt didn't result in any cleanup"
* Start GC in backround
* Bump GC period and IdealBatchSize for Badger
* BadgerDatabase RewindData
* Boolean badger flag -> string database flag
* cosmetic change
The precompile at 0x09 wraps the BLAKE2b F compression function:
https://tools.ietf.org/html/rfc7693#section-3.2
The precompile requires 6 inputs tightly encoded, taking exactly 213
bytes, as explained below.
- `rounds` - the number of rounds - 32-bit unsigned big-endian word
- `h` - the state vector - 8 unsigned 64-bit little-endian words
- `m` - the message block vector - 16 unsigned 64-bit little-endian words
- `t_0, t_1` - offset counters - 2 unsigned 64-bit little-endian words
- `f` - the final block indicator flag - 8-bit word
[4 bytes for rounds][64 bytes for h][128 bytes for m][8 bytes for t_0]
[8 bytes for t_1][1 byte for f]
The boolean `f` parameter is considered as `true` if set to `1`.
The boolean `f` parameter is considered as `false` if set to `0`.
All other values yield an invalid encoding of `f` error.
The precompile should compute the F function as specified in the RFC
(https://tools.ietf.org/html/rfc7693#section-3.2) and return the updated
state vector `h` with unchanged encoding (little-endian).
See EIP-152 for details.
* graphql, internal/ethapi: extend eth_call
This PR offers the third option parameter for eth_call API.
Caller can specify a batch of contracts for overriding the
original account metadata(nonce, balance, code, state).
It has a few advantages:
* It's friendly for debugging
* It's can make on-chain contract lighter for getting rid of
state access functions
* core, internal: address comments
* params: add IsIstanbul to config + rules
IstanbulBlock, used to determine if the config IsIstanbul, is currently
left nil until an actual block is chosen.
* params, core/vm: implement EIP-1108
Old gas costs for elliptic curve operations are given the PreIstanbul
prefix, while current gas costs retain the unprefixed names. The actual
precompile implementations are the same, so they are factored out into
common functions that are called by the pre-Istanbul and current
precompile structs. Finally, an Istanbul precompile list is added that
references the new precompile structs, which in turn reference the new
gas costs.
* params: fix fork ordering, add missing chain compatibility check
* params, core/vm: deprecating gastable, part 1
* core/vm, params: deprecate gastable, use both constant and dynamic gas
* core/vm, params: remove gastable, remove copypaste
* core/vm: make use of the chainrules
* interpreter: make tracing count constant+dynamic gas
* core/vm: review concerns (param/method name changes)
* core/vm: make use of chainrules more
* eth: chain config (genesis + fork) ENR entry
* core/forkid, eth: protocol independent fork ID, update to CRC32 spec
* core/forkid, eth: make forkid a struct, next uint64, enr struct, RLP
* core/forkid: change forkhash rlp encoding from int to [4]byte
* eth: fixup eth entry a bit and update it every block
* eth: fix lint
* eth: fix crash in ethclient tests
This PR adds some hardening in the lower levels of the protocol stack, to bail early on invalid data. Primarily, attacks that this PR protects against are on the "annoyance"-level, which would otherwise write a couple of megabytes of data into the log output, which is a bit resource intensive.
This PR fixes an issue in chain indexer. Currently chain indexer will
validate whether the stored data is canonical by comparing section head
and canonical hash. But the header of the checkpoint may not exist in
the database. We should skip validation for sections below the
checkpoint.
* core/state, cmd/geth: streaming json output dump cmd + optional code+storage
* dump: add option to continue even if preimages are missing
* core, evm: lint nits
* cmd: use local flags for dump, omit empty code/storage
* core/state: fix state dump test
* core: move TxPool reorg and events to background goroutine
This change moves internal queue re-shuffling work in TxPool to a
background goroutine, TxPool.runReorg. Requests to execute runReorg are
accumulated by the new scheduleReorgLoop. The new loop also accumulates
transaction events.
The motivation for this change is making sends to txFeed synchronous
instead of sending them in one-off goroutines launched by 'add' and
'promoteExecutables'. If a downstream consumer of txFeed is blocked for
a while, reorg requests and events will queue up.
* core: remove homestead check in TxPool
This change removes tracking of the homestead block number from TxPool.
The homestead field was used to enforce minimum gas of 53000 for
contract creations after the homestead fork, but not before it. Since
nobody would want configure a non-homestead chain nowadays and contract
creations usually take more than 53000 gas, the extra correctness is
redundant and can be removed.
* core: fixes for review comments
* core: remove BenchmarkPoolInsert
This is useless now because there is no separate code path for
individual transactions anymore.
* core: fix pending counter metric
* core: fix pool tests
* core: dedup txpool announced events, discard stales
* core: reorg tx promotion/demotion to avoid weird pending gaps
* core: reinit chain from freezer in batches
* core/rawdb: concurrent database reinit from freezer dump
* core/rawdb: reinit from freezer in sequential order
