* Revert "remove tombstones (#426)"
This reverts commit aa6bab40e8.
* tombstones don't hide storage or account anymore
* auto-format code by prettier (similar to gofmt)
* wow, it works.....
* small simplification, but need make it more clear
* rebase to master
* rebase to master
* rebase to master
* re-run ci
* clean test files
* remove allocations related to "remove incarnation" actions
* invalidate startKeyNoInc when startKey changed
* dbutils.RemoveIncarnationFromKey - doesn't do allocation
* another way to check if account has storage
* cleanup
* v0 of walk by db version
* save progress, to switch to another task. Put tombstone is still not correct.
* place tombstone only if exists something to hide
* db-based implementation
* db-based implementation
* db-based implementation
* fix prop check
* improve prop check logic
* Need custom logic to skip subtree for account and storage buckets because storage bucket has incarnation in key
* rebase to master
* remove all tombstones when account deleted
* remove all tombstones when account deleted
* added db integrity check
* don't rely on account.Root because it valid only about last incarnation
* remove all tombstones when account deleted
* dial with incarnation in MultiWalk2
* dial with incarnation in MultiWalk2
* when fixedbytes=40 resolver did compare cacheKey with storageKey without removing incarnation
* rebase to master
* rebase to master
Co-authored-by: alex.sharov <alex.sharov@lazada.com>
* implement NoValueCursor interface
* cleanup
* fix tests
* add more stats data to ui
* can't display error
* re-open DB low-level net interface when changing db
* re-open DB low-level net interface when changing db
* fix problem with displaying errors
* run ci
* improve prop check logic
* storage tombstones integrity checks UI
* storage page
* make DB configurable
Co-authored-by: alex.sharov <alex.sharov@lazada.com>
* badger v2 investigation
* buf pool - use native New method and avoid double checks
* db.Open prototype
* db.Tx/Bucket/Cursor prototypes
* Chained config
* Item concept added
* save changes to test on master
* make hack resumable
* Design document v0
* Cursor concept
* less brackets syntax of cursor builder
* benchmarks
* cleanup fs
* test for context cancelations
* test for context cancelations
* test for cursor.Prefix option
* add ForEachKey method
* add ForEachKey method
* add naming explanation
* experiment of non-pointers cursor/bucket
* .Bucket() and .Cursor() doesn't returns error
* .Bucket() and .Cursor() doesn't returns error
* .Bucket() and .Cursor() doesn't returns error
* remove CursorOpts concept
* more test-cases
* simplify open api
* Tx, Bucket, Cursor - now are interfaces
* Tx, Bucket, Cursor - now are interfaces
* switch to interfaces
* rebase master
Co-authored-by: alex.sharov <alex.sharov@lazada.com>
* #remove debug prints
* remove storage-mode="i"
* minnet re-execute hack with checkpoints
* minnet re-execute hack with checkpoints
* rollback to master setup
* mainnet re-exec hack
* rollback some changes
* v0 of "push down" functionality
* move all logic to own functions
* handle case when re-created account already has some storage
* clear path for storage
* try to rely on tree structure (but maybe need to rely on DB because can be intra-block re-creations of account)
* fix some bugs with indexes, moving to tests
* tests added
* make linter happy
* make linter happy
* simplify logic
* adjust comparison of keys with and without incarnation
* test for keyIsBefore
* test for keyIsBefore
* better nibbles alignment
* better nibbles alignment
* cleanup
* continue work on tests
* simplify test
* check tombstone existence before pushing it down.
* put tombstone only when account deleted, not created
* put tombstone only when account has storage
* make linter happy
* test for storage resolver
* make fixedbytes work without incarnation
* fix panic on short keys
* use special comparison only when working with keys from cache
* add blockNr for better tracing
* fix: incorrect tombstone check
* fix: incorrect tombstone check
* trigger ci
* hack for problem block
* more test-cases
* add test case for too long keys
* speedup cached resolver by removing bucket creation transaction
* remove parent type check in pruning, remove unused copy from mutation.put
* dump resolving info on fail
* dump resolving info on fail
* set tombstone everytime for now to check if it will help
* on unload: check parent type, not type of node
* fix wrong order of checking node type
* fix wrong order of checking node type
* rebase to new master
* make linter happy
* rebase to new master
* place tombstone only if acc has storage
* rebase master
* rebase master
* rebase master
* rebase master
Co-authored-by: alex.sharov <alex.sharov@lazada.com>
* squash commits
* enable storage cache
* make linter happy
* fix subtree prefix len check
* save cahnges to test master
* remove restriction on prefix len
* fix comparison of last bits
* fix wrong alignment
* remove debug prints
* commit current state
* commit current state
* avoid changing state of resolver from multiwalk
* remove debug code
* remove debug code
* remove debug code
* remove unnecessary copy
* make code more readable
* reduce rebuildHashes initial resolution
* fix test after rebase to master
* make code more readable
* improve pruner
* pruner add IntermediateCache bucket
* fix panic in Walk on short keys
* reduce allocations for storage keys decompression by increasing default buffer size
* re-run CI
* fix iterator behaviour
* rename cache to hash for unification
* re-run ci
* avoid using underlying DB
* hash all subtree nodes before unload
* fix getNode method
* need to check node type, not parent - before put to hashBucket
* return back parent type check, doesn't work without it.
* don't recalculate hash again
* move unloadFunc from trie to pruner
* rename bucket to shorter name
* rename bucket to shorter name
* clean
* rebase to master
* Enable testGetNodeData by default
* Crude TestHashMapLeak
* nodeFlag ->nodeRef
* linter
* accountNode shouln't be in hashMap since only branch and short nodes are the standart ones
* Finalize hash eviction logic
* small changes to TestHashMapLeak
* Fix for the incorrect hash
Co-authored-by: ledgerwatch <akhounov@gmail.com>
* Use TrieDbState for tx pool
* Not initialise tx pool until state is loaded
* Add preimage
* Fix account
* Print codehash
* Print correct code hash
* Print incarnatin
* Print incarnatin
* Use proper incarnation
* Print dbValue
* Actually fix
* Actually fix
* Fix verifySnapshot
* readAccount to get code hash
* Next incarnation
* Print addrHashes with 0 incarnations
* Print storage history
* Print storage history
* Print storage history
* Print storage history
* Print all storage history
* print change set keys
* print change set keys
* print change set keys
* print change set keys
* Not print codebucket info
* Fixes
* Fix for incarnation
* Fix for storage history bucket
* Try to fix the leak
* Try to fix the leak
* Try to fix the leak
* Try to fix the leak
* Try to fix the leak
* Try to fix the leak
* Try to fix the leak
* Fix embedded nodes
* Hasher
* Fix
* Test fixes
* Add experimental debug flag
* Fix tx_pool_test
* Disable GetNodeData test unless in experiment
* Fix more tests
* Fix lint and revert some changes
* Fix lint
* Fix lint
* add context
* extract chain events
* run commit in goroutines
* mine only on canonical
* typo
* linters
* fmt
* mark unused methods
* restore stress test
* test single miner
* remove unsafe Trie storage
* remove locks from miner
* restore interrupt
* remove result goroutine
* remove unconfirmedBlocks
* cherry-pick 04a1d475ff1a36ad8f92fec80385df18c52bdc1f
* extract uncles
* one miner succeeded
* restore context cancel
* cleanup
* skip an unstable test
* remove pending state
* use context instead of interrupt func
* calculate sealHash only once
* comment out unstable test
* after merge
* fix after merge
Co-authored-by: ledgerwatch <akhounov@gmail.com>
* add env INTERMEDIATE_TRIE_CACHE
* try to use assert.New() pattern
* Fix "maligned" linter warnings to reduce space consumption of structs:
core/types/accounts/account.go:18:14: struct of size 136 bytes could be of size 128 bytes (maligned)
type Account struct {
--
trie/node.go:44:10: struct of size 80 bytes could be of size 72 bytes (maligned)
duoNode struct {
--
trie/resolve_set.go:28:17: struct of size 56 bytes could be of size 48 bytes (maligned)
type ResolveSet struct {
--
trie/resolver.go:34:15: struct of size 88 bytes could be of size 72 bytes (maligned)
type Resolver struct {
--
trie/visual.go:32:17: struct of size 104 bytes could be of size 96 bytes (maligned)
type VisualOpts struct {
* added prefix tree to analyses to reduce memory usage
* make new partition every day
* merge concepts of reporter and snapshot
* tests for .FirstKey() and .NextKey()
* save state
* remove repair
* save state
* remove emptydb check
* save state
* add walkAsOf test
* add WalkAsOf and MultiWalkAsOf tests
* deployed contracts counter
* reference counter for contract code
* drop storage root&contract hash for changesets
* start incarnation is 1(save state)
* fix ReorgOverSelfDestruct test
* hack fix TestReorgOverSelfDestruct
* test benchmark
* cleanup
* remove useless debug
* remove print trie
* return remove subtrie call to updateTrieRoot
* save state
* add mutation test
* remove useless test
* fix
* added mutation commit test
* rename experiment to thin history
* thin history mutation commit test
* fix ethdb tests
* getAsOf test
* add test&fix history index
* fix test
* make test for index search
* compute trie root incarnation fix
* tests fixes
* done job in case of panic
* fix lint
* fix&test for bad incarnation
* fix initial incarnation for genesis
* fix lint
* fix changeset test
* fix storage ranges test
* fix lint
* move set incarnation to create contract
* add comment
Co-authored-by: ledgerwatch <akhounov@gmail.com>
Co-authored-by: Evgeny Danilenko <6655321@bk.ru>
This change adds tests for the virtual clock and aligns the interface
with the time package by renaming Cancel to Stop. It also removes the
binary search from Stop because it complicates the code unnecessarily.
Gollvm has very aggressive dead code elimination that completely
removes one of these two benchmarks. To prevent this, use the
result of the benchmark (a boolean), and to be "fair", make the
transformation to both benchmarks.
To be reliably assured of not removing the code, "use" means
assigning to an exported global. Non-exported globals and
//go:noinline functions are possibly subject to this optimization.
* accounts/abi/bind: Accept function ptr parameter
They are translated as [24]byte
* Add Java template version
* accounts/abi/bind: fix merge issue
* Fix CI
* core: reinit chain from freezer in batches
* core/rawdb: concurrent database reinit from freezer dump
* core/rawdb: reinit from freezer in sequential order
* core, eth: some fixes for freezer
* vendor, core/rawdb, cmd/geth: add db inspector
* core, cmd/utils: check ancient store path forceily
* cmd/geth, common, core/rawdb: a few fixes
* cmd/geth: support windows file rename and fix rename error
* core: support ancient plugin
* core, cmd: streaming file copy
* cmd, consensus, core, tests: keep genesis in leveldb
* core: write txlookup during ancient init
* core: bump database version
* core/vm: remove function call for stack validation from evm runloop
* core/vm: separate gas calc into static + dynamic
* core/vm: optimize push1
* core/vm: reuse pooled bigints for ADDRESS, ORIGIN and CALLER
* core/vm: use generic error message for jump/jumpi, to avoid string interpolation
* testdata: fix tests for new error message
* core/vm: use 64-bit memory calculations
* core/vm: fix error in memory calculation
* core/vm: address review concerns
* core/vm: avoid unnecessary use of big.Int:BitLen()
* common/fdlimit: cap on MacOS file limits, fixes#18994
* common/fdlimit: fix Maximum-check to respect OPEN_MAX
* common/fdlimit: return error if OPEN_MAX is exceeded in Raise()
* common/fdlimit: goimports
* common/fdlimit: check value after setting fdlimit
* common/fdlimit: make comment a bit more descriptive
* cmd/utils: make fdlimit happy path a bit cleaner
* Initial work on a graphql API
* Added receipts, and more transaction fields.
* Finish receipts, add logs
* Add transactionCount to block
* Add types and .
* Update Block type to be compatible with ethql
* Rename nonce to transactionCount in Account, to be compatible with ethql
* Update transaction, receipt and log to match ethql
* Add query operator, for a range of blocks
* Added ommerCount to Block
* Add transactionAt and ommerAt to Block
* Added sendRawTransaction mutation
* Add Call and EstimateGas to graphQL API
* Refactored to use hexutil.Bytes instead of HexBytes
* Replace BigNum with hexutil.Big
* Refactor call and estimateGas to use ethapi struct type
* Replace ethgraphql.Address with common.Address
* Replace ethgraphql.Hash with common.Hash
* Converted most quantities to Long instead of Int
* Add support for logs
* Fix bug in runFilter
* Restructured Transaction to work primarily with headers, so uncle data is reported properly
* Add gasPrice API
* Add protocolVersion API
* Add syncing API
* Moved schema into its own source file
* Move some single use args types into anonymous structs
* Add doc-comments
* Fixed backend fetching to use context
* Added (very) basic tests
* Add documentation to the graphql schema
* Fix reversion for formatting of big numbers
* Correct spelling error
* s/BigInt/Long/
* Update common/types.go
* Fixes in response to review
* Fix lint error
* Updated calls on private functions
* Fix typo in graphql.go
* Rollback ethapi breaking changes for graphql support
Co-Authored-By: Arachnid <arachnid@notdot.net>
* first impl of eth_getProof
* fixed docu
* added comments and refactored based on comments from holiman
* created structs
* handle errors correctly
* change Value to *hexutil.Big in order to have the same output as parity
* use ProofList as return type
This PR implements les.freeClientPool. It also adds a simulated clock
in common/mclock, which enables time-sensitive tests to run quickly
and still produce accurate results, and package common/prque which is
a generalised variant of prque that enables removing elements other
than the top one from the queue.
les.freeClientPool implements a client database that limits the
connection time of each client and manages accepting/rejecting
incoming connections and even kicking out some connected clients. The
pool calculates recent usage time for each known client (a value that
increases linearly when the client is connected and decreases
exponentially when not connected). Clients with lower recent usage are
preferred, unknown nodes have the highest priority. Already connected
nodes receive a small bias in their favor in order to avoid accepting
and instantly kicking out clients.
Note: the pool can use any string for client identification. Using
signature keys for that purpose would not make sense when being known
has a negative value for the client. Currently the LES protocol
manager uses IP addresses (without port address) to identify clients.
This package was meant to hold an improved 256 bit integer library, but
the effort was abandoned in 2015. AFAIK nothing ever used this package.
Time to say goodbye.
Allow the --abi flag to be given - to indicate that it should read the
ABI information from standard input. It expects to read the solc output
with the --combined-json flag providing bin, abi, userdoc, devdoc, and
metadata, and works very similarly to the internal invocation of solc,
except it allows external invocation of solc.
This facilitates integration with more complex solc invocations, such
as invocations that require path remapping or --allow-paths tweaks.
Simple usage example:
solc --combined-json bin,abi,userdoc,devdoc,metadata *.sol | abigen --abi -
* common: delete StringToAddress, StringToHash
These functions are confusing because they don't parse hex, but use the
bytes of the string. This change removes them, replacing all uses of
StringToAddress(s) by BytesToAddress([]byte(s)).
* eth/filters: remove incorrect use of common.BytesToAddress
* common/fdlimit: Move fdlimit files to separate package
When go-ethereum is used as a library the calling program need to set
the FD limit.
This commit extract fdlimit files to a separate package so it can be
used outside of go-ethereum.
* common/fdlimit: Remove FdLimit from functions signature
* common/fdlimit: Rename fdlimit functions
This adds type and struct field context to error messages.
Instead of "hex string of odd length" users will now see "json: cannot
unmarshal hex string of odd length into Go struct field SendTxArgs.from
of type common.Address".
Byte padding function should return the given slice if the length is
smaller or equal rather than *only* smaller than.
This fix improves almost all EVM push operations.
* p2p/discover, p2p/discv5: add marshaling methods to Node
* p2p/netutil: make Netlist decodable from TOML
* common/math: encode nil HexOrDecimal256 as 0x0
* cmd/geth: add --config file flag
* cmd/geth: add missing license header
* eth: prettify Config again, fix tests
* eth: use gasprice.Config instead of duplicating its fields
* eth/gasprice: hide nil default from dumpconfig output
* cmd/geth: hide genesis block in dumpconfig output
* node: make tests compile
* console: fix tests
* cmd/geth: make TOML keys look exactly like Go struct fields
* p2p: use discovery by default
This makes the zero Config slightly more useful. It also fixes package
node tests because Node detects reuse of the datadir through the
NodeDatabase.
* cmd/geth: make ethstats URL settable through config file
* cmd/faucet: fix configuration
* cmd/geth: dedup attach tests
* eth: add comment for DefaultConfig
* eth: pass downloader.SyncMode in Config
This removes the FastSync, LightSync flags in favour of a more
general SyncMode flag.
* cmd/utils: remove jitvm flags
* cmd/utils: make mutually exclusive flag error prettier
It now reads:
Fatal: flags --dev, --testnet can't be used at the same time
* p2p: fix typo
* node: add DefaultConfig, use it for geth
* mobile: add missing NoDiscovery option
* cmd/utils: drop MakeNode
This exposed a couple of places that needed to be updated to use
node.DefaultConfig.
* node: fix typo
* eth: make fast sync the default mode
* cmd/utils: remove IPCApiFlag (unused)
* node: remove default IPC path
Set it in the frontends instead.
* cmd/geth: add --syncmode
* cmd/utils: make --ipcdisable and --ipcpath mutually exclusive
* cmd/utils: don't enable WS, HTTP when setting addr
* cmd/utils: fix --identity
Metadata is provided as JSON string, rather than as JSON object. This
ensures that we can decode to a set of bytes that will be consistent
with the swarm hash embedded in the code, without worrying about
ambiguities of spacing, ordering, or escaping.
This commit makes the wrapper types more generally applicable.
encoding.TextMarshaler is supported by most codec implementations (e.g.
for yaml).
The tests now ensure that package json actually recognizes the custom
marshaler implementation irrespective of how it is implemented.
The Uint type has new tests, too. These are tricky because uint size
depends on the CPU word size. Turns out that there was one incorrect
case where decoding returned ErrUint64Range instead of ErrUintRange.
* common/math: optimize PaddedBigBytes, use it more
name old time/op new time/op delta
PaddedBigBytes-8 71.1ns ± 5% 46.1ns ± 1% -35.15% (p=0.000 n=20+19)
name old alloc/op new alloc/op delta
PaddedBigBytes-8 48.0B ± 0% 32.0B ± 0% -33.33% (p=0.000 n=20+20)
* all: unify big.Int zero checks
Various checks were in use. This commit replaces them all with Int.Sign,
which is cheaper and less code.
eg templates:
func before(x *big.Int) bool { return x.BitLen() == 0 }
func after(x *big.Int) bool { return x.Sign() == 0 }
func before(x *big.Int) bool { return x.BitLen() > 0 }
func after(x *big.Int) bool { return x.Sign() != 0 }
func before(x *big.Int) int { return x.Cmp(common.Big0) }
func after(x *big.Int) int { return x.Sign() }
* common/math, crypto/secp256k1: make ReadBits public in package math
* common: remove CurrencyToString
Move denomination values to params instead.
* common: delete dead code
* common: move big integer operations to common/math
This commit consolidates all big integer operations into common/math and
adds tests and documentation.
There should be no change in semantics for BigPow, BigMin, BigMax, S256,
U256, Exp and their behaviour is now locked in by tests.
The BigD, BytesToBig and Bytes2Big functions don't provide additional
value, all uses are replaced by new(big.Int).SetBytes().
BigToBytes is now called PaddedBigBytes, its minimum output size
parameter is now specified as the number of bytes instead of bits. The
single use of this function is in the EVM's MSTORE instruction.
Big and String2Big are replaced by ParseBig, which is slightly stricter.
It previously accepted leading zeros for hexadecimal inputs but treated
decimal inputs as octal if a leading zero digit was present.
ParseUint64 is used in places where String2Big was used to decode a
uint64.
The new functions MustParseBig and MustParseUint64 are now used in many
places where parsing errors were previously ignored.
* common: delete unused big integer variables
* accounts/abi: replace uses of BytesToBig with use of encoding/binary
* common: remove BytesToBig
* common: remove Bytes2Big
* common: remove BigTrue
* cmd/utils: add BigFlag and use it for error-checked integer flags
While here, remove environment variable processing for DirectoryFlag
because we don't use it.
* core: add missing error checks in genesis block parser
* common: remove String2Big
* cmd/evm: use utils.BigFlag
* common/math: check for 256 bit overflow in ParseBig
This is supposed to prevent silent overflow/truncation of values in the
genesis block JSON. Without this check, a genesis block that set a
balance larger than 256 bits would lead to weird behaviour in the VM.
* cmd/utils: fixup import
Reworked the EVM gas instructions to use 64bit integers rather than
arbitrary size big ints. All gas operations, be it additions,
multiplications or divisions, are checked and guarded against 64 bit
integer overflows.
In additon, most of the protocol paramaters in the params package have
been converted to uint64 and are now constants rather than variables.
* common/math: added overflow check ops
* core: vmenv, env renamed to evm
* eth, internal/ethapi, les: unmetered eth_call and cancel methods
* core/vm: implemented big.Int pool for evm instructions
* core/vm: unexported intPool methods & verification methods
* core/vm: added memoryGasCost overflow check and test
Reworked the EVM gas instructions to use 64bit integers rather than
arbitrary size big ints. All gas operations, be it additions,
multiplications or divisions, are checked and guarded against 64 bit
integer overflows.
In additon, most of the protocol paramaters in the params package have
been converted to uint64 and are now constants rather than variables.
* common/math: added overflow check ops
* core: vmenv, env renamed to evm
* eth, internal/ethapi, les: unmetered eth_call and cancel methods
* core/vm: implemented big.Int pool for evm instructions
* core/vm: unexported intPool methods & verification methods
* core/vm: added memoryGasCost overflow check and test
The crash when compiling stdin was fixed in solc 0.3.6 (released
2016-08-10). While here, simplify the test so it runs with any solc
version.
Fixes#3484. The byte code was different for each run because recent
solc embeds the swarm hash of contract metadata into the code. When
compiling from stdin the name in the metadata is constant.
The registrar was broken, unmaintained and there is a much better
replacement: ENS.
(cherry picked from commit 6ca8f57b08d550613175260cab7633adcacbe6ab)
This commit implements EIP158 part 1, 2, 3 & 4
1. If an account is empty it's no longer written to the trie. An empty
account is defined as (balance=0, nonce=0, storage=0, code=0).
2. Delete an empty account if it's touched
3. An empty account is redefined as either non-existent or empty.
4. Zero value calls and zero value suicides no longer consume the 25k
reation costs.
params: moved core/config to params
Signed-off-by: Jeffrey Wilcke <jeffrey@ethereum.org>
