* add logging to loader
* use pure tx in etl loading, logs in mutation commit
* clean
* bletter logging and more cleanup
* bletter logging and more cleanup
* increase batch size to 500M
* better batch commit logging
* async fsync
* sync fsync
* sync fsync
* unify logging
* fix corner-case when etl can use empty bucket name
* fix tests
* better logging
* better logging
* rebase master
* remove lmdb.NoMetaSync flag for now
* consistent walk and multi-walk
* clean
* sub tx
* add consistent multi-put
* implement dupsort support in one new cursor method
* clear
* core: avoid modification of accountSet cache in tx_pool
when runReorg, we may copy the dirtyAccounts' accountSet cache to promoteAddrs
in which accounts will be promoted, however, if we have reset request at the
same time, we may reuse promoteAddrs and modify the cache content which is
against the original intention of accountSet cache. So, we need to make a new
slice here to avoid modify accountSet cache.
* core: fix flatten condition + comment
Co-authored-by: Felix Lange <fjl@twurst.com>
This replaces the two-stage shutdown scheme with the one we
use almost everywhere else: a single quit channel signalling
termination.
Co-authored-by: Felix Lange <fjl@twurst.com>
* eth/downloader: refactor downloader + queue
downloader, fetcher: throttle-metrics, fetcher filter improvements, standalone resultcache
downloader: more accurate deliverytime calculation, less mem overhead in state requests
downloader/queue: increase underlying buffer of results, new throttle mechanism
eth/downloader: updates to tests
eth/downloader: fix up some review concerns
eth/downloader/queue: minor fixes
eth/downloader: minor fixes after review call
eth/downloader: testcases for queue.go
eth/downloader: minor change, don't set progress unless progress...
eth/downloader: fix flaw which prevented useless peers from being dropped
eth/downloader: try to fix tests
eth/downloader: verify non-deliveries against advertised remote head
eth/downloader: fix flaw with checking closed-status causing hang
eth/downloader: hashing avoidance
eth/downloader: review concerns + simplify resultcache and queue
eth/downloader: add back some locks, address review concerns
downloader/queue: fix remaining lock flaw
* eth/downloader: nitpick fixes
* eth/downloader: remove the *2*3/4 throttling threshold dance
* eth/downloader: print correct throttle threshold in stats
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
# Conflicts:
# eth/downloader/downloader.go
# eth/downloader/downloader_test.go
# eth/downloader/queue.go
# eth/downloader/statesync.go
# eth/fetcher/block_fetcher.go
Solves issue#20582. Non-executable transactions should not be evicted on each tick if there are no promote transactions or if a pending/reset empties the pending list. Tests and logging expanded to handle these cases in the future.
core/tx_pool: use a ts for each tx in the queue, but only update the heartbeat on promotion or pending replaced
queuedTs proper naming
# Conflicts:
# core/tx_pool.go
* core: added local tx pool test case
* core, crypto: various allocation savings regarding tx handling
* core/txlist, txpool: save a reheap operation, avoid some bigint allocs
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
Reverting because this change started handling account balances as
uint64 in the transaction pool, which is incorrect.
This reverts commit af5c97aebe1d37486635521ef553cb8bd4bada13.
# Conflicts:
# core/types/transaction.go
This change further improves the performance of RLP encoding by removing
allocations for big.Int and [...]byte types. I have added a new benchmark
that measures RLP encoding of types.Block to verify that performance is
improved.
# Conflicts:
# core/types/block_test.go
# rlp/encode.go
# rlp/encode_test.go
* core: use uint64 for total tx costs instead of big.Int
* core: added local tx pool test case
* core, crypto: various allocation savings regarding tx handling
* Update core/tx_list.go
* core: added tx.GasPriceIntCmp for comparison without allocation
adds a method to remove unneeded allocation in comparison to tx.gasPrice
* core: handle pools full of locals better
* core/tests: benchmark for tx_list
* core/txlist, txpool: save a reheap operation, avoid some bigint allocs
Co-authored-by: Martin Holst Swende <martin@swende.se>
# Conflicts:
# common/math/integer.go
# core/tx_list.go
# core/tx_pool.go
# core/types/transaction.go
* core, crypto: various allocation savings regarding tx handling
* core: reduce allocs for gas price comparison
This change reduces the allocations needed for comparing different transactions to each other.
A call to `tx.GasPrice()` copies the gas price as it has to be safe against modifications and
also needs to be threadsafe. For comparing and ordering different transactions we don't need
these guarantees
* core: added tx.GasPriceIntCmp for comparison without allocation
adds a method to remove unneeded allocation in comparison to tx.gasPrice
* core/types: pool legacykeccak256 objects in rlpHash
rlpHash is by far the most used function in core that allocates a legacyKeccak256 object on each call.
Since it is so widely used it makes sense to add pooling here so we relieve the GC.
On my machine these changes result in > 100 MILLION less allocations and > 30 GB less allocated memory.
* reverted some changes
* reverted some changes
* trie: use crypto.KeccakState instead of replicating code
Co-authored-by: Martin Holst Swende <martin@swende.se>
# Conflicts:
# core/tx_list.go
# core/tx_pool.go
# core/types/block.go
# core/types/transaction.go
# trie/committer.go
# trie/hasher.go
This PR implements the EVM state transition tool, which is intended
to be the replacement for our retesteth client implementation.
Documentation is present in the cmd/evm/README.md file.
Co-authored-by: Felix Lange <fjl@twurst.com>
# Conflicts:
# cmd/evm/main.go
# core/state/dump.go
# core/state/state_test.go
# tests/state_test_util.go
* core/vm: fix incorrect computation of discount
During testing on Yolov1 we found that the way geth calculates the discount
is not in line with the specification. Basically what we did is calculate
128 * Bls12381GXMulGas * discount / 1000 whenever we received more than 128 pairs
of values. Correct would be to calculate k * Bls12381... for k > 128.
* core/vm: better logic for discount calculation
* core/vm: better calculation logic, added worstcase benchmarks
* core/vm: better benchmarking logic
Once we detect an invalid transaction during recovering signatures, we should
directly exclude this transaction to avoid validating the signatures hereafter.
This should optimize the validations times of transactions with invalid signatures
to only one time.
* return error from rawdb
* squash
* v14
* improve performance of put
* clean
* clean
* hide feature behind ENV variable
* cleanup
* cleanup
* disable ipc and make Readme less confusing (people thought points are depend on each-other)
* fix test
* cleanup
* cleanup
* save state
* cleanup
* add walkasof test for storage with chunks
* fix lint
* fix lint
* remove useless comments
* fix dump test
* move accountRange test to plain state
* get last block fixes
* cmd/tester to initiate reorg
* Second commit
* Turn on profile
* More
* Unwind function
* More fixes
* Before fixing tests
* Fix compile error
* Fix lint
* Fix formatting
* Fix lint
* Fix nil in tests
* Fix lint
* Decouple txpool from rawdb
* Fix some tx tests
* One more test
* Fix the rest of tests in tx_pool_test.go
* Fix formatting
* close goruotines
* fix using defer
* close pm managers a bit later
* fmt
* finalizer for in-mem db
* after cr
* revert finalizer
Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
* cmp to reference db if --reference_chaindata provided
* graceful shutdown
* add ./cmd/integration to makefile
* add interruption support for different stages
* don't compare errors directly
* save state
* walkAsOf by cs test passed
* cs search for plain state
* save state
* fix accounts tests
* refactor walkAsOf account tests
* fix storage test
* refactor walkAsOf storage tests
* fix lint
* fix test
* save state
* save state
* add test with fixed bits
* fmt
* add stages check
* fix lint
* fix lint
* remove obsoleted methods
```
go run ./cmd/integration reset --chaindata=...
go run ./cmd/integration state_stages -h
go run ./cmd/integration state_stages --chaindata=... --verbosity=3 --block=2_000_000 --unwind=10 --unwind_every=1_000
```
Also, it inherits flags from geth:
```
--pprof.cpuprofile=./cpu.out // to file
--pprof --pprof.port=6060 // launch pprof server
--metrics // sends to prometheus
```
* TestWatchNoDir not to be parallel
* Print ks and inc
* Print addrHash
* Change the buffer
* Print loading
* Skip
* More logging
* Error out earlier
* Handle empty codes
* Handle empty codes
* Remove logging
* Compare states
* Not do stage5
* compareBucket
* Preimage
* Clearer errors
* No need to clean up contract code
* Restore stage6
* Printing
* Skip the skipping
* Print all
* Change buffer type
* Add limit to stage5
* Always fail
* Remove exessive logging
* Restore buffer type
* Revert
* Print when exception
* Reenable skipping
* Skip storage items for deleted contracts
* not shortcut
* Remove removeAccount
* Re-enable state hashing
* Default to plain state
* Disable hashing state
* Reenable reset5
* Print unfurl list
* Enable removingAccount
* No printing
* Reenable stage5 commit
* Swap order of stages
* Prevent backwards promotion, reset tx lookup
* reset finish
* Introduce storage item replacement
* See if unwind works
* Restore removingAccount
* Don't do removeAccount for unwinding
* Possible fix
* Proper(er) fix
* Don't exclude unwinding
* Remove unwinding flag
* Fix formatting
* Fix lint
* Not to ignore blocks if they cause reorg
* Fix test, separate stages again
* Fix TestUnwind
* Fix stage
* Swap unwinding
* Revert to unwinding flag
* Print unfurl list
* Print
* Print inside receive
* Print after
* No printing
* Cleanup
* Not use blockCache when doing GetBlock
This adds a new API method on core.BlockChain to allow interrupting
running data inserts, and calls the method before shutting down the
downloader.
The BlockChain interrupt checks are now done through a method instead
of inlining the atomic load everywhere. There is no loss of efficiency from
this and it makes the interrupt protocol a lot clearer because the check is
defined next to the method that sets the flag.
# Conflicts:
# core/blockchain.go
# light/lightchain.go
This PR reimplements the light client server pool. It is also a first step
to move certain logic into a new lespay package. This package will contain
the implementation of the lespay token sale functions, the token buying and
selling logic and other components related to peer selection/prioritization
and service quality evaluation. Over the long term this package will be
reusable for incentivizing future protocols.
Since the LES peer logic is now based on enode.Iterator, it can now use
DNS-based fallback discovery to find servers.
This document describes the function of the new components:
https://gist.github.com/zsfelfoldi/3c7ace895234b7b345ab4f71dab102d4
# Conflicts:
# cmd/utils/flags.go
# core/forkid/forkid.go
# les/client.go
# les/client_handler.go
# les/commons.go
# les/distributor.go
# les/enr_entry.go
# les/fetcher.go
# les/lespay/client/valuetracker.go
# les/metrics.go
# les/peer.go
# les/protocol.go
# les/retrieve.go
# les/server.go
# les/serverpool.go
# les/test_helper.go
# les/utils/expiredvalue.go
# les/utils/weighted_select.go
# les/utils/weighted_select_test.go
# params/bootnodes.go
* Fix body fetch
* Reduce spurious reorgs
* Exit the sync cycle after unwinds
* Fix out of range
* No stalling check for staged sync
* Disable failing tests
* Remove duplicate log message
* Fix UnwindTest and add assertions
* Fix formatting
* Cleanup
* Fix off by one error with bodies
* Remove rollback
* remove unused slice from MultiPut
* mutation: reuse tuples slice and preallocate bucketPuts
* use bucketPool in kv_lmdb
* remove duplicated check of context status
* more benchmarks
* remove reusage of puts
* save state
* txlookup full results
* save state
* save state
* remove experiments
* some fix&lint
* add end key to txLookup and index generation
* change log message
* change log
* fix lint
* lint
* fix test
* Euphemerally -> Ephemerally
* Move StorageMode to ethdb and pass it to PrepareStagedSync
* linter
* Remove StorageModeThinHistory and move SetStorageModeIfNotExist into storage_mode.go
* Optionally write receipts in the execute stage
* memory profiler
* linter
* proper linter fix
* linter
* typo
* Merge stateDb with changeDb so that all-or-nothing is commited in stage_execute
* Start from 9m7
* Regenerate IH + receipts
* Only stats for iH bucket
* Persist receipts
* Go all in
* Start from block 10m
* Convert DbState to use plain state
* Fix findHistory
* Hard-code export
* More fixes
* Fix test
* Fix formatting
* Introduce PlainDbState
* Actually return PlainDbState
* Fix formatting
* Fix name style lint
* Fix linters
* Fix history_test
* Fix blockchain_test
* Fix compile error
* Bucket stats from all buckets
* uint256 in rlp
* uint256 rather than big.Int in Transation
* linters
* more linters
* still linters
* Reduce garbage in writeUint256
* Experiment with GC in writeByteArray
* Misc GC optimisations
* unsafe experiment with writeByteArray
* uint256 in rlp
* uint256 rather than big.Int in Transation
* linters
* more linters
* still linters
* Reduce garbage in writeUint256
* Experiment with GC in writeByteArray
* db based version of PrefixByCumulativeWitnessSize
* db based version of PrefixByCumulativeWitnessSize
* retain all in Trie by default
* fix WitnessLen logic in calcTrie roots
* Rename IntermediateTrieWitnessLenBucket to IntermediateWitnessLenBucket
* handle corner cases in WL
* Use correct incarnation for IH bucket
* use name WitnessSize
* save progress towards db-only witness estimation
* results from trie and from db are still different
* less recursion
* correct incarnation in CumulativeSearch
* reuse results from previous Tick, separate concepts of parent and startKey
* experiment: if not including trie structure to WitnessSize will reduce cumulative error
* tool to generate all IH and tool to calculate assessment of cumulative error
* tool to generate all IH
* Calculate totalWitnessSize based on DB data - then schedule will not overrun state during MGR cycle
* better stats
* Calculate totalWitnessSize based on DB data - then schedule will not overrun state during MGR cycle
* Calculate totalWitnessSize based on DB data - then schedule will not overrun state during MGR cycle
* calculate ticks size distribution
* estimate cumulative error
* fix linter
* resetIH from scratch if needed
* cleanup
* fix test
* fix test
* Not hash, keep the files
* Calculate savings
* Fix
* Fix
* Fix
* Fix
* RestAPI to support local boltdb
* Not error on read-only db
* Changes so far
* Continue
* More
* Roll back a bit
* Restore newline
* something compiles
* Fix restapi
* Fix block number
* Fix reads
* Use plain writer
* Maps for storage reads and writes
* Clean up coersions
* Fix accounts/abi/bind
* Fix tests
* More fixes
* more fixes
* More fixes
* Fixes
* Fixed core/state
* Fixed eth tests
* Move code, fix linter
* Fix test
* Fix linter
* Fix linter
* Fix linter, badger_db to support AbstractKV
* Increase IdealBatchSize for badger
* Fix linter
* Fix linter
* Produce less garbage in GetState
* Still playing with mem allocation in GetCommittedState
* Pass key by pointer in GetState as well
* linter
* Avoid a memory allocation in opSload
* Use uint256.Int rather than common.Hash for storage values to reduce memory allocation in opSload & opSstore
* linter
* linters
* small clean up
* "Unwind" for the execution phase when plain state is selected
* test stub (fails)
* tests (one with incarnations fails)
* test fixups
* fix tests: cleanup contract code bucket
* Produce less garbage in GetState
* Still playing with mem allocation in GetCommittedState
* Pass key by pointer in GetState as well
* linter
* Avoid a memory allocation in opSload
* save state
* add current index feature
* fix test
* remove logs
* Only execute 1000 blocks
* Reset history index
* Correct action
* Increase batch size
* Increase chunk size, print memory stats
* Fix linter
* Remove unused from
* Split into 2 staged
* Use storage history gen
* remove log
* Not to run tx_cacher in staged mode
* Not to recover during stage 2
* Not to recover during stage 2
* Remove counter
Co-authored-by: b00ris <b00ris@mail.ru>
This is a resubmit of #20668 which rewrites the problematic test
without any additional goroutines. It also documents the test better.
The purpose of this test is checking whether log events are sent
correctly when importing blocks. The test was written at a time when
blockchain events were delivered asynchronously, making the check hard
to pull off. Now that core.BlockChain delivers events synchronously
during the call to InsertChain, the test can be simplified.
Co-authored-by: BurtonQin <bobbqqin@gmail.com>
# Conflicts:
# core/blockchain_test.go
