Commit Graph

397 Commits

Author SHA1 Message Date
Alex Sharov
d2286bff1c
Naive implementation of MGR P2P SubProtocol (#403)
* use NoValues cursor where possible

* add ctx

* fix broken logs

* rebase master

* rebase master

* simplify generators

* hack to measure space distribution

* naive epoch and chunking implementation

* make stateless loop cancelable

* make stateless loop cancelable

* remove one rlp layer

* eth64 protocol support - add forkId to status message
2020-03-25 15:40:30 +00:00
Alex Sharov
b490192e67
Use KV Abstraction in RestAPI (#400)
* Introduce NoValuesCursor. From() method is useless because can be replaced by Seek().`
* implement NoValueCursor interface
* use abstract db in restapi
* cleanup .md
2020-03-24 09:12:55 +07:00
Alex Sharov
965d1101f4
Intermediate hash phase 4 (#396)
* another way to check if account has storage

* cleanup

* v0 of walk by db version

* save progress, to switch to another task. Put tombstone is still not correct.

* place tombstone only if exists something to hide

* db-based implementation

* db-based implementation

* db-based implementation

* fix prop check

* improve prop check logic

* Need custom logic to skip subtree for account and storage buckets because storage bucket has incarnation in key

* rebase to master

* remove all tombstones when account deleted

* remove all tombstones when account deleted

* added db integrity check

* don't rely on account.Root because it valid only about last incarnation

* remove all tombstones when account deleted

* dial with incarnation in MultiWalk2

* dial with incarnation in MultiWalk2

* when fixedbytes=40 resolver did compare cacheKey with storageKey without removing incarnation

* rebase to master

* rebase to master

Co-authored-by: alex.sharov <alex.sharov@lazada.com>
2020-03-20 11:30:14 +00:00
Alex Sharov
84ae2eb1f8
Intermediate Hashes UI (#395)
* implement NoValueCursor interface

* cleanup

* fix tests

* add more stats data to ui

* can't display error

* re-open DB low-level net interface when changing db

* re-open DB low-level net interface when changing db

* fix problem with displaying errors

* run ci

* improve prop check logic

* storage tombstones integrity checks UI

* storage page

* make DB configurable

Co-authored-by: alex.sharov <alex.sharov@lazada.com>
2020-03-20 10:06:14 +00:00
Alex Sharov
f9f54f012d
Tester phase 2 (#390)
* Add revive and phoenix

* store enode address to file, then read it from tester

* store enode address to file, then read it from tester

* rebase master

* fix miss-type

* dbg p2p-sub-protocol, add self-destruct test case

* re-create blockFetcher

* exit syncer loop and start new one

* rebase to master

* use core.GenerateChain

* root miss-match

* introduce reduceComplexity flag

* fix transfer to 0 account

* cleanup

* test-case for intermediate cache

* clean

* clean

* clean

* fix handler panic

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
Co-authored-by: alex.sharov <alex.sharov@lazada.com>
2020-03-15 16:10:07 +00:00
Alex Sharov
ce96cf75b2
Intermediate hash phase 3 (#377)
* #remove debug prints

* remove storage-mode="i"

* minnet re-execute hack with checkpoints

* minnet re-execute hack with checkpoints

* rollback to master setup

* mainnet re-exec hack

* rollback some changes

* v0 of "push down" functionality

* move all logic to own functions

* handle case when re-created account already has some storage

* clear path for storage

* try to rely on tree structure (but maybe need to rely on DB because can be intra-block re-creations of account)

* fix some bugs with indexes, moving to tests

* tests added

* make linter happy

* make linter happy

* simplify logic

* adjust comparison of keys with and without incarnation

* test for keyIsBefore

* test for keyIsBefore

* better nibbles alignment

* better nibbles alignment

* cleanup

* continue work on tests

* simplify test

* check tombstone existence before pushing it down.

* put tombstone only when account deleted, not created

* put tombstone only when account has storage

* make linter happy

* test for storage resolver

* make fixedbytes work without incarnation

* fix panic on short keys

* use special comparison only when working with keys from cache

* add blockNr for better tracing

* fix: incorrect tombstone check

* fix: incorrect tombstone check

* trigger ci

* hack for problem block

* more test-cases

* add test case for too long keys

* speedup cached resolver by removing bucket creation transaction

* remove parent type check in pruning, remove unused copy from mutation.put

* dump resolving info on fail

* dump resolving info on fail

* set tombstone everytime for now to check if it will help

* on unload: check parent type, not type of node

* fix wrong order of checking node type

* fix wrong order of checking node type

* rebase to new master

* make linter happy

* rebase to new master

* place tombstone only if acc has storage

* rebase master

* rebase master

* rebase master

* rebase master

Co-authored-by: alex.sharov <alex.sharov@lazada.com>
2020-03-11 10:31:49 +00:00
ledgerwatch
48d4ac202c
Revive devp2p tester for further simulation testing (#387)
* First fixes

* Fixes in the tester

* fix hash announcement

* Tester fixes

* Fix lint

* After-merge fixes

* Fix lint

* Fix lint

* Fix lint

* Fix lint

* Fix lint

* Fix lint

* Clean up fetchers in the test

* Make tests not run forever

* Fix test

* Fix tester

* Increase time a bit

* Fix lint

* Fix tests

* Removed log

* Fix test

* Remove commented out code
2020-03-04 22:25:40 +00:00
b00ris
1658fd6f4f Merge branch 'master' of github.com:ledgerwatch/turbo-geth into split_changeset_bucket_final 2020-02-28 14:57:03 +03:00
Felix Lange
7452f55c8b cmd/geth: enable DNS discovery by default (#20660)
* node: expose config in service context

* eth: integrate p2p/dnsdisc

* cmd/geth: add some DNS flags

* eth: remove DNS URLs

* cmd/utils: configure DNS names for testnets

* params: update DNS URLs

* cmd/geth: configure mainnet DNS

* cmd/utils: rename DNS flag and fix flag processing

* cmd/utils: remove debug print

* node: fix test
2020-02-27 17:21:20 +03:00
b00ris
83e65fdd81 migrations
fix build

add storage mode check

add intermitiate migration state&snapshot migrations
2020-02-26 00:25:30 +03:00
Alex Sharov
fdbba5202b
Trie: store self-destructed accounts (#355)
* squash commits

* enable storage cache

* make linter happy

* fix subtree prefix len check

* save cahnges to test master

* remove restriction on prefix len

* fix comparison of last bits

* fix wrong alignment

* remove debug prints

* commit current state

* commit current state

* avoid changing state of resolver from multiwalk

* remove debug code

* remove debug code

* remove debug code

* remove unnecessary copy

* make code more readable

* reduce rebuildHashes initial resolution

* fix test after rebase to master

* make code more readable

* improve pruner

* pruner add IntermediateCache bucket

* fix panic in Walk on short keys

* reduce allocations for storage keys decompression by increasing default buffer size

* re-run CI

* fix iterator behaviour

* rename cache to hash for unification

* re-run ci

* avoid using underlying DB

* hash all subtree nodes before unload

* fix getNode method

* need to check node type, not parent - before put to hashBucket

* return back parent type check, doesn't work without it.

* don't recalculate hash again

* move unloadFunc from trie to pruner

* rename bucket to shorter name

* rename bucket to shorter name

* clean

* rebase to master
2020-02-12 13:52:59 +00:00
ledgerwatch
767a374c2e
Tx pool use triedbstate, fixes to GetNodeData and move it to experimental (#364)
* Use TrieDbState for tx pool

* Not initialise tx pool until state is loaded

* Add preimage

* Fix account

* Print codehash

* Print correct code hash

* Print incarnatin

* Print incarnatin

* Use proper incarnation

* Print dbValue

* Actually fix

* Actually fix

* Fix verifySnapshot

* readAccount to get code hash

* Next incarnation

* Print addrHashes with 0 incarnations

* Print storage history

* Print storage history

* Print storage history

* Print storage history

* Print all storage history

* print change set keys

* print change set keys

* print change set keys

* print change set keys

* Not print codebucket info

* Fixes

* Fix for incarnation

* Fix for storage history bucket

* Try to fix the leak

* Try to fix the leak

* Try to fix the leak

* Try to fix the leak

* Try to fix the leak

* Try to fix the leak

* Try to fix the leak

* Fix embedded nodes

* Hasher

* Fix

* Test fixes

* Add experimental debug flag

* Fix tx_pool_test

* Disable GetNodeData test unless in experiment

* Fix more tests

* Fix lint and revert some changes

* Fix lint

* Fix lint
2020-02-10 17:05:32 +00:00
Alex Sharov
0dd4386a37
Intermediate trie hashes phase 1 (#334)
* add intermediate trie hashes bucket
* update docs
* use version of CompressNibbles with buf pool (less GC) but without io.ByteWriter (slow)
2020-01-31 11:11:20 +07:00
Péter Szilágyi
5a48d50469 Revert "eth: refactor creation of EthAPIBackend (#20476)" (#20536)
This reverts commit a1bc0e3cb69d6080b884728a60d2f92392b4c392.
2020-01-30 13:36:30 +02:00
Jonathan Gimeno
11fdbc9334 eth: refactor creation of EthAPIBackend (#20476) 2020-01-30 13:36:30 +02:00
Evgeny Danilenko
5b4f352acb
Restore Ethash mining (#231)
* initial

* mining

* remove debug

* debug

* restore random seed in the mining tests

* green tests

* fix blockchain tests

* fix lint

* init miner only if asked

* linters

* do not store trie as singlton

* fmt

* new trieDbState constructor
2019-12-10 16:12:21 +03:00
Martin Holst Swende
fffac1a5ed consensus/ethash, params: eip-2384: bump difficulty bomb (#20347)
* consensus/ethash, params: implement eip-2384: bump difficulty bomb

* params: EIP 2384 compat checks

* consensus, params: add Muir Glacier block number (mainnet,ropsten) + official name

* core/forkid: forkid tests for muir glacier

* params/config: address review concerns

* params, core/forkid: review nitpicks

* cmd/geth,eth,les: add override option for muir glacier

* params: nit fix
2019-12-09 12:56:07 +01:00
Igor Mandrigin
1e231a8a9a
Add storage mode flags as letters. (#229) 2019-12-06 12:19:00 +01:00
Evgeny Danienko
3cc627421f Merge branch 'master' into fix-data-races 2019-11-15 15:22:22 +03:00
Alexey Akhunov
14b45ea50b Downloader flag 2019-11-11 16:58:50 +00:00
Evgeny Danienko
4431ffebf2 goroutine leaks 2019-11-07 18:44:43 +03:00
Alexey Akhunov
fe01bccbb8 Apply Turbo-Geth modifications to go-ethereum codebase 2019-11-01 21:52:03 +01:00
Nguyen Kien Trung
52a967cfab eth: remove unused field (#20049) 2019-09-11 01:38:42 +03:00
Péter Szilágyi
e39b2a2bde
acmd, core, eth, les: support --override.istanbul 2019-08-23 12:09:27 +03:00
Péter Szilágyi
983f92368b
core/forkid: implement the forkid EIP, announce via ENR (#19738)
* eth: chain config (genesis + fork) ENR entry

* core/forkid, eth: protocol independent fork ID, update to CRC32 spec

* core/forkid, eth: make forkid a struct, next uint64, enr struct, RLP

* core/forkid: change forkhash rlp encoding from int to [4]byte

* eth: fixup eth entry a bit and update it every block

* eth: fix lint

* eth: fix crash in ethclient tests
2019-07-08 18:53:47 +03:00
Péter Szilágyi
22411919da
cmd, eth, les, param: drop --override.constantinople 2019-07-02 14:14:59 +03:00
gary rong
f7cdea2bdc all: on-chain oracle checkpoint syncing (#19543)
* all: implement simple checkpoint syncing

cmd, les, node: remove callback mechanism

cmd, node: remove callback definition

les: simplify the registrar

les: expose checkpoint rpc services in the light client

les, light: don't store untrusted receipt

cmd, contracts, les: discard stale checkpoint

cmd, contracts/registrar: loose restriction of registeration

cmd, contracts: add replay-protection

all: off-chain multi-signature contract

params: deploy checkpoint contract for rinkeby

cmd/registrar: add raw signing mode for registrar

cmd/registrar, contracts/registrar, les: fixed messages

* cmd/registrar, contracts/registrar: fix lints

* accounts/abi/bind, les: address comments

* cmd, contracts, les, light, params: minor checkpoint sync cleanups

* cmd, eth, les, light: move checkpoint config to config file

* cmd, eth, les, params: address comments

* eth, les, params: address comments

* cmd: polish up the checkpoint admin CLI

* cmd, contracts, params: deploy new version contract

* cmd/checkpoint-admin: add another flag for clef mode signing

* cmd, contracts, les: rename and regen checkpoint oracle with abigen
2019-06-28 10:34:02 +03:00
gary rong
c0a034ec89 eth, les: reject stale request (#19689)
* eth, les: reject stale request

* les: reuse local head number
2019-06-11 10:40:32 +03:00
gary rong
4e0c1a1a6b eth, les: reject light client connection is server is not synced (#19616)
* eth, les: reject light client connection is server is not synced

* eth, les: rename function and variables

* les: format
2019-05-26 19:15:05 +03:00
Péter Szilágyi
006c21efc7
cmd, core, eth, les, node: chain freezer on top of db rework 2019-05-16 10:39:29 +03:00
Péter Szilágyi
9effd64290
core, eth, trie: bloom filter for trie node dedup during fast sync (#19489)
* core, eth, trie: bloom filter for trie node dedup during fast sync

* eth/downloader, trie: address review comments

* core, ethdb, trie: restart fast-sync bloom construction now and again

* eth/downloader: initialize fast sync bloom on startup

* eth: reenable eth/62 until we properly remove it
2019-05-13 15:28:01 +03:00
gary rong
6269e5574c miner: polish miner configuration (#19480)
* cmd, eth, miner: disable advance sealing if user require

* cmd, console, miner, les, eth: wrap the miner config

* eth: remove todo

* cmd, miner: revert noadvance flag

The reason for this is: if the transaction execution is even longer
than block time, then this kind of transactions is DoS attack.
2019-04-23 10:08:51 +03:00
gary rong
d5cae48bae accounts, cmd, internal: disable unlock account on open HTTP (#17037)
* cmd, accounts, internal, node, rpc, signer: insecure unlock protect

* all: strict unlock API by rpc

* cmd/geth: check before printing warning log

* accounts, cmd/geth, internal: tiny polishes
2019-04-04 14:03:10 +03:00
Péter Szilágyi
ed34a5e08a
cmd, core, eth: support disabling the concurrent state prefetcher 2019-04-01 11:52:11 +03:00
Péter Szilágyi
ac3e7c9b3d
eth: remove redundant chain config fields 2019-03-27 13:23:08 +02:00
Péter Szilágyi
054412e335
all: clean up and proerly abstract database access 2019-03-06 13:35:03 +02:00
Felföldi Zsolt
c2003ed63b les, les/flowcontrol: improved request serving and flow control (#18230)
This change

- implements concurrent LES request serving even for a single peer.
- replaces the request cost estimation method with a cost table based on
  benchmarks which gives much more consistent results. Until now the
  allowed number of light peers was just a guess which probably contributed
  a lot to the fluctuating quality of available service. Everything related
  to request cost is implemented in a single object, the 'cost tracker'. It
  uses a fixed cost table with a global 'correction factor'. Benchmark code
  is included and can be run at any time to adapt costs to low-level
  implementation changes.
- reimplements flowcontrol.ClientManager in a cleaner and more efficient
  way, with added capabilities: There is now control over bandwidth, which
  allows using the flow control parameters for client prioritization.
  Target utilization over 100 percent is now supported to model concurrent
  request processing. Total serving bandwidth is reduced during block
  processing to prevent database contention.
- implements an RPC API for the LES servers allowing server operators to
  assign priority bandwidth to certain clients and change prioritized
  status even while the client is connected. The new API is meant for
  cases where server operators charge for LES using an off-protocol mechanism.
- adds a unit test for the new client manager.
- adds an end-to-end test using the network simulator that tests bandwidth
  control functions through the new API.
2019-02-26 12:32:48 +01:00
gary rong
7fd0ccaa68 core: remove unnecessary fields in logs, receipts and tx lookups (#17106)
* core: remove unnecessary fields in log

* core: bump blockchain database version

* core, les: remove unnecessary fields in txlookup

* eth: print db version explicitly

* core/rawdb: drop txlookup entry struct wrapper
2019-02-21 15:14:35 +02:00
Péter Szilágyi
d6225ab846
cmd/utils, eth: relinquish GC cache to read cache in archive mode 2019-02-07 14:36:25 +02:00
Martin Holst Swende
43e8efe895
accounts, eth, clique, signer: support for external signer API (#18079)
* accounts, eth, clique: implement external backend + move sighash calc to backend

* signer: implement account_Version on external API

* accounts/external: enable ipc, add copyright

* accounts, internal, signer: formatting

* node: go fmt

* flags: disallow --dev in combo with --externalsigner

* accounts: remove clique-specific signing method, replace with more generic

* accounts, consensus: formatting + fix error in tests

* signer/core: remove (test-) import cycle

* clique: remove unused import

* accounts: remove CliqueHash and avoid dependency on package crypto

* consensus/clique: unduplicate header encoding
2019-02-05 11:23:57 +01:00
gary rong
d5cad488be core, eth: fix database version (#18429)
* core, eth: fix database version

* eth: polish error message
2019-01-11 13:49:12 +02:00
Martin Holst Swende
83a9a73b89 cmd/geth, core, eth: implement Constantinople override flag (#18273)
* geth/core/eth: implement constantinople override flag

* les: implemnent constantinople override flag for les clients

* cmd/geth, eth, les: fix typo, move flag to experimentals
2018-12-11 14:19:03 +02:00
Ryan Schneider
48b70ecff1
cmd, eth: Add support for --whitelist <blocknum>=<hash>,... flag
* Rejects peers that respond with a different hash for any of the passed in block numbers.
* Meant for emergency situations when the network forks unexpectedly.
2018-12-10 14:30:06 +02:00
Péter Szilágyi
434dd5bc00
cmd, core, eth, light, trie: add trie read caching layer 2018-11-15 12:22:13 +02:00
gary rong
457e930f27 eth, miner: prefer locally generated uncles vs remote ones (#17715)
* core, eth: fix dependency cycle

* eth, miner: perfer to locally generated uncle
2018-09-21 00:11:55 +03:00
gary rong
ba0a8b7887 core, eth: fix dependency cycle (#17720) 2018-09-20 20:02:15 +03:00
gary rong
d6254f827b all: protect self-mined block during reorg (#17656) 2018-09-20 15:09:30 +03:00
Guillaume Ballet
da29332c5f core/vm: add switches to select evm+ewasm interpreters (#17687)
Interpreter initialization is left to the PRs implementing them.
Options for external interpreters are passed after a colon in the
`--vm.ewasm` and `--vm.evm` switches.
2018-09-20 10:44:35 +03:00
Péter Szilágyi
e8f229b82e
cmd, core, eth, miner, params: configurable gas floor and ceil 2018-08-29 12:40:12 +03:00
gary rong
c1c003e4ff consensus, miner: stale block mining support (#17506)
* consensus, miner: stale block supporting

* consensus, miner: refactor seal signature

* cmd, consensus, eth: add miner noverify flag

* cmd, consensus, miner: polish
2018-08-28 16:59:05 +03:00