Commit Graph

1960 Commits

Author SHA1 Message Date
Alex Sharov
64de47642e
Problem: forkFilter variable are catched by goroutine when peer connected (not when status msg received) - it means to change it tester must reconnect. (#429) 2020-04-06 08:53:52 +01:00
ledgerwatch
e5fe539216
Computation of index size + binary search in the indices (#428)
* Calculate index sizes

* Better counting

* Try binary search

* More compact code

* Handle the case of empty index
2020-04-05 18:27:25 +01:00
Alex Sharov
66ce27174d
don't rebuild - because it evicting IH (#427) 2020-04-05 07:33:06 +01:00
Alex Sharov
aa6bab40e8
remove tombstones (#426) 2020-04-05 07:32:28 +01:00
ledgerwatch
19cd0370d3
Fixes in the stateless prototype after error on the block 2675337 (state clearing) (#425)
* Stop

* return

* Bugfixes

* Cleanup
2020-04-04 08:18:22 +01:00
Alex Sharov
9324d83cb2
ethdb.KV interface - pase 2 (#420)
* - handle cursor.Prefix on server
- move state reports to KV interface

* add CmdCursorSeekKey

* tests for abstract_kv

* avoid reading configs of databases

* avoid reading configs of databases

* make linter happy

* make linter happy

* cleanup

* port badger features from original implementation

* try to fix test

* try to fix test

* .Close() don't return error anymore - defer friendly

* try to enable badger now

* try to enable badger now

* badger can't run on CI yet

* badger can't run on CI yet

* re-run ci

* skip ctx cancelation for badger
2020-04-04 08:18:10 +01:00
ledgerwatch
9bb7f3056d
Fix hanging resolver (#424)
* Debugging the cached resolver

* make logic of comparison more explicit

* fix stuck on block 696817

* Remove tracing

* Fix CI

* Search for root hash

Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
2020-04-01 17:04:41 +01:00
Giulio rebuffo
c5e503634a
tx lookup File Write (hack.go) (#421)
* tx lookups via file insertions

* cleanup

* added heap + buffer to genTx2

* Validation Process

* fix

* removed profile

* remove batch.Commit() in validation

* separated validation
2020-04-01 17:03:51 +01:00
Alex Sharov
508d497644
commit simulator_genesis.json file generated by go run ./cmd/tester genesis > simulator_genesis.json (#419) 2020-03-31 07:52:13 +01:00
Alex Sharov
20df391732
don't pass common ctx to commit block, to avoid cancelation in the middle of commit (#418) 2020-03-31 09:08:25 +07:00
Alex Sharov
c8cbdb8c47
improve changeSetChecks to accept genesis.json (#415) 2020-03-30 19:56:53 +01:00
Alex Sharov
286ef9ea1f
enable intermediate trie hashes by default (#414) 2020-03-30 17:01:24 +01:00
Alex Sharov
c2e71ffc03
[wip] [Tester] - different values in fork and non-fork (#413)
* save progress

* increment in fork only

* skip some steps in fork

* tester switch database to ./simulator file

* correct signer
2020-03-30 13:31:49 +01:00
Alex Sharov
a364cfded6
[Tester] fix fork generator (#410)
fix coinbase
use same genBlock function
switch inside genBlock - now works on blockNumber instead of nonce
fill file in own function
add verbosity cli flag
2020-03-27 20:12:26 +07:00
Andrew Ashikhmin
82b80c2ebe
Thin history: Fix an error in mutation.getNoLock (#409)
* CheckChangeSets and thin history

* small code de-duplication

* small code clean-up

* Fix an error in mutation.getNoLock

* CheckChangeSets: truly make historyfile = chaindata by default
2020-03-26 21:52:22 +00:00
Alex Sharov
8d8e41cd85
Call bufio.Flash() in tester (#407)
* call bufio.Flush

* rerun ci

* handle errors better

* correct sleep
2020-03-26 21:52:05 +00:00
Igor Mandrigin
5ff3948129
Remove error param from NewTrieDbState (#408) 2020-03-26 16:21:15 +03:00
ledgerwatch
6dcb8ade46
Convenience changes to checkChangeSets and stateless (#406)
* Allow separation of files

* Try next block

* Enable history

* use logging

* No check option

* Option to write history
2020-03-25 20:18:46 +00:00
Alex Sharov
d2286bff1c
Naive implementation of MGR P2P SubProtocol (#403)
* use NoValues cursor where possible

* add ctx

* fix broken logs

* rebase master

* rebase master

* simplify generators

* hack to measure space distribution

* naive epoch and chunking implementation

* make stateless loop cancelable

* make stateless loop cancelable

* remove one rlp layer

* eth64 protocol support - add forkId to status message
2020-03-25 15:40:30 +00:00
Alex Sharov
b490192e67
Use KV Abstraction in RestAPI (#400)
* Introduce NoValuesCursor. From() method is useless because can be replaced by Seek().`
* implement NoValueCursor interface
* use abstract db in restapi
* cleanup .md
2020-03-24 09:12:55 +07:00
Alex Sharov
1fb8749638
Make AccountCreation do determined amount of operations and Move .Put out of .View (#402)
* move .Put out of .View and remove deletion of all tombstones when delete acc

* move .Put out of .View and remove deletion of all tombstones when delete acc
2020-03-23 22:14:05 +00:00
Igor Mandrigin
4f4b395aa4
Introduce code node & get rid of code map (#398)
* introduce code node

* replace codeMap with code touches

* fix a comment

* fixups to tests

* fix compile error

* fix getnodedata tests

* add tests and test stubs

* add more test stubs

* add test method bodies

* add and fix more tests on trie for new codenode

* add test change code between blocks

* fix crash in stateless

* remove unneded files

* remove comment

* fix deleted account code

* fix resolve set builder for code nodes
2020-03-23 22:10:36 +00:00
Alex Sharov
965d1101f4
Intermediate hash phase 4 (#396)
* another way to check if account has storage

* cleanup

* v0 of walk by db version

* save progress, to switch to another task. Put tombstone is still not correct.

* place tombstone only if exists something to hide

* db-based implementation

* db-based implementation

* db-based implementation

* fix prop check

* improve prop check logic

* Need custom logic to skip subtree for account and storage buckets because storage bucket has incarnation in key

* rebase to master

* remove all tombstones when account deleted

* remove all tombstones when account deleted

* added db integrity check

* don't rely on account.Root because it valid only about last incarnation

* remove all tombstones when account deleted

* dial with incarnation in MultiWalk2

* dial with incarnation in MultiWalk2

* when fixedbytes=40 resolver did compare cacheKey with storageKey without removing incarnation

* rebase to master

* rebase to master

Co-authored-by: alex.sharov <alex.sharov@lazada.com>
2020-03-20 11:30:14 +00:00
Andrew Ashikhmin
abadbdfb80
[Issue 340] Preserve the original when a contract is self-destructed and then its address is touched in the same block (e.g. # 156634) (#397)
* root was unused in BlockChain.StateAt

* TestDoubleAccountRemoval

* Preserve the original when a contract is self-destructed and then its address is touched in the same block (e.g. #
156634)
2020-03-20 10:08:13 +00:00
Alex Sharov
84ae2eb1f8
Intermediate Hashes UI (#395)
* implement NoValueCursor interface

* cleanup

* fix tests

* add more stats data to ui

* can't display error

* re-open DB low-level net interface when changing db

* re-open DB low-level net interface when changing db

* fix problem with displaying errors

* run ci

* improve prop check logic

* storage tombstones integrity checks UI

* storage page

* make DB configurable

Co-authored-by: alex.sharov <alex.sharov@lazada.com>
2020-03-20 10:06:14 +00:00
Alex Sharov
5f8b1b75d5
Storage Tombstones UI (#394)
* implement NoValueCursor interface

* cleanup

* fix tests

Co-authored-by: alex.sharov <alex.sharov@lazada.com>
2020-03-16 22:00:48 +00:00
winsvega
36b2e89ce8 geth retesteth: increase retesteth default http timeouts (#20767) 2020-03-16 16:40:19 +03:00
Chris Chinchilla
b7415664d4 docs: correct clef typo (#20705) 2020-03-16 16:40:19 +03:00
ledgerwatch
2a04d5bfef
Cleanup and fix the bug in the block 258217 (#392)
* Cleanup in blockchain.go

* Fix

* Fix

* Update hack

* Test fix
2020-03-15 17:15:16 +00:00
Alex Sharov
f9f54f012d
Tester phase 2 (#390)
* Add revive and phoenix

* store enode address to file, then read it from tester

* store enode address to file, then read it from tester

* rebase master

* fix miss-type

* dbg p2p-sub-protocol, add self-destruct test case

* re-create blockFetcher

* exit syncer loop and start new one

* rebase to master

* use core.GenerateChain

* root miss-match

* introduce reduceComplexity flag

* fix transfer to 0 account

* cleanup

* test-case for intermediate cache

* clean

* clean

* clean

* fix handler panic

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
Co-authored-by: alex.sharov <alex.sharov@lazada.com>
2020-03-15 16:10:07 +00:00
Andrew Ashikhmin
aeed1657c7
Issue #340: Re-execute all historical transaction in read-only mode and check ChangeSets (#388)
* Clean up code duplication between IntraBlockState's FinalizeTx & CommitBlock

* checkChangeSets command

* linter

* First attempt at checking account changes

* Reuse runBlock in CheckChangeSets

* linter

* linter

* Optionally include no-changes in the ChangeSets

* linter

* Detect storage changes for account change sets

* Fix post-merge compilation errors

* Use database format compatible with !debug.IsThinHistory()

* PrintChangedAccounts in ChangeSetWriter

* Avoid out-of-bounds access

* Storage changes

* hack FirstContractIncarnation

* Call ChangeSetWriter only once per block
2020-03-11 16:54:09 +01:00
Alex Sharov
c5ffc971c5
[WIP] Badger v2 (#378)
* badger v2 investigation

* buf pool - use native New method and avoid double checks

* db.Open prototype

* db.Tx/Bucket/Cursor prototypes

* Chained config

* Item concept added

* save changes to test on master

* make hack resumable

* Design document v0

* Cursor concept

* less brackets syntax of cursor builder

* benchmarks

* cleanup fs

* test for context cancelations

* test for context cancelations

* test for cursor.Prefix option

* add ForEachKey method

* add ForEachKey method

* add naming explanation

* experiment of non-pointers cursor/bucket

* .Bucket() and .Cursor() doesn't returns error

* .Bucket() and .Cursor() doesn't returns error

* .Bucket() and .Cursor() doesn't returns error

* remove CursorOpts concept

* more test-cases

* simplify open api

* Tx, Bucket, Cursor - now are interfaces

* Tx, Bucket, Cursor - now are interfaces

* switch to interfaces

* rebase master

Co-authored-by: alex.sharov <alex.sharov@lazada.com>
2020-03-11 11:02:37 +00:00
Alex Sharov
ce96cf75b2
Intermediate hash phase 3 (#377)
* #remove debug prints

* remove storage-mode="i"

* minnet re-execute hack with checkpoints

* minnet re-execute hack with checkpoints

* rollback to master setup

* mainnet re-exec hack

* rollback some changes

* v0 of "push down" functionality

* move all logic to own functions

* handle case when re-created account already has some storage

* clear path for storage

* try to rely on tree structure (but maybe need to rely on DB because can be intra-block re-creations of account)

* fix some bugs with indexes, moving to tests

* tests added

* make linter happy

* make linter happy

* simplify logic

* adjust comparison of keys with and without incarnation

* test for keyIsBefore

* test for keyIsBefore

* better nibbles alignment

* better nibbles alignment

* cleanup

* continue work on tests

* simplify test

* check tombstone existence before pushing it down.

* put tombstone only when account deleted, not created

* put tombstone only when account has storage

* make linter happy

* test for storage resolver

* make fixedbytes work without incarnation

* fix panic on short keys

* use special comparison only when working with keys from cache

* add blockNr for better tracing

* fix: incorrect tombstone check

* fix: incorrect tombstone check

* trigger ci

* hack for problem block

* more test-cases

* add test case for too long keys

* speedup cached resolver by removing bucket creation transaction

* remove parent type check in pruning, remove unused copy from mutation.put

* dump resolving info on fail

* dump resolving info on fail

* set tombstone everytime for now to check if it will help

* on unload: check parent type, not type of node

* fix wrong order of checking node type

* fix wrong order of checking node type

* rebase to new master

* make linter happy

* rebase to new master

* place tombstone only if acc has storage

* rebase master

* rebase master

* rebase master

* rebase master

Co-authored-by: alex.sharov <alex.sharov@lazada.com>
2020-03-11 10:31:49 +00:00
ledgerwatch
f210116e65
Semantics: Integrate Z3 into the build (#370)
* Just files

* Fix lint

* First attempt at linking

* More semantics

* Add more arguments

* Added z3 dependency

* Added integration with z3

* Try to fix build

* Add m library

* Try to fix ints

* Separate init/destroy, create sorts
2020-03-06 08:54:21 +00:00
ledgerwatch
48d4ac202c
Revive devp2p tester for further simulation testing (#387)
* First fixes

* Fixes in the tester

* fix hash announcement

* Tester fixes

* Fix lint

* After-merge fixes

* Fix lint

* Fix lint

* Fix lint

* Fix lint

* Fix lint

* Fix lint

* Clean up fetchers in the test

* Make tests not run forever

* Fix test

* Fix tester

* Increase time a bit

* Fix lint

* Fix tests

* Removed log

* Fix test

* Remove commented out code
2020-03-04 22:25:40 +00:00
ledgerwatch
80f06b279a
Fix --download-only mode (#374)
* Fix download only

* Fix lint

* Reset references

* Only reset on error

* Potential fixes

* no NPE

* no NPE

* Not use multi-put

* Reduce ideal batch size for download only

* Handle tds == nil

* remove nested mutation

* Return multiput

* Better reporting

* Reduce batch size for download only

* Avoid extra copying

* Avoid extra copying

* IdealBatchSize

* Not write tx lookup entries

* Larger batches

* Go back to normal batch size

* Fix lint

* Gen tx lookup

* print progress

* Add filling up the lookup array

* Show tx count

* Introduce second round

* Add generating tx lookup

* Fix lint

* properly stop at specified block

* measure the duration of the last phase

* not to fail if the bucket is not found

* Fix lint

* Alternative tx generation

* Fix out of memory

* Fix out of memory

* Split in parts to conserve memory

* Copy keys

* Fix lint

* Fix lint
2020-03-01 09:00:14 +00:00
b00ris
1658fd6f4f Merge branch 'master' of github.com:ledgerwatch/turbo-geth into split_changeset_bucket_final 2020-02-28 14:57:03 +03:00
Igor Mandrigin
edcd16c7ab post-merge fixups 2020-02-27 17:22:25 +03:00
Felix Lange
e4e1447e15 cmd/ethkey: speed up test by using weaker scrypt parameters (#20680) 2020-02-27 17:21:20 +03:00
Felix Lange
7452f55c8b cmd/geth: enable DNS discovery by default (#20660)
* node: expose config in service context

* eth: integrate p2p/dnsdisc

* cmd/geth: add some DNS flags

* eth: remove DNS URLs

* cmd/utils: configure DNS names for testnets

* params: update DNS URLs

* cmd/geth: configure mainnet DNS

* cmd/utils: rename DNS flag and fix flag processing

* cmd/utils: remove debug print

* node: fix test
2020-02-27 17:21:20 +03:00
winsvega
86f56c9079 cmd/geth retesteth: add eth_getBlockByHash (#20621) 2020-02-27 17:21:20 +03:00
Felix Lange
85941edbea cmd/devp2p: fix Route53 TXT record splitting (#20626)
For longer records and subtree entries, the deployer created two
separate TXT records. This doesn't work as intended because the client
will receive the two records in arbitrary order. The fix is to encode
longer values as "string1""string2" instead of "string1", "string2".
This encoding creates a single record on AWS Route53.
2020-02-27 17:21:19 +03:00
meowsbits
9ebb2a4d2f cmd/geth: add 'dumpgenesis' command (#20191)
Adds the 'geth dumpgenesis' command, which writes the configured
genesis in JSON format to stdout. This provides a way to generate the
data (structure and content) that can then be used with the 'geth init'
command.
2020-02-27 17:20:36 +03:00
Martin Holst Swende
f3630b8949 retesteth: clean txpool on rewind, default dao support (#20596) 2020-02-27 17:20:36 +03:00
Felix Lange
cf03897b25 rpc: check module availability at startup (#20597)
Fixes #20467

Co-authored-by: meowsbits <45600330+meowsbits@users.noreply.github.com>
2020-02-27 17:20:36 +03:00
Guillaume Ballet
80fcbd3040 console, internal/jsre: use github.com/dop251/goja (#20470)
This replaces the JavaScript interpreter used by the console with goja,
which is actively maintained and a lot faster than otto. Clef still uses otto
and eth/tracers still uses duktape, so we are currently dependent on three
different JS interpreters. We're looking to replace the remaining uses of otto
soon though.
2020-02-27 17:20:36 +03:00
Guillaume Ballet
91a19add0b cmd/evm: accept --input for disasm command (#20548) 2020-02-27 17:20:36 +03:00
Martin Holst Swende
2f469b2c81 cmd/geth/retesteth: use canon head instead of keeping alternate count (#20572) 2020-02-27 17:20:36 +03:00
Martin Holst Swende
906f989762 cmd: implement abidump (#19958)
* abidump: implement abi dump command

* cmd/abidump: add license
2020-02-27 17:20:36 +03:00
Felix Lange
5f7f638ac5 log, internal/debug: delete RotatingFileHandler (#20586)
* log: delete RotatingFileHandler

We added this for the dashboard, which is gone now. The
handler never really worked well and had data race and file
handling issues.

* internal/debug: remove unused RotatingFileHandler setup code
2020-02-27 17:20:36 +03:00