Commit Graph

42 Commits

Author SHA1 Message Date
Alex Sharov
08590211e6
Add defer collector.Close() (#2351)
* add defer collector.Close

* add defer collector.Close
2021-07-13 09:17:21 +07:00
Alex Sharov
a78cb54132
step towards kv interface (#2311) 2021-07-07 23:15:49 +07:00
b00ris
57473175ff
Body snapshot (#2100)
* fix test

* get rid of ObjectDatabase

* sn_builder_prototype2

* save state

* save state

* integration step1

* fix lint

* fix

* fix test

* integrate migrator.finish

* fix lint

* fix build

* fix typo

* save state

* body snapshot test

* unique tx

* body snapshot generation using walk

* move methods out of test

* block data verification added

* fix lint

* test with remove works correctly

* fix lint

* remove experiment test

* fix test

* add comment

* add second layer of remove test

* rename test

* fix typos

* fix lint

* revert testdata

* body snapshot migration save state

* fix body snapshot migration

* fix after merge

* remove debug test

* debug windows build

* fix build

* fix

* fix lint

* debug

* fix

* fix windows build

* simplify snapshot management&&get rid of lazy tx

* fix lint

* fix windows path

* debug

* debug

* debug

* debug

* remove geometry experiments

* skip windows tests

* clean

* fix

* fix ;int
2021-07-06 23:33:26 +01:00
Alex Sharov
ba902f7ef3
separate_config_from_node_and_eth (#2289) 2021-07-04 08:50:32 +01:00
ledgerwatch
3f74cbf3a7
Update snapshot_builder_test.go (#2290) 2021-07-04 08:49:18 +01:00
Alex Sharov
59d05dc5fe
hide file exists err (#2218) 2021-06-22 11:09:45 +01:00
Alex Sharov
a535e8f30c
switch to mdbx-go module, and db-tools to libmdbx submodule (#2199)
* move mdbx to submodule

* move mdbx to submodule

* move mdbx to submodule

* move mdbx to submodule

* move mdbx to submodule

* move mdbx to submodule

* move mdbx to submodule

* move mdbx to submodule

* move mdbx to submodule

* move mdbx to submodule

* move mdbx to submodule

* move mdbx to submodule

* move mdbx to submodule
2021-06-19 21:30:12 +01:00
Alex Sharov
bc271f92ce
To break dependency to db implementation - move all db classes to "ethdb/kv" and leave in "ethdb" only interfaces (#2195) 2021-06-19 15:21:53 +07:00
Alex Sharov
e9f510c2b7
erigon as lib (#2186) 2021-06-18 10:35:11 +07:00
Artem Vorotnikov
af836a6200
Nuke LMDB (#2167) 2021-06-16 13:57:58 +03:00
BitBaseBit
7ed337cdcc
Implemented panic handling, graceful shutdown and reporting for all goroutines that don't explicitly handle them. (#2153)
* implemented crash reporting for all goroutine panics that aren't handled explicitly

* implemented crash reporting for all goroutine panics that aren't handled explicitly

* changed node defaults back to originals after testing

* implemented panic handling for all goroutines that don't explicitly handle them, outputting the stack trace to a file in crashreports

* handling panics on all goroutines gracefully

* updated missing call

* error assignment

* implemented suggestions

* path.Join added

* implemented Evgeny's suggestions

* changed path.Join to filepath.Join for cross-platform

* added err check

* updated RecoverStackTrace to LogPanic

* updated closures

* removed call of common.Go to some goroutines

* updated scope capture

* removed testing files

* reverted back to original method, I feel like its less intrusive

* update filename for clarity
2021-06-13 17:41:39 +01:00
Alex Sharov
5ba3ea162a
Simulated backend and genesis to kv (#2104)
* move to kv

* move to kv

* move to kv

* move to kv

* move to kv

* move to kv

* merge

* merge

* merge

* merge

* merge

* merge

* merge

* merge
2021-06-05 16:17:04 +01:00
b00ris
66c7c669fb
Snapshot builder simplification (#2064)
* fix test

* get rid of ObjectDatabase

* sn_builder_prototype2

* save state

* save state

* integration step1

* fix lint

* fix

* fix test

* integrate migrator.finish

* fix lint

* fix build

* fix typo

* fix
2021-06-04 13:30:02 +01:00
Alex Sharov
e34d73f162
Dont exec all blocks in 1 tx (#2076)
* save

* save

* save

* save
2021-06-02 08:15:17 +01:00
b00ris
49f1937e61
Fix flacky test (#2008)
* fix test

* get rid of ObjectDatabase
2021-05-25 09:25:16 +01:00
Alex Sharov
0be3044b7e
rename (#1978)
* rename

* rename "make grpc"

* rename "abi bindings templates"

* rename "abi bindings templates"
2021-05-20 19:25:53 +01:00
Alex Sharov
215623da80
Windows support (#1944) 2021-05-20 23:46:12 +07:00
Alex Sharov
9d66aa0fe3
better support for mdbx in snapshots (#1971) 2021-05-20 18:12:07 +07:00
Alex Sharov
cceaf75b16
use t.TempDir() (#1955) 2021-05-18 19:13:16 +07:00
Alex Sharov
617baac56b
move db tests to kv interface (#1946) 2021-05-17 16:54:29 +07:00
ledgerwatch
f6ac04e53e
Skip flaky tests (#1943)
* Update send_transaction_test.go

* Update snapshot_builder_test.go
2021-05-16 19:09:12 +01:00
Alex Sharov
2ff6ce7d92
move staged sync to tx (#1897) 2021-05-14 18:00:20 +07:00
b00ris
48733c31cd
test fix (#1930) 2021-05-13 21:04:47 +01:00
ledgerwatch
128f446e7c
Download v2 fixes (#1903)
* [download.v2] Fix endianness of Difficulty in Status message

* Add comments

* Finish stage to use RwTx

* Print header age in stage 1

* Print new block messages

* TopSeenHeight to be updated from any header

* block in the future handling and skip the test

* Fix downloader test

* Fix test

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2021-05-09 08:22:08 +01:00
b00ris
565a4250d6
Snapshot sync headers stage (#1836)
* save state

* snapshot update works

* save state

* snapshot migrator

* tx test

* save state

* migrations stages refactor

* refactor snapshot migrator

* compilation fixed

* integrate snapshot migrator

* goerli sync headers

* debug async snapshotter on goerly

* move verify headers, remove experiments, fix remove old snapshot

* save state

* refactor snapshotsync injection

* fix deadlock

* replace snapshot generation stage logic to migrate method

* change done for body snapshot

* clean

* clean&&change deleted value

* clean

* fix hash len

* fix hash len

* remove one of wrap methods, add remove snapshots on start

* add err check

* fix shadowing

* stages unwind order debug

* matryoshka experiments

* steam test

* fix build

* fix test

* fix lint

* fix test

* fix test datarace

* add get test

* return timeout

* fix mdbx overlap

* fix after merge

* change epoch size

* clean todo

* fix

* return testdata

* added return from sndownloader gorutine

* fix review comments

* Fix

* More info

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2021-05-08 09:45:40 +01:00
Evgeny Danilenko
984daa4d7a
Clique snapshot timer fixed (#1789)
* fix clique snapshot storage by timer

* after merge

* remove debug

* remove debug

* debug logging

* correct t.Fatal
2021-04-23 20:47:06 +03:00
Alex Sharov
22b9522112
RPCDaemon: add --datadir flag, use --chaindata only if wanna redefine default chaindata folder (#1747) 2021-04-19 12:44:14 +07:00
Alex Sharov
35872a4708
[merge after release] remove etl onLoadCommit hook, because etl doesn't manage transactions anymore (it's low-level code) (#1692) 2021-04-08 18:03:33 +07:00
Artem Vorotnikov
b71c6323a0
KVGetter, replace old geth interfaces with it (#1676) 2021-04-05 20:04:58 +07:00
Alex Sharov
fbfc43da7e
etl to use rwtx (#1660) 2021-04-02 18:22:25 +07:00
Alex Sharov
ef77eb3053
tx.RwCursor() to return err (first step of removing lazy cursors) (#1656) 2021-04-02 13:36:49 +07:00
Artem Vorotnikov
d4c10c9a47
Port rpcdaemon to KV interface (#1627) 2021-03-30 12:53:54 +03:00
Artem Vorotnikov
6a6f5469a1
KV: Split RO and RW transactions (#1577) 2021-03-21 20:15:25 +07:00
b00ris
c03da8fb8e
Split header prefix bucket to headers, td and canonical (#1556)
* split headers prefix

* migration

* fix downloader bug

* test for migration

* fix lint

* uncomment t.Prallel

* fix postprocessing test
2021-03-19 12:54:47 +00:00
Alex Sharov
7fd1f19435
txdb lazy cursors creation, grpc version up (#1440)
* txdb lazy, grpc version up

* tidy

* fix lint
2021-01-15 09:38:09 +00:00
ledgerwatch
baafc90bc6
Fixup after merge (#1432)
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2021-01-02 20:22:35 +00:00
b00ris
8db5790838
State snapshot sync (#1417)
* move experiments to new branch&reorganise kv_snapshot

* walk&modify tests

* added delete from snapshot tests

* fmt

* state snapshot debug

* snapshot validation passed. copy state snapshot

* debug

* snapshot cursor.Prev test

* Prev works correct. Added Current check

* add err check

* added walk forward and backward test

* before refactoring

* refactoring

* execution with snapshot debug

* fix

* remove useless test

* before dupcursor implimentation

* tests with prev and delete works

* execution based on state snapshot passed

* remove useless tests

* blocks to 1140000 passed

* clean verifier

* cleanup state generation

* clean verify && seeder

* remove debug code

* tests passed

* fix lint

* save state

* test passed

* fix lint

* add state hash

* fix lint
2021-01-02 19:28:37 +00:00
Alex Sharov
d7d9ce0a8e
fix go vet (#1425) 2020-12-26 10:18:16 +07:00
Artem Vorotnikov
8c0ecaeea9
Remove StageData (#1419) 2020-12-21 11:01:37 +00:00
Alex Sharov
4565bd43c1
Mdbx v0.9.2 (#1373)
* up devel

* up devel

* up devel

* up devel

* up devel

* up devel

* up devel

* up devel

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* switch to master

* mdbx 0.9.2

* mdbx 0.9.2

* %<<$!%%%%%%<o'{}k.###

* %<<$!%%%%%%<o'{}k.###

* merge

* merge

* merge

* merge

* ad

* ad

* ad

* ad
2020-11-28 14:26:28 +00:00
Igor Mandrigin
393c9965ae
rpcdaemon: subscriptions, newHeads (#1359)
* fix `make grpc` on new checkouts

* update proto files

* add some stub

* prototype with fake events

* notifying about events

* pass events

* events are being sent

* transfer headers to filters

* create the “filters” struct

* implement new heads

* PoC of New Heads subscription

* fix keep alive

* fixups for the client

* add “type” to the event

* support header event type on client

* better stage refactor

* fixup for the eth backend

* fixups

* fix tests

* fix tests

* fix linters

* address comments

* remove unused log
2020-11-17 19:13:41 +00:00
b00ris
6464da7670
Remote snapshot downloader (#1343)
* save state

* save state

* save state

* refactoring

* fix

* save state

* save state

* fmt

* fix lint

* restore torrents for external downloader

* fix lint

* download

* skip debug test

* debug

* remote debug

* small cli fixes

* skip debug test

* external snapshot predownloader

* get rid of remote downloader

* fix lint

* clean makefile

* fix lint

* fix lint

* cleanup

* fix ci

* fmt

* remove proto from interfaces

* Squashed 'interfaces/' content from commit acd02bb94

git-subtree-dir: interfaces
git-subtree-split: acd02bb94c5a421aa8f8d1fd76cd8aad668e9fcb
2020-11-13 16:16:47 +00:00