Commit Graph

78 Commits

Author SHA1 Message Date
Alex Sharov
26fdf9169d
move all packages from "internal" folder - to simplify users live (#5857) 2022-10-25 09:58:25 +07:00
Alex Sharov
86551e82f4
e3: parallel build missed indices (#5817) 2022-10-21 13:45:20 +07:00
Alex Sharov
606ce5c99a
add logging cli flags to various cmd (#5816) 2022-10-21 12:36:17 +07:00
Alex Sharov
b8dbb53d3b
e3: write history and indices to etl (#5742) 2022-10-15 08:20:58 +07:00
Alex Sharov
b12b0d4627
e3: history bsc and mainnet snapshots (#5725) 2022-10-13 09:46:32 +07:00
Alex Sharov
a2e51a2469
e3: prune limited amount before commit #675 (#5693) 2022-10-11 11:25:13 +07:00
Alex Sharov
6a7f8fb2a6
erigon3: cli command to force merge snapshots (#5680) 2022-10-10 09:47:01 +07:00
Alex Sharov
ca9aa4723c
Compress params change (#5631)
as of https://github.com/ledgerwatch/erigon-lib/pull/651
2022-10-05 17:54:54 +07:00
Alex Sharov
c48d51a582
snapshot index workers amount - based on available RAM and CPU (#5524) 2022-10-05 17:07:01 +07:00
ledgerwatch
94f4ea805d
Fixing hive SideChain reorg test (#5620)
the root cause is that when `inMemoryExecution` lambda gets created in
the `eth/backend.go`, it captures the reference of
`backend.notifications`, and so the execution of side-forks actually
adds notifications to there, and it all gets sent out to tx pool (and
RPC daemon) at the end of the stage loop (regardless of whether there
was forkchoice update or not)

so we can create a separate notification, but then somehow flush it to
the "main" nofitications when the in-memory exec state is flushed

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2022-10-05 05:42:38 +01:00
Alex Sharov
fa16d5e855
erigon3: step toward background snapshots build (#5588) 2022-10-02 10:21:17 +07:00
Alex Sharov
f4a7d2cf6d
erigon3: build .vi after download (#5568) 2022-09-29 12:14:51 +07:00
Alex Sharov
9cac57bcc9
erigon3: set seedable files to 32 steps #5534 2022-09-27 10:51:14 +07:00
Alex Sharov
da7ca3cacf
erigon3: build .efi after download #654 (#5523) 2022-09-26 15:51:39 +07:00
Alex Sharov
ef9e296efd
erigon method to test decompression speed (#5464) 2022-09-22 14:11:39 +07:00
Alex Sharov
50811f9cd6
"erigon snapshots ram" command (#5449)
* save

* save

* save
2022-09-21 09:18:37 +07:00
Alex Sharov
9fb8a190bc
erigon22: folder snapshots/history (#5351) 2022-09-18 17:41:01 +07:00
Enrique Jose Avila Asapche
473ca51830
Adding back stage snapshots (#5331)
* added snapshots into stages list && deleted snapshots from defaul unwind

* added verkle trie

* using the same sentries client hd

* need to safe the stage progress of headers as well
2022-09-12 10:26:00 +07:00
Leonard Chinonso
7495986515
Reverting Adding snapshots as a separate stage commit (#5329)
* reverting Adding snapshots as a separate stage commit

* more fixes to merge conflicts
2022-09-10 19:05:18 +02:00
Enrique Jose Avila Asapche
6d1d3a9f47
moved retiring of blocks from senders into snapshot stage (#5290) 2022-09-09 21:48:16 +07:00
Alex Sharov
caa23b9532
erigon22: simplify interhashes stage (#5315) 2022-09-08 14:02:50 +07:00
Enrique Jose Avila Asapche
8fc7c78620
Bor txlookup (#5260)
* printing

* logging

* spam off

* ops

* clean up

* deleted all bor related stuff in snapshots

* adding bor tx to BorTxLookUp

* added readBorTxLookUp

* getting bor tx from bor lookup

* comments

* deleted bor related stuff

* deleting bor tx

* writing borTxLookUp in stage_txlookup

* using bytes

* little fixes

* updated erigon-lib

* using biEndian

* clean up

* setting bytes for big int

* log

* logging

* more logs

* delete logs
2022-09-02 17:41:58 +03:00
Enrique Jose Avila Asapche
12331e018a
Fix bor txlookup (#5170) 2022-08-30 17:23:16 +03:00
Alex Sharov
69fbfc9bef
Snapshots: Erigon update list in db only after Downloader confirmation, rpcdaemon read list from db (#5150) 2022-08-23 16:28:07 +07:00
Alex Sharov
4594ce5ef7
erigon22: history.v2 flag, align rpcdaemon22 (#5016)
* save

* save

* save

* save

* save
2022-08-12 16:13:14 +07:00
Alex Sharov
aee4b53788
save list of snapshots in db (#4777)
* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save
2022-07-28 16:57:38 +07:00
Andrew Ashikhmin
c1f848746d
Update consensus tests to v11 (#4724)
* Consensus tests update 11

* Add GrayGlacier fork

* Wire currentRandom from tests into PREVRANDAO

* Serenity engine

* Support insertion of PoS blocks in MockSentry

* Introduce marshallTypedTransactionsAsRlpStrings arg into (*Block) RawBody()

* Revert "Introduce marshallTypedTransactionsAsRlpStrings arg into (*Block) RawBody()"

This reverts commit 903fca572be03c7de33318ce177a03a4be34927b.

* Post-merge fix

* Don't wait for Beacon Chain in tests

* Skip powToPosBlockRejection transition test

* ForkChoice in insertPoSBlocks

* Add withPosDownloader arg to MockWithEverything in order to fix TestPoSDownloader
2022-07-26 09:35:38 +02:00
Alex Sharov
68e35417fc
RetireBlocks: encapsulate delete logic 2022-07-23 11:09:16 +07:00
Alex Sharov
cd8b10f57e
snapshot merger: smaller interface (#4786)
* save

* save

* save
2022-07-22 16:54:05 +07:00
Alex Sharov
66758c7960
RetireBlocks: less arguments (#4785)
* save

* save
2022-07-22 13:44:42 +07:00
Alex Sharov
e146b66e35
more usage of dirs object #4390 2022-06-07 11:54:04 +07:00
Alex Sharov
a53642b4bf
datadir.Dirs configuration object to group dir config (#4387) 2022-06-07 10:24:50 +07:00
Alex Sharov
f838268e44
[wip] Snapshots: more runtime invariants check (#4350) 2022-06-03 18:50:01 +07:00
ledgerwatch
bcbb8bec65
Try to prevent double block retire (#4330)
* Try to prevent double block retire

* Remove print

* More careful setting of segmentsMax

* Print snapshots max

* Clean lists

* Print

* Use Reopen

* Fix

* Remove prints

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-06-02 16:58:38 +01:00
Alex Sharov
0795bcb865
integration: validate syncmode flag (#4300) 2022-05-30 09:59:38 +07:00
Alex Sharov
66248c4bfb
Torrent: verify all files (#4270) 2022-05-26 12:27:44 +07:00
Alex Sharov
b1b5a319e1
It's safe now to open snapshots at app start (#4216) 2022-05-20 14:46:04 +07:00
Alex Sharov
4e7807251a
Ensure snapshotDir not created by accident (#4164)
* save

* save

* save
2022-05-17 09:40:45 +07:00
Alex Sharov
4ba1f03efe
Snapshots: atomic dir, step 3 (#4103) 2022-05-10 09:29:44 +07:00
Alex Sharov
32d8ab34d9
atomic snapshot dir, step 2 (#4093)
* atomic snapshot dir

* atomic snapshot dir
2022-05-07 14:53:21 +07:00
ledgerwatch
ba987cf262
Remove preverified hashes (#4088)
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-05-06 15:15:01 +01:00
Alex Sharov
2655d57288
reduce downloader deps (#4010)
* reduce downloader deps

* reduce downloader deps
2022-04-28 12:13:30 +07:00
Alex Sharov
58c86d550d
Snapshots: Max index parallelism to 4 (#3925)
* save

* save

* Readme update: built-in rpc daemon and syncmode=snap

* save

* save

* save
2022-04-21 17:49:42 +07:00
Alex Sharov
6aa4999729
Parallel indexing (#3921) 2022-04-21 10:34:10 +07:00
Alex Sharov
8d81c3b0f8
Rwdir fix2 (#3915) 2022-04-19 08:55:46 +07:00
Alex Sharov
e1d7cc3d3f
Enable: syncmode=snap by default (goerli, mainnet, bsc) (#3910) 2022-04-18 14:40:40 +07:00
ledgerwatch
78a58047dc
Fix in recompression (#3884)
* Fix in recompression

* Fix error handling

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-04-14 11:07:17 +07:00
ledgerwatch
156da607e7
More compact representation of huffman trees in the seg files (#3875)
* More compact representation of huffman trees in the seg files

* Update to latest erigon-lib

* recompress

* recompress

* recompress

* uncompress: buf fix

* Update to latest erigon-lib

* Revert default sync mode

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
2022-04-13 16:29:44 +01:00
alex.sharov
482e06ead3 uncompress: buf fix 2022-04-13 16:35:21 +07:00
alex.sharov
594477ab23 uncompress: buf fix 2022-04-13 12:45:22 +07:00