Commit Graph

51 Commits

Author SHA1 Message Date
Alex Sharov
606ce5c99a
add logging cli flags to various cmd (#5816) 2022-10-21 12:36:17 +07:00
Alex Sharov
9fb8a190bc
erigon22: folder snapshots/history (#5351) 2022-09-18 17:41:01 +07:00
Alex Sharov
25d85d88c5
state erigon22: use right db (#5028) 2022-08-12 21:45:09 +07:00
Alex Sharov
f18a5b0864
integration to pass mdbx.Accede flag (#4719) 2022-07-15 13:17:07 +07:00
Alex Sharov
935975bebd
methods to configure db (#4412)
* save

* save

* save
2022-06-09 09:45:30 +07:00
Alex Sharov
7c79d6d453
Cfg dirs (#4388) 2022-06-07 11:00:37 +07:00
Alex Sharov
d4d4c2ec13
Open Integration: use same db limit (#4259) 2022-05-25 16:22:02 +07:00
Alex Sharov
c9cee7a884
path -> filepath (path package is for urls) (#3493) 2022-02-12 20:33:09 +07:00
Alex Sharov
702631bf6f
prohibit change prune mode (#2908) 2021-11-03 20:05:52 +07:00
alex.sharov
74168d9426 remove debug prints 2021-10-06 14:05:22 +07:00
Alex Sharov
21785780bf
IntermediateHash stage - switch from incremental to re-generate mode - if jump > 100K blocks (#2781) 2021-10-05 15:32:18 +07:00
Andrea Lanfranchi
00d627bdb0
Remove config.name (const "erigon") from paths (#2531)
* Remove config.name (const "erigon") from paths

* Fix unit test

* Safety check

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2021-08-18 16:24:03 +07:00
Alex Sharov
6bd44eb26c
move kv to erigon-lib (#2467) 2021-07-29 18:53:13 +07:00
Alex Sharov
5069558752
Apache licensed logger (#2460) 2021-07-29 17:23:23 +07:00
Alex Sharov
838e5f9ef2
Move bucket constants into kv package, move kv interface to kv package (#2455) 2021-07-28 09:47:38 +07:00
Alex Sharov
ba902f7ef3
separate_config_from_node_and_eth (#2289) 2021-07-04 08:50:32 +01:00
Alex Sharov
2b59b45216
Do log tables size, db metrics - avoid concurrency, check stale readers hourly (#2216)
* add table size logs and metrics

* log stale readers

* - don't collect db metrics in background
- do check stale readers once an hour

* execution to update metrics when print logs

* hide file exists err

* hide file exists err
2021-06-22 11:08:47 +01:00
Alex Sharov
bc271f92ce
To break dependency to db implementation - move all db classes to "ethdb/kv" and leave in "ethdb" only interfaces (#2195) 2021-06-19 15:21:53 +07:00
Artem Vorotnikov
af836a6200
Nuke LMDB (#2167) 2021-06-16 13:57:58 +03:00
Alex Sharov
925d17a238
remove lmdb from hack, integration, state packages. remove lmdb dbtools (#2146) 2021-06-11 20:31:37 +07:00
Alex Sharov
eee871aedf
Add db label (later will add txpool db) (#2081)
* add db label - later will add txpool db

* move commit marker to kv_mdbx behind Chain label

* Update migrations.go

* clean

Co-authored-by: ledgerwatch <akhounov@gmail.com>
2021-06-04 15:56:49 +01:00
Evgeny Danilenko
74847d77e6
Rename to Erigon (#2018)
* turbo-geth to erigon

* tg, turbo to erigon
2021-05-26 11:35:39 +01:00
Alex Sharov
450c79e3ab
move integration to kv (#1996) 2021-05-23 12:41:42 +07:00
ledgerwatch
729bfab988
Split DBSchema version depending on the database type (#1990)
* Split DBSchema version depending on the database type

* Fix test

* Fix test

* Fix lint

* Update migrations.go

* Update bucket.go

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2021-05-22 10:20:43 +01:00
Alex Sharov
545fe32f68
Mdbx - make it default db. Lazy buckets renaming. (#1982) 2021-05-22 10:03:02 +07:00
Alex Sharov
0be3044b7e
rename (#1978)
* rename

* rename "make grpc"

* rename "abi bindings templates"

* rename "abi bindings templates"
2021-05-20 19:25:53 +01:00
ledgerwatch
a99d22c313
Trace filter 4 + fixes for download.v2 (#1907)
* Introduce TraceCallSet bucket and fill it up

* Fix compile

* Reset TraceCallSet

* Add database.verbosity

* Warn about dropped messages

* Unwind for call traces

* Fixes for headers download

* Fix for the slow stage 3

* Remove wakeup channels

* Permit system for peer load balancing

* Reduce aggressiveness

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2021-05-10 16:07:22 +01:00
ledgerwatch
4a214d38e4
Fix integration after snapshot PR (#1901)
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2021-05-08 11:42:34 +01:00
Alex Sharov
00f700675b
add --datadir parameter to integration, snapshot generator, header downloader (#1749) 2021-04-19 14:25:26 +07:00
Artem Vorotnikov
d4c10c9a47
Port rpcdaemon to KV interface (#1627) 2021-03-30 12:53:54 +03:00
Alex Sharov
aff859edc0
Mining stage (#1554) 2021-03-23 16:00:07 +07:00
Alex Sharov
060f3b9a60
Grafana: add Btree metrics (#1547) 2021-03-10 10:58:57 +07:00
Alex Sharov
4fe019a099
add periodical db statistic metrics (#1545) 2021-03-09 13:34:13 +07:00
Alex Sharov
8538ec85b3
don't open db in exclusive mode if no migrations (#1524) 2021-03-01 11:02:22 +07:00
Alex Sharov
a139f31dcd
Exclusive lock manual (#1302)
* exclusive lock for chaindata folder

* exclusive lock for chaindata folder

* exclusive lock for chaindata folder

* mdb tools

* resolve abs path

* rpcdaemon to check exclusive lock

* rpcdaemon to check exclusive lock

* exclusive lock for mdbx

* exclusive lock for mdbx
2020-10-28 09:52:15 +00:00
Alex Sharov
531d36efcb
mdbx support (#1235) 2020-10-28 10:18:10 +07:00
ledgerwatch
d2e5cc5cb5
Better migration - receipts_cbor_encode not to increase size of the database (#1250)
* Add print_migration command to integration

* Not run migrations for print commands

* Reverse migration

* Fix lint

* Add datadir to run_migrations

* Add lmdb.mapSize to integration

* Make lmdb.mapSize effective

* Fix

* receipt sizes

* fix print

* Switch to another version of lmdb

* Collector from existing files

* Not load

* Commit after clearing

* Add methods for clear/drop buckets by multiple transactions

* Fix to lmdb

* Not remove temp files

* Not load

* Re-enable loading

* Prints

* Try to fix refill

* Migration in 3 steps

* tidy mod

* Fix lint

* Fix tests

* Migrations for CBOR receipts in 3 txs

* Fix lint

Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
2020-10-17 14:00:12 +01:00
ledgerwatch
234e35851c
option to specify LMDB map size for cmd/integration commands (#1238)
* Add lmdb.mapSize to integration

* Make lmdb.mapSize effective

* Use mapSize for first opening

* Fix for cobra

* Extend lmdb.mapSize for all commands

* Fix lint
2020-10-13 23:42:04 +01:00
ledgerwatch
cd706d5081
Download/seed block bodies and block headers via BitTorrent (#1218)
* save state

* torrent experiments

* torrent experiment passed

* fixes after merge

* snapshot headers processing passed

* save state

* save state

* download headers works after snapshot processing

* save state

* save state

* save state

* save state

* add lazy load tx to snapshots, increase number of trackers

* save state

* speedup getting info

* change logging

* move to turbo package

* save state

* save state

* save state

* cleanup

* save state

* add test test

* save state

* lmdb debugging

* fix readonly mode

* save state

* fix build

* sync works

* save state

* save state

* save state

* allow cmd stages stageSenders use snapshots

* debugging failed hashing

* remove experimental tests

* remove torrent experimental tests

* fix lint

* extract snapshot wrapper

* metainfo checker

* add remote seeder

* add logs

* update gomod

* remove useless code

* fix lint&remove useless code

* extract verify snapshot to separated command

* skip debug test

* fix test

* change type of seedSnapshot flag

* add eth logger to torrent lib

* skip debug test

* add Close method

* review fixes

* fix lint

* tidy mods

* Fix compile

* Fix lint

* Fix rpcdaemon running in the docker

Co-authored-by: b00ris <b00ris@mail.ru>
Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
2020-10-13 13:56:16 +01:00
ledgerwatch
1af6cbe37b
Revert "Download/seed headers and block bodies via BitTorrent (#1195)" (#1202)
This reverts commit 94360212cb.
2020-10-07 21:49:44 +01:00
ledgerwatch
94360212cb
Download/seed headers and block bodies via BitTorrent (#1195)
* save state

* torrent experiments

* torrent experiment passed

* fixes after merge

* snapshot headers processing passed

* save state

* save state

* download headers works after snapshot processing

* save state

* save state

* save state

* save state

* add lazy load tx to snapshots, increase number of trackers

* save state

* speedup getting info

* change logging

* move to turbo package

* save state

* save state

* save state

* cleanup

* save state

* add test test

* save state

* lmdb debugging

* fix readonly mode

* save state

* fix build

* sync works

* save state

* save state

* save state

* allow cmd stages stageSenders use snapshots

* debugging failed hashing

* remove experimental tests

* remove torrent experimental tests

* fix lint

* extract snapshot wrapper

* metainfo checker

* add remote seeder

* add logs

* update gomod

* remove useless code

* fix lint&remove useless code

* extract verify snapshot to separated command

* skip debug test

* fix test

* change type of seedSnapshot flag

* add eth logger to torrent lib

* skip debug test

* add Close method

* review fixes

* fix lint

Co-authored-by: b00ris <b00ris@mail.ru>
Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
2020-10-07 13:15:49 +01:00
ledgerwatch
bdb0155856
Revert "Add sync by bittorrent snapshots (#1160)" (#1194)
This reverts commit 4ebade3583.
2020-10-06 22:31:57 +01:00
b00ris
4ebade3583
Add sync by bittorrent snapshots (#1160)
* save state

* torrent experiments

* torrent experiment passed

* fixes after merge

* snapshot headers processing passed

* save state

* save state

* download headers works after snapshot processing

* save state

* save state

* save state

* save state

* add lazy load tx to snapshots, increase number of trackers

* save state

* speedup getting info

* change logging

* move to turbo package

* save state

* save state

* save state

* cleanup

* save state

* add test test

* save state

* lmdb debugging

* fix readonly mode

* save state

* fix build

* sync works

* save state

* save state

* save state

* allow cmd stages stageSenders use snapshots

* debugging failed hashing

* remove experimental tests

* remove torrent experimental tests

* fix lint

* extract snapshot wrapper

* metainfo checker

* add remote seeder

* add logs

* update gomod

* remove useless code

* fix lint&remove useless code

* extract verify snapshot to separated command

* skip debug test

* fix test

* change type of seedSnapshot flag

* add eth logger to torrent lib

* skip debug test

* add Close method

* review fixes

* fix lint

Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-10-06 20:24:48 +01:00
Alex Sharov
f827f86c11
Rpcdaemon as lib 2 (#943) 2020-08-20 10:52:27 +07:00
Alex Sharov
1dcc2b141a
Rpcdaemon as lib (#940)
* share config object

* create default config and logger

* move db connection to common func

* move server start to cli package

* clear

* clear

* rename cli to rpc

* use unified SetupLogger func

* make all root flag persistent

* use common flags in different packages

* use common flags in different packages

* move TraceTx method to eth package

* use native slice flags

* create package "turbo"

* disable geth api

* disable geth api

* move more data types to turbo/adapter package

* add support for customApiList

* run more

* run more

* run more

* dog-food

* move DoCall

* move DoCall

* fix tests

* fix test
2020-08-19 12:46:20 +01:00
Alex Sharov
091819a51c
adopt --metrics.addr flag in integration (#889) 2020-08-11 06:38:34 +07:00
Igor Mandrigin
8600616c3d geth 1.9.18 post-rebase fixups 2020-08-08 17:33:35 +02:00
Alex Sharov
d458c4cc1e
Migrations: use stage name as db key (#868) 2020-08-05 17:13:35 +07:00
ledgerwatch
0514501937
Revert "Use stage name as db key (#858)" (#867)
This reverts commit 7290bf519e.
2020-08-04 11:05:27 +01:00
Alex Sharov
7290bf519e
Use stage name as db key (#858) 2020-08-04 16:25:28 +07:00