Commit Graph

12945 Commits

Author SHA1 Message Date
alex.sharov
78e5bd37e4 try to use exclusive lock from prometheus 2020-10-26 19:08:36 +07:00
ledgerwatch
7554428884
POC of header downloader - splitting into 2 processes: sentry + downloader (via gRPC) (#1291)
* Splitting sentry and downloader - the beginning

* A bit more

* More on sentry

* More gRPC

* Sentry and downloader separated

* Update binding for stable version of grpc

* Better bufferSize flag

* Fix lint

* Send pelanties

* Fix lint

* Remove hard-coded tips on connect

* Tidy the logs a bit

* Deal with hardTips on Recovery

* Print hard tips

* Hide empty anchors

* Request headers after receiving a message

* Better waking up

* Print hard-coded block numbers

* Print outgoing requests

* Debug logging

* In the middle protection

* Sentry not to lose peers when core disconnects
2020-10-25 22:05:37 +00:00
Evgeny Danilenko
b6687fc9aa
initial (#1296) 2020-10-25 21:34:00 +00:00
ledgerwatch
dd2c047cdf
[WIP] Defrag utility for LMDB database (#1268)
* Initial

* Read freelist pages

* Fix

* Fix lint

* Fix lint

* Fix lint
2020-10-25 10:10:55 +00:00
Alex Sharov
6f27e3043b
lmdb_readonly_dont_create_folder (#1293) 2020-10-25 08:39:09 +00:00
Alex Sharov
331dcd45eb
Store receipts separately - one record per tx (#1271)
* squash

* add --database flag to integration

* clean

* split to 2 buckets

* split to 2 buckets

* split to 2 buckets

* split to 2 buckets

* split to 2 buckets

* save progress

* save progress

* improve test

* improve test

* save progress

* change app logic

* change app logic

* return err from rawdb package

* don't clean automatically

* don't clean automatically

* clean

* clean

* clean

* don't rely on `make clean`

* improve cbor code

* clean

* clean

* clean

* fix tests

* rebase master

* stop on error: headers stage

* make TxDb walk and multiwalk safe

* Fix panics

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-10-25 08:38:55 +00:00
Ricardo Guilherme Schmidt
e9572ae981
Fix infinite rpc.gascap on eth_estimateGas (#1292)
* Fix infinite rpc.gascap on eth_estimateGas

* use `gasCap.Sign() > 0` instead of `gasCap.Uint64() != 0`
2020-10-25 08:19:59 +00:00
Alex Sharov
c2f41d7723
stop on error: headers stage (#1289) 2020-10-24 18:05:12 +01:00
Thomas Jay Rush
1af3e09d75
1209 automate documentation (#1290)
* Automate documentation

* Updating

* Updating

* Fixing lint

* Updating readthedocs documentation

* Updating testing

* Updating testing
2020-10-24 18:03:52 +01:00
Alex Sharov
e85b388061
Protect from wrong class use: make ethdb.Tx incompatible with rawdb.DatabaseReader (#1288)
* protect_from_wrong_class_use

* more cases
2020-10-24 09:09:20 +01:00
Alex Sharov
77a4b3305d
rawdb to return errors: part 1 (#1286)
* ReadBlockByNumber and ReadChainConfig return error

* Let's make rawdb package to return error. But by small steps.
High-level methods need to move first:

ReadBlockByNumber
ReadChainConfig
ReadBlockByHash
ReadHeaderByHash
WriteChainConfig
WriteDatabaseVersion
WriteBlock
DeleteTd
WriteTd
ReadTd
2020-10-24 07:57:09 +01:00
Alex Sharov
6584b43b12
Allow start read transactions from ethdb.Database interface (#1287)
* allow_read_transactions

* fix lint

* fix lint

* more cases
2020-10-24 07:55:43 +01:00
Alex Sharov
5e53c79f50
stable version of grpc codegen, drop kv_remote_v1 support (#1285) 2020-10-24 07:54:03 +01:00
alex.sharov
c043a7c876 prometheus docs for developers 2020-10-24 10:15:53 +07:00
Alex Sharov
76f1b05cb2
Move tmpdir definition to app-start, move migrations folder inside tmpdir (#1282)
* extract tmpdir to app-level-code

* extract tmpdir to app-level-code

* save progresss
2020-10-23 12:18:45 +01:00
Alex Sharov
0bc61c06ed
ethdb readme, db objects diagram (#1281)
* db objects diagram

* db objects diagram

* db objects diagram

* db objects diagram

* db objects diagram

* db objects diagram

* db objects diagram
2020-10-23 12:17:16 +01:00
Alex Sharov
dd5258d524
save progresss (#1283) 2020-10-23 11:52:18 +01:00
Igor Mandrigin
bf044f73ef
Update README.md 2020-10-22 17:44:28 +02:00
Igor Mandrigin
3bec338aa4
Update README.md 2020-10-22 17:40:37 +02:00
Igor Mandrigin
468f246371 begin 2020.10.03 release cycle 2020-10-22 12:30:48 +02:00
ledgerwatch
fc5138c0e2
Update skip analysis (#1277) 2020-10-22 11:16:39 +01:00
Alex Sharov
1e237069e9
migrations: make tmp dir name predictable and unique (#1276) 2020-10-22 10:30:04 +02:00
ledgerwatch
88acfc55ca
Vary batch size for execution stage (#1274)
* Vary batch size for execution stage

* Fix in logging

* Improve log of timings

* Log improvements

* Log improvements

* Fix tests

* More log improvments, fix tests

* Fix tests

* Better logging for Bodies and Interhash

* Fix tests

* Minor fixes

* Fix test
2020-10-21 18:01:40 +01:00
alex.sharov
716a0f2dba link dashboard from main readme 2020-10-21 11:33:10 +07:00
alex.sharov
a1d0cd3a99 improve prometheus docs 2020-10-21 11:26:23 +07:00
alex.sharov
ee0abb3095 improve prometheus docs 2020-10-21 11:24:22 +07:00
alex.sharov
761af4f372 rebase master 2020-10-21 11:23:22 +07:00
alex.sharov
67c295ef15 rebase master 2020-10-21 11:22:25 +07:00
Evgeny Danilenko
44562fd07a
Jump tables data race (#1245)
* initial

* fix

* fmt

* linters

* linters

* debug

* fix data races

* safe close

* remove jump table clonning

* revert fix
2020-10-20 22:23:56 +01:00
Thomas Jay Rush
2aa6fb50ca
Cleans up some custom RPC commands (#1270)
* Moving a few files to be more consistent

* Collecting together custom code under tg_ namespace

* Small cleanups

* Updating test cases

* Uncomment eth_gasPrice but still unimplemented
2020-10-20 22:16:28 +01:00
Alex Sharov
9e7ed07fc6
Remove restapi and debug-ui from readme and docker-compose (#1263)
* rm_restapi_from_readme

* rm_restapi_from_readme

* rm_restapi_from_readme

* rm_restapi_from_readme
2020-10-20 09:20:24 +01:00
Giulio rebuffo
0a4386737e
Uses number instead of others to repressenting progressing in accord to block number (#1260)
* logs fixed with number=

* better logs

* removed lol
2020-10-19 22:46:26 +01:00
Alex Sharov
44bbe97a1f
Migration idempotency: handle corner-cases of CriticalCollector (#1257)
* save progress of db migration

* save progress of db migration

* delete

* delete

* delete

* delete

* delete

* print full key if it has 8 leading zeroes

* print full key if it has 8 leading zeroes

* print full key if it has 8 leading zeroes
2020-10-19 20:20:18 +01:00
Giulio rebuffo
f359527356
better logs (#1259) 2020-10-19 20:17:21 +01:00
canepat
5b62588ad3
Update description of table Headers (#1258)
* Update description of table Headers

* Complete the table Headers description
2020-10-19 20:15:27 +01:00
alex.sharov
dd21d07a7d print full key if it has 8 leading zeroes 2020-10-19 21:03:50 +07:00
Alex Sharov
5dbe1724be
Bitmap etl (#1227)
* save progress

* save progress

* save progress

* save progress

* etl bimaps

* etl bimaps

* etl bimaps

* etl bimaps

* a

* a

* a

* a

* a

* a

* a

* save progress

* save progress

* save progress

* a

* fix_race_condition_on_zstd_build

* clean

* clean

* clean

* clean

* clean

* go mod tidy

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* fix corner case

* commit every

* call traces etl

* call traces etl

* call traces etl

* return missed error

* go mod tidy

* go mod tidy

* go mod tidy

* rebase to master

* remove trash

* print full key if it has 8 leading zeroes
2020-10-19 14:11:01 +01:00
Alex Sharov
d7bcb29802
do commit by don't save partial progress (#1256) 2020-10-19 07:43:30 +01:00
Evgeny Danilenko
3bec75cdc4
Extract nonstandard APIs and introduce tg namespace (#1248)
* initial

* update rpc deamon readme

* use read-only transactions
2020-10-18 20:44:28 +01:00
Giulio rebuffo
0fa82ef2dd
added stage 2 documentation (#1251) 2020-10-18 16:03:04 +01:00
ledgerwatch
bdef07f5f2
Fix cmd/state checkChangeSets and calltraces (#1252)
* Fix to checkChangeSets

* Fix lint

* Update skip analysis

* Fix

* Update skip analysis again
2020-10-18 08:31:23 +01:00
ledgerwatch
d2e5cc5cb5
Better migration - receipts_cbor_encode not to increase size of the database (#1250)
* Add print_migration command to integration

* Not run migrations for print commands

* Reverse migration

* Fix lint

* Add datadir to run_migrations

* Add lmdb.mapSize to integration

* Make lmdb.mapSize effective

* Fix

* receipt sizes

* fix print

* Switch to another version of lmdb

* Collector from existing files

* Not load

* Commit after clearing

* Add methods for clear/drop buckets by multiple transactions

* Fix to lmdb

* Not remove temp files

* Not load

* Re-enable loading

* Prints

* Try to fix refill

* Migration in 3 steps

* tidy mod

* Fix lint

* Fix tests

* Migrations for CBOR receipts in 3 txs

* Fix lint

Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
2020-10-17 14:00:12 +01:00
b00ris
e5f6412da2
Update snapshot hash (#1243)
* remove useless bucket

* update snapshot hashes

* remove log
2020-10-17 08:48:20 +01:00
Giulio rebuffo
78c4ddea03
added documentation for stage 1 (#1246) 2020-10-17 08:47:17 +01:00
Evgeny Danilenko
946d4fac5b
ForkID api (#1244)
* forkID api

* merge fork calls

* optional param
2020-10-16 15:15:10 +03:00
Thomas Jay Rush
c6c82b7569
Finish moving functions to consistent locations to ease testing/documentation of RPC (#1241)
* Cleaning up comments to start

* Renaming get_receipts.go to eth_receipts.go

* Stubbing out RPC mining routines

* Move block-related commands to eth_block.go

* Move transaction-related commands to eth_txs.go

* Move account-related commands to eth_accounts.go

* Move system-related commands to eth_system.go

* Final cleanup

* Updating testing
2020-10-14 16:59:42 +01:00
Alex Sharov
a8f38ba1c3
fix db opening (#1240) 2020-10-14 13:05:54 +01:00
Thomas Jay Rush
dcdcc1a30f
Fixes issue #1232 (#1233)
* Fixes issue #1232

* Adding/updating test cases

* Updating as per comments

* Use tx instead of db in GetModifiedAccounts and clean up a bit

* option to specify LMDB map size for cmd/integration commands (#1238)

* Add lmdb.mapSize to integration

* Make lmdb.mapSize effective

* Use mapSize for first opening

* Fix for cobra

* Extend lmdb.mapSize for all commands

* Fix lint

* Fixes issue #1232

* Adding/updating test cases

* Updating as per comments

Co-authored-by: ledgerwatch <akhounov@gmail.com>
2020-10-14 12:01:44 +01:00
ledgerwatch
234e35851c
option to specify LMDB map size for cmd/integration commands (#1238)
* Add lmdb.mapSize to integration

* Make lmdb.mapSize effective

* Use mapSize for first opening

* Fix for cobra

* Extend lmdb.mapSize for all commands

* Fix lint
2020-10-13 23:42:04 +01:00
ledgerwatch
cd706d5081
Download/seed block bodies and block headers via BitTorrent (#1218)
* save state

* torrent experiments

* torrent experiment passed

* fixes after merge

* snapshot headers processing passed

* save state

* save state

* download headers works after snapshot processing

* save state

* save state

* save state

* save state

* add lazy load tx to snapshots, increase number of trackers

* save state

* speedup getting info

* change logging

* move to turbo package

* save state

* save state

* save state

* cleanup

* save state

* add test test

* save state

* lmdb debugging

* fix readonly mode

* save state

* fix build

* sync works

* save state

* save state

* save state

* allow cmd stages stageSenders use snapshots

* debugging failed hashing

* remove experimental tests

* remove torrent experimental tests

* fix lint

* extract snapshot wrapper

* metainfo checker

* add remote seeder

* add logs

* update gomod

* remove useless code

* fix lint&remove useless code

* extract verify snapshot to separated command

* skip debug test

* fix test

* change type of seedSnapshot flag

* add eth logger to torrent lib

* skip debug test

* add Close method

* review fixes

* fix lint

* tidy mods

* Fix compile

* Fix lint

* Fix rpcdaemon running in the docker

Co-authored-by: b00ris <b00ris@mail.ru>
Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
2020-10-13 13:56:16 +01:00