mirror/erigon-pulse
1
0
Fork 0
mirror of https://gitlab.com/pulsechaincom/erigon-pulse.git synced 2024-12-23 04:03:49 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
13,577 Commits 5 Branches 14 Tags 294 MiB
d4c10c9a47
Commit Graph

64 Commits

Author SHA1 Message Date
Artem Vorotnikov
d4c10c9a47
Port rpcdaemon to KV interface (#1627) 2021-03-30 12:53:54 +03:00
Artem Vorotnikov
6a6f5469a1
KV: Split RO and RW transactions (#1577) 2021-03-21 20:15:25 +07:00
Artem Vorotnikov
b5f9cdd7b2
KV: split Sequence into ReadSequence and MakeSequence (#1569) 2021-03-20 21:12:54 +07:00
Artem Vorotnikov
01f6542729
KV: Remove reserve, putcurrent, cmp, dcmp (#1552) 2021-03-20 16:35:02 +07:00
Alex Sharov
ba37706681
move remote eth backend from ethdb to core package (#1567) 2021-03-20 10:52:00 +07:00
ledgerwatch
81ea5bab78
More updates to downloader, new p2psentry protocol (#1559) 
* Initial commit

* Add sentry gRPC interface

* p2psentry directory

* Update README.md

* Update README.md

* Update README.md

* Add go package

* Correct syntax

* add external downloader interface (#2)

* Add txpool (#3)

* Add private API (#4)

* Invert control.proto, add PeerMinBlock, Separare incoming Tx message into a separate stream (#5)

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>

* Separate upload messages into its own stream (#6)

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>

* Only send changed accounts to listeners (#7)

* Txpool interface doc (#9)

* More additions

* More additions

* Fix locking

* Intermediate

* Fix separation of phases

* Intermediate

* Fix test

* More transformations

* New simplified way of downloading headers

* Fix hard-coded header sync

* Fixed syncing near the tip of the chain

* Add architecture diagram source and picture (#10)

* More fixes

* rename tip to link

* Use preverified hashes instead of preverified headers

* Fix preverified hashes generation

* more parametrisation

* Continue parametrisation

* Fix grpc data limit, interruption of headers stage

* Add ropsten preverified hashes

* Typed hashes (#11)

* Typed hashes

* Fix PeerId

* 64-bit tx nonce

* Disable penalties

* Add goerli settings, bootstrap nodes

* Try to fix goerly sync

* Remove interfaces

* Add proper golang packages, max_block into p2p sentry Status

* Prepare for proto overhaul

* Squashed 'interfaces/' content from commit ce36053c2

git-subtree-dir: interfaces
git-subtree-split: ce36053c24db2f56e48ac752808de60afa1dfb4b

* Change EtherReply to address

* Adaptations to new types

* Switch to new types

* Fixes

* Fix formatting

* Fix lint

* Lint fixes, reverse order in types

* Fix lint

* Fix lint

* Fix lint

* Fix test

* Not supporting eth/66 yet

* Fix shutdown

* Fix lint

* Fix lint

* Fix lint

* return stopped check

Co-authored-by: Artem Vorotnikov <artem@vorotnikov.me>
Co-authored-by: b00ris <b00ris@mail.ru>
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: lightclient <14004106+lightclient@users.noreply.github.com>
Co-authored-by: canepat <16927169+canepat@users.noreply.github.com>
 2021-03-19 21:24:49 +00:00
Artem Vorotnikov
0b2e4df0ff
Align Cursor API with LMDB/MDBX (#1558) 2021-03-19 14:45:01 +07:00
Alex Sharov
4fe019a099
add periodical db statistic metrics (#1545) 2021-03-09 13:34:13 +07:00
ledgerwatch
6a1f000cf7
Bring kv interfaces (#1493) 
* Remove interfaces

* Squashed 'interfaces/' content from commit 0941b0992

git-subtree-dir: interfaces
git-subtree-split: 0941b09926db64934ba7dd161fb9ca2a20ba25b8

* Change generation accordingly

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
 2021-02-13 07:41:47 +00:00
Artem Vorotnikov
5c34713e71
Remove nested transactions (#1483) 2021-02-10 17:04:22 +00:00
Artem Vorotnikov
739c0d30a7
Remove DupFixed (#1482) 2021-02-10 16:56:44 +07:00
Artem Vorotnikov
f7c5ac42d2
Remove unused fields from remoteCursor (#1437) 2021-01-09 12:29:04 +07:00
Andrew Ashikhmin
8a877c227a
Integrate Silkworm execution (#1344) 
* Expose C Handle of lmdbTx

* LoadExecutionFunctionPointer

* silkworm ExecuteBlocks

* linter

* CLI flag for SilkwormExecutionFunc

* linter

* Call SilkwormExecutionFunc in SpawnExecuteBlocksStage

* Commit transaction after Silkworm execution

* Fix batch initialization

* Make batch_size uint64_t instead of size_t

* max_block in silkworm_execute_blocks

* More accurate logProgress

* Use CHandle exposed by lmdb-go

* Enable Silkworm on Linux only

* Update lmdb-go

* Error when attempting to use Silkworm not on Linux

* Move SilkwormFlag from cmd/utils/flags.go to turbo/cli/flags.go

* Integrate Silkworm into cmd/integration

* Check against ReaderBuilder/WriterBuilder/ChangeSetHook in Silkworm execution

* Refactor SpawnExecuteBlocksStage

* linters

* Small clean-ups

* Move ChangeSetHook inside executeBlockWithGo

* No need to tamper with batch size in executeBlocksWithSilkworm

* fix

* Fix

* Fixup

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
 2020-11-28 15:08:02 +00:00
Igor Mandrigin
393c9965ae
rpcdaemon: subscriptions, newHeads (#1359) 
* fix `make grpc` on new checkouts

* update proto files

* add some stub

* prototype with fake events

* notifying about events

* pass events

* events are being sent

* transfer headers to filters

* create the “filters” struct

* implement new heads

* PoC of New Heads subscription

* fix keep alive

* fixups for the client

* add “type” to the event

* support header event type on client

* better stage refactor

* fixup for the eth backend

* fixups

* fix tests

* fix tests

* fix linters

* address comments

* remove unused log
 2020-11-17 19:13:41 +00:00
Alex Sharov
b3f1915d09
ChangeSets dupsort (#1342) 
* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* squash

* squash

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* history_early_stop

* history_early_stop

* vmConfig with ReadOnly false

* auto_increment

* auto_increment

* rebase master

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
 2020-11-16 12:08:28 +00:00
Alex Sharov
1dd283efe2
Sequence for each table (#1354) 
* history_early_stop

* auto_increment

* auto_increment

* tx_table
 2020-11-14 13:48:29 +00:00
Alex Sharov
9712e858e2
Method delete to accept second param: allow delete dupsort values (#1297) 
* aloow_delete_dupsort_values

* check canonical receipt

* exclusive lock for mdbx

* readme_load_dump_tables

* readme_load_dump_tables

* readme_load_dump_tables

* Remove IdealBatchSize change

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
 2020-10-29 13:19:31 +00:00
Alex Sharov
531d36efcb
mdbx support (#1235) 2020-10-28 10:18:10 +07:00
Alex Sharov
331dcd45eb
Store receipts separately - one record per tx (#1271) 
* squash

* add --database flag to integration

* clean

* split to 2 buckets

* split to 2 buckets

* split to 2 buckets

* split to 2 buckets

* split to 2 buckets

* save progress

* save progress

* improve test

* improve test

* save progress

* change app logic

* change app logic

* return err from rawdb package

* don't clean automatically

* don't clean automatically

* clean

* clean

* clean

* don't rely on `make clean`

* improve cbor code

* clean

* clean

* clean

* fix tests

* rebase master

* stop on error: headers stage

* make TxDb walk and multiwalk safe

* Fix panics

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
 2020-10-25 08:38:55 +00:00
Alex Sharov
e85b388061
Protect from wrong class use: make ethdb.Tx incompatible with rawdb.DatabaseReader (#1288) 
* protect_from_wrong_class_use

* more cases
 2020-10-24 09:09:20 +01:00
Alex Sharov
5e53c79f50
stable version of grpc codegen, drop kv_remote_v1 support (#1285) 2020-10-24 07:54:03 +01:00
ledgerwatch
b747ab5324
[WIP] CallTraces index (#1157) 
* Initial commit for CallTraces index

* Fix compilation

* fix lint, add comment

* Fix integration

* Add Close function to ethdb.Cursor, fix some compile errors

* Try to stop cursor leak in Get

* Fix compile errors in RPC daemon

* Fix compile errors

* fixing another way

* Some fixes

* More fixes

* More fixes

* More fixes

* Fixes to core/state

* Fix lint

* Fix lint

* Fixes

* Stage caching for call trace stage

* Add mem stats

* Try to stop the leak

* Turn off debug

* Chunks for 10k blocks

* Print

* Revert "Print"

This reverts commit 5ffada4828d61e00e5dad1ca12c98258dfbbad00.

* Revert "Chunks for 10k blocks"

This reverts commit cfb9d498e782e5583d41c30abf0e2137da27383e.

* Trying to fix the leak

* Don't compute receipts in re-tracing

* Not compose block

* Print speed, fix receipts, bigger caches

* Fix lint

* Utilise changeset info

* Counters

* Use NoReceipts and ReadOnly

* ReadOnly is incompatible with caching

* Skip test leaking transactions

* Fix block test

* Change disable message for call-traces stage

* Use block option for call traces integration

* Fix retracing due to incarnation
 2020-10-12 09:39:04 +01:00
Alex Sharov
9cf10a8f85
Rpcdaemon: Gracefull shutdown, better grpc stream close (#1196) 
* save progress

* GracefulShutdown grpc server, send to server close signal before canceling context

* GracefulShutdown grpc server, send to server close signal before canceling context

* GracefulShutdown json server

* GracefulShutdown json server

* fix lint

* fix lint

* can

* clean

* test

* test

* test

* increase keepalive timeout, increase vegeta timeout, mark server-side errors

* increase keepalive timeout, increase vegeta timeout, mark server-side errors

* hack hugeFreelist

* up streams limit

* up streams limit

* up streams limit

* up streams limit

* up streams limit

* up streams limit

* up streams limit

* up streams limit

* up streams limit

* up streams limit

* up streams limit

* fix_race_condition_on_zstd_build
 2020-10-10 07:06:54 +01:00
ledgerwatch
4b5a333583
Revert the context unbundling change (#1203) 2020-10-07 22:53:04 +01:00
ledgerwatch
70ebf59642
Context cleanup and make context independent (#1193) 2020-10-06 22:07:57 +01:00
Alexey Akhunov
eb5e01ff5d Do not cancel context when closing remote cursor 2020-10-06 17:11:30 +01:00
Alex Sharov
48cf5fb43e
History-style sharding for bitmap indices (#1159) 2020-10-02 10:54:11 +07:00
Alex Sharov
0559d56dae
Pin protoc version and store it in ./build/bin (#1154) 
* use ./build/bin folder to store protobuf binaries

* use ./build/bin folder to store protobuf binaries
 2020-09-29 15:09:50 +01:00
ledgerwatch
7bb9cb4274
Remove BloomStatus from the API (#1155) 
* Remove BloomStatus from the API

* Trying to find missing space

* Fix lint
 2020-09-29 15:09:29 +01:00
Alex Sharov
e02d6acc7d
bitmap indices for logs (#1124) 
* save progress

* try now

* don't create bloom inside rlpDecode

* don't create bloom inside ApplyTransaction

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* rename method

* print timings

* print timings

* print timings

* sort before flush

* fix err lint

* clean

* move tests to transactions

* compressed version

* up bound

* up bound

* more tests

* more tests

* more tests

* more tests

* better removal

* clean

* better performance of get/put methods

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* optimize rpcdaemon

* fix test

* fix rpcdaemon

* fix test

* simplify

* simplify

* fix nil pointer

* clean

* revert some changes

* add some logs

* clean

* try without optimize

* clean

* clean

* clean

* clean

* try

* move log_index to own stage

* move log_index to own stage

* integration add log_index stage

* integration add log_index stage

* clean

* clean

* print timing

* remove duplicates at unwind

* extract truncateBitmaps func

* try detect

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* add blackList of topics

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* sharding 1

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 3

* sharded 3

* sharded 3

* speedup things by putCurrent and putReserve

* clean

* optimize trim

* clean

* remove blacklist

* add more info to err

* ?

* clean

* clean

* clean

* clean

* clean

* working version

* switch to cgo version of roaring bitmaps

* clean

* clean

* clean

* clean

* more docs

* clean

* clean

* fix logs bloom field

* Fix debug_getModifiedAccountsByNumber

* Try to fix crash

* fix problem with "absent block"

* fix problem with "absent block"

* remove optimize method call

* remove roaring iterator

* fix problem with rebuild indicess

* remove debug prints

* tests for eth_getLogs involving topics

* add tests for new stage, speparate topics into 2 buckets

* version up

* remove debug logs

* remove debug logs

* remove bloom filter implementation

* Optimisation

* Optimisatin not required, make rpctest lenient to geth errors

* Lenient to geth failures

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
 2020-09-28 18:18:36 +01:00
Giulio rebuffo
d33302c2be
Mutual TLS authentication (for now without verifying the Common Name) (#1113) 
* added tls auth

* added client side

* put --tls

* fixed flag

* Add key/cert generation instructions, turn off Common Name verification

* Add CLI arguments and Warning

* Lint

* Update the doc about Internal IP

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
 2020-09-19 15:16:04 +01:00
Giulio rebuffo
23d89c0941
Added TLS handshake to RPCDaemon (#1089) 
* added eth_getStorageAt

* used uint32

* now its 256

* incarnation

* added TLS handshake

* code minified

* lint

* minified client

* Update flags.go

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
 2020-09-11 21:17:37 +01:00
Alex Sharov
62fe81e4be
IH stage speedup and lmdb custom comparators support (#1080) 
* etl.Loader - allow use of custom comparator

* log timing

* try now

* try now

* more performance

* etl.Loader - allow use of custom comparator

* working version

* simplify IH cursor

* clean

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* clean

* add only unwind support

* squash

* squash

* clean

* fix test

* clean

* clean

* clean
 2020-09-10 13:35:58 +01:00
Alex Sharov
c45a710ce6
Warmup logs, less overhead, warmup code bucket (#1054) 
* warmup logs and less overhead.

* warmup logs and less overhead.

* move WarmUp to common func
 2020-09-08 08:28:37 +01:00
ledgerwatch
3a92b2b39d
Fix for RPC daemon leak (#1059) 
* Start memory prof

* Fix rpctest

* Fix rpctest

* Attempt to fix the leak

* Remove http pprof
 2020-09-05 21:58:51 +01:00
Alex Sharov
b9bdd003a7
KV: Raw (no custom logic) cursors for DupSort and DupFixed buckets (#1020) 2020-09-04 10:54:15 +07:00
Evgeny Danilenko
e4f495fa44
Get logs (#1028) 
* it compiles

* after recent master

* fix linters warnings

* grpcV7

* go mod tidy

* unmarshall adresses or adress

* fix linters

* after cr

* after cr

* after cr

* after cr

* fix tests

* remove dev version

* it compiles

* mod tidy

* fix bin deps

* use stable version of grpc

* switch back to master constructor

* switch back to master constructor

* add a bit docs

* add a bit docs

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
 2020-09-03 08:51:19 +01:00
Alex Sharov
49351b0091
KV: to support multiple databases and multiple set of buckets (#1002) 
* don't use global variables

* up deps

* clean

* use joined flags

* use single non-global buckets config

* clean

* copy map

* fix tests

* fix tests

* clean

* clean
 2020-08-30 18:34:18 +01:00
Alex Sharov
9091ce62bb
KV reduce features amount (#975) 
* move IdealBatchSize method to Batch object :-)
remove cursor.Walk
remove kv.IdealBatchSize

* remove bolt_db.go file

* move method NoValueCursor to tx object
 2020-08-26 07:03:50 +01:00
Evgeny Danilenko
3b40819444
grpc regenerate (#936) 
* grpc regenerate

* use string type for id in net_version
 2020-08-18 20:22:49 +03:00
Alex Sharov
081d02920d
Remove "Bucket" class from KV abstraction (#928) 
* drop Bucket object from KV abstraction

* add some non-public methods

* rebase master

* rebase master
 2020-08-17 08:24:59 +01:00
Alex Sharov
b2e35eb63f
rename "remote2" to "remote" (#932) 2020-08-17 07:47:06 +01:00
Alex Sharov
e04c4ebe58
KV: move bucket.Get and bucket.Cursor methods to Tx object (#914) 2020-08-14 13:41:18 +07:00
Alex Sharov
16f09be0a8
add method .Last() (#909) 2020-08-12 10:49:52 +07:00
Alex Sharov
e89ad5d6aa
sub tx support (#895) 2020-08-11 17:35:59 +07:00
Alex Sharov
d9d9e14f45
change bucket type to string (#894) 2020-08-11 06:55:32 +07:00
Alex Sharov
7a1b892022
remove .SeekTo() method (#901) 2020-08-11 06:41:15 +07:00
ledgerwatch
0514501937
Revert "Use stage name as db key (#858)" (#867) 
This reverts commit 7290bf519e.
 2020-08-04 11:05:27 +01:00
Alex Sharov
7290bf519e
Use stage name as db key (#858) 2020-08-04 16:25:28 +07:00
Alex Sharov
0a367001d1
implement db stats page (#803) 2020-07-29 05:31:46 +01:00