Commit Graph

170 Commits

Author SHA1 Message Date
Alex Sharov
8873903605
Snapshots table (#510)
* save

* save
2022-07-04 10:56:35 +06:00
Alex Sharov
7aa540e6fa
save (#508) 2022-07-01 10:10:23 +06:00
Giulio rebuffo
b2c211d9bf
Refactor sequence handling in mutation 2022-06-25 01:05:27 +02:00
Giulio rebuffo
541dbe8c47
SeekBothRange instead of FirstDup 2022-06-25 01:04:51 +02:00
Giulio rebuffo
8e0df524d8
Added more functionality to the mutation. 2022-06-24 22:10:45 +02:00
ledgerwatch
46bebb3317
[erigon2.2] Add ReadIndices aggregator to collect data (#500)
* [erigon2.2] Add ReadIndices aggregator to collect data

* Try

* Fix for history access

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-06-20 08:39:29 +01:00
Giulio rebuffo
addc05bae3
Plaxeholder values for dbsize and pagesize 2022-06-19 00:46:31 +02:00
Giulio rebuffo
d57a16c7f6
Implemented ForAmount and MDBX sequencing (#497) 2022-06-18 21:24:30 +02:00
Giulio rebuffo
988a4b5510
Added update txn function 2022-06-17 23:59:04 +02:00
Giulio rebuffo
280c5e9dcc
Added memory mutation to Erigon lib (#493) 2022-06-16 08:01:48 +01:00
ledgerwatch
e2c6ef0058
[erigon2.2] Fixes for inverted indices and domains for the prototype (#489)
* Better control of compress/uncompressed

* Add new function

* more careful pruning

* Printf

* Printf

* Fix DupSort

* Remove copying in prune

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-06-13 19:32:13 +01:00
Alex Sharov
e4b9cc3884
Reclaimable space in logs (#484)
* save

* save

* save

* save

* save

* save
2022-06-09 13:37:28 +01:00
Alex Sharov
19086759eb
methods to configure db #483 2022-06-09 09:43:12 +07:00
ledgerwatch
157b4299e4
[erigon2] Continuation on domains and inverted indices, putting things together (#476)
* Add scan files tests, create new aggregator type

* Fix lint

* windows test fix

* Add delelte test

* AggCollation

* More functions to Aggregator

* More aggregator functions

* Update

* More functions

* More functions

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2022-06-02 21:40:58 +01:00
ledgerwatch
37d9944da9
[erigon2] State domains (move functionality out of aggregator) (#436)
* Domain

* First functions

* change year

* More on domain

* More to test

* More on test

* More on domains

* buildFiles

* More on domains

* Collation test

* Fix collate

* Add test for decompressors

* Restructure history tables

* Split history into 2 tables

* Fix lint

* Check index files in the test

* Close files

* Add file scanning

* Fix lint

* Fix lint

* Add readFromFiles

* Add ef history idx file

* Start cleanup

* More to cleanup, test for ef history

* More test

* Add prune to test

* Test for prune and fix

* Start history access

* History test

* Test for LastDup

* Fix one lint

* Workaround

* History tests

* Debug

* Fix

* Fix in history

* Fix lint

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@alexs-macbook-pro.home>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
Co-authored-by: Alex Sharp <alexsharp@alexs-mbp.lan>
2022-05-24 18:59:57 +01:00
Alex Sharov
9821dcdff5
kv_mdbx: atomic closed flag (#464) 2022-05-24 12:20:19 +07:00
Alex Sharov
4e257396f4
Kv: GetBool helper (#445)
* save

* save
2022-05-10 14:51:32 +07:00
Alex Sharov
9f5b2b6391
don't log inside library about db close #442 2022-05-10 08:55:32 +07:00
Alex Sharov
3c52757526
Downloader tables (#422)
* save

* save

* save
2022-04-21 10:43:46 +07:00
Alex Sharov
1ac7357005
KV: ReadAhead helper #419 2022-04-18 09:37:01 +07:00
Andrew Ashikhmin
8c89288cbc
Add LastForkchoice table (#405)
* Deprecate TransitionBlock

* Add ForkchoiceHead/Safe/Finalized tables

* Merge Forkchoice* into LastForkchoice
2022-03-31 10:51:23 +02:00
Alex Sharov
54629aaf7c
Mdbx fix after v0.11.6 (#398)
* save

* save
2022-03-27 07:23:35 +01:00
ledgerwatch
4e8d577d1d
[erigon2] Switch back to MDBX (#388)
* Switch back to MDBX

* Fix test

* No restore

* Fix overwrite

* Fix change file opening

* Verify state vs change files

* Add WriteMap

* Print

* Not to go to the end of the file when reading

* Fix rebuild

* prefixLen

* Print

* Print

* Print

* Fix for rebuild

* Remove prints

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2022-03-23 14:35:13 +00:00
Alex Sharov
83951a1d62
Enable more linters (#381) 2022-03-19 11:38:37 +07:00
Alex Sharov
2193d570a6
Mdbx: WriteMap fallback on open error (#369)
* save

* save
2022-03-16 11:59:11 +00:00
battlmonstr
3123b6d895
kv: add NodeRecords nodedb table (#359) 2022-03-10 13:15:15 +01:00
battlmonstr
f54e4c7eae
kv: use SetSyncPeriod from mdbx-go v0.22.16 (#358) 2022-03-10 13:04:08 +01:00
Alex Sharov
6512e3c941
add emptyWordsCount field to .seg file header (breaking .seg format) (#355)
* up torrent

* save

* save

* save

* save

* save

* save

* save
2022-03-10 07:48:37 +00:00
battlmonstr
009358ab2f
kv: expose mdbx SafeNoSync and OptSyncPeriod options (#356) 2022-03-10 10:25:11 +07:00
Alex Sharov
a2cbecf373
[wip] KV: limit amount of parallel read txs (#352)
* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* a

* a

* a

* a

* save

* save

* save

* save

* save
2022-03-07 12:21:59 +00:00
Alex Sharov
56d528b1f0
save (#353) 2022-03-04 09:13:15 +07:00
Alex Sharov
5be29adf1a
Grpc err utils (#334)
* save

* save
2022-02-18 09:40:11 +07:00
alex.sharov
a2ca4cf55f save 2022-02-17 10:10:23 +07:00
Alex Sharov
7aec036bdd
remotedb server: ro db (#313) 2022-02-11 16:02:57 +07:00
Alex Sharov
8e4618c57d
state and newHead: 1 goroutine per subscriber #309 2022-02-10 08:26:08 +07:00
alex.sharov
d814adb13e save 2022-02-03 16:28:43 +07:00
alex.sharov
02c407a14e save 2022-02-03 16:28:32 +07:00
Enrique Jose Avila Asapche
a5c2d83108
Added bor tables (#286)
Co-authored-by: Krishna Upadhyaya <krishnau1604@gmail.com>
2022-01-28 21:34:03 +00:00
ledgerwatch
df95313a31
Add blockGasLimit to the state stream (#261)
* Remove interfaces for replacement

* Squashed 'interfaces/' content from commit eff9f8f

git-subtree-dir: interfaces
git-subtree-split: eff9f8f7273dda7c78575f2474e3103da61825db

* Add blockGasLimit to the state stream

* Pass blockGasLimit into txpool

* Add NotTooMuchGas bit, refactored promote

* Replace effectiveTip with minTip in the queues and slices

* Fix lint

* Refactor

* Improve terminology of ordering function

* Optimise baseFee change

* cleanup

* Safe operations

* Fix UnsafeAdd

* also print pendingBaseFee

* trace txpool_content

* trace txpool_content

* user sender ID inverted index to speed up txpool_content

* copy rlp from Db

* remove debug printing

* change from string to []byte

* safe storage of sender in the map

* Fix compile

* safe storage of sender in the map

* Print errors

* Prints

* Prints

* Prints

* Change worst function

* Change worst function

* Change worst function

* Change worst function

* Fix minTip

* Fix minTip

* Warn on db error, but not stop

* Fix interface

* More prints

* Cleanup

* Cleanup

* Differentiate baseFee pool

* Bump version of remove KV interface

* Not print every block

* Not print every block

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-01-25 10:20:11 +00:00
Alex Sharov
1c92f7bc00
mdbx: consensus label #265 2022-01-22 12:49:25 +07:00
alex.sharov
ec2a482087 allow configure pagesize 2022-01-22 12:46:22 +07:00
Alex Sharov
bcef5f091c
allow configure pagesize (#264) 2022-01-22 12:13:35 +07:00
Alex Sharov
79eb27d3f1
Helper to prohibit cli flag changes (#262) 2022-01-22 10:48:22 +07:00
Alex Sharov
ac8c29cacf
kv helpers (#255) 2022-01-20 15:31:01 +07:00
Giulio rebuffo
ac26beb190
added cumulative index tables (#253) 2022-01-20 08:25:03 +00:00
Alex Sharov
0f80e9941f
Switch to parallel compressor (#244) 2022-01-18 12:55:20 +07:00
Alex Sharov
5042e16218
enforce exactly once semantic in BigChunks helper 2022-01-17 16:49:02 +07:00
alex.sharov
3e856aba62 format docs 2022-01-13 11:08:05 +07:00
ledgerwatch
ae6f7202d2
[erigon2] Introduce commitment files and table (#201)
* Introduce commitment files and table

* Introduce commitment files and table

* Introduce words buffer

* Move changes out of aggregator into the writer

* Start calc commitment

* Add commitment package

* More on hex patricia

* More

* account decorator

* More

* Add test

* More

* More to the test

* Use hex in tests

* More

* More

* More

* More

* Simplified cells

* Add delBitmap

* More advanced MockState

* More compact unfolding

* Separation of hashed Keys

* Sepatate downHashedKey and upHashedKey

* Carry extension node through accounts

* optimised row allocations

* Fix encoding/decoding, add trace flag

* Added account and storage into the cells

* Make accountKeyLen global setting

* Remove BranchNodeUpdaqte intermediate

* Started on computeHash

* Initial hash calculations

* Fix lint

* Change account encoding

* Fix commitment issues

* Fix lint

* Fix lint

* Fix lint

* Trace

* Small fix and tracing

* Print branch hashes, fix empty

* Fix

* Fix

* Fix

* Fix

* Print accountFn

* Trigger accountFn

* Trigger accountFn

* Another fix for extension nodes

* Return root hash and set trace

* Print depth for computeCellHash

* Fix for storage leaves

* Fix for storage leaves

* Fix for storage leaves

* Fix for storage leaves

* Fix for storage leaves

* Fix for storage leaves

* Ineffectual deletes

* Trace needUnfolding

* Fix

* Fix

* Fix

* trace deletes

* trace deletes

* trace deletes

* trace deletes

* trace deletes

* not overwrite downHashedKey in updateAccount

* simplify needUnfolding

* simplify needUnfolding

* simplify needUnfolding

* simplify needUnfolding

* simplify needUnfolding

* propagate deletes when unfolding

* remove deletes when folding

* remove deletes when folding

* remove upHashedKey when delete cell

* remove upHashedKey when delete cell

* aggregate storage changes later

* remove del bit when updating account

* remove del bit when updating account

* remove del bit when updating account

* perform delete after code update

* perform delete after code update

* perform delete after code update

* perform delete after code update

* perform delete after code update

* perform delete after code update

* latest update has precedence when aggregating

* Revert "latest update has precedence when aggregating"

This reverts commit 0234ea6c3e1a0b2e7599df7c470d4b8b8d56736a.

* latest update has precedence when aggregating

* Revert "latest update has precedence when aggregating"

This reverts commit cfa7b75327e4d5d1236f905c3c04eb640b21838d.

* Introduce FinishTx

* Separate commitment separation from computation

* Prevent spurious delete

* Spurious deletes

* Trace param

* Preserve upHashedKey when unfolding

* Fix

* Fix

* Correctly compute firstInsert

* Try to fix delete + fold

* Remove commented out

* updateStorage to remove deleted flag

* rename upHashedKey to extension, fix clear-up

* Carry extension up if account plain key is present

* Carry extension up if account plain key is present

* Not to fail for deleteAccount

* Not to fail for deleteAccount

* Not to fail for deleteAccount

* Print

* Print

* Print

* Print

* Print

* Print

* Print

* Print

* Copy keys before putting them into commTree

* Preserve changes to AccountData

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-01-12 11:21:52 +00:00
Alex Sharov
1647faec37
Fix bigChunk helper (#229) 2022-01-12 10:46:26 +07:00
Alex Sharov
e98482670a
mdbx: fix gc "retry" issue (slowness of gc during commit) (#227)
* mdbx: fix gc "retry" issue (slowness of gc during commit)

* mdbx: fix gc "retry" issue (slowness of gc during commit)
2022-01-11 08:12:20 +00:00
Alex Sharov
2fe3f06702
Decompressor internal file path getter (#220) 2021-12-31 18:42:43 +07:00
alex.sharov
dd363aea75 Revert "save"
This reverts commit ef20b06b01.
2021-12-27 15:11:24 +07:00
alex.sharov
ef20b06b01 save 2021-12-27 15:11:00 +07:00
Alex Sharov
53c160deb1
save (#197) 2021-11-30 07:06:57 +00:00
brendan-kelly
057925ba08
Add ParliaSnapshot table to database for Binance Smart Chain support (#190)
* Add ParliaSnapshot table to database for Binance Smart Chain support

* Add schema description for ParliaSnapshot table
2021-11-25 06:48:55 +00:00
Giulio rebuffo
4c973f1ee4
Added CurrentExecutionPayload Bucket (#178)
* Added CurrentExecutionPayload Bucked

* lint
2021-11-20 09:40:26 +07:00
Alex Sharov
a4ee647902
Recsplit: move files read/write helpers to erigon-lib (#176) 2021-11-19 12:03:36 +07:00
Alex Sharov
3256008c58
Canonical tx ids (#174)
* save

* save

* save
2021-11-18 12:02:03 +00:00
Giulio rebuffo
71603ba29a
TransitionKey added for proof-of-stake transition (#170)
* transition data

* Update tables.go
2021-11-17 00:17:13 +01:00
Alex Sharov
921a20df2b
mdbx: set back big augment limit (#157) 2021-11-14 12:20:49 +07:00
ledgerwatch
fd19ad8148
State aggregator (#114)
* State aggregator

* Compile fix

* More

* Add

* More

* More on aggregator

* Writes (still incorrect)

* Move table names

* More

* Start of aggregation

* Change files instead of db

* More on change files

* More

* More

* Dealing with state and change files

* More

* More

* More boilerplate

* More

* More

* Iteration over storage

* More boilerplate

* More fixes

* Insert flag

* More

* Unit test

* Add more to the test

* Expand the test a bit

* More testing

* Keep fixing the test

* More fixes to the test

* Clean up DB tables upon aggregation

* More fixes

* Remove update/insert indicator from returned values

* Add assertions

* close files before deleting

* close files before deleting

* close files before deleting

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2021-11-13 12:12:29 +00:00
alex.sharov
9b7e0459cf save 2021-10-28 21:52:22 +07:00
Alex Sharov
5d84a53060
Pool2 integration test support (#136) 2021-10-28 18:45:18 +07:00
Alex Sharov
b11097448f
Mdbx reduce augment limit to 8*default (#134) 2021-10-27 13:17:11 +07:00
Alex Sharov
585354391b
force some rw tx when db open, to allow mdbx up version signature (#125) 2021-10-22 10:13:11 +07:00
Alex Sharov
ec0155d4ed
Mdbx v0.11.0 (#123) 2021-10-22 09:12:39 +07:00
alex.sharov
9eae3ce63a add codeEvict list metric 2021-10-12 15:59:35 +07:00
alex.sharov
f066f52b04 add codeEvict list metric 2021-10-12 15:16:25 +07:00
Alex Sharov
a745f2391c
Remove debug lines from cache (#109) 2021-10-12 11:17:33 +07:00
Alex Sharov
6a77e30374
Code cache (#108) 2021-10-11 21:25:22 +07:00
Alex Sharov
e4ae867351
,save (#101) 2021-10-06 15:14:58 +07:00
Alex Sharov
04d7bfe3ce
Fix race in cache (#100) 2021-10-05 16:06:22 +07:00
Alex Sharov
cc20173b74
MDBX: Don't block when open db from another process if main process holds long RW tx (#99) 2021-10-05 15:30:06 +07:00
Alex Sharov
33aa4498f0
Pool: speedup iteration over all txs (#94) 2021-09-28 15:54:02 +07:00
Alex Sharov
ffac34a86f
Pool: break data-dependency between effectiveTip and other fields (#93) 2021-09-27 19:45:10 +07:00
Alex Sharov
53afdc9b69
--mdbx.augment.limit (#92) 2021-09-27 09:49:17 +07:00
Alex Sharov
f5fa03d63f
Pool: switch to PendingBaseFee (#90) 2021-09-26 19:47:11 +07:00
Alexandr Borodulin
7740382188
Update tables.go (#86)
Co-authored-by: Aleksandr Borodulin <a.borodulin@axioma.lv>
2021-09-22 17:34:29 +07:00
Alex Sharov
13b0978d86
grpc enable instrumentation (#84) 2021-09-21 10:10:59 +07:00
Alex Sharov
6a7eeccad7
Pool: discard reasons, LRU (#80) 2021-09-20 12:44:29 +07:00
Alex Sharov
1c9602c785
revert 2421 2021-09-20 10:41:02 +07:00
Alex Sharov
dfc14352d2
Pool: chain config, non-mainnet (#78) 2021-09-18 20:58:20 +07:00
Alexandr Borodulin
d32bc94cf8
Update tables.go (#76)
Co-authored-by: Aleksandr Borodulin <a.borodulin@axioma.lv>
2021-09-17 16:28:59 +07:00
Alex Sharov
8163e0cd93
Pool: cache based on txID (#75) 2021-09-17 09:56:04 +07:00
Alexandr Borodulin
1f3720472a
fixes ledgerwatch/erigon#2421 (#72)
* fixes ledgerwatch/erigon#2421

* Update tables.go

Co-authored-by: Aleksandr Borodulin <a.borodulin@axioma.lv>
Co-authored-by: ledgerwatch <akhounov@gmail.com>
2021-09-15 17:08:22 +01:00
Alex Sharov
71308b02b3
Pool: switch cache to tx.ID(), more metrics (#73) 2021-09-15 20:15:32 +07:00
Alex Sharov
aa7acc120c
Pool: expose mdbx's txID to remote_kv (#71) 2021-09-15 13:11:34 +07:00
Alex Sharov
645eb83eca
Pool: mining rpc, remove block num from db, started atomic, more metrics, basic eviction (#70) 2021-09-14 22:01:33 +07:00
Alex Sharov
5b8e019246
Pool: add coherent state cache (#68) 2021-09-13 14:31:15 +07:00
Alex Sharov
fddc5719b3
Coherent cache: basic implementation (#64) 2021-09-10 14:18:00 +07:00
Alex Sharov
c506d17f52
save (#63) 2021-09-09 10:43:41 +07:00
Alex Sharov
0b4e528fac
Pool: no state persistance (#62) 2021-09-08 19:21:13 +07:00
Alex Sharov
c7739b89a8
Pool: state diff direct client, tx parse fuzzing (#59) 2021-09-07 09:52:17 +07:00
Alex Sharov
b1435d3679
don't start tx on shutdown (#56) 2021-09-02 16:33:23 +07:00
Alex Sharov
3b7a086a68
Dont start remote tx on shutdown (#55)
* don't start tx on shutdown

* don't start tx on shutdown
2021-09-02 15:41:29 +07:00
Alex Sharov
38a9bb9d7f
Pool: remove redundant prefix in txs table (#49) 2021-08-31 21:29:55 +07:00
Alex Sharov
29bf077da4
Pool: eviction works, use correct minBaseFee (#48) 2021-08-30 19:26:29 +07:00
alex.sharov
e0b3248964 save 2021-08-30 10:07:16 +07:00
alex.sharov
9ac04671d0 add state check assert routine 2021-08-28 10:17:49 +07:00