Commit Graph

141 Commits

Author SHA1 Message Date
Alex Sharov
a2cbecf373
[wip] KV: limit amount of parallel read txs (#352)
* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* a

* a

* a

* a

* save

* save

* save

* save

* save
2022-03-07 12:21:59 +00:00
Alex Sharov
56d528b1f0
save (#353) 2022-03-04 09:13:15 +07:00
Alex Sharov
5be29adf1a
Grpc err utils (#334)
* save

* save
2022-02-18 09:40:11 +07:00
alex.sharov
a2ca4cf55f save 2022-02-17 10:10:23 +07:00
Alex Sharov
7aec036bdd
remotedb server: ro db (#313) 2022-02-11 16:02:57 +07:00
Alex Sharov
8e4618c57d
state and newHead: 1 goroutine per subscriber #309 2022-02-10 08:26:08 +07:00
alex.sharov
d814adb13e save 2022-02-03 16:28:43 +07:00
alex.sharov
02c407a14e save 2022-02-03 16:28:32 +07:00
Enrique Jose Avila Asapche
a5c2d83108
Added bor tables (#286)
Co-authored-by: Krishna Upadhyaya <krishnau1604@gmail.com>
2022-01-28 21:34:03 +00:00
ledgerwatch
df95313a31
Add blockGasLimit to the state stream (#261)
* Remove interfaces for replacement

* Squashed 'interfaces/' content from commit eff9f8f

git-subtree-dir: interfaces
git-subtree-split: eff9f8f7273dda7c78575f2474e3103da61825db

* Add blockGasLimit to the state stream

* Pass blockGasLimit into txpool

* Add NotTooMuchGas bit, refactored promote

* Replace effectiveTip with minTip in the queues and slices

* Fix lint

* Refactor

* Improve terminology of ordering function

* Optimise baseFee change

* cleanup

* Safe operations

* Fix UnsafeAdd

* also print pendingBaseFee

* trace txpool_content

* trace txpool_content

* user sender ID inverted index to speed up txpool_content

* copy rlp from Db

* remove debug printing

* change from string to []byte

* safe storage of sender in the map

* Fix compile

* safe storage of sender in the map

* Print errors

* Prints

* Prints

* Prints

* Change worst function

* Change worst function

* Change worst function

* Change worst function

* Fix minTip

* Fix minTip

* Warn on db error, but not stop

* Fix interface

* More prints

* Cleanup

* Cleanup

* Differentiate baseFee pool

* Bump version of remove KV interface

* Not print every block

* Not print every block

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-01-25 10:20:11 +00:00
Alex Sharov
1c92f7bc00
mdbx: consensus label #265 2022-01-22 12:49:25 +07:00
alex.sharov
ec2a482087 allow configure pagesize 2022-01-22 12:46:22 +07:00
Alex Sharov
bcef5f091c
allow configure pagesize (#264) 2022-01-22 12:13:35 +07:00
Alex Sharov
79eb27d3f1
Helper to prohibit cli flag changes (#262) 2022-01-22 10:48:22 +07:00
Alex Sharov
ac8c29cacf
kv helpers (#255) 2022-01-20 15:31:01 +07:00
Giulio rebuffo
ac26beb190
added cumulative index tables (#253) 2022-01-20 08:25:03 +00:00
Alex Sharov
0f80e9941f
Switch to parallel compressor (#244) 2022-01-18 12:55:20 +07:00
Alex Sharov
5042e16218
enforce exactly once semantic in BigChunks helper 2022-01-17 16:49:02 +07:00
alex.sharov
3e856aba62 format docs 2022-01-13 11:08:05 +07:00
ledgerwatch
ae6f7202d2
[erigon2] Introduce commitment files and table (#201)
* Introduce commitment files and table

* Introduce commitment files and table

* Introduce words buffer

* Move changes out of aggregator into the writer

* Start calc commitment

* Add commitment package

* More on hex patricia

* More

* account decorator

* More

* Add test

* More

* More to the test

* Use hex in tests

* More

* More

* More

* More

* Simplified cells

* Add delBitmap

* More advanced MockState

* More compact unfolding

* Separation of hashed Keys

* Sepatate downHashedKey and upHashedKey

* Carry extension node through accounts

* optimised row allocations

* Fix encoding/decoding, add trace flag

* Added account and storage into the cells

* Make accountKeyLen global setting

* Remove BranchNodeUpdaqte intermediate

* Started on computeHash

* Initial hash calculations

* Fix lint

* Change account encoding

* Fix commitment issues

* Fix lint

* Fix lint

* Fix lint

* Trace

* Small fix and tracing

* Print branch hashes, fix empty

* Fix

* Fix

* Fix

* Fix

* Print accountFn

* Trigger accountFn

* Trigger accountFn

* Another fix for extension nodes

* Return root hash and set trace

* Print depth for computeCellHash

* Fix for storage leaves

* Fix for storage leaves

* Fix for storage leaves

* Fix for storage leaves

* Fix for storage leaves

* Fix for storage leaves

* Ineffectual deletes

* Trace needUnfolding

* Fix

* Fix

* Fix

* trace deletes

* trace deletes

* trace deletes

* trace deletes

* trace deletes

* not overwrite downHashedKey in updateAccount

* simplify needUnfolding

* simplify needUnfolding

* simplify needUnfolding

* simplify needUnfolding

* simplify needUnfolding

* propagate deletes when unfolding

* remove deletes when folding

* remove deletes when folding

* remove upHashedKey when delete cell

* remove upHashedKey when delete cell

* aggregate storage changes later

* remove del bit when updating account

* remove del bit when updating account

* remove del bit when updating account

* perform delete after code update

* perform delete after code update

* perform delete after code update

* perform delete after code update

* perform delete after code update

* perform delete after code update

* latest update has precedence when aggregating

* Revert "latest update has precedence when aggregating"

This reverts commit 0234ea6c3e1a0b2e7599df7c470d4b8b8d56736a.

* latest update has precedence when aggregating

* Revert "latest update has precedence when aggregating"

This reverts commit cfa7b75327e4d5d1236f905c3c04eb640b21838d.

* Introduce FinishTx

* Separate commitment separation from computation

* Prevent spurious delete

* Spurious deletes

* Trace param

* Preserve upHashedKey when unfolding

* Fix

* Fix

* Correctly compute firstInsert

* Try to fix delete + fold

* Remove commented out

* updateStorage to remove deleted flag

* rename upHashedKey to extension, fix clear-up

* Carry extension up if account plain key is present

* Carry extension up if account plain key is present

* Not to fail for deleteAccount

* Not to fail for deleteAccount

* Not to fail for deleteAccount

* Print

* Print

* Print

* Print

* Print

* Print

* Print

* Print

* Copy keys before putting them into commTree

* Preserve changes to AccountData

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-01-12 11:21:52 +00:00
Alex Sharov
1647faec37
Fix bigChunk helper (#229) 2022-01-12 10:46:26 +07:00
Alex Sharov
e98482670a
mdbx: fix gc "retry" issue (slowness of gc during commit) (#227)
* mdbx: fix gc "retry" issue (slowness of gc during commit)

* mdbx: fix gc "retry" issue (slowness of gc during commit)
2022-01-11 08:12:20 +00:00
Alex Sharov
2fe3f06702
Decompressor internal file path getter (#220) 2021-12-31 18:42:43 +07:00
alex.sharov
dd363aea75 Revert "save"
This reverts commit ef20b06b01.
2021-12-27 15:11:24 +07:00
alex.sharov
ef20b06b01 save 2021-12-27 15:11:00 +07:00
Alex Sharov
53c160deb1
save (#197) 2021-11-30 07:06:57 +00:00
brendan-kelly
057925ba08
Add ParliaSnapshot table to database for Binance Smart Chain support (#190)
* Add ParliaSnapshot table to database for Binance Smart Chain support

* Add schema description for ParliaSnapshot table
2021-11-25 06:48:55 +00:00
Giulio rebuffo
4c973f1ee4
Added CurrentExecutionPayload Bucket (#178)
* Added CurrentExecutionPayload Bucked

* lint
2021-11-20 09:40:26 +07:00
Alex Sharov
a4ee647902
Recsplit: move files read/write helpers to erigon-lib (#176) 2021-11-19 12:03:36 +07:00
Alex Sharov
3256008c58
Canonical tx ids (#174)
* save

* save

* save
2021-11-18 12:02:03 +00:00
Giulio rebuffo
71603ba29a
TransitionKey added for proof-of-stake transition (#170)
* transition data

* Update tables.go
2021-11-17 00:17:13 +01:00
Alex Sharov
921a20df2b
mdbx: set back big augment limit (#157) 2021-11-14 12:20:49 +07:00
ledgerwatch
fd19ad8148
State aggregator (#114)
* State aggregator

* Compile fix

* More

* Add

* More

* More on aggregator

* Writes (still incorrect)

* Move table names

* More

* Start of aggregation

* Change files instead of db

* More on change files

* More

* More

* Dealing with state and change files

* More

* More

* More boilerplate

* More

* More

* Iteration over storage

* More boilerplate

* More fixes

* Insert flag

* More

* Unit test

* Add more to the test

* Expand the test a bit

* More testing

* Keep fixing the test

* More fixes to the test

* Clean up DB tables upon aggregation

* More fixes

* Remove update/insert indicator from returned values

* Add assertions

* close files before deleting

* close files before deleting

* close files before deleting

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2021-11-13 12:12:29 +00:00
alex.sharov
9b7e0459cf save 2021-10-28 21:52:22 +07:00
Alex Sharov
5d84a53060
Pool2 integration test support (#136) 2021-10-28 18:45:18 +07:00
Alex Sharov
b11097448f
Mdbx reduce augment limit to 8*default (#134) 2021-10-27 13:17:11 +07:00
Alex Sharov
585354391b
force some rw tx when db open, to allow mdbx up version signature (#125) 2021-10-22 10:13:11 +07:00
Alex Sharov
ec0155d4ed
Mdbx v0.11.0 (#123) 2021-10-22 09:12:39 +07:00
alex.sharov
9eae3ce63a add codeEvict list metric 2021-10-12 15:59:35 +07:00
alex.sharov
f066f52b04 add codeEvict list metric 2021-10-12 15:16:25 +07:00
Alex Sharov
a745f2391c
Remove debug lines from cache (#109) 2021-10-12 11:17:33 +07:00
Alex Sharov
6a77e30374
Code cache (#108) 2021-10-11 21:25:22 +07:00
Alex Sharov
e4ae867351
,save (#101) 2021-10-06 15:14:58 +07:00
Alex Sharov
04d7bfe3ce
Fix race in cache (#100) 2021-10-05 16:06:22 +07:00
Alex Sharov
cc20173b74
MDBX: Don't block when open db from another process if main process holds long RW tx (#99) 2021-10-05 15:30:06 +07:00
Alex Sharov
33aa4498f0
Pool: speedup iteration over all txs (#94) 2021-09-28 15:54:02 +07:00
Alex Sharov
ffac34a86f
Pool: break data-dependency between effectiveTip and other fields (#93) 2021-09-27 19:45:10 +07:00
Alex Sharov
53afdc9b69
--mdbx.augment.limit (#92) 2021-09-27 09:49:17 +07:00
Alex Sharov
f5fa03d63f
Pool: switch to PendingBaseFee (#90) 2021-09-26 19:47:11 +07:00
Alexandr Borodulin
7740382188
Update tables.go (#86)
Co-authored-by: Aleksandr Borodulin <a.borodulin@axioma.lv>
2021-09-22 17:34:29 +07:00
Alex Sharov
13b0978d86
grpc enable instrumentation (#84) 2021-09-21 10:10:59 +07:00
Alex Sharov
6a7eeccad7
Pool: discard reasons, LRU (#80) 2021-09-20 12:44:29 +07:00
Alex Sharov
1c9602c785
revert 2421 2021-09-20 10:41:02 +07:00
Alex Sharov
dfc14352d2
Pool: chain config, non-mainnet (#78) 2021-09-18 20:58:20 +07:00
Alexandr Borodulin
d32bc94cf8
Update tables.go (#76)
Co-authored-by: Aleksandr Borodulin <a.borodulin@axioma.lv>
2021-09-17 16:28:59 +07:00
Alex Sharov
8163e0cd93
Pool: cache based on txID (#75) 2021-09-17 09:56:04 +07:00
Alexandr Borodulin
1f3720472a
fixes ledgerwatch/erigon#2421 (#72)
* fixes ledgerwatch/erigon#2421

* Update tables.go

Co-authored-by: Aleksandr Borodulin <a.borodulin@axioma.lv>
Co-authored-by: ledgerwatch <akhounov@gmail.com>
2021-09-15 17:08:22 +01:00
Alex Sharov
71308b02b3
Pool: switch cache to tx.ID(), more metrics (#73) 2021-09-15 20:15:32 +07:00
Alex Sharov
aa7acc120c
Pool: expose mdbx's txID to remote_kv (#71) 2021-09-15 13:11:34 +07:00
Alex Sharov
645eb83eca
Pool: mining rpc, remove block num from db, started atomic, more metrics, basic eviction (#70) 2021-09-14 22:01:33 +07:00
Alex Sharov
5b8e019246
Pool: add coherent state cache (#68) 2021-09-13 14:31:15 +07:00
Alex Sharov
fddc5719b3
Coherent cache: basic implementation (#64) 2021-09-10 14:18:00 +07:00
Alex Sharov
c506d17f52
save (#63) 2021-09-09 10:43:41 +07:00
Alex Sharov
0b4e528fac
Pool: no state persistance (#62) 2021-09-08 19:21:13 +07:00
Alex Sharov
c7739b89a8
Pool: state diff direct client, tx parse fuzzing (#59) 2021-09-07 09:52:17 +07:00
Alex Sharov
b1435d3679
don't start tx on shutdown (#56) 2021-09-02 16:33:23 +07:00
Alex Sharov
3b7a086a68
Dont start remote tx on shutdown (#55)
* don't start tx on shutdown

* don't start tx on shutdown
2021-09-02 15:41:29 +07:00
Alex Sharov
38a9bb9d7f
Pool: remove redundant prefix in txs table (#49) 2021-08-31 21:29:55 +07:00
Alex Sharov
29bf077da4
Pool: eviction works, use correct minBaseFee (#48) 2021-08-30 19:26:29 +07:00
alex.sharov
e0b3248964 save 2021-08-30 10:07:16 +07:00
alex.sharov
9ac04671d0 add state check assert routine 2021-08-28 10:17:49 +07:00
alex.sharov
d4cf078cfc add state check assert routine 2021-08-27 15:28:23 +07:00
Willian Mitsuda
ed0dabb416 Fix comment for issuance bucket 2021-08-27 01:52:47 -03:00
Willian Mitsuda
f5c06a9add Add issuance bucket 2021-08-26 15:57:20 -03:00
alex.sharov
334df676b2 persistence 2021-08-25 19:44:21 +07:00
alex.sharov
07df05a767 persistence 2021-08-22 17:39:47 +07:00
alex.sharov
d3f68af522 persistence 2021-08-22 17:06:38 +07:00
alex.sharov
0d2b0c494f dbg 2021-08-20 16:17:34 +07:00
alex.sharov
20e2b5e744 close db label log 2021-08-18 15:06:11 +07:00
alex.sharov
de4c298281 add txpool tables 2021-08-17 10:10:54 +07:00
alex.sharov
60024e7108 remove grpc from remote db 2021-08-15 11:59:25 +07:00
alex.sharov
d8973e89cc remove grpc from remote db 2021-08-15 11:58:07 +07:00
alex.sharov
ad711366fe save isLocalHistory to db 2021-08-13 12:23:14 +07:00
Alex Sharov
9d8ca5579b
Merge pull request #9 from ledgerwatch/fix_kv_metrics
kv metrics
2021-07-29 17:41:03 +07:00
alex.sharov
5f20dea6b6 kv metrics 2021-07-29 17:33:03 +07:00
alex.sharov
41bc91f454 fuzz example 2021-07-29 16:30:06 +07:00
alex.sharov
0f965734bb log 2021-07-28 13:31:35 +07:00
alex.sharov
ee1d85bc82 move to log15 2021-07-28 11:50:45 +07:00
alex.sharov
89f7ccfa74 move mdbx and kv 2021-07-28 10:43:36 +07:00
alex.sharov
81bf1f09b3 move mdbx and kv 2021-07-28 10:37:25 +07:00
alex.sharov
8ddf4d68b1 move mdbx and kv 2021-07-28 10:36:06 +07:00