Commit Graph

388 Commits

Author SHA1 Message Date
Alex Sharov
da31494cc3
transactional migrations (#1079) 2020-09-08 20:39:43 +01:00
Alex Sharov
c45a710ce6
Warmup logs, less overhead, warmup code bucket (#1054)
* warmup logs and less overhead.

* warmup logs and less overhead.

* move WarmUp to common func
2020-09-08 08:28:37 +01:00
b00ris
23e995838f
Fix readonly mode (#1077)
* fix readonly mode

* fix vet
2020-09-08 07:19:06 +01:00
Alex Sharov
afe808c88e
time based reopen tx (#1069) 2020-09-07 12:47:08 +07:00
ledgerwatch
3a92b2b39d
Fix for RPC daemon leak (#1059)
* Start memory prof

* Fix rpctest

* Fix rpctest

* Attempt to fix the leak

* Remove http pprof
2020-09-05 21:58:51 +01:00
Alex Sharov
e690103c2a
Docs for Cursors and KV (#1046) 2020-09-05 14:26:24 +07:00
Alex Sharov
7e71c4512e
restrict metric by 1 bucket (#1052) 2020-09-05 13:09:47 +07:00
Alex Sharov
95e985ef8f
add commit event to dashboard (#1050) 2020-09-05 10:10:17 +07:00
Alex Sharov
0c75304261
Os page cache monitoring (#1049) 2020-09-05 09:36:42 +07:00
Alex Sharov
b9bdd003a7
KV: Raw (no custom logic) cursors for DupSort and DupFixed buckets (#1020) 2020-09-04 10:54:15 +07:00
Evgeny Danilenko
e4f495fa44
Get logs (#1028)
* it compiles

* after recent master

* fix linters warnings

* grpcV7

* go mod tidy

* unmarshall adresses or adress

* fix linters

* after cr

* after cr

* after cr

* after cr

* fix tests

* remove dev version

* it compiles

* mod tidy

* fix bin deps

* use stable version of grpc

* switch back to master constructor

* switch back to master constructor

* add a bit docs

* add a bit docs

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
2020-09-03 08:51:19 +01:00
Alex Sharov
e02f2f6dfc
Revert "simplify cursor.Next and cursor.First implementation (#1019)" (#1039)
This reverts commit f19bb0a345.
2020-09-03 13:21:11 +07:00
Alex Sharov
f19bb0a345
simplify cursor.Next and cursor.First implementation (#1019) 2020-09-01 12:50:23 +01:00
ledgerwatch
c06dbdadf9
Use Append for changesets and receipts (#1018)
* Use Append for changesets and receipts

* Fix lint

* Delete all receipts on unwind

* Fix lint

* Fix unwind keys for changeset

* Fix test

* Use writeReceipts flag

* Revert to Append

* Delete receipts too when resetting
2020-09-01 07:48:25 +01:00
Alex Sharov
49351b0091
KV: to support multiple databases and multiple set of buckets (#1002)
* don't use global variables

* up deps

* clean

* use joined flags

* use single non-global buckets config

* clean

* copy map

* fix tests

* fix tests

* clean

* clean
2020-08-30 18:34:18 +01:00
Alex Sharov
5ce73d438a
fix integration (#999) 2020-08-29 15:27:20 +07:00
Alex Sharov
07ca5c643a
eth_syncing (#991)
* eth_syncing

* linters

* fix readme

* cleanup cli params

* go mod tidy

* remove memsize ui dep

* remove bloomfilter

* implement net_peerCount

* remove legacy flags
2020-08-29 08:24:50 +01:00
Alex Sharov
df05a581dd
Add tx pool stage to transaction (#981)
* add tx_pool stage to tx

* clean

* don't call commit in read tx

* don't call commit in read tx

* clean

* more logs

* fix tx pool race

* more logs

* clean
2020-08-28 07:27:48 +01:00
Alex Sharov
714e5041e2
Better logs3 (#976) 2020-08-26 19:06:51 +07:00
Alex Sharov
9091ce62bb
KV reduce features amount (#975)
* move IdealBatchSize method to Batch object :-)
remove cursor.Walk
remove kv.IdealBatchSize

* remove bolt_db.go file

* move method NoValueCursor to tx object
2020-08-26 07:03:50 +01:00
Alex Sharov
522287ac18
Transactional cycle (#966)
* v0

* v1

* v3

* v4

* clean

* temporary fix of txpool

* Add debug logs about tx start/commit

* Add debug logs about tx start/commit

* Add debug logs about tx start/commit

* add condition

* tx pool to not hold own db

* try to enable TxPool in integration

* exclude tx pool from tx

* exclude tx pool from integration

* reduce limit

* fix integration

* clean

* clean

* clean

* clean

* clean

* exclude tx pool unwind

* exclude tx pool unwind in integration

* fix integration tx pool

* fix commit

* fix current stage after unwind

* fix current stage after unwind

* fix linter

* move unwind of tx_pool after unwind of unwind of senders, then all stages from body to tx_pool will be inside tx.

* move body and headers unwind out of tx

* fix unwind order after reboot

* add support external tx to exec stage

* clean

* clean

* clean

* clean

* clean

* add logs

* better id check

* better id check
2020-08-26 07:02:10 +01:00
ledgerwatch
8728e54a72
Another step in migrating tests to staged sync (#970)
* Log in GetAsOf

* Default plain state

* trace

* More test fixes

* Fix lint

* Fix

* Fix lint

* Fix lint

* Fix lint

* Fix lint

* Fix for events

* Repair events

* Fix lint

* Fix lint

* Change back to InsertChain
2020-08-24 18:00:40 +01:00
Alex Sharov
43a4e34a6e
[to discuss] exec blocks without ObjectDb: Mutation over TxDb (#947)
* use mutation over tx

* clean

* clear

* add .CommitAndBegin() method

* clean

* increase timings for logging

* return ideal batch size
2020-08-24 12:07:59 +01:00
Alex Sharov
a6be18b915
ticker-based logs (#954)
* timer-based logs

* timer-based logs

* delegate progress calculation to user

* delegate progress calculation to user

* delegate progress calculation to user

* clear

* add logs to senders recovery

* use default dir in integration

* more logs

* more logs
2020-08-22 12:12:33 +02:00
Alex Sharov
ebbaa55672
sub-transaction to not call fsync, sub-transactions to not call runtime.LockThread() (#945) 2020-08-21 07:31:43 +01:00
ledgerwatch
01f211f3b8
Revert "IH: no retain list (#946)" (#948)
This reverts commit f86ed64288.
2020-08-20 13:37:34 +01:00
Alex Sharov
f86ed64288
IH: no retain list (#946)
* experiment of not using retain list

* better handle sub-tx

* save progress

* don't touch collector

* don't run collector on sub-tx because of `mdb_cursor_put: MDB_TXN_FULL: Transaction has too many dirty pages - transaction too big`

* clear

* clear
2020-08-20 11:42:43 +01:00
b00ris
eb46dd68df
Account range testing (#938)
* save state

* save state

* save new timeouts

* compare passed

* save state

* cleanup
2020-08-19 07:33:49 +01:00
Evgeny Danilenko
3b40819444
grpc regenerate (#936)
* grpc regenerate

* use string type for id in net_version
2020-08-18 20:22:49 +03:00
Alex Sharov
081d02920d
Remove "Bucket" class from KV abstraction (#928)
* drop Bucket object from KV abstraction

* add some non-public methods

* rebase master

* rebase master
2020-08-17 08:24:59 +01:00
Alex Sharov
b2e35eb63f
rename "remote2" to "remote" (#932) 2020-08-17 07:47:06 +01:00
Alex Sharov
0e253e7336
lmdb transactions of unlimited size (#918)
* add logging to loader

* use pure tx in etl loading, logs in mutation commit

* clean

* bletter logging and more cleanup

* bletter logging and more cleanup

* increase batch size to 500M

* better batch commit logging

* async fsync

* sync fsync

* sync fsync

* unify logging

* fix corner-case when etl can use empty bucket name

* fix tests

* better logging

* better logging

* rebase master

* remove lmdb.NoMetaSync flag for now

* consistent walk and multi-walk

* clean

* sub tx

* add consistent multi-put

* implement dupsort support in one new cursor method

* clear
2020-08-17 07:45:52 +01:00
Igor Mandrigin
d848233275 Merge branch 'master' into geth-1.9.19
# Conflicts:
#	cmd/geth/consolecmd.go
#	cmd/rpcdaemon/service/service.go
#	cmd/utils/flags.go
#	eth/backend.go
2020-08-15 21:55:09 +02:00
Igor Mandrigin
201524c33e fix node tests 2020-08-15 20:21:30 +02:00
Alex Sharov
bf596c26f3
[merge after release] dupsort of plain state (#913)
* dupsort of plain state

* rebase master
2020-08-15 08:11:40 +01:00
Alex Sharov
e04c4ebe58
KV: move bucket.Get and bucket.Cursor methods to Tx object (#914) 2020-08-14 13:41:18 +07:00
Alex Sharov
f7deef79bb
drop bucket log message (#917) 2020-08-13 13:40:28 +07:00
Evgeny Danilenko
5d39fe8873
RPCDeamon coinbase, net_version (#911)
* use eth.backend

* net_version

* add coinbase to the console test
2020-08-12 14:47:59 +01:00
Alex Sharov
16f09be0a8
add method .Last() (#909) 2020-08-12 10:49:52 +07:00
Alex Sharov
760fe0f0b6
DupSort of hash state (#896) 2020-08-12 09:57:55 +07:00
Evgeny Danilenko
3980fa7d45
Grps eth_sendTransaction (#882)
* implementation

* tidy gomod

* linters

* fix cmd test

* fix

* fix lint
2020-08-11 22:09:30 +01:00
Alex Sharov
6de7eb1bec
correct_thread_lock_on_non_managed_tx (#905) 2020-08-11 18:53:47 +07:00
Alex Sharov
7addf34f64
fix wrong merge (#907) 2020-08-11 18:24:39 +07:00
Alex Sharov
e89ad5d6aa
sub tx support (#895) 2020-08-11 17:35:59 +07:00
Alex Sharov
695f8f8904
allow multi-rollback (#904) 2020-08-11 17:35:19 +07:00
Alex Sharov
d9d9e14f45
change bucket type to string (#894) 2020-08-11 06:55:32 +07:00
Alex Sharov
7a1b892022
remove .SeekTo() method (#901) 2020-08-11 06:41:15 +07:00
Alex Sharov
091819a51c
adopt --metrics.addr flag in integration (#889) 2020-08-11 06:38:34 +07:00
Igor Mandrigin
8600616c3d geth 1.9.18 post-rebase fixups 2020-08-08 17:33:35 +02:00
Alex Sharov
f41959f030
drop badger support (#869)
* drop badger support

* rebase master
2020-08-05 16:33:45 +01:00
Alex Sharov
d458c4cc1e
Migrations: use stage name as db key (#868) 2020-08-05 17:13:35 +07:00
Igor Mandrigin
e732ccd438 add lmdb.mapSize param 2020-08-05 10:17:50 +02:00
ledgerwatch
0514501937
Revert "Use stage name as db key (#858)" (#867)
This reverts commit 7290bf519e.
2020-08-04 11:05:27 +01:00
Alex Sharov
7290bf519e
Use stage name as db key (#858) 2020-08-04 16:25:28 +07:00
Alex Sharov
a5c1089d58
optimize .get (#863) 2020-08-04 10:31:37 +07:00
Alex Sharov
ecc94a63f0
DupSort for Plain and Hashed state buckets (behind feature flags) (#854)
* return error from rawdb

* squash

* v14

* improve performance of put

* clean

* clean

* hide feature behind ENV variable

* cleanup

* cleanup

* disable ipc and make Readme less confusing (people thought points are depend on each-other)

* fix test

* cleanup

* cleanup
2020-08-02 12:53:08 +01:00
b00ris
a84230d45c
fix streams leak (#847) 2020-08-01 16:18:31 +01:00
Alex Sharov
9a5c2a1242
reduce mmap limit size (raspbery-pi failed on 4TB) (#840) 2020-08-01 11:59:40 +07:00
Alex Sharov
6c4ec563e1
fix - change private api from ui (#829) 2020-07-30 11:17:09 +01:00
Alex Sharov
b6399a2256
better error messages (#824) 2020-07-30 13:55:40 +07:00
ledgerwatch
7bc45fab80
Remove preimage storage mode, write receipts by default (#809) 2020-07-29 14:00:39 +01:00
Alex Sharov
0a367001d1
implement db stats page (#803) 2020-07-29 05:31:46 +01:00
Alex Sharov
47b7b2cf40
lmdb_is_not_friendly_to_empty_values (#795) 2020-07-29 05:30:29 +01:00
Alex Sharov
f500148b86
remove BoltDatabase, BadgerDatabase, RemoteDatabase classes (replaced by objectDb) (#794) 2020-07-27 22:59:54 +01:00
Alex Sharov
c4a1ffa4f0
Grpc-based remoteDb (#788) 2020-07-27 19:15:48 +07:00
Alex Sharov
79a49e9e44
avoid touching closed db (#789)
* avoid touching closed db

* fix tx rollback
2020-07-27 07:10:50 +01:00
Alex Sharov
b2bcbaa895
Compact flag (#783)
* compact flag

* compact flag

* compact flag

* compact flag

* compact flag
2020-07-25 08:35:08 +01:00
Alex Sharov
7cc0b7e391
On mac lmdb can't start with error "cannot allocate memory" until I not reduced magic number (#779) 2020-07-24 14:47:03 +01:00
Alex Sharov
1a075b10f1
re-create bucket on b.Clear - to easier change bucket settings (for example by releasing new version or just by switching git branches) (#766) 2020-07-21 15:33:30 +07:00
Alex Sharov
e65b743878
Prometheus dashboard revisit (#767) 2020-07-21 15:33:03 +07:00
Alex Sharov
a7e5c4b453
Remove remote_chain package, fix rpc methods: eth_blockNumber and eth_getBlockByNumber (#769) 2020-07-21 15:19:04 +07:00
Alex Sharov
621e3a2074
Ci lmdb - reduce memory usage (#762) 2020-07-21 08:58:00 +07:00
Alex Sharov
1faa9dbcd4
add senders bucket to py (#763) 2020-07-20 16:11:47 +07:00
ledgerwatch
b765ff0e33
cmd/tester initiates staged sync, and unwind (but not reorg) (#758)
* First push

* Remove batch from Blockchain

* Fix compile error in cmd/integration

* Fix compile errors in cmd/state

* Fix compile errors

* Save changes

* Fix core tests (skip tx tests)

* More test fixes

* Fix compile error

* Fix formatting

* Fix formatting

* Fix lint

* Fixes

* Skip some miner tests

* Fix tests

* Cleanup
2020-07-19 09:11:53 +01:00
Alex Sharov
7f60d1b535
Remove load start key from etl (#757)
* limit incremental step size of stages 5 and 6

* remove limit, make incremental

* remove_load_start_key_from_etl

* switch to "struct of arrays" from "array of structs"

* switch to "struct of arrays" from "array of structs"

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-07-18 10:17:04 +01:00
Alex Sharov
03e136c7b8
remove all pools from bolt, badger, lmdb (#750) 2020-07-18 09:39:18 +01:00
Alex Sharov
c8a594dec6
Limit incremental step size of stages 5 and 6. (#754) 2020-07-17 12:52:09 +07:00
ledgerwatch
887fc4f99c
GenerateChain without using TrieDbState (#726)
* Different chain generation

* More fixes

* Fix TestSideLogRebirth

* Fix the rest of core tests

* Fix accounts/abi/bind

* Fix compile errors for cmd/pics

* Fix consensus/clique tests

* More fixes

* Fix core/state tests

* Fix eth tests

* Fix eth/downloader tests

* Fix more tests

* Fix compile errors

* Some more fixes

* Save

* Exists

* Fix last tests

* Clean up

* Fix formatting

* Fix linter

* Fix formatting

* Fix linter

* Fix lint

* Fix lint

* Fix lint

* Fix lint

* Fix lint

* Fix lint

* Fix lint

* Fix lint
2020-07-09 07:15:28 +01:00
b00ris
eca0f233ec
[WIP] WalkAsOf (#724)
* save state

* walkAsOf by cs test passed

* cs search for plain state

* save state

* fix accounts tests

* refactor walkAsOf account tests

* fix storage test

* refactor walkAsOf storage tests

* fix lint

* fix test

* save state

* save state

* add test with fixed bits

* fmt

* add stages check

* fix lint

* fix lint

* remove obsoleted methods
2020-07-09 07:13:45 +01:00
Alex Sharov
fb933bc9e3
move lmdb-go under ledgerwatch org (#720) 2020-07-07 16:18:09 +07:00
Alex Sharov
305f8aff02
Integration tests2 (#714)
```
go run ./cmd/integration reset --chaindata=...
go run ./cmd/integration state_stages -h
go run ./cmd/integration state_stages  --chaindata=... --verbosity=3 --block=2_000_000 --unwind=10 --unwind_every=1_000 
```
Also, it inherits flags from geth:
```
--pprof.cpuprofile=./cpu.out   // to file
--pprof --pprof.port=6060     // launch pprof server
--metrics                  //  sends to prometheus 
```
2020-07-07 11:00:25 +07:00
Alex Sharov
2aadd94acb
remove mgr feature and stateSize calculation feature (#711) 2020-07-05 07:17:27 +01:00
ledgerwatch
49181a298b
Special bucket for transaction senders (#707)
* Senders in the separate bucket

* Pre-create txSenders bucket

* Fix compile error

* Fix compile error

* Fix compile error, remove block transformations

* Fix formatting

* Fix test
2020-07-03 08:23:49 +01:00
Alex Sharov
a7f5b1ca08
KV: defer friendly rollback (#696)
* AbstractKV: defer friendly rollback

* AbstractKV: defer friendly rollback
2020-06-30 04:48:46 +01:00
Alex Sharov
63c998c91f
Only useful IH cursor (#694)
* cursor which returns only useful IH

* cursor which returns only useful IH
2020-06-28 13:51:18 +01:00
Alex Sharov
488a368e5b
remove "k,v,err" fields in kv_bolt, they are useful only in badger (#690) 2020-06-27 15:27:06 +07:00
Alex Sharov
26ccde4614
use lmdb.Append method (#684)
* use lmdb.Append method

* linter happiness
2020-06-24 07:54:21 +01:00
Alex Sharov
e3755a0df2
lmdb append (#678) 2020-06-19 18:19:29 +07:00
Alex Sharov
38f63f9af2
make lmdb default db (#675)
* make lmdb default db

* switch nodes db to lmdb

* log errors in nodedb

* fix miss typing

* fix default test target in makefile

* report error of .Get method

* add debug info

* run tests with debug

* run tests with debug

* remove default db from TEST_DB env

* cleanup

* wait for goroutines finish

* wait for goroutines finish before close db

* wait for goroutines finish before close db

* wait for goroutines finish before close db

* wait for goroutines finish before close db
2020-06-18 19:13:58 +01:00
Alex Sharov
e09b386fc3
lmdb support empty values, but not empty keys (#674) 2020-06-18 12:20:33 +07:00
Alex Sharov
5b051d9ab3
Lmdb: avoid empty key values in put and seek (#673)
* avoid 0 length key/values and seek arguments

* less abstractions in .get method, and copy memory from C space without unsafe conversion into go memory

* fix test
2020-06-17 10:57:48 +01:00
b00ris
e8b617bbf2
Add sort.Search to account changeset (#671)
* fast search

* fix

* remove copy bytes
2020-06-16 15:03:35 +01:00
Alex Sharov
57777e7c60
Prepare codebase for future default DB change (#670)
* Add kv.tx.bucket.Clear() and db.ClearBuckets() methods

* Add kv.tx.bucket.Clear() and db.ClearBuckets() methods

* choose db based on file suffix

* implement db.id method

* implement db.id method

* use ethdb.NewDatabase method

* use ethb.MustOpen method

* cleanup

* support TEST_DB env flag

* create db path automatically needed

* bolt - don't change prefix on happy path
2020-06-16 14:36:16 +01:00
Alex Sharov
8d04689041
close db in tests (#669) 2020-06-16 12:05:40 +07:00
Alex Sharov
57358730a4
Minor lmdb related improvements (#667)
* don't call initCursor on happy path

* don't call initCursor on happy path

* don't run stale reads goroutine for inMem mode

* don't call initCursor on happy path

* remove buffers from cursor object - they are useful only in Badger implementation

* commit kv benchmark

* remove buffers from cursor object - they are useful only in Badger implementation

* remove buffers from cursor object - they are useful only in Badger implementation

* cancel server before return pipe to pool

* try  to fix test

* set field db in managed tx
2020-06-15 13:30:54 +01:00
Alex Sharov
862296d291
Lmdb cursors not always return to pool (#664)
* set Pooled=true

* don't call initCursor on happy path
2020-06-14 21:05:52 +07:00
Alex Sharov
bcac51697b
Enable NoReadahead by default (#660)
* Enable NoReadahead by default

* don't call lmdb.IsNotFound on happy path

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-06-13 14:12:35 +01:00
Alex Sharov
b78a6336d2
dont re-read whole db to calculate file size (#658) 2020-06-13 13:41:38 +01:00
Alex Sharov
b05fafc628
Optimize multi put (#657)
* remove unused slice from MultiPut

* mutation: reuse tuples slice and preallocate bucketPuts

* use bucketPool in kv_lmdb

* remove duplicated check of context status

* more benchmarks

* remove reusage of puts
2020-06-12 18:59:06 +01:00
Alex Sharov
15096f273d
Remove ctx from Open. Stop goroutines on Close. (#650)
* remove ctx from MustOpen

* remove ctx from Open. Stop goroutines on Close.

* remove ctx from Open. Stop goroutines on Close.

* remove ctx from remote open (we have DialTimeout field to manage connection timeouts)

* enable RawReads and add native implementation of Get/Has methods
2020-06-12 10:31:21 +01:00