Commit Graph

324 Commits

Author SHA1 Message Date
Alex Sharov
49351b0091
KV: to support multiple databases and multiple set of buckets (#1002)
* don't use global variables

* up deps

* clean

* use joined flags

* use single non-global buckets config

* clean

* copy map

* fix tests

* fix tests

* clean

* clean
2020-08-30 18:34:18 +01:00
Alex Sharov
5ce73d438a
fix integration (#999) 2020-08-29 15:27:20 +07:00
Alex Sharov
07ca5c643a
eth_syncing (#991)
* eth_syncing

* linters

* fix readme

* cleanup cli params

* go mod tidy

* remove memsize ui dep

* remove bloomfilter

* implement net_peerCount

* remove legacy flags
2020-08-29 08:24:50 +01:00
Alex Sharov
df05a581dd
Add tx pool stage to transaction (#981)
* add tx_pool stage to tx

* clean

* don't call commit in read tx

* don't call commit in read tx

* clean

* more logs

* fix tx pool race

* more logs

* clean
2020-08-28 07:27:48 +01:00
Alex Sharov
714e5041e2
Better logs3 (#976) 2020-08-26 19:06:51 +07:00
Alex Sharov
9091ce62bb
KV reduce features amount (#975)
* move IdealBatchSize method to Batch object :-)
remove cursor.Walk
remove kv.IdealBatchSize

* remove bolt_db.go file

* move method NoValueCursor to tx object
2020-08-26 07:03:50 +01:00
Alex Sharov
522287ac18
Transactional cycle (#966)
* v0

* v1

* v3

* v4

* clean

* temporary fix of txpool

* Add debug logs about tx start/commit

* Add debug logs about tx start/commit

* Add debug logs about tx start/commit

* add condition

* tx pool to not hold own db

* try to enable TxPool in integration

* exclude tx pool from tx

* exclude tx pool from integration

* reduce limit

* fix integration

* clean

* clean

* clean

* clean

* clean

* exclude tx pool unwind

* exclude tx pool unwind in integration

* fix integration tx pool

* fix commit

* fix current stage after unwind

* fix current stage after unwind

* fix linter

* move unwind of tx_pool after unwind of unwind of senders, then all stages from body to tx_pool will be inside tx.

* move body and headers unwind out of tx

* fix unwind order after reboot

* add support external tx to exec stage

* clean

* clean

* clean

* clean

* clean

* add logs

* better id check

* better id check
2020-08-26 07:02:10 +01:00
ledgerwatch
8728e54a72
Another step in migrating tests to staged sync (#970)
* Log in GetAsOf

* Default plain state

* trace

* More test fixes

* Fix lint

* Fix

* Fix lint

* Fix lint

* Fix lint

* Fix lint

* Fix for events

* Repair events

* Fix lint

* Fix lint

* Change back to InsertChain
2020-08-24 18:00:40 +01:00
Alex Sharov
43a4e34a6e
[to discuss] exec blocks without ObjectDb: Mutation over TxDb (#947)
* use mutation over tx

* clean

* clear

* add .CommitAndBegin() method

* clean

* increase timings for logging

* return ideal batch size
2020-08-24 12:07:59 +01:00
Alex Sharov
a6be18b915
ticker-based logs (#954)
* timer-based logs

* timer-based logs

* delegate progress calculation to user

* delegate progress calculation to user

* delegate progress calculation to user

* clear

* add logs to senders recovery

* use default dir in integration

* more logs

* more logs
2020-08-22 12:12:33 +02:00
Alex Sharov
ebbaa55672
sub-transaction to not call fsync, sub-transactions to not call runtime.LockThread() (#945) 2020-08-21 07:31:43 +01:00
ledgerwatch
01f211f3b8
Revert "IH: no retain list (#946)" (#948)
This reverts commit f86ed64288.
2020-08-20 13:37:34 +01:00
Alex Sharov
f86ed64288
IH: no retain list (#946)
* experiment of not using retain list

* better handle sub-tx

* save progress

* don't touch collector

* don't run collector on sub-tx because of `mdb_cursor_put: MDB_TXN_FULL: Transaction has too many dirty pages - transaction too big`

* clear

* clear
2020-08-20 11:42:43 +01:00
b00ris
eb46dd68df
Account range testing (#938)
* save state

* save state

* save new timeouts

* compare passed

* save state

* cleanup
2020-08-19 07:33:49 +01:00
Evgeny Danilenko
3b40819444
grpc regenerate (#936)
* grpc regenerate

* use string type for id in net_version
2020-08-18 20:22:49 +03:00
Alex Sharov
081d02920d
Remove "Bucket" class from KV abstraction (#928)
* drop Bucket object from KV abstraction

* add some non-public methods

* rebase master

* rebase master
2020-08-17 08:24:59 +01:00
Alex Sharov
b2e35eb63f
rename "remote2" to "remote" (#932) 2020-08-17 07:47:06 +01:00
Alex Sharov
0e253e7336
lmdb transactions of unlimited size (#918)
* add logging to loader

* use pure tx in etl loading, logs in mutation commit

* clean

* bletter logging and more cleanup

* bletter logging and more cleanup

* increase batch size to 500M

* better batch commit logging

* async fsync

* sync fsync

* sync fsync

* unify logging

* fix corner-case when etl can use empty bucket name

* fix tests

* better logging

* better logging

* rebase master

* remove lmdb.NoMetaSync flag for now

* consistent walk and multi-walk

* clean

* sub tx

* add consistent multi-put

* implement dupsort support in one new cursor method

* clear
2020-08-17 07:45:52 +01:00
Igor Mandrigin
d848233275 Merge branch 'master' into geth-1.9.19
# Conflicts:
#	cmd/geth/consolecmd.go
#	cmd/rpcdaemon/service/service.go
#	cmd/utils/flags.go
#	eth/backend.go
2020-08-15 21:55:09 +02:00
Igor Mandrigin
201524c33e fix node tests 2020-08-15 20:21:30 +02:00
Alex Sharov
bf596c26f3
[merge after release] dupsort of plain state (#913)
* dupsort of plain state

* rebase master
2020-08-15 08:11:40 +01:00
Alex Sharov
e04c4ebe58
KV: move bucket.Get and bucket.Cursor methods to Tx object (#914) 2020-08-14 13:41:18 +07:00
Alex Sharov
f7deef79bb
drop bucket log message (#917) 2020-08-13 13:40:28 +07:00
Evgeny Danilenko
5d39fe8873
RPCDeamon coinbase, net_version (#911)
* use eth.backend

* net_version

* add coinbase to the console test
2020-08-12 14:47:59 +01:00
Alex Sharov
16f09be0a8
add method .Last() (#909) 2020-08-12 10:49:52 +07:00
Alex Sharov
760fe0f0b6
DupSort of hash state (#896) 2020-08-12 09:57:55 +07:00
Evgeny Danilenko
3980fa7d45
Grps eth_sendTransaction (#882)
* implementation

* tidy gomod

* linters

* fix cmd test

* fix

* fix lint
2020-08-11 22:09:30 +01:00
Alex Sharov
6de7eb1bec
correct_thread_lock_on_non_managed_tx (#905) 2020-08-11 18:53:47 +07:00
Alex Sharov
7addf34f64
fix wrong merge (#907) 2020-08-11 18:24:39 +07:00
Alex Sharov
e89ad5d6aa
sub tx support (#895) 2020-08-11 17:35:59 +07:00
Alex Sharov
695f8f8904
allow multi-rollback (#904) 2020-08-11 17:35:19 +07:00
Alex Sharov
d9d9e14f45
change bucket type to string (#894) 2020-08-11 06:55:32 +07:00
Alex Sharov
7a1b892022
remove .SeekTo() method (#901) 2020-08-11 06:41:15 +07:00
Alex Sharov
091819a51c
adopt --metrics.addr flag in integration (#889) 2020-08-11 06:38:34 +07:00
Igor Mandrigin
8600616c3d geth 1.9.18 post-rebase fixups 2020-08-08 17:33:35 +02:00
Alex Sharov
f41959f030
drop badger support (#869)
* drop badger support

* rebase master
2020-08-05 16:33:45 +01:00
Alex Sharov
d458c4cc1e
Migrations: use stage name as db key (#868) 2020-08-05 17:13:35 +07:00
Igor Mandrigin
e732ccd438 add lmdb.mapSize param 2020-08-05 10:17:50 +02:00
ledgerwatch
0514501937
Revert "Use stage name as db key (#858)" (#867)
This reverts commit 7290bf519e.
2020-08-04 11:05:27 +01:00
Alex Sharov
7290bf519e
Use stage name as db key (#858) 2020-08-04 16:25:28 +07:00
Alex Sharov
a5c1089d58
optimize .get (#863) 2020-08-04 10:31:37 +07:00
Alex Sharov
ecc94a63f0
DupSort for Plain and Hashed state buckets (behind feature flags) (#854)
* return error from rawdb

* squash

* v14

* improve performance of put

* clean

* clean

* hide feature behind ENV variable

* cleanup

* cleanup

* disable ipc and make Readme less confusing (people thought points are depend on each-other)

* fix test

* cleanup

* cleanup
2020-08-02 12:53:08 +01:00
b00ris
a84230d45c
fix streams leak (#847) 2020-08-01 16:18:31 +01:00
Alex Sharov
9a5c2a1242
reduce mmap limit size (raspbery-pi failed on 4TB) (#840) 2020-08-01 11:59:40 +07:00
Alex Sharov
6c4ec563e1
fix - change private api from ui (#829) 2020-07-30 11:17:09 +01:00
Alex Sharov
b6399a2256
better error messages (#824) 2020-07-30 13:55:40 +07:00
ledgerwatch
7bc45fab80
Remove preimage storage mode, write receipts by default (#809) 2020-07-29 14:00:39 +01:00
Alex Sharov
0a367001d1
implement db stats page (#803) 2020-07-29 05:31:46 +01:00
Alex Sharov
47b7b2cf40
lmdb_is_not_friendly_to_empty_values (#795) 2020-07-29 05:30:29 +01:00
Alex Sharov
f500148b86
remove BoltDatabase, BadgerDatabase, RemoteDatabase classes (replaced by objectDb) (#794) 2020-07-27 22:59:54 +01:00