erigon-pulse/common/dbutils/bucket.go

448 lines
16 KiB
Go
Raw Normal View History

package dbutils
import (
"sort"
2020-08-10 23:55:32 +00:00
"strings"
Integration tests 1 (#1793) * Initial commit * Add sentry gRPC interface * p2psentry directory * Update README.md * Update README.md * Update README.md * Add go package * Correct syntax * add external downloader interface (#2) * Add txpool (#3) * Add private API (#4) * Invert control.proto, add PeerMinBlock, Separare incoming Tx message into a separate stream (#5) Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local> * Separate upload messages into its own stream (#6) Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local> * Only send changed accounts to listeners (#7) * Txpool interface doc (#9) * Add architecture diagram source and picture (#10) * Typed hashes (#11) * Typed hashes * Fix PeerId * 64-bit tx nonce * Add proper golang packages, max_block into p2p sentry Status (#12) * Add proper golang packages, max_block into p2p sentry Status * Change EtherReply to address Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local> * Add Rust infrastructure (#13) * DB stats methods removed by https://github.com/ledgerwatch/turbo-geth/pull/1665 * more p2p methods (#15) * add mining methods (#16) * First draft of Consensus gRPC interface (#14) * Update Rust build * Fix interfaces in architecture diagram (#17) * Fix KV interface provider * Fix Consensus interface provider * drop java attributes (#18) * tx pool remove unused import (#19) * ethbackend: add protocol version and client version (#20) * Add missing ethbackend I/F (#21) * Add interface versioning mechanism (#23) Add versioning in KV interface Co-authored-by: Artem Vorotnikov <artem@vorotnikov.me> * spec of tx pool method (#24) * spec of tx pool method (#25) * Update version.proto * Refactor interface versioning * Refactor interface versioning * Testing interface * Remove tree * Fix * Build testing protos * Fix * Fix * Update to the newer interfaces * Add ProtocolVersion and ClientVersion stubs * Hook up ProtocolVersion and ClientVersion * Remove service * Add compatibility checks for RPC daemon * Fix typos * Properly update DB schema version * Fix test * Add test for KV compatibility| * Info messages about compability for RPC daemon * DB schema version to be one key * Update release intructions Co-authored-by: Artem Vorotnikov <artem@vorotnikov.me> Co-authored-by: b00ris <b00ris@mail.ru> Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local> Co-authored-by: lightclient <14004106+lightclient@users.noreply.github.com> Co-authored-by: canepat <16927169+canepat@users.noreply.github.com> Co-authored-by: Alex Sharov <AskAlexSharov@gmail.com> Co-authored-by: canepat <tullio.canepa@gmail.com> Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2021-04-24 15:46:29 +00:00
"github.com/ledgerwatch/erigon-lib/gointerfaces/types"
)
Integration tests 1 (#1793) * Initial commit * Add sentry gRPC interface * p2psentry directory * Update README.md * Update README.md * Update README.md * Add go package * Correct syntax * add external downloader interface (#2) * Add txpool (#3) * Add private API (#4) * Invert control.proto, add PeerMinBlock, Separare incoming Tx message into a separate stream (#5) Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local> * Separate upload messages into its own stream (#6) Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local> * Only send changed accounts to listeners (#7) * Txpool interface doc (#9) * Add architecture diagram source and picture (#10) * Typed hashes (#11) * Typed hashes * Fix PeerId * 64-bit tx nonce * Add proper golang packages, max_block into p2p sentry Status (#12) * Add proper golang packages, max_block into p2p sentry Status * Change EtherReply to address Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local> * Add Rust infrastructure (#13) * DB stats methods removed by https://github.com/ledgerwatch/turbo-geth/pull/1665 * more p2p methods (#15) * add mining methods (#16) * First draft of Consensus gRPC interface (#14) * Update Rust build * Fix interfaces in architecture diagram (#17) * Fix KV interface provider * Fix Consensus interface provider * drop java attributes (#18) * tx pool remove unused import (#19) * ethbackend: add protocol version and client version (#20) * Add missing ethbackend I/F (#21) * Add interface versioning mechanism (#23) Add versioning in KV interface Co-authored-by: Artem Vorotnikov <artem@vorotnikov.me> * spec of tx pool method (#24) * spec of tx pool method (#25) * Update version.proto * Refactor interface versioning * Refactor interface versioning * Testing interface * Remove tree * Fix * Build testing protos * Fix * Fix * Update to the newer interfaces * Add ProtocolVersion and ClientVersion stubs * Hook up ProtocolVersion and ClientVersion * Remove service * Add compatibility checks for RPC daemon * Fix typos * Properly update DB schema version * Fix test * Add test for KV compatibility| * Info messages about compability for RPC daemon * DB schema version to be one key * Update release intructions Co-authored-by: Artem Vorotnikov <artem@vorotnikov.me> Co-authored-by: b00ris <b00ris@mail.ru> Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local> Co-authored-by: lightclient <14004106+lightclient@users.noreply.github.com> Co-authored-by: canepat <16927169+canepat@users.noreply.github.com> Co-authored-by: Alex Sharov <AskAlexSharov@gmail.com> Co-authored-by: canepat <tullio.canepa@gmail.com> Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2021-04-24 15:46:29 +00:00
// DBSchemaVersion
var DBSchemaVersion = types.VersionReply{Major: 3, Minor: 0, Patch: 0}
Integration tests 1 (#1793) * Initial commit * Add sentry gRPC interface * p2psentry directory * Update README.md * Update README.md * Update README.md * Add go package * Correct syntax * add external downloader interface (#2) * Add txpool (#3) * Add private API (#4) * Invert control.proto, add PeerMinBlock, Separare incoming Tx message into a separate stream (#5) Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local> * Separate upload messages into its own stream (#6) Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local> * Only send changed accounts to listeners (#7) * Txpool interface doc (#9) * Add architecture diagram source and picture (#10) * Typed hashes (#11) * Typed hashes * Fix PeerId * 64-bit tx nonce * Add proper golang packages, max_block into p2p sentry Status (#12) * Add proper golang packages, max_block into p2p sentry Status * Change EtherReply to address Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local> * Add Rust infrastructure (#13) * DB stats methods removed by https://github.com/ledgerwatch/turbo-geth/pull/1665 * more p2p methods (#15) * add mining methods (#16) * First draft of Consensus gRPC interface (#14) * Update Rust build * Fix interfaces in architecture diagram (#17) * Fix KV interface provider * Fix Consensus interface provider * drop java attributes (#18) * tx pool remove unused import (#19) * ethbackend: add protocol version and client version (#20) * Add missing ethbackend I/F (#21) * Add interface versioning mechanism (#23) Add versioning in KV interface Co-authored-by: Artem Vorotnikov <artem@vorotnikov.me> * spec of tx pool method (#24) * spec of tx pool method (#25) * Update version.proto * Refactor interface versioning * Refactor interface versioning * Testing interface * Remove tree * Fix * Build testing protos * Fix * Fix * Update to the newer interfaces * Add ProtocolVersion and ClientVersion stubs * Hook up ProtocolVersion and ClientVersion * Remove service * Add compatibility checks for RPC daemon * Fix typos * Properly update DB schema version * Fix test * Add test for KV compatibility| * Info messages about compability for RPC daemon * DB schema version to be one key * Update release intructions Co-authored-by: Artem Vorotnikov <artem@vorotnikov.me> Co-authored-by: b00ris <b00ris@mail.ru> Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local> Co-authored-by: lightclient <14004106+lightclient@users.noreply.github.com> Co-authored-by: canepat <16927169+canepat@users.noreply.github.com> Co-authored-by: Alex Sharov <AskAlexSharov@gmail.com> Co-authored-by: canepat <tullio.canepa@gmail.com> Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2021-04-24 15:46:29 +00:00
// Buckets
// Dictionary:
// "Plain State" - state where keys arent' hashed. "CurrentState" - same, but keys are hashed. "PlainState" used for blocks execution. "CurrentState" used mostly for Merkle root calculation.
// "incarnation" - uint64 number - how much times given account was SelfDestruct'ed.
2021-05-31 08:29:46 +00:00
/*
PlainStateBucket logical layout:
Contains Accounts:
key - address (unhashed)
value - account encoded for storage
Contains Storage:
key - address (unhashed) + incarnation + storage key (unhashed)
value - storage value(common.hash)
Physical layout:
2021-06-16 10:57:58 +00:00
PlainStateBucket and HashedStorageBucket utilises DupSort feature of MDBX (store multiple values inside 1 key).
-------------------------------------------------------------
key | value
-------------------------------------------------------------
[acc_hash] | [acc_value]
[acc_hash]+[inc] | [storage1_hash]+[storage1_value]
| [storage2_hash]+[storage2_value] // this value has no own key. it's 2nd value of [acc_hash]+[inc] key.
| [storage3_hash]+[storage3_value]
| ...
[acc_hash]+[old_inc] | [storage1_hash]+[storage1_value]
| ...
[acc2_hash] | [acc2_value]
...
*/
const PlainStateBucket = "PlainState"
2021-05-31 08:29:46 +00:00
//PlainContractCodeBucket -
//key - address+incarnation
//value - code hash
const PlainContractCodeBucket = "PlainCodeHash"
2021-05-31 08:29:46 +00:00
/*
2021-05-31 09:49:39 +00:00
AccountChangeSetBucket and StorageChangeSetBucket - of block N store values of state before block N changed them.
Because values "after" change stored in PlainState.
Logical format:
2021-05-31 08:29:46 +00:00
key - blockNum_u64 + key_in_plain_state
value - value_in_plain_state_before_blockNum_changes
2021-05-31 08:29:46 +00:00
Example: If block N changed account A from value X to Y. Then:
AccountChangeSetBucket has record: bigEndian(N) + A -> X
PlainStateBucket has record: A -> Y
2021-05-31 08:29:46 +00:00
See also: docs/programmers_guide/db_walkthrough.MD#table-history-of-accounts
As you can see if block N changes much accounts - then all records have repetitive prefix `bigEndian(N)`.
MDBX can store such prefixes only once - by DupSort feature (see `docs/programmers_guide/dupsort.md`).
Both buckets are DupSort-ed and have physical format:
AccountChangeSetBucket:
key - blockNum_u64
value - address + account(encoded)
StorageChangeSetBucket:
key - blockNum_u64 + address + incarnation_u64
value - plain_storage_key + value
*/
const AccountChangeSetBucket = "AccountChangeSet"
const StorageChangeSetBucket = "StorageChangeSet"
2021-05-31 08:29:46 +00:00
const (
//HashedAccountsBucket
// key - address hash
// value - account encoded for storage
// Contains Storage:
//key - address hash + incarnation + storage key hash
//value - storage value(common.hash)
HashedAccountsBucket = "HashedAccount"
HashedStorageBucket = "HashedStorage"
2021-05-31 05:30:30 +00:00
)
2021-05-31 08:29:46 +00:00
/*
AccountsHistoryBucket and StorageHistoryBucket - indices designed to serve next 2 type of requests:
2021-05-31 05:30:30 +00:00
1. what is smallest block number >= X where account A changed
2. get last shard of A - to append there new block numbers
2021-05-31 09:49:39 +00:00
Task 1. is part of "get historical state" operation (see `core/state:GetAsOf`):
If `db.Seek(A+bigEndian(X))` returns non-last shard -
then get block number from shard value Y := RoaringBitmap(shard_value).GetGte(X)
and with Y go to ChangeSets: db.Get(ChangeSets, Y+A)
If `db.Seek(A+bigEndian(X))` returns last shard -
then we go to PlainState: db.Get(PlainState, A)
2021-05-31 05:30:30 +00:00
Format:
- index split to shards by 2Kb - RoaringBitmap encoded sorted list of block numbers
(to avoid performance degradation of popular accounts or look deep into history.
Also 2Kb allows avoid Overflow pages inside DB.)
- if shard is not last - then key has suffix 8 bytes = bigEndian(max_block_num_in_this_shard)
- if shard is last - then key has suffix 8 bytes = 0xFF
It allows:
- server task 1. by 1 db operation db.Seek(A+bigEndian(X))
- server task 2. by 1 db operation db.Get(A+0xFF)
2021-05-31 08:29:46 +00:00
see also: docs/programmers_guide/db_walkthrough.MD#table-change-sets
2021-05-31 05:30:30 +00:00
AccountsHistoryBucket:
key - address + shard_id_u64
value - roaring bitmap - list of block where it changed
StorageHistoryBucket
key - address + storage_key + shard_id_u64
value - roaring bitmap - list of block where it changed
*/
var AccountsHistoryBucket = "AccountHistory"
var StorageHistoryBucket = "StorageHistory"
2021-05-31 05:30:30 +00:00
var (
//key - contract code hash
//value - contract code
CodeBucket = "Code"
//key - addressHash+incarnation
//value - code hash
ContractCodeBucket = "HashedCodeHash"
// IncarnationMapBucket for deleted accounts
//key - address
//value - incarnation of account when it was last deleted
IncarnationMapBucket = "IncarnationMap"
//TEVMCodeBucket -
//key - contract code hash
//value - contract TEVM code
ContractTEVMCodeBucket = "TEVMCode"
)
/*TrieOfAccountsBucket and TrieOfStorageBucket
hasState,groups - mark prefixes existing in hashed_account table
hasTree - mark prefixes existing in trie_account table (not related with branchNodes)
hasHash - mark prefixes which hashes are saved in current trie_account record (actually only hashes of branchNodes can be saved)
@see UnmarshalTrieNode
@see integrity.Trie
+-----------------------------------------------------------------------------------------------------+
| DB record: 0x0B, hasState: 0b1011, hasTree: 0b1001, hasHash: 0b1001, hashes: [x,x] |
+-----------------------------------------------------------------------------------------------------+
| | |
v | v
+---------------------------------------------+ | +--------------------------------------+
| DB record: 0x0B00, hasState: 0b10001 | | | DB record: 0x0B03, hasState: 0b10010 |
| hasTree: 0, hasHash: 0b10000, hashes: [x] | | | hasTree: 0, hasHash: 0, hashes: [] |
+---------------------------------------------+ | +--------------------------------------+
| | | | |
v v v v v
+------------------+ +----------------------+ +---------------+ +---------------+ +---------------+
| Account: | | BranchNode: 0x0B0004 | | Account: | | Account: | | Account: |
| 0x0B0000... | | has no record in | | 0x0B01... | | 0x0B0301... | | 0x0B0304... |
| in HashedAccount | | TrieAccount | | | | | | |
+------------------+ +----------------------+ +---------------+ +---------------+ +---------------+
| |
v v
+---------------+ +---------------+
| Account: | | Account: |
| 0x0B000400... | | 0x0B000401... |
+---------------+ +---------------+
Invariants:
- hasTree is subset of hasState
- hasHash is subset of hasState
- first level in account_trie always exists if hasState>0
- TrieStorage record of account.root (length=40) must have +1 hash - it's account.root
- each record in TrieAccount table must have parent (may be not direct) and this parent must have correct bit in hasTree bitmap
- if hasState has bit - then HashedAccount table must have record according to this bit
- each TrieAccount record must cover some state (means hasState is always > 0)
- TrieAccount records with length=1 can satisfy (hasBranch==0&&hasHash==0) condition
- Other records in TrieAccount and TrieStorage must (hasTree!=0 || hasHash!=0)
*/
const TrieOfAccountsBucket = "TrieAccount"
const TrieOfStorageBucket = "TrieStorage"
2021-04-08 02:46:11 +00:00
const (
// DatabaseInfoBucket is used to store information about data layout.
DatabaseInfoBucket = "DbInfo"
SnapshotInfoBucket = "SnapshotInfo"
BittorrentInfoBucket = "BittorrentInfo"
// Data item prefixes (use single byte to avoid mixing data types, avoid `i`, used for indexes).
HeaderPrefixOld = "h" // block_num_u64 + hash -> header
HeaderNumberBucket = "HeaderNumber" // headerNumberPrefix + hash -> num (uint64 big endian)
HeaderCanonicalBucket = "CanonicalHeader" // block_num_u64 -> header hash
HeadersBucket = "Header" // block_num_u64 + hash -> header (RLP)
HeaderTDBucket = "HeadersTotalDifficulty" // block_num_u64 + hash -> td (RLP)
BlockBodyPrefix = "BlockBody" // block_num_u64 + hash -> block body
EthTx = "BlockTransaction" // tbl_sequence_u64 -> rlp(tx)
BlockReceiptsPrefix = "Receipt" // block_num_u64 -> canonical block receipts (non-canonical are not stored)
Log = "TransactionLog" // block_num_u64 + txId -> logs of transaction
bitmap indices for logs (#1124) * save progress * try now * don't create bloom inside rlpDecode * don't create bloom inside ApplyTransaction * clean * clean * clean * clean * clean * clean * clean * clean * rename method * print timings * print timings * print timings * sort before flush * fix err lint * clean * move tests to transactions * compressed version * up bound * up bound * more tests * more tests * more tests * more tests * better removal * clean * better performance of get/put methods * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * optimize rpcdaemon * fix test * fix rpcdaemon * fix test * simplify * simplify * fix nil pointer * clean * revert some changes * add some logs * clean * try without optimize * clean * clean * clean * clean * try * move log_index to own stage * move log_index to own stage * integration add log_index stage * integration add log_index stage * clean * clean * print timing * remove duplicates at unwind * extract truncateBitmaps func * try detect * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * add blackList of topics * clean * clean * clean * clean * clean * clean * clean * clean * sharding 1 * sharded 2 * sharded 2 * sharded 2 * sharded 2 * sharded 2 * sharded 2 * sharded 2 * sharded 2 * sharded 2 * sharded 2 * sharded 2 * sharded 2 * sharded 2 * sharded 2 * sharded 2 * sharded 2 * sharded 3 * sharded 3 * sharded 3 * speedup things by putCurrent and putReserve * clean * optimize trim * clean * remove blacklist * add more info to err * ? * clean * clean * clean * clean * clean * working version * switch to cgo version of roaring bitmaps * clean * clean * clean * clean * more docs * clean * clean * fix logs bloom field * Fix debug_getModifiedAccountsByNumber * Try to fix crash * fix problem with "absent block" * fix problem with "absent block" * remove optimize method call * remove roaring iterator * fix problem with rebuild indicess * remove debug prints * tests for eth_getLogs involving topics * add tests for new stage, speparate topics into 2 buckets * version up * remove debug logs * remove debug logs * remove bloom filter implementation * Optimisation * Optimisatin not required, make rpctest lenient to geth errors * Lenient to geth failures Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-09-28 17:18:36 +00:00
// Stores bitmap indices - in which block numbers saw logs of given 'address' or 'topic'
// [addr or topic] + [2 bytes inverted shard number] -> bitmap(blockN)
// indices are sharded - because some bitmaps are >1Mb and when new incoming blocks process it
2021-06-16 10:57:58 +00:00
// updates ~300 of bitmaps - by append small amount new values. It cause much big writes (MDBX does copy-on-write).
bitmap indices for logs (#1124) * save progress * try now * don't create bloom inside rlpDecode * don't create bloom inside ApplyTransaction * clean * clean * clean * clean * clean * clean * clean * clean * rename method * print timings * print timings * print timings * sort before flush * fix err lint * clean * move tests to transactions * compressed version * up bound * up bound * more tests * more tests * more tests * more tests * better removal * clean * better performance of get/put methods * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * optimize rpcdaemon * fix test * fix rpcdaemon * fix test * simplify * simplify * fix nil pointer * clean * revert some changes * add some logs * clean * try without optimize * clean * clean * clean * clean * try * move log_index to own stage * move log_index to own stage * integration add log_index stage * integration add log_index stage * clean * clean * print timing * remove duplicates at unwind * extract truncateBitmaps func * try detect * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * add blackList of topics * clean * clean * clean * clean * clean * clean * clean * clean * sharding 1 * sharded 2 * sharded 2 * sharded 2 * sharded 2 * sharded 2 * sharded 2 * sharded 2 * sharded 2 * sharded 2 * sharded 2 * sharded 2 * sharded 2 * sharded 2 * sharded 2 * sharded 2 * sharded 2 * sharded 3 * sharded 3 * sharded 3 * speedup things by putCurrent and putReserve * clean * optimize trim * clean * remove blacklist * add more info to err * ? * clean * clean * clean * clean * clean * working version * switch to cgo version of roaring bitmaps * clean * clean * clean * clean * more docs * clean * clean * fix logs bloom field * Fix debug_getModifiedAccountsByNumber * Try to fix crash * fix problem with "absent block" * fix problem with "absent block" * remove optimize method call * remove roaring iterator * fix problem with rebuild indicess * remove debug prints * tests for eth_getLogs involving topics * add tests for new stage, speparate topics into 2 buckets * version up * remove debug logs * remove debug logs * remove bloom filter implementation * Optimisation * Optimisatin not required, make rpctest lenient to geth errors * Lenient to geth failures Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-09-28 17:18:36 +00:00
//
// if last existing shard size merge it with delta
// if serialized size of delta > ShardLimit - break down to multiple shards
// shard number - it's biggest value in bitmap
LogTopicIndex = "LogTopicIndex"
LogAddressIndex = "LogAddressIndex"
bitmap indices for logs (#1124) * save progress * try now * don't create bloom inside rlpDecode * don't create bloom inside ApplyTransaction * clean * clean * clean * clean * clean * clean * clean * clean * rename method * print timings * print timings * print timings * sort before flush * fix err lint * clean * move tests to transactions * compressed version * up bound * up bound * more tests * more tests * more tests * more tests * better removal * clean * better performance of get/put methods * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * optimize rpcdaemon * fix test * fix rpcdaemon * fix test * simplify * simplify * fix nil pointer * clean * revert some changes * add some logs * clean * try without optimize * clean * clean * clean * clean * try * move log_index to own stage * move log_index to own stage * integration add log_index stage * integration add log_index stage * clean * clean * print timing * remove duplicates at unwind * extract truncateBitmaps func * try detect * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * add blackList of topics * clean * clean * clean * clean * clean * clean * clean * clean * sharding 1 * sharded 2 * sharded 2 * sharded 2 * sharded 2 * sharded 2 * sharded 2 * sharded 2 * sharded 2 * sharded 2 * sharded 2 * sharded 2 * sharded 2 * sharded 2 * sharded 2 * sharded 2 * sharded 2 * sharded 3 * sharded 3 * sharded 3 * speedup things by putCurrent and putReserve * clean * optimize trim * clean * remove blacklist * add more info to err * ? * clean * clean * clean * clean * clean * working version * switch to cgo version of roaring bitmaps * clean * clean * clean * clean * more docs * clean * clean * fix logs bloom field * Fix debug_getModifiedAccountsByNumber * Try to fix crash * fix problem with "absent block" * fix problem with "absent block" * remove optimize method call * remove roaring iterator * fix problem with rebuild indicess * remove debug prints * tests for eth_getLogs involving topics * add tests for new stage, speparate topics into 2 buckets * version up * remove debug logs * remove debug logs * remove bloom filter implementation * Optimisation * Optimisatin not required, make rpctest lenient to geth errors * Lenient to geth failures Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-09-28 17:18:36 +00:00
// CallTraceSet is the name of the table that contain the mapping of block number to the set (sorted) of all accounts
// touched by call traces. It is DupSort-ed table
// 8-byte BE block number -> account address -> two bits (one for "from", another for "to")
CallTraceSet = "CallTraceSet"
// Indices for call traces - have the same format as LogTopicIndex and LogAddressIndex
// Store bitmap indices - in which block number we saw calls from (CallFromIndex) or to (CallToIndex) some addresses
CallFromIndex = "CallFromIndex"
CallToIndex = "CallToIndex"
TxLookupPrefix = "BlockTransactionLookup" // hash -> transaction/receipt lookup metadata
BloomBitsPrefix = "BloomBits" // bit (uint16 big endian) + section (uint64 big endian) + hash -> bloom bits
ConfigPrefix = "Config" // config prefix for the db
// Chain index prefixes (use `i` + single byte to avoid mixing data types).
BloomBitsIndexPrefix = "BloomBitsIndex" // BloomBitsIndexPrefix is the data table of a chain indexer to track its progress
2020-08-04 09:25:28 +00:00
// Progress of sync stages: stageName -> stageData
SyncStageProgress = "SyncStage"
// Position to where to unwind sync stages: stageName -> stageData
SyncStageUnwind = "SyncStageUnwind"
CliqueBucket = "Clique"
CliqueSeparateBucket = "CliqueSeparate"
CliqueSnapshotBucket = "CliqueSnapshot"
CliqueLastSnapshotBucket = "CliqueLastSnapshot"
2020-08-04 09:25:28 +00:00
// this bucket stored in separated database
InodesBucket = "Inode"
// Transaction senders - stored separately from the block bodies
Senders = "TxSender" // block_num_u64 + blockHash -> sendersList (no serialization format, every 20 bytes is new sender)
// headBlockKey tracks the latest know full block's hash.
2020-08-10 23:55:32 +00:00
HeadBlockKey = "LastBlock"
// migrationName -> serialized SyncStageProgress and SyncStageUnwind buckets
// it stores stages progress to understand in which context was executed migration
// in case of bug-report developer can ask content of this bucket
Migrations = "Migration"
Sequence = "Sequence" // tbl_name -> seq_u64
HeadHeaderKey = "LastHeader"
Epoch = "Epoch"
)
// Keys
var (
//StorageModeHistory - does node save history.
StorageModeHistory = []byte("smHistory")
//StorageModeReceipts - does node save receipts.
StorageModeReceipts = []byte("smReceipts")
//StorageModeTxIndex - does node save transactions index.
StorageModeTxIndex = []byte("smTxIndex")
//StorageModeCallTraces - does not build index of call traces
StorageModeCallTraces = []byte("smCallTraces")
//StorageModeTEVM - does not translate EVM to TEVM
StorageModeTEVM = []byte("smTEVM")
2020-08-10 23:55:32 +00:00
Integration tests 1 (#1793) * Initial commit * Add sentry gRPC interface * p2psentry directory * Update README.md * Update README.md * Update README.md * Add go package * Correct syntax * add external downloader interface (#2) * Add txpool (#3) * Add private API (#4) * Invert control.proto, add PeerMinBlock, Separare incoming Tx message into a separate stream (#5) Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local> * Separate upload messages into its own stream (#6) Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local> * Only send changed accounts to listeners (#7) * Txpool interface doc (#9) * Add architecture diagram source and picture (#10) * Typed hashes (#11) * Typed hashes * Fix PeerId * 64-bit tx nonce * Add proper golang packages, max_block into p2p sentry Status (#12) * Add proper golang packages, max_block into p2p sentry Status * Change EtherReply to address Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local> * Add Rust infrastructure (#13) * DB stats methods removed by https://github.com/ledgerwatch/turbo-geth/pull/1665 * more p2p methods (#15) * add mining methods (#16) * First draft of Consensus gRPC interface (#14) * Update Rust build * Fix interfaces in architecture diagram (#17) * Fix KV interface provider * Fix Consensus interface provider * drop java attributes (#18) * tx pool remove unused import (#19) * ethbackend: add protocol version and client version (#20) * Add missing ethbackend I/F (#21) * Add interface versioning mechanism (#23) Add versioning in KV interface Co-authored-by: Artem Vorotnikov <artem@vorotnikov.me> * spec of tx pool method (#24) * spec of tx pool method (#25) * Update version.proto * Refactor interface versioning * Refactor interface versioning * Testing interface * Remove tree * Fix * Build testing protos * Fix * Fix * Update to the newer interfaces * Add ProtocolVersion and ClientVersion stubs * Hook up ProtocolVersion and ClientVersion * Remove service * Add compatibility checks for RPC daemon * Fix typos * Properly update DB schema version * Fix test * Add test for KV compatibility| * Info messages about compability for RPC daemon * DB schema version to be one key * Update release intructions Co-authored-by: Artem Vorotnikov <artem@vorotnikov.me> Co-authored-by: b00ris <b00ris@mail.ru> Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local> Co-authored-by: lightclient <14004106+lightclient@users.noreply.github.com> Co-authored-by: canepat <16927169+canepat@users.noreply.github.com> Co-authored-by: Alex Sharov <AskAlexSharov@gmail.com> Co-authored-by: canepat <tullio.canepa@gmail.com> Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2021-04-24 15:46:29 +00:00
DBSchemaVersionKey = []byte("dbVersion")
BittorrentPeerID = "peerID"
CurrentHeadersSnapshotHash = []byte("CurrentHeadersSnapshotHash")
CurrentHeadersSnapshotBlock = []byte("CurrentHeadersSnapshotBlock")
CurrentBodiesSnapshotHash = []byte("CurrentBodiesSnapshotHash")
CurrentBodiesSnapshotBlock = []byte("CurrentBodiesSnapshotBlock")
)
// Buckets - list of all buckets. App will panic if some bucket is not in this list.
// This list will be sorted in `init` method.
// BucketsConfigs - can be used to find index in sorted version of Buckets list by name
2020-08-10 23:55:32 +00:00
var Buckets = []string{
Intermediate hash phase 3 (#377) * #remove debug prints * remove storage-mode="i" * minnet re-execute hack with checkpoints * minnet re-execute hack with checkpoints * rollback to master setup * mainnet re-exec hack * rollback some changes * v0 of "push down" functionality * move all logic to own functions * handle case when re-created account already has some storage * clear path for storage * try to rely on tree structure (but maybe need to rely on DB because can be intra-block re-creations of account) * fix some bugs with indexes, moving to tests * tests added * make linter happy * make linter happy * simplify logic * adjust comparison of keys with and without incarnation * test for keyIsBefore * test for keyIsBefore * better nibbles alignment * better nibbles alignment * cleanup * continue work on tests * simplify test * check tombstone existence before pushing it down. * put tombstone only when account deleted, not created * put tombstone only when account has storage * make linter happy * test for storage resolver * make fixedbytes work without incarnation * fix panic on short keys * use special comparison only when working with keys from cache * add blockNr for better tracing * fix: incorrect tombstone check * fix: incorrect tombstone check * trigger ci * hack for problem block * more test-cases * add test case for too long keys * speedup cached resolver by removing bucket creation transaction * remove parent type check in pruning, remove unused copy from mutation.put * dump resolving info on fail * dump resolving info on fail * set tombstone everytime for now to check if it will help * on unload: check parent type, not type of node * fix wrong order of checking node type * fix wrong order of checking node type * rebase to new master * make linter happy * rebase to new master * place tombstone only if acc has storage * rebase master * rebase master * rebase master * rebase master Co-authored-by: alex.sharov <alex.sharov@lazada.com>
2020-03-11 10:31:49 +00:00
AccountsHistoryBucket,
StorageHistoryBucket,
CodeBucket,
ContractCodeBucket,
HeaderNumberBucket,
Intermediate hash phase 3 (#377) * #remove debug prints * remove storage-mode="i" * minnet re-execute hack with checkpoints * minnet re-execute hack with checkpoints * rollback to master setup * mainnet re-exec hack * rollback some changes * v0 of "push down" functionality * move all logic to own functions * handle case when re-created account already has some storage * clear path for storage * try to rely on tree structure (but maybe need to rely on DB because can be intra-block re-creations of account) * fix some bugs with indexes, moving to tests * tests added * make linter happy * make linter happy * simplify logic * adjust comparison of keys with and without incarnation * test for keyIsBefore * test for keyIsBefore * better nibbles alignment * better nibbles alignment * cleanup * continue work on tests * simplify test * check tombstone existence before pushing it down. * put tombstone only when account deleted, not created * put tombstone only when account has storage * make linter happy * test for storage resolver * make fixedbytes work without incarnation * fix panic on short keys * use special comparison only when working with keys from cache * add blockNr for better tracing * fix: incorrect tombstone check * fix: incorrect tombstone check * trigger ci * hack for problem block * more test-cases * add test case for too long keys * speedup cached resolver by removing bucket creation transaction * remove parent type check in pruning, remove unused copy from mutation.put * dump resolving info on fail * dump resolving info on fail * set tombstone everytime for now to check if it will help * on unload: check parent type, not type of node * fix wrong order of checking node type * fix wrong order of checking node type * rebase to new master * make linter happy * rebase to new master * place tombstone only if acc has storage * rebase master * rebase master * rebase master * rebase master Co-authored-by: alex.sharov <alex.sharov@lazada.com>
2020-03-11 10:31:49 +00:00
BlockBodyPrefix,
BlockReceiptsPrefix,
TxLookupPrefix,
BloomBitsPrefix,
ConfigPrefix,
BloomBitsIndexPrefix,
DatabaseInfoBucket,
IncarnationMapBucket,
ContractTEVMCodeBucket,
CliqueSeparateBucket,
CliqueLastSnapshotBucket,
CliqueSnapshotBucket,
SyncStageProgress,
SyncStageUnwind,
PlainStateBucket,
PlainContractCodeBucket,
AccountChangeSetBucket,
StorageChangeSetBucket,
Senders,
HeadBlockKey,
HeadHeaderKey,
Migrations,
bitmap indices for logs (#1124) * save progress * try now * don't create bloom inside rlpDecode * don't create bloom inside ApplyTransaction * clean * clean * clean * clean * clean * clean * clean * clean * rename method * print timings * print timings * print timings * sort before flush * fix err lint * clean * move tests to transactions * compressed version * up bound * up bound * more tests * more tests * more tests * more tests * better removal * clean * better performance of get/put methods * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * optimize rpcdaemon * fix test * fix rpcdaemon * fix test * simplify * simplify * fix nil pointer * clean * revert some changes * add some logs * clean * try without optimize * clean * clean * clean * clean * try * move log_index to own stage * move log_index to own stage * integration add log_index stage * integration add log_index stage * clean * clean * print timing * remove duplicates at unwind * extract truncateBitmaps func * try detect * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * clean * add blackList of topics * clean * clean * clean * clean * clean * clean * clean * clean * sharding 1 * sharded 2 * sharded 2 * sharded 2 * sharded 2 * sharded 2 * sharded 2 * sharded 2 * sharded 2 * sharded 2 * sharded 2 * sharded 2 * sharded 2 * sharded 2 * sharded 2 * sharded 2 * sharded 2 * sharded 3 * sharded 3 * sharded 3 * speedup things by putCurrent and putReserve * clean * optimize trim * clean * remove blacklist * add more info to err * ? * clean * clean * clean * clean * clean * working version * switch to cgo version of roaring bitmaps * clean * clean * clean * clean * more docs * clean * clean * fix logs bloom field * Fix debug_getModifiedAccountsByNumber * Try to fix crash * fix problem with "absent block" * fix problem with "absent block" * remove optimize method call * remove roaring iterator * fix problem with rebuild indicess * remove debug prints * tests for eth_getLogs involving topics * add tests for new stage, speparate topics into 2 buckets * version up * remove debug logs * remove debug logs * remove bloom filter implementation * Optimisation * Optimisatin not required, make rpctest lenient to geth errors * Lenient to geth failures Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-09-28 17:18:36 +00:00
LogTopicIndex,
LogAddressIndex,
SnapshotInfoBucket,
CallTraceSet,
CallFromIndex,
CallToIndex,
Log,
Sequence,
EthTx,
TrieOfAccountsBucket,
TrieOfStorageBucket,
HashedAccountsBucket,
HashedStorageBucket,
BittorrentInfoBucket,
HeaderCanonicalBucket,
HeadersBucket,
HeaderTDBucket,
}
// DeprecatedBuckets - list of buckets which can be programmatically deleted - for example after migration
2020-08-10 23:55:32 +00:00
var DeprecatedBuckets = []string{
HeaderPrefixOld,
CliqueBucket,
Intermediate hash phase 3 (#377) * #remove debug prints * remove storage-mode="i" * minnet re-execute hack with checkpoints * minnet re-execute hack with checkpoints * rollback to master setup * mainnet re-exec hack * rollback some changes * v0 of "push down" functionality * move all logic to own functions * handle case when re-created account already has some storage * clear path for storage * try to rely on tree structure (but maybe need to rely on DB because can be intra-block re-creations of account) * fix some bugs with indexes, moving to tests * tests added * make linter happy * make linter happy * simplify logic * adjust comparison of keys with and without incarnation * test for keyIsBefore * test for keyIsBefore * better nibbles alignment * better nibbles alignment * cleanup * continue work on tests * simplify test * check tombstone existence before pushing it down. * put tombstone only when account deleted, not created * put tombstone only when account has storage * make linter happy * test for storage resolver * make fixedbytes work without incarnation * fix panic on short keys * use special comparison only when working with keys from cache * add blockNr for better tracing * fix: incorrect tombstone check * fix: incorrect tombstone check * trigger ci * hack for problem block * more test-cases * add test case for too long keys * speedup cached resolver by removing bucket creation transaction * remove parent type check in pruning, remove unused copy from mutation.put * dump resolving info on fail * dump resolving info on fail * set tombstone everytime for now to check if it will help * on unload: check parent type, not type of node * fix wrong order of checking node type * fix wrong order of checking node type * rebase to new master * make linter happy * rebase to new master * place tombstone only if acc has storage * rebase master * rebase master * rebase master * rebase master Co-authored-by: alex.sharov <alex.sharov@lazada.com>
2020-03-11 10:31:49 +00:00
}
type CmpFunc func(k1, k2, v1, v2 []byte) int
type BucketsCfg map[string]BucketConfigItem
type Bucket string
2020-10-28 03:18:10 +00:00
type DBI uint
type BucketFlags uint
const (
Default BucketFlags = 0x00
ReverseKey BucketFlags = 0x02
DupSort BucketFlags = 0x04
IntegerKey BucketFlags = 0x08
IntegerDup BucketFlags = 0x20
ReverseDup BucketFlags = 0x40
)
type BucketConfigItem struct {
2020-10-28 03:18:10 +00:00
Flags BucketFlags
// AutoDupSortKeysConversion - enables some keys transformation - to change db layout without changing app code.
// Use it wisely - it helps to do experiments with DB format faster, but better reduce amount of Magic in app.
// If good DB format found, push app code to accept this format and then disable this property.
AutoDupSortKeysConversion bool
IsDeprecated bool
2020-10-28 03:18:10 +00:00
DBI DBI
// DupFromLen - if user provide key of this length, then next transformation applied:
// v = append(k[DupToLen:], v...)
// k = k[:DupToLen]
// And opposite at retrieval
// Works only if AutoDupSortKeysConversion enabled
2021-06-30 03:31:36 +00:00
DupFromLen int
DupToLen int
}
var BucketsConfigs = BucketsCfg{
HashedStorageBucket: {
2020-10-28 03:18:10 +00:00
Flags: DupSort,
AutoDupSortKeysConversion: true,
DupFromLen: 72,
DupToLen: 40,
},
AccountChangeSetBucket: {
ChangeSets dupsort (#1342) * change_set_dup * change_set_dup * change_set_dup * change_set_dup * change_set_dup * change_set_dup * change_set_dup * change_set_dup * change_set_dup * change_set_dup * change_set_dup * change_set_dup * change_set_dup * change_set_dup * change_set_dup * change_set_dup * change_set_dup * change_set_dup * change_set_dup * change_set_dup * change_set_dup * working version * working version * working version * working version * working version * working version * working version * working version * working version * working version * working version * working version * working version * working version * working version * working version * working version * working version * working version * working version * working version * working version * working version * working version * working version * working version * working version * working version * working version * working version * working version * working version * working version * working version * working version * working version * working version * working version * working version * working version * working version * working version * working version * working version * aa * aa * aa * aa * aa * aa * aa * aa * aa * aa * aa * aa * aa * aa * aa * squash * squash * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * history_early_stop * history_early_stop * vmConfig with ReadOnly false * auto_increment * auto_increment * rebase master Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-11-16 12:08:28 +00:00
Flags: DupSort,
},
StorageChangeSetBucket: {
ChangeSets dupsort (#1342) * change_set_dup * change_set_dup * change_set_dup * change_set_dup * change_set_dup * change_set_dup * change_set_dup * change_set_dup * change_set_dup * change_set_dup * change_set_dup * change_set_dup * change_set_dup * change_set_dup * change_set_dup * change_set_dup * change_set_dup * change_set_dup * change_set_dup * change_set_dup * change_set_dup * working version * working version * working version * working version * working version * working version * working version * working version * working version * working version * working version * working version * working version * working version * working version * working version * working version * working version * working version * working version * working version * working version * working version * working version * working version * working version * working version * working version * working version * working version * working version * working version * working version * working version * working version * working version * working version * working version * working version * working version * working version * working version * working version * working version * aa * aa * aa * aa * aa * aa * aa * aa * aa * aa * aa * aa * aa * aa * aa * squash * squash * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * fix * history_early_stop * history_early_stop * vmConfig with ReadOnly false * auto_increment * auto_increment * rebase master Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-11-16 12:08:28 +00:00
Flags: DupSort,
},
PlainStateBucket: {
2020-10-28 03:18:10 +00:00
Flags: DupSort,
AutoDupSortKeysConversion: true,
DupFromLen: 60,
DupToLen: 28,
},
CallTraceSet: {
Flags: DupSort,
},
}
func sortBuckets() {
sort.SliceStable(Buckets, func(i, j int) bool {
2020-08-10 23:55:32 +00:00
return strings.Compare(Buckets[i], Buckets[j]) < 0
})
}
func DefaultBuckets() BucketsCfg {
return BucketsConfigs
}
func UpdateBucketsList(newBucketCfg BucketsCfg) {
newBuckets := make([]string, 0)
for k, v := range newBucketCfg {
if !v.IsDeprecated {
newBuckets = append(newBuckets, k)
}
}
Buckets = newBuckets
BucketsConfigs = newBucketCfg
reinit()
}
func init() {
reinit()
}
func reinit() {
sortBuckets()
for _, name := range Buckets {
_, ok := BucketsConfigs[name]
if !ok {
BucketsConfigs[name] = BucketConfigItem{}
}
}
for _, name := range DeprecatedBuckets {
_, ok := BucketsConfigs[name]
if !ok {
BucketsConfigs[name] = BucketConfigItem{}
}
tmp := BucketsConfigs[name]
tmp.IsDeprecated = true
BucketsConfigs[name] = tmp
}
}