prysm-pulse/shared/bytesutil/bytes.go
Radosław Kapka 12403d249f
[Feature] - API Middleware (#8926)
* HTTP proxy server for Eth2 APIs (#8904)

* Implement API HTTP proxy server

* cleanup + more comments

* gateway will no longer be dependent on beaconv1

* handle error during ErrorJson type assertion

* simplify handling of endpoint data

* fix mux v1 route

* use URL encoding for all requests

* comment fieldProcessor

* fix failing test

* change proxy port to not interfere with e2e

* gzl

* simplify conditional expression

* Move appending custom error header to grpcutils package

* add api-middleware-port flag

* fix documentation for processField

* modify e2e port

* change field processing error message

* better error message for field processing

* simplify base64ToHexProcessor

* fix json structs

* Run several new endpoints through API middleware (#8922)

* Implement API HTTP proxy server

* cleanup + more comments

* gateway will no longer be dependent on beaconv1

* handle error during ErrorJson type assertion

* simplify handling of endpoint data

* fix mux v1 route

* use URL encoding for all requests

* comment fieldProcessor

* fix failing test

* change proxy port to not interfere with e2e

* gzl

* simplify conditional expression

* Move appending custom error header to grpcutils package

* add api-middleware-port flag

* fix documentation for processField

* modify e2e port

* change field processing error message

* better error message for field processing

* simplify base64ToHexProcessor

* fix json structs

* /eth/v1/beacon/states/{state_id}/validators

* /eth/v1/beacon/states/{state_id}/validators/{validator_id}

* /eth/v1/beacon/states/{state_id}/validator_balances

* /eth/v1/beacon/states/{state_id}/committees

* allow skipping base64-encoding for query params

* /eth/v1/beacon/pool/attestations

* replace break with continue

* Remove unused functions (#8924)

Co-authored-by: terence tsao <terence@prysmaticlabs.com>

* Process SSZ-serialized beacon state through API middleware (#8925)

* update field names

* Process SSZ-serialized beacon state through API middleware

* revert changes to go.mod and go.sum

* Revert "Merge branch '__develop' into feature/api-middleware"

This reverts commit 7c739a8fd71e2c1e3a14be85abd29a59b57ae9b5, reversing
changes made to 2d0f8e012ecb006888ed8e826b45625a3edc2eeb.

* update ethereumapis

* update validator field name

* update deps.bzl

* update json tags (#8942)

* Run `/node/syncing` through API Middleware (#8944)

* add IsSyncing field to grpc response

* run /node/syncing through the middleware

Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>

* Return HTTP status codes other than 200 and 500 from node and debug endpoints (#8937)

* error codes for node endpoints

* error codes for debug endpoints

* better comment about headers

* gzl

* review comments

* comment on return value

* update fakeChecker used for fuzz tests

* fix failing tests

* Allow to pass URL params literally, without encoding to base64 (#8938)

* Allow to pass URL params literally, without encoding to base64

* fix compile error

Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>

* Process SSZ-serialized beacon state through API middleware (#8925)

* update field names

* Process SSZ-serialized beacon state through API middleware

* revert changes to go.mod and go.sum

* Revert "Merge branch '__develop' into feature/api-middleware"

This reverts commit 7c739a8fd71e2c1e3a14be85abd29a59b57ae9b5, reversing
changes made to 2d0f8e012ecb006888ed8e826b45625a3edc2eeb.

* update ethereumapis

* update validator field name

* update deps.bzl

* update json tags (#8942)

* Run `/node/syncing` through API Middleware (#8944)

* add IsSyncing field to grpc response

* run /node/syncing through the middleware

Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>

* Return HTTP status codes other than 200 and 500 from node and debug endpoints (#8937)

* error codes for node endpoints

* error codes for debug endpoints

* better comment about headers

* gzl

* review comments

* comment on return value

* update fakeChecker used for fuzz tests

* fix failing tests

* Allow to pass URL params literally, without encoding to base64 (#8938)

* Allow to pass URL params literally, without encoding to base64

* fix compile error

Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>

* unused import

* Return correct status codes from beacon endpoints (#8960)

* Various API Middleware fixes (#8963)

* Return correct status codes from `/states` endpoints

* better error messages in debug and node

* better error messages in state

* returning correct error codes from validator endpoints

* correct error codes for getting a block header

* gzl

* fix err variable name

* fix nil block comparison

* test fixes

* make status enum test better

* fix ineffectual assignment

* make PR unstuck

* return proper status codes

* return uppercase keys from /config/spec

* return lowercase validator status

* convert requested enum values to uppercase

* validator fixes

* Implement `/beacon/headers` endpoint (#8966)

* Refactor API Middleware into more manageable code  (#8984)

* move endpoint registration out of shared package

* divide main function into smaller components

* return early on error

* implement hooks

* implement custom handlers and add documentation

* fix test compile error

* restrict package visibility

* remove redundant error checking

* rename file

* API Middleware unit tests (#8998)

* move endpoint registration out of shared package

* divide main function into smaller components

* return early on error

* implement hooks

* implement custom handlers and add documentation

* fix test compile error

* restrict package visibility

* remove redundant error checking

* rename file

* api_middleware_processing

* endpoints

* gzl

* remove gazelle:ignore

* merge

* Implement SSZ version of `/blocks/{block_id}` (#8970)

* Implement SSZ version of `/blocks/{block_id}`

* add dependencies back

* fix indentation in deps.bzl

* parameterize ssz functions

* get block ssz

* update ethereumapis dependency

* gzl

* Do not reuse `Endpoint` structs between API calls (#9007)

* code refactor

* implement endpoint factory

* fix test

* fmt

* include pbs

* gaz

* test naming fixes

* remove unused code

* radek comments

* revert endpoint test

* bring back bytes test case

* move `signedBeaconBlock` to `migration` package

* change `fmt.Errorf` to `errors.Wrap`

* capitalize SSZ

* capitalize URL

* more review feedback

* rename `handleGetBlockSSZ` to `handleGetBeaconBlockSSZ`

* rename `IndexOutOfRangeError` to `ValidatorIndexOutOfRangeError`

* simplify parameter names

* test header

* more corrections

* properly allocate array capacity

Co-authored-by: terence tsao <terence@prysmaticlabs.com>
Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
Co-authored-by: Nishant Das <nishdas93@gmail.com>
2021-06-15 10:28:49 -05:00

343 lines
8.5 KiB
Go

// Package bytesutil defines helper methods for converting integers to byte slices.
package bytesutil
import (
"encoding/binary"
"errors"
"math/bits"
"regexp"
types "github.com/prysmaticlabs/eth2-types"
)
// ToBytes returns integer x to bytes in little-endian format at the specified length.
// Spec defines similar method uint_to_bytes(n: uint) -> bytes, which is equivalent to ToBytes(n, 8).
func ToBytes(x uint64, length int) []byte {
makeLength := length
if length < 8 {
makeLength = 8
}
bytes := make([]byte, makeLength)
binary.LittleEndian.PutUint64(bytes, x)
return bytes[:length]
}
// Bytes1 returns integer x to bytes in little-endian format, x.to_bytes(1, 'little').
func Bytes1(x uint64) []byte {
bytes := make([]byte, 8)
binary.LittleEndian.PutUint64(bytes, x)
return bytes[:1]
}
// Bytes2 returns integer x to bytes in little-endian format, x.to_bytes(2, 'little').
func Bytes2(x uint64) []byte {
bytes := make([]byte, 8)
binary.LittleEndian.PutUint64(bytes, x)
return bytes[:2]
}
// Bytes3 returns integer x to bytes in little-endian format, x.to_bytes(3, 'little').
func Bytes3(x uint64) []byte {
bytes := make([]byte, 8)
binary.LittleEndian.PutUint64(bytes, x)
return bytes[:3]
}
// Bytes4 returns integer x to bytes in little-endian format, x.to_bytes(4, 'little').
func Bytes4(x uint64) []byte {
bytes := make([]byte, 8)
binary.LittleEndian.PutUint64(bytes, x)
return bytes[:4]
}
// Bytes8 returns integer x to bytes in little-endian format, x.to_bytes(8, 'little').
func Bytes8(x uint64) []byte {
bytes := make([]byte, 8)
binary.LittleEndian.PutUint64(bytes, x)
return bytes
}
// Bytes32 returns integer x to bytes in little-endian format, x.to_bytes(32, 'little').
func Bytes32(x uint64) []byte {
bytes := make([]byte, 32)
binary.LittleEndian.PutUint64(bytes, x)
return bytes
}
// FromBytes4 returns an integer which is stored in the little-endian format(4, 'little')
// from a byte array.
func FromBytes4(x []byte) uint64 {
empty4bytes := make([]byte, 4)
return binary.LittleEndian.Uint64(append(x[:4], empty4bytes...))
}
// FromBytes8 returns an integer which is stored in the little-endian format(8, 'little')
// from a byte array.
func FromBytes8(x []byte) uint64 {
return binary.LittleEndian.Uint64(x)
}
// ToBytes4 is a convenience method for converting a byte slice to a fix
// sized 4 byte array. This method will truncate the input if it is larger
// than 4 bytes.
func ToBytes4(x []byte) [4]byte {
var y [4]byte
copy(y[:], x)
return y
}
// ToBytes32 is a convenience method for converting a byte slice to a fix
// sized 32 byte array. This method will truncate the input if it is larger
// than 32 bytes.
func ToBytes32(x []byte) [32]byte {
var y [32]byte
copy(y[:], x)
return y
}
// ToBytes48 is a convenience method for converting a byte slice to a fix
// sized 48 byte array. This method will truncate the input if it is larger
// than 48 bytes.
func ToBytes48(x []byte) [48]byte {
var y [48]byte
copy(y[:], x)
return y
}
// ToBytes64 is a convenience method for converting a byte slice to a fix
// sized 64 byte array. This method will truncate the input if it is larger
// than 64 bytes.
func ToBytes64(x []byte) [64]byte {
var y [64]byte
copy(y[:], x)
return y
}
// ToBool is a convenience method for converting a byte to a bool.
// This method will use the first bit of the 0 byte to generate the returned value.
func ToBool(x byte) bool {
return x&1 == 1
}
// FromBytes2 returns an integer which is stored in the little-endian format(2, 'little')
// from a byte array.
func FromBytes2(x []byte) uint16 {
return binary.LittleEndian.Uint16(x[:2])
}
// FromBool is a convenience method for converting a bool to a byte.
// This method will use the first bit to generate the returned value.
func FromBool(x bool) byte {
if x {
return 1
}
return 0
}
// FromBytes48 is a convenience method for converting a fixed-size byte array
// to a byte slice.
func FromBytes48(x [48]byte) []byte {
return x[:]
}
// FromBytes48Array is a convenience method for converting an array of
// fixed-size byte arrays to an array of byte slices.
func FromBytes48Array(x [][48]byte) [][]byte {
y := make([][]byte, len(x))
for i := range x {
y[i] = x[i][:]
}
return y
}
// Trunc truncates the byte slices to 6 bytes.
func Trunc(x []byte) []byte {
if len(x) > 6 {
return x[:6]
}
return x
}
// ToLowInt64 returns the lowest 8 bytes interpreted as little endian.
func ToLowInt64(x []byte) int64 {
if len(x) > 8 {
x = x[:8]
}
return int64(binary.LittleEndian.Uint64(x))
}
// SafeCopyBytes will copy and return a non-nil byte array, otherwise it returns nil.
func SafeCopyBytes(cp []byte) []byte {
if cp != nil {
copied := make([]byte, len(cp))
copy(copied, cp)
return copied
}
return nil
}
// Copy2dBytes will copy and return a non-nil 2d byte array, otherwise it returns nil.
func Copy2dBytes(ary [][]byte) [][]byte {
if ary != nil {
copied := make([][]byte, len(ary))
for i, a := range ary {
copied[i] = SafeCopyBytes(a)
}
return copied
}
return nil
}
// ReverseBytes32Slice will reverse the provided slice's order.
func ReverseBytes32Slice(arr [][32]byte) [][32]byte {
for i, j := 0, len(arr)-1; i < j; i, j = i+1, j-1 {
arr[i], arr[j] = arr[j], arr[i]
}
return arr
}
// PadTo pads a byte slice to the given size. If the byte slice is larger than the given size, the
// original slice is returned.
func PadTo(b []byte, size int) []byte {
if len(b) > size {
return b
}
return append(b, make([]byte, size-len(b))...)
}
// SetBit sets the index `i` of bitlist `b` to 1.
// It grows and returns a longer bitlist with 1 set
// if index `i` is out of range.
func SetBit(b []byte, i int) []byte {
if i >= len(b)*8 {
h := (i + (8 - i%8)) / 8
b = append(b, make([]byte, h-len(b))...)
}
bit := uint8(1 << (i % 8))
b[i/8] |= bit
return b
}
// ClearBit clears the index `i` of bitlist `b`.
// Returns the original bitlist if the index `i`
// is out of range.
func ClearBit(b []byte, i int) []byte {
if i >= len(b)*8 {
return b
}
bit := uint8(1 << (i % 8))
b[i/8] &^= bit
return b
}
// MakeEmptyBitlists returns an empty bitlist with
// input size `i`.
func MakeEmptyBitlists(i int) []byte {
return make([]byte, (i+(8-i%8))/8)
}
// HighestBitIndex returns the index of the highest
// bit set from bitlist `b`.
func HighestBitIndex(b []byte) (int, error) {
if len(b) == 0 {
return 0, errors.New("input list can't be empty or nil")
}
for i := len(b) - 1; i >= 0; i-- {
if b[i] == 0 {
continue
}
return bits.Len8(b[i]) + (i * 8), nil
}
return 0, nil
}
// HighestBitIndexAt returns the index of the highest
// bit set from bitlist `b` that is at `index` (inclusive).
func HighestBitIndexAt(b []byte, index int) (int, error) {
bLength := len(b)
if b == nil || bLength == 0 {
return 0, errors.New("input list can't be empty or nil")
}
start := index / 8
if start >= bLength {
start = bLength - 1
}
mask := byte(1<<(index%8) - 1)
for i := start; i >= 0; i-- {
if index/8 > i {
mask = 0xff
}
masked := b[i] & mask
minBitsMasked := bits.Len8(masked)
if b[i] == 0 || (minBitsMasked == 0 && index/8 <= i) {
continue
}
return minBitsMasked + (i * 8), nil
}
return 0, nil
}
// Uint64ToBytesLittleEndian conversion.
func Uint64ToBytesLittleEndian(i uint64) []byte {
buf := make([]byte, 8)
binary.LittleEndian.PutUint64(buf, i)
return buf
}
// Uint64ToBytesBigEndian conversion.
func Uint64ToBytesBigEndian(i uint64) []byte {
buf := make([]byte, 8)
binary.BigEndian.PutUint64(buf, i)
return buf
}
// BytesToUint64BigEndian conversion. Returns 0 if empty bytes or byte slice with length less
// than 8.
func BytesToUint64BigEndian(b []byte) uint64 {
if len(b) < 8 { // This will panic otherwise.
return 0
}
return binary.BigEndian.Uint64(b)
}
// EpochToBytesLittleEndian conversion.
func EpochToBytesLittleEndian(i types.Epoch) []byte {
return Uint64ToBytesLittleEndian(uint64(i))
}
// EpochToBytesBigEndian conversion.
func EpochToBytesBigEndian(i types.Epoch) []byte {
return Uint64ToBytesBigEndian(uint64(i))
}
// BytesToEpochBigEndian conversion.
func BytesToEpochBigEndian(b []byte) types.Epoch {
return types.Epoch(BytesToUint64BigEndian(b))
}
// SlotToBytesBigEndian conversion.
func SlotToBytesBigEndian(i types.Slot) []byte {
return Uint64ToBytesBigEndian(uint64(i))
}
// BytesToSlotBigEndian conversion.
func BytesToSlotBigEndian(b []byte) types.Slot {
return types.Slot(BytesToUint64BigEndian(b))
}
// IsHex checks whether the byte array is a hex number prefixed with '0x'.
func IsHex(b []byte) (bool, error) {
if b == nil {
return false, nil
}
return regexp.Match("^(0x)[0-9a-fA-F]+$", b)
}