Skip to main content

y.go

y.go - Overview​

This file provides a collection of utility functions and data structures used throughout the Badger database. It includes functionalities for file operations, byte slice manipulation, key handling, throttling, data conversion, and rate monitoring.

Detailed Documentation​

Constants​

  • ErrEOF: Indicates an end of file when trying to read from a memory mapped file.
  • ErrCommitAfterFinish: Indicates that write batch commit was called after finish.
  • Sync: Flag indicating that O_DSYNC should be set on the underlying file.
  • ReadOnly: Flag to open the file in read-only mode.
  • datasyncFileFlag: Holds the O_DSYNC (datasync) flag for platforms that support it.
  • CastagnoliCrcTable: A CRC32 polynomial table.

Type: Flags​

type Flags int

Represents flags used when opening files.

Function: OpenExistingFile​

func OpenExistingFile(filename string, flags Flags) (*os.File, error)

Opens an existing file. Errors if the file does not exist.

  • Parameters:
    • filename: The name of the file to open.
    • flags: Flags to control how the file is opened (e.g., read-only, sync).
  • Returns:
    • *os.File: A pointer to the opened file.
    • error: An error if the file cannot be opened.

Function: CreateSyncedFile​

func CreateSyncedFile(filename string, sync bool) (*os.File, error)

Creates a new file (using O_EXCL). Errors if the file already exists.

  • Parameters:
    • filename: The name of the file to create.
    • sync: A boolean indicating whether to open the file with the O_DSYNC flag.
  • Returns:
    • *os.File: A pointer to the created file.
    • error: An error if the file cannot be created.

Function: OpenSyncedFile​

func OpenSyncedFile(filename string, sync bool) (*os.File, error)

Creates the file if one doesn't exist.

  • Parameters:
    • filename: The name of the file to open or create.
    • sync: A boolean indicating whether to open the file with the O_DSYNC flag.
  • Returns:
    • *os.File: A pointer to the opened or created file.
    • error: An error if the file cannot be opened or created.

Function: OpenTruncFile​

func OpenTruncFile(filename string, sync bool) (*os.File, error)

Opens the file with O_RDWR | O_CREATE | O_TRUNC.

  • Parameters:
    • filename: The name of the file to open or create.
    • sync: A boolean indicating whether to open the file with the O_DSYNC flag.
  • Returns:
    • *os.File: A pointer to the opened or created file.
    • error: An error if the file cannot be opened or created.

Function: SafeCopy​

func SafeCopy(a, src []byte) []byte

Copies a byte slice src to a without allocating new memory if a has sufficient capacity.

  • Parameters:
    • a: The destination byte slice.
    • src: The source byte slice.
  • Returns:
    • []byte: The modified destination byte slice a containing a copy of src.

Function: Copy​

func Copy(a []byte) []byte

Copies a byte slice and returns the copied slice.

  • Parameters:
    • a: The byte slice to copy.
  • Returns:
    • []byte: A new byte slice containing a copy of a.

Function: KeyWithTs​

func KeyWithTs(key []byte, ts uint64) []byte

Generates a new key by appending the timestamp ts to the given key.

  • Parameters:
    • key: The original key.
    • ts: The timestamp to append.
  • Returns:
    • []byte: The new key with the timestamp appended.

Function: ParseTs​

func ParseTs(key []byte) uint64

Parses the timestamp from the key bytes.

  • Parameters:
    • key: The key containing the timestamp.
  • Returns:
    • uint64: The parsed timestamp.

Function: CompareKeys​

func CompareKeys(key1, key2 []byte) int

Compares two keys, considering both the key and the timestamp.

  • Parameters:
    • key1: The first key.
    • key2: The second key.
  • Returns:
    • int: An integer indicating the comparison result (-1, 0, or 1).

Function: ParseKey​

func ParseKey(key []byte) []byte

Parses the actual key from the key bytes by removing the timestamp.

  • Parameters:
    • key: The key containing the timestamp.
  • Returns:
    • []byte: The key without the timestamp.

Function: SameKey​

func SameKey(src, dst []byte) bool

Checks for key equality ignoring the version timestamp suffix.

  • Parameters:
    • src: The first key.
    • dst: The second key.
  • Returns:
    • bool: True if the keys are equal (ignoring timestamps), false otherwise.

Type: Slice​

type Slice struct {
	buf []byte
}

Holds a reusable buffer.

Function: Resize​

func (s *Slice) Resize(sz int) []byte

Resizes the Slice's buffer (or makes a new one) and returns a slice in that buffer of length sz.

  • Parameters:
    • sz: The desired size of the slice.
  • Returns:
    • []byte: A slice of length sz backed by the Slice's buffer.

Function: FixedDuration​

func FixedDuration(d time.Duration) string

Returns a string representation of the given duration with the hours, minutes, and seconds.

  • Parameters:
    • d: The duration to format.
  • Returns:
    • string: A formatted string representing the duration.

Type: Throttle​

type Throttle struct {
	once      sync.Once
	wg        sync.WaitGroup
	ch        chan struct{}
	errCh     chan error
	finishErr error
}

Throttle allows a limited number of workers to run at a time.

Function: NewThrottle​

func NewThrottle(max int) *Throttle

Creates a new throttle with a max number of workers.

  • Parameters:
    • max: The maximum number of concurrent workers.
  • Returns:
    • *Throttle: A pointer to the new Throttle object.

Function: Do​

func (t *Throttle) Do() error

Should be called by workers before they start working. It blocks if the maximum number of workers are already working.

  • Returns:
    • error: An error if any previous worker has encountered an error.

Function: Done​

func (t *Throttle) Done(err error)

Should be called by workers when they finish working.

  • Parameters:
    • err: An error encountered during the worker's execution, or nil if no error occurred.

Function: Finish​

func (t *Throttle) Finish() error

Waits until all workers have finished working. It would return any error passed by Done.

  • Returns:
    • error: An error if any worker has encountered an error.

Function: U16ToBytes​

func U16ToBytes(v uint16) []byte

Converts the given Uint16 to bytes.

  • Parameters:
    • v: The uint16 value to convert.
  • Returns:
    • []byte: A byte slice representing the uint16 value.

Function: BytesToU16​

func BytesToU16(b []byte) uint16

Converts the given byte slice to uint16.

  • Parameters:
    • b: The byte slice to convert.
  • Returns:
    • uint16: The uint16 value represented by the byte slice.

Function: U32ToBytes​

func U32ToBytes(v uint32) []byte

Converts the given Uint32 to bytes.

  • Parameters:
    • v: The uint32 value to convert.
  • Returns:
    • []byte: A byte slice representing the uint32 value.

Function: BytesToU32​

func BytesToU32(b []byte) uint32

Converts the given byte slice to uint32.

  • Parameters:
    • b: The byte slice to convert.
  • Returns:
    • uint32: The uint32 value represented by the byte slice.

Function: U32SliceToBytes​

func U32SliceToBytes(u32s []uint32) []byte

Converts the given Uint32 slice to byte slice.

  • Parameters:
    • u32s: The uint32 slice to convert.
  • Returns:
    • []byte: A byte slice representing the uint32 slice.

Function: BytesToU32Slice​

func BytesToU32Slice(b []byte) []uint32

Converts the given byte slice to uint32 slice.

  • Parameters:
    • b: The byte slice to convert.
  • Returns:
    • []uint32: A uint32 slice represented by the byte slice.

Function: U64ToBytes​

func U64ToBytes(v uint64) []byte

Converts the given Uint64 to bytes.

  • Parameters:
    • v: The uint64 value to convert.
  • Returns:
    • []byte: A byte slice representing the uint64 value.

Function: BytesToU64​

func BytesToU64(b []byte) uint64

Converts the given byte slice to uint64.

  • Parameters:
    • b: The byte slice to convert.
  • Returns:
    • uint64: The uint64 value represented by the byte slice.

Function: U64SliceToBytes​

func U64SliceToBytes(u64s []uint64) []byte

Converts the given Uint64 slice to byte slice.

  • Parameters:
    • u64s: The uint64 slice to convert.
  • Returns:
    • []byte: A byte slice representing the uint64 slice.

Function: BytesToU64Slice​

func BytesToU64Slice(b []byte) []uint64

Converts the given byte slice to uint64 slice.

  • Parameters:
    • b: The byte slice to convert.
  • Returns:
    • []uint64: A uint64 slice represented by the byte slice.

Type: page​

type page struct {
	buf []byte
}

page struct contains one underlying buffer.

Type: PageBuffer​

type PageBuffer struct {
	pages []*page

	length       int // Length of PageBuffer.
	nextPageSize int // Size of next page to be allocated.
}

PageBuffer consists of many pages. A page is a wrapper over []byte. PageBuffer can act as a replacement of bytes.Buffer.

Function: NewPageBuffer​

func NewPageBuffer(pageSize int) *PageBuffer

Returns a new PageBuffer with the first page having size pageSize.

  • Parameters:
    • pageSize: The initial size of the first page.
  • Returns:
    • *PageBuffer: A pointer to the new PageBuffer object.

Function: Write​

func (b *PageBuffer) Write(data []byte) (int, error)

Writes data to PageBuffer b.

  • Parameters:
    • data: The byte slice to write.
  • Returns:
    • int: The number of bytes written.
    • error: An error, if any.

Function: WriteByte​

func (b *PageBuffer) WriteByte(data byte) error

Writes a single byte to PageBuffer.

  • Parameters:
    • data: The byte to write.
  • Returns:
    • error: An error, if any.

Function: Len​

func (b *PageBuffer) Len() int

Returns the length of the PageBuffer.

  • Returns:
    • int: The number of bytes currently stored in the PageBuffer.

Function: pageForOffset​

func (b *PageBuffer) pageForOffset(offset int) (int, int)

Returns pageIdx and startIdx for the offset.

  • Parameters:
    • offset: The offset into the PageBuffer.
  • Returns:
    • int: The index of the page containing the offset.
    • int: The index within the page where the offset is located.

Function: Truncate​

func (b *PageBuffer) Truncate(n int)

Truncates PageBuffer to length n.

  • Parameters:
    • n: The new length of the PageBuffer.

Function: Bytes​

func (b *PageBuffer) Bytes() []byte

Returns the whole Buffer data as a single []byte.

  • Returns:
    • []byte: A new byte slice containing all the data in the PageBuffer.

Function: WriteTo​

func (b *PageBuffer) WriteTo(w io.Writer) (int64, error)

Writes the whole buffer to w.

  • Parameters:
    • w: The io.Writer to write to.
  • Returns:
    • int64: The number of bytes written.
    • error: An error, if any.

Function: NewReaderAt​

func (b *PageBuffer) NewReaderAt(offset int) *PageBufferReader

Returns a reader which starts reading from offset in page buffer.

  • Parameters:
    • offset: The offset to start reading from.
  • Returns:
    • *PageBufferReader: A pointer to the new PageBufferReader object.

Type: PageBufferReader​

type PageBufferReader struct {
	buf      *PageBuffer // Underlying page buffer.
	pageIdx  int         // Idx of page from where it will start reading.
	startIdx int         // Idx inside page - buf.pages[pageIdx] from where it will start reading.
}

PageBufferReader is a reader for PageBuffer.

Function: Read​

func (r *PageBufferReader) Read(p []byte) (int, error)

Reads up to len(p) bytes.

  • Parameters:
    • p: The byte slice to read into.
  • Returns:
    • int: The number of bytes read.
    • error: An error, if any.

Function: NewKV​

func NewKV(alloc *z.Allocator) *pb.KV

Creates a new pb.KV object, allocating memory from the provided allocator if available.

  • Parameters:
    • alloc: An optional z.Allocator for memory allocation.
  • Returns:
    • *pb.KV: A pointer to the new pb.KV object.

Function: IBytesToString​

func IBytesToString(size uint64, precision int) string

Converts size in bytes to human-readable format.

  • Parameters:
    • size: The size in bytes.
    • precision: The decimal precision to use in the output string.
  • Returns:
    • string: A human-readable string representing the size.

Type: RateMonitor​

type RateMonitor struct {
	start       time.Time
	lastSent    uint64
	lastCapture time.Time
	rates       []float64
	idx         int
}

The RateMonitor struct is used to monitor the rate of data transmission. It keeps track of the number of bytes sent over time and calculates the average rate.

Function: NewRateMonitor​

func NewRateMonitor(numSamples int) *RateMonitor

Creates a new RateMonitor.

  • Parameters:
    • numSamples: The number of samples to use when smoothing the rate.
  • Returns:
    • *RateMonitor: A pointer to the new RateMonitor object.

Function: Capture​

func (rm *RateMonitor) Capture(sent uint64)

Captures the current number of sent bytes.

  • Parameters:
    • sent: The current number of sent bytes. This number should be monotonically increasing.

Function: Rate​

func (rm *RateMonitor) Rate() uint64

Returns the average rate of transmission smoothed out by the number of samples.

  • Returns:
    • uint64: The average rate of transmission in bytes per second.

Code Examples​

Example of using KeyWithTs and ParseTs:

import { U64ToBytes, BytesToU64 } from './y';

// Convert uint64 to bytes
const uint64Value = 1234567890;
const byteArray = U64ToBytes(uint64Value);
console.log('Uint64 as bytes:', byteArray);

// Convert bytes to uint64
const convertedUint64 = BytesToU64(byteArray);
console.log('Bytes as Uint64:', convertedUint64);

Getting Started Relevance​