Skip to main content

memtable.go

memtable.go - Overview​

This file defines the memTable structure, which manages an in-memory skiplist and a write-ahead log (WAL) for storing data. It handles writing data to both the skiplist and WAL, replaying the WAL on startup to recover the skiplist's state, and managing the lifecycle of the WAL.

Detailed Documentation​

memTable​

type memTable struct {
	sl         *skl.Skiplist
	wal        *logFile
	maxVersion uint64
	opt        Options
	buf        *bytes.Buffer
}
  • Purpose: Represents an in-memory table that stores data in a skiplist and WAL.
  • Fields:
    • sl: A pointer to the skl.Skiplist used for storing key-value pairs in memory.
    • wal: A pointer to the logFile used for writing data to disk.
    • maxVersion: The maximum timestamp encountered in the memTable.
    • opt: The options used to configure the memTable.
    • buf: A buffer used for encoding entries before writing to the WAL.

(*DB) openMemTables​

func (db *DB) openMemTables(opt Options) error
  • Purpose: Opens existing memtables from disk during database initialization. It reads memtable files from the directory, parses their IDs, opens them, replays their WALs to reconstruct the skiplists, and adds them to the database's immutable memtable list (db.imm).
  • Parameters:
    • opt: The database options.
  • Returns: An error if any operation fails, or nil on success.

(*DB) openMemTable​

func (db *DB) openMemTable(fid, flags int) (*memTable, error)
  • Purpose: Opens a specific memtable file by its ID. It initializes a new memTable with a skiplist and a WAL, and replays the WAL to populate the skiplist.
  • Parameters:
    • fid: The file ID of the memtable.
    • flags: The flags to use when opening the file (e.g., os.O_RDWR, os.O_RDONLY).
  • Returns: A pointer to the opened memTable and an error, if any.

(*DB) newMemTable​

func (db *DB) newMemTable() (*memTable, error)
  • Purpose: Creates a new memTable with a new file ID.
  • Returns: A pointer to the new memTable and an error, if any.

(*DB) mtFilePath​

func (db *DB) mtFilePath(fid int) string
  • Purpose: Generates the file path for a memtable file given its ID.
  • Parameters:
    • fid: The file ID of the memtable.
  • Returns: The file path of the memtable.

(*memTable) SyncWAL​

func (mt *memTable) SyncWAL() error
  • Purpose: Synchronizes the WAL to disk.
  • Returns: An error if the sync fails.

(*memTable) isFull​

func (mt *memTable) isFull() bool
  • Purpose: Checks if the memTable is full based on the skiplist size and WAL size.
  • Returns: true if the memTable is full, false otherwise.

(*memTable) Put​

func (mt *memTable) Put(key []byte, value y.ValueStruct) error
  • Purpose: Writes a key-value pair to the memTable. It writes the entry to the WAL and inserts it into the skiplist.
  • Parameters:
    • key: The key to write.
    • value: The value to write, including value, meta, usermeta, and expiresat.
  • Returns: An error if writing to the WAL fails.

(*memTable) UpdateSkipList​

func (mt *memTable) UpdateSkipList() error
  • Purpose: Replays the WAL to update the skiplist. This is typically called during memTable initialization or recovery.
  • Returns: An error if iterating over the WAL fails.

(*memTable) IncrRef​

func (mt *memTable) IncrRef() {
	mt.sl.IncrRef()
}
  • Purpose: Increments the reference count of the underlying skiplist.

(*memTable) DecrRef​

func (mt *memTable) DecrRef() {
	mt.sl.DecrRef()
}
  • Purpose: Decrements the reference count of the underlying skiplist. When the reference count reaches zero, the skiplist is deallocated, and the WAL file is deleted.

(*memTable) replayFunction​

func (mt *memTable) replayFunction(opt Options) func(Entry, valuePointer) error
  • Purpose: Returns a function that replays a WAL entry by inserting it into the skiplist.
  • Parameters:
    • opt: The database options.
  • Returns: A function that takes an Entry and a valuePointer as input and returns an error.

logFile​

type logFile struct {
	*z.MmapFile
	path string
	lock     sync.RWMutex
	fid      uint32
	size     atomic.Uint32
	dataKey  *pb.DataKey
	baseIV   []byte
	registry *KeyRegistry
	writeAt  uint32
	opt      Options
}
  • Purpose: Represents a log file (WAL) on disk.
  • Fields:
    • MmapFile: Embedded mmap file.
    • path: The path to the log file.
    • lock: A read-write mutex to protect access to the file and its memory map.
    • fid: The file ID of the log file.
    • size: Atomic uint32 representing the current size of the log file.
    • dataKey: Encryption key.
    • baseIV: Base initialization vector for encryption.
    • registry: Key registry.
    • writeAt: Atomic uint32 representing the current write offset in the log file.
    • opt: The database options.

(*logFile) Truncate​

func (lf *logFile) Truncate(end int64) error
  • Purpose: Truncates the log file to the specified size.
  • Parameters:
    • end: The new size of the log file.
  • Returns: An error if the truncation fails.

(*logFile) encodeEntry​

func (lf *logFile) encodeEntry(buf *bytes.Buffer, e *Entry, offset uint32) (int, error)
  • Purpose: Encodes an entry to the buffer.
  • Parameters:
    • buf: The buffer to encode the entry into.
    • e: The entry to encode.
    • offset: The offset of the entry in the log file.
  • Returns: The number of bytes written to the buffer and an error, if any.

(*logFile) writeEntry​

func (lf *logFile) writeEntry(buf *bytes.Buffer, e *Entry, opt Options) error
  • Purpose: Writes an entry to the log file.
  • Parameters:
    • buf: A buffer used for encoding the entry.
    • e: The entry to write.
    • opt: The database options.
  • Returns: An error if the write fails.

(*logFile) decodeEntry​

func (lf *logFile) decodeEntry(buf []byte, offset uint32) (*Entry, error)
  • Purpose: Decodes an entry from a byte slice.
  • Parameters:
    • buf: The byte slice to decode.
    • offset: The offset of the entry in the log file.
  • Returns: A pointer to the decoded Entry and an error, if any.

(*logFile) decryptKV​

func (lf *logFile) decryptKV(buf []byte, offset uint32) ([]byte, error)
  • Purpose: Decrypts the key and value in the given buffer.
  • Parameters:
    • buf: The buffer containing the encrypted key and value.
    • offset: The offset of the entry in the log file.
  • Returns: The decrypted key and value as a byte slice, and an error if decryption fails.

(*logFile) keyID​

func (lf *logFile) keyID() uint64
  • Purpose: Returns the ID of the data key used to encrypt the log file.
  • Returns: The data key ID.

(*logFile) encryptionEnabled​

func (lf *logFile) encryptionEnabled() bool
  • Purpose: Checks if encryption is enabled for the log file.
  • Returns: true if encryption is enabled, false otherwise.

(*logFile) read​

func (lf *logFile) read(p valuePointer) (buf []byte, err error)
  • Purpose: Reads data from the log file at the given value pointer.
  • Parameters:
    • p: The value pointer indicating the offset and length of the data to read.
  • Returns: The data read from the log file and an error, if any.

(*logFile) generateIV​

func (lf *logFile) generateIV(offset uint32) []byte
  • Purpose: Generates an initialization vector (IV) for encryption using the base IV and the entry's offset in the log file.
  • Parameters:
    • offset: The offset of the entry in the log file.
  • Returns: The generated IV.

(*logFile) doneWriting​

func (lf *logFile) doneWriting(offset uint32) error
  • Purpose: Called after writing to the log file is complete. It synchronizes the file to disk and truncates it to the written offset.
  • Parameters:
    • offset: The offset to truncate the file to.
  • Returns: An error if syncing or truncating the file fails.

(*logFile) iterate​

func (lf *logFile) iterate(readOnly bool, offset uint32, fn logEntry) (uint32, error)
  • Purpose: Iterates over the entries in the log file, calling the provided function for each entry.
  • Parameters:
    • readOnly: Indicates whether the iteration is read-only.
    • offset: The offset to start iterating from.
    • fn: The function to call for each entry.
  • Returns: The end offset of the iteration and an error, if any.

(*logFile) zeroNextEntry​

func (lf *logFile) zeroNextEntry() {
	z.ZeroOut(lf.Data, int(lf.writeAt), int(lf.writeAt+maxHeaderSize))
}
  • Purpose: Zeros out the next entry in the log file to prevent corruption in case of a crash.

(*logFile) open​

func (lf *logFile) open(path string, flags int, fsize int64) error
  • Purpose: Opens the log file. It either opens an existing file or creates a new one, and initializes the logFile struct.
  • Parameters:
    • path: The path to the log file.
    • flags: The flags to use when opening the file (e.g., os.O_RDWR, os.O_CREATE).
    • fsize: The initial size of the file if it's newly created.
  • Returns: An error if opening the file fails.

(*logFile) bootstrap​

func (lf *logFile) bootstrap() error
  • Purpose: Initializes a new log file with the key ID and base IV.
  • Returns: An error if any operation fails.

Code Examples​

None.

Getting Started Relevance​