Skip to main content

manifest.go

manifest.go - Overview​

This file defines the Manifest struct and related functions for managing the MANIFEST file in a Badger database. The MANIFEST file stores the state of the database, including information about LSM files and their levels. It handles reading, writing, and rewriting the MANIFEST file to ensure data consistency and efficient operation.

Detailed Documentation​

Manifest​

type Manifest struct {
	Levels []levelManifest
	Tables map[uint64]TableManifest

	// Contains total number of creation and deletion changes in the manifest -- used to compute
	// whether it'd be useful to rewrite the manifest.
	Creations int
	Deletions int
}
  • Purpose: Represents the contents of the MANIFEST file, describing the startup state of the database.
    • Levels: Information about LSM tree levels.
    • Tables: Information about specific tables in the LSM tree.
    • Creations: Total number of creation changes in the manifest.
    • Deletions: Total number of deletion changes in the manifest.

createManifest​

func createManifest() Manifest
  • Purpose: Creates and initializes a new Manifest object.
  • Returns: A new Manifest object with initialized Levels and Tables.

levelManifest​

type levelManifest struct {
	Tables map[uint64]struct{} // Set of table id's
}
  • Purpose: Contains information about LSM tree levels in the MANIFEST file.
    • Tables: A set of table IDs present at this level.

TableManifest​

type TableManifest struct {
	Level       uint8
	KeyID       uint64
	Compression options.CompressionType
}
  • Purpose: Contains information about a specific table in the LSM tree.
    • Level: The level of the table in the LSM tree.
    • KeyID: The Key ID of the table.
    • Compression: The compression type used for the table.

manifestFile​

type manifestFile struct {
	fp        *os.File
	directory string

	// The external magic number used by the application running badger.
	externalMagic uint16

	// We make this configurable so that unit tests can hit rewrite() code quickly
	deletionsRewriteThreshold int

	// Guards appends, which includes access to the manifest field.
	appendLock sync.Mutex

	// Used to track the current state of the manifest, used when rewriting.
	manifest Manifest

	// Used to indicate if badger was opened in InMemory mode.
	inMemory bool
}
  • Purpose: Holds the file pointer and other metadata for the MANIFEST file. It manages appending changes, rewriting, and ensuring atomicity.
    • fp: File pointer to the MANIFEST file.
    • directory: Directory where the MANIFEST file is stored.
    • externalMagic: External magic number used by the application.
    • deletionsRewriteThreshold: Threshold for the number of deletions before rewriting the manifest.
    • appendLock: Mutex to guard appends to the manifest file.
    • manifest: Current state of the manifest.
    • inMemory: Indicates if Badger is running in in-memory mode.

ManifestFilename​

const (
	// ManifestFilename is the filename for the manifest file.
	ManifestFilename                  = "MANIFEST"
	manifestRewriteFilename           = "MANIFEST-REWRITE"
	manifestDeletionsRewriteThreshold = 10000
	manifestDeletionsRatio            = 10
)
  • Purpose: Defines constants related to the MANIFEST file, including filenames and rewrite thresholds.
    • ManifestFilename: Filename for the manifest file ("MANIFEST").
    • manifestRewriteFilename: Filename for the manifest rewrite file ("MANIFEST-REWRITE").
    • manifestDeletionsRewriteThreshold: Default threshold for deletions before rewriting (10000).
    • manifestDeletionsRatio: Ratio of deletions to total changes that triggers a rewrite (10).

asChanges​

func (m *Manifest) asChanges() []*pb.ManifestChange
  • Purpose: Returns a slice of pb.ManifestChange objects representing the current state of the Manifest. These changes can be used to recreate the manifest.
  • Parameters:
    • m: A pointer to the Manifest object.
  • Returns: A slice of *pb.ManifestChange objects.

clone​

func (m *Manifest) clone() Manifest
  • Purpose: Creates a deep copy of the Manifest object.
  • Parameters:
    • m: A pointer to the Manifest object.
  • Returns: A new Manifest object that is a clone of the original.

openOrCreateManifestFile​

func openOrCreateManifestFile(opt Options) (
	ret *manifestFile, result Manifest, err error)
  • Purpose: Opens an existing MANIFEST file or creates a new one if it doesn't exist.
  • Parameters:
    • opt: Badger Options containing configuration details like directory and read-only mode.
  • Returns:
    • *manifestFile: A pointer to the manifestFile object.
    • Manifest: The Manifest object representing the state of the database.
    • error: An error object, or nil if successful.

helpOpenOrCreateManifestFile​

func helpOpenOrCreateManifestFile(dir string, readOnly bool, extMagic uint16,
	deletionsThreshold int) (*manifestFile, Manifest, error)
  • Purpose: Helper function for openOrCreateManifestFile that encapsulates the logic for opening or creating the MANIFEST file.
  • Parameters:
    • dir: The directory where the MANIFEST file is located.
    • readOnly: A boolean indicating whether the database should be opened in read-only mode.
    • extMagic: The external magic number.
    • deletionsThreshold: The threshold for deletions before rewriting the manifest.
  • Returns:
    • *manifestFile: A pointer to the manifestFile object.
    • Manifest: The Manifest object representing the state of the database.
    • error: An error object, or nil if successful.

close​

func (mf *manifestFile) close() error
  • Purpose: Closes the MANIFEST file.
  • Parameters:
    • mf: A pointer to the manifestFile object.
  • Returns: An error object, or nil if successful.

addChanges​

func (mf *manifestFile) addChanges(changesParam []*pb.ManifestChange) error
  • Purpose: Atomically writes a batch of changes to the MANIFEST file. It also handles rewriting the manifest if necessary.
  • Parameters:
    • mf: A pointer to the manifestFile object.
    • changesParam: A slice of *pb.ManifestChange objects representing the changes to be applied.
  • Returns: An error object, or nil if successful.

syncFunc​

var syncFunc = func(f *os.File) error { return f.Sync() }
  • Purpose: A variable holding a function that synchronizes the file to disk. This is exposed for testing purposes.
  • Parameters:
    • f: A pointer to the os.File object.
  • Returns: An error object, or nil if successful.

magicText​

var magicText = [4]byte{'B', 'd', 'g', 'r'}
  • Purpose: Defines the magic text used to identify a Badger MANIFEST file.

badgerMagicVersion​

const badgerMagicVersion = 8
  • Purpose: Defines the version number of the Badger MANIFEST file format.

helpRewrite​

func helpRewrite(dir string, m *Manifest, extMagic uint16) (*os.File, int, error)
  • Purpose: Helper function to rewrite the MANIFEST file. It writes the current state of the manifest to a new file, then atomically renames it to replace the old MANIFEST file.
  • Parameters:
    • dir: The directory where the MANIFEST file is located.
    • m: A pointer to the Manifest object representing the current state.
    • extMagic: The external magic number.
  • Returns:
    • *os.File: A pointer to the new MANIFEST file.
    • int: The number of net creations.
    • error: An error object, or nil if successful.

rewrite​

func (mf *manifestFile) rewrite() error
  • Purpose: Rewrites the MANIFEST file by calling helpRewrite.
  • Parameters:
    • mf: A pointer to the manifestFile object.
  • Returns: An error object, or nil if successful.

countingReader​

type countingReader struct {
	wrapped *bufio.Reader
	count   int64
}
  • Purpose: A reader that wraps another reader and counts the number of bytes read.
    • wrapped: The wrapped bufio.Reader.
    • count: The number of bytes read.

Read​

func (r *countingReader) Read(p []byte) (n int, err error)
  • Purpose: Reads data into the provided byte slice and updates the byte count.
  • Parameters:
    • p: The byte slice to read data into.
  • Returns:
    • n: The number of bytes read.
    • err: An error, if any.

ReadByte​

func (r *countingReader) ReadByte() (b byte, err error)
  • Purpose: Reads a single byte and updates the byte count.
  • Returns:
    • b: The byte read.
    • err: An error, if any.

errBadMagic & errBadChecksum​

var (
	errBadMagic    = errors.New("manifest has bad magic")
	errBadChecksum = errors.New("manifest has checksum mismatch")
)
  • Purpose: Defines error variables for bad magic number and checksum mismatch in the MANIFEST file.

ReplayManifestFile​

func ReplayManifestFile(fp *os.File, extMagic uint16) (Manifest, int64, error)
  • Purpose: Reads the manifest file and constructs a Manifest object representing the state of the database. It also returns the last offset after a completely read manifest entry.
  • Parameters:
    • fp: A pointer to the os.File object representing the MANIFEST file.
    • extMagic: The external magic number.
  • Returns:
    • Manifest: The Manifest object representing the state of the database.
    • int64: The last offset after a completely read manifest entry.
    • error: An error object, or nil if successful.

applyManifestChange​

func applyManifestChange(build *Manifest, tc *pb.ManifestChange) error
  • Purpose: Applies a single manifest change to the provided Manifest object.
  • Parameters:
    • build: A pointer to the Manifest object to be modified.
    • tc: A pointer to the pb.ManifestChange object representing the change to be applied.
  • Returns: An error object, or nil if successful.

applyChangeSet​

func applyChangeSet(build *Manifest, changeSet *pb.ManifestChangeSet) error
  • Purpose: Applies a set of manifest changes to the provided Manifest object.
  • Parameters:
    • build: A pointer to the Manifest object to be modified.
    • changeSet: A pointer to the pb.ManifestChangeSet object containing the changes to be applied.
  • Returns: An error object, or nil if successful.

newCreateChange​

func newCreateChange(
	id uint64, level int, keyID uint64, c options.CompressionType) *pb.ManifestChange
  • Purpose: Creates a new pb.ManifestChange object for creating a table.
  • Parameters:
    • id: The ID of the table.
    • level: The level of the table.
    • keyID: The Key ID of the table.
    • c: The compression type.
  • Returns: A pointer to the new pb.ManifestChange object.

newDeleteChange​

func newDeleteChange(id uint64) *pb.ManifestChange
  • Purpose: Creates a new pb.ManifestChange object for deleting a table.
  • Parameters:
    • id: The ID of the table to be deleted.
  • Returns: A pointer to the new pb.ManifestChange object.

Code Examples​

No examples provided in the original file.

Getting Started Relevance​