The Ledger DB is responsible for the following tasks:

• Maintaining the in-memory ledger state at the tip: When we try to extend our chain with a new block fitting onto our tip, the block must first be validated using the right ledger state, i.e., the ledger state corresponding to the tip.

• Maintaining the past \(k\) in-memory ledger states: we might roll back up to \(k\) blocks when switching to a more preferable fork. Consider the example below:

  

  Our current chain's tip is \(C_2\), but the fork containing blocks \(F_1\), \(F_2\), and \(F_3\) is more preferable. We roll back our chain to the intersection point of the two chains, \(I\), which must be not more than \(k\) blocks back from our current tip. Next, we must validate block \(F_1\) using the ledger state at block \(I\), after which we can validate \(F_2\) using the resulting ledger state, and so on.

  This means that we need access to all ledger states of the past \(k\) blocks, i.e., the ledger states corresponding to the volatile part of the current chain. Note that applying a block to a ledger state is not an invertible operation, so it is not possible to simply unapply \(C_1\) and \(C_2\) to obtain \(I\).

  Access to the last \(k\) ledger states is not only needed for validating candidate chains, but also by the:

  • Local state query server: To query any of the past \(k\) ledger states.
  • Chain sync client: To validate headers of a chain that intersects with any of the past \(k\) blocks.
• Providing LedgerTables at any of the last \(k\) ledger states: To apply blocks or transactions on top of ledger states, the LedgerDB must be able to provide the appropriate ledger tables at any of those ledger states.

• Storing snapshots on disk: To obtain a ledger state for the current tip of the chain, one has to apply all blocks in the chain one-by-one to the initial ledger state. When starting up the system with an on-disk chain containing millions of blocks, all of them would have to be read from disk and applied. This process can take hours, depending on the storage and CPU speed, and is thus too costly to perform on each startup.

  For this reason, a recent snapshot of the ledger state should be periodically written to disk. Upon the next startup, that snapshot can be read and used to restore the current ledger state, as well as the past \(k\) ledger states.

• Flushing LedgerTable differences: The running Consensus has to periodically flush chunks of differences from the DbChangelog to the BackingStore, so that memory is off-loaded to the backing store, and if the backing store is an on-disk implementation, reduce the memory usage.

Note that whenever we say ledger state we mean the ExtLedgerState blk mk type described in Ouroboros.Consensus.Ledger.Basics.

Resource management in the LedgerDB

The LedgerDB has currently 3 backends it can use:

• InMemory: This backend is pure except for tracing. No resources are allocated.
• LMDB: This backend allocates a BackingStore and BackingStoreValueHandles on it. The BackingStore does not necessarily need to be closed as described in the LMDB documentation:

Closing a database handle is not necessary, but lets mdb_dbi_open() reuse the handle value.

Therefore, the BackingStore is not allocated in any resource or tracked in any way as a resource.

For the value handles, all the usages of those are bracketed or tracked in a resource registry, so they will be closed individually when an exception arrives. The key difference is that in V1, the value handle cannot outlive the Forker (see "Forker management in the running node" below), while in V2 the resources (handles) can outlive the forkers and be moved to the LedgerDB.

• LSM: This backend allocates a BlockIOFS and a Session. Using the session, new Tables are allocated, but closing the session closes any existing Tables handles.

Both the BlockIOFS and the Session are stored in the ldbResources of the LedgerDB, and closing the LedgerDB will release them. The LedgerDB will be closed by closing the ChainDB which is tracked in the top-level registry. Therefore we don't need to keep track of the Table handles nor we need to further keep track of the BlockIOFS and the Session.

Forker management in the running node

The openForkerAtTarget method of the LedgerDB type is the lowest-level method for opening a Forker. This comment describes the tree formed by definition-and-use edges rooted at openForkerAtTarget. It doesn't describe the entire tree, but rather just enough to confirm that Forkers will not be leaked during the extended execution of the running node. There are a few helpful clarifications to make before elaborating that tree.

• This comment is only concerned with the running node. In contrast, the shutting down node is addressed by the "Resource management in the LedgerDB" section above. There are two key ideas. First, the shut down routines don't open additional Forkers. Second, it's OK for the shut down routines to not necessarily close their local/owned Forkers, since the Forker backends either don't require that or already take care of open handles when their top-level "close" method is called.
• Some subtrees, like the withTipForker subtree, are irrelevant to this comment, because they explicitly use bracket or a short-lived ResourceRegstiry and so can't contribute to any leaks. To clarify: there would be leaks if those brackets were indefinitely nested or if the registry outlived multiple iterations, etc. But that itself would be an unacceptable stack leak, for example. Such unbounded nesting/registries are therefore beyond the scope of this comment.
• Tools (like db-analyser) and tests are also beyond the scope of this comment, so those subtrees are mentioned but not elaborated.
• It turns out that the resulting tree is currently merely a list of such uninteresting subtrees, so the nub of this comment can be linear despite describing a tree.

At the time of writing, the (linear spine of the) def-use tree is as follows.

• openForkerAtTarget is used directly only to define withTipForker (bracketed) and openReadOnlyForker.
• openReadOnlyForker is used directly only in db-analyser (tool), in tests, and to define openReadOnlyForkerAtPoint.
• openReadOnlyForkerAtPoint is used directly only to define allocInRegistryReadOnlyForkerAtPoint (registered), to define withReadOnlyForkerAtPoint (bracketed), and to construct a MempoolLedgerDBView.
• The Forker part of MempoolLedgerDBView is used directly only in initMempoolEnv (called by openMempool) and in 'implSyncWithLedger. If an exception arrives during openMempool, then node will shutdown, so leaks are not a concern here. The syncing-with-ledger use is bracketed via modifyMVar_ to close the old forker just before replacing it with the new one.

(image code)

>>> import Image.LaTeX.Render
>>> import Control.Monad
>>> import System.Directory
>>> 
>>> createDirectoryIfMissing True "docs/haddocks/"
>>> :{
>>> either (error . show) pure =<<
>>>  renderToFile "docs/haddocks/ledgerdb-switch.svg" defaultEnv (tikz ["positioning", "arrows"]) "\
>>> \ \\draw (0, 0) -- (50pt, 0) coordinate (I);\
>>> \  \\draw (I) -- ++(20pt,  20pt) coordinate (C1) -- ++(20pt, 0) coordinate (C2);\
>>> \  \\draw (I) -- ++(20pt, -20pt) coordinate (F1) -- ++(20pt, 0) coordinate (F2) -- ++(20pt, 0) coordinate (F3);\
>>> \  \\node at (I)  {$\\bullet$};\
>>> \  \\node at (C1) {$\\bullet$};\
>>> \  \\node at (C2) {$\\bullet$};\
>>> \  \\node at (F1) {$\\bullet$};\
>>> \  \\node at (F2) {$\\bullet$};\
>>> \  \\node at (F3) {$\\bullet$};\
>>> \  \\node at (I) [above left] {$I$};\
>>> \  \\node at (C1) [above] {$C_1$};\
>>> \  \\node at (C2) [above] {$C_2$};\
>>> \  \\node at (F1) [below] {$F_1$};\
>>> \  \\node at (F2) [below] {$F_2$};\
>>> \  \\node at (F3) [below] {$F_3$};\
>>> \  \\draw (60pt, 50pt) node {$\\overbrace{\\hspace{60pt}}$};\
>>> \  \\draw (60pt, 60pt) node[fill=white] {$k$};\
>>> \  \\draw [dashed] (30pt, -40pt) -- (30pt, 45pt);"
>>> :}