Our storage is ZFS-style. In general, it seems that (on conceptual level) we managed to obtain each and every integrity property of ZFS, but re-optimized for RDBMS purposes.

Data Storage - Following ZFS Blueprints

In MECHLOVE, data storage (those pages which don’t belong to WAL) is handled by tablespaces. However, unlike most RDBMS out there, our tablespaces do NOT rewrite pages in place; instead, in ZFS spirit, we are doing full-scale CoW for on-disk data, which is enabled by always-in-memory uberblocks. 

Some subtle distinctions from ZFS:

  • Our uberblock on-disk representation is not a ring, but an A/B flip-flop. Our analysis shows that for RDBMS (unlike for file systems) it is extremely important to avoid silent degradation into the previous version, and an A/B flip-flop provides natural detection for this kind of failure. We do NOT claim ZFS is wrong here; RDBMS and ZFS are just solving subtly different problems.
  • While we do provide end-to-end checksums, we do not need multiple levels in the Merkle tree; in fact, our Merkle tree is always 1-level: our uberblock simply has checksums of each and every page within the checkpoint. This is a perfectly valid variation of the Merkle tree, which doesn’t change its integrity properties.

Compared to traditional databases, following ZFS blueprints gives us the following important advantages:

  • CoW provides natural protection from torn writes (and that’s without any need for performance-heavy patches such as MySQL DWB or Postgre’s FPW).
  • End-to-end data integrity (based on 1-level Merkle trees for Data Storage, but see below re. WAL/ZIL)
  • Self-healing (well, when we implement our own RAID, which we plan to do). As a side benefit of implementing our own software RAID in an RDBMS context, it seems that we will NOT need those write-intent bitmaps a la mdadm (relevant information is already in WAL), saving a bit of performance. 
  • Scrubbing and Repairing

On Checkpoints, Write Coalescing and Write Amplification

As with any sane RDBMS, with MECHLOVE data pages are not written to disk right away; they're written either when a cache eviction happens, or as part of a checkpoint.

Our checkpoints (as with any production-level RDBMS) are fuzzy, in the sense that they don't stop the world. What actually happens is that whenever we need to write a checkpoint (for example, to limit recovery time), is the following: we simply scan all the immutable pages of the snapshot that corresponds to the "last-materialized CSN", remove version patches from them, and write each page to the disk.

This approach provides all the usual benefits of fuzzy checkpoints (such as write coalescing), plus it ensures that our checkpoint pages are always perfectly self-consistent; as discussed below, it serves as a foundation to avoid undo during recovery.

One common concern whenever someone hears "CoW" for database pages is write amplification. However, in our model, we're writing exactly the same stuff as conventional RDBMS does, and therefore do not have any substantial write amplification (note that the size of the uberblock is negligible by design). If anything, this design allows us to avoid certain sources of write amplification; in particular, CoW pages mean that there is no risk of "torn pages" on a crash, and therefore (as mentioned above) protections such as DWB or FPW are not necessary.

WAL

WAL is an all-important part of RDBMS design, so of course we didn’t take it lightly. 

WAL Integrity-Wise - Arguably Subtly Better Than ZFS ZIL

Integrity-wise, we started from following ZFS ZIL blueprints, but then we realized that we can do better than that. At least in our understanding, ZFS ZIL is not protected by end-to-end checksums (ZIL checksums are local only); to combat this problem, ZIL provides some checks to ensure integrity against some specific failure modes (such as lost writes and misdirected writes). 

However, our WAL goes further than ZIL integrity-wise; it uses a hash chain, with each subsequent WAL record storing the hash of the previous one (and hashed itself as well). While, due to the circular nature of the WAL, it is not possible to trace this hash chain all the way to the root of trust, for our threat model (which is non-malicious failures), the hash chain provides very natural and inherent protection from singular failures such as lost writes and misplaced writes. NB: if ZIL does use hash chains - our apologies (our conclusion relies on the comment “The zio_eck_t contains a zec_cksum which for the intent log is the sequence number of this log block.”). 

We’ve heard that ZIL abolished the idea of hash chains to facilitate parallel recovery, but for RDBMS recovery is sequential anyway, so hash chains are not expected to hurt. 

In addition, we keep all the ZIL integrity checks, including record checksums and sequence numbers. We contend that due to the added hash chains, our WAL is more resilient to non-malicious failures than ZIL. While the difference is not that significant in practice, as we were designing a new system, we chose to implement this improvement (it is pretty much for free, at least in our model). 

WAL DB-Wise - Template-Based Idempotent Logical Logging with Physical Hints

DB-Wise, all the major RDBMS we know about use some variation of ARIES “physiological” logging; ARIES was introduced in 1992, and is solid, but it effectively enforces slot-based pages. And slot-based pages, perfectly fine in 1992, are extremely inefficient in 2026; more specifically, they’re cache-unfriendly, latchless-unfriendly, and SIMD-unfriendly. Uncoupling page layout from WAL unties our hands to use much more efficient page layouts (see Part 3 for details). 

As a result, we’ll be using idempotent low-level logical logging, where each update is written to WAL along the lines of “UPDATE USERS SET FIELD=? WHERE PK=?”. All updates are written to WAL by PK (even if the original SQL wasn’t by PK), and all the values involved are written as firm end values (no increments, etc.). This makes the log idempotent (which is a firm prerequisite to being feasible for WAL), and removes all the need to re-run logic such as constraints etc.; replaying such a WAL record over a valid checkpoint cannot possibly fail. 

No undo

One important feature of our WAL is that (because our Data Storage is CoW, and our checkpoint pages are perfectly consistent) it seems that we do NOT need to undo during recovery (we need only to redo); this will reduce WAL file size (we won’t need old-values, only new-values), and simplify recovery logic. Even more importantly, it greatly simplifies handling of the schema changes during recovery (DDL becomes just yet another frame to be processed).

Performance Optimizations

Now to performance optimizations. First, to speed up redo, our WAL frame may contain a list of the affected PageIDs (as physical hints for replay); this information is completely optional for correctness, but it will reduce the amount of work involved in recovery. 

Another optimization is “templates”. In short, instead of writing a tuple (enum-of-USERS-table,list-of-enums-of-fields-updated, PK-value,serialized-field-values) we first write a special frame (template #N,enum-of-USERS-table,list-of-enums-of-fields-updated), and then can re-use this template via frames (template-ID,PK-value,serialized-field-values). These templates will naturally correspond to SQL prepared statements involved (not necessarily as 1:1), and can significantly reduce both frame sizes and, more importantly, composing/parsing times. 

WAL as Single Source of Truth

Look ma, no control file! 

Unlike most of the popular contemporary RDBMS, we do NOT have a “control file” that lives alongside the WAL. Instead, we treat WAL as a single source of truth. 

Control-file-less Recovery

To find out which WAL files have to be scanned during recovery, we’ll include CSN-of-checkpoint-started-in-previous-WAL-file and CSN-of-checkpoint-finished-in-previous-WAL-file into the header of each WAL file; then, we can scan only the headers of WAL files (from the last one towards the first ones) to find the point of recovery. 

No conflicts

TBH, the decision to avoid a control file is not as important as the other design decisions we've made here; however, WAL being the Single Source of Truth means no chance for the control file to disagree with WAL, which, in turn, avoids introducing (and the need to handle) some rather ugly failure modes.

Why Not Run on Top of Existing ZFS?

We feel that our approach provides several advantages over running on top of existing ZFS, mostly related to:

  • Different task definitions for file system and RDBMS (see above for examples of such mismatches).
  • Avoiding unnecessary-for-RDBMS layers - and traversing these layers takes time. 
  • Having caches in app space, which helps to avoid going through the userspace-kernel boundary.