The write-ahead log, explained by rebuilding one

Durability sounds like a hardware problem. It is mostly a discipline problem, and the write-ahead log is the discipline. The rule is one sentence: write your intent to a durable log before you touch the real data.

Why write twice

If you update a data page in place and the machine dies halfway, you have a page that is neither the old value nor the new one. There is no way to know what you were doing. The WAL fixes this by recording the intent first:

1. append {op, key, value} to the log
2. fsync the log
3. apply the change to the main store

If step 3 never happens, recovery replays the log and finishes the job. The log is the source of truth until the change is safely in place.

fsync is the whole game

Appending to a file is cheap. Making the operating system promise the bytes are actually on disk is not. That promise is fsync, and forgetting it is how people ship a "durable" store that loses data on power failure.

def append(self, record: bytes) -> None:
    self.f.write(len(record).to_bytes(4, "big"))
    self.f.write(record)
    self.f.flush()
    os.fsync(self.f.fileno())  # the line that makes it durable

Everything before the fsync is a suggestion. Everything after it is a fact.

Recovery is just replay

On startup, you read the log from the last checkpoint and reapply each record. Because the operations are idempotent by design, replaying a record that was already applied is harmless.

• Checkpoint so the log does not grow forever.
• Truncate only what you have proven is safely in the main store.
• Verify each record with a checksum, because a torn write at the tail is normal.

Rebuild one of these once and the database stops feeling like magic. It is just a log, an fsync, and the patience to replay.