Living document

Deterministic simulation testing in celld: exploring execution orders and reproducing bugs

How celld controls time and execution order to find bugs with deterministic simulation testing, explained through interactive diagrams, pseudocode, and a race it discovered.

Updated: 9/19/2026

celld
Deterministic Simulation Testing
Concurrency
Durable Objects

We are building celld, a runtime for running Cloudflare Workers and Durable Objects applications on your own machines. It can run with an S3-compatible object store as its only external service dependency.

To make it reliable, we need to test how it behaves across different combinations of events and failures—and be able to reproduce the problems we find. One of the tools we use is deterministic simulation testing (DST). It lets us run celld’s code in a controlled environment, explore different execution orders, and repeat a failing run to investigate its cause.

Our simulator is still under development and is not included in celld’s public repository. However, it has already found previously unknown bugs in celld. In this article, we will walk through what DST is, how we make it work with celld, and how we reproduced and fixed one of the bugs it found.

What deterministic simulation testing does

Distributed systems run across machines that communicate over a network. Peter Deutsch’s The Eight Fallacies of Distributed Computing lists common false assumptions, including “The network is reliable” and “Latency is zero.” We have to design for unreliable communication and for machines that can stop or restart at any time.

A bug can depend on a particular sequence of delayed messages, failed saves, and server restarts. Those events can happen in a different order on the next test run, making the failure difficult to reproduce.

This is where DST comes in. It runs the system’s real code in an environment controlled by a simulator. For example, the simulator can let data be saved but delay the reply, or even make a save fail. It can also move the simulated clock forward until a retry is due, without waiting for that time to pass in the real world. As execution progresses, the test checks that the system upholds its guarantees under these conditions.

The simulator uses random choices to explore different orders of events and failures. A seed is the starting value for the random-number generator. With the same code, test settings, and seed, the simulator reproduces the same sequence, intermediate states, and result. We can explore different runs by changing the seed, then repeat a failing run to trace where things went wrong.

Running celld’s production code in a controlled environment

A cell in celld runs application code and has its own SQLite database.

celld handles events such as incoming requests, completed storage operations, and timer firings. The code that selects the next event is separate from the code that handles it. This lets the simulator control the order of events while running the same event-handling code as production.

The simulator can also control when asynchronous tasks run, advance a simulated clock, and simulate delays or failures when celld saves data to S3-compatible object storage.

First, let’s take a brief look at a simplified replay that shows a successful request to insert one row into a cell’s SQLite table.

Watch for the gap between the write becoming durable in object storage and celld receiving confirmation. Other requests or background tasks can run in between. The simulator controls these steps separately so we can explore different execution orders and look for race conditions.

Press Next to step through choosing an action, animating it, and revealing its result.

Candidates

Generate an insert request.

Chosen

Choose
Run
Result

No insert request has been generated yet.

Step 0 / 24

Enable JavaScript to step through the recording. The complete sequence is also available as text.

For fixed code and test settings, the seed determines the requests and execution order. This excerpt follows a request to insert one row into a cell’s SQLite table.

The seed determines which available action the simulator selects next, including request generation and delivery. Only the candidates relevant to this illustration are shown. Clicking Next advances through choosing an action, running it, and showing its result. These are presentation phases, not simulator ticks. The ▶ marker identifies the components selected to advance; ◆ marks the resulting changes. celld handles the events with its production code. The code that inserts the row is written for the test. The client and object storage also use test implementations.

A simplified excerpt from a recorded development run. Cell A is the cell handling this request; the other cells illustrate that each cell has its own SQLite database. Concurrent work and intermediate actions are omitted.

Generate an insert request
Chosen: Generate an insert request.
Candidates: Generate an insert request.
The simulator uses the seed to choose a request that inserts a row into a SQLite table. The request is ready to be sent to Cell A.
Send the request to Cell A
Chosen: Send the request to Cell A.
Candidates: Send the request to Cell A.
The simulator lets the client send the prepared request to Cell A. The client then waits for a response.
Insert a row into SQLite
Chosen: Let Cell A process the insert request.
Candidates: Let Cell A process the insert request.
The test code inserts a row into Cell A’s SQLite table. The client has not yet received a success response.
Hold the response until data is durable
Chosen: Let celld check whether it can reply.
Candidates: Let celld check whether it can reply.
celld holds the response while it checks that the database changes are durably saved. The simulator can advance other work during this wait.
Upload the database changes
Chosen: Send the SQLite changes; let object storage receive them.
Candidates: Send the SQLite changes; let object storage receive them.
celld sends a batch of SQLite changes to object storage. The simulated object store receives the data, but the write has not completed yet.
Complete the write to object storage
Chosen: Complete the storage write.
Candidates: Complete the storage write.
The simulator lets the write take effect. The data is saved, but the success result has not yet been delivered to celld.
Deliver the result and resume waiting work
Chosen: Deliver the storage write result to celld.
Candidates: Deliver the storage write result to celld.
After the storage result arrives, celld resumes and confirms that the database changes are durable.
Return a successful response
Chosen: Let celld check again whether it can reply.
Candidates: Let celld check again whether it can reply.
After celld completes its checks, it releases the response. The client receives a success response for the row insertion.

The seed controls choices such as which requests to generate, what runs next, whether storage operations succeed or fail, and which faults to inject and when. In pseudocode:

const simulation = startSimulation({
  code: celldCode,
  requests: createRequestGenerator(seed),
  environment: createTestEnvironment(seed),
  scheduler: createScheduler(seed),
});

while (simulation.hasWork()) {
  const candidates = simulation.availableActions();
  const action = simulation.scheduler.choose(candidates);

  simulation.run(action);
  checkInvariants(simulation);
}

Each candidate is an operation the simulator can advance: a request, a task, a clock change, or a storage operation. The test configuration limits which outcomes are allowed, including whether faults can occur. celld’s production code handles the resulting events.

After each action, an independent checker uses observed responses and stored data to test invariants: conditions the system must uphold throughout a run.

An alarm race the simulator found

Next, let’s dive into another, more interesting run that actually helped us discover a real bug in celld.

The bug found by the simulator was a race in celld’s alarm handling. An alarm schedules a cell to run application code at a specified time. The cell need not stay in memory until then. celld can unload an idle cell to make room for others and load it again in time to run its alarm.

The alarm’s scheduled time is stored in the cell’s SQLite database. To find which cells to wake without opening every database, celld scans wake entries in object storage. Each entry identifies a cell and when to reactivate it. Once the cell is active, celld reads the scheduled time from SQLite to determine when to run the alarm.

When we found this bug, celld reused wake entries to avoid unnecessary writes to object storage. Suppose an application deletes a 10:00 alarm and sets a new one for 10:05. If the 10:00 wake entry is still present, it can bring the cell back early at 10:00. celld reads 10:05 from SQLite and waits until then to run the alarm. If celld needs to evict the cell during that wait, it first ensures that a wake entry for 10:05 is in place.

Deleting an alarm clears its scheduled time from SQLite. Its wake entry in object storage is removed later by a separate cleanup task. In this example, celld returns success to the client once the SQLite change has been saved to object storage, without waiting for that cleanup. The simulator can delay this task while advancing other work.

The bug appeared when cleanup from an earlier alarm deletion ran after a new alarm had been set successfully. It deleted a wake entry that the new alarm still needed.

Once the client receives confirmation that an alarm is set, a usable wake entry must remain until the alarm runs or is canceled. Here, usable means it can wake the same cell at or before the alarm’s scheduled time. We check this invariant below.

Candidates

Delete Cell A's 10:00 alarm.

Chosen

Choose
Run
Result
Check

Cell A has a 10:00 alarm in SQLite and a 10:00 wake entry in object storage.

Step 0 / 21

Enable JavaScript to step through the recording. The complete sequence is also available as text.

This is an illustrated reconstruction of a race found during development, rather than an exact trace replay.

The simulator selects which requests and pending operations advance. Only the candidates relevant to this illustration are shown. Clicking Next advances through choosing an action, running it, showing its result, and checking the invariant when applicable. These are presentation phases, not simulator ticks. The ▶ marker identifies the selected component; ◆ marks the resulting changes. At the key interleaving, two candidates show which operation advances and which waits. celld’s code decides how to process the events. The test checks the wake information after an alarm is set successfully. This check is fixed, not chosen by the seed.

The sequence describes the implementation at discovery. Each displayed step groups several lower-level actions. All states and the check result are prepared for playback; the browser does not run celld or its simulator.

Delete the 10:00 alarm
Chosen: Delete Cell A's 10:00 alarm.
Candidates: Delete Cell A's 10:00 alarm.
The alarm is gone from SQLite. The 10:00 wake entry remains in storage; its deletion task has not run yet.
Acknowledge the deletion
Chosen: Let celld reply to the deletion request.
Candidates: Let celld reply to the deletion request. Run cleanup of the 10:00 wake entry.
The client receives a success response for deleting the alarm. Cleanup of the 10:00 wake entry still waits.
Set a 10:05 alarm first
Chosen: Set Cell A's alarm for 10:05.
Candidates: Set Cell A's alarm for 10:05. Run cleanup of the 10:00 wake entry.
The 10:05 alarm reuses the 10:00 wake entry: the cell can wake early and wait until 10:05.
Acknowledge the new setting
Chosen: Let celld reply to the 10:05 alarm request.
Candidates: Let celld reply to the 10:05 alarm request. Run cleanup of the 10:00 wake entry.
Setting the 10:05 alarm succeeds. The reused 10:00 wake entry must remain available.
Run the delayed cleanup
Chosen: Run cleanup of the 10:00 wake entry.
Candidates: Run cleanup of the 10:00 wake entry.
Acting on the earlier deletion, the delayed cleanup task requests deletion of the 10:00 wake entry.
The check catches lost wake information
Chosen: Let object storage apply the deletion.
Candidates: Let object storage apply the deletion.
The 10:00 wake entry is gone, even though the 10:05 alarm was set successfully.

In the run above, the application received success responses for both deleting the 10:00 alarm and setting the new 10:05 alarm. The 10:05 alarm remained in SQLite, but the 10:00 wake entry it relied on was gone. That loss is what the checker detected.

The alarm could still run at 10:05 as long as the node kept that time in memory. If the node stopped or crashed, that information would be lost. The missing wake entry could then prevent celld from finding the cell again. The alarm could be missed even though its scheduled time remained in SQLite and the client had received a success response.

The simulator found this failing order by exploring requests and pending work. We had not written a test that prescribed this sequence.

Reproducing the bug and preventing regressions

We replayed the alarm failure with the same code, test settings, and seed. This let us repeatedly inspect the moment the old cleanup task deleted the wake entry needed by the new alarm, without having to find the failing order again.

We then reduced the failure to three application requests: set an alarm, delete it, and set a new one. Reproducing the failure also requires delaying the old wake-entry cleanup task until the new alarm has been set successfully, then letting it run. We kept those requests and that task order as a regression test.

The regression test checks intermediate states as well as the final result. If the required wake entry disappeared briefly and later work restored it, checking only the end would miss the failure. We therefore check that a usable wake entry remains when the old cleanup task runs and whenever a storage change takes effect.

Since then, we have changed the wake-entry design. Each new alarm setting gets its own entry in object storage. A delayed deletion for an older alarm can only remove that older entry, leaving the new one intact. Old entries are removed once celld has confirmed they are no longer needed.

Improving the simulator to make celld more robust

celld’s simulator is still under development, but it has already found previously unknown bugs in the production code. We have fixed these bugs and kept regression tests for them. The alarm race in this article is one example.

The simulator controls the environment around celld’s engine. Tests of the full runtime complement it by covering JavaScript execution and real network I/O.

Next, we will expand the operations and failures the simulator can combine. For example, a node could crash while a write is stalled by a storage outage, then restart before storage is available again. We also want to vary which requests arrive as the node recovers.

We will add checks for more of celld’s guarantees. When celld tells a client that a write succeeded, for example, the data must survive a node restart. Like the alarm check above, these checks will run during retries and recovery so that a later repair cannot hide a temporary loss.

When we find a failing execution, we can follow its progress, fix the cause, and retain a test to catch the same problem in the future. We want to keep improving the simulator so that we can repeat that process and make celld a more robust, reliable distributed system.