The GenServer Is a Cache, Postgres Is the Queue

29 Oct 2026

I was looking at the Camelot board one afternoon and it showed three tasks running. The profile page, on the same screen, said two running and one queued.

Both were correct. The board was showing me tasks that had been dispatched; the profile was showing me tasks that actually held a runner slot. One of those three cards was not running at all. It was waiting in line, and nothing in the interface said so.

That is a UI bug, and I fixed it. But it seems to be a UI bug that only exists because of a design decision underneath it, and that decision is the interesting part.

Why there is a queue at all

An agent run is expensive in two directions. It costs money in tokens, and it costs a container with a real Elixir toolchain in it, on nodes that do not have infinite memory.

So concurrency has to be capped. Globally, and also per user, because one enthusiastic person should not be able to starve everyone else.

The obvious way to write that is to refuse work when you are full. Return {:error, :at_capacity}, let the caller retry.

I did not want that. From the pool’s moduledoc:

Sessions enqueue immediately and wait their turn — the pool never refuses a dispatch. Per-user and global caps control concurrency, not admission. Wait time is a UX signal (and a future monetisation lever via per_user_max_overrides).

Refusing a dispatch pushes the problem onto every caller. Each one needs retry logic, backoff, and some way to tell the user what happened. Accepting it and making them wait keeps all of that in one place. And the waiting itself is information: if your tasks queue for a long time, that is something I want to be able to show you, and eventually something you could pay to change.

So enqueue/3 always succeeds:

@spec enqueue(String.t(), String.t(), pid()) ::
        {:ok, %{position: non_neg_integer()}}
def enqueue(user_id, session_id, from_pid) when is_pid(from_pid) do
  GenServer.call(@name, {:enqueue, user_id, session_id, from_pid})
end

It returns your position in line. Zero means you were dispatched immediately.

The queue lives in Postgres, not the GenServer

A GenServer holding a queue is probably the most natural thing to write in Elixir. State goes in the struct, :queue from Erlang’s standard library does the heavy lifting, and it is fast and correct and has no race conditions.

It is also completely gone the moment you deploy.

I deploy Camelot several times a week. If the queue lived only in GenServer state, every deploy would silently drop every queued session. Not crash, just drop. The user would see a card that said “queued” forever, with nothing anywhere to pick it up again, because the only record that it was waiting died with the process.

So the queue does not live in the GenServer. It lives in Postgres, as Session rows with status: :queued, and the GenServer is a fast index over those rows:

The authoritative queue is the set of Camelot.Agents.Session{status: :queued} rows in the DB. The GenServer state is just a fast index over those rows, rebuilt on boot by Camelot.Runtime.Reconciler calling tick/0.

This last part is the whole trick. On boot, the reconciler (the same one from the first article in this series) walks the durable rows and rebuilds the index. The queue survives the deploy because it was never in memory in the first place. Memory just had a copy.

What stays in GenServer state, and why

The GenServer state is still doing real work. It just holds derived data:

defstruct active: %{},
          queue: %{},
          waiters: %{},
          monitors: %{},
          global_max: 20,
          per_user_max: 2,
          per_user_max_overrides: %{}

queue is a map of user id to an Erlang :queue, so popping the next session for a user is O(1) rather than an ordered database query on every release.
active is a map of user id to a MapSet of session ids, so cap checks are set operations.
waiters maps a session id to the pid that is waiting to be told it can start.
monitors maps a monitor ref back to the user and session it belongs to.

Dispatch is then pleasantly boring:

defp dispatch_user(state, user_id) do
  cond do
    total_active(state) >= state.global_max ->
      state

    active_count(state, user_id) >= effective_cap(state, user_id) ->
      state

    true ->
      case pop_queue(state, user_id) do
        {nil, state} ->
          state

        {session_id, state} ->
          grant(state, user_id, session_id)
          state |> add_active(user_id, session_id) |> drop_waiter(session_id)
      end
  end
end

Two caps, a pop, a grant. Every one of those calls is against in-memory data, which is exactly what the index is for.

The important property is that none of this is information you could not reconstruct. Delete the whole struct, replay the :queued and :running rows, and you are back where you were.

Waiters and the one OTP feature I actually rely on

Granting a slot is a message:

defp grant(state, _user_id, session_id) do
  case Map.get(state.waiters, session_id) do
    nil ->
      Logger.warning("RunnerPool: no waiter for #{session_id}")

    pid ->
      send(pid, {:runner_slot, session_id})
  end
end

The TaskRunner for that task is sitting in handle_info waiting for {:runner_slot, session_id}. When it arrives, it builds the spec and starts the container.

But what happens if the waiter dies while it is waiting? Or while it is running? Its slot would be held by nobody, forever, and the pool would slowly leak capacity until it stopped dispatching anything.

And this is a place where OTP does exactly what I need, with no work from my side:

def handle_info({:DOWN, ref, :process, _pid, _reason}, state) do
  case Map.pop(state.monitors, ref) do
    {nil, _} ->
      {:noreply, state}

    {{user_id, session_id}, monitors} ->
      state =
        %{state | monitors: monitors}
        |> remove_from_active(user_id, session_id)
        |> remove_from_queue(user_id, session_id)
        |> drop_waiter(session_id)
        |> dispatch_global()

      broadcast(state)
      {:noreply, state}
  end
end

Every waiter is monitored when it enqueues. If it dies, the slot is released and the next session in line gets dispatched immediately.

I have written in the previous article about how much work it is to rebuild supervision for processes that live in someone else’s container. This is the contrast. Inside the VM, “notice when the thing holding a resource dies” is four lines, and they are exactly right.

Back to the board

So, back to the card that said it was running.

Because the pool never refuses a dispatch, “I asked for a runner” and “I have a runner” are different states, and from the outside they look identical. The task row says the same thing either way. The only place that knows the difference is the pool’s in-memory index.

So the pool exposes a snapshot for the UI, the board asks it whether a task holds a slot or is still in line, and now a queued card says it is waiting.

This is the trade-off you accept when you choose “always accept, make them wait” over “refuse when full”. Rejection is self-documenting: the caller gets an error and has to deal with it. Queueing is invisible by default. You have to go and surface it on purpose, or your users will think the application is broken while it is in fact working exactly as designed.

Conclusion

For myself I took two rules out of this story, and now I apply both without thinking:

If it must survive a restart, it is a row. GenServer state is a cache over durable state, never the source of truth for it. The test is simple: kill the process and ask what was lost. If the answer is “work the user is still waiting for”, it belonged in the database.

A queue that never rejects has to be visible. The moment you decide to absorb load rather than refuse it, you have taken on the obligation to tell someone they are waiting.

Next time, a debugging story with a much shorter root cause: a LiveView that reconnected in a loop in production and never in test, for reasons that had nothing to do with production.