A Practical Guide To Implementing The Actor Model In Rust: Tokio And Custom Schedulers

A comprehensive technical exploration of a practical guide to implementing the actor model in rust: tokio and custom schedulers, covering key concepts, practical implementations, and real-world applications.
Contents
This is an excellent foundation for a deep dive. The introduction perfectly captures the pain of traditional concurrency. Let’s expand this into a comprehensive, in-depth exploration. We need to add more conceptual depth, concrete code examples in multiple languages (Rust and Python for their contrasting approaches to concurrency), a deeper exploration of error handling and distributed systems, and a practical walkthrough of a real-world-like application.
The Tyranny of the Lock: Why Your Concurrent Code is Broken (And How the Actor Model Can Save It)
Let’s be honest: writing correct, high-performance concurrent code is the hardest thing you can do in modern software development. We often talk about it glibly—“just use a thread pool,” “throw an Arc on it,” “it’s fine because it’s just a read.” But deep down, every experienced systems programmer has a horror story. A deadlock that took three days to reproduce. A data race that only manifested on a specific CPU architecture under full load. A seemingly simple refactor that introduced a subtle livelock, silently destroying throughput.
The traditional model—shared mutable state protected by locks—is fundamentally brittle. It requires the programmer to be a perfect oracle, predicting every possible interleaving of execution and ordering of memory operations. You are essentially fighting the CPU and the compiler for control. And the weapon you are given? A mutex. A blunt instrument in a world requiring surgical precision.
But what if you didn’t have to share state at all? What if you could build your system not as a single, fragile blob of data, but as a collection of independent, isolated agents that only communicate through a strict message-passing protocol?
This is the promise of the Actor Model.
For decades, this paradigm has been the secret weapon of massively concurrent systems like WhatsApp (powered by Erlang/OTP) and Akka on the JVM. The core concept is elegant in its simplicity:
- Everything is an Actor. A lightweight computational entity.
- Each actor has a mailbox (queue). Other actors send messages to this mailbox.
- Actors are isolated. No shared state. An actor can only change its own private state.
- Actors process messages one at a time. This effectively serializes access to the actor’s state, making locks unnecessary.
It’s the ultimate form of “don’t call us, we’ll call you.” An actor doesn’t expose its internal data; it doesn’t have public methods that mutate its fields. Instead, it receives a message, processes it, and optionally sends messages (including itself) or creates new actors. It’s a paradigm shift from “thinking in objects” to “thinking in processes.”
Chapter 1: The Anatomy of Failure: Dissecting the Lock-Based Nightmare
To truly appreciate the Actor Model, we must first descend into the depths of the shared-state hell from which it offers escape. Let’s dissect the specific pathologies of lock-based concurrency.
The Four Horsemen of the Concurrency Apocalypse
Data Races: This is the most insidious. A data race occurs when two threads access the same memory location concurrently, at least one of them is writing, and there’s no synchronization enforcing a specific order. The C++ and Rust memory models explicitly make this “undefined behavior.” In practice, it means your program might work perfectly in debug mode, but crash mysteriously in release. Your compiler might reorder operations so drastically that your carefully placed
volatilekeyword is useless. You are not just fighting the hardware; you are fighting the optimizer.Deadlocks: The classic. Thread A holds lock L1 and waits for lock L2. Thread B holds lock L2 and waits for lock L1. Neither can proceed. They are locked in an eternal, silent embrace. The solution? A strict hierarchy for lock acquisition. But in a complex codebase, with layers of abstr… abstraction and callbacks, maintaining this hierarchy is a nightmare.
Livelocks: Deadlock’s more frustrating cousin. Threads are not blocked; they are busy doing nothing useful. They are constantly acquiring and releasing locks in a perfect, unproductive dance. Imagine two people trying to pass each other in a narrow hallway, both stepping aside in the same direction, blocking each other indefinitely. The system is “alive,” but completely stalled.
Starvation: A thread repeatedly fails to acquire a lock because other threads are always faster. This is often a subtle effect of thread scheduling and lock fairness policies. A high-priority thread can be starved by a lower-priority one that’s holding a lock, leading to priority inversion, a classic real-time systems problem.
A Concrete Example: The Broken Bank Transfer
Let’s imagine a simple bank transfer system. We have two accounts, A and B, and we want to transfer $100 from A to B. The naïve lock-based approach looks like this (pseudo-code):
void transfer(Account *a, Account *b, int amount) {
lock(a->mutex);
lock(b->mutex);
a->balance -= amount;
b->balance += amount;
unlock(b->mutex);
unlock(a->mutex);
}
This is a deadlock waiting to happen. If thread 1 calls transfer(account1, account2, 100) and thread 2 calls transfer(account2, account1, 50), we have a classic circular wait. The standard fix is to enforce a global lock ordering:
void transfer(Account *a, Account *b, int amount) {
if (a < b) {
lock(a->mutex);
lock(b->mutex);
} else {
lock(b->mutex);
lock(a->mutex);
}
a->balance -= amount;
b->balance += amount;
unlock(b->mutex);
unlock(a->mutex);
}
This works, but it’s fragile. What if a third account type, SavingsAccount, is introduced? What if the logic becomes more complex, involving fees, exchange rates, and audit logs? The lock ordering quickly becomes a global invariant that every new piece of code must respect. It’s a ticking time bomb.
This is the fundamental problem: the programmer must manually enforce a global ordering on all shared resources. The Actor Model eliminates this necessity by eliminating the shared resources themselves.
Chapter 2: The Actor Model: A New Covenant for Concurrency
The Actor Model, formalized by Carl Hewitt in 1973, isn’t just a library or a design pattern; it’s a fundamental computational model, as foundational as the Turing Machine or the Lambda Calculus. It provides a different set of primitives for building systems.
The Three Primitives
An actor can do exactly three things:
- Create more actors.
- Send messages to other actors (including itself).
- Designate what to do with the next message.
That’s it. No shared state. No locks. The only way for two actors to interact is through asynchronous, immutable messages placed in their mailboxes.
Let’s break down the implications of this:
- Isolation: Each actor is a fortress. Its internal state is its own. It never has to worry about another thread corrupting its data because no other thread can see it. This eliminates data races at the architectural level.
- Serialization: An actor’s mailbox acts as a perfect queue. Messages are processed one at a time, from start to finish. This means that within the context of processing a single message, an actor’s state is owned outright. No locks are needed because the execution is effectively single-threaded.
- Asynchrony: When you send a message, you don’t block waiting for a reply. You fire-and-forget. The sender and receiver are decoupled in time. This allows for massive parallelism and fault tolerance.
Erlang/OTP: The Godfather of Actor Systems
Erlang was designed by Ericsson in the 1980s for building telecommunication switches, which demanded extreme reliability (99.9999999% uptime, or “five nines”). The developers accidentally invented a near-perfect implementation of the Actor Model (though they called them “processes” and “messages”).
Erlang’s key innovation for the Actor Model is supervision and fault tolerance.
- Let it Crash: Erlang processes are incredibly lightweight (a few hundred bytes of memory). The philosophy is not to write defensive code that handles every possible error. Instead, if a process encounters an unexpected error, it dies. That’s it. It stops executing.
- Supervisors: Other processes, called supervisors, are responsible for monitoring child processes. When a child dies, the supervisor has a pre-defined strategy: restart the child, restart all children, or stop itself. This creates a hierarchical tree of fault tolerance.
- The Link: Processes can be linked. If one linked process dies, a signal is sent to the other, which can then decide how to handle it. This prevents error propagation from corrupting independent parts of the system.
This means you can write a system that is self-healing. A bug causes a process to crash? The supervisor restarts it. A memory leak in a process? It dies, freeing its memory, and is reborn clean. This is a fundamentally different approach from the Java/C# world, where a single uncaught exception in a thread can bring down the entire application.
Implementation in Rust: The actix Framework
Rust, with its focus on zero-cost abstractions and fearless concurrency, is a natural fit for the Actor Model. The actix framework is a powerful implementation, though it’s worth noting it uses asynchronous message passing, not the strict “one message at a time” model of Erlang. Let’s model our bank transfer using actix.
First, define our Account actor:
use actix::{Actor, Context, Handler, Message};
use std::collections::HashMap;
// --8<-- [start:account_actor]
// The state of our account actor
#[derive(Default)]
struct Account {
id: u32,
balance: i64,
// Maybe a history of transactions?
operations: Vec<Transaction>,
}
#[derive(Debug)]
struct Transaction {
from: u32,
to: u32,
amount: i64,
timestamp: std::time::Instant,
}
// Messages that our account can receive
#[derive(Message)]
#[rtype(result = "i64")]
struct GetBalance;
#[derive(Message)]
#[rtype(result = "bool")]
struct Deposit {
from: u32,
amount: i64,
}
#[derive(Message)]
#[rtype(result = "Result<i64, String>")]
struct Withdraw {
to: u32,
amount: i64,
}
impl Actor for Account {
type Context = Context<Self>;
}
// Handler for GetBalance message
impl Handler<GetBalance> for Account {
type Result = i64;
fn handle(&mut self, _msg: GetBalance, _ctx: &mut Context<Self>) -> Self::Result {
self.balance
}
}
// Handler for Deposit message
impl Handler<Deposit> for Account {
type Result = bool;
fn handle(&mut self, msg: Deposit, _ctx: &mut Context<Self>) -> Self::Result {
self.balance += msg.amount as i64;
self.operations.push(Transaction {
from: msg.from,
to: self.id,
amount: msg.amount as i64,
timestamp: std::time::Instant::now(),
});
true
}
}
// Handler for Withdraw message
impl Handler<Withdraw> for Account {
type Result = Result<i64, String>;
fn handle(&mut self, msg: Withdraw, _ctx: &mut Context<Self>) -> Self::Result {
if self.balance >= msg.amount as i64 {
self.balance -= msg.amount as i64;
self.operations.push(Transaction {
from: self.id,
to: msg.to,
amount: msg.amount as i64,
timestamp: std::time::Instant::now(),
});
Ok(self.balance)
} else {
Err(format!("Insufficient funds in account {}. Balance: {}, needed: {}", self.id, self.balance, msg.amount))
}
}
}
// --8<-- [end:account_actor]
Now, the transfer logic. This is not a method on an account; it’s a separate actor or even just an asynchronous main function that orchestrates the other actors.
use actix::prelude::*;
#[actix::main]
async fn main() {
// Start two account actors
let addr1 = Account::create(|ctx| Account {
id: 1,
balance: 1000,
..Default::default()
});
let addr2 = Account::create(|ctx| Account {
id: 2,
balance: 500,
..Default::default()
});
// Transfer $100 from account 2 to account 1
let amount = 100;
let from_addr = addr2.clone();
let to_addr = addr1.clone();
// Send a withdraw message to account 2
let result = from_addr.send(Withdraw { to: 1, amount }).await;
match result {
Ok(Ok(_)) => {
// Withdraw succeeded, now deposit
let _ = to_addr.send(Deposit { from: 2, amount }).await;
println!("Transfer of ${} completed successfully.", amount);
}
Ok(Err(e)) => {
println!("Transfer failed: {}", e);
}
Err(_) => {
println!("Failed to communicate with account actor.");
}
}
// Check balances
let bal1 = addr1.send(GetBalance).await.unwrap();
let bal2 = addr2.send(GetBalance).await.unwrap();
println!("Account 1: ${}, Account 2: ${}", bal1, bal2);
}
What’s different?
- No locks: The
Accountactor’shandlemethods run in a single-threaded context. It modifiesself.balancedirectly, without any mutex. This is impossible to get wrong. - Error handling is explicit: The
Withdrawmessage returns aResult. The transfer logic checks it. If the withdraw fails (insufficient funds), the deposit is never made. The system is consistent by design. - Asynchronous by default: The
.awaitcall ensures we don’t block an OS thread while waiting for the actor’s response. Theactixruntime handles the scheduling.
This is a simple example, but it scales. Imagine a trading system with thousands of account actors. Each one is its own little world. There’s no global lock pool. No risk of deadlock. The only synchronization is through message passing.
Implementation in Python: The pypy and ray Approaches
Python has a fundamentally different relationship with concurrency. The Global Interpreter Lock (GIL) prevents true parallel execution of Python bytecode within a single process. However, for I/O-bound or embarrassingly parallel tasks, we can still use the Actor Model.
Let’s build a similar system using the pypy actor library (which uses green threads) or, more practically for modern Python, the ray framework, which is designed for distributed and actor-based computing.
import ray
from typing import Dict, List
@ray.remote
class AccountActor:
def __init__(self, account_id: int, initial_balance: int):
self.id = account_id
self.balance = initial_balance
self.transactions: List[Dict] = []
def get_balance(self) -> int:
return self.balance
def deposit(self, from_id: int, amount: int) -> bool:
self.balance += amount
self.transactions.append({
'from': from_id,
'to': self.id,
'amount': amount,
'type': 'deposit'
})
return True
def withdraw(self, to_id: int, amount: int) -> (bool, str):
if self.balance >= amount:
self.balance -= amount
self.transactions.append({
'from': self.id,
'to': to_id,
'amount': amount,
'type': 'withdraw'
})
return True, "OK"
else:
return False, f"Insufficient funds. Balance: {self.balance}, needed: {amount}"
# Initialize Ray
if not ray.is_initialized():
ray.init()
# Create two account actors
account1 = AccountActor.remote(1, 1000)
account2 = AccountActor.remote(2, 500)
# Transfer logic
amount = 100
result = ray.get(account2.withdraw.remote(1, amount))
if result[0]:
ray.get(account1.deposit.remote(2, amount))
print("Transfer successful")
else:
print(f"Transfer failed: {result[1]}")
# Check balances
bal1 = ray.get(account1.get_balance.remote())
bal2 = ray.get(account2.get_balance.remote())
print(f"Account 1: ${bal1}, Account 2: ${bal2}")
Crucial differences from the lock-based approach:
ray.remote: This decorator turns a Python class into an actor. It can be run on a local machine or across a cluster. The actor’s methods are invoked remotely via.remote(). The actor’s state is not shared; it’s encapsulated within the actor’s process (or potentially a separate machine).- Immutability of messages: The arguments to
.remote()calls are serialized and sent to the actor. They are effectively immutable from the caller’s perspective. - Fault isolation: If this actor crashes (e.g., due to a bug in the
withdrawmethod), the other actors and the rest of the system remain unaffected. Ray provides mechanisms for fault tolerance in production.
This example demonstrates that the Actor Model isn’t just for systems languages. It provides a robust mental model for concurrency in dynamically-typed languages as well, helping you avoid the Python-specific pitfalls of the GIL.
Chapter 3: Beyond the Basics: Advanced Actor Model Patterns
The simple bank transfer is a good start, but the true power of the Actor Model emerges with more complex patterns.
1. The Supervisor Tree: Building Resilient Systems
In Erlang/OTP, supervision is core. Let’s imagine we have a service that fetches stock prices from an external API. This service is an actor that can fail (network timeout, malformed response). We can create a supervisor that monitors it.
% In Erlang syntax (pseudo-code for brevity)
-module(stock_supervisor).
-behaviour(supervisor).
init(_Args) ->
% Define the child process specification
ChildSpec = #{id => stock_fetcher,
start => {stock_fetcher, start_link, []},
restart => temporary, % Don't restart on crash for a temporary issue
shutdown => 5000,
type => worker,
modules => [stock_fetcher]},
% Strategy: one for one (restart only the dead child)
{ok, {{one_for_one, 5, 10}, [ChildSpec]}}.
The supervisor strategy could be:
- one_for_one: Restart just the crashed child.
- one_for_all: Restart all children (if the failure is catastrophic and all are compromised).
- rest_for_one: Restart the crashed child and all children started after it (if there’s a dependency chain).
This creates a “crash early, crash often” mentality. You don’t try to handle every network error in the stock fetcher. You let it crash, and the supervisor resets the state and tries again. This is far more robust than nested try-catch blocks that might leave the system in an inconsistent state.
2. The Ask Pattern (Request-Response)
We already saw this in Rust with send(msg).await. An actor sends a message and expects a reply. This is how you get information out of an isolated actor. In Erlang, you use the ! operator to send a message and a receive block to wait for a reply, often including a message identifier to correlate responses.
% Actor A
ask_balance(AccountPid) ->
AccountPid ! {self(), get_balance},
receive
{AccountPid, Balance} -> Balance
after 5000 -> % Timeout
timeout_error
end.
% Actor B (Account)
handle_get_balance({FromPid, get_balance}) ->
FromPid ! {self(), my_balance}.
This pattern is not just for queries. It’s the foundation of remote procedure calls (RPC) in distributed systems, where the “receive” block might be a promise or a future.
3. The Event Sourcing Pattern
An actor’s private state is precious. What if you need to reconstruct it after a crash? The Actor Model naturally supports event sourcing: instead of modifying state directly, an actor processes a series of events. The actor’s state is the cumulative result of applying all past events. This is exactly what the transactions list is doing in our Rust example!
Example: A Bank Account as an Event Stream
Instead of having a balance field and mutating it, an account actor could process a stream of events:
enum Event {
Deposited { from: u32, amount: i64 },
Withdrawn { to: u32, amount: i64, reason: String },
AccountCreated { id: u32, initial_balance: i64 },
FeeApplied { amount: i64, description: String },
}
impl Account {
fn apply_event(&mut self, event: Event) {
match event {
Event::Deposited { amount, .. } => self.balance += amount,
Event::Withdrawn { amount, .. } => self.balance -= amount,
Event::AccountCreated { initial_balance, .. } => self.balance = initial_balance,
Event::FeeApplied { amount, .. } => self.balance -= amount,
}
}
fn handle_command(&mut self, cmd: Command) -> Vec<Event> {
match cmd {
Command::Deposit { from, amount } => {
self.apply_event(Event::Deposited { from, amount });
vec![Event::Deposited { from, amount }]
}
Command::Withdraw { to, amount } if self.balance >= amount => {
self.apply_event(Event::Withdrawn { to, amount, reason: "withdrawal".to_string() });
vec![Event::Withdrawn { to, amount, reason: "withdrawal".to_string() }]
}
_ => vec![],
}
}
}
Now, if the actor crashes and restarts, it can reload all past events from a durable log and replay them to reconstruct its state. This is the foundation of CQRS (Command Query Responsibility Segregation) and Event Sourcing, providing a complete audit trail and powerful debugging capabilities.
4. The Router Pattern
Sometimes you want to distribute work across a pool of worker actors. An actor router is an actor that receives messages and forwards them to a pool of workers using a routing strategy (round-robin, random, smallest mailbox, consistent hashing).
// Using actix's built-in router feature
use actix::prelude::*;
use rand::Rng;
struct Worker {
id: u32,
}
impl Actor for Worker {
type Context = Context<Self>;
}
// Message for workers
#[derive(Message)]
#[rtype(result = "()")]
struct DoWork(String);
impl Handler<DoWork> for Worker {
type Result = ();
fn handle(&mut self, msg: DoWork, _ctx: &mut Context<Self>) {
println!("Worker {} processing: {}", self.id, msg.0);
// Simulate work
std::thread::sleep(std::time::Duration::from_millis(rand::thread_rng().gen_range(10..100)));
}
}
// A simple router actor
struct Router {
workers: Vec<Addr<Worker>>,
}
impl Actor for Router {
type Context = Context<Self>;
}
impl Handler<DoWork> for Router {
type Result = ();
fn handle(&mut self, msg: DoWork, _ctx: &mut Context<Self>) {
// Round-robin routing
let index = some_consistent_hash(&msg.0) % self.workers.len();
self.workers[index].do_send(msg); // Fire and forget
}
}
This is a powerful pattern for building scalable microservices. A single “dispatcher” actor can distribute incoming API requests to a pool of worker actors that handle business logic, database queries, and external calls.
Chapter 4: The Actor Model in the Real World: Scaling from a Single Machine to a Cluster
The Actor Model isn’t just for in-process concurrency. It’s the foundation of some of the most scalable distributed systems on the planet.
Case Study 1: WhatsApp (Erlang/OTP)
WhatsApp’s claim to fame is handling 2 million+ concurrently connected users per server. How?
- Lightweight Processes: Each user is represented by an Erlang process (an actor). These processes are not OS threads; they are scheduled by the Erlang VM. You can have hundreds of thousands of them on a single server.
- State as a Process: The user’s chat state, connection state, and message history are all part of that process’s private state. There is no shared database for the session state.
- Fault Tolerance: If a process crashes, only that user’s connection drops for a fraction of a second. The supervisor restarts it, and the user reconnects. This makes the overall system incredibly robust.
- Hot Code Swapping: Erlang allows you to upgrade the code of a live system without stopping it. You can deploy a new version of the routing logic or the message processing engine without taking the service down. This is a critical feature for a 24/7 global service.
Case Study 2: Akka (Scala/Java)
Akka is a toolkit for building concurrent and distributed applications on the JVM. It’s heavily inspired by Erlang/OTP.
- Location Transparency: In Akka, an actor is referenced by a path, like
akka://system@host:port/user/my-actor. The system hides whether the actor is in the same process, on a different machine, or behind a load balancer. Sending a message to a remote actor is syntactically identical to sending to a local one. - Cluster Formation: Akka can form a dynamic cluster of actor systems. Actors can be deployed across the cluster, and the system handles node failures, split-brain scenarios, and automatic rebalancing.
- Persistence: Akka Persistence is an event-sourcing library that allows actors to persist their state to a durable store (e.g., Cassandra, Kafka, PostgreSQL). This is used by companies like Intel, Samsung, and PayPal for building robust, scalable backends.
The Challenge: Distributed State Management
The Actor Model doesn’t make distributed computing easy; it makes it explicit. You can’t avoid the CAP theorem (Consistency, Availability, Partition Tolerance). You have to choose your trade-offs.
- Consistency: If you use event sourcing and persistence, you can achieve strong consistency by ensuring all events are processed in order.
- Availability: If a node fails, the actors on that node are unreachable. You must have a replication strategy or a failover mechanism. This is where the “supervisor” pattern goes distributed. A supervisor on another node can take over responsibility for a failed actor.
- Partition Tolerance: Network partitions are inevitable. The Actor Model’s isolation means that a partitioned actor can still function correctly in its own little world, but it might not have the latest data. You must decide how to handle stale data during a partition.
Tools like Apache Kafka work beautifully with the Actor Model. An actor can be a Kafka consumer or producer. The actor’s mailbox can be backed by a Kafka topic, providing durable, fault-tolerant message delivery. The Actor Model gives you the architecture; the messaging infrastructure provides the resilience.
Chapter 5: The Critique: When is the Actor Model NOT the Right Tool?
Like any powerful abstraction, the Actor Model has its downsides and is not a silver bullet.
- Complexity for Simple Problems: Using actors for a simple counter that is only ever touched by one thread is overkill. The overhead of a mailbox, a scheduler, and message serialization (even in-process) is not trivial. The actor model shines when complexity is high, not low.
- Debugging and Testing: Debugging a distributed system of actors is notoriously hard. You can’t just set a breakpoint and step through messages. You need sophisticated tracing, logging, and snapshotting (e.g., Erlang’s
observertool, or tools like Lightrun). - Deadlock is still possible in message passing: While you can’t have a resource deadlock (two actors holding locks), you can have a communication deadlock (or starvation). For example:
- Actor A sends a message to B and waits for a reply.
- Actor B sends a message to A and waits for a reply.
- Neither can process the other’s message because they are both blocking on a reply. This is a deadlock in the message passing layer. The solution is to always use a timeout and an async ask pattern.
- Message Loss: In a distributed system, messages can be lost. The Actor Model doesn’t guarantee delivery. You must build your own retry and acknowledgment mechanisms, which adds complexity.
- Performance Overhead: For very high-throughput, low-latency operations (like a network packet router), the overhead of mailbox scheduling can be a problem. Languages like Rust and C++ with careful optimization are often better for these use cases. The LMAX Disruptor pattern is an alternative for ultra-low-latency systems that avoids the overhead of actors.
Conclusion: Reclaiming Control from the Tyrant
The lock is a tyrant. It demands total control over the essence of your program, the order of operations. It punishes the slightest oversight with a system-wide deadlock or a silent data corruption. The Actor Model is a rebellion against this tyranny. It is a declaration of independence from shared state.
You don’t fight the CPU for control. You don’t orchestrate a complex ballet of thread synchronization. You design a system of independent, message-passing entities—actors—that collaborate by sending immutable messages. Each actor is a king in its own castle, safe from the chaos outside.
The Actor Model doesn’t make concurrency easy—it makes it possible to reason about. It forces you to think about the communication between components, not the state inside them. It embraces failure as a natural part of computation, building self-healing systems that can survive the death of individual parts.
The journey from lock() and unlock() to actor.tell(msg) is a fundamental shift in how you think about computation. It is a shift from a world where you fight to control shared resources to a world where you simply coordinate independent agents. The path is not without its own challenges—distributed consistency, debugging, and careful message design—but it offers a path to building systems that are truly concurrent, resilient, and scalable.
So, the next time you find yourself spending hours debugging a deadlock or trying to prove that your lock-free data structure is correct, consider the Actor Model. It might just be the liberation your concurrent code needs. Your future self, and your users, will thank you.