Why our voice agents run customer code in WebAssembly

How every customer ships their own logic into a live phone call — and why the interesting part turned out not to be the sandbox.

A call that needs four backends

An EMI reminder call, about nine seconds in. The agent has just asked for a payment date and the customer says, in Hinglish, that they already paid yesterday.

To say anything useful next, the agent has to:

• look up the loan account in the lender's loan management system
• check whether a payment posted in the last 48 hours — which lives in a different system, run by a different vendor
• work out that the payment posted but hasn't reconciled yet, and produce a reference number for it
• write the outcome back into the lender's CRM so the collections team doesn't call again tomorrow

Four systems, and roughly a second to do it in before the silence starts sounding like a dropped call.

Now the part that actually shapes the architecture: no two of our customers have the same four systems. Authentication is a static API key here, mutual TLS there, and at a third customer a JWT we have to sign with a private key they issued us. One returns amounts in paise, as strings. Another nests the outstanding balance three levels deep under a field called RESP_DTLS. A third paginates a response that is never longer than one page.

This post is about what we do with that. It isn't about prompt engineering and it isn't about models — the model in this story is doing the easy part.

Configuration was the right first answer, and then it wasn't

Our first version of this was declarative. A campaign could define an API call: method, URL, headers, some templating for the request body, and a mapping from the response into variables the agent could speak.

It worked for a surprisingly long time, because it covers the genuinely common case — fetch one thing, read a field out of it.

Then the requests started arriving. Call this endpoint, and if the status is one of these three values, call a second one. Retry that one twice, but only on a 502. Compute an HMAC over a canonical string built from four fields in a specific order, and put it in a header. Convert this date, which arrives as DDMMYYYY with no separators, into something a human would say out loud. Sum an array. Sum it, unless one element has a flag set.

Each of these is reasonable. Each one, on its own, is a small addition to the config format. Eighteen months of them and you have invented a programming language — one with no type checker, no tests, no debugger and no local development story. If your configuration format has grown an if, you're writing a programming language. You just haven't admitted it, so nobody has built you a stack trace.

So we let customers ship code instead. Which immediately created two problems worth taking seriously.

Constraint 1: one customer's bad code cannot end another customer's call

Our orchestrator is not a request-response service. A single process carries many concurrent live phone calls, each one a stateful audio pipeline with a human attached to it.

That changes what a bug costs. In a normal API service, code that hangs gets one request killed and a 500 logged. In ours, a tenant integration that blocks forever doesn't produce an error page — it produces dead air, for someone who is mid-sentence, and then a hangup.

So "run the customer's code in our process, in our language, on our event loop" was never going to work. But the obvious escape hatch — give each plugin its own process, or its own container — doesn't survive contact with the latency budget either. We have roughly a second for the entire turn, and inference and speech synthesis have already claimed most of it. Spawning a container per tool call spends the whole budget on the spawn. Keeping warm sandbox pools per tenant per pod, across thousands of campaigns, is a fleet management problem we'd rather not invent.

What we wanted was an isolation boundary measured in microseconds.

Constraint 2: customer logic ships on the customer's clock

The second problem is organisational, and it's the one that actually drove the decision.

A customer changes a field name in their payments API on a Tuesday afternoon. If their integration logic lives inside our monorepo, fixing that means a platform release: our CI, our review, our deploy train, our rollback risk, and a pod restart that drops live calls. One customer's urgency becomes the whole platform's release pressure, and every such fix carries a non-zero chance of breaking a tenant who had nothing to do with it.

We wanted a deployment unit that is one customer's logic — versioned on its own, rolled out and rolled back on its own, with a blast radius that stops at the edge of that campaign. And we wanted it live without restarting anything, because restarting means dropping calls.

That's a short list, and it points fairly directly at one answer.

Why WebAssembly, and why Wasmtime

A WebAssembly module starts with no ambient authority at all. It cannot open a socket, read a file, look at the clock or learn its own hostname. Every capability it has is one the host deliberately handed it.

This inverted default is the whole reason we're here. Every other sandbox we evaluated starts from "a process, which can do everything" and then fences things off — seccomp filters, namespaces, a read-only root. You're subtracting from a permissive baseline, one forgotten syscall away from a bad afternoon. WebAssembly starts at zero and you add. When the code isn't yours, addition is a much safer direction to be working in.

Two other properties mattered:

• Instantiation is cheap. A module is compiled once and instantiated per call, in microseconds. That's cheap enough that we give every call its own fresh instance rather than sharing one between calls — so there is no mutable state that can leak from one conversation to the next, or from one tenant to another. Isolation between tenants isn't a policy we enforce; it's a consequence of the instance lifetime.
• The runtime has resource metering as a first-class primitive. Fuel and epoch-based interruption mean a host can put a hard deadline on guest execution, and linear memory can be given a ceiling, without the guest cooperating. That isn't a property you get from a scripting language embed.

We run Wasmtime underneath, because it's mature, it has those metering primitives built rather than bolted on, and it embeds cleanly into the Python process that runs our orchestrator.

We looked seriously at microVMs. Firecracker-class isolation is genuinely stronger than ours, and if the threat model were "actively hostile tenant attempting escape" we'd be having a different conversation. Our threat model is closer to "a correct-looking integration, written under deadline pressure, that occasionally does something catastrophic by accident" — and for that, a microVM per tool call is not a cost worth paying. We're not zealots about this. If the threat model changes, the answer changes.

A note on where we actually are. It would be tidier to claim we run every plugin under a hard fuel budget and a memory ceiling. In truth, the guest-side deadline is the part of this design we're still tightening; what carries the weight in production today is the capability boundary, the per-call instance lifetime, and hard timeouts on the host side of every call a plugin can make outward. That last one exists because of a lesson worth passing on: a plugin blocked on a stalled upstream will hold the worker thread it's running on until that request returns, and a default HTTP client will wait forever. Connect and read timeouts on the host side of the boundary were not a nice-to-have — they were the difference between a slow backend and a leaking thread pool.

Why we write the plugins in Rust

Once you've chosen WebAssembly, you have to choose what compiles to it. We standardised on Rust, and the deciding factor was less glamorous than you'd hope: binary size.

Every plugin is an artifact we fetch over the network, hold in a size-capped cache, and instantiate on the call path. Size shows up three times — as cold-start latency the first time a pod sees a campaign, as memory per resident plugin across thousands of campaigns, and as cache eviction pressure that turns yesterday's warm plugin into today's cold start.

These are orders of magnitude for the kind of plugin we write, not a benchmark — your mileage will vary with what you pull in. But the shape of the table is stable, and the shape is the point: Rust is the only row where the binary is mostly your code.

Two things came along for free and turned out to matter as much:

No garbage collector means no pause. A GC pause inside a tool call isn't a p99 line on a dashboard, it's a gap in a sentence. Rust's memory model gives us latency we can reason about instead of latency we have to measure and hope about.

The type system is doing real work. This code computes outstanding balances, parses dates from four different formats, and signs requests. It's exactly the code where a silent None or an implicit string-to-number coercion becomes a customer-facing incident. Having the compiler refuse to build the plugin is a much better place to find that out than having an agent say it out loud to a borrower.

The contract is two functions: discover, then execute

We didn't want a plugin registry. Registries drift — someone adds a function, forgets to declare it, and the source of truth quietly disagrees with reality.

So plugins are self-describing. The contract is small enough to state in three lines:

• init — hand the plugin its context for this call: campaign configuration, journey variables, customer and call identifiers.
• list_functions — return the tools this plugin provides, as schemas, in the shape the model already expects.
• execute_function — given a function name and arguments, do the thing and return a result.

At call setup the plugin manager resolves which plugins the campaign pins, instantiates each one, initialises it, and asks it what it can do. The schemas that come back are wrapped into callables and registered with the model as its tools. When the model emits a tool call, we route it by name back through the single execute_function entrypoint on the right instance — one export, not one export per tool, which is what keeps the host side free of any per-tool knowledge.

Where a campaign loads more than one plugin, we namespace function names with the plugin's slug on the way out and strip the prefix on the way back in, so two plugins can both expose a get_status without either knowing the other exists.

sequenceDiagram
    autonumber
    participant M as Language model
    participant O as Orchestrator
    participant PM as Plugin manager
    participant S as WASM sandbox
    participant B as Customer backends

    Note over O,PM: Call setup
    O->>PM: resolve pinned plugin versions
    PM->>PM: cache lookup → instantiate
    PM->>S: init(call context)
    PM->>S: list_functions()
    S-->>PM: tool schemas
    PM-->>O: registered tools
    O->>M: here are your tools

    Note over M,B: Mid-call — "I already paid yesterday"
    M->>O: tool_call: check_payment_status(loan_id)
    O->>PM: dispatch
    PM->>S: execute_function(name, args)
    S->>B: look up loan (via host function)
    B-->>S: raw response
    S->>B: check recent payments
    B-->>S: raw response
    S->>S: reconcile, convert, decide
    S-->>PM: clean, flat result
    PM-->>M: tool result
    M->>O: "Yes, I can see a payment from yesterday…"

Adding a tool means adding a function and republishing. Nothing on our side is edited, reviewed or deployed.

One deliberate choice in that diagram: when a plugin throws, we don't fail the turn. The error comes back as the tool's result, and the model gets to recover in language — apologise, ask a clarifying question, move on. A failed integration should sound like a person who doesn't know something, not like a dropped call.

The plugin is a facade, not a proxy

This is the part that surprised us, and it's now the main reason we'd make this choice again even if isolation had never been a concern.

The naive way to wire a backend into an agent is one tool per HTTP endpoint. get_loan, get_payments, get_reference_number. It feels clean. It's a bad idea, for three reasons:

It makes the model the orchestrator. Three endpoints means three round trips through inference, each one a chance for the model to call them out of order, skip one, or decide it has enough after two. You've moved your business logic into a probabilistic system and you can't unit test it.

It makes the model a JSON parser. The raw response goes into the context window — every envelope field, every null, every RESP_DTLS. The model now has to infer that outstanding_amt: "4520000" is in paise. Usually it does. Occasionally the customer is told they owe forty-five lakh rupees.

It makes vendor changes your problem forever. Every field rename ripples into prompts, and prompts don't have a compiler.

So our plugins expose one tool where a naive integration would expose four. check_payment_status(loan_id) is a single call from the model's point of view. Inside the sandbox it's a sequence: hit the loan system, hit the payments system, compare timestamps, apply that customer's reconciliation window, convert paise to rupees, format the date the way a person actually says dates, and return a flat struct with four fields and no nulls.

The model gets an answer, not a payload. All the multi-step orchestration, all the vendor-specific decoding, and all the business rules that have to be exactly right live in typed, tested code behind a stable interface.

The model should decide what to do. It shouldn't have to decode RESP_DTLS.

The side effect is latency. Four backend calls behind one tool call is one inference round trip instead of four — and the backend calls happen inside a compiled binary sitting close to the network, not between two token streams.

The plugin cannot open a socket

Everything a plugin reaches, it reaches by asking the host. In practice that's a small and deliberately boring set of capabilities:

• Outbound HTTP, proxied. The plugin hands the host a request description; the host makes the call, through a forward proxy inside our cluster. That single door is where timeouts and retry policy are enforced, where stable egress IPs come from — customers' firewalls need something to allowlist — where redaction happens before anything is logged, and where every external call gets traced and attributed to a call and a campaign.
• Cryptographic helpers — signing a JWT, hashing a canonical string. Note what this buys: the customer's private key never enters guest memory. The plugin asks for a signature; it doesn't get the key. A plugin that's compromised, or simply wrong, can't exfiltrate a credential it was never given.
• Scoped key-value storage, at campaign scope and at call scope, with different lifetimes. A plugin can memoise an expensive lookup for the duration of a call, or cache a token across a campaign, without us handing it a database connection.
• Structured logging and analytics, so plugin authors can debug without inventing their own channel out.
• Writes back into the call — output variables, call metadata, journey events — which is how a plugin's result becomes something the rest of the platform acts on after the call ends. A value a plugin writes is authoritative; it beats whatever our post-call extraction would have inferred from the transcript.
• Limited control over the live pipeline. This is the one that goes furthest. A plugin can, for instance, start a form-fill flow: send the caller an SMS link, and at the same time tell the conversation's idle detector to stand down and check back later — so the agent waits quietly while someone types, instead of asking "are you still there?" three times.

Every one of these is a place we can enforce policy, add a metric, or cut off a misbehaving tenant, because there's exactly one door and we built it.

From git push to a running call

Plugin source lives in its own repository, outside the platform monorepo, as a normal Cargo workspace — one crate per customer campaign, plus a small shared crate wrapping the host functions. That normality is the point: a customer's engineers don't have to learn our world to work in it. They get cargo test and a type checker.

Publishing goes through a versioned control plane, driven by a small CLI in CI: build, then promote. Promote is a sequence of four calls — create the plugin, create a draft version, get a presigned URL and PUT the binary straight to object storage, then activate the version. Active versions are immutable; you can't upload over one. The binary never passes through our API, and the storage layout is tenant-scoped the whole way down: domain, then campaign, then plugin, then version.

CI authenticates against a separate, narrow API surface gated by a build token rather than by the human OIDC path — GitHub Actions doesn't have a Google identity, and we'd rather give it a scoped door of its own than teach it to impersonate a person.

flowchart TD
    A[git push to plugin repo] --> B[CI: cargo build, wasm32 target]
    B --> C[Run plugin tests]
    C --> D[Create draft version]
    D --> E[PUT binary via presigned URL]
    E --> F[Activate → immutable]
    F --> G[Pin campaign to new version]

    G --> H{Next call on this campaign}
    H --> I[Resolve pinned version → S3 path]
    I --> J{Local disk cache?}
    J -->|hit| N[Instantiate, run, free]
    J -->|miss| K{Shared cache?}
    K -->|hit| M[Populate disk, instantiate]
    K -->|miss| L[Fetch from object storage]
    L --> M
    M --> N

    classDef ci fill:#eef4ff,stroke:#5b7cc4,color:#12203a
    classDef run fill:#eefaf1,stroke:#4f9d6b,color:#0f2a1a
    class A,B,C,D,E,F,G ci
    class H,I,J,K,L,M,N run

Three tiers underneath: a local disk cache on each pod, LRU with a size cap and a TTL; a shared cache in front of storage so a fresh pod doesn't cold-start every campaign it's never seen; and object storage as the source of truth.

The cache key is worth a paragraph, because it's where the naive version breaks. Plugins are fetched over presigned URLs, and a presigned URL contains a fresh signature and expiry every time you mint one — so keying the cache on the URL gives you a permanent miss rate of 100%. We hash the URL with the query string stripped. What's left is the tenant-scoped path, which already ends in the version, so the key is effectively (domain, campaign, plugin, version).

Which is what makes hot reload boring, and boring is the goal. There is no invalidation message, no cache-bust, no coordination. Activating a new version changes which version the campaign resolves to; the next call computes a different key, misses once, fetches, and warms. Plugin instances are per-call and freed at the end of the call, so there's nothing long-lived holding the old binary open. No restart, no dropped calls, no deploy. Rollback is re-pinning the previous version, which is already sitting warm in every cache that ever saw it.

We instrument this like the dependency it is: cache hit and miss rates per plugin, fetch duration, binary size distribution, eviction counts by reason. Binary size especially is a number we watch rather than one we assumed — it's what quietly decides whether a cold start is invisible or audible.

The same machinery runs everything else

Tool calling during a conversation is the most visible use, but it's not the only one. The same contract, the same sandbox and the same cache run most of a call's lifecycle. Different exported function, different typed request and response, identical machinery underneath:

• Before we dial, a plugin gets the call context and can enrich it, override the number, or cancel the call outright — the cheapest call is the one a customer's own system tells you not to make.
• After the call, a plugin receives the transcript, extracted answers, outcome, duration and a link to the recording, and writes results wherever that customer needs them.
• Around rescheduling and warm transfers, plugins decide whether and when to try again, and what should happen after a handoff.
• Around the model itself, a plugin can transform the message history before every inference call — which is how customer-specific context management happens without customer-specific code in our inference path.

One of these deserves singling out, because it's the clearest example of a plugin doing something a config format never could: a pre-call hook can return a list of keywords to boost, and those go into the speech recogniser before the call connects. A customer who knows this journey is about a specific product can tell the ASR what proper nouns to expect on this particular call. Customer code, in Rust, in a sandbox, tuning transcription accuracy for one conversation.

That breadth is most of the value of having built it properly once. There is one runtime to secure, one cache to tune, one set of dashboards, and one thing a plugin author has to learn.

A synchronous sandbox inside an asynchronous orchestrator

One implementation detail, because it bit us and it will bite anyone who does this.

A WebAssembly call is synchronous and CPU-bound. Our orchestrator is a Python asyncio process with many live calls on one event loop. Call into a sandbox directly from a coroutine and you've blocked the loop — and not for the call that made the tool call. For every call on that pod.

Forty milliseconds of blocking is a scheduling hiccup in a web service. Spread across thirty concurrent conversations it's thirty small gaps in thirty sentences, and people notice gaps in sentences long before anyone notices latency on a dashboard.

So instantiation and execution both run on a worker thread pool, and the event loop awaits the result. The wrinkle is that host functions still need to reach back into the live pipeline — the form-fill example above has to find the conversation it belongs to — so the call context is explicitly copied into the worker thread rather than assumed. Unglamorous, and the difference between a sandbox that's invisible and one that makes the whole pod sound slightly broken.

What's next

The honest summary is that this started as an isolation problem and turned into an interface problem. We adopted WebAssembly because we needed customer code that couldn't take down a pod full of live calls. We kept it because the plugin boundary turned out to be the right place to put everything the model shouldn't be doing — the orchestration, the decoding, the arithmetic that has to be exact.

There's a lot we've skipped. How plugins get tested against a simulated call before they go anywhere near a phone line. What happens when a plugin is correct but the backend behind it is down, which is a far more common failure than a bad plugin. How you version a tool schema when the model has already been told the old one. Why we ended up running a second, non-WASM plugin engine behind the same registry for batch workloads, and what that taught us about where the sandbox is worth its cost. And the thing we get asked about most: how you give a customer's engineering team enough of a local development loop that they'll actually write plugins instead of filing tickets.

We'll cover those. We'll also cover the tenant plugin that appeared to be leaking memory for a week and turned out to be a thread pool quietly filling up with requests to a server that had stopped answering.