Build a Speech-to-Speech Phone Agent with Cloudonix CXML and Node.js

2026-11-08 · Cloudonix Team

Wiring a speech-to-speech model to a phone number looks simple until the first caller interrupts it. Now you're managing live audio, model state and business tools while a person waits on the line. Cloudonix gives you a clean split: CXML controls the call, a WebSocket carries the audio and your Node.js service runs the agent.

A phone agent is a stateful media service

A chat demo can get away with a request and a response. A phone agent cannot. The call stays open, audio arrives continuously and the model may still be speaking when the caller changes their mind. Add a calendar lookup or an order update and your little voice demo has become a stateful communications service.

The model is only one dependency. You also need public call control, TLS for the media socket, per call state and a safe way to run tools. If you tie those pieces to one vendor's helper library, changing the model later can mean rewriting the telephony adapter too.

That coupling also makes cost hard to read. Cloudonix doesn't use the usual per minute platform pricing model, which is useful when you're testing long conversations. You'll still pay for your model, hosting and any carrier services, so prototype with the same call lengths you expect in production.

Important: Keep two contracts separate in your design. Cloudonix CXML controls the call and opens the stream, while your AI provider defines its own session, audio and tool events.

Put CXML at the edge and keep the model replaceable

We're going to put Cloudonix at the telephony edge and keep the agent portable. An inbound call reaches a Cloudonix voice application. That application fetches CXML from Node. The <Connect><Stream> instruction opens a bidirectional audio WebSocket to your media service. Your service then creates one speech-to-speech session, perhaps with Rime Prism, and connects that session to the call.

The same design works for outbound calls through the Cloudonix API. The model can change without changing the CXML document. Your business tools also stay in Node, where you can test them without owning a phone call.

sequenceDiagram
    participant P as Caller
    participant C as Cloudonix session
    participant N as Node.js application
    participant A as AI provider session
    P->>C: Call reaches the voice application
    C->>N: Request the CXML document
    N-->>C: Return Connect and Stream instructions
    C->>N: Open the audio WebSocket
    N->>A: Create one AI session
    loop While the call is active
        C->>N: Caller audio
        N->>A: Forward audio
        A-->>N: Response audio or tool request
        N-->>C: Response audio
    end
    N-->>C: Close the stream when finished

Cloudonix owns the phone session while Node owns the model session and its tools.

The diagram uses conceptual audio messages, not vendor event names. The Cloudonix <Stream> reference documents the CXML connection and its attributes. Use the full media protocol reference for your deployment when writing the WebSocket handler. Copying start, media, mark or clear handlers from another platform would be guessing with production audio.

Answer with CXML, then open the stream

Start with the smallest useful voice application. This CXML sends caller audio to your service and lets the service return audio on the same stream.

<?xml version="1.0" encoding="UTF-8"?>
<Response>
  <Connect>
    <Stream url="wss://agent.example.com/media" track="inbound_track" />
  </Connect>
  <Hangup />
</Response>

When Cloudonix runs it, execution waits inside <Connect> until the stream terminates. Only then does the document continue to <Hangup>. The inbound_track setting sends audio from the caller to your WebSocket. Cloudonix also supports outbound_track and both_tracks; use those only when you actually need the other direction, or you may end up feeding your agent its own reply.

The url must be a valid WebSocket URL. A failed connection stops the application, so a public wss:// endpoint and a valid certificate belong in your first test, not on a cleanup list for later.

Serve the document with Node built-ins

You don't need a CPaaS SDK to return that document. Node's built-in HTTP server is enough for a development webhook.

import { createServer } from "node:http";

const streamUrl = process.env.STREAM_URL;
if (!streamUrl?.startsWith("wss://") || /[&<>"']/.test(streamUrl)) {
  throw new Error("STREAM_URL must be a simple public wss:// URL");
}

const cxml = `<?xml version="1.0" encoding="UTF-8"?>
<Response>
  <Connect>
    <Stream url="${streamUrl}" track="inbound_track" />
  </Connect>
  <Hangup />
</Response>`;

createServer((req, res) => {
  if (req.url !== "/voice") {
    res.writeHead(404).end();
    return;
  }
  res.writeHead(200, { "Content-Type": "text/xml; charset=utf-8" });
  res.end(cxml);
}).listen(Number(process.env.PORT || 5050));

This server builds the CXML once and responds at /voice. Set STREAM_URL to your public media socket, expose port 5050 through a TLS-enabled tunnel and point the Cloudonix voice application at the public /voice URL. Keep the XML content type. And yes, the rest of the media service still has to exist. CXML is the doorbell, not the whole house.

Use the Cloudonix API for outbound agent calls

An outbound agent starts from your backend instead of an inbound number. Cloudonix documents POST /calls/<domain-name-or-id>/application for this. The request can name an existing voice application, a CXML URL or an inline CXML document. Using a URL keeps the outbound and inbound paths on the same app.

Use the API origin and authorization scheme documented for your account. Here, CLOUDONIX_API_BASE has no trailing slash and CLOUDONIX_AUTHORIZATION contains the complete Authorization header value. Node's built-in fetch handles the request without a telephony SDK.

const domain = encodeURIComponent(process.env.CLOUDONIX_DOMAIN);
const response = await fetch(
  `${process.env.CLOUDONIX_API_BASE}/calls/${domain}/application`,
  {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      "Authorization": process.env.CLOUDONIX_AUTHORIZATION
    },
    body: JSON.stringify({
      destination: "+14155550123",
      "caller-id": "+14155550100",
      url: "https://agent.example.com/voice"
    })
  }
);

if (!response.ok) {
  throw new Error(`Cloudonix API ${response.status}: ${await response.text()}`);
}

When the destination answers, Cloudonix connects the call to the voice application. That application returns the same <Connect><Stream> document and your media service takes over. Optional send-digits can enter DTMF after answer and before the application runs, which is useful when a test call has to pass an auto attendant.

Warning: This is a privileged API. The credential must have system, tenant or domain administrator authorization for the domain, and Cloudonix returns HTTP 401 for insufficient authorization. Keep the credential in your backend and check the response before logging a call as started.

The Node service owns audio, tools and interruptions

Your WebSocket handler owns the messy part. Create a fresh state container for each call: stream identifier, AI session, audio buffers, tool abort controller and playback queue. Never put those in module globals. Two simultaneous calls would happily chew on each other's state.

Decode the audio format documented for your Cloudonix stream before sending it to the model. If the model requires another sample rate, resample after decoding and low-pass before downsampling. Keep filter and encoder state per call so chunk boundaries don't add clicks. Don't assume another platform's audio encoding or base64 JSON envelope applies to Cloudonix. Confirm the wire format first, then write a small codec adapter with tests using recorded audio.

Backpressure matters too. If the model produces replies faster than the call can play them, your queue grows and interruptions feel broken. Bound the queue, measure queued audio duration and discard audio only according to a deliberate playback policy. Forward silence when the stream supplies it, since many speech engines use audio energy and timing to detect the end of a turn.

Tool calls need the same care. Validate arguments against a schema, apply timeouts and make side effects idempotent. A model request is untrusted input with a nice voice. Return a compact result to the model and log the correlation ID you actually use, without putting API keys or sensitive call audio in application logs.

Log enough to reconstruct a failure without turning your logs into a second call recording system. Keep the call identifier available to your app, the model session identifier when available, stream reconnects, audio queue high-water mark, tool duration and terminal errors. Avoid raw audio and full transcripts by default. If your deployment has GDPR or HIPAA obligations, decide retention and access controls before testing with real customer data. Your Node service and AI provider have their own responsibilities.

Walk through an interruption and a booking tool

Say you're building an after-hours booking line. The caller asks for the first open appointment tomorrow. While the agent starts explaining the availability, the caller adds, “Actually, make it next week.” Your media service should stop creating outbound audio for the old turn and forward the new speech immediately. The model can then request your availability tool with revised dates.

Stopping your local queue only covers audio you haven't sent yet. Audio already buffered downstream needs behavior defined by the actual Cloudonix stream protocol. Verify whether your deployment supports playback control or completion feedback before promising precise “heard up to here” accounting. You can still build solid interruption handling without inventing mark semantics.

sequenceDiagram
    participant P as Caller
    participant C as Cloudonix session
    participant N as Node.js media service
    participant A as AI provider session
    participant T as Tool handler
    A-->>N: Start spoken response
    N-->>C: Send response audio
    P->>C: Speak over the response
    C->>N: Deliver new caller audio
    N->>N: Stop queued local playback
    N->>A: Forward the interruption audio
    A-->>N: Request a tool call
    N->>T: Validate arguments and run
    T-->>N: Return the tool result
    N->>A: Submit the result
    A-->>N: Generate a revised spoken response
    N-->>C: Send response audio

A caller interruption and a tool request share the same per call conversation state.

For the tool round, your server validates the date range, calls the scheduling system and returns availability to the same AI session. If the caller speaks again while the tool runs, cancel obsolete work where possible and let the provider's conversation semantics decide whether the result still applies. Keep a per call turn ID so a slow response can't book the wrong thing.

Test this scenario with real overlap, not polite turn-taking. Start a long reply, interrupt it halfway through and place a tool call immediately afterward. Watch three clocks: time to first audio, queued audio duration and tool latency.

Then test the ugly cases. Close the AI session while audio is queued, drop the WebSocket during a tool request and make the tool time out. The call should end or recover according to a policy you chose, rather than whatever exception happened to surface first.

Where this build fits and where OPBX is shorter

This architecture is a good fit when your agent needs custom tools, model choice or control over audio handling. Cloudonix gives you the phone session and CXML stream. Node gives you a familiar place for orchestration. The trade-off is that you own media protocol compliance, codec testing, scaling and tool safety. A speech-to-speech model doesn't remove that work.

You also need the full stream reference before writing the media handler. The CXML and API pages are enough to build call control, but they aren't enough to guess a media wire protocol. Treat protocol verification as a proof of concept task with test calls and recorded fixtures.

If your actual goal is an AI receptionist for a business PBX, writing a media server may be too much. OPBX, the MIT licensed open source PBX built on Cloudonix, can route calls to 17 AI voice providers over SIP or WebSocket. Its AI load balancers support round robin, priority and percentage strategies, with automatic failover and fallback to an extension, ring group or IVR.

OPBX also gives you extensions, queues, business hours and call records in a self hosted stack. You'll still configure an AI provider and design the conversation. You just won't start by implementing audio plumbing. For a highly custom agent, keep the Node design. For PBX routing and provider failover, OPBX is the shorter path.

Make the edges boring first

Start with one inbound number, one CXML endpoint and one read-only tool. Get a clean recording of the audio path, then test interruption and outbound calls through the API. Once those edges are boring, swap models or add tools without touching call control.

The Cloudonix DevZone has the CXML <Stream> reference and call control API details. If you'd rather inspect a complete business PBX, clone OPBX on GitHub and trace its AI routing before deciding how much of the stack you want to own.

Frequently asked questions

What role does CXML play in a Cloudonix speech-to-speech phone agent?

CXML is the call-control layer. When a call reaches your Cloudonix voice application, it fetches a CXML document from your Node.js service, and instructions like <Connect><Stream> tell Cloudonix to open a bidirectional audio WebSocket to your media service. Execution waits inside <Connect> until the stream terminates.

Do I need to rewrite my telephony code if I switch speech-to-speech models later?

No — that's the point of the architecture. Cloudonix CXML handles call control and the audio stream, while your AI provider defines its own session, audio and tool events in your Node.js service. Keeping these two contracts separate means the model can change without changing the CXML document.

How does Cloudonix pricing work for testing voice agents?

Cloudonix doesn't use the usual per-minute platform pricing model, which is helpful when testing long conversations. However, you still pay separately for your model, hosting and any carrier services, so it's wise to prototype with the same call lengths you expect in production.

Can I reuse WebSocket handlers like start, media, mark or clear from another telephony platform?

No — those event names come from other platforms' media protocols, and copying them would be guessing with production audio. Use the Cloudonix <Stream> reference and the full media protocol documentation when writing your WebSocket handler.

Does this architecture work for outbound calls too?

Yes. The same design works for outbound calls made through the Cloudonix API — Cloudonix still controls the call and opens the stream, while your Node.js service creates the AI session and runs the agent.

Where should my business logic and tool integrations live?

Keep them in your Node.js service alongside the AI session. That way tools like calendar lookups or order updates stay testable without owning a phone call, and they remain independent of the telephony layer.

What infrastructure do I need beyond the speech-to-speech model?

The model is only one dependency. You also need public call control (which Cloudonix provides), TLS for the media WebSocket, per-call state management and a safe way to run tools — which is why a phone agent should be treated as a stateful media service, not a simple request-response demo.