Stop making your personal AI agent transcribe first

A personal AI agent is a small program that does things for you: dims the lights, starts a timer, moves a meeting, sends a message. Talking to it is the natural way to use it, because your hands are usually busy. The hard part isn’t the actions. It’s deciding, every time you speak, which action you meant, whether you were talking to the agent at all, and how sure it needs to be before it acts.

This post walks through a design where Tazo, an audio-native decision model, makes that call straight from the audio. You write the list of actions, Tazo returns a probability for each, and your own thresholds decide whether the agent acts, confirms or ignores.

Why voice commands are harder than they look

The usual recipe is speech-to-text, then an LLM that reads the transcript and picks a tool. It works in a demo. In a living room or at a desk, a few things go wrong:

• Everything goes through the LLM. “Lights off” and “draft a reply to Priya” take the same slow path, so the simplest commands pay the full price.
• The agent can’t tell who you’re talking to. A transcript of “so I told him to send it tomorrow” looks like a command. Said to a colleague, it isn’t one.
• Hesitation disappears. “Uh, maybe move my three o’clock?” and “Move my three o’clock.” can transcribe to nearly the same words. One is a decision. The other is a question you’d like answered. We covered the same gap in what an audio decision model is.
• There’s no sense of how sure the agent is. An LLM that names a tool gives you the tool, not a number you can set a bar against. Acting on a guess is fine for the lights and a bad idea for a payment.

None of this is a reason to drop the LLM. It’s a reason to stop using it for the one job it’s worst suited to: a fast, repeated, multiple-choice decision about live audio.

The design: audio in, action out

Treat “what should I do?” as a pick from a known list, the same way a voice agent picks between respond, wait and escalate. Your actions are the options. Tazo hears the audio and returns calibrated probabilities over them, with no transcript in between.

There are three parts to build:

1. An action list. Plain names for what your agent can do, plus one for “this isn’t for you”.
2. A decision call. Send Tazo the audio and the list, get probabilities back.
3. A policy. A threshold per action, and what the agent does on each side of it.

The LLM stays in the design, but later and less often. It fills in details that can’t be a multiple-choice answer, like the text of a note or the body of an email.

Step 1: write the action list

Start with five to eight actions you’d use every day. Each one is a short, concrete name. Add two options that aren’t actions at all, so the agent has somewhere to put speech it shouldn’t act on:

• lights_off, set_timer, add_event, move_event: everyday actions that are cheap to undo.
• send_message, delete_or_purchase: actions with consequences, which you’ll treat more carefully.
• not_for_me: speech aimed at someone else, a TV, or nobody.
• none_of_these: aimed at the agent, but not something it can do.

not_for_me is what makes an always-listening agent bearable. Without it, every sentence in the room is forced into one of your actions. none_of_these is the polite way to say “I heard you, I can’t do that yet”, instead of doing the nearest wrong thing.

Because the options are part of the request, adding a capability later means adding a name to this list. It doesn’t mean retraining anything.

Step 2: ask Tazo what the moment needs

Each time you speak, send Tazo the audio along with your action list. It returns a probability for every option, and the highest one is the agent’s best guess at what you meant.

That’s the whole routing layer. It’s one call instead of a transcription step and an LLM round trip, which is why simple commands can run on a much shorter path. Tazo runs as a standalone API, so it fits next to whatever you already use for the microphone, speech-to-text and speaking back.

Step 3: turn probabilities into behavior

The probability is only useful if you decide in advance what to do with it. Give every action its own bar, set by what a mistake costs.

Then walk through the same few checks every time:

1. If the best guess is none_of_these, say the agent can’t do that yet.
2. If its probability is below that action’s bar, ask a short question instead of acting, like “Move your 3 pm meeting?”
3. If the action is not_for_me, stay quiet.
4. If the action is marked to confirm, read it back and wait for a yes.
5. Otherwise, run it.

The numbers above are examples. Pick yours by recording a few dozen real commands and a few dozen near misses, then check where the probabilities land. Because the probabilities are calibrated, a threshold of 0.9 means roughly what it says, and you can move it up or down on purpose.

Three habits make this feel good:

• Cheap and undoable acts early. If a wrong light switch costs you one more command, a low bar is fine.
• Anything visible to other people gets a read-back. Even at 0.9, “Send ‘running ten minutes late’ to Sam?” costs a second and saves an awkward message.
• Anything you can’t undo always confirms, however high the number is.

What it looks like in practice

Here is the same agent with three different utterances. Each one asks the same question and gets a different answer.

The third card is the one a transcript-first agent gets wrong. The words contain “send”, but the audio sounds like a conversation with another person, so not_for_me wins. The second card is the one most agents run straight through: the same words with a rising, uncertain delivery should lead to a question, not an action.

The values in the diagram are illustrative. Your own thresholds should come from your own recordings.

Step 4: fill in the details

Choosing the action is only half of a command. “Add lunch with Sam on Friday” also needs a title and a date. There are two ways to get them, depending on the shape of the answer:

• When the answer is one of a known set, ask Tazo again. Which lamp: kitchen, bedroom or desk? Which meeting: the three that are on your calendar this afternoon? Read the options from your own data and ask which one you said. This keeps the whole path audio-first, and the same threshold logic applies.
• When the answer is free-form, use speech-to-text and an LLM. The text of a note, the body of a message, a calendar title. Run it after the action is chosen, so the LLM only works on commands that are worth working on.

That split is the main reason the agent feels fast. Most of what you say to a personal agent is short and falls into the first group. The LLM is saved for the minority that really need words.

Handling the awkward moments

A few behaviors are worth designing for from the start:

• Corrections. “No, the other one” is a decision about the last action. Give the agent an undo_last option and a correct_last option, and keep the last result around for a few seconds.
• Interruptions. If the agent is speaking and you start talking, that’s the same kind of call a voice agent makes about backchannels and barge-in. Add stop_talking as an option and let it win when it should.
• Low confidence on everything. When no option clears its bar, ask a short question instead of guessing. “Did you want the lights or the timer?” is cheap and keeps trust.
• Starting early. For read-only actions such as checking the weather or your calendar, you can begin the lookup while you’re still speaking. That’s speculative tool calling, and it fits naturally once the audio is already being scored.

Where to start

You don’t need the whole agent to find out whether this works. Build the smallest useful version first:

1. Pick three actions you’d use daily, plus not_for_me.
2. Record twenty real commands and twenty things you’d say near the agent that aren’t commands.
3. Run them through the routing call and look at where the probabilities fall.
4. Set a threshold per action from that, and make everything risky confirm.
5. Add actions one at a time, and re-check the thresholds when you do.

Your LLM can keep deciding what to write. Tazo decides what you meant.

A personal agent doesn’t need to understand everything you say. It needs to know when you’re talking to it, what you want, and how sure it is.