Ada on the Godot kiosk in an end-to-end test against the real server. Nadia is one of the test guests.

AI and voice

Getting 150 builders from the lobby to Room 209 without a queue

0 wrong guests admitted in 28 noisy scenes

A voice concierge on a lobby screen for a meetup night. She gives directions, checks guests in by QR pass or by name, and suggests people to meet. Check-in is plain Python rules; the model only talks and picks tools.

AI Tinkerers Doha meets in the Tech 2 building at Qatar Science and Technology Park. For Round 4, on 19 October 2026, the organisers approved about 150 builders for Room 209 on floor 2. Most had never been in the building, and most would arrive in the same half hour, while one or two volunteers ran the desk.

I built Ada for that half hour and gave the talk about her at the event. She's a voice concierge on a lobby screen: she gives directions, checks guests in by QR pass or by name, and names people worth meeting. The model talks and picks tools, and plain Python rules decide who gets in. In 28 scenes spoken over crowd noise, nobody was admitted who shouldn't have been.

28
end-to-end scenes, crowd noise mixed into the audio
0
wrong guests admitted
0 to 1
cloud calls per check-in

Watching one guest's evening

A talk about a kiosk is easy to fake with slides. A slide can't show what the kiosk heard in a noisy lobby, or what it got wrong.

I opened the deck on a recording instead: one guest, start to finish, on the real Godot kiosk and server with lobby chatter in the microphone. The captions show what Whisper heard, slips included. Pick a step to jump to it.

Starting from the research

A meetup lobby has problems a language model can't see: a rush at the door, strangers who won't say their name out loud, and a room too loud to hear in. I had no time for interviews before the event, so the design started from published research and the event's own data.

I read check-in benchmarks, wayfinding studies, work on talking to strangers, and Whisper's own noise results. Each finding turned into one rule for Ada.

  1. Most guests arrive in the first half hour.

    Check-in vendors plan for 60 to 70% of attendees arriving within 30 minutes of doors opening, and aim for under 60 seconds per person. Self-service kiosks see 15 to 30% of first-time users needing help.

    So: Ada has to clear the 6:00 to 6:30 rush on her own and send only the exceptions to the desk.

    Nunify, Event check-in time benchmarks (vendor data, 2026) ↗

  2. People get lost inside buildings too.

    Wayfinding research puts the cause in three places: the building's layout, the signs, and the visitor's own picture of the space. First-time visitors have no picture yet.

    So: Directions name a landmark and a count ("the lifts to the right of reception", "the third door"), and a second kiosk on floor 2 picks the visitor up.

    Carlson, Hölscher, Shipley & Dalton, Getting Lost in Buildings, Current Directions in Psychological Science (2010) ↗

  3. People hold back from talking to strangers, and misjudge how it goes.

    After a first conversation, people consistently underestimate how much the other person liked them. Separately, professional networking for instrumental reasons makes many people feel uncomfortable.

    So: Ada can make the first move for a guest: name someone who works on their topic and say where to find them.

    Boothby, Cooney, Sandstrom & Clark, The Liking Gap in Conversations, Psychological Science (2018); Casciaro, Gino & Kouchaki, Administrative Science Quarterly (2014) ↗

  4. Speaking to a device in public feels exposed.

    In studies of voice assistants, people were less willing to say private information out loud in public places than at home.

    So: A guest never has to say their name: holding up the QR pass is the first option, and Ada never reads names or the guest list aloud.

    Easwara Moorthy & Vu, Privacy Concerns for Use of Voice Activated Personal Assistant in the Public Space, IJHCI (2014) ↗

  5. Speech recognition gets worse fast as a room gets louder.

    Whisper's own paper shows every model it tested degrading as background noise rises, with white noise and with pub noise. People also raise their voice and change how they speak in noise (the Lombard effect).

    So: Push-to-talk marks exactly when the guest is speaking, the reply is captioned on screen, and anything with a right answer (check-in) goes through rules, not recognised words.

    Radford et al., Robust Speech Recognition via Large-Scale Weak Supervision (2022); Zollinger & Brumm, The Lombard effect, Current Biology (2011) ↗

  6. Doha is an English-speaking room with many first languages.

    About 88% of Qatar's residents are foreign nationals. Arabic is the official language and English is the language of business.

    So: Ada speaks English by default and handles accented speech and names from many languages. A name has to match one guest clearly, so a mishearing goes to the desk instead of admitting the wrong person.

    Demographics of Qatar, Wikipedia (read 19 Oct 2026) ↗

Framing the problem for four people

One kiosk serves four people with different stakes. A first-time guest wants the room and no queue. A presenter wants to set up early and be found afterwards. The volunteer at the desk wants only the hard cases. The organisers want an accurate headcount and a guest list nobody ever reads out.

I turned those stakes into three questions to design for:

Primary

Nadia Rahman

First-time attendee
“I just want to find the room and not stand in a line.”

Goals

  • Find Room 209 fast
  • Check in without fuss
  • Meet someone working on Arabic speech

Pains

  • Unfamiliar building, signs she can't see in the crowd
  • Doesn't want to say her name to a screen in a busy lobby
  • Walking up to strangers

How they use Ada

  • Directions
  • QR check-in
  • Find people (she opted in)
Primary

A presenter

One of the night's three live demos
“Tell me where to set up, and point people my way.”

Goals

  • Get to the front table before 6:40
  • Be easy to find afterwards for people who share the topic

Pains

  • Stuck in the check-in rush before setup
  • Answering where-is-the-room on the way in

How they use Ada

  • Presenter line on check-in: "Presenters sit at the front table"
  • Named in find-people for their demo's topics
Stakeholder

Joseph Mensah

Volunteer at the event desk
“If the kiosk handles the easy ones, I can deal with the real problems.”

Goals

  • Keep the line short during the rush
  • Handle only the exceptions

Pains

  • Looking up names one by one
  • Passes used twice
  • Being the directions desk too

How they use Ada

  • Ada sends waitlist, reused-pass and not-found cases to him
Stakeholder

The organisers

AI Tinkerers Doha team
“Nobody reads out our guest list. Ever.”

Goals

  • Accurate headcount
  • No guest data spoken or sent anywhere it shouldn't go

Pains

  • Walk-ins they can't admit
  • A model that might improvise a yes

How they use Ada

  • Rules decide check-in
  • QR decoded on the machine
  • Opt-in networking only

Edge personas

  • Priya NairBrings a colleague: a party of two on one pass.plus_ones 1
  • Daniel OrtizOn the waitlist. Needs a kind, clear no and where to go next.waitlist
  • Sara KimCancelled her RSVP, then came anyway.cancelled
  • Nadia KarimShares a first name with Nadia Rahman, and opted out of networking.opt-out

I drew the personas from the research and the event's data, with no interviews yet, so they are proto-personas and every guest is invented. The four edge cases at the bottom shaped the check-in rules more than the main four did.

Mapping Nadia's evening

Nadia Rahman's evening · feeling from −2 (stressed) to +2 (delighted)

+2+10−1−2

Check in

6:06 PM · feeling +2
Doing
Presses C, then Q, and holds her phone up.
Thinking
Please don't make me spell my name.
What Ada does
"You're on the list, Nadia." A pass card appears with the room and gate time.
Pain
Glare or a tilted phone; a pass that's already been used.
Opportunity
Decide in rules in under a second; send real problems to the desk.

Nadia is the primary persona: a speech researcher new to Doha, with her QR pass on her phone. Her evening runs from two days before the event to leaving through the gates at 9:10, and it dips twice: arriving in a loud lobby, and walking up to strangers.

I drew the map before building anything. Each dip became a feature. The room number goes on screen because the lobby is loud, and Ada names people who opted in because Nadia won't walk up to random ones.

Storyboarding four moments, then breaking them

The storyboard has four panels: the lobby, the pass scan, the second kiosk on floor 2, and networking. Each one looked simple on paper and stopped being simple once I listed what could go wrong in it.

I gave every panel its edge cases, and each edge case became a test scene. The stills come from those test runs, so they show the lobby kiosk: the floor 2 kiosk is still only a panel in the storyboard.

Keeping admission out of the model

A language model that can take actions will sometimes take the wrong one. OWASP lists this as Excessive Agency ↗. At a door, the wrong action means admitting someone who isn't on the list, or telling a stranger who else is here.

A first name alone gets the same reply whether one guest or two share it, so the answer gives nothing away.

I split the job in two. The model understands the guest and picks a tool. Rules in server/meetup.py decide every check-in and write the sentence Ada says, so the model never sees a full name or a pass code.

What the model does

  • Understands what the guest asked
  • Picks one of 14 tools
  • Writes the spoken reply

What plain Python rules do

  • Check the event time
  • Check RSVP status
  • Check earlier check-ins
  • Read the QR pass with OpenCV, no model call

a pass scan makes no cloud call at all; a check-in by name makes at most one

The rules can return 16 outcomes, and every one ends somewhere a guest can act on: admitted, asked for more, sent to the desk, refused, or asked to try again. Filter them by what happens next.

  • checked_inAdmitted"You're on the list, Nadia." Pass card, directions, what's on now.
  • welcome_backAdmittedSeen at the other kiosk: "Hi again, Nadia."
  • already_checked_inAdmittedWithin 2 minutes: "You're all set." Later: the original time, and a pointer to the desk.
  • need_surnameAskFirst name only. Same reply whether one or two guests share it.
  • need_nameAsk"What name is the RSVP under?"
  • name_not_foundDeskNo clear match. Ada never guesses between guests or lists names.
  • unknown_codeDeskA well-formed pass that isn't on the list.
  • wrong_eventRefuseA pass for another night.
  • not_a_passRefuseAny other QR code, like a flyer link.
  • waitlistedDeskKind no, and what to do next.
  • rsvp_cancelledDeskThe guest cancelled; the desk decides.
  • closedRefuseAfter gates close at 10 PM.
  • not_todayRefuseThe pass is real but the event is another day.
  • event_cancelledRefuseThe whole night is off.
  • no_codeRetryNothing readable: hold it closer, screen bright, press Q again.
  • camera_unavailableRetryOffers the spoken route instead.

Names are the tricky part, because Whisper spells them however it likes. The server compares how names sound, so “Kabib Bi Han” can match “Cabibihan”, but only when one guest is the clear winner: at least 0.72 alike, and 0.15 ahead of the next closest guest. A first name alone gets the same reply whether one guest or two share it, so the answer gives nothing away.

Tracing each flow to the function

Two programs share one Mac. Godot draws the lobby and owns Ada's body. A Python server built on Pipecat listens, thinks and speaks. They talk over one local WebSocket, and only the text of each turn leaves the machine.

The stack is chosen so nothing personal leaves the building.

I named every box after the real function, so the talk could follow one question from the microphone to Ada's footsteps. Pick a flow and step through it.

"Where's the AI meetup?"

The guest just talks; Silero VAD and Smart Turn decide when they have finished. Whisper and Kokoro run on the Mac; only the text of the turn goes to the model.

Godot kioskWebSocketServer on the MacKeysmain._unhandled_inputVoice clientMuseumVoiceClient: mic, socket, speakerMessage routermain._on_voice_messageHUDcaptions, status, pass cardAda's bodyAvatarGuide: walk, face, lipsFrame loopSceneTree, 60 fpsWebSocket :8766JSON text + AUD0 audiotransport.inputMuseumSerializer.deserializeMuseumBridge_handle_message, emit, scan_passRulesWhisper (MLX)large-v3-turbo, on the MacUserTranscriptEmittercaptions; drops stray tapsLocalRAGInjectorLobbyData.retrieval_contextContext aggregatorLLMContext, one turn at a timeOpenAILLMServiceOpenRouter: gemma-4-31b-it, cloudCloudExpressionCueParsertakes off [pleased]AssistantTranscriptEmittercaption of the replyKokoro TTSKokoroTTSService.run_tts, 24 kHzInteractionStateEmitterstates and latency metricstransport.outputMuseumSerializer.serializetool_handler14 tools, every argument checkedRulesCheck-in rulesMeetupNight: check_in_*, find_peopleRulesCheckInStoreJSON file, fcntl lockRulesQRScannerOpenCV, on the Mac

This stepOn this flowNot usedCloudRulesPlain code the model can’t override

01 / 19Keys

main._unhandled_input

Space pressed: voice.set_push_to_talk_active(true), guide.set_state("listening")

The stack is chosen so nothing personal leaves the building. Whisper and Kokoro run on the Mac's GPU, OpenCV reads the pass locally, and the model sees only the words of the turn and the guide entries.

Kiosk

  • Godot4.7
    Local

    Draws the lobby, Ada and the HUD; runs the frame loop, navigation and input

    A real 3D scene with a navigation mesh, so Ada can walk a guest somewhere instead of pointing at a map

  • GDScript
    Local

    Kiosk logic: message router, voice client, avatar, HUD

    No build step; the scripts reload with the scene

  • GL Compatibility renderer
    Local

    OpenGL 4.1 through Metal on Apple silicon

    Light enough to share the laptop GPU with Whisper and Kokoro (51 fps in rehearsal)

  • VRM + MToon addonsVRM 1.0
    Local

    Lets Ada wear VRM avatars beside the Ready Player Me body

    Swap the body (M key) without touching the face or lip-sync code

Transport

  • WebSocketwebsockets 15.0.1
    Local

    One socket on 127.0.0.1:8766: JSON messages plus AUD0 binary audio

    Godot speaks WebSocket natively; one ordered channel for both audio and events

Voice pipeline

  • Pipecat1.1.0
    Local

    Frame pipeline: speech in, context, model, speech out, interruptions

    Barge-in and turn handling come built in; each step is a small FrameProcessor

  • Python3.13.5
    Local

    The server, the rules and the tests

    Pipecat, MLX and OpenCV are all Python

Speech in

  • Whisper large-v3-turbo, 4-bitmlx-whisper 0.4.3, MLX 0.32.0
    Local

    Transcribes the turn when Space is released, English, temperature 0

    Runs on the Mac's GPU, so no guest audio leaves the building

Model

  • gemma-4-31b-itvia OpenRouter, Cerebras first
    Cloud

    Answers and calls tools; max 140 tokens, temperature 0.2, no reasoning

    Fast first token on Cerebras; only the text of the turn and the guide entries are sent

Speech out

  • Kokoro v1.0kokoro-onnx 0.6.1
    Local

    Ada's voice, 24 kHz mono, made sentence by sentence and sent as it's ready

    Small enough to run beside Whisper; sounds natural at 1.05× speed

  • MMS TTS
    Local

    Filipino and English voices, used when MUSEUM_TTS isn't kokoro or Kokoro's files are missing; Whisper then detects the language

    Kokoro speaks English only; this keeps a Filipino path and a fallback

Rules and data

  • Check-in rules
    Local

    Every check-in outcome, find_people, event phases

    Deterministic and unit-tested; the model can't admit anyone

  • CheckInStore
    Local

    Who has checked in, as a JSON file with an fcntl lock and atomic writes

    Safe if two kiosk processes share it; nothing to install

  • Visitor guide
    Local

    Facts retrieved into each turn

    Answers come from the guide, with the source shown on screen

  • OpenCV QRCodeDetectoropencv-python-headless 5.0.0.93
    Local

    Reads the pass off the webcam frame, trying several views of each frame

    Pass check-in needs no model call, and no image leaves the Mac

This site and testing

  • React + ViteReact 18.3.1, Vite 5.4.11
    Build tool

    This presentation

    One page for the story, the architecture and the live view

  • Node22.22.3
    Build tool

    Tails the trace files and streams them by Server-Sent Events; serves function source

    Standard library only, no packages

  • JSONL trace
    Build tool

    Both processes append one line per event

    A file both sides can write, and any tool can read

  • pytest + e2e runnerpytest 9.1.1
    Build tool

    Unit tests, plus 28 scenes spoken through the real stack over crowd noise

    Tests the whole loop as a guest would use it, not only the parts

Inside Godot, one frame runs in a fixed order, with audio on its own thread beside it. That order is why a guest can cut Ada off and the walk to the lifts keeps going.

  1. 01

    Input

    main._unhandled_inputMuseumPlayer._unhandled_input

    Space, C, Q, mouse look. Each key becomes a message or a state change.

  2. 02

    Physics (60 Hz)

    MuseumPlayer._physics_processAvatarGuide._physics_process

    Visitor movement and the gaze ray; Ada's navigation step, gaze turn, and AvatarFace.update for blinks, mouth and expression.

  3. 03

    Process (every frame)

    main._processMuseumVoiceClient._process

    Locator arrow and the trace heartbeat. The socket: poll(), _read_packets() into message_received and _receive_audio, then _capture_microphone() while Space is held.

  4. 04

    Signals

    main._on_voice_messageAvatarGuide.feed_audio

    Messages route to the HUD and Ada inside the same frame, through Godot signals.

  5. 05

    Render

    RenderingServer

    The lobby, Ada and the HUD, through the GL compatibility renderer.

  6. 06

    Audio thread

    AudioStreamGeneratorAudioEffectCaptureAudioEffectSpectrumAnalyzer

    Runs beside the frame loop: plays Ada's queued speech, fills the mic capture buffer, measures the 7 speech bands for the lips.

Testing 28 scenes in a loud room

Unit tests can't tell you whether a guest gets through the door. So every scene from the storyboard was spoken through the whole stack, Godot to server and back, over synthetic lobby chatter from ten macOS voices.

A mishearing makes Ada ask again. It can't admit anyone, because admission never trusts the transcript.

When a loud room got in the way, it was the hearing: Whisper turned “AI Tinkerers” into “AIVP” in a busy lobby, a guest cut in as a check-in landed, and the loudest room drowned the question. Ada asked again each time, and none of it let in the wrong guest.

Two of the scenes are on video at the top of this section. Scene 27 is the loudest room; scene 23 is a guest cutting Ada off mid-answer.

Pick a scene to see what the guest said, what Whisper heard, what Ada said back, and how loud the room was.

28 scenes marks a failed scene

Wayfinding 4

Pass check-in 11

Spoken check-in 5

Event info and people 5

Turn-taking in noise 3

Scene 2 · Wayfinding · failed

Directions in a busy lobby

Noise 5 dB SNR
Guest said
“Hi! Where's the AI meetup?”
Whisper heard
“Hi, where's the AIVP?” · 40% words wrong
Ada said
“I do not have a detail for the AIVP in my guide. Please ask at reception.”
Time to first audio
7.3 s
Why it failed
missing ['209']

Holding to six principles

A kiosk at a door makes a small call for every guest, and a guest notices each one that goes wrong: a name read out to the room, a wrong admission, a reply lost in the noise, a refusal with nowhere to go next.

I held every decision above to six principles: one for each of those four, one for knowing when a guest has finished talking, and one for keeping personal data on the Mac. Each principle has a measure the test run reports.

PrincipleWhat it means in the build
Rules decide, the model talks.Check-in outcomes come from server/meetup.py. The model only chooses when to call it, and the rules wrote the line Ada says.
Just talk, or hold to talk.Ada listens hands-free and works out when the guest has finished. In a room too loud for that, push-to-talk lets Space mark the turn, and taps under 0.4 s are dropped. The 28 test scenes used push-to-talk.
Nothing personal leaves the Mac.Speech recognition, QR reading and the guest list stay on the machine. The model only sees an outcome, never the full name or code.
Opt in, or stay invisible.Only presenters and guests who opted in can be named in networking answers.
Always a person to go to.Every refusal ends with the event desk outside Room 209.
Say it and show it.Every reply is captioned, and the pass card stays on screen, for loud rooms and for guests who are hard of hearing.
MeasureResultWhere it comes from
Check-in, pass shown to decisionunder 1 spass_read metric, measured on every scan
Start of the turn to Ada's first wordmedian 5.9 s (1.4 to 16 s)first_audio_s over the scenes where Ada spoke. It includes the guest speaking; pass check-ins, with no speech and no model, are 1.4 to 3 s
Scenes run in a loud room28tools/record_e2e.sh, 19 Oct 2026
Wrong admissions0every admit path is a rule, tested in server/tests/test_meetup.py
Names or list read out0organiser and "is X here" scenes pass
Time to first word includes the guest speaking. Pass check-ins, with no speech and no model, take 1.4 to 3 seconds.

Planning the next round

The demo runs one kiosk on one Mac with invented guests. Before it meets a real door, it needs the second floor, a better pass reader, and people who aren't me.

I split what's left into two lists. The build list comes from the rehearsal logs, where OpenCV missed passes on bright phone screens and stray words from the room started replies. The research list checks my proto-personas against real guests, starting with a count at the desk from 6:00 to 6:30 at the next event.

Build next

  • The floor 2 kiosk and the lift ride: designed in the storyboard, not built
  • A stronger QR decoder, tested with real AI Tinkerers passes on bright phone screens
  • Voice barge-in tuned against a real room's echo
  • Drop stray words like "uh" or "okay" before they start a reply

Research next

  • Watch the desk at the next event from 6:00 to 6:30: count arrivals, time each check-in, note every question that isn't check-in.
  • Five short interviews after the demos: what was hard about getting here and meeting people?
  • Run the kiosk beside the desk for a week, opt-in, and compare queue length with and without it.
  • Test with Arabic-accented and South Asian-accented English speakers, and with a guest who is hard of hearing.

Notes

Sources

  1. LLM06: Excessive Agency ↗, OWASP Top 10 for LLM Applications, 2025.
  2. Pipecat ↗, the open-source voice pipeline the server is built on.
  3. Radford et al., Robust Speech Recognition via Large-Scale Weak Supervision ↗ (Whisper), 2022.
  4. The research findings above link their own sources.

More projects