Turning doctor-patient conversations into structured, EHR-ready records with a custom ambient AI scribe
Healthcare
EHR Integration
AI consulting
BACKGROUND
About the client
The client is a digital health company building AI clinical documentation for multi-specialty care, across general practice, psychiatry, physiotherapy, and long-term care. Their clinicians, like clinicians everywhere, were losing hours each day to writing up visits, and the company's promise was to give that time back: have the conversation, and let the record write itself.
The ambition is simple to state and hard to build. Recording a visit is easy. Turning a real, messy doctor-patient conversation into a structured, accurate, EHR-ready record that a clinician will trust and sign is not. Off-the-shelf transcription gave them a wall of text, not a usable clinical note, and it did not handle protected health information to the standard healthcare demands.
They came to us to build the pipeline that closes that gap: from the spoken conversation to a structured record in the EHR, reliable across real clinical conditions, accurate where a mistake would matter, and compliant by design.
The ambition is simple to state and hard to build. Recording a visit is easy. Turning a real, messy doctor-patient conversation into a structured, accurate, EHR-ready record that a clinician will trust and sign is not. Off-the-shelf transcription gave them a wall of text, not a usable clinical note, and it did not handle protected health information to the standard healthcare demands.
They came to us to build the pipeline that closes that gap: from the spoken conversation to a structured record in the EHR, reliable across real clinical conditions, accurate where a mistake would matter, and compliant by design.
Domain
Ambient clinical documentation
Compliance
HIPAA
Platform
iOS, Android, Web
Speech to text
Deepgram, Whisper / OpenAI, Mistral
AI & infrastructure
OpenAI, Azure OpenAI, Mistral, MayaAI, AWS S3, IpInfo, Stripe, Sentry
The challenges
1
Speech to text that survives a real exam room
A single transcription engine was not enough. One provider hallucinated on noisy or poor-quality audio, another timed out under load at peak hours, and accents and multi-speaker exchanges broke naive setups. The scribe had to keep working in real clinical conditions, not just in a quiet demo.
2
Accuracy where a mistake reaches the care plan
Transcription was only step one. When an LLM drafts a summary or a care plan, it can weight what the patient said the same as what the clinician instructed, and a single wrong dosage or instruction is a clinical risk, not a typo. This was the step that separated a usable tool from a liability.
3
Protected health information, handled to standard
Every second of audio and every line of transcript is PHI. It had to be obfuscated where appropriate, stored securely, access-controlled by geography and role, and observable when something went wrong, without slowing the clinician down. Compliance could not be a layer added at the end.
4
From transcript to finished, EHR-ready record
A wall of text saves no one time. The output had to be a structured record shaped for the specialty, general practice, psychiatry, physiotherapy, long-term care, with the right fields extracted and written into the EHR so the manual data entry disappeared. That last mile held most of the value, and most of the work.
What we built
Multi-engine speech-to-text layer
We ran speech to text across several engines (Deepgram, Whisper / OpenAI, and Mistral) behind one interface, so the active model could be switched or fallen back to without touching the app.
Each engine was tuned for clinical audio: parameters set against real recordings rather than defaults, with handling for noise, multi-speaker exchanges, and long sessions.
MayaAI handled on-the-fly obfuscation so sensitive terms were protected at the transcription boundary.
Each engine was tuned for clinical audio: parameters set against real recordings rather than defaults, with handling for noise, multi-speaker exchanges, and long sessions.
MayaAI handled on-the-fly obfuscation so sensitive terms were protected at the transcription boundary.
Generation and verification, on two different models
Accuracy came from cross-checking, not from one bigger prompt. One model generated the structured record or care plan; a second, deliberately different model verified it against the same source and knowledge base.
Using two distinct models (for example OpenAI and Azure OpenAI, or Mistral) caught the errors a single model is blind to, the wrong dosage, the patient preference mistaken for a clinical instruction.
Specialty templates shaped the output for general practice, psychiatry, physiotherapy, and long-term care.
Using two distinct models (for example OpenAI and Azure OpenAI, or Mistral) caught the errors a single model is blind to, the wrong dosage, the patient preference mistaken for a clinical instruction.
Specialty templates shaped the output for general practice, psychiatry, physiotherapy, and long-term care.
EHR write-back, automation, and compliance by design
The verified, structured output was written back into the record, so the clinician reviewed a finished note instead of retyping one, and the manual follow-up work was automated away.
Audio and transcripts lived in encrypted AWS S3, access was controlled by geography and role (IpInfo), errors were traced in real time (Sentry), and plan and access management ran through Stripe.
PHI handling, storage, and observability were built into the pipeline from the first commit, not bolted on for an audit.
Audio and transcripts lived in encrypted AWS S3, access was controlled by geography and role (IpInfo), errors were traced in real time (Sentry), and plan and access management ran through Stripe.
PHI handling, storage, and observability were built into the pipeline from the first commit, not bolted on for an audit.
Achieved results
The results
Transcription that holds up in real conditions
Multi-engine fallback meant a noisy room, an accent, a long session, or a provider outage no longer stopped the scribe. The system stayed usable where single-engine setups quietly fail.
Records a clinician will actually sign
Generation checked by an independent verification model produced structured, specialty-specific records and care plans that cut manual entry and lowered the risk of a clinically meaningful error reaching the EHR.
Compliant and observable by design
PHI was obfuscated, encrypted, geo and role access-controlled, and fully traceable, so the scribe met a HIPAA standard without adding friction for the clinician or a scramble before an audit.
Book a live demo to see it in action
TECHNOLOGIES
Tech stack
RELEVANT CASES
Check our more healthcare stories
Contact Us
Please provide your info and inquiry details below.
