Baatein AI

Problem

Talking to an AI today is like meeting a stranger every time. It forgets your daughter's name and the friend you meant to call. It wants pure English or pure Hindi, while India speaks a mix. And what it does remember, you can't see, correct or delete.

The people who most need someone to talk to, like an older parent living alone, are the ones these tools serve worst.

Approach

Baatein is a voice friend you call from the browser. Three ideas shape it:

• It speaks your mix. "Aaj ghutne mein thoda dard tha" ("My knee hurt a bit today") works as naturally as English. Older users get a slower, respectful style ("Kamala ji").
• It remembers only with your yes. Before keeping anything it asks "Kya main yeh yaad rakhun?" ("Shall I remember this?"). Transcripts are an internal log it can never search.
• It follows up. "Shanti se baat hui?" ("Did you talk to Shanti?") It tracks the stories you're living through and brings up dates when they're due.

You stay in control just by saying it: "Mira nahi, Myra hai" corrects a memory, "Yeh bhool jao" forgets one, "Aaj kuch save mat karna" saves nothing today, and "Job ki baat mat karna" avoids a topic.

The working rule through the whole build was to fix a problem at the layer that causes it (data, model, code structure) and measure the options, rather than adding prompt instructions to cover it up.

Architecture

1. Before the call, the agent reads its notes: your name, the last call, plans you mentioned, anything due soon. It greets you first, and the code picks the greeting, so a second call on the same day doesn't ask again how your day is.
2. During the call, speech goes through Sarvam to Claude and back through Sarvam's voice. Each turn searches memory in the background without holding the reply. A turn made only of filler words isn't searched at all.
3. After the call, one structured Claude call writes a short summary and follow-ups, naming only what you agreed to keep.
4. The memory layer connects the three. Each call starts from it and adds to it, only with your yes.

The whole thing deploys as one Docker image on Railway, running the API and the agent side by side with SQLite on a volume, and LiveKit Cloud carries the audio.

Evaluation

What was tested	Result
Real conversations. 39 scenarios from a 45-situation library: grief, corrections, "don't save this", topics to avoid, an older person repeating a story, multi-day call sequences	All pass, twice in a row, against the real model
Finding the right memory. 31 questions in each language, plus small talk that should find nothing	Right memory first: 29/31 English, 28/31 Hindi, 24/31 Hinglish
Voices. 9 voices on the same streamed reply, listening for tone changes mid-sentence	Picked the steadiest
Code. Unit, contract, integration and API tests in CI on every change	100% branch coverage, strict type checks

Retrieval benchmark

I measured the embedding model rather than patching prompts. Memories are stored as short English sentences while speech arrives in another language and script, so retrieval has to work across them. The corpus is one persona's 34 memories, asked 31 questions in each form, plus small talk that should find nothing.

Right memory first / in the top 5, out of 31:

Model	Hindi	Hinglish	Keywords	Query time
MiniLM-L12 (the old default)	25 / 28	9 / 13	28 / 30	4 ms
multilingual-e5-large, local	26 / 30	16 / 25	28 / 30	34 ms
embeddinggemma-300m, local	29 / 31	20 / 27	29 / 31	16 ms
llama-text-embed-v2, 2048 dims, hosted	28 / 31	24 / 31	29 / 31	~450 ms

For the chosen model, AUC against small talk is 0.98 for English, 0.89 for Hindi and 0.80 for Hinglish. Each language gets its own measured similarity cut-off, because one shared value dropped most of the right answers.

Should a turn be searched at all?

An evaluation on 252 labelled sentences compared ways to skip pointless searches:

Option	Searches skipped	Useful hits lost	Time
Filler-word list (chosen)	30% of live turns	0 of 134	~0
Character n-gram classifier	16%	2	0.1 ms
Local MiniLM classifier	13–18%	0–4	7 ms, +800 MB per worker
Ask Claude	Most small talk	0	1.0 s typical, 3.4 s worst

On the production path, small talk returning memories fell from 19 to 9 with no useful hit lost, and live turns sent to Pinecone fell from 80 to 55.

Caveats

• The benchmark dataset is small and comes from one persona, so timings are relative.
• Odia and Kannada are in the code but hidden from signup, because retrieval is weaker there (15/24 and 17/25 first), and their test sentences still need review by native speakers.

Tradeoffs

Decision	Benefit	Cost
Own memory store on SQLite, not Mem0	Covers storylines, reminders and summaries; no extra infrastructure	I own the ranking and the migrations
Hosted embeddings	Best for the three languages in focus; no 1.7 GB model in each worker	About 450 ms a search, so results can arrive a turn late
Replies don't wait for the search	No added silence on ordinary turns	A memory can land one turn late, which is why Claude also has a recall tool
Strict consent	Trust, and nothing kept without a yes	It sometimes has to ask
Deletion by tombstone, confirmed on the next message	Provable, and one misheard sentence can't erase a memory	One extra step
"Forget me" only in the app, with the password	A misheard sentence or a borrowed phone can't wipe everything	Not available by voice
Reminders only inside a call	It never promises something it can't keep	No alerts until there's a phone or push channel

Outcome

All six build phases are done, and it's live at baatein-ai.experiship.com with invite-only test accounts.

Next is a mobile app for reminders with real alerts, then more languages, personas people can choose, and a daily check-in call for older users.

What I learned

Most "the model got it wrong" moments weren't model problems. A reminder it never brought up was a fact missing from context. A check-in asked twice was state nobody tracked. Fixing those in code held up far better than another line of prompt, and measuring the options kept me from building things that sounded good: asking Claude to decide when to search would have been slower than the search it saved.