Epistemic Noise
All notes

Note · 2 min

Speaker embeddings

Speaker embeddings make voice matching look almost trivial.

Take a speech segment, map it to a fixed-dimensional vector, compare two vectors with cosine similarity and decide whether the recordings probably contain the same speaker.

The difficult part is everything before that comparison.

An embedding model will still return a perfectly valid vector for a clipped recording, a two-second fragment, mostly silence, heavy channel noise or a diarization segment containing two people. The dimensionality is correct. The representation may still be useless.

So speaker matching begins with segmentation and quality control. I want enough usable speech, limited silence and clipping, confidence that the segment belongs to one speaker, and preferably multiple usable segments when the recording allows it. Aggregating good evidence is usually safer than pretending every crop deserves equal trust.

The similarity threshold is also not universal. A threshold taken from a benchmark may behave very differently under another language, microphone, codec, call channel or noise distribution. It has to be calibrated on in-domain same-speaker and different-speaker examples, with the operating point chosen according to the cost of false accepts and false rejects.

And in high-consequence applications, similarity should usually produce candidates rather than verdicts.

A voice embedding is evidence about identity. It is not identity itself.