Talkie's dictation runs on NVIDIA's Parakeet. We converted it to Core ML ourselves, starting from NVIDIA's own weights, and got it running on the Mac's Neural Engine at 477 MB.
When you hold the dictation key in Talkie and say a sentence, the words land at your cursor about half a second after you let go. In that half second a neural network called Parakeet reads the sound and writes it out. NVIDIA trained it and released it under an open license. It has 600 million parameters and was built to run on NVIDIA's GPUs, inside NVIDIA's NeMo toolkit, in Python.
A Mac runs none of that. To get Parakeet onto one, the model has to be translated into Apple's Core ML format, made small enough to download, and arranged so that the Neural Engine, the part of Apple Silicon built for this kind of math, does the heavy lifting.
We wanted to do that translation ourselves, from NVIDIA's published weights. If dictation gets something wrong, we want to be able to trace it to a step we wrote. If we want a smaller or faster build, we want to be able to make one. Here is how the conversion went, what we chose along the way, and what came out the other end.
Parakeet works in stages. A preprocessor turns the raw audio into a mel spectrogram, a picture of which frequencies are loud at each moment. An encoder, which holds nearly all of the model's weight, reads that picture and writes a summary of each 80-millisecond slice. A small decoder remembers the words written so far, and a joint network combines that memory with one slice at a time to choose the next piece of a word.
The "TDT" in the model's name stands for Token-and-Duration Transducer. Along with each token, the joint predicts how many slices to skip before it looks again. Silence and long vowels get skipped, so the decoder runs far fewer times than there are slices.
We started on Archie, our Linux workstation, where NeMo is installed. It has no GPU, and the export doesn't need one. There we loaded NVIDIA's checkpoint, traced each of the four pieces into TorchScript, and saved what each piece outputs for a fixed test input. Those saved outputs became our reference: whatever we built later had to reproduce them.
The conversion to Core ML ran on a MacBook Air with coremltools 9.0, which is built against torch 2.7.0, so we pinned torch there.
Our first encoder converted cleanly, loaded cleanly, and produced output of the right shape. Then we compared it with NeMo's saved output and got a cosine similarity of 0.82, where 1.0 means identical. The traced PyTorch model, before Core ML touched it, scored 0.74, so the conversion had faithfully copied a bad trace.
The conversion succeeded, and the model was wrong. Only the comparison against NeMo noticed.
The fault was a line we had added for convenience. Before tracing, our export script called NeMo's transcribe() to save a reference transcript. When transcribe() finishes, it puts the model back in evaluation mode but leaves the encoder inside it in training mode. Ask the model and it says it's evaluating; ask the encoder and it says it's training. A trace records whatever the code does on the run it watches, so ours recorded two things meant only for training. Dropout zeroes a random tenth of the values in each layer. Batch normalization scales each recording by its own statistics instead of the ones the model learned.
Cosine similarity treats each output as an arrow and measures the angle between the two. At 1.0 they point the same way.
Each line compares a trace with NeMo in evaluation mode, on the same 10.4-second recording, after the subsampling step and after each of the encoder's 24 layers. The faint lines are other runs of the same trace.
Dropout made the trace random, which is why it never scored the same twice. Core ML leaves dropout out when it converts a model, but it kept the batch normalization, and batch normalization alone scores 0.818. That is the 0.82 we started with. Traced in evaluation mode, the encoder matched NeMo at 1.0 after every layer. The 16-bit Core ML build scored 0.9966, and what's left is rounding.
A matching encoder was one check. The one that counts is whether the whole model writes the same words. We ran our 16-bit test build, before any compression, and FluidInference's published Core ML conversion of the same model over 200 recordings from LibriSpeech's clean test set, 4,634 words in all. They wrote identical text for 179. Fig. 6 has the other 21.
Most of the 21 come down to spelling: jeweller's against jeweler's, colour against color, one word or two. The rest sit mostly at the ends of recordings, where FluidInference's build tends to add a word nobody said and ours, once, dropped the last five. Across all 200, FluidInference's build made 123 word errors and ours 131. That is a small sample, and not the build we ship; Fig. 11 scores the shipped build on the full test sets.
At 16 bits per weight, the encoder is 1.1 GB: a long download before someone can dictate their first sentence, and a lot to hold in memory. So we compressed it with palettes.
A palette works like a paint-by-number kit. Instead of storing each weight as a 16-bit number, we pick 64 representative values and store each weight as a 6-bit index into that list. Core ML looks the real value up when it runs.
The question is which weights share a palette. One palette per tensor saved the most space and lost the most accuracy, and a histogram of the weights showed why: in one encoder tensor, a single outlier sat about 88 times farther from the center than a typical weight. To reach it, the palette had to spread its 64 entries thin, and only 32 of them landed where nearly all the weights were.
Giving each group of 16 output channels its own palette fixed it. An outlier now claims only its own group's palette, and the other groups spend all 64 entries on ordinary weights. We skip tensors with fewer than 2,048 values, where a palette costs more than it saves. That leaves 294 of the encoder's 636 tensors compressed, and nearly all of its bytes.
The k-means step that builds the palettes runs in parallel worker processes, and at first they hung forever. Python's multiprocessing on macOS starts each worker by re-importing the script, and our script had no __main__ guard. With the guard, the palettized encoder built in 782 seconds.
We compressed the encoder several ways and ran each build over LibriSpeech's full test sets: 5,559 recordings, about 10.7 hours of people reading books aloud. The "clean" set is clear speech; the "other" set is harder to hear.
Pick a build to compare it here and in Fig. 12.
The 8-bit build matched the palettes for accuracy but came out 128 MB larger and slower. One palette per tensor was the smallest and the least accurate. Keeping 48 sensitive constants at 16-bit recovered much of that loss. One palette per 16 channels, which we call pal6, had the lowest error on clean speech, stayed within a hair of the best on the harder set, and ran as fast as anything we built. We shipped it.
We put FluidInference's published conversion through the same test. It scored 2.92% on clean speech and 4.22% on the harder set. Over the 105,000 words in both sets, pal6 wrote 137 fewer wrong ones.
These rates come from our own decoding and text cleanup, applied the same way to every build here, FluidInference's included. They sit slightly above NVIDIA's published figures for the original model.
Apple Silicon offers three places to run a model: the CPU, the GPU and the Neural Engine. The Neural Engine is the fastest and uses the least power, but it accepts only certain operations in certain shapes. Core ML decides where each operation runs when it compiles the model, and we can ask it for that plan.
Slowed 16 times. Medians of ten runs on a MacBook Air.
Of the encoder's 1,427 operations, 1,399 run on the Neural Engine. The other 28 come before the first layer: they turn the recording's length into masks, which means whole numbers and true-or-false values the Neural Engine doesn't handle. Most of the work is matrix multiplication, and the Neural Engine gets through a 15-second window in 33 milliseconds. Held to the GPU, the same window takes 107; on the CPU, 154. The preprocessor runs on the GPU at full 32-bit precision, because turning sound into a spectrogram is sensitive to rounding, and the error would flow into everything after it. The decoder and joint are tiny and run once per step, so the CPU handles them without a round trip to another chip.
The first time a Mac loads the model, macOS compiles the encoder for that particular Neural Engine. It takes 90 to 113 seconds. The result is cached against the model's folder, so later loads take 1.7 seconds. Move the folder and the compile starts over.
Our checks start with arithmetic and end on a Mac that had never seen the model. Each one sits closer to real use than the one before.
Vox's own benchmark and the same clips sent through Talkie as dictations each missed 5 of 743 words, so the trip through the app leaves the result alone. On a clean install, Vox downloaded the model without an account, and the files matched ours byte for byte.
Vox, the speech engine inside Talkie, reads a catalog that says where each model lives. Shipping meant putting the files on Hugging Face, with credit to NVIDIA under the same CC-BY-4.0 license, and pointing the catalog's parakeet:v3 entry at them.
@ 743fa35parakeet:v3 points at that revisionmerged 5985681Oct 31, 2026477 MBWhen we rebuilt Talkie against the new Vox, its agent fetched the model on its own and started transcribing.
Parakeet v3 understands 25 European languages, and we have measured only English. We tested on M1 and M2 Macs; M3 and M4 compile their own Neural Engine plans, and we haven't checked them. The first-launch compile still takes a minute and a half, which we'd like to shorten. And the conversion scripts still live outside the Vox repository; moving them in would let anyone rebuild the model from NVIDIA's checkpoint.
Until then, the model that hears you in Talkie is one we built, measured, and can build again.
Correction, November 2: an earlier version of this post credited NeMo's _prepare_for_export() with fixing the trace. The cause was transcribe() leaving the encoder in training mode.