Teaching TranslateGemma Moroccan Darija: What Worked, What Broke, and What 32.8 BLEU Actually Means

Moroccan Darija is not an obscure language to the people who speak it, but it can look remarkably obscure to a language model. It has no universally agreed written standard, moves freely between Arabic and Latin scripts, tolerates enormous variation in spelling, and borrows naturally from other languages. A sentence that looks completely ordinary in a Moroccan conversation may therefore sit surprisingly far from the text distributions on which even strong multilingual models learned to translate.

That makes Darija an interesting test of what multilingual models actually learn. If a capable translation model performs poorly on it, is the problem that the model is too small, that Darija is intrinsically difficult to model, or simply that the language has not been represented well enough during training? I wanted to test the last possibility before reaching for more scale.

I fine-tuned Google’s TranslateGemma 4B for Moroccan Darija-to-English translation, training it on both Arabic- and Latin-script text. On an external evaluation derived from TerjamaBench, BLEU rose from 7.49 to 32.82, chrF from 31.15 to 52.10, while TER fell from 116.11 to 53.53. The most interesting result came from Latin-script Darija: 3.59 BLEU before fine-tuning, 31.91 after it. A model that had the capacity to translate the language apparently lacked much of the exposure required to recognize one of the ways people actually write it.

The numbers are the clean part of the experiment. Getting them was not. The training run crossed GPU providers before it finished, checkpoint recovery broke along the way, and several apparent model problems turned out to have little to do with the model itself. Those complications ended up making the project more useful than a successful fine-tuning run would have been on its own: it became an experiment in data, evaluation, training infrastructure, and eventually in how much useful translation capability can be packed into a model small enough to run on a laptop.

Why TranslateGemma 4B

TranslateGemma is Google’s translation-focused Gemma family. It uses a dedicated translation template where the source and target languages are explicitly provided instead of relying on a generic conversational prompt. The 4B checkpoint was especially attractive for this project because it sits in a useful middle ground: large enough to inherit substantial multilingual and translation capability, while still small enough to fine-tune on a single RTX 4090 and later run locally on consumer hardware.

I could have started with a 12B or 27B model. That would also have made every training mistake several times more expensive. For a narrow problem such as Darija to English translation, I wanted to first find out how far the smallest serious member of the family could be pushed.

The task has another annoying property. Darija has no dedicated language code in TranslateGemma, and people write it in both Arabic script and Latin-script Arabizi. During training, both were therefore passed through the Arabic-to-English translation route while the model learned from the examples what Moroccan Darija actually looked like.

That slightly awkward setup turned out to work much better than I expected.

The data, in a nutshell

The model was trained from a corpus of 500,350 semantic rows, built from human DODa data plus a much larger controlled synthetic expansion. Each semantic row connects the same meaning across English, Arabic-script Darija, and Latin-script Darija. During preparation, each row could therefore become both an Arabic→English and an Arabizi→English example, while the two representations were kept together during splitting to avoid leakage across splits. Human DODa examples were sampled 3× during training, and I removed 57 semantic groups that overlapped with the external benchmark before the split. After expansion and human-data upsampling, the final training stream contained 1,185,690 directional examples. I wrote separately about how that dataset was generated in How We Expanded DODa to 500K Darija Examples, because there is enough trouble in synthetic data to deserve its own article.

Training a 4B model without training 4B parameters

I used QLoRA rather than full fine-tuning. The base model was loaded in 4-bit NF4, the vision part was frozen, and LoRA adapters were attached to the language-model attention and MLP projections. Training used bfloat16 compute on RTX 4090 GPUs, an effective batch size of roughly 64, gradient checkpointing, and a single epoch. The complete epoch contained 18,527 optimizer steps and took about 28 hours of active training time, split across RunPod and Vast.ai after the mid-run migration.

The calendar time was longer because the run included checkpoint transfers, environment failures, the cloud migration, and evaluation. I prefer quoting active training time here because it is the more meaningful compute number.

The first version of the training setup was much less satisfying. GPU utilization spent far too much time around 20 to 30 percent and the projected runtime was ugly. That is where the project gave me one of its more useful reminders: before buying faster hardware, check whether the hardware you already rented is actually doing anything.

I changed the data and batching path, added throughput-aware configuration, and got the 4090 spending much more of its time on the model instead of waiting for Python. The training estimate improved enough that finishing the experiment became reasonable.

Validation loss reached its minimum around step 13,000 and rose slightly afterward. I still completed the planned single epoch because the increase was modest, the learning rate was already decaying toward zero, and token-level validation loss was not the external translation metric I ultimately cared about. I preserved the intermediate checkpoints; in a future run, I would evaluate the best-validation checkpoint against TerjamaBench as well.

Then I moved the training run to another cloud

RunPod worked well, but I was paying around $0.74/hour for the 4090 option I was using. At the point where most of the expensive experimental mistakes had already happened, I found suitable Vast.ai capacity around $0.40/hour.

Those are the prices I personally saw during this run, rather than general price claims about either platform.

Because checkpoints were already being pushed to Hugging Face, the migration looked conceptually simple:

stop RunPod
        ↓
start RTX 4090 on Vast
        ↓
resume training (checkpoints are already stored on HF)

No manual download step needed since checkpoints are persisted to Hugging Face during training.

The first Vast environment used a PyTorch version below 2.6 and hit a safety check requiring PyTorch 2.6 or newer because of CVE-2025-32434, a critical vulnerability affecting torch.load in earlier releases. Moving to PyTorch 2.6 fixed that problem, but also exposed a checkpoint-compatibility issue of its own.

PyTorch 2.6 also changed torch.load so that weights_only=True became the default, a deliberate security improvement that also introduced backward-compatibility consequences for some serialized objects.

The Hugging Face checkpoint itself contained several types of state:

• model / adapter
• optimizer
• scheduler
• trainer state
• RNG state

The failure happened when restoring rng_state.pth. The safer unpickler rejected a NumPy reconstruction object:

Unsupported global:
numpy.core.multiarray._reconstruct

At this stage, starting again from step zero was technically possible and emotionally unacceptable.

The recovery was fairly surgical. I downloaded the checkpoint from Hugging Face, temporarily removed only rng_state.pth, and resumed the Trainer from everything else.

That preserved the model weights, optimizer state, learning-rate scheduler, trainer state, and global step. It did sacrifice exact random-number continuity across the migration.

This distinction matters. A checkpoint can be good enough to recover training without being sufficient for bit-for-bit reproduction of the next batch sequence.

When the resumed progress bar jumped to roughly 13,001 / 18,527, I knew the important state had survived.

So, did the model actually get better?

Training loss is useful, but a translation model eventually has to translate something it has not seen.

For the external evaluation I used TerjamaBench, a manually reviewed Moroccan Darija benchmark created by AtlasIA. It contains 850 entries spanning Arabic-script Darija, Arabizi, and English, with categories covering things such as code-switching, spelling variation, idioms, humor, dates, named entities, dialectal variation, and longer sentences.

The published benchmark is primarily presented in the English→Darija direction. My task was Darija→English, so I reversed it and evaluated both Darija representations against their English reference:

• 850 Arabic-script Darija → English
• 850 Arabizi → English

One detail matters here: AtlasIA’s published benchmark results use a filtered English→Darija setup, while I evaluated all 850 available rows in the opposite direction, including the dialect-variation category. The scores in this article should therefore be treated as a separate reversed TerjamaBench evaluation, rather than compared directly with the published TerjamaBench leaderboard.

The original TranslateGemma checkpoint and the fine-tuned model were evaluated through the same generation path.

BLEU rewards n-gram overlap with the reference translation. chrF performs a similar comparison at character level and is often more forgiving of legitimate wording differences. TER estimates how much editing would be required to transform the model output into the reference, so lower is better.

None of these metrics understands language like a human evaluator does. Together, though, the direction is difficult to misread. BLEU increased by more than 25 points, chrF gained about 21 points, and TER was cut by more than half.

The script breakdown is more revealing:

The model learned Arabizi particularly aggressively. That was one of the main reasons for building every training example around paired Arabic and Latin representations of the same meaning, and the final numbers suggest that the strategy transferred outside the training corpus.

What the aggregate score hides

The overall score hides a lot.

Long sentences were among the strongest categories, reaching 47.62 BLEU in Arabic script and 46.05 in Arabizi. Mixed-language examples were also strong at 41.04 and 43.18 respectively. Named entities, common phrases, and misspellings performed reasonably well too.

Then we arrive at idioms.

Arabic-script idioms scored 6.11 BLEU.

Arabizi idioms scored 2.63.

Half a million rows were still not enough to solve idiomatic language, where translating the words correctly can produce entirely the wrong translation.

That is useful information. Dialectal variation also stayed around 19 BLEU, while humor, religion, and numbers/dates remained below the overall score. These categories tell me much more about the next dataset iteration than another random 500,000 generated sentences would.

One result in the benchmark also deserves a small warning. Arabic single-word examples produced 0 BLEU while chrF was 62.66. Corpus BLEU becomes unreliable on tiny, very short subsets because higher-order n-grams may simply not exist. The zero makes for a dramatic screenshot and a terrible conclusion. For that category, chrF and direct inspection are much more useful.

The 70 BLEU result I would rather not advertise

The internal test set produced an overall BLEU score of 69.98.

If I wanted the largest number possible in a LinkedIn post, I could probably stop there.

The split tells the real story:

The synthetic examples were generated from the same broad process as most of the training corpus. They are held out, so the model did not memorize those exact rows, but they still share the linguistic preferences of the generator. A model trained extensively on that distribution becomes very good at that distribution.

The human subsets land around 30 BLEU. The external TerjamaBench score lands at 32.82.

That agreement is much more reassuring to me than the 70.

How good is 32.82 in the wider Darija landscape?

There still is no clean public Darija→English leaderboard where every open system is evaluated on exactly the same test set, so comparing reported BLEU values requires some discipline.

Rosca, Issam, and Spanakis reported 26.79 BLEU for their best Darija→English system in their 2025 low-resource MT study, evaluated on their DODa test setup.

A much newer open model, oddadmix/50M-Darija-English-v1, reports 39.72 BLEU and 53.22 chrF on its own held-out 3,000-pair evaluation set. It is particularly impressive because it has only about 52M parameters and handles both Arabic-script Darija and Arabizi.

Those numbers cannot be ranked directly against my 32.82 because the datasets differ. A model can gain or lose many BLEU points simply by changing domains, reference style, sentence length, or how close the test distribution is to its training data.

The claim I am comfortable making today is narrower: this fine-tuned TranslateGemma is competitive with strong open Darija→English work, and its external Arabizi performance is particularly encouraging.

The obvious next experiment is to run the strongest open models on these same 1,700 reversed TerjamaBench inputs. If I do that, the comparison becomes much more interesting and much harder to argue with.

The surprisingly nice ending: it runs on my Mac

Once the cloud training was finished, I wanted a very ordinary test. Type Darija, get English back.

I loaded the 4B base model and the LoRA adapter on my M4 MacBook Air with 24 GB of unified memory through Apple’s MPS backend. My first attempt produced rows of <pad> tokens because of a local precision issue. After moving the inference path to bfloat16 and matching the exact prompt/tokenization route used during evaluation, the translations started behaving normally.

What surprised me afterward was the speed.

I have not benchmarked it properly yet, so I am intentionally avoiding numbers here. Subjectively, though, the interaction feels fast enough that the model behaves like a local translator rather than a research checkpoint you wait around for.

That matters more than I expected when I chose the 4B model.

A formal benchmark should come next: warm latency, first-token latency, output tokens per second, p50, p95, and peak unified memory across a representative prompt set. Until then, “surprisingly fast on my Mac” is an observation rather than a performance claim.

Still, it is a nice observation.

What I would do differently

The biggest lesson from this project is that the model itself caused fewer problems than everything surrounding it.

I would benchmark throughput before committing to any multi-day training estimate. I would test checkpoint restoration early, while restarting still feels cheap. I would pin the entire runtime more aggressively, including container, PyTorch, Transformers, PEFT, CUDA, and evaluation libraries. I would also treat the remote checkpoint repository as part of the system architecture from the start, because the ability to throw away a GPU instance and continue somewhere else ended up being one of the most valuable properties of the whole setup.

The evaluation changed my view of the data too. Another million generic synthetic sentences would probably increase the number on the dataset card more than the usefulness of the model. Idioms, regional variation, cultural expressions, humor, and difficult human examples are where I would spend the next annotation budget.

And I would keep the 4B model.

The larger TranslateGemma variants may eventually improve quality, but this run showed that a relatively small model can move from 7.49 to 32.82 BLEU, almost eliminate its Arabic-versus-Arabizi gap, handle code-switched Darija surprisingly well, and still remain practical enough to run interactively on a laptop.

For the next iteration, better Darija looks like a more promising investment than a bigger model.