If you want to score generated text automatically, you need a function that takes a model output and a reference answer and returns a number. The obvious family of such functions compares words. The obvious problem with that family is that two sentences can share no words and mean the same thing. Embedding-based metrics exist to close that gap, and they close some of it.
Where n-gram metrics break
BLEU and ROUGE, the long-standing defaults, count overlapping word sequences. BLEU is precision-oriented and came from machine translation; ROUGE is recall-oriented and came from summarisation. Both are fast, deterministic and easy to explain, which is most of why they survived.
They fail in a specific, predictable way. “The meeting was postponed” and “They pushed the meeting back” carry the same information and share one content word. An n-gram metric scores that pair as a near-total miss. It will equally happily score a fluent paraphrase below a clumsy near-copy, which means optimising against it pushes your system toward mimicry rather than meaning.
Embedding similarity
The simplest alternative: embed both texts, take the cosine similarity of the two vectors. Paraphrases now score high, because the embedding model was trained to put them near each other.
The weakness is that a single vector per document compresses hard. Negation is the notorious case — “the transaction succeeded” and “the transaction did not succeed” embed close together in most models despite being opposites. Anything where a small lexical change flips the meaning is a place this metric will quietly mislead you.
BERTScore
BERTScore keeps more resolution. Rather than one vector per text, it embeds every token in context, then greedily matches each token in the candidate to its most similar token in the reference and averages the resulting similarities. Because the embeddings are contextual, the same word in different senses gets different vectors, and you can compute precision, recall and F1 variants just as with n-gram metrics.
It correlates with human judgement noticeably better than BLEU or ROUGE on most tasks. It is also slower, it depends on which underlying model you use — so scores are not comparable across configurations — and it still rewards saying the same things rather than saying correct things.
The limit all of them share
Every metric here measures similarity to a reference. That is a proxy for quality, and it is a poor one whenever a good answer can differ from the reference — which is to say, for most open-ended generation. A response can be more accurate, better structured and more useful than your reference answer and score badly for it.
They also require references to exist. For a live product with no ground truth, the entire family is unavailable, which is why LLM-as-judge has taken over so much of this territory: it scores against a rubric instead of a reference, so it works where no reference exists.
Practical guidance
Use embedding metrics where they are strong: as fast regression checks in CI, where you have references and you care about whether today’s change broke something rather than about absolute quality. They are cheap, deterministic and repeatable, which makes them good gatekeepers and bad judges.
Do not report them as quality scores to anyone making a decision. A BERTScore of 0.87 means nothing on its own; it is only interpretable as a delta against your own previous run. And whatever you automate, keep a small human-reviewed set alongside it. Every automatic metric drifts away from what you actually care about, and the human set is how you notice.
