Same Words, Different Meaning: Can Vector Retrieval Tell Scope Apart?

My starting intuition was a piece of mathematics. "For all δ there exists ε" and "there exists ε for all δ" use the same words, but they do not mean the same thing. Swap the order of the quantifiers and the statement changes — in one direction every δ gets its own ε, in the other a single ε has to work for all δ. The words are nearly identical. The meaning is not.

That raised a question about the machinery underneath most retrieval-augmented systems: can vector retrieval tell the two apart? If a textbook contains both versions — the correct statement and its scope-swapped twin — and a user asks about one of them, does the retriever return the right one, or does it hand back both and leave the distinction to something downstream?

My prediction, going in, was blunt: several state-of-the-art embeddings, times several distance measures, would all fail. This essay reports two experiments that tested that prediction — a small probe, and then a scaled-up version with forty more minimal pairs. The prediction turned out to be half right, and the half that was wrong is the interesting part.

The design: minimal pairs and a WIN rule

The experiment is built from minimal pairs: two sentences that share nearly all their words and differ only in scope — quantifier order, negation placement, or connective grouping — with genuinely different truth conditions. Each pair comes with a distinguishing scenario, a concrete situation where one twin is true and the other is false. If no such scenario exists, the pair is not a pair; it is the same statement twice.

Each side of each pair gets one query, written as a paraphrase of that side's reading. The judgment rule is strict and pairwise: a query counts as a WIN only if the correct segment scores above its scope-swapped twin. Ranking the right segment first is not required; beating the twin is. That is the whole test. Everything else — distractors, other segments, absolute scores — is context.

One subtlety matters for the weak side of a quantifier swap. A weak reading ("every patient has some doctor") is logically entailed by the strong reading ("some doctor treats every patient"), so a plain paraphrase of the weak side cannot exclude the strong world. The weak-side queries therefore use mixed scenarios — different doctors for different patients — which is the only phrasing that pins the reader to the weak side.

The methods under test:

• A hand-written BM25 (k1 = 1.5, b = 0.75) as the keyword control.
• Three bi-encoders with mean pooling: MiniLM-L6-v2, bge-base-en-v1.5, and e5-base-v2 (e5 with its specified query:/passage: prefixes).
• Four distances — cosine, dot, Euclidean, Manhattan — each computed on raw vectors and on L2-normalized vectors.
• A cross-encoder reranker, ms-marco-MiniLM-L-6-v2, scored over the full corpus. This is the key control: a bi-encoder never sees the query and the passage together, but a cross-encoder reads both at once, so if anything can see scope, it should be this one.

Everything ran on CPU (torch 2.14.1, transformers 5.20.2).

The ε-δ pair, worked

My original example swapped δ and ε in the other direction. The experiment's ε-δ pair is the same kind of quantifier-order swap, with the variables in the roles they play in the standard continuity definition: pointwise (∀ε∃δ) against uniform (∃δ∀ε).

• E1 (pointwise): "For every epsilon > 0, there exists a delta > 0 such that for all x, ..."
• E2 (swapped): "There exists a delta > 0 such that for every epsilon > 0, for all x, ..."

The two sentences are, as bags of words, identical. Here is what the first experiment scored on them (normalized cosine, except the cross-encoder):

BM25, the pure word-bag control, scored the pair as an exact tie both times (6.929 = 6.929 and 7.876 = 7.876) — as it must, since the word bags are the same. The embeddings do slightly better than a tie: they win by 0.005 to 0.015. That margin is the first hint of the shape of the whole result. Some order information does get into the vectors. It is just not strong, and it is not reliable.

Experiment 1: the probe

The first run used 14 segments: four minimal pairs plus six distractor passages (a limit definition, a derivative definition, sequence convergence, a syllogism, differentiable-implies-continuous, an open-set definition). The four pairs were the ε-δ pair above, one propositional pair ("It is not true that both Alice and Bob passed", ¬(A∧B), against "Alice did not pass, and Bob passed", (¬A)∧B), and two first-order pairs (every student likes some teacher against some teacher liked by every student; and the mother version of the same swap). Eight queries, one per side.

Two facts about distances came out of this run and survived the scale-up. On normalized vectors, cosine, dot, and Euclidean rankings are completely identical — a mathematical identity, not an experimental coincidence. And raw dot product is the least stable measure (MiniLM raw dot scored 4/8 here), because it folds vector length into the comparison. Do not use raw dot.

The probe's verdict: my prediction was half right. There was no total failure — bge and e5 each scored 7/8. But the propositional pair, which every method won, is also the pair whose twins differ most on the surface, so it proves the least. The real test is the hard pairs, where the word bags are identical or nearly so — and there, every model lost at least one query, every winning margin was between 0.003 and 0.015, and even the cross-encoder lost two. The probe was too small to stop there, though. Eight queries is a hint, not a result.

Experiment 2: the scale-up

The second run kept the first run's 14 segments and 8 queries untouched and added 40 new minimal pairs: 20 propositional-logic pairs (negation scope, connective grouping, implication scope, only-if versus if, if-and-only-if, inclusive versus exclusive or, unless, inverse conditionals), 15 first-order quantifier swaps across different domains (patients and doctors, cities and mayors, problems and solutions, employees and managers, songs and singers, keys and locks, courses and teachers, books and authors, dogs and owners, students and tutors, villages and wells, teams and coaches, articles and editors, gardens and gardeners, movies and directors), and 5 negation-quantifier pairs ("not every" against "no", "some did not" against "none did"). That is 94 segments and 88 queries. The run finished with exit code 0 in about four minutes.

Per-method results

The best bi-encoder is e5 at 65/80 on the new pairs (81.2%), then bge at 62/80 (77.5%), then MiniLM at 52/80 (65.0%). The cross-encoder, the one method that reads the query and the passage together, scores 53/80 (66.2%) — below both bge and e5. Its middling first-run score was not a small-sample fluke; at scale it falls below the bi-encoders it was supposed to beat.

Per-type breakdown (headline methods, normalized cosine)

Negation-quantifier pairs are the easiest (70–90% across methods), propositional pairs sit in the middle (65–80%), and quantifier swaps are the hardest with the smallest margins. That ordering makes sense: "no" and "not every" are themselves strong lexical signals, while a pure quantifier swap changes almost nothing on the surface.

How big are the wins? Margin analysis

A win is not a win if the margin is noise. The table below shows the distribution of (correct score − twin score) for normalized cosine; the cross-encoder column is a logit difference, on a different scale, so it is only comparable within its own column.

Read the e5 swap row: 73.3% of its swap margins fall within ±0.015, and 17 of its 24 swap wins are narrow wins. The best model on the hardest pairs is winning by noise. bge is steadier on swaps (mean +0.021, median +0.025) but still has 30% of its margins in the narrow band. Negation-quantifier margins are far larger (bge mean +0.074), because the distinguishing words are themselves strong signals.

The cross-encoder is left out of this chart because its scores are logits on a different scale; the 0.015 band does not apply to it.

Recall is fine. Precision is the failure.

On quantifier swaps, the twin is in the top 3 every single time, for all three bi-encoders and for the cross-encoder. Retrieval always hands over both contradictory versions. So the failure is not that the right passage is missing. It is that the wrong passage is right next to it, and the ranking between them is a coin flip. Any system that retrieves top-k and reasons over the result is being handed contradictory material and asked to sort it out downstream.

Two more precise facts. First, weak readings are harder than strong readings: on swap pairs, weak-side (∀∃) queries beat strong-side (∃∀) queries at 7/15 versus 12/15 for MiniLM, 11/15 versus 15/15 for bge (bge is perfect on the strong side), and 13/15 versus 11/15 for e5. Models drift toward the shorter, more canonical strong sentence. Second, difficulty is concentrated, not spread out: 8 of the 80 new queries are lost by all three embeddings (P01_a, P02_a, P04_b, P07_a, P14_a, P20_b, F11_a, N04_b) — all weak-reading or negated-weak-reading forms — while 46 of 80 are won by all three.

And BM25 on swaps deserves its own line: 14/30 with a mean margin of −0.69. Word overlap does not merely fail to help here; it systematically favors the wrong twin.

The first run reproduces exactly

The strongest check that the scale-up did not change the experiment: the first run's 8 queries, re-scored inside the 94-segment corpus, come out WIN/LOSE-identical for all five headline methods.

The protocol did not drift, and the first run's numbers were not a small-corpus artifact. The direction is stable.

What this means

The honest summary is not "retrieval fails." It is narrower and more useful: retrieval succeeds at recall and fails at the last step. On the hardest pairs — the ones where the words are the same and only scope differs — the retriever reliably returns both versions and then ranks them by a margin smaller than noise. A language model reading the top-k results receives two passages that contradict each other, with no signal in the ranking about which one answers the question.

That locates the fix. It is not at recall — widening the net, adding more passages, or retrieving more candidates will not separate twins that already sit next to each other at the top. It has to be at ranking or verification: a step that actually compares the candidate passages against the question's scope — which quantifier binds what, where the negation lands — and checks the answer against it, rather than trusting a similarity score that was never trained to see scope. Until such a step exists, any precise material — mathematics, law, contracts, specifications — retrieved this way carries a quiet risk: the passage that looks right and means the opposite is always one rank away.

Caveats that stay attached

• The queries are paraphrases written for the experiment, not phrases a real user typed. Different phrasing could move the results.
• The passages are single sentences. Real textbook blocks are 512-token chunks, where the scope signal could be diluted by surrounding text or strengthened by context. That was not tested.
• Only English was tested. Languages that mark quantifier scope differently were not tested.
• 88 queries is still probe scale, not a benchmark — but the exact reproduction of the first run inside the bigger corpus says the direction is stable.

Reproducibility

The runnable scripts (exp.py, exp2.py), the raw outputs (exp-output.txt, exp2-output.txt), and the full pair inventory with logical forms and distinguishing scenarios live in the quant-order-exp directory. The first run's files were left byte-identical when the second run was built; the second run is a separate script and a separate record, not an edit of the first.

The question I started with has a sharper answer now. Vector retrieval does not see scope. It sees something correlated with scope — enough to beat a word-bag tie by a hundredth of a point, not enough to trust when the words are the same and the meaning is not.