The model that picks your feed never sees most of the catalogue. A cheaper one already threw it away

Recommendation is almost certainly the most widely deployed machine learning in the world. It decides what plays next, what appears in a shop's listing, which video loads before you have finished the previous one. Everyone has a theory about "the algorithm", and almost none of those theories describe what the system is actually shaped like — which is a pity, because the shape explains more than the theories do.

You cannot score a billion things

Start with the constraint that determines the architecture. A model good enough to judge whether you specifically will like one specific item is expensive to run — it looks at your history, the item's properties, the time of day, dozens or hundreds of signals. Running that model against a catalogue of ten million items, for every request, within the fraction of a second a user will tolerate, is impossible by several orders of magnitude.

So no real system does. Every production recommender is staged.

A cheap retrieval step reduces millions of candidates to a few hundred, using methods fast enough to run at that scale. Then the expensive ranking model scores only those few hundred and orders them. Often a third step adjusts the final list for variety, freshness or business rules, because a ranked list of ten near-identical items scores well and reads badly.

The consequence is the part that gets missed. The good model never sees anything the cheap model discarded. If retrieval fails to surface an item, no amount of sophistication downstream can rescue it — it was never a candidate. The quality of what you are shown is set mostly by that first, crude stage, which is exactly the same structural point that governs retrieval-augmented generation: the clever component is capped by the dumb one in front of it.

How candidates get found

The oldest and still most important idea is collaborative filtering, and it requires no understanding of the content whatsoever.

Build a giant sparse table of who engaged with what. You do not need to know that two films are both thrillers; you need to know that people who watched one tended to watch the other. That co-occurrence is the signal. Factorise the table and you get compact vectors for every user and every item, positioned so that proximity means "tends to co-occur" — at which point recommendation becomes a nearest-neighbour search in the same way semantic search does.

Modern systems mostly use two-tower models: one network encodes the user, another encodes the item, and they are trained so that a user's vector lands near the items they engaged with. The advantage is operational rather than statistical — item vectors can be computed in advance and indexed, so serving a request means encoding one user and searching, not scoring a catalogue.

Alongside this sits content-based retrieval, using the item's own attributes, which exists mainly to cover the case collaborative filtering cannot handle at all.

Cold start makes popularity bias structural

A brand-new item has no interaction history. The co-occurrence table has an empty row for it, so the collaborative signal is not weak — it is absent. The same is true of a brand-new user.

Systems therefore fall back on what they do have: the item's attributes, and popularity. New things get shown because they resemble old things that worked, and widely-engaged things get shown because engagement is the one signal that is never missing.

This is worth stating carefully, because it is usually described as a value judgement by a platform. It is not primarily that. Popularity bias falls out of sparse data. A system with no information about an item, optimising to be right on average, will reasonably prefer the item it has evidence about — and the effect compounds, because being shown produces the very data that would justify showing it again. Deliberate countermeasures exist, and they cost measured performance in the short term, which is why they need an explicit decision rather than emerging on their own.

The model is not trained on what people like. It is trained on what people did, about the things it chose to show them.

The loop that hides its own mistakes

Here is the deepest problem in the field, and it is not about optimisation at all.

Training data comes from logs: item shown, user clicked or did not. But the shown items were chosen by the previous version of the model. You have abundant data about what happens when the model's favoured items are displayed, and no data whatsoever about the items it declined to display. The missing observations are not a random sample — they are precisely the items the system already judged unpromising.

So a model that is wrong about a category of items stops generating the evidence that would reveal the error. It shows them less, so they accumulate no engagement history, so they look even less promising, so it shows them less. The mistake becomes self-confirming, and it is invisible in offline evaluation because offline evaluation uses the same biased logs.

The standard responses are all partial and all worth knowing. Deliberate exploration: reserve a small fraction of slots for items the model is uncertain about, accepting a known cost now to buy information. Importance weighting: reweight logged events by how likely the old system was to have shown them, which corrects some of the bias if you recorded those probabilities — and you can only do that if someone thought to log them. Off-policy evaluation: estimate how a new model would have performed using old logs, which works within limits and fails when the new model wants to show genuinely different things.

And the one that actually settles arguments: controlled online experiments. Because the system influences the data that measures it, offline metrics routinely disagree with live outcomes, and the live test is the only honest arbiter. Teams that skip it ship improvements that are not.

Optimising the measurable thing

Briefly, because it is the same pattern this site has described in model training: the signal that is easiest to collect is a click, and a click is a poor proxy for whether anything valuable happened. Optimise it directly and you get systems that reward provocation and regret. The response has been multi-objective training — watch duration, later return visits, explicit ratings, survey-calibrated estimates of satisfaction — which helps and does not resolve the underlying difficulty, because the thing you actually want to increase happens over months and cannot be attributed to any single recommendation.

What "filter bubble" does and does not mean

The popular account has a model deciding to narrow your world. That is not what happens; nothing in the system has a preference about your worldview. What is real is the loop described above — a system that learns from what it showed will tend to keep showing more of it, and the alternatives quietly stop accumulating evidence.

Whether that measurably changes people's political attitudes is genuinely contested, and several large field experiments have found smaller effects than the popular account assumes. The loop is not in doubt. Its consequences for individual beliefs are, and that distinction is worth holding onto rather than collapsing in either direction.

Why it matters for students and researchers

In India this machinery increasingly decides commercial outcomes rather than just entertainment. Whether a small seller in a tier-three city is ever surfaced in a marketplace listing, whether a regional-language creator finds an audience, whether a lending product reaches a first-time borrower — all of these run through retrieval and cold start. The structural bias toward items with history is therefore a bias against everyone new, which in a fast-growing market means most participants.

The transferable lesson is broader than recommenders. Any system whose outputs influence the data it will later learn from has this property: it can be confidently, stably wrong, and its own metrics will not tell it. Credit scoring, hiring screens, policing deployment and fraud rules all share the structure. Learning to spot it — and knowing that the answer is deliberate exploration plus a real experiment, not a better offline score — is one of the more valuable habits available to anyone building with data.

Frequently asked questions

How does a recommendation system actually work?

In stages. A fast retrieval step selects a few hundred candidates from millions, then a slower, more accurate ranking model scores only those and orders them. A final pass often adjusts the list for variety or freshness.

What is collaborative filtering?

It is recommending based on patterns of co-engagement rather than on content: if people who interacted with one item tended to interact with another, those items are treated as related. The table of interactions is factorised into vectors so similarity becomes distance.

What is the cold start problem?

It is the absence of interaction history for a new item or a new user, which leaves collaborative methods with nothing to work from. Systems fall back on attributes and popularity, which is why new entrants are structurally disadvantaged.

Why do recommendation systems reinforce their own mistakes?

Because they train on logs of what they chose to show. Items the model declined to show accumulate no engagement data, so they continue to look unpromising, and the error never produces the evidence that would correct it.

Why are A/B tests necessary if offline metrics exist?

Because offline metrics are computed from data the previous model generated, so they carry its biases. A live controlled experiment is the only measurement that reflects what happens when the new system chooses what to show.