Why Data Minimisation and Detection Pull Apart

Detection and data minimisation want opposite things from the same system. Recognising that the tension is structural, rather than a problem someone can design away, is the starting point for handling it honestly.

Detection improves with breadth and history

Distinguishing coordinated activity from ordinary traffic depends on comparison, and comparison requires a baseline built from many sessions over a meaningful period.

More attributes per session also help, since coordination usually reveals itself through a combination of properties rather than through any single one.

Every argument for better detection is therefore an argument for collecting more and keeping it longer, and the argument is technically sound on its own terms.

Minimisation requires the opposite

Data protection principles require collecting only what is necessary for a stated purpose and retaining it only as long as that purpose requires.

Necessity is judged against the purpose rather than against what would be useful, and "it improves our model" is not a necessity argument in the sense the principle intends.

Retention limits bite hardest, because the historical baselines detection depends on are exactly what a short retention period removes.

Aggregation resolves part of the conflict

Baselines are statistical, so they can often be maintained as aggregates rather than as records of individual sessions.

Deriving features early and discarding the raw observations preserves most of the analytical value while removing the detailed record of what a specific person did.

This does not work for everything, since investigating a specific incident requires the underlying detail, but it covers the bulk of what long retention was being used for.

Tiered retention matches purpose to duration

Different data supports different purposes on different timescales, so a single retention period for everything is either too long for some of it or too short for the rest.

Short retention for raw session data, longer retention for aggregates, and case-scoped retention for confirmed incidents is the pattern that satisfies both requirements without pretending the tension does not exist.

Each tier needs a stated justification, and the exercise of writing those justifications usually reveals data that nothing actually needs.

The unresolved part should be stated

Some detection capability is genuinely lost under strict minimisation, and claiming otherwise leads to systems that quietly retain more than their documentation describes.

Being explicit about what is given up allows the trade to be made deliberately at the right level of the organisation, rather than being decided by whichever engineer set the retention configuration and moved on.