Order-preserving dedupe

dedupe.py

Run the Python Dedupe test next to the implementation for concrete examples.

If you've ever been paged at 3am because a queue consumer ran twice, you know duplicates aren't just a data-quality nit — they're a delivery bug. This utility removes duplicates from a list while keeping the order in which items first appeared. No third-party packages, no service to stand up, no config to drift.

Python Dedupe relies solely on the standard library. That means it works in a bare container, in a lambda, or on whatever box your cron lands on. Fewer moving parts, fewer things to break.

The approach is straightforward: iterate once, track what you've seen in a set, and append only the first occurrence. Order is preserved by construction, not by a sort afterward. It's idempotent by design — running it on already-deduped input is a no-op, which is exactly what you want when a retry fires and the previous run already finished the job.

Here's the core pattern:

def dedupe(items):
    seen = set()
    result = []
    for item in items:
        if item not in seen:
            seen.add(item)
            result.append(item)
    return result

That's it. The set gives you O(1) membership checks, the list keeps insertion order, and you're done. Memory grows with unique items, not the whole input, so it's fine for large streams that have a lot of repeats.

One thing to watch: this uses equality and hashing, so items need to be hashable. If you're dealing with unhashable types like dicts, you'll need a key function. The test file covers that case, so run it before you rely on this in a prod path.

The test suite lives right next to the implementation. Run it with python -m pytest or just execute the test file directly — it's standard-library only, so there's no test runner to install either. If the tests pass, you can trust the behavior; if they don't, you've caught a regression before it becomes a missed-job alert.