can't break enrichment processing back to shell with ctrl-c

[HongPong]

during enrichment ctrl-c (pressing a few times) gives

Enriching facility NORTH LAKE CORRECTIONAL FACILITY...
^CEnriching facility NORTHEAST OHIO CORRECTIONAL CENTER...
Enriching facility NORTHERN REGIONAL JAIL...
Enriching facility NORTHWEST STATE CORRECTIONAL CENTER...
^CEnriching facility OLDHAM COUNTY DETENTION CENTER...
^CProcess ForkProcess-2:
Process ForkProcess-3:
Traceback (most recent call last):
Traceback (most recent call last):
  File "/installs/python/3.13.3/lib/python3.13/multiprocessing/process.py", line 313, in _bootstrap
    self.run()
    ~~~~~~~~^^
  File "/installs/python/3.13.3/lib/python3.13/multiprocessing/process.py", line 313, in _bootstrap
    self.run()
    ~~~~~~~~^^

/installs/python/3.13.3/lib/python3.13/multiprocessing/connection.py", line 395, in _recv
    chunk = read(handle, remaining)
KeyboardInterrupt
KeyboardInterrupt
^C
^C^C^C^C^C

etc and the shell is not reset back to normal. i think it would be best if we could break out.

process such as

/share/mise/installs/python/3.13.3/bin/uv run python main.py --scrape --enrich

can be halted by kill and the process number as normal.

[johnseekins]

So it does exit on ctrl-c:

$ uv run python main.py --enrich
ICE Detention Facilities Scraper by the Open Security Mapping Project. MIT License.
Did not supply --scrape or --load-existing. Proceeding with default data set (20 facilities)
Starting data enrichment with external sources...
Enriching facility BAKER COUNTY SHERIFF DEPT....
Enriching facility SAUK COUNTY SHERIFF...
Enriching facility NORTHWEST ICE PROCESSSING CENTER...
^CEnriching facility LOGAN COUNTY JAIL...
Enriching facility JEFFERSON COUNTY JAIL...
Enriching facility GRAYSON COUNTY JAIL...
Enriching facility COLLIER COUNTY NAPLES JAIL CENTER...
Traceback (most recent call last):
  File "/home/john/github/ice_detention_scraper/main.py", line 167, in <module>
    main()
    ~~~~^^
  File "/home/john/github/ice_detention_scraper/main.py", line 154, in main
    facilities_data = enrich_facility_data(facilities_data, args.enrich_workers)
  File "/home/john/github/ice_detention_scraper/enrichers/general.py", line 27, in enrich_facility_data
    for res in pool.map(_enrich_facility, facilities_data["facilities"].items()):
               ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/john/.local/share/mise/installs/python/3.13.3/lib/python3.13/concurrent/futures/process.py", line 617, in _chain_from_iterable_of_lists
    for element in iterable:
                   ^^^^^^^^
  File "/home/john/.local/share/mise/installs/python/3.13.3/lib/python3.13/concurrent/futures/_base.py", line 619, in result_iterator
    yield _result_or_cancel(fs.pop())
          ~~~~~~~~~~~~~~~~~^^^^^^^^^^
  File "/home/john/.local/share/mise/installs/python/3.13.3/lib/python3.13/concurrent/futures/_base.py", line 317, in _result_or_cancel
    return fut.result(timeout)
           ~~~~~~~~~~^^^^^^^^^
  File "/home/john/.local/share/mise/installs/python/3.13.3/lib/python3.13/concurrent/futures/_base.py", line 451, in result
    self._condition.wait(timeout)
    ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/home/john/.local/share/mise/installs/python/3.13.3/lib/python3.13/threading.py", line 359, in wait
    waiter.acquire()
    ~~~~~~~~~~~~~~^^
KeyboardInterrupt

Just not like you'd expect. You have to wait...'cause there are subprocesses that...don't appear to be properly receiving the signal.

I agree we should explore this more, but it feels like somewhat lower priority than other things.

[HongPong]

Yeah I agree it seems like they are getting the message eventually so it is not such a big deal. i'll leave open as 'nice to have' but we don't need to worry for now.

[HongPong]

The AI thing suggested this to me which is cool but i am not gonna mess with it for now..

---

Let me look at the enrichment code to understand how multiprocessing is being used:Now I can see the problem clearly. The code uses ProcessPoolExecutor with pool.map(), which spawns forked processes. When you hit Ctrl+C:

1. The main process receives SIGINT (KeyboardInterrupt)
2. But pool.map() blocks waiting for results, and forked child processes don't get cleanly interrupted
3. The child processes are stuck in I/O (HTTP requests) and won't terminate until those complete

Here's a fix that makes Ctrl+C work cleanly:

from concurrent.futures import ProcessPoolExecutor, as_completed
import copy
from enrichers import (
    openstreetmap,
    wikidata,
    wikipedia,
)
from schemas import (
    facilities_schema,
)
import signal
import time
from utils import logger


def _init_worker():
    """Make child processes ignore SIGINT so only the parent handles it."""
    signal.signal(signal.SIGINT, signal.SIG_IGN)


def enrich_facility_data(facilities_data: dict, workers: int = 3) -> dict:
    """wrapper function for multiprocessing of facility enrichment"""
    start_time = time.time()
    logger.info("Starting data enrichment with external sources...")
    enriched_data = copy.deepcopy(facilities_schema)
    total = len(facilities_data["facilities"])
    processed = 0

    with ProcessPoolExecutor(max_workers=workers, initializer=_init_worker) as pool:
        futures = {
            pool.submit(_enrich_facility, item): item[0]
            for item in facilities_data["facilities"].items()
        }
        try:
            for future in as_completed(futures):
                res = future.result()
                enriched_data["facilities"][res[0]] = res[1]  # type: ignore [index]
                processed += 1
                logger.info("  -> Finished %s, %s/%s completed", res[1]["name"], processed, total)
        except KeyboardInterrupt:
            logger.warning("\nInterrupted! Shutting down workers...")
            pool.shutdown(wait=False, cancel_futures=True)
            raise

    logger.info("Data enrichment completed!")
    enriched_data["enrich_runtime"] = time.time() - start_time
    logger.info(" Completed in %s seconds", enriched_data["enrich_runtime"])
    return enriched_data


# ... existing code ...

What changed and why

1. _init_worker + initializer=_init_worker — Each child process is told to ignore SIGINT. This prevents the messy tracebacks from child processes all independently catching the signal and dumping stack traces. Only the parent process handles Ctrl+C.

2. pool.submit() + as_completed() instead of pool.map() — With pool.map(), the iterator blocks internally in a way that makes it hard to catch KeyboardInterrupt promptly. Using submit() + as_completed() gives us an explicit loop where we can catch the interrupt between result retrievals.

3. except KeyboardInterrupt — When you press Ctrl+C, we catch it, call pool.shutdown(wait=False, cancel_futures=True) to immediately stop the pool without waiting for pending tasks, then re-raise so the program exits cleanly.

This means a single Ctrl+C will now cleanly shut everything down instead of leaving zombie processes you have to kill manually.

---

After calling pool.shutdown(...), the code does raise — which re-throws the same KeyboardInterrupt exception up to the caller (main()). Since main() doesn't catch it either, Python exits with the standard KeyboardInterrupt traceback (or you could catch it in main() for an even cleaner exit).

Without the raise, the function would silently continue to the logger.info("Data enrichment completed!") line and try to return incomplete data as if everything succeeded.