1. CI Logging Enhancements:
- Added comprehensive diagnostics with process tree, network listeners, file descriptors
- Added timestamps at every stage (before/during/after pytest)
- Added trap EXIT to always show diagnostics
- Added immediate process checks after pytest finishes
- Added sub-shell execution with immediate cleanup
2. Fixed Subprocess PIPE Blocking:
- Changed Colab mode from PIPE to DEVNULL to prevent blocking
- PIPE without reading can cause parent process to wait indefinitely
3. Pytest Session Hooks:
- Added pytest_sessionstart to log initial state
- Added pytest_sessionfinish for aggressive cleanup before exit
- Shows all child processes and their status
This should reveal exactly where the hang is happening.
Based on excellent diagnostic suggestions, implemented multiple fixes:
1. Diagnostics:
- Added faulthandler to dump stack traces 10s before CI timeout
- Enhanced CI script with trap handler to show processes/network on timeout
- Added diag() function to capture pstree, processes, network listeners
2. ZMQ Socket Timeouts (critical fix):
- Added RCVTIMEO=1000ms and SNDTIMEO=1000ms to all client sockets
- Added IMMEDIATE=1 to avoid connection blocking
- Reduced searcher timeout from 30s to 5s
- This prevents infinite blocking on recv/send operations
3. Context.instance() Fix (major issue):
- NEVER call term() or destroy() on Context.instance()
- This was causing blocking as it waits for ALL sockets to close
- Now only set linger=0 without terminating
4. Enhanced Process Cleanup:
- Added _reap_children fixture for aggressive session-end cleanup
- Better recursive child process termination
- Added final wait to ensure cleanup completes
The 180s timeout was happening because:
- ZMQ recv() was blocking indefinitely without timeout
- Context.instance().term() was waiting for all sockets
- Child processes weren't being fully cleaned up
These changes should prevent the hanging completely.
Fixed the actual root cause instead of just masking it in tests:
1. Root Problem:
- C++ side's ZmqDistanceComputer creates ZMQ connections but doesn't clean them
- Python 3.9/3.13 are more sensitive to cleanup timing during shutdown
2. Core Fixes in SearcherBase and LeannSearcher:
- Added cleanup() method to BaseSearcher that cleans ZMQ and embedding server
- LeannSearcher.cleanup() now also handles ZMQ context cleanup
- Both HNSW and DiskANN searchers now properly delete C++ index objects
3. Backend-Specific Cleanup:
- HNSWSearcher.cleanup(): Deletes self.index to trigger C++ destructors
- DiskannSearcher.cleanup(): Deletes self._index and resets state
- Both force garbage collection after deletion
4. Test Infrastructure:
- Added auto_cleanup_searcher fixture for explicit resource management
- Global cleanup now more aggressive with ZMQ context destruction
This is the proper fix - cleaning up resources at the source, not just
working around the issue in tests. The hanging was caused by C++ side
ZMQ connections not being properly terminated when is_recompute=True.
Based on excellent analysis from user, implemented comprehensive fixes:
1. ZMQ Socket Cleanup:
- Set LINGER=0 on all ZMQ sockets (client and server)
- Use try-finally blocks to ensure socket.close() and context.term()
- Prevents blocking on exit when ZMQ contexts have pending operations
2. Global Test Cleanup:
- Added tests/conftest.py with session-scoped cleanup fixture
- Cleans up leftover ZMQ contexts and child processes after all tests
- Lists remaining threads for debugging
3. CI Improvements:
- Apply timeout to ALL Python versions on Linux (not just 3.13)
- Increased timeout to 180s for better reliability
- Added process cleanup (pkill) on timeout
4. Dependencies:
- Added psutil>=5.9.0 to test dependencies for process management
Root cause: Python 3.9/3.13 are more sensitive to cleanup timing during
interpreter shutdown. ZMQ's default LINGER=-1 was blocking exit, and
atexit handlers were unreliable for cleanup.
This should resolve the 'all tests pass but CI hangs' issue.
