fix: increase outer timeout to 360s to respect pytest's 300s timeout

The outer shell timeout must be larger than pytest's internal timeout (300s)
to allow pytest to handle its own timeout gracefully and perform cleanup.

Changes:
- Increased outer timeout from 180s to 360s (300s + 60s buffer)
- Made timeouts configurable via environment variables
- Added clear documentation about timeout hierarchy
- Display timeout configuration at runtime

Timeout hierarchy:
1. Individual test: 20s (markers)
2. Pytest session: 300s (pyproject.toml)
3. Outer shell: 360s (for cleanup)
4. GitHub Actions: 6 hours (default)

This prevents the outer timeout from killing pytest before it can finish
its own timeout handling, which was likely causing the hanging issues.
This commit is contained in:
Andy Lee
2025-08-08 22:48:40 -07:00
parent 60eef4b440
commit 742c9baabc

View File

@@ -265,6 +265,11 @@ jobs:
limit-access-to-actor: true
- name: Run tests with pytest
# Timeout hierarchy:
# 1. Individual test timeout: 20s (see pyproject.toml markers)
# 2. Pytest session timeout: 300s (see pyproject.toml [tool.pytest.ini_options])
# 3. Outer shell timeout: 360s (300s + 60s buffer for cleanup)
# 4. GitHub Actions job timeout: 6 hours (default)
env:
CI: true # Mark as CI environment to skip memory-intensive tests
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
@@ -349,8 +354,24 @@ jobs:
echo "🔌 Pre-test network state:"
ss -ltn 2>/dev/null | grep -E "555[0-9]|556[0-9]" || echo "No embedding server ports open"
echo "🏃 Running pytest with 180s timeout..."
timeout --preserve-status --signal=INT --kill-after=10 180 bash -c '
# Set timeouts - outer must be larger than pytest's internal timeout
# IMPORTANT: Keep PYTEST_TIMEOUT_SEC in sync with pyproject.toml [tool.pytest.ini_options] timeout
PYTEST_TIMEOUT_SEC=${PYTEST_TIMEOUT_SEC:-300} # Default 300s, matches pyproject.toml
BUFFER_SEC=${TIMEOUT_BUFFER_SEC:-60} # Buffer for cleanup after pytest timeout
OUTER_TIMEOUT_SEC=${OUTER_TIMEOUT_SEC:-$((PYTEST_TIMEOUT_SEC + BUFFER_SEC))}
echo "⏰ Timeout configuration:"
echo " - Pytest internal timeout: ${PYTEST_TIMEOUT_SEC}s (from pyproject.toml)"
echo " - Cleanup buffer: ${BUFFER_SEC}s"
echo " - Outer shell timeout: ${OUTER_TIMEOUT_SEC}s (${PYTEST_TIMEOUT_SEC}s + ${BUFFER_SEC}s buffer)"
echo " - This ensures pytest can complete its own timeout handling and cleanup"
echo "🏃 Running pytest with ${OUTER_TIMEOUT_SEC}s outer timeout..."
# Export for inner shell
export PYTEST_TIMEOUT_SEC OUTER_TIMEOUT_SEC BUFFER_SEC
timeout --preserve-status --signal=INT --kill-after=10 ${OUTER_TIMEOUT_SEC} bash -c '
echo "⏱️ Pytest starting at: $(date)"
echo "Running command: pytest tests/ -vv --maxfail=3 --tb=short --capture=no"
@@ -382,7 +403,7 @@ jobs:
echo "🔚 Timeout command exited with code: $EXIT_CODE"
if [ $EXIT_CODE -eq 124 ]; then
echo "⚠️ TIMEOUT TRIGGERED - Tests took more than 180 seconds!"
echo "⚠️ TIMEOUT TRIGGERED - Tests took more than ${OUTER_TIMEOUT_SEC} seconds!"
echo "📸 Capturing full diagnostics..."
diag