TASK_CONTEXT_API_ROUTING_ERROR.md

# Task Context: HuggingFace API Routing Error

## Current Work: Debugging Agent Display Failure on HuggingFace Deployment

### Critical Error Summary

The SAAP platform deployed on HuggingFace is experiencing complete failure of agent display functionality. The frontend loads successfully, but agents cannot be retrieved from the backend.

**Error Pattern:**
```
GET /api/v1/agents HTTP/1.1" 500 Internal Server Error
RuntimeError: Not found (from spa_static_files.py line 27)
```

**Success Indicators:**
- Platform initializes successfully ✅
- 7 agents loaded into database ✅
- colossus connection successful ✅
- OpenRouter client initialized ✅
- Frontend static assets serve correctly ✅
- API endpoint fails catastrophically ❌

### Key Technical Concepts

**Architecture:**
- **Backend:** Python FastAPI with hybrid agent support (colossus + OpenRouter)
- **Frontend:** Vue.js SPA with Vite build system
- **Deployment:** Docker containerized on HuggingFace Spaces
- **Database:** SQLite with SQLAlchemy ORM
- **Agents:** 7 base templates (Jane, John, Lara, Theo, Justus, Leon, Luna Alesi)

**Critical Components:**
1. `backend/main.py` - FastAPI application initialization and middleware ordering
2. `backend/spa_static_files.py` - Custom static file handler (line 27 = error source)
3. `backend/api/agents.py` - Agent API endpoints (should handle /api/v1/agents)
4. `backend/services/agent_manager_hybrid.py` - Hybrid agent manager service

**Pydantic Warning (Secondary Issue):**
```
UserWarning: Valid config keys have changed in V2:
* 'schema_extra' has been renamed to 'json_schema_extra'
```

### Relevant Files and Code

**Error Source Location:**
- File: `backend/spa_static_files.py`
- Line: 27
- Method: `get_response`
- Error: `raise RuntimeError("Not found")`

**Stack Trace Analysis:**
```
Starlette Middleware Chain:
1. proxy_headers.py
2. CORS middleware  
3. Exception middleware
4. Routing (fails here - goes to StaticFiles instead of API route)
5. StaticFiles handler (spa_static_files.py line 105)
6. get_response method (spa_static_files.py line 27) → RuntimeError
```

**Request Flow (ACTUAL):**
```
Frontend → GET /api/v1/agents 
  → Starlette routing 
  → StaticFiles middleware catches route (WRONG!)
  → spa_static_files.py:get_response() 
  → RuntimeError("Not found")
```

**Request Flow (EXPECTED):**
```
Frontend → GET /api/v1/agents 
  → Starlette routing 
  → FastAPI agent router 
  → agent_api.py:get_agents()
  → Return agent list JSON
```

### Problem Solved Thus Far

**Initialization Success:**
- Database connection established
- 7 agents registered from templates
- colossus server connection verified (14.06s test successful)
- OpenRouter client initialized with free model fallback
- Hybrid agent manager operational

**Configuration Verified:**
- Environment: `OPENROUTER_API_KEY` warning present (expected in free mode)
- Database: `sqlite:///./saap_production.db` created successfully
- Providers: Both colossus (✅) and OpenRouter (✅) initialized

### Pending Tasks and Next Steps

**Immediate Investigation Required:**

1. **Examine spa_static_files.py implementation**
   ```python
   # Need to review line 27 in get_response method
   # Understand why it's intercepting API routes
   ```

2. **Check main.py middleware ordering**
   ```python
   # Verify API router mounted BEFORE static file handler
   # Pattern should be:
   app.include_router(api_router)  # FIRST
   app.mount("/", SPAStaticFiles())  # LAST (catch-all)
   ```

3. **Verify route registration**
   ```python
   # Confirm /api/v1/agents endpoint properly registered
   # Check FastAPI route debugging output
   ```

4. **Fix Pydantic V2 deprecation warning**
   ```python
   # Replace 'schema_extra' with 'json_schema_extra' in Pydantic models
   ```

**Direct Quote from Error Logs:**

> "ich bekomme folgende Fehler in huggingface und die Aagenten werden aus diesem Grund nicht angezeigt:
> 
> INFO: 10.20.38.235:38000 - "GET /api/v1/agents HTTP/1.1" 500 Internal Server Error
> ERROR: Exception in ASGI application
> Traceback (most recent call last):
>   File "/app/spa_static_files.py", line 27, in get_response
>     raise RuntimeError("Not found")
> RuntimeError: Not found"

**Where Development Left Off:**

The user encountered this error in production HuggingFace deployment. The platform starts successfully, all backend services initialize correctly, but the critical `/api/v1/agents` endpoint fails with a routing error. The static file handler is incorrectly intercepting API requests that should be handled by FastAPI routers.

**Root Cause Hypothesis:**

The middleware/route mounting order in `backend/main.py` likely has the SPA static file handler registered before (or with higher priority than) the API routes, causing all requests to be evaluated by the static file handler first. When `/api/v1/agents` doesn't match a static file, the handler raises `RuntimeError("Not found")` instead of allowing the request to continue to FastAPI routers.

**Required Fix Pattern:**

```python
# backend/main.py - CORRECT PATTERN

app = FastAPI()

# 1. Register API routes FIRST (specific routes)
app.include_router(agent_router, prefix="/api/v1")
app.include_router(hybrid_router, prefix="/api/v1")

# 2. Mount static files LAST (catch-all fallback)
app.mount("/", SPAStaticFiles(directory="frontend/dist"))
```

**Test Verification Needed:**
- Confirm `/api/v1/agents` returns JSON agent list (not 500)
- Verify static assets still load (/, /assets/*)
- Check agent display in frontend UI
- Validate both API and SPA work without conflicts

**Technical Context - File Structure:**
```
backend/
├── main.py (← FIX NEEDED: middleware ordering)
├── spa_static_files.py (← line 27 raising error)
├── api/
│   ├── agents.py (← should handle /api/v1/agents)
│   └── hybrid_endpoints.py
└── services/
    └── agent_manager_hybrid.py
```

**Environment Details:**
- Python: 3.11
- FastAPI: Latest (from requirements.txt)
- Uvicorn: Running on 0.0.0.0:7860
- Platform: HuggingFace Spaces
- Deployment: Docker container

**Error Frequency:** 
Continuous - every frontend refresh attempts to fetch agents and fails. The error repeats every few seconds as the frontend retries the request.

## Debugging Methodology to Apply

Per Rules/debugging-workflows.md - Four-Phase Framework:

**Phase 1: Root Cause Investigation** (IN PROGRESS)
- ✅ Read error messages - RuntimeError from spa_static_files.py line 27
- ✅ Stack trace analyzed - Starlette routing → StaticFiles → error
- ⚠️ Need to read spa_static_files.py and main.py to confirm hypothesis
- ⚠️ Need to trace middleware ordering

**Phase 2: Pattern Analysis** (PENDING)
- Find working FastAPI + SPA patterns
- Compare against current implementation  
- Identify middleware ordering differences

**Phase 3: Hypothesis Testing** (PENDING)
- Hypothesis: Middleware ordering causes route interception
- Test: Reorder middleware in main.py
- Verify: API routes accessible, SPA still works

**Phase 4: Implementation** (PENDING)
- Create failing test for /api/v1/agents endpoint
- Fix middleware ordering
- Verify all tests pass
- Deploy and validate on HuggingFace

## Integration Requirements

**Security Considerations:**
- API endpoints must not expose sensitive data
- CORS properly configured for HuggingFace domain
- Error messages sanitized (no stack traces to frontend)

**Code Quality Standards:**
- Fix Pydantic V2 deprecation warnings
- Follow FastAPI best practices for middleware ordering
- Ensure clear separation: API routes vs static file serving

**Testing Requirements:**
- Integration test: GET /api/v1/agents returns 200 + agent list
- Integration test: GET / returns 200 + SPA HTML
- Integration test: GET /assets/* returns 200 + static assets
- Coverage target: ≥75% (MCP/tool-heavy architecture acceptable)

## Critical Files to Examine

1. **backend/spa_static_files.py** - Line 27 error source
2. **backend/main.py** - Middleware mounting order
3. **backend/api/agents.py** - Agent endpoint implementation
4. **backend/services/agent_manager_hybrid.py** - Agent retrieval logic

## Success Criteria

- [ ] `/api/v1/agents` returns 200 status with valid JSON
- [ ] Agents display in frontend UI
- [ ] Static assets continue loading correctly
- [ ] No RuntimeError exceptions
- [ ] Pydantic deprecation warning resolved
- [ ] Route precedence clearly documented