LLM routing & AI platforms
Connecting model selection, inference, observability, and private infrastructure for procurement workflows.
Architecture overview of the implemented components; the routing policy and gateway smoke tests have separate verification scopes.
The problem
Procurement workflows need different model capabilities, structured responses, and controlled escalation. A shared model platform also needs a consistent API, useful telemetry, and restricted access to its applications and databases.
What I built
- A Python routing framework using LangGraph, LiteLLM, and Pydantic, with 10 procurement-agent policies across three model tiers. Selection checks capabilities and context limits; structured-output validation and bounded escalation handle failures.
- A centralized LiteLLM gateway with Docker Compose, PostgreSQL, and Caddy HTTPS. Seven model endpoints passed smoke tests spanning text generation, Thai-language embeddings, and image-based OCR.
- Langfuse and OpenTelemetry integration linking workflow and model-call traces with token usage, latency, and cost.
Operating the platform
I provisioned PostgreSQL, Redis, Neo4j, and Qdrant with individual developer credentials and restricted administrative access. Self-hosted NetBird provides private DNS and trusted HTTPS, with verified developer and administrator boundaries and closure of public application and database ports.
Prometheus and Grafana provide server, gateway, and service monitoring through two dashboards containing 26 panels.
Verification and trade-offs
The integration checkpoint recorded 173 passing tests, with pytest, Ruff, mypy, and uv covering routing, validation, failure handling, and telemetry privacy. During a controlled 30-second tracing-service outage, a real gateway inference completed before the tracing service resumed.
The outage check establishes inference isolation, not guaranteed trace delivery. Endpoint checks were smoke tests, not sustained-load benchmarks. The three-tier routing profile was verified separately from the live endpoint tests; it does not establish procurement-task quality for those models.
Engineering takeaway
Routing rules, model capability validation, and deployment verification solve different problems. Keeping their evidence separate makes platform behavior easier to reason about and prevents a successful smoke test from becoming an unsupported quality claim.