Highlights5
- Hybrid retrieval: a FAISS and Neo4j layer cut p95 query latency 60% while holding 95% recall.
- Latency reduction: Redis caching and SQL query rewrites took API latency from 198ms to 20ms, nearly 90%, verified under simulated peak load.
- ETL pipeline: an idempotent pipeline with Postgres constraints and validation checks raised dataset accuracy to 95% across 10,000+ faculty profiles ingested a day.
- Production APIs: delivered 20+ REST APIs for the NSF-funded Knowledge Alliance tool on .NET 8, Dapper and MediatR, secured with JWT and queued via AWS SQS.
- Recommendations and search: a FastAPI pgvector embedding service surfaced collaborator recommendations at 85% top-k precision, and an n-gram ranking model lifted search relevance 15% over baseline.
More detail
All 9 points
Retrieval
- Stood up a FastAPI embedding microservice over Postgres/pgvector for faculty-profile similarity, surfacing collaborator recommendations at 85% top-k precision.
- Lifted search relevance 15% over baseline with a bigram and n-gram ranking model over cleaned faculty profile text.
- Tuned the pgvector index against exact search rather than by feel, trading recall for latency deliberately and recording where the curve bends.
- Chose the FAISS index by measurement: a flat index is exact and does not survive the corpus growing, so the switch to a partitioned index came with its recall cost stated.
Data pipeline
- Made the ETL idempotent on a natural key so a re-run after a partial failure updates rather than duplicates, which is what let the pipeline be retried without a cleanup script.
- Put the data-quality gates in Postgres rather than in the loader: uniqueness, not-null and range constraints, so bad rows fail at the boundary instead of being discovered in a report.
Platform APIs
- Delivered 20+ production REST APIs for NSF-funded Knowledge Alliance tool using .NET 8 + Dapper + MediatR clean architecture, secured with JWT and queued via AWS SQS.
- Used MediatR pipeline behaviours for validation, logging and error shaping, so a new endpoint inherits the cross-cutting rules instead of reimplementing them, with long work pushed to SQS to keep the API responsive.
Performance and load
- Wrote pytest unit + Locust load suites against FastAPI endpoints, establishing throughput and latency SLOs prior to release.
Tools
- Python
- PostgreSQL
- Pandas
- FAISS
- FastAPI
- .NET 8
- Dapper
- MediatR
- AWS SQS
- Redis
- Neo4j
- Docker
- Git
- REST APIs
- Locust