Description
Summary ABOUT THE PROJECT We're building a multi-agent voice and chat platform that lets teams create customer-facing AI agents grounded in their own uploaded documents, test them in a sandbox, publish to production, and deploy via REST API, real-time voice sessions, or an embeddable widget. We need an expert backend engineer to own the intelligence and streaming layer — a unified engine that drives agentic AI across both text endpoints and low-latency, bidirectional voice channels. Each agent maintains its own synchronized knowledge state, prompts, and conversation history. This role is 100% backend: data ingestion, RAG engineering, prompt orchestration, and real-time voice infrastructure. No frontend dashboards or UI work. Your deliverables are clean, robust backend logic, APIs (REST/WebSockets), and streaming pipelines. TECH STACK TypeScript, Next.js (App Router), Postgres with pgvector, Drizzle ORM, Vercel AI SDK, Google Gemini (chat, embeddings, live voice), OAuth, Stripe billing. WHAT YOU'LL BUILD 1. Multimodal RAG Pipeline Engine - High-performance ingestion pipeline for PDFs, DOCX, markdown, CSV, HTML, JSON, images, audio, and video - Optimal chunking strategies (overlap-aware, sentence/paragraph boundaries) - High-quality vector embeddings and semantic search via pgvector with HNSW indexing - Fully isolated, per-agent knowledge bases 2. Real-Time Voice Streaming (WebSockets) - Low-latency backend for bidirectional voice conversations - Integration with streaming audio models (e.g. Gemini Live) using ephemeral tokens and WebSocket sessions - Interruption-aware sessions with RAG-enriched system prompts - Transcript persistence, usage metering, and sandbox vs. production behavior with API key and origin-based access control 3. Multi-Agent Isolation & Prompt Routing - Secure per-agent configuration: each user runs multiple independent agents (support, sales, onboarding), each with its own system prompt, knowledge namespace, voice settings, API credentials, and conversation logs - Routing layer that assembles the correct context and enforces complete data isolation between agents 4. Context Assembly Layer - Context-stitching engine that merges platform instructions, per-agent prompts, live conversation history, visitor/session metadata, and retrieved RAG fragments into unified payloads for chat streaming and voice inference 5. Lead Capture, Tracking & CRM Integration - Pre-chat forms, client-side event tracking SDK, server-side form submission APIs - OAuth-based CRM sync with per-agent field mapping and idempotent export - Structured payloads and background jobs to push leads and session data downstream, extensible for future qualification/scoring logic IDEAL BACKGROUND - Strong TypeScript backend experience on Next.js App Router (API routes, server-side logic) - Deep hands-on RAG experience: document parsing, chunking, embedding pipelines, pgvector search - Vercel AI SDK and Google Gemini (chat streaming, embeddings, live voice APIs) - Postgres (Neon or similar), Drizzle ORM or equivalent, solid migration discipline - Real-time audio/voice engineering: WebSockets, ephemeral token flows, low-latency session design - SaaS architecture: OAuth, API key management, rate limiting, Stripe subscriptions, webhooks - Bonus: embeddable widget APIs, CRM OAuth integrations, multimodal embeddings WHY THIS ROLE MATTERS You'll own the engine that makes AI agents accurate, fast, and production-ready across text and voice — keeping every agent's knowledge, credentials, and conversations cleanly isolated. Your work sits at the intersection of RAG, real-time streaming, and developer-facing deployment APIs.