For Engineers: Implementation Details & Stack
Engineers want specifics they can reproduce. Our production path pairs an LLM for dialogue with a tool layer, a retrieval index, and an evaluation harness that fails the build when quality drops. Channel adapters normalize web, mobile, and third-party widgets into one event schema. Sessions store compact state, not full history dumps. Compression rules keep token size sane for long DIY flows.
Retrieval uses chunked corpora with metadata filters for product line, locale, and clearance. Embedding models are chosen for short official English text typical of contractor SOPs. Rerankers run when multi-document conflicts appear. Citation IDs travel with answers so support can verify. If retrieval returns empty, the dialogue asks one ordered follow-up or escalates. Guessing is banned by default.
Tool design starts from OpenAPI-style contracts with mock servers. Product recommendation logic from the beauty agent taught us to keep business calculations outside free text. Eligibility math, inventory checks, and plan limits stay deterministic. The model only selects and explains. Timeouts, retries with jitter, and dead-letter queues cover flaky upstreams common on legacy middle tiers.
Optimization work targets hot intents first. Caching hashed query results, shrinking bloated system prompts, and moving low-stakes paths to smaller models cut cost without hurting satisfaction. Edge cases include concurrent edits on tickets, multi-tenant bleed prevention, and users who paste giant logs. Guardrails drop oversized payloads with a clear message. Human agents receive a truncated, linked artifact instead.
Local deployment patterns favor US regions close to Northern Virginia traffic. Feature flags flip languages, brands, and experimental tools per tenant. Observability hooks emit spans on retrieval, model marry, and tool calls. Engineers debug a single conversation ID across dashboards. That short path is what keeps night pages rare.