Why We Moved from Serverless to Edge for AI Workflows

Reducing cold starts and managing API latency when orchestrating multiple LLM calls simultaneously.

When building applications that run sequential LLM processes, every millisecond counts. Standard serverless functions (like AWS Lambda or Vercel Serverless) can suffer from cold starts that delay responses by several seconds. At Siyol Technologies, we deploy latency-sensitive routes to Edge runtimes to ensure fast, responsive performance.

1. Serverless Cold Starts vs. Edge Runtime

Serverless functions require booting a complete container runtime (such as Node.js) when a request comes in, which can take anywhere from 1.5 to 5 seconds. Edge runtimes run on lightweight V8 isolates, starting up in under 50ms and executing globally close to the user.

For AI tasks, Edge functions excel at orchestrating multiple API requests (e.g. calling OpenAI and updating a database) without adding latency from server cold starts.

2. Setting Up an Edge Route for Streaming Response

Below is an example of a Next.js Edge API route (`/api/chat/stream`) configured for the Edge runtime. It streams responses to the user in real-time, reducing initial page load times.

// Route completions over Server-Sent Events (SSE) using Next.js Edge Runtime
export const runtime = "edge";

export async function POST(req: Request) {
  const { prompt } = await req.json();
  const response = await fetch("https://api.openai.com/v1/chat/completions", {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      "Authorization": `Bearer ${process.env.OPENAI_API_KEY}`
    },
    body: JSON.stringify({
      model: "gpt-4o-mini",
      messages: [{ role: "user", content: prompt }],
      stream: true
    })
  });

  return new Response(response.body, {
    headers: {
      "Content-Type": "text/event-stream",
      "Cache-Control": "no-cache"
    }
  });
}

3. Database Connections at the Edge

While Edge functions provide low latency, they present challenges for traditional database connections. Connecting directly to a relational database (like PostgreSQL) from thousands of edge nodes can quickly exhaust connection pools. This can lead to connection drops and slow query response times.

To manage database operations from the Edge, we use HTTP-based connection proxies or serverless databases (like Supabase pg-meta, Turso, or Neon). These database clients route requests over HTTP, allowing edge functions to connect securely without maintaining open sockets.

4. Edge Runtime Development Guidelines

• No Node.js Native Modules: Edge runs on V8. Libraries that rely on built-in modules like 'fs' or 'child_process' will fail.
• Strict Memory Limits: Keep function sizes small. Standard Edge functions are capped at 50MB of memory.
• Select Global DB Connections: Use HTTP-based database clients (e.g. Neon, Turso, Supabase pg-meta) to avoid database connection pool issues at the Edge.
• Short Execution Windows: Edge functions must start returning headers within 25 seconds of receiving a request.

By deploying latency-sensitive routes to Edge runtimes, we ensure that Siyol's web apps respond instantly to users globally, while keeping cloud infrastructure costs manageable.