[Performance] AI microservice handles requests synchronously - long LLM inference blocks the Go HTTP handler goroutine

[anshul23102]

Problem

If the Go HTTP handler calls the AI provider synchronously and waits for the LLM response before returning, long inference times (10-30 seconds for large outputs) hold the handler goroutine for the full duration. Under load, this exhausts the goroutine pool and causes request queuing:

func handleAIRequest(w http.ResponseWriter, r *http.Request) {
    // Blocks for 10-30 seconds:
    response, err := openai.CreateCompletion(r.Context(), openai.CompletionRequest{
        Model:     "gpt-4o",
        Prompt:    prompt,
        MaxTokens: 2000,
    })
    if err != nil {
        http.Error(w, err.Error(), 500)
        return
    }
    json.NewEncoder(w).Encode(response)
}

Proposed Fix

Use Server-Sent Events (SSE) with streaming to return tokens progressively:

func handleAIRequest(w http.ResponseWriter, r *http.Request) {
    w.Header().Set("Content-Type", "text/event-stream")
    w.Header().Set("Cache-Control", "no-cache")
    w.Header().Set("Connection", "keep-alive")

    flusher, ok := w.(http.Flusher)
    if !ok {
        http.Error(w, "Streaming unsupported", http.StatusInternalServerError)
        return
    }

    stream, err := openaiClient.CreateCompletionStream(r.Context(), openai.CompletionRequest{
        Model:  "gpt-4o",
        Prompt: prompt,
        Stream: true,
    })
    if err != nil {
        fmt.Fprintf(w, "data: {"error": "%s"}\n\n", err.Error())
        return
    }
    defer stream.Close()

    for {
        response, err := stream.Recv()
        if errors.Is(err, io.EOF) {
            fmt.Fprintf(w, "data: [DONE]\n\n")
            flusher.Flush()
            return
        }
        fmt.Fprintf(w, "data: %s\n\n", response.Choices[0].Text)
        flusher.Flush()
    }
}

This frees the goroutine from blocking, reduces perceived latency (first token in < 1 second), and allows the server to handle far more concurrent requests.

Acceptance Criteria

• LLM responses streamed via SSE, not buffered and returned all at once
• First token returned to client in under 2 seconds
• Request context cancellation (client disconnect) stops the upstream LLM stream
• Concurrent request capacity at least 10x higher than the synchronous baseline (measured with wrk or hey)

[anshul23102]

Hi @AnkanMisra, I'd like to work on this issue as part of GSSoC 2026. I'll refactor the AI microservice to handle LLM inference asynchronously (via goroutines/worker queue) so long-running requests no longer block the Go HTTP handler. Could you please assign it to me?

/assign
/claim

[pisum-sativum]

@AnkanMisra I want to work on this project as a part of Gssoc 2026 kindly assign me this issue.Pls add the gssoc tags.

[github-actions]

Assigned @pisum-sativum.