Use LitServe instead of FastAPI (enhancement)

[Youho99]

I just came across this LitServe repository, which promises 2x performance over FastAPI.

LitServe is designed by the creators of pytorch-lightning, and is specifically designed for deploying AI models.

It might be interesting to test this to compare performance for SimpleAI.

[lhenault]

Hello! Thanks for flagging this project I wasn't aware of. It sounds interesting and I'll try it soon.

I think it makes sense to keep an eye on alternatives to FastAPI, but:

• Looking at the benchmark I'm not really convinced by their claim: the only relevant graph is the second one, where FastAPI has 6 requests per second versus 8 for LitServe on CPU (the "optimized" version doesn't seem fair as they should do the same with FastAPI then), and 45 vs 57 with a GPU. So it's more about 50% speed up in a benchmark with vested interests and no idea of statistical significance or other nice things. So I'm a bit cautious about this (although it's promising and a decent increase of performance potentially).
• For LLM use cases the underlying framework is rarely the bottleneck
• As you've been experiencing it yourself, there is more than raw speed to consider and the FastAPI ecosystem is convenient and reliable, for things like logging, CORS, middlewares, community, etc... There is a reason why FastAPI is a benchmark that other projects try to challenge regularly.

Cheers!

[Youho99]

I mentioned this project in fact more to know if for SimpleAI, it could be an advantage or not.

From what you mention, FastAPi seems to remain the best solution, in the absence of significant performance gains.

Maybe when the library has been tested by the community, its use will become clearer and indispensable.

[Youho99]

You can move this issue as a discussion