pipeline parallel fails at high load

[yunoJ]

Hi team,

I've been stress-testing Sarathi and pipeline parallel doesn't pass the assert statement

sarathi-serve/sarathi/core/datatypes/sequence.py

Line 90 in f17201a

assert num_tokens > 0

during the middle of the execution.

The CLI arguments I'm passing to sarathi/benchmark/main.py are:

python main.py  \
    --synthetic_request_generator_config_num_requests 1000   \
    --parallel_config_tensor_parallel_size 1   \
    --parallel_config_pipeline_parallel_size 4    \
    --interval_generator_config_type POISSON \
    --poisson_request_interval_generator_config_qps 100000

I think there might be a bug in the scheduler. Would you mind helping me out?

[AgrawalAmey]

@yunoJ we are aware of some regressions in pipeline parallelism. i will port the fix from our internal repository sometime soon. we have some other optimizations which are tied to those fixes, which we need to merge first.