[FEAT]: support SFT as functions

[lwaekfjlk]

Description

We need to support SFT as one mode besides RL to allow warmup.

Additional Information

No response

[zhusq20]

SFT Support for OpenTinker

Add Supervised Fine-Tuning (SFT) support to the OpenTinker framework. This enables users to perform SFT on models using the same "Training as a Service" paradigm as the existing RL functionality.

Code Reusability Insights

To minimize redundancy and maintain consistency, the SFT implementation will reuse the following existing components:

• DataProto Serialization: Use serialize_dataproto and deserialize_dataproto for efficient cross-process data transfer.
• Client Infrastructure: Leverage HTTPTrainingClient and SchedulerClient for API communication and job submission.
• Scheduler Logic: Extend the existing JobSchedulerActor to manage SFT job lifecycles and GPU allocation.
• Config System: Utilize the established Hydra-based configuration patterns.
• Logging & Tracking: Hook into the existing Tracking class for uniform WandB/console logging.

Proposed Changes

Scheduler

[MODIFY] job_scheduler.py

• Extend JobType to include SFT.
• Update _launch_server to handle SFT jobs. Specifically, it should use a different server script (launch_sft_server.py) when the job type is SFT.

Server

[NEW] http_sft_server.py

• Implement SFTTrainingServerBackend which wraps verl.trainer.sft_trainer.SFTTrainer or FSDPSFTTrainer.
• Expose endpoints:
  • /api/v1/train_step: Receive a batch of data and perform one SFT update.
  • /api/v1/validate: Run validation on a batch.
  • /api/v1/save_checkpoint: Save the current model state.
  • /api/v1/health: Standard health check.

[NEW] launch_sft_server.py

• Entry point for the SFT training server.
• Handles Ray initialization (if needed) and starts the FastAPI server.
• Uses Hydra for configuration management.

Client

[NEW] sft_client.py

• A specialized client for SFT training.
• Submits the SFT job to the scheduler.
• Loads local datasets and streams batches to the training server.
• Manages the training loop and validation frequency.