[misc] feat: support fsspec (gs:// , s3://) sources in copy_to_local

[dkondoetsy]

Support GCS and S3 in copy_to_local via fsspec.

copy_to_local handled local + hdfs:// only. So configs that point data.train_files / model paths at gs:// (or other fsspec object stores) are treated as local and failed to resolve.

Add fsspec-backed fetching:

• is_fsspec_path(): detects a remote fsspec scheme (gs://, s3://, ...) and is kept distinct from is_non_local()
  (hdfs://, fetched by hdfs_io) and from local paths / HF model ids. is_non_local() is unchanged

• copy_local_path_from_hdfs() now also fetches fsspec paths; _fetch_remote() dispatches hdfs:// -> hdfs_io.copy,
  otherwise it uses fsspec's fs.get (file or dir), reusing the existing md5 cache + filelock + directory-record machinery.

• fsspec is imported lazily with a clear error naming the backend to install (gcsfs for gs://, s3fs for s3://); added fsspec to requirements.txt.

Tests (tests/utils/test_fs_on_cpu.py) cover is_fsspec_path classification, a copy_to_local fetch via the always-available in-memory fsspec backend (no network), and unchanged local-path passthrough.

Why not use fsspec for hdfs?

verl's hdfs_io.copy is tuned to a cluster's HDFS CLI/env (Kerberos auth, HADOOP_HOME/CLASSPATH, libhdfs). fsspec's HDFS goes through pyarrow's HadoopFileSystem (needs a JVM + classpath) or webhdfs (different endpoint/auth). Equivalence isn't guaranteed.

This PR was developed with AI assistance (Claude Code). I have reviewed every changed line and run the tests.

Checklist

• Couldn't find directly related PRs. Very tangentially related: Add S3 FSDP Checkpointing Support to Verl #1373

Test

tests/utils/test_fs_on_cpu.py (new cases, using the always-available in-memory fsspec
backend — no network/gcsfs needed):

• test_is_fsspec_path — scheme classification (gs/s3 → fsspec; hdfs/file/local/HF-id → not).
• test_copy_to_local_fetches_fsspec_file — fetch a single file to the local md5 cache.
• test_copy_to_local_local_path_passthrough — local paths returned unchanged (no regression).
• test_copy_to_local_rejects_fsspec_glob — a glob fails loud (no silent single-shard).
• test_copy_to_local_fsspec_dir_trailing_slash — memory://dir/ fetches recursively.

Run: pytest tests/utils/test_fs_on_cpu.py

API and Usage Example

No API change — same copy_to_local signature; remote schemes are now resolved.

from verl.utils.fs import copy_to_local
local = copy_to_local("gs://my-bucket/gsm8k/train.parquet")  # requires `gcsfs` installed
# e.g. data.train_files=gs://my-bucket/gsm8k/train.parquet now resolves on the worker node

Design & Code Changes

• is_fsspec_path(): detects a remote fsspec scheme (gs://, s3://, …) via regex, excluding
  hdfs:// (kept on hdfs_io) and file:///local/HF-ids. is_non_local() is left
  hdfs-only, so its other callers (rm_dataset, checkpoint managers) are unchanged.
• _fetch_remote(): dispatches hdfs:// → hdfs_io.copy, otherwise fsspec fs.get
  (file or dir). Raises on a multi-path glob match instead of silently using one shard.
  fsspec imported lazily with a clear "install gcsfs/s3fs" error.
• copy_local_path_from_hdfs(): guard broadened to is_non_local(src) or is_fsspec_path(src),
  reusing the existing md5 cache + filelock + directory-record machinery. Trailing slash is
  stripped for fsspec dir URLs.
• fsspec added to requirements.tx

Checklist Before Submitting

• Searched for similar PRs: https://github.com/verl-project/verl/pulls?q=is%3Apr+copy_to_local+fsspec+gcs — none found; main still has is_non_local → hdfs-only. Not a duplicate.
• Read the Contribute Guide.
• Apply pre-commit checks: pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always
• Add / Update the documentation. No user-facing change.
• Add unit or end-to-end test(s) to the CI workflow to cover all the code. If not feasible, explain why: ...
• Once your PR is ready for CI, send a message in the ci-request channel in the verl Slack workspace. (If not accessible, please try the Feishu group (飞书群).)
• If your PR is related to the recipe submodule, please also update the reference to the submodule commit via git submodule update --remote or cd recipe && git pull origin main.

[gemini-code-assist]

Code Review

This pull request adds support for remote object-store URLs (such as gs:// and s3://) using fsspec. It updates verl/utils/fs.py to detect fsspec paths, fetch remote files/directories, and handle trailing slashes, while adding comprehensive unit tests. The feedback suggests wrapping both the fsspec import and the get_fs_token_paths call in a try-except block to ensure that missing backend errors (like gcsfs or s3fs) are properly caught and reported with a helpful error message.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[CLAassistant]

All committers have signed the CLA.

[dkondoetsy]

@eric-haibin-lin, @tongyx361 when you have a chance.