Add HdfsRemoteLogIO.from_config and register hdfs remote logging scheme

[jason810496]

Part of #70265 (related: #67056).

Why

#67056 decoupled remote logging from the hardcoded branches in airflow_local_settings.py: core and
the Task SDK now resolve the handler via ProvidersManager dispatch on the
[logging] remote_base_log_folder URL scheme, instantiating the provider class through a no-arg
from_config() classmethod. This issue migrates the hdfs scheme.

What

• Add HdfsRemoteLogIO.from_config() in
  providers/apache/hdfs/src/airflow/providers/apache/hdfs/log/hdfs_task_handler.py, mirroring
  the legacy branch:
  

  airflow/airflow-core/src/airflow/config_templates/airflow_local_settings.py

  Lines 287 to 302 in 104ad12

  	elif remote_base_log_folder.startswith("hdfs://"):
  	from airflow.providers.apache.hdfs.log.hdfs_task_handler import HdfsRemoteLogIO
  	_default_conn_name_from("airflow.providers.apache.hdfs.hooks.webhdfs", "WebHDFSHook")
  	REMOTE_TASK_LOG = HdfsRemoteLogIO(
  	**cast(
  	"dict[str, Any]",
  	{
  	"base_log_folder": BASE_LOG_FOLDER,
  	"remote_base": urlsplit(remote_base_log_folder).path,
  	"delete_local_copy": delete_local_copy,
  	}
  	| _io_kwargs,
  	)
  	)

  
  — including the [logging] remote_task_handler_kwargs IO-kwargs merge and expanduser on
  base_log_folder, so behavior is unchanged for existing configs.
• Register the hdfs scheme under a remote-logging: section in
  providers/apache/hdfs/provider.yaml and mirror it in
  providers/apache/hdfs/src/airflow/providers/apache/hdfs/get_provider_info.py.
• Add tests mirroring TestS3RemoteLogIOFromConfig in
  providers/amazon/tests/unit/amazon/aws/log/test_s3_task_handler.py.
• Verify end to end with a real system test: set up the actual backend yourself (a real
  service or a local equivalent), run a task with remote logging enabled, and confirm logs are
  uploaded and read back through the new dispatch path. Include the setup and verification
  results in the PR description.

Notes

The legacy branch passes urlsplit(remote_base_log_folder).path as remote_base — the
hdfs://host part is stripped and only the path is kept. Mirror this in from_config.

Reference

Merged examples to follow: #69817 (s3), #69816 (cloudwatch). If from_config raises on a bad
config, the shared factory falls back to the legacy path, so this is not a breaking change.

The legacy branch derives a default connection id from WebHDFSHook.default_conn_name; handle the
connection id the same way the merged amazon PRs do (the hook reads
[logging] remote_log_conn_id itself).

[Vamsi-klu]

@jason810496 I'd like to take this one, thanks!

For the end-to-end verification I'll run a single-node Hadoop container locally with WebHDFS enabled, so no external setup is needed. I'll follow the merged amazon examples (#69817, #69816) and mirror the urlsplit(...).path handling noted in the issue.