Progress reporting

[michaeljones]

Thank you for the project.

Would you be open to the idea of having some kind of progress reporting behaviour for the copy? I don't have a clear sense of the best approach to progress reporting in this situation. It might end up being that the user provides a closure which is called after each file is created or with some kind event description to say if it is a file or directory and then the user would be responsible for managing that information.

I guess if the user walks the whole source directory before the copy then they'd have an idea of how many files and directories were present and so could infer some progress by keeping a tally of progress closure calls vs the total count.

I'm not sure if this makes sense. A colleague has pointed out that Window has struggled with progress reporting in this situation for a long time so it might not be that easy. It certainly would struggle in a situation with lots of small files and one or two very large files. In which case the initial directory walk would need to gather all the paths & file sizes and the progress reporting would have to report the path of each file copied.

[michaeljones]

The specific use case I have at the moment, is to copy a large number of small files so I've gone with the file count approach on what is currently a private fork of the code.

I've added a file_count function to the public API:

pub fn file_count<P: AsRef<Path>>(source: P) -> Result<usize, std::io::Error> {
    let mut entry_count = 0;

    for _entry in walkdir::WalkDir::new(source)
        .into_iter()
        .filter_map(|e| e.ok())
    {
        entry_count += 1;
    }

    Ok(entry_count)
}

Which counts the files in the same way as the copy logic approaches them.

Then I've added a copy_dir_with_progress function:

/// Copy a directory from `source` to `dest`, creating `dest`, with minimal options.
pub fn copy_dir_with_progress<P: AsRef<Path>, Q: AsRef<Path>>(
    source: P,
    dest: Q,
    progress_callback: impl Fn(usize),
) -> Result<(), std::io::Error> {
    CopyBuilder {
        source: source.as_ref().to_path_buf(),
        destination: dest.as_ref().to_path_buf(),
        overwrite_all: false,
        overwrite_if_newer: false,
        overwrite_if_size_differs: false,
        exclude_filters: vec![],
        include_filters: vec![],
    }
    .run_with_progress(progress_callback)
}

Where the callback function is just given an index of the current file being copied and on the client side I convert that to a percentage using the total count reported by the file_count function and then report that to the user. It seems to work. It doesn't seem completely ideal to walk the whole source directory first but it turns out to be manageable in my situation from what I can tell even on Windows.

[woelper]

Hi! I think this is a nice feature request!
The only issue I see is that this costs a bit performance, especially for large folders / many files as this would require getting the number of files beforehand. If this is optional I can see this being useful!

[woelper]

Hi! I added a pr here:
#15

I tried to build something that feels ergonomic - once you add a .with_progress() to the builder, the number of files will be automatically generated. Here is some example code:

    CopyBuilder::new(
        "from",
        "to",
    )
    .with_progress(|all, done| {
        info!("copied {done}/{all}");
    })
    .run()
    .unwrap();

[michaeljones]

Thank you for engaging with the concept :) I really appreciate it. I think your API looks clean and useful. Having it provide the total count to the callback is smart!

I'll try to find the time in the next day or so to test this with our code base if that would be useful.

[woelper]

I updated the code to not require send and sync. Does this work better for you?

[woelper]

I released a new version today on crates.io. Thank you again for the request.