artifact: Intermittent download skip on Windows with periodic jobs using crons and multiple tasks sharing same bucket (regression from #27398)

[javiersanchez-alt]

Nomad version
Nomad v2.0.3
Also reproduced on v1.11.2 and v1.11.3. Last known good version: v1.11.1.

Operating system and Environment details

OS: Windows Server
Driver: raw_exec
Job type: periodic batch (using crons scheduler)
Related issue / suspected regression source: #27398

Issue
Since Nomad 1.11.2, periodic jobs on Windows intermittently skip the artifact download phase entirely. The task starts directly without going through Received, Task Setup, or Downloading Artifacts events, causing it to execute with missing files and fail with exit code 1.
The regression is consistently reproduced when a job has two or more tasks each with their own artifact block, both downloading from the same S3/MinIO bucket, scheduled via crons. In those cases the client appears to skip the download phase on subsequent periodic invocations, likely because the artifact caching logic introduced in #27398 evaluates files left on disk from a prior run as already present.
The fix in #27398 (which resolved the renameat ... Access is denied issue on Windows) appears to have introduced this new behavior.

Reproduction steps

Run a periodic job using crons with the raw_exec driver on a Windows node.
Define two or more tasks, each with its own artifact block pointing to the same S3/MinIO bucket.
Let the job trigger multiple times via its cron schedule (typically fails within 3–5 runs).
Observe that some allocs skip the download phase entirely and start the task with missing files.

Expected Result
Every alloc should emit the full lifecycle event sequence:
Received → Task Setup → Downloading Artifacts → Started

Actual Result
Some allocs skip directly to Started with no Downloading Artifacts event, then fail because expected files are absent:

Failing alloc — Downloading Artifacts event missing

Time Type Description
2026-06-17T19:51:54Z Not Restarting Exceeded allowed attempts 3
2026-06-17T19:51:54Z Terminated Exit Code: 1
2026-06-17T19:51:54Z Started Task started by client
2026-06-17T19:51:37Z Restarting Task restarting in 17.1s
2026-06-17T19:51:37Z Terminated Exit Code: 1
2026-06-17T19:51:17Z Started Task started by client

Successful alloc — correct behavior

Time Type Description
2026-06-17T19:51:09Z Terminated Exit Code: 0
2026-06-17T19:50:48Z Started Task started by client
2026-06-17T19:50:46Z Downloading Artifacts Client is downloading artifacts
2026-06-17T19:50:45Z Task Setup Building Task Directory
2026-06-17T19:50:45Z Received Task received by client

Job file (if appropriate)
hcljob "example" {
datacenters = ["dc1"]
type = "batch"

periodic {
crons = ["*/5 * * * *"]
prohibit_overlap = true
}

group "grp" {

task "app" {
  driver = "raw_exec"

  artifact {
    source      = "s3::https://minio.example.com/bucket/app.zip"
    destination = "local/"
    options {
      aws_access_key_id     = "..."
      aws_access_key_secret = "..."
    }
  }

  artifact {
    source      = "s3::https://minio.example.com/bucket/config.yml"
    destination = "local/app/etc"
    options {
      aws_access_key_id     = "..."
      aws_access_key_secret = "..."
      archive               = false
    }
  }

  config {
    command = "cmd.exe"
    args    = ["/c", "cd /d local\\app && app.exe -f config.yml"]
  }
}

task "sidecar" {
  driver = "raw_exec"

  artifact {
    source      = "s3::https://minio.example.com/bucket/sidecar.zip"
    destination = "local/"
    options {
      aws_access_key_id     = "..."
      aws_access_key_secret = "..."
    }
  }

  artifact {
    source      = "s3::https://minio.example.com/bucket/sidecar-config.yml"
    destination = "local/sidecar/etc"
    options {
      aws_access_key_id     = "..."
      aws_access_key_secret = "..."
      archive               = false
    }
  }

  config {
    command = "cmd.exe"
    args    = ["/c", "cd /d local\\sidecar && sidecar.exe -f sidecar-config.yml"]
  }
}

}
}

Both tasks download from the same MinIO bucket (minio.example.com/bucket). The intermittent skip is observed on the second or subsequent cron-triggered runs when artifact files from a prior alloc remain on disk.

[mismithhisler]

Hi @javiersanchez-alt thanks for the great bug report. I'll point out that there was another artifact PR that was merged right after the v1.11.1 release (so landed in 1.11.2) but didn't appear in the changelog.

#26892

This introduced some larger artifact changes. I'm going to take a look at this and the PR you linked and see if there is anything obvious here.

[mismithhisler]

Hmm now that I look at this closer, its really odd the task is not emitting an event for downloading the artifact... That indicates it's not even running the code from both of the linked PR's.

We show the last 10 events for a task, its completely possible after 3 restarts that we would drop the first couple events that would include downloading the artifact. I'm also going to assume you maybe just left off a couple events which is why there are not 10?

[javiersanchez-alt]

Hi @mismithhisler, thanks for your fast response.

In this case, the logs I sent were just an example cause it happens completely at random. There could be 3 executions and the last 2 fail, or they could all be completely fine. There is no exact pattern for when the artifact download starts failing, since it's a silent error with no logs.

[javiersanchez-alt]

Here's an example of 2 allocations, same HCL definition and one success and the other with failed status:

The success task downloaded an .yml artifact correctly on the alloc folder without any errors

This is an extract of the hcl definitions that doesn't work properly (This appears three times in the HCL configuration across different tasks. Same hcl definition):

And here are the Logs with no errors:

Jun 18, '26 19:45:23 +0000 | Restarting | Task restarting in 17s -- | -- | -- Jun 18, '26 19:45:23 +0000 | Terminated | Exit Code: 1 Jun 18, '26 19:45:23 +0000 | Started | Task started by client Jun 18, '26 19:45:07 +0000 | Restarting | Task restarting in 16s Jun 18, '26 19:45:06 +0000 | Terminated | Exit Code: 1 Jun 18, '26 19:45:06 +0000 | Started | Task started by client Jun 18, '26 19:45:01 +0000 | Downloading Artifacts | Client is downloading artifacts Jun 18, '26 19:45:00 +0000 | Task Setup | Building Task Directory Jun 18, '26 19:45:00 +0000 | Received | Task received by client

The Allocation that worked without errors was because it successfully downloaded the file to the destination.

Version v1.11.2

And with v1.11.1 version there's no error:

[javiersanchez-alt]

[mismithhisler]

My guess is in geometry-proxy there is no storage.yml in the allocation with the bug?

[javiersanchez-alt]

Exactly. In the allocation with the bug, storage.yml is not downloaded into the /etc folder. If you look at the allocation starting with 52cc in the previous image, the /etc folder was created because the destination for storage.yml specifies it, and it downloads correctly. Meanwhile, in the allocation with the bug (ee28), there is no /etc folder because the download didn't execute, even though it did download other components, leaving the downloads half-done.

[mismithhisler]

Out of curiosity do you have any client.artifact parameters set? Specifically disable_artifact_inspection?

Also, does this only happen when you have 2 artifacts with overlapping destinations? I noticed that one artifact had a destination of local/ and the other had local/app/etc.

[javiersanchez-alt]

Nope, here's the HCL Definition without URL's and without credentials:

job "mtiler-proxy" {
  datacenters = ["onpremises"]
  type        = "batch"

  meta {
    mine        = "cerrejon"
    environment = "qa"
    release     = "v0.1.0"
  }

  periodic {
    crons            = ["*/5 * * * *"]
    prohibit_overlap = true
  }

  group "grp-mtiler-proxy" {
    task "polygons" {
      driver = "raw_exec"

      artifact {
        source      = "s3::XX/bin/mine_tiler/v0.1.0/geometry-proxy_v0.1.0.zip"
        destination = "local/"
        options {
          aws_access_key_id     = "XX"
          aws_access_key_secret = "XX"
        }
      }
      artifact {
        source      = "s3::XX/bin/mine_tiler/v0.1.0/geo/configs/storage.yml"
        destination = "local/geometry-proxy/etc"
        options {
          aws_access_key_id     = "XX"
          aws_access_key_secret = "XX"
          archive               = false
        }
      }
      artifact {
        source      = "s3::XX/bin/mine_tiler/v0.1.0/geo/configs/polygons.json"
        destination = "local/geometry-proxy"
        options {
          aws_access_key_id     = "XX"
          aws_access_key_secret = "XX"
          archive               = false
        }
      }
      config {
        command = "cmd.exe"
        args = [
          "/c",
          "cd /d local\\geometry-proxy && geometry-proxy.exe -f polygons.json"
        ]
      }
      resources {
        cpu    = 200
        memory = 256
      }
      kill_timeout = "300s"
    }

    task "sequence-daily" {
      driver = "raw_exec"

      artifact {
        source      = "s3::XX/bin/mine_tiler/v0.1.0/geometry-proxy_v0.1.0.zip"
        destination = "local/"
        options {
          aws_access_key_id     = "XX"
          aws_access_key_secret = "XX"
        }
      }
      artifact {
        source      = "s3::XX/bin/mine_tiler/v0.1.0/geo/configs/storage.yml"
        destination = "local/geometry-proxy/etc"
        options {
          aws_access_key_id     = "XX"
          aws_access_key_secret = "XX"
          archive               = false
        }
      }
      artifact {
        source      = "s3::XX/bin/mine_tiler/v0.1.0/geo/configs/sequence-daily.json"
        destination = "local/geometry-proxy"
        options {
          aws_access_key_id     = "XX"
          aws_access_key_secret = "XX"
          archive               = false
        }
      }
      config {
        command = "cmd.exe"
        args = [
          "/c",
          "cd /d local\\geometry-proxy && geometry-proxy.exe -f sequence-daily.json"
        ]
      }
      resources {
        cpu    = 200
        memory = 256
      }
      kill_timeout = "300s"
    }

    task "sequence-weekly" {
      driver = "raw_exec"

      artifact {
        source      = "s3::XX/bin/mine_tiler/v0.1.0/geometry-proxy_v0.1.0.zip"
        destination = "local/"
        options {
          aws_access_key_id     = "XX"
          aws_access_key_secret = "XX"
        }
      }
      artifact {
        source      = "s3::XX/bin/mine_tiler/v0.1.0/geo/configs/storage.yml"
        destination = "local/geometry-proxy/etc"
        options {
          aws_access_key_id     = "XX"
          aws_access_key_secret = "XX"
          archive               = false
        }
      }
      artifact {
        source      = "s3::XX/bin/mine_tiler/v0.1.0/geo/configs/sequence-weekly.json"
        destination = "local/geometry-proxy"
        options {
          aws_access_key_id     = "XX"
          aws_access_key_secret = "XX"
          archive               = false
        }
      }
      config {
        command = "cmd.exe"
        args = [
          "/c",
          "cd /d local\\geometry-proxy && geometry-proxy.exe -f sequence-weekly.json"
        ]
      }
      resources {
        cpu    = 200
        memory = 256
      }
      kill_timeout = "300s"
    }


   
  }
}

NOTE: The curious thing is that it only happens with storage.yml. The problem doesn't occur with the .zip, which is also fetched from the same URL multiple times.
Note: I know we need to optimize the HCL so it only downloads once, which would probably save us from this whole issue, but we have many HCL definitions that could be compromised

[mismithhisler]

@javiersanchez-alt I think i've been able to reproduce this error using 3 artifacts with similar destinations to what you have posted (I was using 2 yesterday and having trouble with getting a repro). At the moment I think this is an issue with how we are merging downloaded artifacts from their tmp folder to the final destination. At first I wrote some tests for the merge code to simulate this issue, but those all passed. However what I missed in my test was the fact that we are downloading and merging all these artifacts in parallel.

I am now seeing various errors like:

failed to download artifact "http://172.31.84.26:8080/storage.txt": RemoveAll download-and-use\local\app: The directory is not empty.

and

failed to download artifact "http://172.31.84.26:8080/storage.txt": RemoveAll download-and-use\local\app: The process cannot access the file because it is being used by another process.

Some of these tasks retry and end up succeeding while others fail completely, and these events overwritten. My initial thought here is that we can keep the concurrent download, but should serialize the merge into the final destination. I'm going to chat with the team and get back to you.

[mismithhisler]

@javiersanchez-alt I do think a workaround at the moment is to download each artifact into it's own directory, and then adjust your args to point to the appropriate file. Although I am not sure if this is feasible with the storage.yml as that doesn't appear to have a flag for it's location.