posts/2026-02-26-argocd-aws-codecommit-credentials - ArgoCD never invokes the configured Git credential helper

[adalsa91]

Hi,

I was trying to follow the approach described in your article about using AWS CodeCommit with ArgoCD via IRSA and the AWS CodeCommit credential helper:

posts/2026-02-26-argocd-aws-codecommit-credentials

I managed to get all of the underlying pieces working:

• IRSA is correctly configured on argocd-repo-server
• The pod can assume the IAM role
• AWS CLI is available inside the repo-server container
• aws codecommit credential-helper works
• git ls-remote https://git-codecommit.<region>.amazonaws.com/v1/repos/<repo> works successfully from inside the repo-server container using the configured helper

I also mounted the Git configuration at /etc/gitconfig (following your separate article about Git configuration and credential caching in ArgoCD), and verified that Git is correctly loading the credential helper configuration.

However, when ArgoCD itself tries to access the repository, it still fails with:

failed to list refs: authentication required:
<NotAuthorizedException>
<Message>SPNEGO token required</Message>
</NotAuthorizedException>

To troubleshoot this, I modified the credential helper command to write to a temporary log file whenever it is invoked. Manual Git operations trigger the helper as expected, but ArgoCD repository validation and application synchronization never invoke it.

This makes me suspect that either:

1. ArgoCD is not using the system Git configuration for repository access in this scenario.
2. Additional configuration is required but not documented in the article.
3. The behavior depends on a specific ArgoCD version.

Could you clarify whether you have successfully validated the full end-to-end flow with ArgoCD actually syncing applications from CodeCommit using this method? If so, would you be able to share the missing configuration required to make ArgoCD invoke the credential helper?

Thanks for the article and for any additional details you can provide.

[simlarsen]

Thanks for the unusually careful write-up. Confirming that aws codecommit credential-helper works when you run git manually inside the repo-server container, but ArgoCD's own repo validation/sync never calls it, is exactly the right thing to test — and it points straight at the root cause. You did the hard part; here's what's going on.

TL;DR

There are two compounding reasons, and the article's Method 3 was wrong on both:

1. Wrong path. ArgoCD runs git with HOME=/dev/null, so a per-user ~/.gitconfig (i.e. /home/argocd/.gitconfig) is never read. Your manual git worked because your shell had a normal $HOME; ArgoCD's git does not.
2. ArgoCD doesn't authenticate through credential.helper at all. It feeds credentials from your repo/repo-creds Secrets through git's askpass mechanism. With no matching Secret, it supplies nothing, and git fails non-interactively — which is the authentication required ... SPNEGO token required you're seeing (that's CodeCommit's response to an unauthenticated request).

Why the helper is never invoked

(1) HOME=/dev/null. ArgoCD's repo-server runs the real git binary and sets HOME=/dev/null (plus GIT_TERMINAL_PROMPT=false) in runCmdOutput (util/git/client.go). Git resolves global/user config relative to $HOME (git-config FILES), so it looks for /dev/null/.gitconfig, which can't exist. ArgoCD's docs confirm it: "Argo CD runs Git with the HOME environment variable set to /dev/null. As a result, global Git configuration is not supported." (git_configuration). So a helper at /home/argocd/.gitconfig is silently ignored — the system path /etc/gitconfig is the only one honored.

(2) ArgoCD uses askpass, not on-disk helpers. For an HTTPS repo, Repository.GetGitCreds does one of two things:

• Matching repo/repo-creds Secret → it returns HTTPSCreds and injects GIT_ASKPASS=argocd + a nonce (util/askpass/server.go); git calls the argocd-git-ask-pass helper, which returns the Secret's username/password over a unix socket.
• No matching Secret (your IRSA-only case) → it falls through to git.NopCreds{}, whose Environ() returns nothing — no GIT_ASKPASS, no credentials. With HOME=/dev/null and GIT_TERMINAL_PROMPT=false, git can't prompt and fails. CodeCommit replies with the SPNEGO/authentication-required error.

One clarification, since it's a common misconception: git consults credential.helpers first, and askpass is only the fallback for whatever the helpers didn't supply (git credential.c, gitcredentials(7)). So it's not that "askpass beats the helper" — it's that ArgoCD never configures any credential.helper, and your helper lived at a path git never reads.

Is /home/argocd/.gitconfig simply the wrong path — and does it work if corrected?

Partly. The correct path is /etc/gitconfig (mount with subPath: gitconfig). ArgoCD does not set GIT_CONFIG_NOSYSTEM, so /etc/gitconfig is read regardless of HOME, and because helpers run before askpass, a helper there can be reached — but only for a repo with no matching repo-creds Secret (the NopCreds path, where there's no askpass to compete). Fixing the path is necessary but not sufficient; it also needs:

• the aws CLI present in the repo-server image (the stock image doesn't have it, so helper = !aws codecommit credential-helper $@ silently fails);
• IRSA on the argocd-repo-server service account with codecommit:GitPull;
• no repo/repo-creds Secret for that URL (a matching Secret routes through askpass and the helper is never consulted).

Be aware this approach is undocumented/unsupported upstream (it relies on the NopCreds fall-through, an implementation detail), and CodeCommit helper tokens are short-lived (~15 min). See argo-cd #3291, #4072, #9284.

What actually works (pick one)

1. Refresh a repo-creds Secret on a schedule (recommended, no static long-lived creds). A CronJob (~every 10 min) on an IRSA service account with codecommit:GitPull runs aws codecommit credential-helper get and writes the short-lived username/password into a repo-creds-labeled Secret. ArgoCD re-reads it on update and authenticates via its normal askpass path — the community-confirmed pattern (#9284).
2. Corrected /etc/gitconfig helper with the three requirements above — works, but unsupported and more brittle.
3. AWS-managed EKS ArgoCD capability (native IRSA CodeCommit) — cleanest if you're on the managed add-on.
4. Static service-specific HTTPS creds or an SSH key in IAM, stored in a repo-creds Secret — fully supported and simplest (Methods 1 and 2 in the article).

Fixed

The article has been updated in #150 (now merged): Method 3 corrects the path to /etc/gitconfig, drops the "no further config needed" claim and spells out the real prerequisites, a new Method 4 documents the credential-refresh CronJob, and a Troubleshooting entry covers exactly the SPNEGO token required failure you hit. Thanks again for the precise report.

[simlarsen]

(Auto-closed by the merge of #150.) Resolved as far as the article goes — happy to reopen if Method 4 (the credential-refresh CronJob) doesn't get you a clean sync against CodeCommit.

[adalsa91]

Thanks a lot for the detailed explanation and for taking the time to investigate this. I really appreciate it, and it's great to see that you've already updated the article.

Just to clarify one thing: before reading your reply I had already switched from ~/.gitconfig to /etc/gitconfig (after finding the ArgoCD documentation about HOME=/dev/null), and I also verified that there were no repo or repo-creds Secrets present for the repository.

In that setup:

• git config --list --show-origin correctly shows the configuration from /etc/gitconfig.
• git ls-remote works perfectly when run manually inside the argocd-repo-server container.
• However, when ArgoCD itself tries to validate or sync the repository, my credential helper is never invoked.

So I still seem to be hitting something different, but your explanation about the askpass flow and NopCreds was very helpful and helped me understand ArgoCD's internals much better.

In any case, I think I'll move forward with the CronJob approach you suggested. It actually looks like a cleaner and more robust solution than relying on the credential helper.

Thanks again for the thorough explanation and for updating the article!