Tim Raymond

Modules are the future of package distribution in Go. They’re in preview mode now, but in 1.13, they’ll be on by default. For projects that only use public dependencies, they work without a hitch. I’ve found that some additional configuration is necessary if you want a smooth experience using private modules. What follows are my experiences setting up both my local environment to fetch private Go modules and a Docker build environment.

The root of the issue with fetching private dependencies is that you need to authenticate to the source of those repositories. Since modules are written using https:// URLs, git will attempt to fetch from, for example, https://github.com//. Ordinarily, git would prompt you for a username and password (if it doesn’t, set the environment variable GIT_TERMINAL_PROMPT=1 ). If this works for you, great! I keep all my passwords in a password manager, so I don’t want to look them up all the time if I can help it.

Git has something called “credential helpers” (see gitcredentials(7))which can provide these usernames and passwords to Git for you. If you’re on macOS, the osxkeychain helper is usually included with Git’s installation. Using this, you’ll get a macOS system prompt for your credentials and they’ll be stored within Keychain once you enter them. You shouldn’t have to enter them again as long as your macOS Keychain is unlocked. Further information about theosxkeychain helper can be found in Github’s documentation.

If you like the idea of credential helpers, but you’re not on macOS, a credential helper probably exists for your platform. If you need to do something custom, you can even write your own using the credentials API.

Credential helpers may not be a workable solution on all hosts. In that situation, the simplest way I have found to fetch private dependencies is to create a “Personal Access Token” in Github with scopes that allow it to read private repositories. Once you have your Personal Access Token, you can create a .netrc file in your home directory formatted like so:

1
2
3

machine github.com
       login ${YOUR_USERNAME}
       password ${YOUR_PERSONAL_ACCESS_TOKEN}

Once you’ve done this, git is smart enough to know how to use this to fetch from all your private repositories.

The major downside of this approach is that it requires having a secret hanging out on disk for anyone with access to read it. Instead, we’d like to use SSH to fetch dependencies, since we can rely on the SSH agent to protect our secrets instead. Unfortunately, we need more configuration to do this.

Since Go modules currently has no ability to recognize that a dependency fetched from, for example, https://github.com/ can also be fetched from ssh://git@github.com/, we need to configure Git to perform that substitution for us. The quickest way to do this is git config --global url."ssh://git@github.com/".insteadOf "https://github.com/". This will create a file at $HOME/.gitconfig (or modify an existing one). If you’d like this to apply to all users, substitute --global with --system.

This will mostly work (remember to start your SSH agent with eval $(ssh-agent)) but you may end up with spurious failures like I did where you end up with Permission denied (publickey) . This doesn’t make any sense at first glance— after all, the agent should have all the keys. If you look closer, you’ll notice that some of your private repositories fetched successfully. What’s going on?

I noticed this behavior when using Docker’s new BuildKit engine to build images. One of the best parts about it is that it allows you to forward an SSH agent into the build process specifically for fetching private repositories. Using an SSH agent within a RUN directive is accomplished like:

1

RUN --mount=type=ssh go mod download

This suffers from the same spurious failures. The reason is that Go will fetch modules concurrently and open multiple SSH connections to Github at the same time. Github appears to have MaxStartups configured on their SSH daemon (or something equivalent), which will start to refuse simultaneous unauthenticated connections when there are many of them. That’s us — we’re making a ton of connections to fetch dependencies.

SSH provides connection multiplexing so that multiple connections to the same host end up going over the same socket. We can use this to tame the thundering herd of Git processes, but it requires some more configuration.

Outside of Docker, this is most cleanly accomplished with the following added to your $HOME/.ssh/config :

1
2
3
4

Host github.com
  ControlMaster auto
  ControlPersist 3600
  ControlPath ~/.ssh/%r@%h:%p

If you’d like multiplexing on all your SSH connections, you can also change Host github.com to Host * .

Within Docker, the situation is slightly more complicated. Since that SSH configuration is specific to your user running on the host machine, SSH running in the Docker builder won’t have any awareness of it. Fortunately, git provides us the environment variable GIT_SSH_COMMAND which we can use to tweak the ssh command that git , and consequently go (since it shells out to git ) will use to fetch our Go modules.

The complete RUN command will look like:

1
2
3

RUN --mount=type=ssh mkdir -p /var/ssh && \
    GIT_SSH_COMMAND="ssh -o \"ControlMaster auto\" -o \"ControlPersist 300\" -o \"ControlPath /var/ssh/%r@%h:%p\"" \
    go mod download

Ugly, I know. Alternatively you can write out the above config to /etc/ssh/ssh_config as another RUN step, but I opted against that since this is the only command that needs this configuration.

The following is a more complete Dockerfile which shows fetching keys from Github, installing dependencies like Git, and also strategic placement of COPY commands so that you don’t have to unnecessarily fetch go modules when you haven’t added any new dependencies from a previous build:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

# syntax=docker/dockerfile:experimental
#
# ^^ That's the important part for the new BuildKit functionalityFROM golang:1.12-alpine3.8# Moving outside of $GOPATH forces modules on without having to set
# ENVs
WORKDIR /src
RUN apk add --no-cache --update \
        openssh-client \
        git \
        ca-certificates# Force fetching modules over SSH
RUN git config --system url."ssh://git@github.com/".insteadOf "https://github.com/"# Fetch github's SSH host keys and compare them to the published
# ones at:
#
# https://help.github.com/en/articles/githubs-ssh-key-fingerprints
RUN set -euo pipefail && \
    mkdir -p -m 0600 ~/.ssh && \
    ssh-keyscan -t rsa github.com >> ~/.ssh/known_hosts && \
    ssh-keygen -F github.com -l -E sha256 \
        | grep -q "SHA256:nThbg6kXUpJWGl7E1IGOCspRomTxdCARLviKw6E5SY8"# Add go.mod and go.sum first to maximize caching
COPY ./go.mod ./go.sum ./
RUN --mount=type=ssh mkdir -p /var/ssh && \
    GIT_SSH_COMMAND="ssh -o \"ControlMaster auto\" -o \"ControlPersist 300\" -o \"ControlPath /var/ssh/%r@%h:%p\"" \
    go mod download # Copy in the project
COPY . . # Compile...
RUN GOOS=linux GOARCH=amd64 go install \
      -ldflags='-w -s -extldflags "-static"' \
      github.com/<yourorg>/<yourproject>/...CMD ["/go/bin/your_binary"]

These are just a few ways that you can authenticate to sources of private dependencies. Ultimately, I believe that cleaner solutions will come from using GOPROXY using something like Athens which can be configured to do all this for you with you only having to set GOPROXYon those hosts that need to fetch private modules. Until those solutions are production-ready, I hope that the solutions provided above might help you to start using Go modules in your own organizations where private dependencies are the norm.