Go modules is a dependency management system which has been introduced since go 1.11. In the project I worked with, each microservice has its own repository and the common libraries reside in multiple private repositories. This setup allows us to properly design shared libraries and versioning. Obviously the first challenge we encountered was downloading dependencies from Gitlab private repositories and below listed a series of problems and solutions I’ve got.
Here is an example of the “go.mod” file I had originally:
go 1.13 require (
gitlab.com/myprivaterepo/common/go/auth v1.0.0
)
1st problem: go modules communicates with git over https, while we extensively use ssh for all communication with private repositories (and I would encourage you to do the same). From here on, I would assume you have already set up ssh certificate and use it for communication with your repositories.
Solution: we need to instruct git to ssh instead by using the following command:
git config --global url."git@gitlab.com:".insteadof="https://gitlab.com"That is equivalent with adding this to your .gitconfig:
[url "git@gitlab.com:"]
insteadOf = https://gitlab.comThe mechanism is similar to other git hosting providers such as github or bitbucket.
2nd problem: gitlab subgroup repositories are not recognised properly. The original issue is here. This happened to me because the repository I wanted to import was in the 2nd level subgroup. This is the error I got:
$ go mod download
> go: gitlab.com/myprivaterepo/common/go/auth@v1.0.0: reading gitlab.com/myprivaterepo/common/go/auth/go/auth/go.mod at revision go/auth/v1.0.0: unknown revision go/auth/v1.0.0Debugging further using go get:
$ go get -v https://gitlab.com/myprivaterepo/common/go/auth
> get "gitlab.com/myprivaterepo/common/go/auth": found meta tag get.metaImport{Prefix:"gitlab.com/myprivaterepo/common", VCS:"git", RepoRoot:"https://gitlab.com/myprivaterepo/common.git"} at //gitlab.com/myprivaterepo/common/go/auth?go-get=1
> get "gitlab.com/myprivaterepo/common/go/auth": verifying non-authoritative meta tag
> go: gitlab.com/myprivaterepo/common/go/auth@v1.0.4: reading gitlab.com/myprivaterepo/common/go/auth/go/auth/go.mod at revision go/auth/v1.0.4: unknown revision go/auth/v1.0.4Observe that gitlab reported an unexpected repo roothttps://gitlab.com/myprivaterepo/common.gitinstead ofhttps://gitlab.com/myprivaterepo/common/go/auth.git
Solution: Instruct go modules to use the right repository by adding this to the end of go.mod:
replace (
gitlab.com/myprivaterepo/common/go/auth => gitlab.com/myprivaterepo/common/go/auth.git v1.0.0
)Alternatively, you could import gitlab.com/myprivaterepo/common/go/auth.git in the go files but this approach is really ugly.
3rd problem: mysteriously, sometimes the error still occurred and would go away if I additionally set GOPRIVATE environment variable:
export GOPRIVATE="gitlab.com/myprivaterepo"I would appreciate if someone could explain this magic to me
4th problem: configure private go modules for CI pipelines. In principle, the same SSH setup I describe above could be used for CI pipelines but I found it pretty complicated to configure with the CI system we have. I experimented with using gitlab access token as well but AWS Codebuild just decided to log out each and every command including my access token which is a huge security issue.
Solution: check in vendored code. I wasn’t a fan of checking in generated code before but I think it comes down to tastes or philosophy. However this approach comes with a number of benefits:
- Newly checked out code could be built without network communication or additional setups.
- Faster build as dependencies are pre-downloaded.
I’ve followed multiple solutions on the internet but the above works best for our local development, CI pipelines and SSH communication for gitlab. It also requires minimal setups. Normally you would only encountered the 1st problem I stated. In case you happened to use gitlab, and that your private repo is under a subgroup, further patches might be required.
Originally posted at: https://techproject.io/
