3

We are setting up our source and build system in AWS. we will have two persistent VMs hosting source and artifactory. We have a bunch of compilers (a few Gigs) we want available via NFS, is this something that should be hosted in artifactory?

I'm looking for the 'best practice' answer. I can't find much info on the thought behind this so any links would be appreciated.

3
  • What is the use case for the compilers? Will they be downloaded onto a VM when it is created? Are the compilers ever updated?
    – Preston Martin
    Feb 2 '18 at 16:54
  • Compilers? Javac?
    – 030
    Feb 2 '18 at 18:07
  • We are NFS mounting them, not downloading them. The compilers are rarely updated, but will be occasionally(every several months). They are CCs for c/c++, altera, etc. (we do all server/embedded/firmware work).
    – scott
    Feb 2 '18 at 20:43
2

In many cases compilers are used as-is, already packaged by 3rd parties. When changing the structure of the build systems changing compilers usually translates into switching to a different version, but which is also pre-packaged. If this is your use case the artifact repository offers a single advantage: availability. Even if the provider decides (for whatever reason) to discontinue the version - you still have your copy in the respository.

But if the compilers receive any updates without version changes or if you do any sort of customisation - even if it's just building them without touching any code - you want to store those particular versions in the artifactory to be able to reliably obtain them at any time.

Side note: how exactly you use the compilers can influence the performance of your builds, especially in very large projects with build times which can take hours. You'd have to analyze your builds' structure and/or usage patterns to figure out if it's worth pursuing optimisations and, if so, in which direction(s).

Directly invoking compilers from NFS is convenient - very little configurations required on the build servers. But performance of massive builds can be very sensitive to the NFS performance, of both the NFS server itself and of the network connecting it to the build server(s). Some artifactories may have direct NFS support. You'd have to measure the impact of switching to such service, in some cases it may still be better to use a dedicated NFS server instead. Or maybe switching to VM images that contain the compilers locally installed.

1
  • We will be using JFrog, which I am pretty sure has NFS support if I am not mistaken.
    – scott
    Feb 5 '18 at 16:40
3

There are a few main reasons why you would use Artifactory (or any other binary repository manager) over a traditional file storage (in your case NFS).

  1. The ability to manage your artifact versions
  2. A central location to access artifacts across infrastructure (which you have accomplished with NFS)
  3. The ability to download/recreate previous versions of the artifacts (you could also use a git/svn repository for this, but it would be contain MUCH more irrelevant metadata about the binary artifacts than would be useful to a user)
  4. You want to control who/what has permissions to access the stored artifacts

If none of these are applicable in your use-case, I would say that Artifactory would be overkill and the effort to create and maintain it would not out way the value you would receive from it. Although I have used it in the past, and it is a great tool for what it aims to accomplish :)

1

One solution that might fit your scenario are environment modules. Given that you only need the compilers on Linux, and have access via a network share (nfs/Samba). To load a toolchain into your terminal, just type

module load gcc

and let autocompletion fill in the version number. It is basically Python’s virtualenv for any Linux program/toolchain/SDK. I have used it extensively and can warmly recommend this tool.

Each compiler would have their own folder which is subdivided by version which allows you to have direct access to all versions. Configuration is defined server-side with PATH and other environment variables. User-side, a bash script has to be installed into users’ .bashrc

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.