In many cases compilers are used as-is, already packaged by 3rd parties. When changing the structure of the build systems changing compilers usually translates into switching to a different version, but which is also pre-packaged. If this is your use case the artifact repository offers a single advantage: availability. Even if the provider decides (for whatever reason) to discontinue the version - you still have your copy in the respository.
But if the compilers receive any updates without version changes or if you do any sort of customisation - even if it's just building them without touching any code - you want to store those particular versions in the artifactory to be able to reliably obtain them at any time.
Side note: how exactly you use the compilers can influence the performance of your builds, especially in very large projects with build times which can take hours. You'd have to analyze your builds' structure and/or usage patterns to figure out if it's worth pursuing optimisations and, if so, in which direction(s).
Directly invoking compilers from NFS is convenient - very little configurations required on the build servers. But performance of massive builds can be very sensitive to the NFS performance, of both the NFS server itself and of the network connecting it to the build server(s). Some artifactories may have direct NFS support. You'd have to measure the impact of switching to such service, in some cases it may still be better to use a dedicated NFS server instead. Or maybe switching to VM images that contain the compilers locally installed.