0

I have a gitlab yaml I've been working on for a bit and while everything works it seems like each job takes quite a bit longer than it should. I tend to think the culprit is that npm files are fetched every time rather than sharing the node_modules cache between jobs.

Short of creating my own docker image, how can I optimize this yaml in a way that reduces the bottlenecks ?

gitlab .yaml

image: node:10.19.0 # https://hub.docker.com/_/node/
# image: node:latest

cache:
  # untracked: true
  key: my-project
  # key: ${CI_COMMIT_REF_SLUG} # per branch
  # key:
  #   files:
  #     - package-lock.json # only update cache when this file changes (not working) @jkr
  paths:
    - .npm/
    - node_modules
    - build

stages:
  - prepare # prepares builds, makes build needed for testing
  - test # uses test:build specifically @jkr
  - build
  - deploy

# before_install:

before_script:
  - npm ci --cache .npm --prefer-offline

prepare:
  stage: prepare
  needs: []
  script:
    - npm install

test:
  stage: test
  needs: [prepare]
  except:
    - schedules
  tags:
    - linux
  script:
    - npm run build:dev
    - npm run test:cicd-deps
    - npm run test:cicd # runs puppeteer tests @jkr
  artifacts:
    reports:
      junit: junit.xml
    paths:
      - coverage/

build-staging:
  stage: build
  needs: [prepare]
  only:
    - schedules
  before_script:
    - apt-get update && apt-get install -y zip
  script:
    - npm run build:staging
    - zip -r build-staging.zip build
  # cache:
  #   paths:
  #     - build
  #   <<: *global_cache
  #   policy: push
  artifacts:
    paths:
      - build-staging.zip

deploy-serv1:
  stage: deploy
  needs: [build-staging]
  tags: [linux]
  only:
    - schedules
  #   # - branches@gitlab-org/gitlab
  before_script:
    - apt-get update && apt-get install -y lftp
  script:
    # temporarily using 'verify-certificate no'
    # for more on verify-certificate @jkr: https://www.versatilewebsolutions.com/blog/2014/04/lftp-ftps-and-certificate-verification.html
    # variables do not work with 'single quotes' unless they are "'surrounded by doubles'"
    - lftp -e "set ssl:verify-certificate no; open myserver.org; user $LFTP_USERNAME $LFTP_PASSWORD; mirror --reverse --verbose build/ /var/www/domains/dev/projects/my-project/build/; bye"
  # environment:
  #   name: staging
  #   url: http://dev.myserver.org/projects/my-project/build
  when: manual
  allow_failure: true

build-production:
  stage: build
  needs: [prepare]
  only:
    - schedules
  before_script:
    - apt-get update && apt-get install -y zip
  script:
    - npm run build:staging
    - zip -r build-prod.zip build
  # cache:
  #   paths:
  #     - build
  #   <<: *global_cache
  #   policy: push
  artifacts:
    paths:
      - build-prod.zip

deploy-serv2:
  stage: deploy
  needs: [build-production]
  tags: [linux]
  only:
    - schedules
    # - master
  before_script:
    - apt-get update && apt-get install -y lftp
  script:
    - sh deploy-prod
  environment:
    name: production
    url: http://www.my-project.org
  when: manual
  allow_failure: true
3

I highly recommend using npm ci instead of npm install. The slow part of installing npm modules is figuring out everything that needs to be installed. If you commit your package-lock.json and use npm ci, it skips that part entirely and just looks at your lockfile to download what's needed. It's significantly faster.

5
  • What are your thoughts on npm install with --prefer-offline ? I was using npm ci but switched to npm i with the usage of prefer offline. As of yet I don't have an opinion on either - other that npm ci FEELS better due do the use of the keyword ci – Jacksonkr Sep 24 '20 at 15:39
  • I've never used --prefer-offline. In fact I didn't know it existed before your comment. I think the main benefit of npm ci is the fact that you know exactly what will be installed and there are no changes made to package.json. It also deletes node_modules before installing packages so you know it's a clean slate every time. I've not yet seen issues with npm ci slowness, but you could use --prefer-offline with npm ci. Here's an example from the gitlab docs: docs.gitlab.com/ee/ci/caching/#caching-nodejs-dependencies – bradym Sep 24 '20 at 18:00
  • Funny you should mention that gitlab doc - those docs are where I pulled most my current implementation from. It seems like the issue now is that compiling / extracting the cache takes longer than fetching all the npm packages. – Jacksonkr Sep 25 '20 at 12:51
  • Circling back - ultimately I deemed my prepare job to be doing double work. I took prepare out which cut my workload in half. I don't know I'd call that optimized but it's certainly much better. – Jacksonkr Sep 25 '20 at 18:51
  • 1
    Ahh, yes. I don't think I noticed you were doing both. And optimization is relative, so yes, it's optimized. :D – bradym Sep 26 '20 at 0:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.