4

We have a pre-receive hook script in a bash shell script that enforces commits with messages having the right ticket number in a pre-defined format. We also have a separate python script that can be called in the pre-receive hook bash script to call PMD to statically analyze source code in a branch that developers push to the target remote branch.

Prior to now, we have been using the pre-receive bash script for validating our development process with respect to our Gitlab server issue tickets.

I have recently attempted to call the python script in the bash script in such a way that our ticketing and milestone validations are done before calling up PMD to expose source code that does not meet the standard. This did not work in that order, so I decided to call the python script to run PMD before the ticket validations. This also resulted in only the python script running alone without calling the rest of the bash script code.

I will appreciate any help. Please kindly find the source code of the pre-receive bash script for when I tried calling python/PMD first. Kindly point me to how I can call the python last after the ticket validations.

#!/bin/bash
#
# pre-receive hook for Commit Check
#
COMPANY_EMAIL="mycorp.org"

readonly PROGNAME=$(basename $0)
readonly PROGDIR=$(readlink -m $(dirname $0))
IS_MERGE=0

check_single_commit()
{
  COMMIT_CHECK_STATUS=1
    echo "Repo >> $REPOSITORY_BASENAME"

    if [[ "$COMMIT_MESSAGE" == "Merge branch"* ]] || [[ "$COMMIT_MESSAGE" == "WIP MI"* ]]; then
      COMMIT_CHECK_STATUS=0
      IS_MERGE=1
    else
    workFlowResult=`java -jar -Dspring.config.location=/home/gitlab/gitlab_custom_hooks/application.properties /home/gitlab/gitlab_custom_hooks/gitlab-tool.jar -prercv "$COMMIT_AUTHOR" "$COMMIT_MESSAGE" "$REPOSITORY_BASENAME"`
    echo "COMMIT_AUTHOR=$COMMIT_AUTHOR, COMMIT_MESSAGE=$COMMIT_MESSAGE, REPOSITORY_BASE=$REPOSITORY_BASENAME"

      echo " >>>>>>>>>>>>>>>>> $workFlowResult >>>>>>>>>>>>>>>>>" >&2
    if [[ "$workFlowResult" == *"PRE_RECEIVE_OK"* ]]; then
      echo " >>>>>>>>>>>>>>>>> $workFlowResult >>>>>>>>>>>>>>>>>" >&2
      COMMIT_CHECK_STATUS=0
    fi

    fi
}

check_all_commits()
{
  REVISIONS=$(git rev-list $OLD_REVISION..$NEW_REVISION)
  IFS='\n' read -ra LIST_OF_REVISIONS <<< "$REVISIONS"
if [ $(git rev-parse --is-bare-repository) = true ]
then
    REPOSITORY_BASENAME=$(basename "$PWD")
else
    REPOSITORY_BASENAME=$(basename $(readlink -nf "$PWD"/..))
fi
    echo REPOSITORY_BASENAME is $REPOSITORY_BASENAME
    REPOSITORY_BASENAME=$(basename "$PWD")
    REPOSITORY_BASENAME=${REPOSITORY_BASENAME%.git}

  for rid in "${!LIST_OF_REVISIONS[@]}"; do
    REVISION=${LIST_OF_REVISIONS[rid]}
    COMMIT_MESSAGE=$(git cat-file commit $REVISION | sed '1,/^$/d')
    COMMIT_AUTHOR=$(git cat-file commit $REVISION | grep committer | sed 's/^.* \([^@ ]\+@[^ ]\+\) \?.*$/\1/' | sed 's/<//' | sed 's/>//' | sed 's/@$COMPANY_EMAIL//')
    check_single_commit

    if [ "$COMMIT_CHECK_STATUS" != "0" ]; then
      echo "Commit validation failed for commit $REVISION" >&2
      exit 1
    fi

  done
}



echo "Checking the coding style, coding standards and look out for potential issues"

if python /home/gitlab/gitlab_custom_hooks/pmd-policy.py; then
   echo "PMD_CHECK_OK"
   PMD_CHECK_STATUS=0

  # Get custom commit message format
    while read OLD_REVISION NEW_REVISION REFNAME ; do
        check_all_commits
    done



    exit 0

else

  echo "PMD_CHECK_FAILED: Review your Code and fix the issues"
  exit 1

fi

Below is the code for the python script that calls PMD last commit to a branch at the Gitlab server.

#!/usr/bin/env python

import subprocess
import sys
import tempfile
import shutil
import os
import errno


pmd = '/home/gitlab/gitlab_custom_hooks/pmd-bin-5.5.4/bin/run.sh'

# implementing check_output for python < 2.7
if not hasattr(subprocess, 'check_output'):
    def check_output(*popenargs, **kwargs):
        if 'stdout' in kwargs:
            raise ValueError('stdout argument not allowed, it will be overridden.')
        process = subprocess.Popen(stdout=subprocess.PIPE, *popenargs, **kwargs)
        output, unused_err = process.communicate()
        retcode = process.poll()
        if retcode:
            cmd = kwargs.get("args")
            if cmd is None:
                cmd = popenargs[0]
            er = subprocess.CalledProcessError(retcode, cmd)
            er.output = output
            raise er
        return output
    subprocess.check_output = check_output


# helper for calling executables
def call(*args, **kwargs):
    return subprocess.check_output(*args, **kwargs).strip()


# helper for calling git
def call_git(cmd, *args, **kwargs):
    return call(['git'] + cmd, *args, **kwargs)


# get all new commits from stdin
def get_commits():
    commits = {}
    for line in sys.stdin:
        old, new, ref = line.strip().split(' ')
        if old == '0000000000000000000000000000000000000000':
            old = '4b825dc642cb6eb9a060e54bf8d69288fbee4904'

        if ref not in commits:
            commits[ref] = []
        commits[ref].append({
            'old': old,
            'new': new,
            'files': get_changed_files(old, new)
            })

    return commits


# get a list of changed files between to commits
def get_changed_files(old, new):
    return call_git(['diff', '--name-only', old, new]).split('\n')


# get filemode, object type (blob,tree,commit), hash for the given file at the
# given commit
def get_change_type(commit, filename):
    return call_git(['ls-tree', commit, filename]).split('\t')[0].split(' ')


commits = get_commits()

# use the latest file commit only
print "Cleaning up file list..."

files = {}
count = 0
for ref, data in commits.iteritems():
    files[ref] = {}
    for commit in data:
        for filename in commit['files']:
            if not filename.lower().endswith('.java'): continue
            files[ref][filename] = get_change_type(commit['new'], filename)
    count += len(files[ref])

print "%d Files to check in %d branches" % (count, len(files))

# create temporary dir and save a copy of the new files
tempdir = tempfile.mkdtemp('git_hook')
for ref, files in files.iteritems():
    for filename, data in files.iteritems():
        dname = os.path.dirname(filename)
        bname = os.path.basename(filename)
        try:
            os.makedirs(os.path.join(tempdir, dname))
        except OSError, exc:
            if exc.errno == errno.EEXIST:  # directory exists already
                pass
            else:
                raise

        with open(os.path.join(tempdir, dname, bname), 'w') as fp:
            fp.write(call_git(['cat-file', data[1], data[2]]))

try:
    # call checkstyle and/or pmd and print output
    print call([pmd, 'pmd', '-d', tempdir, '-f', 'text', '-R', 'rulesets/java/basic.xml,rulesets/java/unusedcode.xml,rulesets/java/imports.xml,rulesets/java/strings.xml,rulesets/java/braces.xml,rulesets/java/clone.xml,rulesets/java/design.xml,rulesets/java/clone.xml,rulesets/java/finalizers.xml,rulesets/java/junit.xml,rulesets/java/migrating.xml,rulesets/java/optimizations.xml,rulesets/java/strictexception.xml,rulesets/java/sunsecure.xml,rulesets/java/typeresolution.xml'])
    print "SUCCESS"
    sys.exit(0)
except subprocess.CalledProcessError, ex:
    print ex.output  # print checkstyle and/or pmd messages
    exit(1)
finally:
    # remove temporary directory
    shutil.rmtree(tempdir)

closed as too broad by 030, Pierre.Vriens, user2640621, Dan Cornilescu, Tensibai Oct 18 '17 at 10:08

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 2
    It's hard to correlate your description of the events with the code. Can you show some logs - how were the scripts invoked, what actually happened, what were you expecting? – Dan Cornilescu May 6 '17 at 2:34
  • @DanCornilescu, it is difficult posting the logs from PMD, as it is copious. The git hook pre-receive is called whenever the Git server (Gitlab in this case) receives a push from the client. It is a standard script that is provided for development teams to customize to suit their workflows. You can fail or allow a push depending on pre-defined conditions. The pre-receive hook, in turn, calls the python script. – sage May 6 '17 at 21:34
  • I mean the significant messages from the two scripts, showing how they relate to each other, like "PMD_CHECK_OK" or "PMD_CHECK_FAILED: Review your Code and fix the issues". Or how did the python script exit? Success/failure? etc. You need to be able to debug your scripts and understand what they do to understand how the end-to-end system works and why. Especially when they're not running on your own servers - post mortem logs is all you get - you need to figure out exactly what happened, you messages must tell you that. – Dan Cornilescu May 6 '17 at 22:17
  • Isn't what you're after just moving the while loop before the if python ? – Tensibai May 9 '17 at 7:54
  • @Tensibai, thank you very much for your reply. I have moved it before the if python statement and I have also moved it after. In each case, it seems that the git branch files are consumed by the first script that processes it. This might require a deeper understanding of how git works under the hood. The only option I think is viable in this instance is to move the Gitlab ticket validation code from the calling pre-receive bash script to the python script while the pre-receive script will only be a wrapper for invoking the python script. What do you think? – sage May 10 '17 at 9:28