Automating creation of new VM and pushing code to server

Question

Automating creating of new VM and pushing code to server

Hi all,

I am writing a integration test for a client-server application. To simulate the production server setup, and to update the code, I set up VMs on my computer. The application runs on a Windows server, so I have to use systems compatible with Windows.

STEP 1: Set up VM properly:

1. Download the OVA file from MS
2. create a new VM from the OVA file
3. reset the mac id
4. set a static dns ip address and a VM ip address
5. open up the port 22
6. install bitvise ssh server
7. transfer public keys to the VM
8. install python 3.5.2
9. update pip
10. install virtualenv systemwide

STEP 2: Deploy code to VM:

1. create a requirements file using pip using: pip freeze -r requirements
2. zip the entire repo using powershell using: powershell.exe -nologo -noprofile -command "& { Add-Type -A 'System.IO.Compression.FileSystem'; [IO.Compression.ZipFile]::ExtractToDirectory('atu.zip', 'atu'); }"
3. using pscp I transfer the files to the VM using: pscp ./atu.zip [email protected]:/C:/Users/ipython_user/Desktop
4. unzip the files again using powershell using: powershell.exe -nologo -noprofile -command "& { Add-Type -A 'System.IO.Compression.FileSystem'; [IO.Compression.ZipFile]::ExtractToDirectory('atu.zip', 'atu'); }"
5. delete the virtualenv folder called env in the folder using plink
6. create a fresh env folder using using virtualenv: virtualenv env
7. using the requirements.txt document, I install the requirements again by direcatly using the pip installed in the virtualenv above using: /path/to/env/Scripts/pip.exe install -r /path/to/requirements.txt
8. run the file with the python distribution installed in the virtualenv using: /path/to/python file_name.py

Because I was trying to execute relatively complex commands, which in turn had multiple quotes (double and single), spaces etc, I had to spend quite a bit of time just debbuging how to run these commands over the network. I have automated most of the deployment part(STEP 2) using a combination of subprocess.call, plink, and pscp. However, I haven't yet automated setting up the VM (STEP 1).

I am guessing, the above set of actions are not just something I am doing, and probably others have done this as well.

Currently, I am using low level tools to acheive this. While this works for setting up Windows VM, it wont probably work for Linux based VMs or if I change python versions use a different language etc etc.

I have also searched a bit and came across this book: https://www.safaribooksonline.com/library/view/test-driven-development-with/9781449365141/ch09.html

where the author goes into how to deploy to a linux server. However, it seems that he is essentially doing what I am doing by directly running commands on the linux server, just through the fabric wrapper. Because ssh doesn't work as well on Windows, I am guessing, I will have to get back to the nitty-grittys of deployment with fabric as well.

Question:

1. Is there a better way to setup the VM, and deploy code to the VMs, while abstracting away the difference between different operating systems, python versions, tools etc?

2. I have also heard of a git based workflow. Is that better than directly pushing the code to the server? Why so?

Answer 1

Instead of using custom scripts which it seems that you are using, I would consider using open-source tools, similar to what Tensibai has suggested in his comments.

I would further split your two Tasks a bit more differently, such that Step 1 would be to just "Set up the VM", and Step 2 would be to "Provision the VM with required programs and configuration".

My 2-cents would be to use tools such as Packer and Ansible.

You would use (for example), the VirtualBox Packer builder, to stand up your VM, which would handle points 1-4 of your Step 1, and then use the Ansible-local provisioner, or something to this effect, to run a playbook, which would handle the rest of Step 1, and all of Step 2.

---

The above suggestion would make your life a lot easier, and would be able to abstract different OS's as well as you'd just write a different Packer template and Ansible task. With installing different Python versions / Tools, you'll just need to update the Ansible role / have several different roles with the specific tools.

There are of course, other tools such as Vagrant, Puppet, and Chef, which can handle what you'd like to achieve, so it's up to you find your preferred tool.

---

Mini breakdown of Ansible modules you can use:

• Open up port: win_firewal_rule_module
• Installing bitvise and python: win_chocolatey
• Setting up pip and virtual envs: pip
• Transferring public keys / requirements.txt etc: win_copy

Etc. The full list of Ansible Windows Modules can be found here.

Answer 2

I use The Foreman for this. It leverages the fog vsphere module to interact with your vSphere server and also works with Google, Microsoft, and Amazon's clouds and other hypervisor systems as well if needed or you need to migrate at some point in the future.

This software stack is backed by Red Hat and re-branded as Red Hat Satellite 6 if you wish to use their licensed and supported version and is specifically designed for management of the Lifecycle of machines (both physical and virtual) and it can also manage Docker containers.

Upon deployment of your VM, you have the following options for configuring your newly deployed VM:

• Kickstart script
• Puppet
• Ansible
• Chef
• SaltStack

As a python shop, you will most likely be comfortable with either SaltStack or Ansible and would want to be sure to install The Foreman with support for your chosen configuration management software.

This software comes with a REST API that can be leveraged to trigger vm deployments automatically from a build pipeline (Eg, Jenkins, Bamboo, TravisCI, CircleCI, etc) or you could have the VM trigger testing after configuration is complete (or both).