So I believe I came up with a somewhat re-usable solution for this problem using devpi. It allows you to use the same Dockerfile for the testing and production image.
Devpi allows me to create my own index which can shadow my production index with versions of software that haven't been officially released.
Steps to make this work:
1. Base image from ARG