1

I have an AWS Glue job written in Python that I would like to perform pyunit tests on. However, from reading and experimenting on my own, there seem to be quite a few dependencies on the Glue service, making this task very impractical if not impossible. My only thought would be to separate the non glue dependent functions in the job into a separate module to perform unit testing on.

Is unit testing with AWS Glue possible, and if so, what are the best practices for doing so?

2
  • 1
    Checkout this solution with codepipeline. aws.amazon.com/blogs/big-data/…
    – user17183
    Sep 19 '19 at 19:26
  • @user17183 Please add this as an answer by explaining the details rather than only adding a link
    – 030
    Dec 19 '19 at 13:57
2

Checkout this solution with codepipeline. https://aws.amazon.com/blogs/big-data/implement-continuous-integration-and-delivery-of-serverless-aws-glue-etl-applications-using-aws-developer-tools/

1
  • 1
    The link looks good. Could you summarize the important elements in the answer to prevent that the information will be lost if the link is deprecated.
    – 030
    Dec 20 '19 at 8:39
1

If you are unit testing, you should be mocking external dependencies like Spark and AWS Glue. Look into Python mocking frameworks like unittest.mock, monkeypatch and pytest-mock. Your unit tests should just test your code and how it interacts with Spark and AWS Glue. Your unit tests should not test Spark and AWS Glue functionality. Do that in your component and integration testing.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.