1

I have an AWS Glue job written in Python that I would like to perform pyunit tests on. However, from reading and experimenting on my own, there seem to be quite a few dependencies on the Glue service, making this task very impractical if not impossible. My only thought would be to separate the non glue dependent functions in the job into a separate module to perform unit testing on.

Is unit testing with AWS Glue possible, and if so, what are the best practices for doing so?

Improve this question
2
  • 1
    Checkout this solution with codepipeline. aws.amazon.com/blogs/big-data/…
    – user17183
    Commented Sep 19, 2019 at 19:26
  • @user17183 Please add this as an answer by explaining the details rather than only adding a link
    – 030
    Commented Dec 19, 2019 at 13:57

2 Answers 2

Reset to default
2

Checkout this solution with codepipeline. https://aws.amazon.com/blogs/big-data/implement-continuous-integration-and-delivery-of-serverless-aws-glue-etl-applications-using-aws-developer-tools/

Improve this answer
1
  • 1
    The link looks good. Could you summarize the important elements in the answer to prevent that the information will be lost if the link is deprecated.
    – 030
    Commented Dec 20, 2019 at 8:39
1

If you are unit testing, you should be mocking external dependencies like Spark and AWS Glue. Look into Python mocking frameworks like unittest.mock, monkeypatch and pytest-mock. Your unit tests should just test your code and how it interacts with Spark and AWS Glue. Your unit tests should not test Spark and AWS Glue functionality. Do that in your component and integration testing.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.