I am an Azure Data Engineer and try to leverage the IaC benefits in this area of Azure Data Factory, Azure Databricks, Azure SQL & Synapse, Azure Power BI, Azure IoT. After some research I figured out that I need two essential parts:
1.Provisioning part
2.Configuration management part
Breaking down to technology I decided to use Terraform for the provisioning of infrastructure like (RG, Network, DBs etc). For the configuration management I am still not sure what to choose between Chef, Puppet and Ansible. There is another idea to use Docker/Packer + Kubernetes.
What are the best practices of IaC and automation when it comes to the Area of Data Engineering/Big Data?