IaC Deployment in Azure with Terraform + Chef vs. Puppet vs. Ansible for Data Engineers

Question

I am an Azure Data Engineer and try to leverage the IaC benefits in this area of Azure Data Factory, Azure Databricks, Azure SQL & Synapse, Azure Power BI, Azure IoT. After some research I figured out that I need two essential parts:

1.Provisioning part
2.Configuration management part

Breaking down to technology I decided to use Terraform for the provisioning of infrastructure like (RG, Network, DBs etc). For the configuration management I am still not sure what to choose between Chef, Puppet and Ansible. There is another idea to use Docker/Packer + Kubernetes.

What are the best practices of IaC and automation when it comes to the Area of Data Engineering/Big Data?

Answer 1

Cloud setup could be done completly by Terraform. Check out terraform provider resources for required parameters ex. Azure Data Factory

Configuration management stuff mostly in use for OS tuning inside Virtual Machines. So maybe you don't need them at all.