My team is on the process of migrating all our cloud applications from AWS to GCP, and we were considering using GKE for our main program, until we got the news that Google will now charge an hourly management fee of $0.10 for each cluster in GKE.
We still haven't started using GKE, and are now considering if it's really the best option for us. Our workload consists of two process-heavy data-analyses pipelines, which we use to run up to 20 analyses simultaneously in different AWS instances, running those once a week, for around 15min to 3h at most (and perhaps will start running it twice a week on the nearby future). On AWS, we currently spawn several instances of EC2 machines for each pipeline, one for each analyses, so we were considering GKE as we could run several analyses on the same cluster, and let GKE do the scaling for us.
I'm also under the impression that this cost is on top of the cost to have one pod running constantly for every nodepool, is that correct? For every nodepool we create inside our cluster, we'd have to pay for a pod to be running in there 24/7 as well, so that's another 'idle cost' to be considered...
We'd only need to run workloads on the cluster for around 15min to 6h each week, so paying for nodes to be up and running 24/7 seems a bit of a waste, but perhaps I'm not seeing the big picture here, as analyses could be run on the same cluster and save us money? Is there an efficient way to shut GKE down while we're not using it - either the pods or, better yet, the whole cluster itself? That'd save us at the very least 690h of running idle every month.
Would killing and deleting all our clusters as soon as the last analyses is done, and recreating it every week, once new analyses are needed, be something feasible? Would there be a cost involved with that? Is that feasible to be done 'automatically'?