Apache Airflow federation - observe remote dag in full detail?

Question

We have numerous airflow instances running in different organizations/teams around the company. While we have built a highly scaleable centralized airflow service, we do have cases where we need to depend on another airflow instance to run a dag.

Our DevOps team often gets tasked with running larger orchestrations across teams, and we prefer to keep a highly detailed view of all tasks, even if they are in a remote airflow.

Is there any good way to monitor, with full fidelity, all of the tasks in a remote airflow? Something similar to the sub-dag operator for a remote airflow would be sufficient.

Answer 1

I spent a long time on this problem and made 2 custom solutions as nothing out of the box exists.

Both steps require you to either use both the experimental and the extra plugin API or to have access to the remote database.

At that point, basically either:

• Make a GUI plugin on a separate page that hits the APIs and shows all tasks (list exec dates, list tasks in exec date, get status of each).
• Make python operator steps in your dag to call out to the state of the remote dag steps with the same name. It's hard to reflect anything outside of running/done though as you need your step to keep checking in case the remote end restarts the step/tries again.

The APIs are slow, so this isn't fun. I'd recommend going for the DB if you have the option.

Airflow 2.0 will fix the APIs apparently, and then this may be a lot easier.