Problem statement- New files arrive on NFS and looking for a solution (using Apache airflow) to perform continuous NFS scan (for new file arrival) and unzip & copy file to another repository (on CentOS machine). Airflow is running as docker image.

  • about file watching, I am doing my opening the shell and running the command to see if file is present, I guess you can make a sensor out of it and do similar as a custom operator for unzip and copy – user6870 Mar 11 '18 at 16:27
  • It's resolved now - I used python package pyinotify and watched pyinotify.WatchManager() for new file. This python watcher triggered task on Airflow using TriggerDagRunOperator (from airflow.operators) – Anil Kumar May 9 '18 at 4:56
  • Could you post an answer? – 030 Dec 23 '19 at 8:53

Browse other questions tagged or ask your own question.