Problem statement- New files arrive on NFS and looking for a solution (using Apache airflow) to perform continuous NFS scan (for new file arrival) and unzip & copy file to another repository (on CentOS machine). Airflow is running as docker image.

  • about file watching, I am doing my opening the shell and running the command to see if file is present, I guess you can make a sensor out of it and do similar as a custom operator for unzip and copy – whoisthis Mar 11 '18 at 16:27
  • It's resolved now - I used python package pyinotify and watched pyinotify.WatchManager() for new file. This python watcher triggered task on Airflow using TriggerDagRunOperator (from airflow.operators) – Anil Kumar May 9 '18 at 4:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.