0

Trying to get an overview of different languages and software for workflows and data pipelines--and thus 'data engineering'--I stumbled upon Common Workflow Language. I thought, well, how common is this? - Got to ask the community, and I hope DevOps is appropriate.

Does someone know how used this language is? Perhaps compared to Apache Airflow or other tools for workflows.

1 Answer 1

2

This question is almost certainly going to be closed "off-topic", but since I have some experience with communities using CWL in the past, I'll provide my bit.

First, a correction to your premise: CWL is a standard, which makes it different to most other workflow tools. It has several implementations. One of the main benefits is that it's platform-agnostic, making workflows highly-portable.

As you can see from their website, the user community of this standard is heavily biased towards research communities. Implementations of the standards are de-facto default toolchains in several smaller communities, mostly in the bioinformatics domain, where workflows can become quite complex.

There is a strong incentive to publish workflows so that

  1. workflows can be attributed correclty
  2. reproducibility can be tested.

If you want to use Apache Airflow, there's an implementation of CWL for that

2
  • Thanks a lot (OMG, the number of alternatives). Do you mind expanding a bit on comparisons with the alternatives? Perhaps 'implementations of CWL for a given tool' and "plain" CWL without needing the additional ones? - that is, if such a question makes sense. Otherwise, please state that such a way is not useful for reason a,b,c. :-)
    – Johan
    Commented May 3, 2023 at 10:21
  • 1
    I think you'll find that at commonwl.org/implementations Commented May 3, 2023 at 11:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.