Airflow/CWL Glossary and FAQ

What is CWL?
What is Airflow
What is CWL-Airflow
What is Docker
What is Docker-compose
What is Conda
What is PostgreSQL

What is CWL?

CWL stands for the Common Workflow Language Citing the CWL website:

CWL is an open standard for describing analysis workflows 
and tools in a way that makes them portable and scalable 
across a variety of software and hardware environments, 
from workstations to cluster, cloud, and high performance 
computing (HPC) environments. CWL is designed to meet the 
needs of data-intensive science, such as Bioinformatics, 
Medical Imaging, Astronomy, High Energy Physics, and Machine Learning.

Apache Airflow is a platform to programmatically author, schedule and monitor workflows. It represents workflows as graphs, visualizes them and provides a nice Graphical User Interface (GUI) to manage, control and monitor workflows.

What is CWL-Airflow 

CWL-Airflow implements CWL by translating CWL workflows into Airflow DAGs, hence providing a nice GUI to manage, control and monitor CWL workflows.

What is Docker 

Docker, is a software framework for managing containers on servers and the cloud.

What is Docker-compose 

Docker-compose is a tool for defining and running multi-container Docker applications.

Airflow uses multiple containers (at least webserver and scheduler) and also interacts with other applications, such as a RDBMSs where it stores it state. These applications can also be deployed as containers. We use docker-compose to orchestrate all required containers.