davvdg/docker-airflow repository overview

⁠docker-airflow

This repository contains Dockerfile of airflow⁠ for Docker⁠'s automated build⁠ published to the public Docker Hub Registry⁠.

⁠Informations

• Based on Debian Jessie official Image debian:jessie⁠ and uses the official Postgres⁠ as backend and RabbitMQ⁠ as queue
• Install Docker⁠
• Install Docker Compose⁠
• Following the Airflow release from Python Package Index⁠

⁠Installation

Pull the image from the Docker repository.

    docker pull puckel/docker-airflow

⁠Build

For example, if you need to install Extra Packages⁠, edit the Dockerfile and than build-it.

    docker build --rm -t puckel/docker-airflow .

⁠Usage

By default, docker-airflow run Airflow with SequentialExecutor :

    docker run -d -p 8080:8080 puckel/docker-airflow

If you want to run other executor, you've to use the docker-compose.yml files provided in this repository.

For LocalExecutor :

    docker-compose -f docker-compose-LocalExecutor.yml up -d

For CeleryExecutor :

    docker-compose -f docker-compose-CeleryExecutor.yml up -d

NB : If you don't want to have DAGs example loaded (default=True), you've to set the following environment variable :

LOAD_EX=n

    docker run -d -p 8080:8080 -e LOAD_EX=n puckel/docker-airflow

If you want to use Ad hoc query, make sure you've configured connections: Go to Admin -> Connections and Edit "mysql_default" set this values (equivalent to values in airflow.cfg/docker-compose.yml) :

• Host : mysql
• Schema : airflow
• Login : airflow
• Password : airflow

Check Airflow Documentation⁠

⁠UI Links

• Airflow: localhost:8080⁠
• Flower: localhost:5555⁠
• RabbitMQ: localhost:15672⁠

When using OSX with boot2docker, use: open http://$(boot2docker ip):8080

⁠Scale the number of workers

Easy scaling using docker-compose:

    docker-compose scale worker=5

This can be used to scale to a multi node setup using docker swarm.

⁠Links

• Airflow on Kubernetes kube-airflow⁠

⁠Wanna help?

Fork, improve and PR. ;-)