Geek Answers Handbook

Airflow: when to use CeleryExecutor and when to use Mesos Executor

I am new to Airflow and trying to understand how we should configure it in our environment (on aws).

I read that Airflow uses celery with the redis broker. How is it different from Mesos? I had not used Celery before, but I tried setting up celery-redis on my dev machine, and it worked with ease. But adding new components means extra monitoring.

Since we already use mesos for our cluster management, I’m trying to understand what I’m missing if I didn’t select celery and go with MesosExecutor?

+4
source share
4 answers

CeleryExecuter Redis Broker. , , , RabbitMQ, Redis Broker.

. MesosExecutor, CeleryExecutor, . , , Airbnb CeleryExecutor .

0

MesosExecutor - Docker, , , .

, Mesos, , Celery MQ - .

0

/ .

, Mesos, , MesosExecutor. , .

Celery + RabbitMQ, MesosExecutor, .

0

You cannot use the MesosExecutor for us. We need an abstraction layer to handle dependencies for work, we cannot (and should not) rely on any dependent application installed on mesos slaves. When the Docker container and / or the Mesos container are supported by the MesosExecutor, we can refer to it. In addition, I like to see dedicated workers inside the marathon. I am working on how to auto-scale workers using a marathon.

0
source

More articles:


All Articles