Geek Answers Handbook

Airflow: how to SSH and run BashOperator from another server

Is there a way to ssh to another server and run BashOperator using Airbnb Airflow? I am trying to run the hive sql command with Airflow, but I need to SSH into another box to start the hive shell. My tasks should look like this:

  • SSH for server1
  • launch hive shell
  • run the hive command

Thanks!

+6
source share
1 answer

I think I just realized:

  • Create an SSH connection in the user interface under Admin> Connection. Note: the connection will be deleted if you reset the database

  • In the Python file add the following

    from airflow.contrib.hooks import SSHHook sshHook = SSHHook(conn_id=<YOUR CONNECTION ID FROM THE UI>)

  • Add SSH statement task

     t1 = SSHExecuteOperator( task_id="task1", bash_command=<YOUR COMMAND>, ssh_hook=sshHook, dag=dag)

Thanks!

+11
source

More articles:


All Articles