Airflow setup on EC2 instance along with DAG management on the server using Jupyter notebook is the easiest and convenient way of managing automated scripts in the Apache Airflow called DAGs. Before we move onto the deployment, let’s know about the Apache Airflow and Jupyter Notebook.
Airflow is a platform to programmatically author, schedule and monitor workflows.
Use Airflow to author workflows as Directed Acyclic Graphs (DAGs) of tasks. The Airflow scheduler executes your tasks on an array of workers while following the specified dependencies. Rich command line utilities make performing complex surgeries on DAGs a snap. The rich user interface makes it easy to visualize pipelines running in production, monitor progress, and troubleshoot issues when needed.
For Details information, click on this link https://airflow.apache.org/docs/stable
The Jupyter Notebook is an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and narrative text. Uses include: data cleaning and transformation, numerical simulation, statistical modeling, data visualization, machine learning, and much more.
Details on official website, https://jupyter.org
Setting Up Apache Airflow and Jupyter Notebook
For setting up Airflow and Jupyter Notebook, you need access to the Ubuntu terminal through SSH for EC2 instance. Secondly, make sure to allow 8080 and 8888 ports in the AWS security groups so that everyone can access these ports which are default ports for airflow and jupyter notebook respectively.
Install the apache airflow the traditional way and open the Custom TCP ports in the EC2 security
group. The ports we will be targeting are 8080 for Apache Airflow and 8888 for Jupyter Notebook.
For details on setting up Apache Airflow and Jupyter Notebook you can follow below link:
The installation can also be done using the venv (Virtual environment) or Anaconda (Python environment manager). In such cases, we would require to provide respective library paths for Airflow and Jupyter Notebook.
To get the Jupyter Notebook exposed to 8888, it is required to generate the config file using below command. To create a jupyter_notebook_config.py file, with all the defaults commented out, you can use the following command line:
jupyter notebook --generate-config
If the current OS is ubuntu, the file can be located at:
You need to edit this file using VIM or Nano command in command line and add below lines in the file
c.NotebookApp.allow_origin = '*'
c.NotebookApp.ip = '0.0.0.0'
We can run airflow using following two commands
airflow webserver -p 8080 (command will expose the airflow interface on 8080 port)
airflow scheduler (with purpose of refreshing DAGs and scheduling and executing tasks)
To run Jupyter Notebook, use below command:
jupyter notebook (it will expose the respective directory listing in which command is executed)
We need to make sure that we are executing above jupyter command in Airflow Home Directory, which is going to allow us to manage the DAGs. The above commands are for testing purposes, we need to run these commands as service so that we don’t have to keep the ssh open.
For running the Apache Airflow and Jupyter Notebook as service in the background, we need to follow below steps:
1. Create a service for Airflow Webserver command using below command:
sudo nano /etc/systemd/system/airflow-webserver.service
Paste the below code in the file:
[Unit] Description=Airflow webserver daemon After=network.target postgresql.service mysql.service redis.service rabbitmq-server.service Wants=postgresql.service mysql.service redis.service rabbitmq-server.service [Service] EnvironmentFile=/home/ubuntu/airflow/airflow User=ubuntu Group=ubuntu Type=simple ExecStart=/usr/bin/sudo /bin/bash -lc 'airflow webserver' Restart=on-failure RestartSec=5s PrivateTmp=true [Install] WantedBy=multi-user.target
2. Create a service file for airflow scheduler command using below command:
sudo nano /etc/systemd/system/airflow-scheduler.service
Paste the below code in the file:
[Unit] Description=Airflow scheduler daemon After=network.target postgresql.service mysql.service redis.service rabbitmq-server.service Wants=postgresql.service mysql.service redis.service rabbitmq-server.service [Service] EnvironmentFile=/home/ubuntu/airflow/airflow User=ubuntu Group=ubuntu Type=simple ExecStart=/usr/bin/sudo /bin/bash -lc 'airflow scheduler' Restart=always RestartSec=5s [Install] WantedBy=multi-user.target
3. Create a service file for airflow scheduler command using below command:
sudo nano /etc/systemd/system/jupyter.service
[Unit] Description=Jupyter Notebook [Service] Type=simple PIDFile=/run/jupyter.pid ExecStart=/usr/bin/sudo /bin/bash -lc 'jupyter-notebook --allow-root' User=ubuntu Group=ubuntu WorkingDirectory=/home/ubuntu/airflow Restart=always RestartSec=10 [Install] WantedBy=multi-user.target
4. Before starting the services, they are required to be enabled using below commands:
systemctl enable airflow-webserver.service systemctl enable airflow-scheduler.service systemctl enable airflow-jupyter.service
5. To run the services, run below commands:
systemctl start airflow-webserver systemctl start airflow-scheduler systemctl start airflow-jupyter
Using above set of instructions, the apache airflow and jupyter notebook will be running smoothly on the EC2 instance.
For any questions or in need of more details, please feel free to reach out to us. Team Iotasol is here to help!