Setting Up Apache Airflow and Jupyter Notebook on AWS EC2 instance

Airflow setup on EC2 instance along with DAG management on the server using Jupyter notebook is the easiest and convenient way of managing automated scripts in the Apache Airflow called DAGs. Before we move onto the deployment, let’s know about the Apache Airflow and Jupyter Notebook.

Apache Airflow

Airflow is a platform to programmatically author, schedule and monitor workflows.

Use Airflow to author workflows as Directed Acyclic Graphs (DAGs) of tasks. The Airflow scheduler executes your tasks on an array of workers while following the specified dependencies. Rich command line utilities make performing complex surgeries on DAGs a snap. The rich user interface makes it easy to visualize pipelines running in production, monitor progress, and troubleshoot issues when needed.

For Details information, click on this link https://airflow.apache.org/docs/stable

Jupyter Notebook

The Jupyter Notebook is an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and narrative text. Uses include: data cleaning and transformation, numerical simulation, statistical modeling, data visualization, machine learning, and much more.

Details on official website, https://jupyter.org

Setting Up Apache Airflow and Jupyter Notebook

For setting up Airflow and Jupyter Notebook, you need access to the Ubuntu terminal through SSH for EC2 instance. Secondly, make sure to allow 8080 and 8888 ports in the AWS security groups so that everyone can access these ports which are default ports for airflow and jupyter notebook respectively.

Install the apache airflow the traditional way and open the Custom TCP ports in the EC2 security group. The ports we will be targeting are 8080 for Apache Airflow and 8888 for Jupyter Notebook.

For details on setting up Apache Airflow and Jupyter Notebook you can follow below link:

https://airflow.apache.org/docs/stable/installation.html#getting-airflow

https://jupyter.readthedocs.io/en/latest/install/notebook-classic.html

The installation can also be done using the venv (Virtual environment) or Anaconda (Python environment manager). In such cases, we would require to provide respective library paths for Airflow and Jupyter Notebook.

To get the Jupyter Notebook exposed to 8888, it is required to generate the config file using below command. To create a jupyter_notebook_config.py file, with all the defaults commented out, you can use the following command line:

jupyter notebook --generate-config

If the current OS is ubuntu, the file can be located at:
/home/ubuntu/.jupyter/jupyter_notebook_config.py

You need to edit this file using VIM or Nano command in command line and add below lines in the file

c.NotebookApp.allow_origin = '*'

c.NotebookApp.ip = '0.0.0.0'

We can run airflow using following two commands

airflow webserver -p 8080 (command will expose the airflow interface on 8080 port)

airflow scheduler (with purpose of refreshing DAGs and scheduling and executing tasks)

To run Jupyter Notebook, use below command:

jupyter notebook (it will expose the respective directory listing in which command is executed)

We need to make sure that we are executing above jupyter command in Airflow Home Directory, which is going to allow us to manage the DAGs. The above commands are for testing purposes, we need to run these commands as service so that we don’t have to keep the ssh open.

For running the Apache Airflow and Jupyter Notebook as service in the background, we need to follow below steps:

1. Create a service for Airflow Webserver command using below command:

sudo nano /etc/systemd/system/airflow-webserver.service

Paste the below code in the file:

                    [Unit]
                    Description=Airflow webserver daemon
                    After=network.target postgresql.service mysql.service redis.service rabbitmq-server.service
                    Wants=postgresql.service mysql.service redis.service rabbitmq-server.service
                    [Service]
                    EnvironmentFile=/home/ubuntu/airflow/airflow
                    User=ubuntu
                    Group=ubuntu
                    Type=simple
                    ExecStart=/usr/bin/sudo /bin/bash -lc 'airflow webserver'
                    Restart=on-failure
                    RestartSec=5s
                    PrivateTmp=true
                    [Install]
                    WantedBy=multi-user.target
                

2. Create a service file for airflow scheduler command using below command:

sudo nano /etc/systemd/system/airflow-scheduler.service

Paste the below code in the file:

                    [Unit]
                    Description=Airflow scheduler daemon
                    After=network.target postgresql.service mysql.service redis.service rabbitmq-server.service
                    Wants=postgresql.service mysql.service redis.service rabbitmq-server.service
                    [Service]
                    EnvironmentFile=/home/ubuntu/airflow/airflow
                    User=ubuntu
                    Group=ubuntu
                    Type=simple
                    ExecStart=/usr/bin/sudo /bin/bash -lc 'airflow scheduler'
                    Restart=always
                    RestartSec=5s
                    [Install]
                    WantedBy=multi-user.target
                

3. Create a service file for airflow scheduler command using below command:

sudo nano /etc/systemd/system/jupyter.service

                    [Unit]
                    Description=Jupyter Notebook
                    [Service]
                    Type=simple
                    PIDFile=/run/jupyter.pid
                    ExecStart=/usr/bin/sudo /bin/bash -lc 'jupyter-notebook --allow-root'
                    User=ubuntu
                    Group=ubuntu
                    WorkingDirectory=/home/ubuntu/airflow
                    Restart=always
                    RestartSec=10
                    [Install]
                    WantedBy=multi-user.target
                

4. Before starting the services, they are required to be enabled using below commands:

                    systemctl enable airflow-webserver.service
                    systemctl enable airflow-scheduler.service
                    systemctl enable airflow-jupyter.service
                

5. To run the services, run below commands:

                    systemctl start airflow-webserver
                    systemctl start airflow-scheduler
                    systemctl start airflow-jupyter
                

Using above set of instructions, the apache airflow and jupyter notebook will be running smoothly on the EC2 instance.

For any questions or in need of more details, please feel free to reach out to us. Team Iotasol is here to help!

Does our work excite you?
envelope
team@iotasol.com
or share your details - we will reach back!