Setting Up Apache Airflow for Local Development

In this guide, we’ll walk through setting up Apache Airflow on a local machine using Conda to manage the Python environment.

1. Create a Conda Environment

First, create a Conda environment that will house the necessary Python libraries for Airflow:

conda create -n airflow python=3.11 -y

2. Activate the Conda Environment

Once the environment is created, activate it with the following command:

conda activate airflow

3. Install Apache Airflow

To install Airflow, you’ll need to use a constraints file specific to your Python and Airflow versions. Start by setting up the environment variables and then proceed with the installation:

export AIRFLOW_HOME=~/airflow

AIRFLOW_VERSION=2.10.0

PYTHON_VERSION="$(python -c 'import sys; print(f"{sys.version_info.major}.{sys.version_info.minor}")')"

CONSTRAINT_URL="https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt"

pip install "apache-airflow==${AIRFLOW_VERSION}" --constraint "${CONSTRAINT_URL}"

4. Verify the Installation

Check if Airflow was installed successfully by running:

airflow version

You should see output similar to:

2.10.0

5. Run Airflow Standalone

Start Airflow in standalone mode, which initializes the database, creates a user, and launches all components:

airflow standalone

6. Access the Airflow UI

To access the Airflow UI, open your web browser and navigate to localhost:8080. You can log in using the admin credentials provided in the terminal when Airflow started. Here, you'll find the default/example DAGs ready to be explored and tested.

7. Running Airflow Components Manually

If you prefer to run each part of Airflow separately instead of using the all-in-one standalone command, you can do so with the following commands:

1. Initialise the database
2. Create an admin user
3. Start the web server
4. Launch the scheduler

airflow db migrate

airflow users create \
    --username khodeprasad \
    --firstname Prasad \
    --lastname Khode \
    --role Admin \
    --email khodeprasad@test.org

airflow webserver --port 8080

airflow scheduler