Apache Airflow Tutorial

 Apache Airflow is a tool for authoring, monitoring & scheduling pipelines. As a result, it is an ideal case of ETL & MLOps pipelines. 

Examples Uses Cases:

  1. Extracting data from many sources, aggregating them, transforming them, and store in a data warehouse.

  2. Extract insights from data and display them in an analytics dashboard

  3. Train, validate, and deploy machine learning models

Key Components:

1. WebServer : Webserver is Airflow’s user interface (UI), which allows you to interact with it without the need for a CLI or an API. From there one can execute, and monitor pipelines, create connections with external systems, inspect their datasets, and many more.

2. Schedular : The scheduler is responsible for executing different tasks at the correct time, re-running pipelines, backfilling data, ensuring tasks completion, etc.

3. Executors : Executors are the mechanism by which pipelines run. There are many different types that run pipelines locally, in a single machine, or in a distributed fashion.

4. Databases: It store all metadata related to pipeline. It supports postgres , mysql & other database as well. 

Airflow Basic Concenpts:

1. DAGs: 
All pipelines are defined as directed acyclic graphs (DAGs). Any time we execute a DAG, an individual run is created. Each DAG run is separate from another and contains a status regarding the execution stage of the DAG. This means that the same DAGs can be executed many times in parallel. 

2. Tasks:
Each node of the DAG represents a Task, meaning an individual piece of code. Each task may have some upstream and downstream dependencies. These dependencies express how tasks are related to each other and in which order they should be executed. Whenever a new DAG run is initialized, all tasks are initialized as Task instances. This means that each Task instance is a specific run for the given task.


3. Operators:

Operators can be viewed as templates for predefined tasks because they encapsulate boilerplate code and abstract much of their logic. Some common operators are BashOperator, PythonOperator, MySqlOperator, S3FileTransformOperator. As you can tell, the operators help you define tasks that follow a specific pattern. For example, the MySqlOperator creates a task to execute a SQL query and the BashOperator executes a bash script.

Operators are defined inside the DAG context manager.

4. Task Dependencies:

To form the DAG’s structure, we need to define dependencies between each task. One way is to use the >> symbol as shown below:

task1 >> task2 >> task3

Note that one task may have multiple dependencies:

task1 >> [task2 , task3]

The other way is through the set_downstream, set_upstream functions:

t1.set_downstream([t2,t3])

5. XComs:

XComs, or cross communications, are responsible for communication between tasks. XComs objects can push or pull data between tasks. More specifically, they push data into the metadata database where other tasks can pull from. That’s why there is a limit to the amount of data that can be passed through them. 

6. Scheduling:

Scheduling jobs is one of the core features of Airflow. This can be done using the schedule_interval argument which receives a cron expression, a datetime.timedelta object, or a predefined preset such as @hourly, @daily etc. A more flexible approach is to use the recently added timetables that let you define custom schedules using Python.


Once we define a DAG, we set up a start date and a schedule interval. If catchup=True, Airflow will create DAG runs for all schedule intervals from the start date until the current date. If catchup=False, Airflow will schedule only runs from the current date.

Backfilling extends this idea by enabling us to create past runs from the CLI irrespective of the value of the catchup parameter:

airflow backfill -s <START_DATE> -e <END_TIME> <DAG_NAME>


Connections & Hooks:

Airflow provides an easy way to configure connections with external systems or services. Connections can be created using the UI, as environment variables, or through a config file. They usually require a URL, authentication info and a unique id. Hooks are an API that abstracts communication with these external systems

Airflow Best Practises:

  1. Idempotency: DAGs and tasks should be idempotent. Reexecuting the same DAG run with the same inputs should always have the same effect as executing it once.

  2. Atomicity: Tasks should be atomic. Each task should be responsible for a single operation and independent from the other tasks

  3. Top-level code: Top-level code should be avoided if it’s not for creating operators or dags because it will affect performance and loading time. All code should be inside tasks, including imports, database access, and heavy computations.

  4. Complexity: DAGs should be kept as simple as possible because high complexity may impact performance or scheduling



Comments

Post a Comment

Popular posts from this blog

Spark Debugging And performance Optimzation

Image
  Introduction to Spark Optimization Optimizing Spark can dramatically improve performance and reduce resource consumption. Typically, optimization in Spark can be approached from three distinct levels: cluster level, code level, and CPU/memory level. Each level addresses different aspects of Spark's operation and contributes uniquely to overall efficiency in distributed processing with Spark. Cluster Level Optimization At the cluster level, optimization involves configuring the Spark environment to efficiently manage and utilize the hardware resources available across the cluster. This includes  tuning resource allocation settings such as executor memory, core counts, and maximizing data locality to reduce network overhead . Effective cluster management ensures that Spark jobs are allocated the right amount of resources,  balancing between under-utilization and over-subscription . Code Level Optimization Code level optimization refers to the improvement of the actual Spa...
Read more

Spark File Format

Image
  File Formats In the realm of data storage and processing, file formats play a pivotal role in defining how information is organized, stored, and accessed. Here's a table summarizing the characteristics, advantages, limitations, and use cases of the mentioned file formats: Format Characteristics Advantages Limitations Use Cases Comma-Separated Values (CSV) Simple, all data as strings, line-based. Simplicity, human readability. Space inefficient, slow I/O for large datasets. Small datasets, simplicity required. XML/JSON Hierarchical, complex structures, human-readable. Suitable for complex data, human-readable. Non-splittable, can be verbose. Web services, APIs, configurations. Avro Row-based, schema in header, write optimized. Best for schema evolution, write efficiency, landing zone suitability (1), predicate pushdown (2). Slower reads for subset of columns. Data lakes, ETL operations, microservices architechture, schema evolving environments. Optimized Row Columnar (ORC) Columna...
Read more

Basics of Spark

SparkContext vs SparkSession While SparkContext is used for accessing Spark features through RDDs, SparkSession provides a single point of entry for DataFrames, streaming, or Hive features including HiveContext, SQLContext or Streaming Context. Also, with the introduction of Spark 2.0, SparkSession was introduced as a new entry point for Spark applications. Lazy Evaluation When you apply a transformation like  sort  to your data in Spark, it's crucial to understand that Spark operates under a principle known as  Lazy Evaluation . This means that when you call a transformation, Spark doesn't immediately manipulate your data. Instead, what happens is that Spark queues up a series of transformations, building a plan for how it will eventually execute these operations across the cluster when necessary. Lazy evaluation in Spark ensures that transformations like   sort   don't trigger any immediate action on the data. The transformation is registered as a part of the ...
Read more