Build Data Engineering Pipelines with Azure Data Factory
- Description
- Curriculum
- FAQ
- Reviews
Hello students!
I am pleased to present this course on one of the most in demand data engineering tools around… Azure Data Factory!
As the demand for cloud-based data integration services continues to skyrocket, there is a huge demand for professionals with knowledge of services like Azure Data Factory. By learning Azure Data Factory, users can enhance their skills and increase their job prospects in the field of data engineering and analytics.
In this course you will primarily be using Azure Data Factory on Microsoft Azure in addition to other services such as Azure Blob Storage, Azure Data Lake Storage Gen 2 and Azure SQL Database.
The course is packed with lectures, code-along videos and a dedicated course project. As an added benefit you will also have lifetime access to all of the lectures…
This course will cover the following topics:
-
Azure Storage Solutions such as Azure Blob Storage and Azure Data Lake Gen2 Storage
-
The basics of Azure Data Factory including the core components such as Linked Services, Datasets, Activities, Data Flows, Pipelines and Integration Runtimes
-
Integrating data from various file formats such as CSV, JSON and Parquet
-
The Copy Activity in Azure Data Factory
-
Data Flows, Control Flow and Transformation Activities in Azure Data Factory
-
Orchestrating Data Integration Workflows
-
How to create Schedules and Triggers to execute your pipelines
-
How to use Parameters and Variables with your Linked Services, Datasets and Pipelines
-
How to use Azure Data Factory with SQL Databases
-
Authentication and Access including Managed Identities, Service Principals and Azure Key Vault
-
21
Copy Activity Overview -
22
Section Objective
-
23
Copy Activity - Order Items Part 1 (Linked Services and Source/Sink Datasets)
-
24
Copy Activity - Order Items Part 2 (Copy Activity and Pipeline)
-
25
Copy Activity - Customers JSON
-
26
Copy Activity - Orders Parquet
-
27
Copy Activity - Stores and Products JSON
-
28
Organising Data Factory Objects in Folders
-
29
Chaining Activities
-
30
Working with Multiple Files as a Source and Copy Behaviour
-
31
Links and Resources
-
32
Data Flow Overview
-
33
Section Objective
-
34
Data Flow Demo and User Interface
-
35
Data Flow Debug (Pricing)
-
36
Data Flow Debug (Row Limit)
-
37
Source and Sink Partitioning
-
38
Select Transformation
-
39
Cast Transformation
-
40
Import Projection
-
41
Derived Column Transformation
-
42
Raw to Cleansed Data Flow Requirement
-
43
Raw to Cleansed Data Flow Walkthrough
-
44
Partitioned Datasets
-
45
Filter and Sort Transformations
-
46
Aggregate Transformation
-
47
Join Transformation
-
48
Conditional Split and Union Transformations
-
49
Cleansed to Structured Data Flow Requirement
-
50
Cleansed to Structured Data Flow Walkthrough
-
51
Structured to Analytics Data Flow Requirement
-
52
Structured to Analytics Data Flow Walkthrough
-
53
Links and Resources
-
60
Data Lake Preparation
-
61
Delete Activity
-
62
Get Metadata Activity
-
63
Lookup Activity
-
64
If Condition and Switch Activities
-
65
Introduction to Parameters in ADF
-
66
Introduction to Variables in ADF
-
67
ForEach Activity
-
68
Metadata Based Approach (Example)
-
69
Course Project Activity: Dynamic Datasets and Folder Structure
-
70
Links and Resources
-
71
Section Overview
-
72
Creating our Azure SQL Resource
-
73
Creating our Schema and Tables
-
74
Adding a Unique Identifier Column to Structured Orders Data
-
75
Creating Linked Service and Datasets for SQL
-
76
Important Note Regarding the Dynamic Datasets
-
77
Copy Activity with SQL Sink
-
78
Data Flow with SQL Sink
-
79
Course Project Activity: Storing our Structured Data in the SQL Database
-
80
Links and Resources
