Azure Databricks administration - ETL Workflow
- Description
- Curriculum
- FAQ
- Reviews
In this Course, you will learn about spark based Azure Databricks, with more and more data growing daily take it any source be it in csv, text , JSON or any other format the consumption of data has been really easy via different IoT system. mobile phones internet and many other devices.
Azure Databricks provides the latest versions of Apache Spark and allows you to seamlessly integrate with open source libraries. Spin up clusters and build quickly in a fully managed Apache Spark environment with the global scale and availability of Azure.
Here is the 30,000 ft. overview of the agenda of the course, what will you learn and how you can utilise the learning into a real world data engineering scenario, this course is tailor made for some one who is coming from a background with no prior knowledge of Databricks and Azure and would like to start off their career in data world, specially around administering Azure Databricks.
Prepare for Azure Databricks Certified Associate Platform Administrator by solving practise questions.
Prepare for interviews and certification by solving quizzes at the end of sessions.
1. What is Databricks?
2. Databricks Components:
a. Notebook
b. Clusters
c. Pool
d. Secrets
e. Databricks CLI
f. Cluster Policy
3. Automate the entire administration activity via Terraform.
4. Mount Azure Blob Storage with Databricks.
5. Automate mount Azure Blob Storage with Databricks.
6. Load CSV data in Azure blob storage
7. Transform data using Scala and SQL queries.
8. Load the transform data into Azure blob storage.
9. Understand about Databricks tables and filessystem.
10. Configure Azure Databricks logging via Log4j and spark listener library via log analytics workspace.
11. Configure CI CD using Azure DevOps.
12. Git provider intergration.
13. Configure notebook deployment via Databricks Jobs.
-
1
Azure Databricks Architechture -
2
Introduction
-
3
Module 1 Agenda
-
4
Introduction to Azure Databricks Workspace.
In this section we will start creating Azure Databricks core component, we would take a tour of Azure databricks and learn about the fundamentals asets like notebook, clusters, pool, admin console, user setting and many more.
-
5
Databricks Clusters
We will create clusters in this session and would be using it run our notebooks on top of this cluster.
-
6
Databricks Pools
Spinning up a cluster can take up sometime, and to reduce the start up time and auto scaling time we would be using databricks pool.
-
7
Databricks Notebooks and magic commands
All of the resources we have been managing so far manually can be managed via Azure Databricks cli as well, in this session we would understand about databricks cli.
-
8
Databricks CLI and DBFS management
In this session we would be automating the entire Infrastructure process by creating terraform config files.
-
9
Administrating Cluster via Terraform
-
10
Databricks Administration Practise -
11
Practise actual scenarios to master databricks administration
-
12
Azure Databricks Platform Admin practice questions
-
13
Mount Azure blob storage to load and save data
-
14
Apply try and catch on Scala application
-
15
Automate blob mount via terraform
-
16
Transform data using scala
-
17
Apply filters using SQL
-
18
Push transformed data into Blob storage
-
19
NYC FireStation Data Schema
-
20
Databricks tables and file system.
-
21
Data transformation Questions
-
22
aggregate, where, filter, orderby,count functions
-
23
use of Min(),max(),columnrenamed() functions
-
24
Structured Streaming
