Master AWS Lambda Functions for Data Engineers using Python
- Description
- Curriculum
- FAQ
- Reviews
Do you want to learn AWS Lambda Functions by building an end-to-end data pipeline using Python as Programming Language and other key AWS Services such as Boto3, S3, Dynamodb, ECR, Cloudwatch, Glue Catalog, Athena, etc? Here is one course using which you will learn AWS Lambda Functions by implementing an end-to-end pipeline by using all the services mentioned.
As part of this course, you will learn how to develop and deploy lambda functions using the zip files, custom docker images as well as layers. Also, you will understand how to trigger lambda functions from Eventsbridge as well as Step Functions.
-
Set up required tools on Windows to develop the code for ETL Data Pipelines using Python and AWS Services. You will take care of setting up Ubuntu using wsl, Docker Desktop, and Visual Studio Code along with Remote Development Extension Kit so that you can develop Python-based applications using AWS Services.
-
Setup Project or Development Environment to develop applications using Python and AWS Services on Windows and Mac.
-
Getting Started with AWS by creating an account in AWS and also configuring AWS CLI as well as Review Data Sets used for the project
-
Develop Core Logic to Ingest Data from source to AWS s3 using Python boto3. The application will be built using Boto3 to interact with AWS Services, Pandas for date arithmetic, and requests to get the files from the source via REST API.
-
Getting Started with AWS Lambda Functions using Python 3.9 Run-time Environment
-
Refactor the application, and build a zip file to deploy as AWS Lambda Function. The application logic includes capturing bookmarks as well as Job Run details in Dynamodb. You will also get an overview of Dynamodb and how to interact with Dynamodb to manage Bookmark as well as Job Run details.
-
Create AWS Lambda Function using a Zip file, deploy using AWS Console and Validate.
-
Troubleshoot issues related to AWS Lambda Functions using AWS Cloudwatch
-
Build a custom docker image for the application and push it to AWS ECR
-
Create AWS Lambda Function using the custom docker image in AWS ECR and then validate.
-
Get an understanding of AWS s3 Event Notifications or s3-based triggers on Lambda Function.
-
Develop another Python application to transform the data and also write the data in the form of Parquet to s3. The application will be built using Pandas by converting 10,000 records at a time to Parquet.
-
Build orchestrated pipeline using AWS s3 Event Notifications between the two Lambda Functions.
-
Schedule the first lambda function using AWS EventsBridge and then validate.
-
Finally, create an AWS Glue Catalog table on the s3 location which has parquet files, and validate by running SQL Queries using AWS Athena.
-
After going through the complete life cycle of Deploying and Scheduling Lambda Function and also validating the data by using Glue Catalog and AWS Athena, you will also understand how to use Layers for Lambda Function.
Here are the key takeaways from this training:
-
Develop Python Applications and Deploy as Lambda Functions by using a Zip-based bundle as well as a custom docker image.
-
Monitor and troubleshoot the issues by going through Cloudwatch logs.
-
The entire application code used for the demo along with the notebook used to come up with core logic.
-
Ability to build solutions using multiple AWS Services such as Boto3, S3, Dynamodb, ECR, Cloudwatch, Glue Catalog, Athena, etc
-
3
Introduction to Getting Started on Windows with Required Tools -
4
Overview of Powershell on Windows 10 or Windows 11
-
5
Setup Ubuntu VM on Windows 10 or 11 using wsl
-
6
Setup Ubuntu VM on Windows 10 or 11 using wsl
-
7
Setup Docker Desktop on Windows
-
8
Validate Docker on Windows using Command Line leveraging Power Shell
-
9
Review Docker Desktop Resource Configurations
-
10
Install Visual Studio Code on Windows
-
11
Install Remote Development Extension Kit for Visual Studio Code
-
12
Install Python 3.9 and Distutils on Windows using wsl Ubuntu
-
13
Review Tools Installed for Application Development using Python and AWS Service
-
14
Setup Project Folder using Visual Studio Code
-
15
Ensure Python 3.9 for the Project
-
16
Create Python Virtual Environment using Python 3
-
17
Install Required Dependencies for the Project using AWS Services
-
18
Ensure AWS CLI to interact with AWS Services using AWS CLI Commands
-
19
Recommendation to use Personal AWS Account for the course
-
20
Setup and Login into AWS Account
-
21
Setup AWS IAM User with Administrator Permissions
-
22
Configure and Validate AWS CLI
-
23
Configure AWS CLI with custom profile as default
-
24
Recap of Date Arithmetic using Python
-
25
Validate Python boto3 to interact with AWS Services
-
26
Setup and Validate Jupyter based Interactive Environment
-
27
Review GHActivity Data Details
-
28
Download GHActivity Data using requests
-
29
Review GHActivity Data using Pandas
-
30
Managing s3 using Python boto3
-
31
Overview of AWS Dynamodb
-
32
Create DynamoDB Table for Job Details
-
33
Create DynamoDB Table for Job Run Details
-
34
Get First Run Details to Copy GHActivity Data to AWS s3
-
35
Get Incremental Load Logic for next file
-
36
Understand AWS s3 concepts such as buckets and objects
-
37
Copying or Uploading Files to AWS s3 as objects using Python boto3
-
38
Writing Python Objects or Data as AWS s3 Objects using boto3
-
39
Save GHActivity Data to AWS s3
-
40
Convert Date Time to Integer Unix Epoch using Python
-
41
Save Job Run Details to DynamoDB Table
-
42
Validate Data Copied to AWS s3 and job run details
-
43
Run and Validate End to End Process
-
44
Introduction to Getting Started with AWS Lambda Functions
-
45
Overview of AWS Lambda and Getting Started using Python 3
-
46
Passing Arguments to AWS Lambda and Processing using Python
-
47
Using Custom Handlers for AWS Lambda Functions using Python 3
-
48
Using AWS Services such as s3 in AWS Lambda Functions
-
49
Recap of handling permissions using AWS IAM Roles and User Groups
-
50
Develop AWS Lambda Function to list objects from AWS S3 Bucket
-
51
Passing Environment Variables to AWS Lambda Functions
-
52
Customizing Resources such as memory used for AWS Lambda Function
-
53
Setup Local Development Environment for AWS Lambda Functions
-
54
Develop logic for AWS Lambda Function using external packages
-
55
Build Zip file to deploy as AWS Lambda Function
-
56
Deploy Application with External Dependencies as AWS Lambda Function
-
57
Understand Problem Statement for Python Application for AWS
-
58
Setup Python Project for AWS Lambda using Visual Studio Code
-
59
Core Logic to upload files to AWS S3 using Python boto3
-
60
Develop Python Application to upload files to AWS s3 using Python boto3
-
61
Build Zip File for Python Application to deploy as AWS Lambda Function
-
62
Deploy Python Application as AWS Lambda Function using Zip File
-
63
Conclusion and request for rating and feedback
-
64
Introduction to Build and Deploy AWS Lambda Function using Zip File
-
65
Update Application Code with Core logic for Ingestion
-
66
Overview of Validating User Defined Functions using Python CLI
-
67
Validate Application using Core Logic to ingest data
-
68
Add Lambda Handler to ingest data to AWS s3
-
69
Build Zip File for Python Application to deploy as AWS Lambda Function
-
70
Upload Python Application Zip File to s3 and deploy as AWS Lambda Function
-
71
Set Custom Handler and required Environment Variables for AWS Lambda Function
-
72
Granting Permissions on AWS s3 and Dynamodb to AWS Lambda Function via Role
-
73
Change Memory and Timeout for AWS Lambda Function and Test
-
74
Recap and Overview of Monitoring Lambda Functions using Cloudwatch
-
75
Limitations of Deploying AWS Lambda Function using Zip file
-
76
Automate Build of AWS Lambda Function using Shell Scripts
-
77
Introduction to Build and Deploy AWS Lambda Function using Custom Docker Imag
-
78
Create Dockerfile for Custom Docker Image for AWS Lambda Function
-
79
Create Custom Docker Image for AWS Lambda Function using Python 3 Run-time
-
80
Validate Custom Docker Image by creating Docker Container
-
81
Run the application using Python CLI in the Docker Container
-
82
Run the Docker Container with the Credentials and Environment Variables
-
83
Validate AWS Lambda Function Locally using Docker and Curl
-
84
Create AWS ECR Repository for Custom Docker Image
-
85
Push Custom Docker Image for AWS Lambda Function to AWS ECR
-
86
Create AWS Lambda Function using Custom Docker Image in AWS ECR
-
87
Run and Validate AWS Lambda Function created using Custom Docker Image
-
88
Create Shell Script to Build and Push Docker Image to AWS ECR
-
89
Add Command to build script to reconfigure AWS Lambda Function to latest docker
