NumPy, Pandas, Matplotlib in Python using Amazon SageMaker

Meet your instructors for the course.

In the first lecture we will be giving an Introduction to AWS SageMaker. We start by creating a new AWS account step by step and also provide you insights to what is python and also what are the aspect of Data science.

We will go ahead and start by opening a free Tier Account with AWS SageMaker. Amazon Sage Maker is a machine learning service that you can use to build, train, and deploy ML models for any use case. AWS SageMaker fits best for us. It provides Jupyter Notebooks for running Python kernels with a compute instance that we can choose as per our data engineering requirements on demand. It’s a good channel to start your professional Machine Learning models and then you can deploy your model with just few clicks.

We will be showing you how to login in to AWS SageMaker Account and also answer frequently asked Questions like

1.What is Amazon SageMaker?

2. How to build a studio in to AWS SageMaker?

It takes care of abstracting a ton of software development skills necessary to accomplish the task while still being highly effective and flexible and cost-effective. Most importantly, it helps you focus on the core ML experiments and supplements the remainder necessary skills with easy abstracted tools similar to our existing workflow.

Below is the provided link to open a free Account on AWS SageMaker

https://aws.amazon.com/sagemaker/

Important Points to be noted down is using SageMaker studio is Free, you only pay for the services that you will be using within the studio, you have many options or services to use ,With the AWS Free Usage Tier*, you can get started with Amazon S3 for free in all regions.

Make sure after using the SageMaker Studio please kindly shutdown all the services as per the hints provided in the lecture video or else AWS will charge for the usage and the running cost.Do raise an service ticket and request the AWS team to avoid any costs incurred.

Please download the Code files for this entire Course from the attached ZIP file. It also contains the PPT's (Presentations) and also the Cheat Sheets.

After opening of an account by using AWS SageMaker our next step is to learn the basics of Python.

Python is a very simple yet very effective coding language. If this is your first time in coding, Don’t worry since we begin with the very basics.

In this lecture we will be giving you an introduction to Python, Python programming is widely used in Artificial Intelligence, Natural Language Processing, Neural Networks and other advanced fields of Computer Science and we will be learning basic of pythons to create simple projects over the period of time.

Many large companies use the Python programming language include NASA, Google, YouTube, Netflix, etc.

It is ideally designed for rapid prototyping of complex applications, We will understand python basics by performing small tasks in the AWS SageMaker environment have the better hands-on practice.

Python Programming Characteristics:

• It provides rich data types and easier to read syntax than any other programming languages

• It is a platform independent scripted language with full access to operating system API's

• Compared to other programming languages, it allows more run-time flexibility

• A module in Python may have one or more classes and free functions

• Libraries in Pythons are cross-platform compatible with Linux, Macintosh, and Windows

• For building large applications, Python can be compiled to byte-code

• Python supports functional and structured programming as well as Object-oriented programming (OOP)

• It supports interactive mode that allows interactive testing and debugging of snippets of code

• In Python, since there is no compilation step, editing, debugging and testing is fast.

In this lecture we will be covering various data structures in python, you can think of a data structure as a way of organizing and storing data such that we can access and modify it efficiently. In python Data structures are like List, Tuples, Set and Dictionary.

In this session we will cover List and Tuples, we will perform the operations and also see their properties.

List: A List is an ordered collection of items (which may be of same or different types) separated by comma and enclosed in square brackets.

Lists are mutable and that means you can add, remove, delete so these operations are doable in the List.

Tuple: Tuple looks similar to list. The only difference is that comma separated items of same or different type are enclosed in parentheses. Individual items follow zero based index, as in list or string.

Difference between List and Tuple:

• List is Mutable

• Tuple is Immutable

• List is Ordered collection of items

• Tuple It is Ordered collection of items

• Items in list can be replaced or changed

• Items in tuple cannot be changed or replaced

Now we will go ahead and start with fundamentals of python and performed various tasks by using data structures in python.

If contents of an object can be modified in place, after it has been instantiated, is a mutable object. On the other hand, any operation on immutable object that tries to modify its contents is prohibited.

We start by creating List, Tuples and we will be explaining through the required examples.

We have already gone through the Data structures like List, Tuples and we studied the some of the Operations performed on List and Tuples and also some of the major properties of the above Data structures.

Now we will discuss in detail about sets and strings.

Set: This is one of the important Python Data Structures. A Python set is a slightly different concept from a list or a tuple. A set, in Python, is just like the mathematical set. It does not hold duplicate values and is unordered. However, it is not immutable, unlike a tuple.

String: Till now, we have discussed numbers as the standard data-types in Python. In this section of the tutorial, we will discuss the most popular data type in Python, i.e., string.

Python string is the collection of the characters surrounded by single quotes, double quotes, or triple quotes. The computer does not understand the characters; internally, it stores manipulated character as the combination of the 0's and 1's.

In Python, strings can be created by enclosing the character or the sequence of characters in the quotes. Python allows us to use single quotes, double quotes, or triple quotes to create the string.

Dictionaries : Finally, we will take a look at Python dictionaries. Think of a real-life dictionary. What is it used for? It holds word-meaning pairs. Likewise, a Python dictionary holds key-value pairs. However, you may not use an un-hash able item as a key.

To declare a Python dictionary, we use curly braces. But since it has key-value pairs instead of single values, this differentiates a dictionary from a set.

Creating a dictionary is as simple as placing items inside curly braces {} separated by commas. While the values can be of any data type and can repeat, keys must be of immutable type (string, number or tuple with immutable elements) and must be unique.

While indexing is used with other data types to access values, a dictionary uses keys. Keys can be used either inside square brackets [] or with the get() method.

If we use the square brackets [], KeyError is raised in case a key is not found in the dictionary. On the other hand, the get() method returns None if the key is not found.

Operators are special symbols in Python that carry out arithmetic or logical computation. The value that the operator operates on is called the operand.

1. Arithmetic operators are used to perform mathematical operations like addition, subtraction, multiplication, etc.

2. Comparison operators are used to compare values. It returns either True or False according to the condition.

3. Logical operators are the and, or, not operators.

We will also be explaining in detail about various operators used in python and perform it.

We will be working with strings and performing data collections operations in python. You’ll cover the basic characteristics of Python dictionaries and learn how to access and manage dictionary data. Once you have finished this tutorial, you should have a good sense of when a dictionary is the appropriate data type to use, and how to do so.

The control flow of a Python program is regulated by conditional statements, loops, and function calls. This section covers the if statement and for and while loops; functions are covered later in this chapter. Raising and handling exceptions also affects control flow;

Decision making is required when we want to execute a code only if a certain condition is satisfied. The if…elif…else statement is used in Python for decision making. In this Lecture we will be discussing on “if else statements” in python those who are new coding may find it tricky or interesting, also run some Basic problems in Aws SageMaker.

Decision making statements available in python are:

• if statement

• if. Else statements

• nested if statements

If statements: Often, you need to execute some statements only if some condition holds, or choose statements to execute depending on several mutually exclusive conditions. The Python compound statement if, which uses if, elif, and else clauses, lets you conditionally execute blocks of statements.

The elif and else clauses are optional. Note that unlike some languages, Python does not have a switch statement, so you must use if, elif, and else for all conditional processing

Functions are essential components in python. They help us to eliminate redundancy in the code. A function can be considered as a tool that takes specific inputs to give us required outputs. In this lecture, we will learn to implement and call functions to perform specific tasks.

We will start with basics and then show you some advance applications. We will be explaining different types of arguments i.e., inputs taken by the function. We will also discuss return statements.

A function is a block of organized, reusable code that is used to perform a single,

related action. As you already know, Python gives you many built-in functions like print(), etc. but you can also create your own functions.

These functions are called user-defined functions.

In Python, a function is a group of related statements that performs

a specific task. Functions help break our program into smaller and

modular chunks. As our program grows larger and larger,

functions make it more organized and manageable. Furthermore,

it avoids repetition and makes the code reusable.

The advantages of using functions are:

• Reducing duplication of code.

• Decomposing complex problems into simpler pieces.

• Improving clarity of the code.

• Reuse of code.

• Information hiding.

In this session you'll learn about functions, what a function is, the syntax, components, and types of functions. Also, you'll learn to create a function in Python. In Python, a function is a group of related statements that performs a specific task

Using lambda() Function with map()

The map() function in Python takes in a function and a list as an argument. The function is called with a lambda function and a list and a new list is returned which contains all the lambda modified items returned by that function for each item.

1. Lambda function: A lambda operator can have any number of arguments but can have only one expression. It cannot contain any statements and returns a function object which can be assigned to any variable.

Lambda functions are syntactically restricted to return a single expression. You can use them as an anonymous function inside other functions. The lambda functions do not need a return statement, they always return a single expression.

A Lambda Function in Python programming is an anonymous function or a function having no name. It is a small and restricted function having no more than one line.

2. Map function: Map functions expect a function object and any number of iterables, such as list, dictionary, etc. It executes the function_object for each element in the sequence and returns a list of the elements modified by the function object.

Python map() applies a function on all the items of an iterator given as input. An iterator, for example, can be a list, a tuple, a set, a dictionary, a string, and it returns an iterable map object. Python map() is a built-in function. ... Using map() with Python built-in functions.

The map() method creates a new array populated with the results of calling a provided function on every element in the calling array.

The map() function applies a given to function to each item of an iterable and returns a list of the results. The returned value from map() (map object) can then be passed to functions like list() (to create a list), set() (to create a set) and so on.

Will be try to understand what exactly these function does and also performing some basic operations on it.

In this session, we will be working with for loops. For loops are used to iterate through lists, tuples, sets and dictionary. We can perform element-wise operations on every element of the iterable.

Python programming language provides following types of loops to handle looping requirements. Python provides three ways for executing the loops. While all the ways provide similar basic functionality, they differ in their syntax and condition checking time.

For in Loop: For loops are used for sequential traversal. For example: traversing a list or string or array etc. In Python, there is no C style for loop, i.e., for (i=0; i<n; i++). There is “for in” loop which is similar to for each loop in other languages.

Using else statement with for loops: We can also combine else statement with for loop like in while loop. But as there is no condition in for loop based on which the execution will terminate so the else block will be executed immediately after for block finishes execution.

For loops can iterate over any iterable object (example: List, Set, Dictionary, Tuple or String).

Exercise:Now with the help of  examples in the video and lets dive deep and see what happens internally here.

1. Make the list (iterable) an iterable object with help of iter() function.

2. Run a infinite while loop and break only if the StopIteration is raised.

3. In the try block we fetch the next element of fruits with next() function.

4. After fetching the element we did the operation to be performed in with the element. (i.e print(fruit))

         Try the above Exercise and solve it on your own.

In previous session we have covered the topics on Loops, which included For Loops and we have run through the operations by using for loops, In this session, we will be working with while loops.

The while loop in Python is used to iterate over a block of code as long as the test expression (condition) is true. We generally use this loop when we don't know the number of times to iterate beforehand.

How do you use a while loop in Python?

With the while loop we can execute a set of statements as long as a condition is true.

• Print i as long as i is less than 6: i = 1. while i < 6: ...

• Exit the loop when i is 3: i = 1. while i < 6: ...

• Continue to the next iteration if i is 3: i = 0. while i < 6: ...

• Print a message once the condition is false: i = 1. while i < 6:

While loop is used when we are not sure about the number of times a condition has to be checked. Unlike for loop, while loop requires a terminating condition.

In Python, "for loops" are called iterators. Just like while loop, "For Loop" is also used to repeat the program. But unlike while loop which depends on condition true or false. ... For Loop iterates with number declared in the range.

In most computer programming languages, a do while loop is a control flow statement that executes a block of code at least once, and then either repeatedly executes the block, or stops executing it, depending on a given boolean condition at the end of the block.

We will also demonstrate the application of while loop as well as for loop using simple examples lets go ahead and dive in to a video.

In this session you will have understanding of python’s operation on topics we have learned so far.

Strings can be created by enclosing characters inside a single quote or double-quotes. Even triple quotes can be used in Python but generally used to represent multi line strings and docstrings.

We can access individual characters using indexing and a range of characters using slicing. Index starts from 0. Trying to access a character out of index range will raise an IndexError. The index must be an integer. We can't use floats or other types, this will result into TypeError.

Python allows negative indexing for its sequences.

The index of -1 refers to the last item, -2 to the second last item and so on. We can access a range of items in a string by using the slicing operator :(colon).

Strings are immutable. This means that elements of a string cannot be changed once they have been assigned. We can simply reassign different strings to the same name.

Let’s try to implement our logic using python and solve a couple of examples. We are working on two simple examples. We will describe the examples in details with every step for better understanding of concepts.

In our previous class we have solve some examples of python and in this session, we will cover the topic on object-oriented programming. what is object-oriented programming?

Objected oriented programming as a discipline has gained a universal following among developers. Python, an in-demand programming language also follows an object-oriented programming paradigm. It deals with declaring Python classes and objects which lays the foundation of OOPs concepts

Object Oriented Programming is a way of computer programming using the idea of “objects” to represents data and methods. It is also, an approach used for creating neat and reusable code instead of a redundant one. the program is divided into self-contained objects or several mini-programs. Every Individual object represents a different part of the application having its own logic and data to communicate within themselves.

Object-oriented programming (OOP) is a method of structuring a program by bundling related properties and behaviors into individual objects.

What are Classes and Objects?

Class: A class is a collection of objects or you can say it is a blueprint of objects defining the common attributes and behavior.

Objects: Objects are an instance of a class. It is an entity that has state and behavior. In a nutshell, it is an instance of a class that can access the data.

In this tutorial, you’ll learn the basics of object-oriented programming in Python.

In this session we will learn about basic component of python which is NumPy Arrays

Let’s solve some operations and understand how to make use of it this. NumPy is a Python library used for working with arrays. NumPy contains a multi-dimensional array and matrix data structures.

It can be utilised to perform a number of mathematical operations on arrays such as trigonometric, statistical, and algebraic.

Its core library for scientific computing, we are using the concept that has already explained in our previous lecture which is nothing but the list. And we are also making use of Matrix which we learnt it in our school time. Since list is a one-dimensional object and we are making use of NumPy to convert one directional object to multi-dimensional object.

We are going to learn how to do operations like element wise addition, subtraction, concatenate by using NumPy also how to reshape or concatenate the elements.

To convert list to an array we are using NumPy and it’s a package and name of the library we are importing here and making use of this we are able to convert any list, tuple in to an array.

By importing NumPy we are performing many operations and there are advanced operations which we will be discussing later.

In our previous session we have learned about NumPy Arrays operations and in this session we will discussing about the pandas library and solve some of the examples and perform some task on it. And also, how to handle data when we have csv or text data and manipulate data.

We have seen previous how to convert list to array, NumPy is a library in pandas which allows you to form multi-dimensional array, we have seen various functions like date, time, dimensions, size, shape etc.

We also did element wise addition, subtraction, concatenate by using NumPy

In computer programming, pandas is a software library written for the Python programming language for data manipulation and analysis.

Pandas is nothing but the data structure for manipulating the data, you will get introduce to panda’s series. It’s mostly used for structured data or labelled data very easily. Its pretty easy has lot of applications while handling structured data and helps to handle any missing values in it.

We will start with the introduction to panda’s series and performs some of the operations by using pandas

Library.

It provides high-performance, easy to use structures and data analysis tools. Unlike NumPy library which provides objects for multi-dimensional arrays,

So let’s talk about pandas series and dive in to it.

In the previous session we have learned about the pandas basis operations and also what ways we can perform the task in python, In today’s session we will be solving problems on how to load the data in pandas by using pandas Data frame attribute and extract values from csv files.

In this tutorial, we will learn to read CSV files with different formats in Python with the help of examples. We are going to exclusively use the csv module built into Python for this task.

By default, a comma is used as a delimiter in a CSV file. However, some CSV files can use delimiters other than a comma. Few popular ones are | and t.

A CSV file is nothing more than a simple text file. However, it is the most common, simple, and easiest method to store tabular data. Once you go through the installation, you can use the read_csv() function to read a CSV file.

You can display the first five rows of the CSV file via the head () method of the Pandas Data Frame. Let’s go and learn about the file handling by using pandas.

Please download the attached source file.

In previous session we have discussed about file handling also how to group a data and how to push a data in to file how to find null values and dimension of the data.

So, you’ve imported a CSV file with the Pandas Python library and had a first look at the contents of your dataset. So far, you’ve only seen the size of your dataset and its first and last few rows. Next, you’ll learn how to examine your data more systematically.

And this session is again upon file handling itself by using pandas. we will be doing some more advanced operation by reading data and goes follow and we will perform some random task on columns and also get the visual representation by manipulating data. we will get introduction to Bar graph, Box Plot.

Later, you’ll meet the more complex categorical data type, which the Pandas Python library implements itself.

In our previous session we have learnt about loading data by using pandas and NumPy, we discussed about handling null values, handling dimensions of the dataset and also how to do grouping.

in this session we will be performing tasks on columns and rows individually.

Now we are good to go for reading a dataset.by using pandas data frame we will try to extract data, this session we will be discussing about the further operations on pandas,

The have some visual representation of data by using pandas. And we will be making use of diabetes data.csv in the session and try to read the data by using pandas.

Also, we will try to understand the data how it really look like, basic operations of data will be the main of for this session and also like use of lambda, modulo such mathematical operations and finishing the model by having visual representation by plotting a graph.

we will be plotting graphs like bar graph and box plot.

Bar Graph: A bar plot or bar chart is a graph that represents the category of data with rectangular bars with lengths and heights that is proportional to the values which they represent. The bar plots can be plotted horizontally or vertically. A bar chart describes the comparisons between the discrete categories.

Box plot: Python box plot tells us how distributed a dataset is. Another use is to analyse how distributed data is across datasets. Such a plot creates a box-and-whisker plot and summarizes many different numeric variables.

let’s go ahead and perform the task with the dataset and plot the graphs.

After learning enough about Pandas and Numpy we are good to go further with understand of Matplotlib. We are going to learn how to make use of this Library what are the advantages of the Matplotlib and how this helps a Data Scientist to take a visual representation of it.

Matplotlib is a comprehensive library for creating static, animated, and interactive visualizations in Python. Matplotlib makes easy things easy and hard things possible. Create. Develop publication quality plots with just a few lines of code.

Matplotlib is not a part of the Standard Libraries which is installed by default when Python, there are several toolkits which are available that extend python matplotlib functionality.

The simplest way to install Matplotlib is to download and install the Anaconda distribution of Python. The Anaconda distribution of Python comes with Matplotlib pre-installed and no further installation steps are necessary.

If you are using Sage Maker or Google Collab, the above step can be ignored.

In our previous session we have learned about Matplotlib in python and Matplotlib has proven to be an incredibly useful and popular visualization tool, but even avid users will admit it often leaves much to be desired.

Seaborn is a Python data visualization library based on Matplotlib. It provides a high-level interface for drawing attractive and informative statistical graphics

Seaborn provides an API on top of Matplotlib that offers sane choices for plot style and colour defaults, defines simple high-level functions for common statistical plot types, and integrates with the functionality provided by Pandas Data Frames.

The main idea of Seaborn is that it provides high-level commands to create a variety of plot types useful for statistical data exploration, and even some statistical model fitting.

We will solve some basic examples to make enough understanding Seaborn in python.

In this session we are going to talk about File Handling in python without using pandas.

Python too supports file handling and allows users to handle files i.e., to read and write files, along with many other file handling options, to operate on files. The concept of file handling has stretched over various other languages, but the implementation is either complicated or lengthy, but alike other concepts of Python, this concept here is also easy and short. Python treats file differently as text or binary and this is important.

Each line of code includes a sequence of characters and they form text file. Each line of a file is terminated with a special character, called the EOL or End of Line characters like comma {,} or newline character. It ends the current line and tells the interpreter a new one has begun. Let’s start with Reading and Writing files.

In this session we are going to talk about Web Scraping, we will be learning about the Beautiful Soup library for web scraping, The internet is an absolutely massive source of data. Unfortunately, the vast majority if it isn’t available in conveniently organized CSV files for download and analysis. If you want to capture data from many websites, you’ll need to try web scraping.

Web scraping is a term used to describe the use of a program or algorithm to extract and process large amounts of data from the web. ... Whether you are a data scientist, engineer, or anybody who analyses large amounts of datasets, the ability to scrape data from the web is a useful skill to have.

First, we need to import all the libraries that we are going to use. Next, declare a variable for the url of the page. Then, make use of the Python urllib2 to get the HTML page of the url declared. Finally, parse the page into Beautiful Soup format so we can use Beautiful Soup to work on it.

To extract data using web scraping with python, you need to follow these basic steps:

• Find the URL that you want to scrape.

• Inspecting the Page.

• Find the data you want to extract.

• Write the code.

• Run the code and extract the data.

• Store the data in the required format.

Web scraping is an automated method used to extract large amounts of data from websites and data on the websites are unstructured. Web scraping helps collect these unstructured data and store it in a structured form. 

Don’t worry if you’re still a total beginner — in this tutorial we’re going to cover how to do web scraping with Python from scratch, starting with some answers to frequently-asked questions about web scraping. Then, we’ll dig into some actual web scraping,

In this session we are going to talk about Computer Vision which means will be making use OpenCV library and the environment that we are going to use is Jupyter Notebook in Anaconda or Google Colab instead of AWS to prevent costs incurred while using Kinesis data streaming services.

Python is a programming language that aims for both new and experienced programmers to be able to convert ideas into code easily.Implementing CV through Python allows developers to automate tasks that involve visualization. While other programming languages support Computer Vision, Python dominates the competition.

OpenCV is a library of programming functions mainly aimed at real-time computer vision. 

Using this, all of the OpenCV array structures gets converted to/from NumPy arrays. This makes it easier to integrate it with other libraries which use NumPy. For example, libraries such as SciPy and Matplotlib.

Let’s se how to make use of OpenCV and learn its applications and see some of the examples. In this lecture, we are focusing on images and videos. In the next lecture, we will work on real-time Data.

In this section, we will use OpenCV to give you an introduction to Computer Vision. At the end of this session you will be able to build your own face detection system. Computer vision is widely used in a lot of real time applications like Augmented Reality, Driver Drowsiness detection etc. This section will give you an insights to the advance level operations that can be done using OpenCV .We will be also showing you hands on demonstration of how you can built your own face detection system.

In the previous session we have learned how to make use of OpenCV and we had use some of the major operations by using OpenCV and ran certain task in this session we will be using the same information only the application would be based on video data.

Computer Vision is one of the hottest topics in artificial intelligence. It is making tremendous advances in self-driving cars, robotics as well as in various photo correction apps. Steady progress in object detection is being made every day. Generative Adversarial Networks (GANs) is also a thing researchers are putting their eyes on these days. Vision is showing us the future of technology and we can’t even imagine what will be the end of its possibilities. Image processing is performing some operations on images to get an intended manipulation.

Python is an excellent choice for these types of image processing tasks due to its growing popularity as a scientific programming language and the free availability of many state-of-the-art image processing tools in its ecosystem.

We are going to run face detection and eyes detection to capture features of the human face. We are using a pretrained model and it’s a real time video so you will get the real time experience and which can help you to make your own model too. So, let’s dive in to the codes now and learn real-time computer vision by using OpenCV.