AWS devops: ElasticSearch at AWS with terraform and ansible
The course consists of 5 main modules.
Teh first one is about how to deploy ES as an AWS service, which is called OpenSearch. Here I will tell you about interesting OpenSearch pitfalls that I discovered during my commercial practice. And believe me – you will not find that information at AWS documentation. You will get a ready terraform module for deploying the Opensearch cluster and we will analyze it step by step together.
In the 2d part I will show how to deploy an ES cluster at ECS (Elastic Container Service). And again we will use the terraform modules for that. I will show you why you can’t use Fargate in that case and how many problems should be resolved to deploy ES at ECS.
Next course section is about how to deploy ES at EC2 instances using docker swarm. I will prove to you that it is the easiest and at the same time the most cheap way to deploy ES at almost any cloud for most small and middle companies with a steady growth pattern. You will get ready terraform modules that will allow you to prepare AWS infrastructure and ansible modules for deploying HA ES cluster upon it.
The 4th part is devoted to autoscaling possibilities of ES cluster using different deployment approaches. We will talk about what we really can autoscale, when we can use it and what big limitations we have in case of working with ES.
5th module is devoted to the theory. If you are already familiar with ES cluster conceptions and everything that is related to shards and ES performance – then you can skip it. But if not – that is the place where you have to start from.
Initial terraform configuration aspects
AWS skeleton configuration
-
3
Terraform repository code structure and environment configurationAt that lecture we are going to investigate the devops repo structure that would be extensively used at almost all further lessons.
-
4
Pre-init terraform module
At that lecture we are going to run a pre init terraform module. that will allow us to create an s3 bucket for keeping terraform state and dynamoDB for keeping locks.
Deploying an Elasticsearch cluster using AWS Openseach
-
5
Network architecture scheme
AWS network architecture scheme
-
6
Part 1: Network terraform module - vpc, subnets, cidr
Network terraform module - vpc, subnets, cidr
-
7
Part 2: Network terraform module - routing tables and security groups
Network terraform module - AWS routing tables and security groups
-
8
Part 3: Network terraform module implementation
Network terraform module implementation
-
9
Terraform bastion module
Terraform bastion module
Deploying an ES cluster using AWS ECS
-
10
What is OpenSearch?
We will try to answer the next questions:
Can we easily use Opensearch instead of ElasticSearch?
Is it the same or different products?
Why does OpenSearch appear at all?
-
11
Part 1: Terraform OpenSearch module - security and permissions
Terraform OpenSearch module for deploying HA OpenSearch cluster - security and permissions
-
12
Part 2: Terraform OpenSearch module - cluster configuration
Terraform OpenSearch module for deploying HA OpenSearch cluster - cluster configuration
-
13
Part 3: Terraform OpenSearch module - network, logs and storage
Terraform OpenSearch module for deploying HA OpenSearch cluster - network, logs, storage
-
14
Part 4: Terraform OpenSearch module - CW alarms, variables
Terraform OpenSearch module for deploying HA OpenSearch cluster - cloudwatch alarms and variables
-
15
IMPORTANT - POSSIBLE ERRORS AND HOW TO RESOLVE THEM -
16
Apply terraform OpenSearch module, domain endpoint and opensearch dashboard
Deploy opensearch cluster, use opensearch dashboard
-
17
OpenSearch monitoring
OpenSearch monitoring possibilities
-
18
Short summary and essential notices that appeared from real commercial practice
Short summary and essential notices that appeared from real commercial practice
-
19
AWS OPENSEARCH BENCHMARKS
Deploying an ES cluster using AWS EC2 and docker swarm
-
20
What is ECS and why not Fargate but EC2
What is ECS, what are ECS types and why we cant use ECS Fargate launch type for deploying Elasticsearch cluster
-
21
ECS base components and architecture scheme
ECS base components and architecture scheme
-
22
Elasticsearch Docker file and AWS ECR
Elasticsearch Docker file and AWS ECR
-
23
ECS IAM profile terraform module
ECS IAM profile module
-
24
ECS EC2 terraform module
ECS EC2 terraform module
-
25
ECS cluster terraform module
ECS cluster terraform module
-
26
Elasticsearch at AWS ECS - short summary
Elasticsearch at AWS ECS - short summary
Autoscaling and Elasticsearch
-
27
What is docker swarm and why it is not dead
What is docker swarm and why it is not dead
-
28
Docker swarm architecture
Docker swarm architecture
-
29
EC2 terraform module
-
30
Ansible inventory, docker and docker swarm ansible roles
Ansible inventory, docker and docker swarm ansible roles
-
31
Ansible Elasticsearch role
Ansible Elasticsearch role
-
32
Elasticsearch and docker swarm - short summary
Elasticsearch and docker swarm - short summary
Course summary - which deployment scenarios is better?
Elasticsearch theory
Other courses related to Elasticsearch
-
35
Inside a cluster
You will learn how to create high available ElasticSearch cluster at production
-
36
ElasticSearch - shards and performance
You will learn what is ElasticSearch shards, how to calculate the number of shards for cluster and what are the main performance issues.
-
37
Indexing secretes
You will learn how to index millions of documents in the most efficient way with zero time downtime
