1. Amazon Elasticsearch Service overview
In this section, let’s explore the amazon elastic search service. And this may be called as amazon es. At the exam, they might call it as amazon es or the elastic search service. All right, so just remember that and the elastic search service is a managed version of elastic search, which is an open source project, and it typically is used with the elk stack. So elk stands for elastic search, log stash, and kibana. So these are three different tools that are generally used together. So elastic search provides search and indexing functionality, logs, as provides a log ingestion mechanism. It’s similar to cloud watch logs. And kibana provides real time dashboards for your elastic search data.
And it’s a visualization tool. And this does require provisioning servers. It’s not a serverless offering. And typical use cases are log analytics, real time application monitoring, security analytics, full text search, click stream analytics, and indexing. Let’s look at the elk stack in a little more detail. Now, elk is a combination of elastic search logs and kibana. Elasticsearch provides us with search and indexing functionality. And the way it works is you send data to elastic search in the form of JSON documents using the API or log stash or even the kinesis firehose. And elasticsearch automatically adds a searchable reference to these documents in the elasticsearch clusters index.
Then we have logstash. This provides us with a log injection mechanism, and it uses what is called as a log stash agent. It’s an alternative to cloud watch logs, and you get to decide on the retention and granularity of the logs. And finally, we have kibana. It’s a visualization tool, and it provides real time dashboards on top of the data that sits in your elasticsearch cluster. This is an alternative to cloud watch dashboards, but it does have a dashboard capabilities than the cloud wash dashboards. All right, that’s about the elk stack. Let’s continue to the next lecture.
2. ElasticSearch Service patterns
Now, in this lecture, let’s look at some of the patches that we can use with elastic search service. So first we have DynamoDB. So we can use elastic search to provide search functionality for the data that’s sitting in our Dynamo tables. So you can use DynamoDB streams along with a lambda function. So the lambda function can consume the data from DynamoDB streams and push that data into elastic search. And now your Dynamo data becomes searchable. So you can use DynamoDB to read your items, and you can use elastic search to search the items in your DynamoDB table. And in the similar fashion, you can also make your Neptune data searchable using Neptune streams.
So you use a lambda function to consume a Neptune stream, and the lambda function will put this data into the elastic search cluster. So now your Neptune graph data becomes searchable. So you can read your items from the Neptune cluster, and you can search those items using the elastic search cluster. All right. Then you can also use elasticsearch with cloud watch logs. You have your cloud watch logs, and you can create subscriptions on top of your cloud watch logs, which can be consumed by a lambda function, which is internally managed by AWS.
And the lambda function can push these logs into elasticsearch to make them searchable. So the subscriptions are something that you can use to get access to a real time feed of log events from cloud watch logs. And you can consume this feed using a lambda function. Or you can also use kinesis data firehose to consume these logs. And then you can further push these logs in real time to elastic search. And elastic search makes this log searchable. So that’s another pattern you can use with elasticsearch. All right, so that’s about it. Let’s continue to the next lecture.
3. Elasticsearch Service – Multi-AZ
Now let’s look at the multiaz options in Elastic search service. So elastic search supports multiaz. So you can have up to three AZ deployments with Elastic search. So you can use a single AZ, two AZ, or a three AZ deployment with your Elastic search cluster. You can deploy these within VPC. And Elastic search uses dedicated master nodes to increase your cluster stability. These dedicated master nodes are distributed across three AZ, even if you select a two AZ deployment. So that’s important to note. So in the example here, we are using Elastic search only in two AZ, but still, we have dedicated master node setup in three AZ.
So that’s something that’s good to remember. But for production use, it’s recommended that you use three AZ deployments. And for multiaz, you should create at least one replica for each index in the cluster. And this kind of makes sense, because without replicas, cross AZ replication doesn’t happen. So this largely defeats the purpose of multiaz. Make sure that you create at least one replica for each index in the cluster. All right, so that’s about it. Let’s continue to the next lecture.
4. Logging options in Elasticsearch Service
Now, let’s talk about logging in elasticsearch service. So the elasticsearch service provides three types of elastic search logs, error logs, search slow logs, and index slow logs. And all these logs are accessible through cloud watch logs. And these logs are disabled by default, and you can enable them from within your elastic search console. Or that’s about logging in elasticsearch. Let’s continue to the next lecture.
5. ElasticSearch Service pricing
Now, let’s talk about pricing for your Elastic search service. First off, you have instance pricing, which is priced per hour. So you have three types of instances to choose from on demand instances, Reserved Instances. So Reserved Instances give you discounts on a one to three year term. And then you also have a third type of instance called as UltraWarm instance. So this is kind of an on demand or managed storage, and this is a new tier type and a cost effective way to store large amount of read only data. So that’s what you use Ultraworm Instances for.
And apart from instance pricing, you also pay for your EBS volumes. You can choose from general purpose or provisioned IOPS or even magnetic volumes. And similar to other AWS services, you pay the standard data transfer charges and elasticsearch supports manual as well as automated snapshots. The automated snapshots come free with a 14 day retention, and manual snapshots are charged on a pergb price. So that’s all about the Elastic Search service. Let’s go into a demo and create an elasticsearch cluster. All right, let’s do that.
6. ElasticSearch Service – Hands on
In this demo, let’s look at how to create an elastic search service domain. So here I am in the Elasticsearch Service console, and we can create a new domain using the Create new domain option here. And domain simply means your Elasticsearch service cluster. All right, we can choose a deployment type, let’s say development and testing, and we can go with the latest elasticsearch version and click Next. And here we configure our domain. So let’s say my es domain, all right? And we can choose an instance type. We can choose the smallest one available here. Let’s go with t too small.So since this is just a demo, I’m going to select t two small as the instance type, number of nodes as one, and we can leave the rest of the settings at the default values.
We don’t need a dedicated master node, so simply click Next and we’ll choose public Access. To keep things simple. Here you can set up fine grant access control if you like. You can configure cognitive authentication. Here you can set up your access policy. So you can choose any of these, let’s say, which is custom access policy, and then we can specify the IP address or the IMR that you should allow. So what I’m going to do is I’m going to find my IP address, and here I’m going to paste in my IP address. You can locate your IP address by googling for my IP address, and it should show you your IP address. All right, and then we can leave rest of the settings to their default values and continue and finally review this information and confirm.
And this is going to create your elastic search service domain. And the domain status is loading, and it’s going to take a while for it to become active. So let’s wait for a while. I’m going to pause the video here and come back once the domain is available. All right, now the domain is active, and you can see that we have an endpoint here. So you can use this endpoint to load data into your elastic search domain. And you can also use it to query your elastic search data.So if you click on this, it’s going to show you the information about your elastic search cluster, all right? And you can use this endpoint to load data using curl. So you can use simple curl commands like curl put command to load data into your elastic search domain, for example.
And then here we see Kibana endpoint. So Cabana is a visualization tool. So if you click through it, it’s going to load the Kibana dashboard. So this is the welcome screen of Kibana, and since we don’t have any data, is going to prompt us to try sample data. Or you can also explore Kibana. So let’s try out the sample data. Click on this and let’s go with sample web logs. The data is loaded. Let’s view it. So this is how Kibana dashboard looks I’m not going to dive deeper into this. If you’re interested, you can definitely explore this further. All right. With that, we come to the end of this section. Let’s continue to the next one.
