18. CloudWatch Logs – Logs Subscriptions + Kinesis Data Firehose…
So our EC two instance sends the log to the access log log stream into Cloud Watch, and this is a streaming log. So this is all real time, okay? And as we go ahead and go to the website, we will get logs in here. So then what if you want to analyze this log in real time and to do real time processing, a lovedata, we need to log data, we need to use subscription. So subscription is a way to get access to real time feed of log events from Cloud Cloud Watch logs and have it delivered to other services. And you need to remember those. There’s three of them. There is Amazon kinesis Stream, amazon kinesis Data Firehose Stream or aws Linda for custom processing, analyzing or loading to other systems.
So this is quite confusing that there are three. So remember, kinesis streams Data, firehose and Linda. Because if you go to the Cloud Watch Logs console, let’s go back to Cloud Watch logs and I click to, for example, access logs go to action for the subscription. Here. The only two options I have is aws Lambda and Amazon Elasticsearch service. So you may say. Wait, stefan, we just read that it is kanisa streams kanye’s data, firehose and lambda. So here we don’t see kinesis. This is weird. And now we have Amazon Elasticsearch Service. So this is really weird, right? And I said, yeah, this is pretty weird. So aws Lambda allows you to send it to any custom location, and that custom location could be Amazon elastic search service.
And so if we were to select this option, this would actually go ahead and create for us a lambda function that would be sending it to Elasticsearch. So this is a managed function by aws, but it would be using it as lambda in the back end. So here the only thing that’s a little bit weird, and let me go back to my Cloud Watch Logs agents, is that we cannot send to kinesis data streams or kinesis data fire hose from the console. Maybe this is something that they will improve in the future. But for now we cannot stream to kinesis Streams or Data Firehose directly from the console. But obviously we can do it from the cli. So now let’s have a look at some using subscription filters and there will be tutorials by aws.
So let’s just discuss from an architecture point of view what would happen if we stream from Cloud Watch into aws Lambda. Okay? A lambda function, then the lambda function in real time would receive all the data. And so in real time could do whatever you want. That lambda function could be sending it to Amazon Elasticsearch as the console allows us to do so, or that lambda function could be loading it into splunk or sending it to Amazon S three, whatever you want, whatever you do with that lambda function, this will happen in real time. Also, if you send it to kinesis Stream, this will happen in real time as well.
You could send all the data into kinesis Stream and get another kind of application reading from the Kennedy Stream and doing stuff with it. Okay, all of this would be in real time. If you use kinesis Data Firehose, though here, it will be using the underlying service firehose. And so this is a near real time service, as we saw from before. That means that even though the data is loaded in real time into Data Firehose, data Firehose will wait for the buffer to be filled, or wait for 60 seconds or 120 seconds or whatever we configured it to before it flushes the buffer into its destination. And I don’t know if you remember, but the destination of Data Firehose could be S Three, it could be elasticsearch, it could be splunk, and that’s it.
And redshift from S three. Obviously. So let’s go ahead and this is why we created a Data Firehose before. Let’s go ahead and configure a subscription filter with Amazon kinesis Data Firehose. So in this example, we’ll create, using the cli, the subscription filter. So we could create a bucket, but we already have one. So we’ll skip to Role to step two. Then we need to create the Im role that will grant Amazon kinesis Data Firehose permission to put data into our Amazon S Three buckets. But we already have done that because if we go to kinesis, we did already create a kinesis Data Firehose, and we did actually make sure that it was able to deliver logs into S Three.
So this is done already, and then we scroll down. Then step eight, we need to create an im role for Cloud Watch logs that will allow it to put data into the Kennedy Data Firehose service. So we need to create this Trust Policy. So let’s go in here. In my Cloud Watch, I’m going to create trust policy cloudwatch subscription. json And I’m going to add this. And the region is going to be EU west one. So Excellence. And next step is we need to do the Create role to create a specific Im role. And we need to specify the Assume Role policy documents. So let’s go and run this command. So I’m going to go ahead in my cli, clear the screen, and say, okay, you’re going to create a role.
This is a role name Excellence. And the Assume Role policy is going to be not here. This is going to be here and under the Cloud Watch directory. And the file name is called Trust Policy. So I’m going to copy this file name in here, and they should work. So let’s verify from within the console that the role has been created. So if I go and look for kinesis here, we get the cwl. So Cloud Watch logs to kneesys. Firehose role. So here we go. But it doesn’t have any permissions yet. So this is where we add permissions. So we’ll create this permission file. And so I’m going to just add this as an inline policy. So I’ll click on add inline policy. I’ll use json and I paste this and we’re saying okay, firehose in the region EU West One.
My account number, I’ll get it from the support center. So here it is. Something us one this account and then I will allow pass roll from this account to this role. Okay, excellent. So this is the policy that’s needed, I’ll review it and when I’m happy, I’m going to call it inline cwl to Firehose policy. Okay, we’ll create this policy and we’re done. So now we have this inline policy attached to our role. So we’re done with this step. And then finally we need to create delivery stream and a subscription filter. So as I said, this could not be done from the ui. I wish it was, but it would be simpler. But we have to use this entire command line in here to create this. So let’s go back into the cli and we need to fill in the blanks.
So here we need to say the destination ARN is going to be Firehose in the region EU West One. Then my account number as I said, which is from my support dashboard. So here we go, my account number is here. And then we need to name the delivery stream. So the one we have created in Firehose. So why don’t we go ahead and find it from the Firehose console. So this is called demo. Firehose excellence. And then the role ARN, we need to use the ARN from the account. So I’ll copy the role. Here we go. And the role name is called cwl to kinesis firehose role. And I’ll make sure that this works with the right region. So EU West One and I’ll specify the profile to be a list DevOps, press Enter and there is a log group that does not exist.
So we need to yes, for the log group name, edit it. So let’s go back up and the log group name we are interested in is called access underscore Logs Log. I think, let’s go back to Cloud Watch access underscore lug. Excellent. And the filter name is called Destination. And yes, of course the filter pattern will just remove it. So we’ll just have quote quotes and this will mean that every single log line should go into the delivery stream. So I’ll press Enter now and this worked. But now what we should be seeing is that every time we send data into the access log, can you see data? Firehose should send it into S three. So why don’t we go ahead and test this. So I need to go back to my EC Two instance, find where it is.
So I’m going to find here my logging instance and I’m going to copy the public dns and I’m just going to refresh the Hello World. I’m going to go to some random url as well to generate some 404 back to Hello World. And now I’m going to wait two minutes until kinesis Firehose hopefully delivers my data into S Rate. So now if I go back to Cloud Watch and refresh this page, now we can see at the very bottom in here for the access log that there is a subscription in here, and it says Firehose demo Firehose. So even though I can’t really do anything from the ui, the console, I can see that there is a Firehose subscription that has been created for me for that metric filter. And I can go in here and remove the subscription filter if I wanted to.
But we need to make sure it works. So to test it, so again, just refresh a few times the page. So now let’s go into S Three, and I will find my bucket and I will go to Cancer Data Firehose. Try to find the folder. Here we go. Here’s my folder. I’m going to select this log file, download it and open it. And I just opened my file. And now we can see the log events directly appearing in my file that was put into S Three by Kenny dev Firehose, and we can get all this information and so on. So the export to S Three has definitely worked. So through this lecture, I want you to remember that you can send through Cloudwatch Logs subscription filters in real time to kinesis and Lambda and Firehose.
But Firehose will be delivering this near real time. Okay, but these are the three kind of destinations for Cloudwatch Logs, and you need to remember those absolutely going into the exam, if you need a more visual way of seeing things. I like this article and it shows you what’s happening. So if I scroll down in here, we can see that the Cloud Watch logs is sending through a destination in here directly data into my Amazon kinesis Firehose, which for us, we used Amazon S Three. But it is totally possible for us to have sent it to a splunk Http event collector, for example, or use Data transformation for an Amazon lambda function and so on, and then finally going into the destination.
So this is quite a nice way. And even though, again, this is super not intuitive, even though in the ui you’re not possible to create subscriptions directly into kinesis. So can you see fairhost only lambda and elasticsearch service directly from the ui? It is definitely possible, as you can see in here, because when we go back to the access log and look at the subscriptions, it says here that it’s subscribed to Firehose through the demo Firehose description and destination that I’ve created. All right, so that’s it. It’s a long hands on, I know, but it’s quite an insightful one and definitely an automation. You need to know as a DevOps so that’s it. I will see you in the next lecture.
19. All Kind Of Logs
So now we have the access log of our httpd application into a log stream. And this access log is pretty standard. It gives us the information around the kind of network calls that were done, from which IP, when it was done, what was the get put or so on that was done, the version of Http and the error or the return code of the call, for example, 200 or 400 or 404 or whatever. Okay, so imagine this is our production application and we’d like to ensure that we don’t have too many four or four errors coming out. So we could go ahead and publish a custom metric, but that would require a little bit of code because we need to implement within our application a way to send four or four metrics to cloudwatch whenever they happen into cloudwatch metrics.
But we already have the log and the log contain this information. So why don’t we go ahead and try to create something called a metrics filter. And this will take the log file and filter it based on some criteria and create a metric out of it. So for example, if we filter for the error code 400, then we get these things and say, okay, these are 400. And what if you do 404? Currently we don’t have one of them. So I’m going to create a second one. I’m going to do slash, this does not exist and press Enter, and this obviously does not exist. So we got a 404. And so if we go into cloud watch logs, hopefully we should see that log line very, very soon. Okay, so let’s go back into the logs, into the access log. One up, sorry in here. And we’re going to create a filter metric filter.
So let’s click on Metric Filter and click on Add metric Filter. So this is at the log group level. In here we should choose a filter pattern so we can see examples. And this could be a way to match log events with 400 Http response. So let’s just use this one, but we’ll say, okay, status code equals not four star 404. Okay? And now we can just test the pattern and it says, okay, out of all these logs lines, you found two matches out of ten. And we can look at the test results in here. So this pattern looks like it’s working, so we’ll keep it as is. And this is great, but there’s obviously a lot of different syntax you could use. The idea is that if there is anything you can extract out of these log lines, then there must be a pattern for it to extract it and create a log metric filter out of it.
So here the idea of saying, okay, look at all these logs and out of this, whenever you have status code equals 400, then please export these and tell me they happened. So we have two out of ten events in the sample log. So we’ll assign a metric to it. And so after creating a metric filter, we’ll create a metric. And so this is the filter name. So this is fine, we’ll keep it as is, and the metrics namespace is going to be called Logmetrics, and the metric name is going to be called 404 Not Found. Okay? And this is a metric that is coming straight out of our filter metric. So we could have advanced settings, for example, the value and the default value. But for now we’ll keep it, keep it as is and create the filter.
Okay, so now this has been created and out of this we’ve created our first metric that is based on the logs filter. So now what if we want to have an alarm and saying, okay, whenever three times we get 404 and five minutes interval, then please send us an email. So let’s create an alarm on top of this metric. And as you can see, we go back to cloudwatch alarms on this. And so we just need to select the namespace, which is our custom metric. So logs metric and the metric name 404 Not Found. And we’ll look at the sum for five minutes and we’ll say whenever you’re greater than three, then you should alert us and send us an alarm to an existing sns topic. Would it be this one? Excellent. And click on Next and say, okay, too many 404, that’s my alarm name. And click on next. And for now we don’t have any data available.
So this is fine, it will not trigger, but so let’s create the alarm and it’s been created, so let’s wait for it to get sufficient data and go into the OK state. So here my metric is showing, okay, now and it has one that it points right here. So what I’m going to do is go to this page and I’m going to refresh it a couple of times just to trigger more 404. And what this will do hopefully, is that it will write a few log lines into my Cloud Watch logs. So if I go back to my Cloud Watch and go to my access log, and this log stream in here and look for 404, we should start seeing a lot more 404. So as you can see, the log was streaming directly from my instance into my Cloud Watch logs to the Cloud Watch Unified Agent. And so hopefully if I go back to my Cloud Watch alarm now and refresh, I should start seeing more data points if it went quick enough into Cloud Watch.
And yes. So here we go. This data point has breached my limits. So now we’re at six, thanks to my metric filters, and hopefully very soon the alarm should go into an alarm state. I won’t wait for that, but you get the idea. So the whole process again is to go to your logs. Click on here as you can see, I have one filter right here, but I could add different metric filters. And by adding a metric filter, then you can create a metric out of it and from that metric, you can create an alarm on top of it. And that alarm can do anything another alarm can do. So remember this going into the exam, the way of creating a metric filter, what it could be used used for, and the process of creating an alarm on top of your metric from your metric filter. And I will see you in the next lecture.
