13. CloudWatch Logs – Console Overview
So now let’s look at cloud watch logs. And for this I’m going to change my region back to the one I’ve been using because I know a lot of logs have already been created. So back into the EU west one region, I’m going to have a look at what I get. So I’ll click on logs and I get all this ui. So let’s go over the ui one by one. So we have log groups. And log groups is just a giant directory of stuff that contains log streams and each log stream will contain the log itself. So if I go to code build and I go to my web app could buildmaster, I have different log streams. And it’s arbitrary the way you want to name your log streams here because this log stream was managed by code build could build used the build ID in the log stream name.
So if I click on this specific log stream, then I’m able to see the log of everything that happened for my code build and look at it. So this is something that’s quite familiar, but now we understand better the relationship between log groups and log streams. So same for kinesis firehose, we also get some information in case there is an event, an error event, it would appear right here. So some of these logs are managed by AWS. Usually they start by slash AWS and then the name of the service. So we have code build, cancun, firehose, lambda and so on. And here the logs that I create myself. They start by ECS first run test definition and we get a different log stream. And so this represents the task ID of running.
And so here we go. We get some information around the log for that specific task. And so we get all these tasks right here. So excellence back into my log group. I can also look at the cloud trail log group which will give me all the information flow cloud trail inside of my accounts. Okay, so we have all these things. So all these logs are here. And there is this column called expire events after. And right now it’s set to never expire. But if I know I’m receiving a lot of club trial and I want to expire data because at some point I don’t need it, it’s going to be an S three anyway. Maybe I want to say retain only one month of data within this one log group.
And so because I apply this at the log group level, all the log streams within this log group will have the same setting. And as you can see here at the log stream level, I cannot apply a retention setting. Okay, so here we have metrics filters and we’ll see what that means in a greater detail in one specific lecture. So for now I won’t go over it. And then finally, if you go in one of these functions, you are able to delete a log stream. If you wanted to, you could create a log stream and just navigate whatever you want. And this will make an api call. oops, and finally I have to cancel that. And finally within the log stream, you are able to filter for events and search, for example, for a four or four if there is one, and find all the four or fours in there.
Or if you’re looking for 200 type of events, then you would have 200 and so on. So there is some searching capability. So this is just a very high level overview of Cloud Watch. We’ll see how we can data include cloudwatch for now, we’ve seen that some AWS services do this for us, but how would we do this ourselves? How would we put our application logs into Cloud Watch? We’ll see this in the next lecture by looking at the Cloud Watch log vision. So see you in the next lecture.
14. CloudWatch – Unified CloudWatch Agent Part I
So let’s go through the new Cloud Watch unified agents and this is a new kind of agent which does metrics and logs at the same time. So it will be a great time to look at this agent in details. So we’re going to launch an instance and I’ll use Amazon linux two and I’ll select this one I’ll use a T two micro and then for the instance detail I will create a new iam role and for that I am role. I’m first going to create create it and say it’s for EC two click on Next permissions and in there I’m going to search for policy and it’s called Cloud Watch. So let’s look at Cloud Watch. And so we have Cloud Watch agents server policy. I think this one will be great.
Let’s look at it in details. So it has logs, put logs events, create log stream and create log groups so it will allow us to log stuff. And also it has the Cloud Watch put metric data so it will allow us to send metrics custom metrics into Cloud Watch but also it has the ssm get parameter. And we’ll see this in this lecture around how we can fetched configuration of the Cloud Watch agent from ssm. So this is perfect, we’ll use this one let’s click on Next tag and Next review and we’ll call this one aws Cloud Watch role for easyto. Okay, this is great. Create the role and let’s verify that everything is correct.
So I’ll go to this role and the policy is there so let’s attach this role into our instance and here it is and for now we will not do anything else. We’ll keep this, we won’t use tags, let’s use tag. Let’s just name our EC two instance logging instance because it’s going to log something and configure Security group this is fine. Ready and launch, launch and I’ll use this key pair yes and launch my instance. Okay, so our instance is launched and we’re going to install apache on it to just have a simple web server and we’ll stream the log of apache onto Cloud Watch. So back on our instance let’s connect to it using EC two instanceconnect and that’s going to be simpler and from this I’m going to install Httpd.
So for this I’m doing studio sum install Httpd and this will install apache for me I’ll say yes and then I’m just going to create an index html file so I’ll say hello world into varindex html. Excellent. So now if you go to our instance in here and then we go to our security group to the inbound rules and we’re going to add the Http rule just so we can test whether or not Http was correctly installed. So here we go. This is our rule that has been edited and back into our Logging instance I’m going to copy the public dns and it’s saying it’s refused because we haven’t started it so that makes sense. So we’ll do pseudo systemctl httpd start httpd so let’s do it again. Start Httpd. And now this should have started my apache server.
So if I refresh this page it now says hello world. Okay, so everything is good. We have installed Httpd and now we would like to do is trim the log from VAR log and in there oops and in there we have different loggings available to us. But the one log we are interested into is Httpd. And then we’ll have in there access log. For example this is one log file and we’ll also get error log if we wanted to in here. So these are two kind of logs that we’d like to have into Cloud Watch. Okay? So let’s get back into Cloud Watch. And so the first thing we have to do, not in Cloud Watch actually, but the first thing we have to do is to install the Unified Cloud Watch agent to get started with Cloud Watch logs.
So there’s this new thing is that before there used to be a Cloud Watch agent for logs specifically and it was called the Cloud Watch Logs Agent. And there used to be a metric system and a script to send data from your EC to instance, for example, Ram and disk info into Cloud Watch metrics as well. So now they have created something called the Unified Cloud Watch Logs cloudwatch Agent. Sorry. And that allows you to both send metrics, custom metrics and logs into Cloud Watch. And the cool thing with it is that you can store and retrieve its configuration into the ssm parameter store and that will allow you to have a quick setup for all your instances if you wanted to have them all configured the same way.
Okay, so what we’ll do is we’ll go ahead through the pain of configuring it. If you go through documentation you’ll find it’s relatively complicated to see how to get started. But thankfully I’ve done this before and I’ve just summarized the information of how to do it here. So the first thing we have to do is to do webgets and download this file. So let’s get back into our EC two instance connect. I’m going to exit the root user and I’m going to issue webgets to download this Amazon cloudwatch Agent Rpm file. Then when we’ve done this we need to install the cloudwatch Agent. So we’ll do pseudo rpm minus U and then look at the file we just downloaded. So here we go.
Now it’s doing this and finally we need to run the wizard and the wizard will go through the configuration of the Cloud Watch agent for us. So let’s launch the wizard and see what it’s happening. So it’s saying which OS are you planning to use for the agents? And we are planning to use linux. The reason why it’s not smart in code smart and saying well, you’re using linux so obviously you want linux is that we could configure the Cloud Watch agent for Windows directly from the window from the linux machine and that would be fine. So that’s why but here we’re doing linux. So one and are you trying to fetch the default region for the east metadata? Are you using EC Two or on premise host? We’re saying, okay, we’re using EC Two.
Which user are planning to run the agent as of? So we can use Root, cw agent or others? I’ll use root. Do you want to turn on the stats ddn? So this demon is used to collect stats D metrics from your applications and this is something that could be quite nice because if your application is a stats D endpoint, then the Clywatch Unified Agent could send this directly into Clywatch. So we’ll say yes, although we will not use this and so on. What port do you want to use? 8125 is absolutely perfect. Okay, now what is the collect interval for the statistic demand? We’ll keep it as 10 seconds. And what is the aggregation interval? We’ll keep it as 60 seconds. Okay, do you want to monitor metrics from collect D? So this is another kind of demand we can collect metrics from, so we’ll say yes.
And do you want to monitor any host metric? For example, cpu, memory, et cetera? Yes, we definitely want to have cpu and memory. Do you want to monitor cpu metrics per Core? And this is something we’ll say yes to. So here, thanks to this Cloud Watch Unified Agent, we’re able to get cpu metrics not just at the aggregate level, the way we see it in the Cloud Watch console, but per Core. So this is great. And do you want to add easy two dimension, for example, image ID, instance ID and so on into the metrics? We’ll say Absolutely. And then would you like to collect your metrics at a high resolution at submitted resolution? And this is something we’ve seen for custom metrics. So we could be every 1 second, every 10 seconds, 30 seconds or 60 seconds.
And to make sure that we don’t pay overpay, we’ll just keep it as 60 seconds. But these are all the options that we can get because we have a custom metric. What default metrics config do you want? Basic standard in advance. And I’ll just ask you to refer to the documentation for this. But we’ll just keep it with Basic and this shows us the configuration that we have as follows for now. So we’re saying okay, the Agent is collecting every six second run as root and here is the dimension that we need and here’s the metrics that we collect and so on and we’re going to collect the disk and also the memory and the memory is used person.
So we’ll get some Ram information into Cloud Watch logs and so on. So are you satisfied with the config? We’ll say yes, and now we’re done now he said do you have any existing Cloud Watch logs here? So this is the old agency because if we had the old agents we would be able to import a configuration file directly into this unified agent but because we don’t have one we’ll just say no. And do we want to monitor any log files? We’ll say yes and log file path is going to be VAR log httpd access underscore log and this is going to represent the access log of our apache server and the log group name could be access Logs. Excellent. We’ll just keep it simple and the log stream name will be instance ID. Perfect.
Do you want to specify any other files? Yes, we’ll say yes absolutely and this time it will be varscore Log Excellence and we’ll keep everything as default. And you want any additional files? I’ll just say no. And this should make it fully ready. So here we go. So we have the entire configure here and as we can see now there is a logs part in this json configuration which says okay, you should collect these files and the collect list that you need to have is these file path with this group name and stream so everything could be editable afterwards. Right? So I say please check the above content of the config and the config is also located in this file name so we can go ahead and copy this.
Excellent. And you can edit it manually if needed to. And the question is do you want to store the config in the ssm Parameter Store? And the answer is yes. What Parameter Store names you want to use for your config? And you need to use Amazon Cloud Watch prefix if you use the managed policy. So Amazon Cloud Watch linux is a great one and I don’t think it’s going to work unless we would have the right im policy for this. So if we go back to im, remember the cloudwatch Agent server policy allows us to do a get parameter as well but we need to edit this to also do a put. So what I’m going to do is attach a policy and I’m going to type Cloud Watch again.
But in here we can do cloudwatch Agent admin policy and this admin policy does everything as before but this time it allows us to do a put parameter as well on the ssm Parameter Store. So let’s select this one, click on attach policy and now this is good. So now if we go back to our instance and go ahead and press Enter to keep this choice and say okay, eus one is a great place to get started and the aus credentials you should use is the one directly from the sdk so we’ll keep it as one. Here we go. It says successfully put conflict to Parameter Store. Amazon Cloud Watch linux program exits now. So I’m just going to pause this video right now. And I’ll see you in the next video to see what happened.
