11. [DVA] CloudFormation Intrinsic Functions
So the exam may ask you about interesting functions, and we’ve seen them. But here is a list of the ones you should absolutely know for the exam. The ref function, the get attribute find and map, the import value, the join sub, and the condition functions So let’s go ahead and have a quick memory refresher on what they are. The ref function is the most important to me. It used to be referencing parameters, and if you referenced a parameter, it would return the value of that parameter. So we’ve seen this in practice when you provided the security group description and resources.
So if you reference another resource in your confirmation template, it will return the physical ID of the underlying resource. So, for example, if we reference an EC2 instance, we will get the EC2 instance ID. Now the shorthand for this is going to be exclamation point ref, and this is what it looks like. So here’s a quick example. We create a subnet, and for the property VPC ID, we refer to the VPC that was created before, so what we get out of it is the VPC ID. As I said, when you reference an identifier, it returns to a physical ID. It’s super important to know. Now, how do we get more information from the resources? Because if we can only get the ID information out of the resources using a ref, then we’re pretty much stuck for some things. As a result, we can use get at, and attributes can be assigned to any resources you create. And so, to basically get a list of all the resources and all the attributes that get exposed from these resources, you will have to look in the documentation. So let’s take a quick look at it to get a feel for it. So if we look at our EC2 instance and go to return values, we can see that there is a ref. So the ref will return the ID of that instance.
Okay? And then, using the get attribute function, we’re able to get the availability zone, the DNS name, the private IP, and so on. So all these things are exposed. So to know what attributes are exposed in each resource, you have to go to the source and look at the documentation. Now, for example, if we want to get the AZ of an easy-to-use machine, we thought we could do it. Here is our resource block. So we have our EC2 instance with an image ID and an instance type, and we want to get the AZ out of it. For example, when we create an EBS volume for this, we’re going to create a new volume under resources, and we’ll say the type is an EC two-volume condition if we had one from before. But the important thing here and there is that if you look at the properties, the size is 100, and the AZ is using the get attribute function of the EC two-dot availability zone. So the EC-2 instance is generated directly from the name on the left, and the dot indicates which attribute we want to obtain from it. So we’d like to obtain the availability zone. So this is quite a popular exam question as well: how do we get this attribute of this resource? And the answer is to use the get at function you have for “find in map,” and we’ve seen this before. We use this finding map function, and the shorthand syntax is that we have to specify the map name, the top-level key, and the second-level key. And so here is a quick summary. In this example, we were using the Find in Map function to find the image ID based on the region we were in and the type of architecture (32 or 64 bits). Import value is to import values that have been exported as outputs in other templates, and for this we use the import value function. So, remember, when we import a value, we simply give the import value name that was exported, and everything should work fine.
Now for join, we haven’t seen it yet, but we can join values with the delimiter. So this is the shorthand syntax we should define for join. We give the delimiter name, and then we provide a comma-delimited list of values to make it slightly more concrete. To create the A column, B column, and C string, we can use the join function with the column, and then we specify A-B-C on the right hand side. So if you do programming, this is quite a common function in programming, so just know about it; sometimes you’ll be asked, “Hey, what is the output of this interesting function?” And if you use join, you know that you have to just put the columns in between the A and the B and the B and the C. OK? Finally, we have the sub function.The sub function is shorthand for substitutes. It’s super handy; it allows you to substitute values within strings, so you can use it with references or pseudo variables, and the string must contain this dollar sign and open with the variable name to do a substitution. So this is a bit obscure, but just remember substitution and it will totally make sense when you see it, if you see it in the exam. Okay? Sub is to substitute values. Finally, conditions are when we define, for example, a condition to only create the production resources, and we can use a bunch of functions in there, so the interesting functions we can use are and equals if not and or, and maybe others that will get added to confirmation over time. but that’s all. For the interesting functions you should use, you will recognise that the function is called inter when there is this FN column or when you have that little exclamation point before a name that represents a function. This is all you need to know about CloudFormation functions. I hope that was helpful, and I will see you in the next lecture.
12. CloudFormation User Data
All right, let’s get advanced with cloud formation. Now let’s talk about user data. So we know we can have user data when an Easy Two instance launches, and we can set it through the console. But there is also a way for us to directly include this in cloud formation directly. And so for this, we’ll pass the entire script through the function called FN base 64, which will convert our script to a base 64 string. That is what user data needs in CloudFormation. So the good news is that when you run the script, all logs, not just confirmation logs, but all user data in general, will be in Varlogcloudinit login. We’ll see this in a second. So let’s see how we can do this right away. Okay, so I’m in cloud formation; make sure you’re in the North Virginia region as well. And there’s a new, redesigned bias confirmation console. It doesn’t change much, but we’ll just click on it just to make sure that it looks like something you may get to see in the future.
All right, so we have this new UI. It’s exactly the same thing; don’t worry. So let’s make a stack so the template is ready, and then we’ll use and upload a template file. But before that, let’s go and see what the template is. So we’ll upload three user data files in YAML. And so if you look at this template, it has one parameter, which is the SSH key we’re going to use. As a result, make sure you have a key pair on hand in US East 1. If it’s not created, then we’re going to create an instance, and it will be in US East 1A. It will use this AMI. This is why we are unable to use USEast One. because AMI is region-based, and then we’ll have a key name refss key. So we’ll be able to SSH into our instance. And then we have an SSH security group that allows us to SSH on port 22 and also view on port 80 because we’ll install a web server. And so the important thing here is user data. As you can see, we pass through the interesting function of base 64.
We pass a script, and this whole script is preceded by a vertical pipe. So the vertical pipe here means that this whole thing is one giant string with new lines. If you delete this, it won’t work. So you need to have that vertical pipette say that all this is a script. And so in the script, it’s very simple. We’ve seen this before. We install HTTPD, power on the system, and then type hello world into the index HTML. So that’s about it. Let’s go and have a look at how we can do it. So we upload a template file. So I’ll choose it and its user data, and then I’ll click on Next. And so the stack name here is—I’ll just call it user data example and SSH key. If you do have a SSH key, it will show up here. Otherwise, you can go to EC 2. So if you go to service EC 2, then on the lefthand side of EC 2, you would go all the way to key pairs, and in key pairs, you could create a key, name it whatever you want, and then it will download it for you. So for me, I’ve already done that. So I’ll just use this confirmation key pair that I have. I’ll click on “next,” and then I won’t specify any option. Click on “next.” The template is an s, and the parameter is SSH key, which equals a confirmation key pair. Everything looks good, so I’ll click on “create a stack.” So let’s go ahead. And now we’re just going to have to wait for our EC2 instance to come up and ensure that the EC2 user data gets applied.
Okay, so it looks like my stack is now complete. So if I go to resources and refresh, I can see that there is my instance in a security group that was created. So I click on my instance, and that will take us straight to the EC2 management console. And in there, we can look at the public IP. For example, here it is. When I visit the public IP URL, I get a Hello World from user data. The cool thing I want to show you is that we can SSH directly into this instance, and we can see that the user data gets executed by doing caver log and then cloud init log. And this is where all the user data output will go. If you press enter, you can see that, for example, this was the command when we were running yum install httpd, so everything got installed and we get the output right here. Then systemctl started http, enabled http, He’d you from user data, and said cloud in it has finished at this time. So if there was any issue in our EC-Tour data script, we would look into that file again. We would examine the varg cloud init output log to determine where the error occurred. which is quite nice when you debug. But something to think about right here is that we created our stack, but our stack was not complete working.
Even if our user data did not work, we would still have gotten a good green tick. So we have to think about how to solve that problem. and we’ll see this in the next lecture. But for now, you’ve learned how to use user data with cloud formation. Congratulations. I will see you at the next lecture. And don’t forget to always delete your stacks right after you’re done.
13. CloudFormation cfn-init
All right, we’ve seen user data, but it’s pretty clunky as soon as you want to have something quite complex and quite structured. So for this, we’ll use CFN init, and there is this block in the metadata of the resource that we’ll see in a second called “Cloud Formation” in it. And if you include it, you can use a script called the CFN script, which stands for “Cloud Formation,” and it helps make basically complex EC2 configurations readable, and the EC2 instance will query our cloud formation service to get the init data and then apply it. And then, as soon as it applies it, all the logs will go to the VAR logcfninit logfile, and we’ll see it in a second. So why does it matter?
Well, you’ll see in a second; I’ll show you. It looks quite a bit more readable than our script. So we have the confirmation service, and it will launch an EC2 instance. On this EC2 instance, as part of the user datascript, I will run CFN Init, and the action of such will cause the EC2 instance’s query confirmation service to retrieve the init data and then apply it. So let’s see how this works now through an example. Okay, so as we can see in three user data YAML files, we had this giant script base 64, and it was a simple one, but it could get way more complicated. Think of all the things you want your EC2 instances to do when they start. So now if you go to CFN init YAML, we have the same parameters and a set key, and we have an instance as well running the same AMI with the same key name and the same security groups. But now the user data is going to be a little bit different. It’s still a base 64, but now we’re on top of it.
Passing through the sub-function, we saw that vertical pipe, and here’s the first thing we’re going to do: update the CFN bootstrap script to get the latest version. And this is just something you have to do. And we have them installed already because we use a Linux AMI, a Linux 2 AMI, and then we’re going to start the CFN init script. So we’ll execute slash opt slash slash bin slash CFN init. Then we pass in the stack ID. So this is a stack ID that represents a pseudo-variable in our stack. Then we want to get the CFN from my instance’s metadata, which is right here on line 26. So we’ll retrieve this metadata, specify the region, which is a stack region, and then, in case of error, say “exit fail” to run CFN in it. So, oops, the metadata itself The metadata itself is going to install a simple Apache http page, and it has the AWS cloud formation in it block. And in there, there’s a config. And we can now read that configuration. I won’t walk you through how I wrote this. The important thing is to know how it understands. for the Sysops exam. So here we have packages, and we say, “Okay, from Yum packages, I want to install HTTPD.
” So here we’ve just said “install HTTPD,” and we should do the exact same thing as when we did Yum install HTTPD before. So we’re good here. Then there are files, and Files says that this file at VAR www HTML index HTML must have some content, and the content is all of these things. So here, as you can see, I had more stuff at H one block, a P block, and it was easier because I could use that vertical pipe again, whereas in the user data script I had to use this echo thing, which was a bit clunky. So here we have more content, more freedom, and we can also set the mode of the file to be equal to 0 0 0 64. Then we can set some comments; we can say “hello,” and we’ll just run the comment “hello world.” And then, finally, we can say that services should start, and we say, “Okay, the HTTPD service should be enabled, and we should ensure that it is running.” And so the cool thing about this cloud formation is that I think it is way more readable than the user data script, which is a basic bash script. As a result, CloudFormation would be used in it. The rest is the exact same thing, but let’s take a look at how this works in the console. So here’s the console, and we’ll go create a stack. The template is complete; we will upload it and select the file Cfninit YAML. I’ll click “next,” then call CFN init example, and for the SSH key, I’ll use the same key as before, then click “next,” and finally, I’ll create a stack. Okay, now our stack is created, and I’ll just wait a minute. Okay, so my stack has been created, and if I go to my E2 management console and refresh and see all my instances, this one has been running, so I can go to the public IP and make sure that the CFN init worked. And it says, “Hey.” Hello. Well, from the EC2 instance, this was created using CFN in it.
So it’s pretty cool that it works, but the cool thing I want to do is SSH into the instance and see what exactly happened. So let’s just go and SSH into this instance, and we have to remove the HTTP copy and paste the mistake. Here we go; we’re inside of it. So now, as I said, there is the far log cloud in its output log that we can look at. And as you can see, we just ran this command CFN in there, but we didn’t really get any output out of it. We don’t see anything. So to see the output of this command, what you have to do is look at the file named VAR log or cfninit log, and this file will contain the output of our CFN init script. So as we can see here, it’s installed HTTPD, it did the command “hello,” it enabled the service HTTPD, and it started the service successfully. So you can see a lot of things in this VAR log directory. You can see all the logs that come out of these confirmation things, so it’s pretty cool. You can also look at this CFN in its command log if you feel it could be interesting as well. But the idea is that you need to understand that if you needed to troubleshoot everything, they would be right here. And here we get the actual outputs of all the commands, which, if we wanted to debug for whatever reason, we could look at this.
So Yummy installed httpd; it worked flawlessly, and then I typed hello, world at the very bottom. It said, “Hey.” Hello, world. And it was successfully completed. So that’s what I find really cool. Now we have more structure, and we have more files to look at in case things go wrong. And I think it is a better way to configure your instances. So the only issue we have is that this instance is still being created, so we have a complete instance, but it makes no difference whether or not our CFN’s script works. We’d like to complete the creation process only after I’m confident that this has worked. So how do we do this? So, I’ll see you there.
14. CloudFormation cfn-signal and wait conditions
Okay, so as I said, we still don’t know how to confirm that the EC2 instance got properly configured and that it was successful after doing a CFN invite Maybe it failed; maybe we don’t want confirmation to complete because our configuration of our instance has failed. And so for this, we can use the CFN signal script. And so for this, we’ll run the CFN signal command right after the CFN unit command, and we’ll tell the confirmation service to keep going or fail based on that outcome.
For this, we’ll need to define a wait condition, and the wait condition will tell the confirmation templates to wait until they receive a signal from CFN signal. So we’ll block the template until it receives a signal and attach a creation policy that works on EC 2 and ASG as well. But the creation policy will dictate how many signals you want to receive and for how long we’re willing to wait. No worries, we’ll see the code in a second, so keep on hanging. However, in terms of the diagram, we have the cloud formation service, and it will launch an easy two instance, just like before. This time, though, there will be a waiting condition. So the wait condition will just wait and put the confirmation service in hang mode, and it will wait until it receives a signal. So in the meantime, the easy-to-instance will run the CFN init script, and we will query the confirmation service to retrieve the init data just like it did before. But then after it’s done, we will run the CFN signal script and we will signal the weight condition that we’re done, and then confirmation will be happy and we’ll be able to move ahead. So let’s see how this works in practice. So in the Cfninit YAML, as we can see, you ran the CFN init file, but that was nothing afterwards. So let’s go to the CFN signal and we’ll look at this. So the exact same parameters, the exact same MYAC-To instance But now if you look at the user data after the CFN init command, we’re going to run the CFN signal command, and it’s basically going through this little minus-e dollar question mark.
So this is a special bash, but it basically says if this works, then send a good signal; if this doesn’t work, then send a bad signal, and we’re going to signal our stack and we’re going to signal our weight condition. So, in this region, I’ll go over the simple weight condition that we’ll define right here at the bottom in a second. And so this is saying if this worked, signal the weight condition that it’s all good; if it didn’t work, then still signal the weight condition that it didn’t work, and then cloud formation will draw its conclusion. The metadata itself hasn’t changed and has the exact same cloud formation in it. So because it worked before, it should work now too. Okay, let’s scroll down. And here is our weight condition. As a result, our current weight condition is a confirmation weight condition. And the creation policy states that we are awaiting a resourcesignal, and the timeout will be, and I’ll just set it to two minutes to be fair. So PT two minutes, that means I’m waiting for a resource signal for two minutes, and when I receive it, I’ll be happy. You can also specify accounts if you want to. You could say count two, but that would make no sense because we’re not sending two signals. That is, if you create multiple simple instances for us, it may count as one because we are only waiting on one instance.
So I’ll just leave it like this. So this sample weight condition is waiting for one instance to signal something within two minutes, and that signal should be obviously positive. And we know that the signal has been sent to it through the command right here. It says “resource: sample weight condition.” The rest is the same. The SSH security group has the same rules. And so the only things we’ve added to this stack are the weight condition and the CFN signal script in our user data. So let’s take a look at how this works. So we are in confirmation. I’ll create a stack, upload a template file, and this time I will upload Cfnsignal YAML. Click on Next, and I will just say CFN Signal Example, and I’ll just remove the typo because that’s pretty bad. Okay, now the SSH key is going to be the same as before. Click on Next, and on the bottom, click on Create Stack. So now the cool thing is that I will not pause the video because I want to show you exactly how things happen in real time. So we’re waiting for things to happen. The SSH Security Group is making great progress, and the sample weight condition is making progress too. Now we’re just waiting for our EC2 instance to start up. So my instance is now in “Create in Progress.” So if I go to my EC2 management console, I should see my EC2 instance that is starting right now. But as you can see, my weight condition is still in Create in Progress.So it’s still waiting. So I’ll refresh again and wait for it to happen. Okay, so my simple-to-instance is now up and running, and if I return to confirmation, this is usually when things end. It would say my instance is complete. So it’s going to show this in a second.
Create Complete. Here we go. My instance. Create complete. And so the stack is still not completed because the simple wait condition is still in effect. Returning to stacks, we can see that Create is still active. So it’s because the wait condition is waiting for a signal, and this will not happen until the EC2 instance has configured itself and uses the CFN signal. So I’m just going to wait a little bit longer for the CFN signal to arrive. And here we go. The sample weight condition has received a success signal with this unique ID. And so that was about 20 seconds after my ECTwo instance had been created, and because it received a proper success signal, we went into the create complete stage. And so that’s the cool thing. The condition had to wait 20 seconds for it to complete. Now, if I go to my EC2 management console and open up my public IP, I will know for sure that it has worked because the CFN signal sent a positive signal for confirmation. So that’s a really cool thing. Now, with weight condition, we are able to basically say, “Wait a little bit until you can say, “Okay, your stack has worked,” because this easy instance will be able to send a signal right away in the next lecture.” We’ll see what happens when things fail, because that’s a very important test case for the examination. But that’s it for now. We’ve just seen that a wait condition allows us to wait for something to complete before we receive a signal. As usual, I will go ahead and delete my stack, and I will see you in the next lecture.
15. CloudFormation cfn-signal failures troubleshooting
You must be able to troubleshoot why the wait condition did not receive the required number of signals from an Amazon EC2 instance, which is one of the most important questions on the exam. So, number one, it’s maybe because the AMI you’re using has the confirmation helper scripts installed but not yet installed. So you need to make sure that it has them. Obviously, if it doesn’t have the help of scripts, you go online to the documentation and figure out how to download them to your instance. Then you need to verify that the CFN init and CFN signal commands were successfully run on the instance. And as we see, there are logs such as the VAR log, the cloud init log, and the varlog CFNinit log that help us debug the instance launch. Very important. Then we can retrieve the logs by logging directly into our instance using SSH. But as we’ll see in this lecture, we won’t be able to do so until we prevent a delete instance on failure. So we’ll see this in the next lecture, but in this lecture we’ll see how things would not work, which will be quite cool.
And then finally, a very tricky one is that your instance must have a connection to the Internet. And so if it’s in a VPC, basically our instance should have internet connectivity through a NAP device if it’s in a private subnet or through an internet gateway if it’s in a public subnet. And we’ll see this in detail when we do the VPC section. But the idea is that if our instance cannot talk to the Internet, it cannot talk to the confirmation service. And if it cannot talk to the confirmation service, well, there is no way for the confirmation service to receive a signal, and things won’t work. So, for example, if you want to know if you have access to the confirmation service from the instance, you can use the AWS Amazon.com curl command. Very simple. So let’s go take a look at how we can trigger a failure for the weight condition. All right, so in CFN signal failure YAML, now it looks like the exact same thing, but the thing that is a failure is that my command hello will say boom, and then we’ll exit with the error code 1. And so when you do exit one, that means that is a bad exit code; the good exit code is zero; one is a bad one. So this will basically trigger a CFN init failure. And so we will signal a failure to confirm. And so let’s go see what happens in that case. So we go to our stacks, we create the stack, and we’re going to upload our failure stack. So it’s number six. Here we go. And click on “next.” Now I’ll just say that in the event of a CFN signal failure, for example, the SSH key will still be the same. And next, and all the way to the bottom, create a stack.
All right, so now we are going to have to wait just a little bit, and I’ll pause the video for the EC2 instance to get created and to report a failure. As we can see, my instance is now in “Create in Progress.” So if I go to my EC2 management console, I should be able to see, yes, my instance running, so now it should be calling CFN in it, and we should wait for it to happen. So my simple wait condition is still active, so it’s essentially waiting for a signal. So now I’ll just refresh and wait a little bit. My instance is now created, so CFN init should be running within a minute. And here we go. We get a “failed to receive one resource” signal within the specified duration. So I think that was even a timeout. This one isn’t even pinging with the CFN signal. So we fail to receive a signal within the specified duration. So Create failed for the sample weight condition, and as a result, my entire CFN signal instance and stack are going to roll back. Rollback means that my instance, as well as my sample weight condition, will be deleted. So, when I go to my Etmanagement console, my instance is currently shutting down. And so the cool thing is that, yes indeed, if the CFN signal doesn’t work, or if the CFN in it doesn’t work, or whatever, then obviously the wait condition fails, we’re happy, and we should roll back. But now the little problem is that I can’t debug why it failed because, as you can see, my instance is shutting down. So that’s a bit of a problem for me, isn’t it? because I can’t SSH into it. So we’ll see in the next lecture how to do this. But the cool thing is that now we get a sample weight condition that fails, and so we’re able to debug why no resource signal was being received. It’s because there was an error in our script. So I’ll see you in the next class.
