6. Implementing S3 Backend
Hey everyone, and welcome back to the Kplabs course. Now, in the early lectures, we were discussing a remote backend and how we could have Terraform store store the TF state file within the remote backend. So in today’s lecture, we’ll actually do the thing practically, and we’ll look into how exactly we can do that. One thing you must decide is which remote backend type you want to use to store the TF state files. Now, basically, terraform has a lot of back ends. You have art factory, console, and so on; you have S3, Swift, and so on. So depending on your requirements, you can use one of them.
So, in today’s lecture, we’ll look at S-3 and how we can use it to implement Terraforms remote state back end. All right? So let me quickly show you the structure. So I have a file called “Kplabs remote heifer back end.” So this is basically the directory, and within the directory I have an EIP underscore TF, which basically creates a simple EIP resource. This is just a three-line code, and within the provider’s TF, I have my access and secret key. So let me actually open up the console, and I’ll just go to the directory. All right, so if I do a Terraform application, let’s just wait. So what this will do is basically create a simple elastic IP resource. All right? So it has created a simple elastic IP resource.
Now, you see, along with that, it has created the TerraForm TF state file locally. Now, this is what we do not intend to do. We intend to store this file remotely. So let’s quickly do the same thing. I’ll just destroy the resource, and we’ll begin from the initial start. All right, so the resource has been destroyed. One thing to keep in mind is that if you do not specify a remote backend, the TerraForm will store the TF state file locally by default. So, if you want TerraForm to store the Testate file in some of the remote back ends, follow these steps. So within this diagram, if you see that you want TerraForm to store this TF state file in one of the remote back ends, then you have to tell TerraForm that this is the back end information and this is where you should be storing the TerraForm TF state file. So let’s begin. So I’ll just delete the TF state file, and since we are going to work with S 3, I am in the S 3 remote backend documentation. Now, the configuration is very, very simple. So let me quickly show you.
I’ll just copy this example configuration and create a file called as backend TF. All right? So this is the backend TF file, which is created in the same directory where my resources are. And I’ll use the configuration that has been presented in the documentation. Now the configuration is very simple. So here you are. Specifying the type of backend, you want, which is the S 3.
Now, within S3, what you have to do is specify in which of S3’s buckets you want the TerraForm TF state file to be stored. And that is actually specified in this specific column. So let’s do one thing; let’s go to step three, and we’ll create a new bucket where our configuration will be stored. So the bucket name would be “kplabs remote hyphen backend,” and our bucket is created. So what I’ll do is put the details in the SV bucket. I’ll say Kplabs’ remote hyphen backend. This is the first one. The second one is that since our bucket was created in the north California region, which is the west one, I’ll put the region here, and the key would be the name of the TF state file.
So let me call it the remote demo TF state. All right, this is the file name that I want, and I’ll go ahead and click save. Now, whenever you configure the back end, you have to initialize the back end as well. As a result, you must run the TerraForm in it. Okay, as you can see, it throws an error stating that there is no configuration or that it is unable to obtain the AWS credentials. Alternatives include going down, where there are two variables: the access key and the secret key. So we have to use those variables in the remote backend TF file as well. So I’ll copy and paste these two variables here, and then let’s do a Terraform in it right now. So currently, it is initializing the back end. Perfect. So the S-3 back end is now initialized. So next time If you use TerraForm, I’ll say yes, and this will result in the creation of a resource. However, now it will store the TF state file in the three buckets that we have specified. So it has created a resource. So, however, if I just refresh the page, you can see there is no TerraForm TF state file.
Now, in order to verify whether the file is already created, let’s go to Kplabs’ remote back end, and you can see this is where the TF state file is actually created. So this is one of the ways in which you can actually configure your TerraForm to store the TF state file remotely. Now, this remote back end is something that can be configured. As we have already discussed, there are a lot of remote backends. Depending upon the requirements, you can choose one of the remote back ends suitable for the use case that your organisation might have. So that concludes the lecture on remote backend. Go ahead and try this out, and if you have any questions, feel free to connect with us. I look forward to seeing you in the next lecture.
7. Challenges with State File locking
Hey everyone, and welcome back to the Kplabs course. So, in today’s lecture, we’ll go over TerraForm state file locking. Now, this is a very important feature that generally runs behind the scenes but provides very important functionality in Terraform, specifically during times of team collaboration. So, let’s go ahead and discuss more about what TerraForm state locking is all about. So generally, behind the scenes, whenever you perform some kind of write operation, TerraForm will lock the state file. Now, the locking of state files is very important, specifically during the right operation, because if someone else performs the right operation on the same TerraForm TF state file resource, then it will actually corrupt your state files.
So, let’s understand this with an example. So, let’s say that person A is terminating the RDS resource, which has an associated RDS state file, all right? Now, at the same time, just assume that this is a Git repository and multiple people are working on the infrastructure. So, person A is terminating the RDS resource, and at the same time, person B is now trying to resize the RDS resource. So, on the one hand, you are trying to terminate, and on the other hand, you are trying to resize the RDS instance. So, if two operations are performed at the same time on the same resource, which has a TFstate file, it would corrupt your TF state file. And this is the reason why. TerraForm locks the TF state file after an operation has been completed. So, to illustrate what I mean, consider the following scenario. So, let me do a TerraForm plan based on what I’ve done. I’ll just copy this a bit with CTRL C, and currently I am in the KP Labs hyphen gate directory. So, I have two command-line windows. And what I’ll do is do a TerraForm plan here.
As a result, whenever you run a TerraForm plan, TerraForm will lock the state file. So, let’s look into how that would work. So, whenever I create a TerraForm plan, TerraForm locks the state file. At the same time, if someone is attempting to perform a similar or correct operation from a different window (let’s assume the command line window as well), TerraForm will not allow them to do so. So, let’s begin. So, here I’m doing TerraForm at the same time. I am doing a TerraForm plan in the second window. Now, in the first window, the plan would work as expected. However, because I ran TerraForm Plan at the same time, you can see in the second window that the error locking state file is already locked and that you will be unable to modify it during the lock process. So if two people try to do the same operation at the same time, or I would say they try to modify or do any operation, this error would occur. Now, if I try it again, you can see that it works perfectly. So this specific locking feature is very important. Now, one of the challenges is that this locking feature is not available for all the backend types.
So we were discussing where we had actually stored our TF state file in the S 3. The locking feature is no longer available in the default use case if you store the TerraForm statefile in the S file. So let’s look into that specific part as well. So I’ll go to Kplabs’ remote iPhone backend, and this is the file. Let me do it for the same I’ll do KP Labs’ remote Hyphen backend. So these are the two console windows. We’ll do the same thing, and we’ll look into what happens. I’ll run a TerraForm plan from here, and even from here, I’ll run the TerraForm plan. So let’s look at what happens now. So in the first window, the refreshing state has occurred, and even in the second window, the refreshing state has occurred.
So you can see that there were no changes here, and the same was true in the second common line window. So, as of now, there is no state locking feature present in the configuration that we have put in place. Now, if you just do a TerraForm plan, it is not going to create much of an issue. But as we have discussed, if we are actually performing the right operation, as we already discussed one example of the right operation, someone is trying to terminate, and at the same instant, there is a second team member who is trying to resize. So there are two different right operations occurring on the same TF state file. If the remote locking is not done, then this would actually corrupt the state file. And this is the reason why, whatever backend type you choose, you have to make sure that it supports locking functionality. Otherwise, at some point in time, your state file will be corrupt.
8. Integrating DynamoDB with S3 for state locking
Hey, everyone, and welcome back to the Kplabs course. So, in the earlier lecture, we were discussing state locking. And what we’ll do is we’ll implement state locking in one of the remote backend types of S3 that we were configuring. So we already looked at the default configurations, and three of them did not support state locking. However, there is a feature through which we can implement state locking. In fact, it is actually stated here that this specific type of remote backend is standard, and it supports state locking with the help of DynamoDB. So what we’ll have to do is, if we want to have state locking based on S3, then we have to configure DynamoDB as well. So let’s take a step back. So within the variables, you see that one variable is basically related to the DynamoDB table.
And this is where you can implement state locking. So what we’ll do is go to DynamoDB. All right, so this is DynamoDB. What I’ll do is go and click on Createtable, and here I’ll say that it’s a three-state lock so that it becomes much easier. Now, within the primary key, you have to define a primary key of type lock ID. So this is the lock ID, and this is the primary key, which is what is suggested that we should be putting in. So I’ll put the primary key over here, and then I’ll go ahead and click on Create. All right, so now we have a three-state lock. This is the DynamoDB table, and this table is getting created. Meanwhile, what we’ll have to do is add this specific entry to our backend TF file.
So if you go to the backend TF file, you have to add one more entry as a DynamoDB underscore table. And I’ll basically put the table name that we have configured. As a result, this table has been created. So now, if you do a state lock, this is the table name, and I’ll click on Save. Perfect. You now have a perfect method for achieving state locking. Functionality is quite simple, actually. So now that we have configured the state locking, let’s verify if it actually works or not. So in order to do that, what I’ll do is stay in the same directory and copy this command, which is TerraForm Plan, so that it becomes much easier. Rather than typing multiple times, I’ll do a Terraform plan here. All right, so we have to do an INI. The init is required because we have actually specified one more type, which is the DynamoDB table. So the plugins related to DynamoDB have to be configured. All right, so let’s do a Terraform plan here, and it’s occurring.
And on the other hand, let’s do a TerraForm plan here as well. And here you see it says “acquiring state lock.” So the state lock has not been acquired because there was an operation being performed. So if you see this over here, let me actually open it up. If you look at the items, you will notice that there are currently no items present over here. But as soon as you do a TerraFormPlan, there will be one item that automatically gets created within this state file. you see over here. So this is the item that got created, and this is actually locking this specific remote demo hyphen TF state file. So this is what the locking functionality is all about, and it is quite simple to implement locking with S 3. So if you’re using a remote backend, make sure that the remote backend always supports the locking operation. So generally, once your TerraForm land gets completed, this specific lock file automatically gets deleted. So it only gets generated when the lock is acquired. So, this is a very simple and brief lecture, but it is very important for the infrastructure. So this is it about this lecture. I hope this has been informative for you, and I look forward to seeing you in the next lecture.
9. Terraform State Management
Hey everyone, and welcome back. Now, in today’s video, we’ll go over TerraForm state management. Now, as your TerraForm usage becomes more advanced, there will be some use cases where you will find that you might have to modify the TerraForm state.
Now, it is important that you never modify the TerraForm state file directly and manually. Instead of that, you can make use of the Terraform state command command. Now, under the TerraForm State, there are multiple subcommands that can be used to perform certain activities. So these are some of the primary lists of sub commands. Let’s quickly explore that. So you have list; you have move; you have pull, push, remove, and show. And this is the associated description for each one of them. So let’s do one thing: let’s discuss each of these state subcommands and also look at it from a demo perspective. Now the first one is on the list. So the TerraForm State list command is used to list the resources that are part of the TerraForm state file. So if you look into this command, “TerraForm state list,” it basically shows you the resources that are part of the state file. So let’s do one thing: let’s quickly have a quick demo so that this can be better understood. So currently I’m in my CLI, and if you see that there is one TF file that is available, and the name is state hypermanagement TF, So let me quickly open this up.
So you can see that this particular TF file creates an easy to instance, a simple Im user, and we store the state file remotely in an S3 bucket. And this is the reason why you do not have a local state that is available. So all of the things that are part of this file are applied and created in AWS. Now, if you do a TerraForm state list, it basically shows you the things that have been part of the state file—basically the Im user and the EC 2 instance. So in fact, if I have to show you, you have this EC2 instance and you have this IAM user, and both of them are being returned. When you do a Terraform state list, the second important one is the move. So the TerraForm State move command is used to move items in a TerraForm state.
So this command is used in many cases, specifically if you intend to rename any existing resource without destroying and recreating it. So we’ll be discussing this second point in detail in a minute. Now, one important part to remember is that due to the destructive nature of this command, it will output a backup copy of the state prior to saving any changes. So let’s do one thing; let’s quickly have a demo on that. Now, as we were discussing, these resources are already created in AWS. So if I have to quickly show you Within the AWS, you see that we have one EasyToInstance that is running, and we also have an IAM user called Load Balancer that was created. So, let us quickly access the state management. So let’s say that we want to change this web application to my EC 2. So let’s try it out and see what exactly happens when you just change this specific aspect here. So now when you do a Terraform plan, you see what exactly it is doing: adding one resource and destroying one resource.
As you can see, it’s adding a double underscore instance to my EC2 instance while also destroying the AWS underscore instance web app. So this is what we are discussing in the second paragraph: if you want to rename an existing resource without destroying and recreating it, because here, when we try to rename it, it is destroying and it is also recreating it. So this is something that we might not always want; we just want to rename the resource. That’s about it. So in such cases, you can make use of the Terraform move. So after clearing the screen, let’s try it out. So you do a Terraform move. Now, when you do that, you have to specify the AWS underscore instance that was the value earlier, which was web app. So if you do a Terraform state list, you will see that you have an AWS underscore instance web app. So this is what you will enter here, and you will also specify the new one, which is “AWS underscore instance dot my EC2.” All right, let’s go ahead and press enter. So it seems we have made some typos. Let’s try it out. So we have made a small typo here. It should be a TerraForm state transition.
All right. So now you see that it says that you have successfully moved one object. So, under state management, you’ll have my easy to. But when you do a Terraform plan over here, you should not see the destruction and recreation of the resource. So currently now it says that there are nochanges and the infrastructure is up to date. The third one is the pull. So the TerraForm state pull command is used to manually download and output the state that is part of the remote location. So basically, it will go ahead and show you the remote state. Now, this is very useful for reading values out of a state file, and you can also make use of JQ to further interact with it. So let’s quickly look into it. Now, as we were looking at this specific demo file that we have, we are making use of a remote state, which is S 3. So if you want to look into the values that are part of this specific state, you can make use of the Terraform state pull command. So let’s write it up. So I’ll say TerraForm State. Pull. And now you can see that you basically get all of the outputs from the state file that are in the S bucket. As a result, you will notice that you have an ARN associated with the in user. The name of our instance is “my EC to now” and so on. The next command is push. So the TerraForm State push command is used to manually upload a local state file to a remote state. So ideally, this command should not be used very often. This is something that is very rarely used. The next one is removed. So the TerraForm State RM command is used to remove items from a TerraForm state. In the service provider state, items removed from the Terraform states are not physically destroyed.
Now, items removed from the TerraForm state are no longer managed by TerraForm. So, for example, if you remove an AWS instance from the state file, the AWS instance will continue to run, but the Terraform plan will no longer see that specific instance. So let’s try it out. So let’s create a screen and do the TerraForm state list yet again. And now let’s go ahead and try to remove this specific instance. So TerraForm State RMScore it is. MYEC Two is an example. So now you can see that it says it has removed the AWS underscore instance MYAC 2. The instance in AWS will continue to run now that this has been removed from the state file. So if you look here, if I quickly refresh, the instance will still be running; the only difference is that it has been removed from the state file and is no longer being managed by the TerraForm.So if you quickly do a Terraform state pull now within the resource, you only see the resource associated with the Im user. There is no resource associated with the EC2 instance that is available.
Now that you have this specific EC to block, it is still available. The next time you run a Terraform plan, it will attempt to create a new EC2 instance. So let’s look into it. So you can see it’s adding something, and if you look here, it’s attempting to create a new EC to instance that’s part of the TF file. The last one is the TerraForm State show. So this command is used to show attributes for a single resource in a Terraform state.
So it can happen that a single TerraForm state can have multiple resources. So if you want to see the attributes and the associated values of a single resource, then you can run the Terraform State show followed by the associated name. So let’s try it out. So in our case, since EC2 instances are not part of the state file, the next thing we can do is create an AWS underscore IAM underscore user named LB. So let’s try it out. So I’ll go with TerraForm States. AWS.Im user LB So now, it will only show you the details associated with the Im user. So that’s the high-level overview of some of the subcommands that you can make use of with the Ten.
10. Importing Existing Resources with Terraform Import
Hey everyone, and welcome back to the Kplabs course. So in today’s lecture, we will be discussing TerraForm import. Now, TerraForm import is one of the features in TerraForm, and I am confident that you will use it at some point. So let’s go ahead and understand what the TerraForm import is all about. Now, it might definitely happen definitely.
Even if you use infrastructure as a code, I am sure there will be one team member who will create resources manually. As a result, some people may secretly create resources that the hotel is unaware of; they were created manually, and this happens. The second use case is that you are using Terraform after manually creating the resources. So it happens in most organisations where the infrastructure is already ready. And after the infrastructure is ready, you go ahead and decide that you need to use TerraForm. And in such a use case, TerraFormimport is a great utility. So let me give you an example. So let’s assume that this is the availability zone and there is an EC2 instance that is created manually.
Now, because this EC2 instance was created manually, any changes you want to make to it must also be done manually. So let’s assume that you want to change the instance type of this EC from T2 micro to MPO large. And since this was done, this EC2 instance was launched manually. Even changing the instance type would be manual. So everything would be manual. This is why, in order to bring this into the infrastructure as code, you must import this specific resource into the TerraForm structure. And this is where the TerraForm import really helps. So let’s look into how we can get this working with the help of the TerraForm import feature.
So currently, I have one EC2 instance, which is called manual. So this EC2 instance is something that we have launched manually, and these are the parameters associated with the EC2 instance. So it has a security group, a specific AMI ID, and a specific key pair name. So now we want to import this specific EC2 instance into a TerraForm structure. So if tomorrow, let’s assume that I want to change some parameters of this EC2 instance, Since this was launched manually, I will have to do things manually. It cannot be done through Terraform. So that’s the reason why we will import this specific EC2 instance into our Terraform. So next time any changes have to be made, they can be done through the terrain itself. So I have created a directory called “TF import.” and within the TF import.
I have a provider, TF. So this is a provider configuration, and that is it. I don’t really have any other things. So what we will do is import something manually; until now, TerraForm will not generate the TF files for you; it will only generate the TF state file. Always remember that. So if you want to manually import this instance, you will have to write the resource configuration yourself. TerraForm will not do that for you, although that is the feature that is going to be upcoming. But as of now, the feature of automated writing of TF files on TerraForm import is not present. So what we’ll do is create a new file. I’ll call it EC2 TF, and we’ll need to create a resource configuration based on the configuration of this EC2 instance. So let’s begin. So I’ll create an AWS resource instance and call it my EC2. The next thing is the AMI. So you will have to put the AMI in. Let’s just figure out the things that we’ll need. One is the AMI. The second is the VPC security group ID. So this is the third thing that we need: the key name.
So let’s get these three things done. So the AMI ID is the one that is specified over here. So I’ll specify the AMI ID over here. Let’s copy it and paste it here. The second is the security group ID. security group ID. We can get it from the console itself. So I’ll copy the security group ID here, and let’s add the security group ID and the key name. Again, we’ll have to refer to the console, get the key name, and I’ll paste it here. So these are the three things.
Now along with that, you also have a tag, which is associated over here. So you see, there is a tag where the key is named and the value is manual. So again, we’ll have to use the tag. So I’ll use the tag where key is equal to name and value is equal to manual. So one last thing that we’ll do is use this subnet ID. It’s always the best practice. We’ll put this subnet ID in and add the subnet ID as well. Subnet ID is the same as the subnet ID where the perfect instance is launched. So I think everything seems to be in order. Just one part is missing: the curly braces to start and end the code. Perfect. Now this is our code. So all we have to do now is connect it. So let me first verify if our code is working as expected. I’ll do a TerraForm in it, and this will download the AWS provider.
So the TerraForm import command will read the state file, create a state file associated with the EC to instance we specify, and then link the state file to the resource configuration. You will see once we run the command. Okay, so our provider has been installed, and I believe there were a few more broad entries. So, if I do a TerraForm plan instance type, you can see that we forgot one parameter, which is the instance underlining type, and this would be two micro. The manual EC2 instance that we have is basically based on EC2. Perfect. So everything seems to be working perfectly now. As you can see, we have a TerraForm strategy. Basically, it will say that we have to add one more instance. So if I do a Terraform apply, it will basically create one more instance that is identical to this specific manual instance. But this is something that we don’t really want. So we have to import the state of this EC2 instance and link it with the manual configuration of Terraform that we have written. So in order to do that, what we do is run a TerraForm import and specify the resource. So this is AWS underscore instance two of my EC two.
And along with that, you also have to specify the instance ID that is associated with the EC2 instance that is created manually. I’ll press Enter, and it will begin importing the state file and link it to the resource that we have configured.
So it has successfully imported the state file. So let’s just wait for a moment. Perfect. So now, if you will see, the import is successful. So once the import is created, you will see there is a TerraForm TF state file, and within the TerraForm TF state file, let’s just quickly verify if everything is perfect. As a result, the public IP address is 48.171.171. Let’s verify. And the public IP address is 48.171.171. So now that this EC2 instance is imported into TerraForm, any changes that we want to make to this EC2 instance have to be done via TerraForm. So let’s do one thing: let’s add one more security group to this specific EC2 instance. So I’ll go to the security groups and let me just copy a random security group ID, and then I’ll add it under the TerraForm EC-2 PF file. So let’s quickly verify.
All right, so now I have added one more security group. So this time when you do a Terraform plan, it will basically say that there will be a change of security group that needs to be added to the instance. All right, so now you see the security group has to be added. So let’s quickly do a Terraform application, and basically what should happen is that currently, if you will see, there is only one security group for remote practical. Once the application has finished, there should be one more default security group that should be added.
So the application is finished. So let’s just quickly verify by refreshing. And now you see there is one more security group called “default,” which is added to the EC instance. And this is how you import the TerraForm—or, more precisely, the manual resources to the TerraForm. So that’s all there is to the TerraForm Import Command. I hope this has been useful for you. It’s quite an easy command and not very complicated. So if you are familiar with writing the TerraForm TF state files, importing is something that will not be very challenging thing. So this is it for this lecture. I hope this has been informative for you, and I look forward to seeing you in the next lecture.
