1. Integrating with GIT for team management
Hey everyone, and welcome back to the KP Labs course. So in today’s lecture, we’ll be discussing team collaboration when it comes to Terraform. So one thing that we have already seen is that, till now, we have been working on the TerraForm code locally. So local changes are not always good because it might happen that your hardware disc might crash or the file might get deleted due to a virus or malware, and your entire TF state files’ entire code might get erased. And this is one of the reasons why you should not always store the code locally.
Now, the second reason why it is not good to have local changes is because it is not done in a collaborative manner. So let’s assume that this is the laptop that we are working with and that we are storing entire code and TF state files here. Now, there is one more person in the team within our organisation who has joined the DevOps team, and he also wants to contribute to the code. So typically in organisations, it’s not like a one-man army that might handle the entire TerraForm code for the entire infrastructure. There will be a team that collaborates and manages the entire infrastructure through a tool like TerraForm. And in this kind of approach, since the code is stored locally, another person in the team won’t be able to directly access the code as well as the TerraForm TF state files. So this is the reason why there should be centralised management. And what happens in centralized management is that you push the code to a central repository, which is git.
And, as I’m sure you’re aware, get allows anyone on your team who wants to work with TerraForm to pull from this git repository and work on the code that you’ve been working on. So what we’ll do in today’s lecture is configure git and push a sample of code into the git repository for our demo purposes. So for the git solution, my personal favourite is Bit bucket. I have always been a fan of Bit bucket, and I have always used it. So depending on your choice, you can use the solution that you need. So for the demo, I will be using Bit bucket. Bit bucket is free, by the way, so you can use it for free. Definitely, it’s not free for enterprises. There are certain restrictions on a free account, but you’ll be able to create a free bid bucket repository.
So, what should we do? Let’s make a repository called Kplabs iPhone demo. In fact, let me refer to it as Kaleb’s Find Git to make it more meaningful. And I’ll click on “create a repository.” All right, so this is the repository. Now if you go a bit down, just click on “I’m starting from scratch,” and it will give you a link to clone. Again, if you’re using a Mac or Linux, installing a git client is a breeze. Brio install git or Yum install git. However, a few additional steps are required for Windows. So basically, you need to download a git for Windows. So, if you simply type git windows, you will be presented with the link from git sem, and you will notice that your download is automatically starting, and the download begins. So in my case, I have already downloaded the exe file to save time.
So we’ll quickly download the git client for Windows. So installation is pretty simple. I’ll use the nano editor; I am a fan of nano. So you can just select the defaults and let it install them for you. All right, so it has been completed. I’ll click on “launch git bash,” and it should launch a nice little bash for us. So we will be cloning this specific repository. So one thing that I’ll quickly do is create a new directory. Let me name this repository, or my repo, a short and simple name. I’ll name this my repo, and from my bash editor, let me do a PWT. I’ll go to my desktop, I’ll go to my repository, and this is where we will be cloning the repository. So let me copy the clone command and I’ll paste it over here. All right. So you see, it has told me that I appear to have a clone repository. That is fine, because I don’t really have anything over here. So in case it asks you for a password, So simply enter the bit bucket password that you created. So this is the git repository that I have cloned. So now if I go to CDKPlabs and git, So there are a few things that we can clone over here.
In my case, I have two files. The first is the service provider, and the second is RDS. Allow me to copy these two files and paste them into the Kplab seven git. So these are the two files that we will be looking forward to committing. So I’ll say “get add” and “commit first git code.” All right, so we’ll just configure the globe. Now you can go ahead and commit, and you can push to the central gate repository. All right, so now your branch has been pushed. So now, if you quickly do a refresh here, You can see that the empty message has vanished. And if you go into the source now, there are two files that are part of the source. Now this is how you can actually commit to git. I’m sure most of you are already aware of how you can do that, but I just thought that it might be a good revision for some of you who are new to git. So this is it. In this lecture, there are a lot of things that we will be discussing in continuation of this lecture. So this is it. I hope this has been informative for you, and I look forward to seeing you in the next.
2. Security Challenges in Commiting TFState to GIT
Hey everyone, and welcome back to the Kplabs course. So in today’s lecture, we look into some of the challenges that most organisations face, specifically when they integrate their infrastructure and code with a Git repository. So in the earlier lecture, we had committed two files to a Git repository. Now, one of them was RDS TF. So if I quickly show you what RDS TF contains, you see, one thing it contains is the username, and the second thing that it contains is the password. One of the most important things to remember is that you should never store passwords, API keys, or any other type of secret within the Git repository. It is very important to know.
Now, let me actually show you what many people do. This is our RDS TF. And, when you create RDS, you must usually provide a username and password. And a password is something that you can include in the TerraForm code. Now, what many smart people will do instead of storing the password here is use some kind of interpolation function. So let me actually use one. So I’ll say file, and instead of storing it, here is what I will do. Let me select this. I’ll create a new file. I’ll say RDS underscore pass TXT, and within this, let me actually open this up from the consoles so that it becomes much more clear. So I create an RDS underscore pass called dotTXT, which is not in the Git repository, it is outside of the Git repository.
So this file will not be committed. All right? So now, within the RDS underscore passing the TXT, what we will do is store our password. I’ll say my secret password is 505, and I’ll click on save. All right? So now we’ll use the function, which we’ll refer to as “trespass TXT.” So this is the function that we will be using. So, this is the code that many of us have seen in track in one of the organizations; they were using this kind of structure, and they said we never store any kind of passwords or secrets within the Git repository. So let me demonstrate the problem that this type of setup causes. If you’re not sure what this will do, it will read the contents of the RDS underscore Pass TXT directory. So this is one directory up. So now let’s do one thing; let’s create, or, in fact, let me open up the command prompt. I’ll go to my desktop and my repository. Now, within my repo, I have a file called RDS underscore pass TXT, which this specific function is referring to.
Now let’s do one thing. Let’s go to Kplabs underscore git, and let’s verify we have all these. So let’s run a Terraform in it to install our AWS provider. All right? So the AWS provider has been configured. Now, if I do a TerraForm plan, it will basically show you that the MySQL RDS database has to be created. So this is the RDS database-related resource. And now, as you can see over hereunder the name, it is Mydb. However, within the password, it is showing that the password is sensitive, so it is not actually showing you the password. So let’s do one thing: let’s actually do a TerraForm application and see what exactly happens. You see, the password is still sensitive, so it is not actually showing you the password. All right, so the DB instance got created.
Now let’s do one thing; let’s go back to our batch of git. Let’s do LS. And now you see there is a TerraForm TF state file. In the most common approach, we’ll do an gait-status, and I’ll simply add all of the changes that have been made, commit them, and push them. Let me just do a git push. All right, so the git has been pushed. So now if we go back to Bit bucket, let me just refresh the page. All right. As you can see, two new files have been added. You don’t really have to add this terrain; you can add it to git ignore. Anyway, my intention for this lesson was to have this TerraForm TF state file added to your git repository. This is something I’ve seen in the majority of organisations and startups; in fact, they all take this approach.
Now, the problem with this kind of approach is that even though you are using interpolation within this password, you are not committing this specific RDS underscore path. TXT, the problem is that whatever TerraForm dot TF state file is created, the password is stored in clear text within the TerraForm dot TF state file. And it’s the same here. You’re back in the same loop, storing your secrets within the Get repository, which is not considered best security practice. As a result, this approach is not highly recommended. And we should be taking a different approach, where we do not commit the TerraForm Testate file within the git repository.
So we’ll be looking more into how we can move the TerraForm TF state file out of the git repository in such a way that it can still enable team collaboration. So, this is it for this lecture. My main aim for this lecture is to show you why TerraForm TF state file committing to the git repository is not a very idle practice, even though you are using some kind of function similar to this. So this is it for this lecture. I hope this has been informative for you. And again, I will be posting these codes after this lecture so that you can write them out in your lab setup, and then we can proceed with the next lecture. So I hope this has been informative for you, and I look forward to seeing you in the next lecture.
3. Module Sources in Terraform
Hey everyone, and welcome back. Now in today’s video, we will be discussing the TerraForm module sources. Now, the source argument within a module block tells TerraForm where to find the source code for the desired child module. So if you look at the sample example, you have a module of console, and within the source, you have given the URL where the module is available. Now there are multiple supported sources. So you have the local path, you have the Terraform registry, you have GitHub, you have HTTP, URLs, SD, buckets, and so on. Now let’s quickly revise the first one, which is the local path. So the local path must begin with either or to indicate that the local path is intended. So if you look into the source code here, you have the dot slash.
That basically means that this is a local path from which the module needs to be sourced. The second important one that we need to know is the git module source. So arbitrary git repositories can be used by prefixing the address with a special git prefix. So if you look over here, within the source over here, you have a git followed by the exact URL where your repository exists. Any valid git URL is now possible after this specific prefix. So from HTTP, you see that this is a valid git URL that can be used along with the supported protocols. So there are two supported protocols in this example. So you have HTTP and SSH, and both of them are supported as part of the git module source. Now, a few more important things that we need to know over here.
So by default, TerraForm will clone and use the default branch that is associated with the appropriate repository. However, there is a way in which you can override that specific configuration with the help of the ref argument. So if you see over here, you have specified a ref argument where you can specify the overall branch as well as the tag names. So this is something that can be done. So let’s do one thing: let’s quickly jump into the practical so that we can understand more about the git-related aspects. Now, as we are discussing, there are multiple module sources that are available. Now, the primary goal of today’s video is to practically explore the one associated with the git. So let’s take a step back. So you have one linked to GitHub, one linked to Bitbucket, and one linked to a generic git repository. So, if you look over here, this is the syntax you need to use. So for today’s demo, what we’ll do is create a git repository called Temp python Repo, which we’ll be using for today’s demo. So what we have for today’s video is a file called demo file TF, and we’ll be using a module with the source code of git. So let’s create a new module. Let’s call it the “demo module,” and within the source let’s specify git followed by the protocol. It can be HTTP or SSH.
Let’s start with http first. So from our repository, let’s quickly do a clone. Here I’ll copy the URL, and let me paste the URL over here; I’ll go ahead and save the change. And from the CLI, we have a demo file tape that is available. Let’s go ahead and do a TARA for a minute. So here you see that it has downloaded the appropriate files from our git repository and that they are stored within the TerraForm modules directory. So let’s quickly explore. So within TerraForm, you have the modules directory; within the modules, you have the demo modules; and basically, these are the files that were part of our git repository. Now, for the first demo, we basically made use of a generic git repository. However, TerraForm directly supports GitHub.
So if you look over here, it supports GitHub, where instead of giving an overall syntax of “git followed by the protocol and the URL,” you can directly specify the overall repository over here. So let’s go ahead and try this approach as well. So let’s go back to our Terraform directory. What we’ll do is go ahead and remove the folder associated with the modules. Until that time, let’s go ahead and try the GitHub format over here. So, if you look at Zeelbora’s overall format on GitHub, followed by the temporary proposer, So I’ll copy this up, and let’s replace this. You can also remove the https. So this is the overall format. As a result, our modules folder has been removed once more. Let’s do another Terraform in it. And as expected, you see the overall files have been configured within the module. So if you would like to verify, if you go to the demo module here, you should see demo TF and README MD. Now, both of these are part of the GitHub repository.
So this is the first part. The second part was where we were discussing references to a specific branch. So currently, what is happening is that the master branch is the default one, and when you do a git clone, the master branch will be referenced. However, at a later stage, you might want to refer to a different branch. So in today’s example, we have two branches: one is the master, and the second is the development. So let’s say you want to refer to the development branch within the source, so let’s look into how that can be done. So if you look at this screenshot, you are making use of a ref argument where you will specify the overall branch name. So let’s try this out. So again, let’s go ahead and remove the modules directly here. Let’s quickly undo the change. So we’ll go back to the generic repository format, and here we’ll make use of ref and specify the branch name, which is development. All right, let’s go ahead and save the change.
And now let’s quickly do a tara for a minute. All right. So here it is basically stating that there is an error with the attempt to download a module. This is primarily because there is a small typo in the name. So let’s quickly fix the typo, and then do another Terraform in it. Great. So things seem to be fine now. Now going back to the modules here, you should see that there are a few more extra files. So apart from the TF and the README, you also have the TerraForm TFS file that was part of the development branch. So let me quickly show you. So for the master branch, you have a demo TF. And if you quickly switch to the development branch here, you should see that you also have a TerraForm TFS file that is present. So that’s the high-level overview of the TerraForm module sources. We also look into the git-based module source and how we can refer to a specific branch. This particular repository of temp hyphen repos that I have is now on GitHub. This is a public repository. So you can go ahead and run the exact code that we had done in today’s demo, and you’ll be able to pull the changes accordingly.
4. Terraform and .gitignore
Hey everyone, and welcome back. Now in today’s video, we will be discussing Terraform and Git ignore. In a nutshell, the git ignore file is a text file that tells git which files or folders in the overall project should be ignored. So you will have a file called dot gitignore, and here you can specify the exact folders or maybe the files that you do not want to commit as part of the get repository. Now, when it comes to TerraForm, depending on the environment, it is recommended to avoid committing certain files to the repository. The first is the TerraForm.dot directory itself. So when you do a TerraForm in it, this file will be recreated. The size of this directory varies depending on the overall provider of plugins. So it is recommended to ignore this as part of the git ignore file.
However, depending on the environment, you may choose to ignore Terraform TF 1 because it may contain sensitive data such as usernames, passwords, or secrets, among other things. So again, this part depends on an environment-by-environment basis. Then you also have the TerraForm TF state file. So this should ideally be stored within the remote site, and the TerraForm or TF state file can also contain the username or password that you might have within your TerraForm configuration. The next file is the crash log file. So again, if the Terraform crashes, then the logs are stored in a file named “crash log.” So this file is also something that you can decide to add as part of the gate ignore.
Now, if you go to GitHub and the gate ignore repository, you will find the gate ignore file for each technology. So let’s do a quick terraform over here and see what they have as part of the gate ignore here. So let me maximize it so that it’s viewable. So the first one is the local TerraForm directory. So this is something that they have added. The second is the TF state file that they have recommended. Then you have the crash log file, and they have also ignored the towers file. Not only TerraForm TF, but also any custom TFW you may have. Then they have a few more files they have added as part of the git ignore. Now, many of you might have already done the “git ignore” part practically within your projects, but to ensure that everyone is on the same page, I will do a small practical on that. So what I have is a git repository called Tempo Repo, which is a public repository that every one of us will be able to clone. And within this repository, there is a simple file called demo TF.
So let’s see what the demo TF has to offer. So it basically has a few locals and a few outputs over here. So let’s do one thing. Let’s go ahead and quickly clone this repository. So I’ll do a git clone. As a result, you must ensure that git is installed on your system. So now, if you do a ls iPhone L, you will see that you have one directory. And within that directory, if you do LS, as expected, we have a demo TF that is available. So let’s do one thing; let’s also create a new TF one. Let us refer to it as TF 1. And I’ll say username is equal to admin, and password is equal to password.
Great. So apart from that, let’s also do a Terraform application. I’ll say yes, and as expected, within the output we have some output that is available. So now, if you do a LS hyphen L, we should have an example TFW file, and we should also have a TerraForm TF state. And if you do a TerraForm in it, and I do LSLA, you should have a dot TerraForm directory as well. So now let’s go ahead and create a new git ignore file. And within this, you can put dot TerraForm, say start TF 1, and even TerraForm TF State. All right. Now, before we save this, let me copy it up. I will not save it for the time being. Now, if you look into the git status, you’ll see that it is trying to commit the TerraForm TF state, the example.tf file, as well as the TerraForm file. And these are the three things that we do not really want to commit to the git repository.
So let’s quickly open up the dot gitegnor file, I’ll paste all three names, and I’ll save the change. So now, if I do LSLA, you should see that there is a dot-gitegnor file that is available. And once the git ignore file is present, if I do gait status, you’ll only see that the only file that it is trying to commit is git ignore, ignoring the TerraForm. It is ignoring the TFWs file as well as the TF state file. Great. So, hopefully, you have a high-level understanding of TerraForm git ignore and some of the files. On a high level, you can disregard it as a component of the get repository.
5. Remote State Management with Terraform
Hey everyone, and welcome back to the Kplabs course. Now, in the get lecture, we were discussing that the TerraForm TF state file can also contain sensitive information. And this is one of the reasons why it is rarely recommended to avoid committing the TFS state files in the get repository. Now, the question that comes up is: how will you work or how will you collaborate if the get repository does not have the TF state file itself? and the answer is through the remote back end.
Now, remote backend is basically a feature of TerraForm that allows you to store the TerraForm TF state files in a central repository that is not get. So let’s look at what I mean by this. So, within the remote back end, you see, I have a laptop. So I have been storing my TerraForm TF files—I would say the TerraForm code—in the central repository. However, the TF state files are generally stored on the remote back end. So the central repository will not have the TF state files, and it will only be part of the remote back ends. So this is one of the things that you can work with. Now, what I can show you is one of the examples so that you can understand it much better. So I have a directory called KP Labs and a remote hyphen backend. And within this, I have a resource called AWS EIP. Now I have already configured, or I would say I have already applied, that resource. And this is the resource that is created through Terraform.
However, if you will see over here in this directory, I don’t really have any TerraForm TF state files. So that TerraForm TF state file is actually not stored locally. It is kept on a remote server. So where exactly is it stored? Let me show you. Currently, one of my remote back ends that I have configured is S 3. And this is where the state file is stored. So if I quickly open the state file in the browser, all right, let me quickly download it in fact. All right. So you see, this is the TerraForm TF state file. Now, this TF state file is actually stored within my three; it is not stored in the local directory. And now you may ask, “How is it done?” And the answer to this is that we have included one more file called “backend TF.” And within backend TF, we have set TerraForm to save the TF state file in this specific remote back end call as S 3.
So let’s go back to our presentation and look at the supported backend types. So currently, TerraForm does support a lot of backend types. One of them is the S. However, there are a lot of backend types that are supported, including console and various others. So, as of now, we have been storing our TF state files in the Git repository. However, from now on, we will look into how we can configure various back ends with our TerraForm repository. Now, when it comes to the back end, there are two major types of back ends that are available. One is the standard backend type, and the other is the enhanced backend type. Now, the standard backend type basically allows you to do state storage and state locking. And the enhanced backend type basically allows all the features of Standard plus remote management. So don’t really worry. These will be discussed in the coming weeks.
