1. S3 Buckets and Objects
Okay? So first, to talk about Amazon S3, we need to talk about buckets. So S Three is a system or a service that allows us to store objects. So files go into buckets or directories. And each bucket must have a globally unique name. As you can see, with the tools we have, we can’t make buckets with names that are already taken. The buckets are defined at the region level. So even though S3 is a global service, buckets are a regional resource, and there is a naming convention that includes no uppercase, no underscore, and three to 63 characters in length. It should not be an IP, and it must start with a lowercase letter or a number. OK? Now in these S-three buckets, we need to create objects. And objects are files, and they must have a key. And what is the key? It is the full path to that file. So if we have a bucket named “my bucket” and an object named “Myfile.TXT,” then the key is “myscorefileTXT,” which is the blue part. But if we have folder structures within our three buckets, so my underscore folder one, another folder, and then my file TXT, the key is the full path. So again, that’s the entire thing in blue. And the key can be decomposed into two things: the key prefix and the object name. So, in the same long example, the underscorefolder one slash, another folder slash is the prefix for myofile TXT. OK? That is the prefix, whereas the object name is my underscore file TXT.
So even though there’s no concept of directories within buckets, just very, very long key names, then the UI will try to trick you into thinking otherwise because we could create directories within history. So what we really have in history are just keys with very long names that contain slashes. Okay? So now let’s go again for these objects. So the object values are the contents of the body. So the maximum object size on Amazon Xray is five terabytes. So five GB, which is a huge object, But you cannot upload more than 5 GB at a time. So that means that if you want to upload a big object of five terabytes, you must divide that object into parts of less than 5 GB and upload these parts independently into what’s called a “multi-part upload.” Now, each object in Amazon’s history can have metadata. So list the key-value pairs that could be system or user metadata. And this is to add information to your objects as well as tags. You can also have key-value pairs and tags, which are very useful when it comes to object security or lifecycle policies. Finally, we’ll see when we go into versioning that there is a version ID on our Amazon S3 budgets, and we’ll see what the value of that is in the versioning lectures.
So without further ado, let’s get into the Amazon S-3 console and do a hands-on. So, let’s take a look at the Amazon S3 for the first time. So I’m just going to type “S Three” in the search bar, and we are going to be redirected to the Amazon S Three console. So as we can see, there are improvements to the console. So hopefully yours looks like mine, but if there is a big, big change, I will obviously update this course, obviously.Okay, so the first thing we can see in this UI is that the top right corner says global. As a result, S Three does not necessitate region selection. So that makes S3 a global service. So in this UI, you will see all your buckets. But as I said, buckets are created for a specific region. So when we go and create a bucket, we will still have to select a region. So this view, this global view, will give us all the buckets across all the regions. But a bucket is tied to a specific region. So let’s go ahead. I’m going to remove the panel on the left-hand side, and I’m going to go ahead and create a bucket. So I’ve entered a bucket name, and the bucket name must be globally unique. So, for example, if I type “test” in here, this bucket name should be taken because probably someone already thought about creating a bucket name for “test.” So it’s going to create the bucket, and it says, “Yes, the bucket with the same name already exists.”
So it doesn’t matter if this bucket doesn’t exist yet in my account; it exists in someone else’s, and so it’s already taken. So instead, I can say “the bucket of Stefan 2020,” and that bucket name should not be taken, and we’re good to go. Then we have to select a region. So, in this case, I choose EU Ireland US One because it is close to me. But you should select a region that’s close to you as well to create your S3 buckets. As you can see, not all the regions have S3 just yet. Okay? So select the region closest to you, and we can see that this bucket is going to be tied to this region, even though it will show up as a global service. Okay, so let’s scroll down and see all the settings. So there are bucket settings to block public access. and I will leave this on. Right now, we’ll do a deep dive on this. This is to prevent your bucket from becoming public by mistake. So for now, we’ll block all public assets and keep our bucket private. And this is to prevent data leaks that can happen in the news. So this was a setting that was added later, last year or two years ago, to really prevent buckets from going public mistakenly. We will get to see the advanced settings one by one in the future. Lectures. So what I’m going to do right now is just take these buckets and click on Create Buckets. Okay, so my bucket has been created, and now I can go to the bucket details either by clicking this link or by clicking the link in the console as well.
So now what I can do is go ahead and upload a file. So for this, I’m going to the Upload button on the top left and then clicking on “Add File.” I’m going to select Coffee.jpg and upload that coffee file. So as you can see, we can either directly click on “Upload Here” or we can go through some options. So let’s just quickly view these options, even though we won’t touch them right now. So I click on “Next,” and here we have some information about the permissions of who can see that file. So my account can read and write to this file. So that’s perfect. And for now, we cannot manage public permissions because they have been blocked by the settings we saw before. Click on “next.” We can look at properties, which depend on how we want to store that object or storage class. And we’ll have a deep dive on this as well, on encryption, metadata, and tags. For now, I’m going to leave everything as is and click on Next. So overall, we haven’t changed anything. I just want to quickly glance through the options, and I’m going to go and click on “Upload.” Now my file is being uploaded to my SRE bucket, and here it is. Perfect. So my file has been uploaded. I can click on it, and a panel on the right hand side comes up and gives us some information about our file. So let’s go ahead and do something. I’m going to right-click on this file and then click on “Open.” So through this method, I am able to view my Coffee JPEG file, and as you can see, it is the image that I’ve uploaded. So this works, but the URLs you can see look a bit long and very complicated. If I go the other way, click on this file and click on the object URL from this panel. So I’m going to click on the object URL here, go back to my file, and open this new tab. It says the XML file does not appear to have any style. It says access is denied. So somehow, using this URL, which is a public URL for my file, in step three, I get a 403 access denied. We can access this file by right-clicking and opening it using this URL. So the reason is that this file is not public. So any time we try to access this file publicly, because for now we haven’t uploaded and updated our buckets to be public, we will not be able to access it.
However, if we use the right click and open command, it actually generates a unique URL. And this is a very special URL that’s very, very long. And this URL is called a pre-signed URL. It’s signed with my credentials on AWS. And so because I have the power to view this file, it’s going to be included in that URL, and I’m going to be able to access this file. We’ll take a closer look at the preset URL later, but for now, we’ve seen that the public version of our file is not accessible because our bucket is private, and that we can access our file privately using the presigned URL. What I can do as well is create a folder, so I can create a folder called Images and click on Save. And then, within this folder, I can go ahead and upload my coffee JPEG file. And now my file has been uploaded yet again. And so, as we can see now, if you look at our three buckets, we have two keys: we have the coffee jpg at the root of our Sri buckets, and we also have a coffee jpg within the key images. So the path images So we have different keys, and it gives us the illusion of having a directory, but it’s actually just a very long name for our file, coffee JPEG. So this is the entire file name. So we have the same file in two different buckets and two different paths. Sorry. And what we can do as well is perform operations on our file. So, for example, we can rename the file or delete the file copy and move, allowing us to perform a wide range of operations on our files. So what I’m going to do just to show you is go to this directory here and I’m going to do “delete,” and this is going to delete my image within the images directory. So I clicked on delete, and here we go. It’s gone. And so that’s just the basics of sbackets. We have a lot more to see, as you can see, a lot more options, and so on, but hopefully you liked it and I will see you in the next lecture.
2. S3 Versioning
So now let’s talk about Amazon’s three versioning strategies. So your files in Amazon S3 can be versioned, but it has to be enabled first at the bucket level, so we’ll do this in the hands-on. So that means that if you re-upload a file version with the same key, then it will overwrite it. but it won’t overwrite it. Actually, it will create a new version of that file. So instead of overwriting the file that already exists, it will create a new file version, and I’m simplifying it here, but it will be version one, then version two, then version three, etc, etc.So it’s best practise to version your buckets in Amazon S3 in order to have all of the file versions for a while because it protects you against unintended deletes because you can restore a previous version and also because you can easily roll back to any previous versions you wanted.
So you need to know a few things, though any files that were not versioned prior to enabling versioning will have the version null, and if you suspend versioning in your bucket, it does not delete the previous versions; it will just make sure that future files do not have a version assigned to them. So let’s go hands-on and see how it works. So here I am in my bucket, and I’m going to go to the properties, and in properties, I have versioning, so I’m going to click on versioning, click on enable versioning, and finally save. So this is to keep multiple versions of an object in the same bucket back in overview, and now we have a new panel called versions and need to click on it. So right now it says “hide,” but I want to go into version “show.” and now we have a different UI in the Amazon S3 console that shows the VERSIONID on top of the file name.
So for this copy of the jpg, the version ID is null because we are enabling versioning after having uploaded a few files, as I said in the theory lecture, and this will have a version ID of null, but let’s go ahead and upload a new file. So I’m going to add a file, and this time I’m going to upload a beach jpg and upload this file, and what we get out of it is that this beach jpg has a version ID, and it’s not 1234. I was making it very simple here. The version ID is a much more complicated string, but as you can see, there is a version here, and it says “latest version,” so I can go ahead and re-upload, for example, that beach file. So, let’s upload that beach file. As we can see, it was successful, and we now have two versions of that file, the previous one and the latest version, which was uploaded at 218 PM and has a different VERSIONID. Even though it was the same file, it was not overridden. It has been uploaded as a new file version. So we can do the same with the coffee. So I can upload, for example, the coffee jpg, and as we can see now, there is a latest version for coffee jpg with a version ID, and even the previous file with the version ID null is still kept in here. Okay, but if we go back to version hide, we only see two files. So let’s do something quite interesting. We’re doing versions hide, and I’m going to take my beach JPEG, right-click, and then I want to delete it, so I’ll say delete and click on delete. So now it seems that my file is gone; we cannot see it in this window, but if we go back to versions past, we can see that the beach jpg is still here, and the thing that gets added for beach jpg is a delete marker. So a delete marker has a file with zero size that has a version ID, and because it is the latest thing on mybeach.jpg, this delete marker made my file hidden.
But what I can always do is restore a previous version. So if I can click on this delete marker and delete my delete marker, what’s going to happen is that now the latest version is one of the previous versions, and if I go to here, my beach JPEG is back. So this is what I mean by preventing unintended deletion. If I go into this beat jpg and unintentionally delete this, it will still show because it was just a delete marker. Now it is possible for you to remove a specific version of a file. For example, if I go to this one and right-click and then do delete, this will delete this specific version ID in a Sparket, and this will for sure delete that version ID permanently. So it’s very important to understand that, and that’s all there is to know about versioning. Obviously, you can suspend the versioning. So, if I suspend the versioning, any previous objects are still present and have an Aversion ID; it only affects future version objects. Okay, so I’m going to go back in versioning and reenable it, but you can play around and see how that works. But versioning is a very powerful feature of Amazon SRE. because now you can keep multiple versions of a file for a long time and ensure you can roll back or restore any previous versions if you want. That’s it for this lecture. I hope you liked it, and I will see you in the next lecture.
3. S3 Encryption
So now let’s talk about Amazon’s three versioning strategies. So your files in Amazon S3 can be versioned, but it has to be enabled first at the bucket level, so we’ll do this in the hands-on. So that means that if you re-upload a file version with the same key, then it will overwrite it. but it won’t overwrite it. Actually, it will create a new version of that file. So instead of overwriting the file that already exists, it will create a new file version, and I’m simplifying it here, but it will be version one, then version two, then version three, etc, etc.
So it’s best practise to version your buckets in Amazon S3 in order to have all of the file versions for a while because it protects you against unintended deletes because you can restore a previous version and also because you can easily roll back to any previous versions you wanted. So you need to know a few things, though any files that were not versioned prior to enabling versioning will have the version null, and if you suspend versioning in your bucket, it does not delete the previous versions; it will just make sure that future files do not have a version assigned to them. So let’s go hands-on and see how it works. So here I am in my bucket, and I’m going to go to the properties, and in properties, I have versioning, so I’m going to click on versioning, click on enable versioning, and finally save. So this is to keep multiple versions of an object in the same bucket back in overview, and now we have a new panel called versions and need to click on it. So right now it says “hide,” but I want to go into version “show.” and now we have a different UI in the Amazon S3 console that shows the VERSIONID on top of the file name.
So for this copy of the jpg, the version ID is null because we are enabling versioning after having uploaded a few files, as I said in the theory lecture, and this will have a version ID of null, but let’s go ahead and upload a new file. So I’m going to add a file, and this time I’m going to upload a beach jpg and upload this file, and what we get out of it is that this beach jpg has a version ID, and it’s not 1234. I was making it very simple here. The version ID is a much more complicated string, but as you can see, there is a version here, and it says “latest version,” so I can go ahead and re-upload, for example, that beach file. So, let’s upload that beach file. As we can see, it was successful, and we now have two versions of that file, the previous one and the latest version, which was uploaded at 218 PM and has a different VERSIONID. Even though it was the same file, it was not overridden. It has been uploaded as a new file version. So we can do the same with the coffee. So I can upload, for example, the coffee jpg, and as we can see now, there is a latest version for coffee jpg with a version ID, and even the previous file with the version ID null is still kept in here. Okay, but if we go back to version hide, we only see two files. So let’s do something quite interesting. We’re doing versions hide, and I’m going to take my beach JPEG, right-click, and then I want to delete it, so I’ll say delete and click on delete. So now it seems that my file is gone; we cannot see it in this window, but if we go back to versions past, we can see that the beach jpg is still here, and the thing that gets added for beach jpg is a delete marker. So a delete marker has a file with zero size that has a version ID, and because it is the latest thing on mybeach.jpg, this delete marker made my file hidden. But what I can always do is restore a previous version.
So if I can click on this delete marker and delete my delete marker, what’s going to happen is that now the latest version is one of the previous versions, and if I go to here, my beach JPEG is back. So this is what I mean by preventing unintended deletion. If I go into this beat jpg and unintentionally delete this, it will still show because it was just a delete marker. Now it is possible for you to remove a specific version of a file. For example, if I go to this one and right-click and then do delete, this will delete this specific version ID in a Sparket, and this will for sure delete that version ID permanently. So it’s very important to understand that, and that’s all there is to know about versioning. Obviously, you can suspend the versioning. So, if I suspend the versioning, any previous objects are still present and have an Aversion ID; it only affects future version objects. Okay, so I’m going to go back in versioning and reenable it, but you can play around and see how that works. But versioning is a very powerful feature of Amazon SRE. because now you can keep multiple versions of a file for a long time and ensure you can roll back or restore any previous versions if you want. That’s it for this lecture. I hope you liked it, and I will see you in the next lecture.
4. S3 Security & Bucket Policies
Okay, so now let’s talk about Amazon SRE security. So it’s very complex, but first you have user-base security. So, our IAM users have IAM policies, and they authorise which API calls should be allowed. And if our user is authorised through our IMS policy to access our Amazon S buckets, then it’s going to be able to do it. Then we have resource-based security. And this is an infamous “three bucket” policy. They’re bucket-wide rules that we can set in the S-3 console. And what they will do is state what principles can and cannot do in our three buckets. This gives us cross-account access to our S3 buckets. We’ll go over the S Three Bucket Policies in great detail. Then there’s object ACL, which is finer grain and allows us to set the access rule at the object level. Finally, bucket ACL is becoming even less common. And these two don’t really come up at the exam. Now, take note of an important principle. So it could be a user; a role can access an S3 object if the IA
M permissions allow it. That means that you have an “im” policy attached to that principle that allows access to your “free” buckets. Or if the resource policy allows it—and usually your rebucket policy does—you need to make sure there is no explicit denial. So, if your user is permitted to access your S3 buckets but your bucket policy expressly prohibits them from doing so, they will be unable to do so. Okay? Okay, so now is the deep dive on Sri bucket policies; they’re JSON-based policies. So JSON is a notation language. So we have a JSON bucket policy here. And this bucket policy allows the general public to learn about our structures. As we can see, the effect allow principal star is enabled. Anyone who performs the action of obtaining an object from the resources bucketstar can do so for any object in my S-3 buckets. So this is great. This allows public access to our S-3 buckets.
So these bucket policies can be applied to your buckets and objects. As a result, both actions allow or deny a set of APIs. The effect is to allow or deny. The principle is the account or the user that this SRE bucket policy applies to. And so some common use cases for Sri bucket policies are to grant public access to a bucket or to force objects to be encrypted at the upload time, or to grant access to another account using cross-account S rebuke policies. As a result, we’ll delve deep into rollback policies. Then there are the buckets for restricting public access. So we’ve seen this firsthand as we begin. So this was a new setting that was created to block objects from being public. if the account had some restrictions.
So here we have four different kinds of settings for blocking public access settings.We have the new access control list, any access control lists, or new public or access point policies. So this is going to block objects and buckets from becoming public if they are granted through any of these methods. Or you can block public and cross-account access to buckets and objects through any public bucket or access point policy. So you don’t need to remember these four different settings, okay? It’s just a summary in here. What you need to remember going into the exam is that there is a way to block public access to your SRE bucket through these settings. The exam will not test you on each of these hese settingThese settings historically were created to prevent company data leaks because there were a lot of leaks of Amazon S-record buckets in the news, and Amazon S-record came up with this way of making sure that an administrator could say, “Hey, none of my buckets are public, by the way,” because of these settings. and that was very popular.
And so if you know that your buckets should never be public, leave them on, and there’s a way to set these at the account level, as we’ll see in the hands on other securities in S Three. On the networking side, you can access S Three privately through VPC Endpoints. So if you have two instances in your VPC without internet access, then they can access three privately through what’s called a VPC endpoint. For logging on it, you can use S-free access logs, which can be stored in the other S-3 buckets. API calls can also be recorded in Cloud Trail, a service that stores API calls in your account. For user security, you have MFA-delete. So multifactor authentication is MFA, in which case if you want to delete a specific version of objects in your bucket, then you can enable MFA delete, and we will need to be authenticated with MFA to be able to delete that object. Finally, there were some presigned URLs that we saw briefly when we reopened that file; there was a very long URL, which is a URL that’s signed with some AWS credentials and is only valid for a limited time. And the use case for it, for example, is to download a premium video from the service if the user is logged in and has purchased that video. So the idea here is that at any point during the exam, you will notice that certain files are only accessible to certain users for a limited time. Think pre-signed URLs. So in the next lecture, we’ll do a hands-on introduction to security to see all these various options. So I’ll see you in the next class.
5. S3 Bucket Policies Hands On
Okay, so let’s have a look at the different security settings for our S3 buckets. So the first one I want to show you is going to be around the “Three Bucket” Policy. So if you go to Permissions and then to Bucket Policy, we are able to set a bucket policy for our S-3 buckets. And so we have to put the JSON document here, and for this we can use either documentation to get some simple policies or use the Policy Generator. So we’ll use the Policy Generator because it is a cool exercise; you don’t need to remember exactly how to use it, but I think it’s a cool exercise, and we’re going to design an Amazon S-3 Policy Bucket Policy that will prevent us from uploading an object unencrypted without the Ads 256 encryption mode. So the selected type of policy is going to be a “three bucket” policy, and then the effect is going to be “deny.” We will deny the principal, which means anyone. As a result, we will deny anyone on the Amazon SRE service the action. So we’re talking about upload. So the action is called “put object.” So, in order to upload, we look for put objects.
As a result, we deny anyone on Amazon for the action putobjects and the Amazon resource name, or ARN. This format will appear frequently throughout this course, and to obtain the ARN, we must return to S 3. So we copy the entire ARN. It starts with the ARN column. As a result, we’ve included it here. And at the end of this, because this is a put object that we apply it to, it needs to be applied to anything within the bucket. So at the end of the ARN, we need to add a slash star, which represents any object within the bucket. This is why we add a slash and a star, and we add the statements. And first we need to add conditions, obviously. So we’re saying that no one in Amazon history has ever had the right to place an object on any object within our buckets. And the first thing is that we want to make sure that there is a header for encryption. So we’re looking for null, and the key is going to be the encryption header. So here we go, we’re going to find Amazon Serverside Encryption elsewhere, and the value is correct. So we’re saying here if this XIMZ server-side encryption header is null, it’s null, which means it doesn’t exist. Then deny it. If it’s null, that means that the object was sent unencrypted. So this is what we want. So we’ll add this statement, which is the first statement in here, and then we’ll add a second statement. So we’ll say the main star effect is false. Again, we’ll find the object here. So put objects, and for the resource name we’ll have “slash st,” so just save them before, and then we’ll add a condition, and we’re saying okay, string not equal. So this is another condition, and we’ll look at the same object header. So we’ll look at the AMZ server-side encry
ption, and we’re saying okay, if it’s not equal to AES 256, which is the encryption used when we have SSEs 3, then deny it. So through this second statement here, what we’re doing is denying any request that does contain this header but is not equal to a yes to 56. We’ll add the statements. Now we have two deny statements, and hopefully that works. We’ll generate the policy, and the conditions did not work, so let’s go back and edit this real quick. So I think the problem I had was around the fact that I did not add the condition, so let me quickly fix this. So staring denies, and it’s good to keep mistakes on camera sometimes because this is something that you can go through on your own as well, so no one is perfect. So I’ll just do this quickly, okay, and then add a condition, and we’ll do it again. We’ll do it twice; I’ll show it once, and then I’ll fix it twice. So Amazon multiplied by three services add encryption value true, and here’s what I didn’t do: click on add condition. So here we go. Now that this condition has been added, okay, so this looks good; add the statement, and the second thing is deny start, and Amazon is free to search for put objects once more. On the same star addition condition, we say string does not equal the same header here, and the final value is 256. Now we’ll add a condition and a statement. Okay, so we have the correct policy, so I’ll generate it, and this time it looks good. So I’m going to go ahead and copy this, paste it in here, and save it. So now we have a bucket policy, and we need to see if it works. So there’s no better way to do this than to try it out. So we’re going to go back to our bucket and upload the file, and this time we’re going to upload our coffee JPEG. Click on “next,” “next,” and I will make sure to have encryption “none,” “next,” and “upload,” and it’s starting, and we get one error. So here there was an error; if we view the details of the error, we’re saying it’s a forbidden error, and if you look at it, it was forbidden because it did not include the correct header, so we had no encryption, and this was denied by the Sri bucket policy. Now, if you upload the same file, “coffee.jpg,” but specify “as 256,” So, by uploading using Amazon S’s three master keys, It should work this time. So, yes, this was a success, and now we can upload. We had the exact same copy as the JPEG. But this time we specify the SSE KMs encryption scheme. Click on “next” and upload. This one should give us an error. Yes, it did give us some errors. Now we have two errors. As you can see, the S Three Bucket policy is active and in effect. So obviously, I did not remember how to do this bucket policy on my own. Okay, just Google “S Three bucket policy, deny encryption” and you’ll find some blogs, for example, that show you some bucket policies that allow you to “deny encryption, correct encryption errors, or deny unencrypted object upload.” So these are things you can find online. But I thought it was really cool to show you this address policy generator, which I think is a really cool tool to create policies. OK, back into Section 3. We also have the access control list, which, to be honest, is not at the exam anymore. So I’m not going to linger on those. However, this is where we can either set the access control list at the bucket level or set the access control list on this object. So if I click on the object and go to permissions, then I can set the ACL at the object level. But again, I’m not going to linger on it because this is out of scope for the exam. Finally, if we obtain permission, we will restrict public access. These are the four settings that I was mentioning to you. They’re described right here, and right now everything is on. So it’s blocking all public access. But if you wanted to make your bucket public, then the first thing you need to do is untick everything and click on Save, which I’m not going to do right now. Okay, but you see where these settings are, and again, you don’t remember the names of the four settings. Just remember that these settings exist and that there is a separate level of protection for your S rebuckets. And as I said, these settings can also be applied at the account level. So, if you go to the left hand side, you can configure your account to block public access. When you know that all the extra buckets in your account should not be public, you should enable them here as an extra security measure. OK, that’s it for this lecture. I hope you liked it. I’ll be in my buckets, and I’ll see you at the next lecture.
