1. Storage Account Strategy Overview
So in this section of the course we’re going to be talking about storage. Now, a reminder that the AZ 303 exam is the Technologies exam and it’s much more focused on the acts of creating storage and understanding the choices that you’re making as you’re making them in terms of storage creation. In this course we’re talking about design and in this video we’re talking about the provisioning aspects. What are the decision factors that go into creating a storage account? Within Microsoft Azure you have two choices of storage. One is an unmanaged storage and the other are managed disks. So unmanaged storage is basically storage, that is a storage account. There are four types of storage in a storage account. One of them is Blobs and those are in the form of containers.
You can also have files in the file share, you have tables and queues. Now the same storage account also supports data lake storage but we’re not going to really get into that in this course. In an unmanaged storage account you are paying per gigabyte. So you can create a storage account, put nothing in it and it would not cost you anything. The moment you put a single file into that storage account you’re going to be paying for the storage account in increments of gigabytes. So if you put only a few bytes, it’s a 1GB charge. Now the storage accounts are actually quite cheap, right? The cloud providers Microsoft, Amazon and such have really driven down storage prices and it does make a lot of sense if you have a lot of files to store long term to store them in the cloud.
On screen you can see a screenshot of the pricing page and I’m highlighting here the pricing per gigabyte. So it’s 1. 8 cents per gigabyte. At the hot tier, of course there’s a volume discount so the more that you store the price comes down slightly from 1. 84 to 1. 77 to 1. 7. Now, if there are other tiers, which we’ll talk about later in this video, actually in the next video we’ll talk about tiers that have even cheaper pricing. You can see one cents per gigabyte and even one 1000th, one 100th cents per gigabyte of the archived tier. Now the other type of storage are called managed disks. Now they’re very different. Managed in the name means that Microsoft is going to take off some of the burden of the disk.
So right now if a disk was to get full in an unmanaged disk situation you’re basically responsible for creating additional storage accounts, moving stuff between them. There’s a lot of sort of responsibility. Now managed disks are again management. Microsoft, they take on some of that responsibility. It typically is for a virtual machine only. So you’re going to use managed disk for the C drive and the D drive and the E drive. Any additional extended storage that you use on a virtual machine that is recommended to be used by managed disk that’s been available now for years. It is sort of the default weight. Now, managed discs are priced differently.
You’re basically choosing a tier to subscribe to and that’s going to be your monthly fee. So for instance, here is the standard tier which is using magnetic spinning hard disk and the smallest one you can get is a 32 gigabyte and it costs a dollar 54 a month. Now, if you do the math, that is more expensive than the unmanaged storage by a bit. So we were looking at 1. 8 cents per gigabyte for unmanaged and this appears to be around five cents per gigabyte and managed. So there’s a price differential. But again, this is more designed towards virtual disk. And so when you create your VM, you’re going to have a 32 gigabyte C drive and a 64 gigabyte D drive. Those are provision storage and basically the entire storage will be used and you pay for it whether you use it or not. So once you’ve provisioned this, the price is as it’s set. Now, in any of these unmanaged and managed situations, you do have the option for premium storage. Now in the managed case, you’re talking about solid state drives, flash drives, it’s a much quicker disk in the unmanaged state. You’re talking about basically paying more for performance. And so it’s, look at the price difference. It’s around ten x or goes from 1. 8 cents to for the premium. What you get is you get a much more responsive storage.
So if the access time to get to the storage file is important to you, not that the access time is poor on the other ones, but this is sort of more optimized towards that, then you will be willing to pay more. Also you pay less for reads and writes to this. So the premium tier is if you have frequently accessed files that you need hyper speed access to. On the manage disk account you can see that there’s premium storage. 32GB is 481. If you recall on the standard storage it is 150 for 32GB. And so now the price has sort of tripled. But again, the speed has been much faster. We can actually see on screen, it says the throwput is 170 megabytes per second. And so you get much quicker access to these files and maybe your VMs are going to run quicker if your disks are premium.
So I think Microsoft does recommend premium disks if you’re going to go this route. Now, unmanaged storage accounts have a limit and it’s been going up over the years. And so it used to be 500 terabytes and two two petabytes and now it’s five petabytes. And if that’s not enough for you there’s, you can open a support ticket and get more. Just to keep in mind, five petabytes is a lot of data and even at 1. 8 cents per gigabyte, or even 1. 7 cents per gigabyte at the top tier, that’s $85,000 per month. Estimated charges for storing five petabytes. So if you’re going to be storing that much data, you are going to be paying for it. But it’s still extremely cheap for the amount there is bandwidth.
Now you can have inbound bandwidth, depending on your location, it goes from five to 20 gigabits per second. So some North American locations are faster and some other locations around the world are a bit slower. And if you do need more inbound bandwidth, if that’s not sufficient to your needs, you can open a support ticket and get more allocated. The outbound bandwidth seems fairly fixed. It’s 50 gigabits per second. That’s pretty fast. Now Microsoft does keep copies of your files in multiple locations for what’s known as redundancy. And the way that this tends to work is basically that they’ll keep three copies of your files in the same data center. Or if you go into the global redundant route, they’ll keep six copies of your files across two data centers.
That way if we ever have a hard disk failure, you know, storage, storage doesn’t fail often, but it does fail from time to time. And so they’ve got additional copies and you won’t lose your, won’t lose your data. They have a really large, I think it’s like 99s durability for files, which means that, you know, once in a, on a million years, they’re going to lose a file from you. So it’s pretty good. I should also mention this concept of reserved capacity. We’ve seen this on a couple of slides already. But you can save money by reserving your storage. And so if you know already you’re going to be using something close to 100 terabytes per month, you can see that the pricing is for one year of reservation on 100 terabytes is $1,545 per month. And if you break that down to the per gigabyte pricing, that’s less than 1. 5 cents per gigabyte. Now remember, the pricing for pay as you go was 1. 8 to 1. 7. And so this is 1. 5, which is, which could be a lot of savings if you go up to the one petabyte, you save just a little bit more. And then when you go out to three years, the pricing drops from 1500 to 1200 per month. And so you’re getting down to year reservation. So if you know that you’re storing a lot in the cloud and you don’t have any plans to reduce that, you can save even more money by going with a reservation.
Talking back around virtual machines. Now you can use unmanaged disks or manage disks for the virtual machines, extended disks. And you can basically add EFGH drives, additional drives to your virtual machines, to your things. And we can see when you go to create the VM inside of Azure that you can actually pick which VM type and instance type supports these additional disks. So the B one Ms only supports two additional disks, whereas the B two S supports four or the B two Ms supports four. So if having additional storage in your virtual machines is important, then choosing a VM instance type that supports as many as you need can is advisable. You can see the operations per second. Also.
2. Setting Access Tiers
So continuing on talking about storage accounts, when you go to create a storage account within Azure, it does ask you what type of storage account you want. Currently as I record this, it gives you three options. There’s the general purpose v one storage account, there’s the general purpose v two storage account and there’s the Blob storage account. So in this video we’re going to talk about why would you choose either of those? First of all, for general purpose v one, it is not recommended that you choose that unless you really have no choice to choose that. General purpose v one is not being, no new features are being added to that. At some point it will go away.
And so if you’re got a lot of storage accounts that have running under general purpose v one and you just want one more, well, you can create another general purpose v one. But if it’s for a brand new application, you may want to look at general purpose v two. It’s got more features and it’s the sort of the future of Azure storage accounts. Now there’s also this concept of the Blob storage account. Now the Blob storage account is what’s called premium performance block blob storage. Now if you create a Blob storage account, you’re getting your storage account on hardware that has been specifically designed to be extremely fast. So it’s using solid state drives, but it’s also tuned to get that first byte out the door as quickly as possible.
Microsoft has said the service level agreement for a Blob storage account is less than ten milliseconds for the first bite. So I think they’re setting the expectation that if you’ve got files and getting the file out to the end user or to that application as quickly as possible is the priority, then you can consider using a Blob storage account. Again, it’s optimized around reading and it’s a little bit more expensive than a general purpose account. But if that’s your need, then that’s a perfect storage account for you. If you go into the general purpose v two storage account, then you have some options.
Now this is the most cases you may want to choose general purpose v two. We know when we create that we have the option between Blobs tables, queues and files. And those general purpose v two storage accounts can handle all four types at the same time. So you can have a set of Blobs in there, you can have files in there. You can use the tables on the same storage account. Now, when it comes to the access of the files inside your Blob section of your GPV two storage accounts, there are three access tiers. Now hot access is the default. It has got the highest cost to store files and the lowest cost to access them.
So the concept here is, again, you upload a file into Azure and they’re going to charge you so many cents per gigabyte to store it. There five cents per gigabyte, and then tens of thousands of people are going to come and want to read your file. And every 10,000 requests, you’re going to pay a few cents as well for the access. So that’s called hot access. The file is there, it’s ready to be used, and that is sort of the default access type. Now, there is a thing called cool access. So let’s say you have a file that you don’t expect a lot of people to want to read. Let’s say it’s a backup file.
You’ve taken a database backup or you’re taking a system backup, you upload that into Azure, you might want to set that into cool access. Cool access actually saves money on the storage. So you would pay half the price, roughly at this point, for a cool access storage than you would for hot access storage. But it does cost slightly more for the access. So if you put something in cool access and you end up needing to access it tens of thousands of times, you may end up paying more. And so for something that needs frequent access, you go into the hot storage account level. If something is not needed frequently, you might want to consider cool access. There is another one called archive access, and this is the lowest storage account cost and the highest access cost.
So now, in the archive world, you’re saving 90% off the cost of hot access. So if hot access is five cents per gigabyte, archive access is half a cent per gigabyte, but the access costs are the highest. And so you can end up paying dollars to get those files back. And why is that? Because basically, this is for long term storage. Let’s say you have your backups and you put them in cool access, but then after 30 days, you don’t need those backups anymore. Now, maybe legally you need to hold on to them, and maybe there’s some for real emergencies, you might want to have a month old backup of something. But you can put those backups into archive storage and save even more money on the storage of it. Because if you ever needed to access that backup some months in the future, then you are willing to pay a little bit more to access it in exchange for the savings.
Now, the thing with archive storage is there’s a bunch of costs relating to it in terms of your limitations. So we remember back in the day that people used to have tape drives to do their backups. They would mount a magnetic tape into a machine, like a big cassette, massive cassette size tape. And then they would back up their computers to a tape, and then they remove the tape from the machine and store that somewhere. Well, you’re not using tape drives for Azure Archive Tier, but it does take hours to get your data back. So if you put files into the archive tier. And then you realize, oh man, I need that file. You can expect it to be five, 6 hours to get that data back. That’s called rehydrating the file.
So it’s an individual command you’re giving to the storage account to get that file back. And it could take some time. Both the cool and the archive tier require you to make a commitment. So in the cool tier, you’re going to be paying for 60 days. Regardless of whether you actually add the file and then delete the file, the cost remains. And the archive tier, it’s 180 days minimum. So both of these tiers require you to have some kind of long term goal to keep those files in there. The cool thing too, is on storage account level, when you create your storage account, you can set the hot or cool as the default. Now, hot is the default already, but if you’ve you want a storage account that only contains cool files, you can set that as a default and then you don’t have to do anything. You just put the file into the storage account and you’re going to get that savings, the 50% savings, because it’s a cool tier by default.
3. Storage Requirements
So, continuing on to talk about storage strategy, let’s talk about how you understand your own storage requirements. So you have applications and you’re looking at Microsoft Azure to create storage for. And how do you make some of those decisions around that? Well, the first thing you should keep in mind is that storage in the cloud is basically unlimited. I mean, I’m sure we could put a number on it, but for all intents and purposes, it’s affinity. Like we can say that a storage account has two petabytes of limit in United States or in Europe, and two petabytes is quite a lot. It’s larger than Library of Congress by many times, etc. You can create multiple storage accounts though.
So a second storage account will get you four petabytes, and the 10th storage account will get you 20 petabytes. There you can get a limit of 250 storage accounts per region. And so if you have access to 20 regions at 250 storage accounts each, that is something ridiculous, like 5000 maximum storage accounts across your subscription. Then you can have multiple subscriptions. Trust me when I say cloud storage is basically unlimited to subject to the limits of your bank account, I guess because it all costs money. The other thing to note about storage is the insane amount of durability. So what would you think if you put a file into Azure an hour ago and then Azure was to have some kind of catastrophe in this region and then they come back online? What are the chances that they lost your file? Well, we can say this, it’s eleven nines, which is 99. 99.
And when you do the math on that, if you gave them a million files, they would lose one of those files every 650,000 years. So that is essentially you will die before they lose one of your files. Right now there are some probably very specific circumstances around the milliseconds between you storing a file in Azure and then them having some kind of failure and the file not getting replicated to the second or third region in that quick time. But they’ve architected this solution such that the file comes in and it immediately gets put into a second system, et cetera. Now, one of the problems you’re going to face, even with this amazing durability, is if you’re storing your files in the eastern US region, and Microsoft is having some kind of trouble with east US region, you would not have access to your files. And so what you need to do, if this is an issue for you, is you need to have globally redundant storage.Microsoft is actually previewing a feature that you can actually initiate a failover.
And so if you have your files stored globally and you know this eastern region was to go down, then you can actually fail over to the other global region and you would then pick it up from there. So there are strategies for protecting your files during an outage. But it is a real problem that when the region goes down and your files are there, there’s going to be some downtime for you. Your application will attempt to do something and it will fail or with the whole region being down and your application is down as well. I mentioned before that storage accounts have limits on the operations per second. So if you’re using an unmanaged storage account, keep an eye on those limits. And you’re going to have to implement some type of Sharding strategy in order to have multiple storage accounts. That’s a manual thing when it comes to unmanaged storage. So yeah, there’s operations challenges when you’re creating a virtual machine. Virtual machines come with what is called local storage.
And so when you look at the instance types in the Azure Portal when you’re going down, you’ll see some of them come with 100GB, 200GB, 300gb, and so that storage is there and you can use it. We talked about having attached storage. So every VM has two or four or six or more extended storages that you can create. And finally there’s this concept of detached storage. So you can create a storage account that you can put files into, you can attach to your VM, but then when you’re done with it, you can detach it and re attach it to another VM. This is a way of moving files around and so you can’t mix and match those. You have to have unmanaged storage accounts for unmanaged VMs.
But yeah, there are strategies around local storage, attached storage, et cetera. If security is a huge concern of yours, keep in mind that Azure Storage accounts have a public endpoint by default. Now, that doesn’t mean that your files are public by default. They’re not. But you do need access keys in order to access a storage account over the Internet or a signature. So if you’ve given someone your access keys or you’ve given someone a signature to access a container, they can access that over the Internet because it’s got a public endpoint. If that is a problem for you, you can attach your storage account to a virtual network. And once you’ve associated your storage account to a virtual network, that actually removes the public endpoint.
And so only VMs on that virtual network can access the storage account. Or if you set up a VNet peering or a network security group allowing traffic to come through, then you can get access to storage account from elsewhere. But assigning your storage account to a virtual network is a good security thing, unless you specifically need public access to your storage account. Might be a smart thing to do.
4. Storage Management
So let’s end this section talking about storage with a discussion of the various tools that Azure gives us to manage our storage accounts. Now, by default, storage accounts are encrypted. So when you go and create a brand new storage account and store some files in it as it sits on the physical disks within the Azure data center, it’s actually sitting in an encrypted state. This is called Encryption at rest. And now it is transparent to us. So we don’t actually handle the keys. We don’t actually encrypt or decrypt. That’s called transparent data encryption. And so that means Microsoft Azure behind the scenes is taking our plaintext, our binary files from us, encrypting it and storing it on a disk in an encrypted state and then in reverse, when we go and request a file, it decrypts it as it comes back and out of Azure. So it’s transparent to us. But that means that Azure is controlling the security keys for that. So one thing you can do is you can modify the encryption settings for your storage account.
And so you can use your own key. And that’s stored in what’s called the Azure Key vault. So you can go and generate a managed key in the Azure Key Vault and then attach this to your storage account. And then at that point, you control the key. So that simply means instead of Azure having the key, you have it. Now, there’s also the concept of the storage lifecycle. Now, we talked about this in the Access Tiers video just a little while ago. But you can actually go into Azure Storage account and set up a lifecycle where stuff is going to sit in storage, in hot tier storage, and then it’ll move to cool storage after so many days that you specify. And you could also optionally have it move to Archive storage after so many days since modification. And so this is basically what they call lifecycle management, which is you just pop the files into your storage account and Azure will automatically move them into the cheaper storage tier after a certain number of days. You can even set it to delete after a certain number of days. So if you have log files or you are just historically chronically something, but you don’t need more than so many days worth, you can just have it delete. There is a software program called Azure Storage Explorer. Now, you can download this and install this on your account. This is a screenshot of the Storage Explorer that’s part of the portal. So similar to this, you can go into your subscription, you can go into storage accounts, and you can see the containers and you can see the files inside the container upload download stuff like that with the software.
It’s basically allowing you to manipulate the contents of an Azure storage account from your own desktop. Now, it’s not necessarily great for moving stuff around and doing large scale copying. There’s a tool called AZ Copy which everyone should be familiar with, which you can download again to your local and point that to a storage. Account and copy files from one storage account to another. Now, the great thing about this is it does it behind the scenes. And it’s not actually downloading the files from Azure to your desktop and then uploading them from your desktop back into the other storage account. It’s actually connecting the two storage accounts behind the scenes. And it doesn’t even have to be the same subscription. If you’re passing in the correct keys, you can also use the secure signature. So AZ copy is a lot of options for copying files. And these are the types of management tools that we have to manage the storage account.
