1. Should you use gsutil or Cloud Storage Transfer Service?
So welcome back. Let’s talk about storage migration and what utilities you should use. How do you know what tools you should use and in what cases? And this is going to be a big part of the exam. Of course, you’re going to hear me talk about this several times because the reason I have a specific section just on storage migration is because it is heavily tested on the test and is also not covered in the cloud architect course. So this is a weakness in the Google Cloud training that does not address some of the areas that are on the test. So that’s why I wanted to do this little special section to make sure that we knew what to expect on this exam.
Okay, so here’s what I’d like you to know and what you want to use as a guideline for the task when you’re answering questions. So the case study questions or just basic questions that they asked don’t have to be case study questions. What it appeared to be—at least the focus of the exam questions—was more focused on how you would migrate to the Google Cloud Platform. That makes perfect sense. They want a cloud architect to understand how to bring data to the cloud platform. Now if we’re just talking about buckets, then we want to think about GS Util or cloud storage. I’m not talking about databases, Bitable, or anything else of the sort.
I’m not even getting into that. So I just wanted to make sure that we’re on the same page and that there is no official answer. But I guess if you’re going to get an official answer, Google Docs is where you want to go. So they have this. Should I use GS, Utile, or cloud storage? Transfer. Well, that’s a great question, and my answer to that is: it depends. And again, there are a lot of different reasons why you may want to use one over another. So you’re going to see me highlight this again when I specifically cover the cloud storage transfer module.
And it’s because you need to know this because you’re going to give up two or three questions on the test if you don’t know if you should use GS Utile or the storage transfer service. So again, when you’re transferring data from Orem, use GS Util. So remember, customer site use of gustily doesn’t mean you can’t use the Storage Transfer Service. It depends on the use case. But this is the sort of thing you want to use on the test. And then what about if you transfer data from another cloud storage provider? Again, you want to use the storage transfer service, right? So if you’re on Amazon, you can use the storage transfer service. That’s great.
Now it says “another cloud storage provider.” Now that there’s really nothing that addresses, in my opinion, storage transfer service per se, how would you migrate from Azure? Or how would you migrate from Rackspace? It isn’t there, but you could certainly use Storage Transfer Service to migrate with tab-delimited files, or TSVs, which I believe is the correct term. But again, just again, for the test, I’m really harping on this because this is an area that I think was not covered if you took the course. So you need to know this. So with that said, to transfer data from On Perm, use gustily. So, once again, you receive a query from the customer site to Google Cloud. The right answer would likely be gustily. If you’re transferring from another service provider like AWS, the right answer is: “What storage transfer service?” Okay, I think I made my point. Let’s continue on. Let’s go and find out more about each of these utilities that you could use to transfer data.
2. Cloud Storage Transfer Service
Welcome back. So let’s talk about an option for you or your customers to be able to transfer your data from in-house or from AWS, say, to Google’s cloud platform. Now one of the reasons this has really cool features is that it’s actually built into the console. But before we go into the console, let’s just talk about what it is and why you may want to use it.
And so let’s say you’re operating in Amazon now, you’re in production, you want to do a storage migration, and again, you don’t have a lot of time to do it. Of course, this is assuming that you have the bandwidth to transfer from one cloud provider to another.
Now, to use the cloud storage transfer service, what you’re going to do is set up a data source for a data sync. Now, a data source is going to be, of course, where you’re transferring data from, and the data sync is where you’re going to go to. Now, a data sync is essentially, basically, a Google Cloud Storage bucket, which is exactly what it is. So you’re going to create a bucket in GCP, and that’s also known as a sync. So I don’t know why they didn’t call it a target; it just would have been simpler. But again, every vendor is going to have an acronym or two that they can pad into their trademark or something. But, once again, for the test, you’ll want to know this because you’ll see a question; in fact, on the beta, there are two questions on the storage transfer service, and you’ll want to know what this is, why you want to use it, and what the benefits are.
So let’s just clarify that. Now, in general, here’s a use case or two for why you might want to use this just so you can put it together. Basically, you could again—actually, it says it right here—back up your data to a bucket from other storage providers. So basically, you want to use cloud storage as a target. Or you could move your data from a storage bucket to a near line bucket. As a result, you could use this to migrate from one Google storage tier to another. So again, let’s say you’re in Asia Pacific and you want to migrate to another region, let’s say, or you want to migrate within the same region and you want to migrate from a highly available storage bucket to a near line, let’s say. Again, this is up to you to decide what scenario makes sense. Some of the advantages include the ability to schedule this. So you could schedule this to make it happen.
Again, there are numerous ways to accomplish this. You could set this up in the console, or you could use this in the API library. So you can download the API if you like. Rest APIs. Again, you could go ahead and use that as well. Now, I like this little tidbit here. Again, this is a good little tidbit about why you might want to use gustily over cloud storage—pardon the slight mouse movement here. Now, we’ll talk about gustily in the following session, but GS until is another command-line tool that does the same thing. Now basically, you want to use GS Utile where you’re on premise, for example, or, again, if you’re going from one provider to another, or perhaps you’re just trying to figure out what tool makes sense. Now again, you need to understand the charges as well before you transfer data.
So you want to pay attention to that stuff as well. But again, you could use it again; that doesn’t mean you can’t use GS Utility with the storage provider. It’s just that Google has optimized the storage-transfer service to transfer from Amazon. So again, it was meant to work with S 3, for example. So just pay attention to why you might want to use each of these. Again, it’s hard to give you a solid use case over. You want to use gustily to transfer data to the cloud. But again, just for test purposes, this is what you need to know. Use US-Util if it’s an on-premises location and bandwidth is not an issue. Now, if you’re transferring on premises and you have bandwidth issues, you want to use import and export. But what about if you’re going with another safer service to Use the storage transfer service. So highlight this for the test. This is in your cribPDF study sheet, which you can download. This is very important because you’re going to get numerous questions on transferring data to the Google Cloud Platform. I would like to say that, just on the non-beta test that I had taken about two months ago, there were at least five questions on transferring data to GCP.
So again, that’s why I separated these. And the other thing is, again, that the course for the architect doesn’t really cover this very well. To get this information, you actually had to take the CP 100 course as well to get some of it, not all of it. So again, this is where you want to pay attention because this is a good chunk of the test. So I think I’ve said enough. Study this; make sure you get this. There will almost certainly be some questions about using the cloud storage transfer service instead of import exporter gustily or GS fuse, for example. So with that said, let’s carry on.
3. Cloud Storage Migrate Buckets from GCP Regions – Demo
I’m over here at the Google Cloud Platform storage dashboard, and I’m over here in the browser section of the dashboard. Now, what I’d like to do in this demo is simply migrate files from one region to another. And that’s one of the use cases for the transfer service. To transfer files from one Google data center in one region to another region, another use case could be AWS, or perhaps your own data center. In this case, we’re just going to go ahead and show you how to deal with Google. It’s simple enough. Now, before I get started, I have a couple of things to point out. I have buckets. Under buckets, you could see that I have two migration buckets.
What I did was just create two buckets so that you don’t have to watch me create buckets again like I did in the previous demo. Now I have a migrate bucket and a migrate bucket. What we’re going to do is migrate files from Europe to the USA. But before we do that, let’s just make sure we got files in there in Europe. So I’ve got 123456 files in there, and this is the bucket EU. Let’s go back and check what’s in the USA bucket. Okay, so there are no files in the USA bucket. What we’d like to do now is transfer the files to the United States. Now the easiest way to do that, because there are only a few files, is to use the console. If we go over here to transfer, you can see that it brings up the transfer message that says “Create transfer.” So it says you could transfer data to your cloud storage buckets from Amazon Simple Storage Service S Three.
You could also do HTTP or HTTPS servers or other buckets. You can schedule one-time transfers or daily transfers. Again, you could go ahead and tag them. You can make prefixes or whatever you want in this case, but for demonstration purposes, let’s go ahead and make a transfer. As you can see, it now says select source. The source we want to use is a Google Cloud Storage bucket. Now, if you want to use AWS, when I click that, all you have to do is put in the URL of the bucket and your credentials. And it works fairly simply as well. You could also go to the list of objects. So you put in the URL of, for example, your web server at your infrastructure, wherever that is—at your work or at a colo. Whatever you’re doing, go ahead and do that as well. Now there’s a course-specific format you want to look at. Make sure you check that out, because this will fail if you don’t follow it to a T. But let’s go to the Google Cloud bucket. Now you can see that it says “Browse.” I’d like to do some quick browsing. Now you can see that it has bucket names: bucket from and bucket to.
We’re not at the destination yet. So the first thing we want to do is what? Select the source, right? So I’m going to go ahead and select the “migrate” bucket from the EU one. I selected that, and you can see the green check mark. Now let’s say, for example, you delete a bucket or you put improper permissions on it; that bucket will not come up green if the API is not able to essentially establish a connection. So it has to be able to find it and be able to establish some kind of connectivity. Essentially. You could also go here and specify prefixes. You could, for example, specify based on the age of the file. This is a good way to configure the transfer. And when we get to the end, we could also tell it to do it every day or whatever the schedule is. So it’s fairly simple.
This is a very simple utility. So let’s go ahead and hide that. Actually, again, we’re not going to use that in this case, but it’s easy enough to do if you want. So let’s click “continue.” Now let’s select the destination. Now if I select USA, you can see that it’s able to deal with that. This is the important part. What about if you’re transferring files from one bucket to another and the bucket containing the files has similar files with the same name? What do you want to do with that? You have to tell it: do I want to overwrite the destination? Do I want to delete objects? Do I want to delete objects from the destination? So basically, you could also tell it to delete files from the destination as it writes to the source, right? So you have some really good capabilities here as well. So take a look at this. But be careful with it because you don’t want to transfer files and then delete those files from the source if you may need to keep them down the road or if you need to archive them first. So pay attention to what you’re doing. This is a great way to ruin a good weekend if you’re not careful.
So in this case, I’m going to just go click Continue. By default, it’ll drop the files in there, and it won’t write over anything or cause issues. Since there are no files in there, I’m not worried about it. Now what I want to do is configure the transfer. I could do it now or at a certain time. I could put in the time, and I could put in the description. By default, it puts in the source and the target. And then it’s only a one-time transfer. However, I could run it daily, and I could also change the name as well.But by default, most customers will likely just use this once. But this is a valid way for you to set up an archive process to do backups. For example, some customers like to drop files in a storage bucket and back them up. Again, that’s up to you. Or delete them at a certain time of the week or month. What we want to do now is simply click Create. You see what I mean? It’s telling you what’s going on. This is actually pretty cool. It tells you it’s starting, and you’ll see this progress. Now there are only a few files. The only challenge is that it’s coming from Europe. could be a pretty slow link. However, this is a Google data center to data center transfer, not one from my home or another data center. This is basically Google talking to Google.
So this should not be too long. There’s only a few files. Now I have noticed that there could be some latency in the browser. That is something that can happen if you use GS util to reduce any kind of latency in viewing or completion. That will work in this case. But again, if you have latency in your link from source to target, then that’s a different story. You may want to use the transfer service again; check out the use case. So we’ll come back to this when it’s done. This could still take five or ten minutes, depending on how busy things are. So the transfer was finished. As you can see, it is completed at 233. We started this probably about four minutes ago. Sometimes you may want to also hit “refresh” just to be sure. Like I said, sometimes there’s some latency, and you just have to refresh. Let’s go back to the browser and see what we have going on. Now let’s check the bucket. We have the files there. Let’s go back to the source bucket, and we have the files there as well. So that’s how you do a very simple file transfer between Google Cloud regions. Remember, there are some data costs that will, of course, cost you.
But be aware that you do have a specific threshold as well. You need to pay attention to that as well. With that said, that is an overview of how to transfer from one bucket to another in different regions. Simple enough. Finally, I’d like to walk you through the transfer appliance. Remember, this is something that you need to order. There is a good cost to this. Google will validate your support contracts, your cloud account, and everything. Usually you’ll get this set up through a partner; that is that section. And if we go to settings, this is where you could select and validate your project access. The API access you see by default It has these permissions, and you have this identity again. You can go over here to find out how you could modify the permissions on the transfer if you so choose. is simple enough, and then there’s interoperability as well. And that’s about all that I had for the storage transfer service.
4. Gsutil
Welcome back. So let’s talk about another tool that you could use to essentially migrate data to your cloud storage. Now, again, this is another utility. This is actually part of the Google Cloud SDK. So, once again, one of the reasons you should use this is essentially, particularly if you need to perform a broader range of object management tasks.
So, for example, if you’re transferring data from one cloud provider or from your in-house storage, you may want to transfer with specific rules for objects. For example, a common use for this I’ve seen at least two customers use is because they want to be able to rename everything. So, for example, if you have a file structure that’s on Amazon, it’s very common for a lot of customers to want to name that AWS or have a prefix in there. As a result, you might want to change the prefix. This is a great tool to do that. Now, what’s really cool is that this is supported by Google, and so you do have it in your SDK kit. So download the SDK. There was a demo I had done at the beginning of the class around this and running GS Util. You just bring up your browser to do that. Again, it’s fairly simplistic to use. You don’t need to know any of the commands or anything else for the exam. But again, play around with this if this is of interest to you.
It’s called GS Util. This is part of the SDK. It uses Python. Now, one of the things you’ll encounter on the test is that they’ll present different scenarios of customers having storage and you’ll need to transfer that storage to the Google Cloud platform. And again, you need to know the right tools for the right situation. So here’s my thought: If you see a scenario where the customer requires significant renaming or restructuring of buckets or objects, then this is really the way to go. So again, on the test, you need to know this and understand which utility. Again, it’s easy to get confused between GS Util, for example, and some of the other tool sets, for example. So just be aware of that. So you don’t want to use FUSE when you should be using GS Util, for example. So with that said, let’s carry on to the next subject area.
5. GCSFUSE
Welcome back. So let’s talk about another capability that’s out there to help you transfer files to the Google Cloud Platform storage bucket. Let’s say, for example, that you currently have your files in there in a directory structure. And, I suppose, native file structure exists with Google Cloud Storage buckets. So again, you’re going to a non-file system. That’s essentially what the cloud platform is: storage buckets that is.
So just keep in mind that if you do require a file or directory structure, you can transfer your files from that structure to Google Cloud platform storage buckets and effectively create one. So this is essentially a plug-in that you must install, and it is known as a diffuser adapter. Now this is supported by the Apache open-source community. Please excuse me while I explain that this was created by Google. And again, it’s not personally efficiently supported by Google support or anything. This is more of a here’s something to help you out kind of thing.
But we’re not going to directly support you with it. So again, just realise that this could be another option to migrate to the Google Cloud platform as well. So especially if you have something like NFS or some kind of specific file structure that you’d like to try to maintain and use buckets for, this is a good solution. So with that said, the link is below. Take a look at it. But for the test, do understand that this is an option that could be selected on the exam. And so you want to understand that if you get a question that is going to be focused on basically translating objects from a file structure to cloud storage, then you need to use a solution more or less like this. So, once again, just a thought; just make sure you understand what it is at a high level. Let’s move on.
6. Offline Media Import / Export
Welcome back. So let’s talk about another option for transferring data to the Google Cloud Platform. Now again, this is probably going to be the most last used resort possible. Again, you’re going to use offline media import and export in a scenario when you have to transfer large amounts of data and you have very little bandwidth to do that. So again, if you’re on a one-meg connection and you get to transfer two terabytes of data, well, this might be a good solution.
So with that said, when you take the test, if you get any questions that, again, ask you about the best solution to transfer large amounts of data with limited bandwidth, this may be one of those solutions. Again, it depends on the whole case study question, but just take a minute to look at this and understand that this is an option as well. Now, Quick will note that this is done by third-party providers, not by Google itself. So you have to go through one of the service providers over here to have them facilitate the transfer. So basically, you copy your data onto drives or tapes or whatever they’ll come by, you send it to them, they’ll pick it up, whatever the agreement is, and then they’ll go ahead and work it out with the Google Cloud Platform to upload your data to your cloud platform. So with that said, this is another option to basically do data transfers that are going to be typically large with low amounts of bandwidth available. So with that said, let’s continue on.
