19. BIG-IP Troubleshooting Part 1
After troubleshooting connectivity issues layer one, layer two, layer three and application related issues, let’s troubleshoot our Fybig IP device. Now, what I have here is just a simple connectivity. We have our client PC connected to our FY big IP via the management network and the reason why is because we are not troubleshooting any traffic going to the servers or from the client. And the issue is very simple. You as an F Five Pip specialist, you just realize hey, I’m not getting any log messages from the GUI. Now, how to verify and what to check to start troubleshooting because we still don’t know what needs to be checked. Is it related to configuration objects such as virtual servers, pools, pool members, nodes? Or is it related to network configuration? Is it related to system configuration.
Now, the right answer for this is log message is in a specific directory. This is stored in a specific directory called Varlog. And if you’re looking for Ltm related messages, it is in a file called Ltm. Now, to verify why it’s not getting any messages, you can start with a command DF DF. This is disk full and this is a command to show the amount of free disk on each mounted disk. The use of all displays that is reported by DF reflects only 90% of full capacity. Meaning if one of the directory reaches 100%, in reality, from our system this is only 90%. Now, looking at the output, you see that that Shm is 1%, config directory is 3% and user 83% VAR is 30% and shared is 13%.
But look, VAR log directory where many modules are residing not only Ltm but also Asmgtm and many more, is already 100%. And again, this is based off from the command DF. By the way, DF as the command this is executed in our advanced shell in our Linux bash. And if you’re familiar with Linux, most probably you’ve already used this command before and H is for human readable. And it is very obvious that this is the reason why you are not seeing any more new log messages and troubleshooting. Big IP is not only related to the GUI or tmSh in your exam you might see this kind of questions that is not related to, again big IP configuration objects GUI. It’s not related to network in general, but related to Linux commands.
20. BIG-IP Troubleshooting Part 2
Just like from the previous example. It’s only the Windows client connecting to the big IP via the management network. And our issue is also related to managing our F five big IP appliance. Now, the first issue is we are unable to access our F five big IP GUI and we need to generate a snapshot and download it. This is the Qk view file because when you open a case more commonly, or there’s a higher percentage that they will ask for your fib iPQ view so they can analyze it from the icon. The problem is, you cannot access your big IP GUI. But good news is, you can access it via cli, via Ssh. So all you need to do is access this via Ssh. And the question next is, how can you generate Qk view file from Cli? I’m here in our F five VIP GUI, and we already know where to download our two K view file from our F five configuration utility.
Under the system module, you go down from the very bottom option, which is the support. You click Support, and you click New Support Snapshot. And from there, generate and upload the Qk view. I help, which is the default. Or you can just generate and download Qk view. Right. But again, in our problem, we cannot access the M five PG IP GUI. And one more thing. It is also telling us when was the last generate and downloaded Qk view file. And I reset. I deleted. So you won’t see any record generated here in our F five big Ipcli. And this is the only access we have based on our second example. Now, generating and downloading Qk view is very easy. All you need to do is go to the tmSh.
Excuse me? Or you can actually use tmSh command on your F five pip advantage tmSh and enter Qk view. What’s going to happen here is it will generate the Qk view file, and we’re done. As you can see, our Qk view file has been generated. It also tells us where to download it. Where can we download it? It can be downloaded under varpdirectory and the name is bigipip one F five trn. com Qkvu. Now, since we don’t have access to the GUI, you can retrieve this file from many different options. You can use Sftp or you can also do Scp secure, copy this specific file VAR, TMP, big IP, and then the destination of your server or your destination of your transfer server. I’m back in the GUI. And as you can see, it shows us the last time we generated Qk view file.
21. BIG-IP Troubleshooting Part 3
Now, our next example is related to overutilization as we have here, we have our Fib IP connected to three servers listening to port 80 and these three servers are added in a pool called Http underscore pool. And this Http pool is associated to our virtual server with an IP address of ten 10110 listening to the port 80. Now, we know that the client sends requests or Http requests to this virtual server. And since there is already a pool associated to the virtual server, we expect load balance of connection from the Vgi down to the three servers. It can be server one, it can be server two, or it can be server three. But the problem is one of the server is overutilized and upon checking in our network map it shows all three members are online and the load balancing is set to round robin.
So again, we’re expecting load balancing of connection to all three pool members and we still don’t know what are the other configurations, we just know that load balancing is configured round robin and all of these objects full members knows are all up and running. What’s the possible cause of the issue? Because there’s only one server that is pretty much getting all of the traffic from many different clients.I am back here in our Fibip and as I mentioned in our problem, we’ve already verified that our application objects are available, including our pool members 170 216 21, two, three and our Http pool as well as our virtual server Http underscore Vs.
Now, we’ve already verified that the load balancing is enabled. If I click the pool Http underscore pool it’s not only help monitors that is enabled, but also the load balancing. And the load balancing method is set to round robin. Now, we also verified that the Ratio and Priority Group is set to default. Because if you change Ratio and Priority Group, this may also the cause of over utilization, where your big IP connects to just one server and it may not connect to other servers because maybe one pool member or one server is assigned to the highest Priority Group. Now, let’s go to our Windows client and test.
I am going to open a web browser and I’m going to type our Https IP address and it is connected to server one. If I click source IP address it verifies that yes, it is connected always. Not just now, not just the next connection. It is always connecting to 170 216 21. And if we go back to our main page, this is our client IP address and it’s using a client for 58261. If I refresh it, it changes. So this means it’s using different session. So it’s not real load balancing at all. It always connects to server one. Now, what’s the possible issue for this? It’s not really an issue, but what’s the configuration that affects this over utilization of a single server? In this case it’s server One.
Now if I go to virtual server and if I select Http underscore Vs you will see that the configuration is pretty basic. Most of these settings are the defaults, but if I click resources there is no I rules. But you see a default persistence profile which is a source address affinity and this may be the reason why we are only connected to one server. And let me do it again. I’m going to connect to the RVs and it’s still persisting to server one. Now to verify it’s really persisting to server One, let’s go to our tmSh. I’m currently here in our Fyb ipadvanced shell. To go to tmSh I will just simply type tmSh and I need to go to the Ltm module in order to see the persistence record.
I’m going to run show persistence persistence records and as you can see it’s persisting to the first server. 172, 170, 216, 21 source average persistence is good for some application because we are able to maintain the connection from one source IP address to that particular server. But here is one common issue for the source address affinity. If a site with thousands or it can be 10,000 or 100,000 of clients and this site is just using one public IP address, they’re sharing this one public IP address. The big IP will persist that one public IP address and it may receive multiple, when I say multiple tens of thousands or hundreds thousands to that one server only. So if you’re going to enable source IP address persistence, you should consider these.
22. BIG-IP Troubleshooting Part 4
This is related from the previous example, previous topology where we have our F five big IP connected to three servers. This three servers is added in a pool called Http underscore pool which is already associated to the virtual server Http underscore vs. Now there is an astronaut that you would like to do and that is to delete this pool. For some reason you want to delete this pool and every time you try to delete it, it is always unsuccessful. What is the possible reasons? We are trying multiple times to delete this pool but we are not able to. And why is that? Let’s try it one more time, shall we? I’m going to select Http pool and by the way, I’m under local traffic pools pool list and I see all created pools, but this is the only pool that we want to delete.
I’m going to select and click delete. Okay, I’m going to confirm that I want to delete this pool. Click delete. I still unable to delete it, but we see the message when it’s the message cannot be deleted because it is used by a virtual server. That makes sense because if we delete this the virtual server might try to search where is that pool? It is associated to me and I was using it for the past few days and it’s all gone now. That’s a good idea to not allow just someone else delete this pool that is currently used by the virtual server. If we go to the virtual server and let’s verify if this pool is really associated to it. This is the properties page and as you can see, most of our configuration or profile configurations are here. If I click Resources you will see not only persistence and I rules configuration but also the pool.
And we verify the Http pool is associated to this virtual server. Now I’m going to detach this Http pool. It’s very easy. All you need to do is click the select Moff and select none. I’m going to click Update now and there you go. Our default pool is none. No default pool for this virtual server. If I go back to my pools configuration, I still see Http pool here and the status still green. Circle doesn’t matter because we know that one of the pool members are up and available. Now I will accept to delete this Http pool one more time. Let’s see if we’re going to be successful. I’m going to click delete now, confirm delete. There you go. We are successful deleting this pool. So it is very obvious the issues because our Http pool is used by the virtual server Http underscore VX.
23. BIG-IP Troubleshooting Part 5
This is our last troubleshooting example, and I save the best for last. Now we have our Big IP and it is actually not related to server One, Two and Three. And the client no, the Big IP is unable to reach the Ihealthf Five, and this is needed in order for us to upload the Qk view file automatically. We don’t need to go to Iheld F Five. com, log into our account, download the Qk view file, and upload it to the Ihealth portal. Now it can be done automatically like what we discussed on the previous section. Here’s the problem. The big IP unable to reach Ihealth F Five and you, as an F Five specialist, needs to troubleshoot while your F Five big IP device unable to reach the F Five Iheld portal.
I’m here in our F Five Big IP cli advance shell, and I will show you how to verify if the destination is reachable or not. First, we’re going to use a command Ping. This allows us to verify if our destination, IP address or name is reachable. I’m going to specify our destination Ihealth F Five, and as you see, the name is translated to an IP address. And you also see a return from the server which is Iheld F Five. As you see, 15 packets is transmitted and 15 is received 0% packet loss. Now, if you want to verify the hops it took from your client station or from the VIP excuse me, to the Ihealth F Five, you can use a command called Trace route. And I will again use Ihealth F five. com as the destination.
It provides us the first hub, which is our default gateway. The second hub is another layer three device in our local area network. How do I know? Well, because it’s using a private IP address. The third hub is our service provider router 4th 5th hop. This is already the service provider backbone, and this is already the agent. It somewhere in between five to maybe nine or 10th hop. And as you can see, the 10th hop is already F Five networks and it went to Santa Clara. And after 11th hop, we don’t see any more the next few hops, maybe because they already block the port or the protocol used by our trace route. But again, this is how to verify if you want to see the hops it took to the destination.
Now, let’s say you want to not only resolve the name to IP address, you want also to verify and see the details of the domain name. Okay? It’s not only Ping or Icmp to do this. We have what we called digihealth fy. com. And as you see, this allows us to get all of the details, not only the IP address 104 2191-1014. This is the same IP address we see from our ping results. We also see the server that resolves this IP address. And we also see the DNS header, the opcode, the status and also the flags. Now let’s troubleshoot based on our issue, first thing I will do is ping the destination ping ihealth F Five. And look, the reason why our big IP device unable to reach Ihealth F Five. com is because it doesn’t recognize this name.
Now, if I do traceroute again Iheld F Five, it is also telling us, but in more detail that this name cannot handle host commandline iheld F Five. com on position one. I don’t really know what exactly means, but it seems to me that it cannot resolve the name to an IP address. Lastly, I’m going to use the Dig command and this will ultimately verify if the name cannot be or can be resolved to an IP address. Dig ihealth f five. And as you see, it’s not resolving an IP address. And look at this warning recursion we requested but not available. We are not getting an IP address resolution at all. So it seems to me that the issue is a DNS issue. Let’s check our fib IP GUI.
If DNS is configured and added and all I need to do is go to the system module under configuration under Device, click DNS. And as you see, there is no DNS added in this DNS configuration page. So I will just simply add a DNS IP address. I will click Add. Maybe someone removed it, or for some reason it was gone. I’m going to click update. Now I will go back to our advance shell and verify if we can reach Ihealth F Five. I’m going to ping ihealth and it’s now responding. If I do dig iheld fi. com, we are now getting an answer and the IP address is 104-21-9110 dot 140. And that’s how you troubleshoot the reachability to a destination host.
24. BIG-IP Troubleshooting Part 6
Big IP common Issues and best Practices when we create application objects such as pool members, pools virtual servers, it is recommended to use a monitor or help monitors, but make sure it’s the correct and right help monitors for that application. In our previous example, we assign a help monitor to our pools that is listening to port 80 and we use a network, an Icmp based help monitor, which is not really recommended. You might as well use Http based health monitors to get the correct status of our pool members. And we also use health monitors most of the time. And this is my personal recommendation. Most of the time we use it on pulls, you assign them to the pools and all pool members will inherit it directly.
It is for me not recommended to use health monitors on a specific pool member or a specific notes. Why? Because if I have 100 nodes, how will I be how will I be able to remember what specific monitor and a specific note I assign with for nodes? It is recommended to assign an Icmp help monitor for the node default, but again, as much as possible, please avoid assigning a help monitor on a specific node or a specific full number. Always check the network map. Well, it is really good to view all of the application objects in one page, and from there you will also be able to see their health status. And one more thing from the network map you will also be able to verify what I rolls is assigned to a virtual server.
Troubleshooting from Cli is also something that I recommend. We must learn how to troubleshoot our big IP from the Cli. Why? Because there are some Cli commands and functionalities that are not available in the GUI, especially from the advanced shell. In the advanced shell we can do ping TCP dump, although we can do TCP dump also in the GUI we can also do DF or this full big top allows us to view the status of our objects in real time. It’s like statistics, but statistics you need to specify what objects like pools virtual server, virtual address in big top it is there in one page, but also in real time. We also use dig. This allows us to resolve names to IP address from analytics.
Operating System use Ihealth for diagnosis well, Ihealth is one of the cool features of our big IP device because we can download a snapshot of our device and upload it to ihealth. From there we able to analyze and check the diagnosis. There is also a recommendation like do we really need to do software upgrade? What are the potential vulnerability and threats in our F five big IP appliance? And lastly, it is recommended to open an F five support case, especially if there’s a critical case in our F five appliance such as Upgrade, or if there are security vulnerability, or if you are just planning to enable high availability or clustering. So those are the cases and examples while you need to open F five support cases.
