198. Lessons Learned (OBJ 4.2)
In this lesson, we’re going to talk about Lessons Learned, which is a key part of your post-report delivery activities. Now, Lessons Learned are an analysis of the events that can provide us insight into how we can improve our penetration testing process in the future. The Lessons Learned process is really a formalized method for us to document the things that we experienced during this engagement. What went right? What went wrong? What can we do better next time? All these are things that should be recorded and our internal organizational processes need to be improved, so that those same issues don’t occur again during the next engagement. For example, maybe our approval process to be able to run an exploit was too slow for us to be able to take advantage of some vulnerabilities we found.
Well, one lesson learned here might be that we need to capture the need to delegate down lower level of approval authority, so that we can get approval to run those exploits quicker, because it held up the entire process of events that we had planned. Now we don’t have to go and redevelop the entire approval process right now, but instead we just want to capture that this was an issue, and it’s something that we need to address later on.
That is the purpose of the Lessons Learned process, to capture those issues and maybe come up with some suggestions of how we can improve it, but we don’t have to solve it here today. Now, to conduct a Lessons Learned meeting you want to structure it using some basic questions. I already gave you a couple of these that we started with. Things like, what went well during our engagement? What didn’t go so well? What could we do better? What didn’t go as planned? But we can also ask other questions.
Like, what can the team do to increase its people skills, so they can communicate better during the engagement? What about our processes and technology, are there things we can do to improve those? What can we do to engage better with our clients, to make sure they’re happier? Are there new vulnerabilities, exploits or other things that we learned about during this engagement? There’s been many engagements I’ve been a part of, but we went to run an exploit that worked every single time before, but guess what? Organizations have gotten smart and they start patching those vulnerabilities, so our go-to exploits don’t work anymore.
We’ll start documenting that in our Lessons Learned to tell the rest of the team, “Hey, don’t rely on that exploit. You need to come up with some new techniques.” These are the things you can capture inside a Lessons Learned. Conversely, maybe you found a way in that was really novel and unique, and you want to share that with the other penetration testing teams in your organization. The Lessons Learned process is a great place to capture those new exploits, and then share them with the rest of the teams.
And then finally, as we look through this entire process of the engagement, what things can we change and what remediations do we have, to fix the issues that we came up with? For example, let’s say the way we were taking all of our data from the assessment and putting it into a central database, wasn’t being very effective for us to be able to actually get data back out of that database, because that tool didn’t support the needs that we had. We can notate that here in the Lessons Learned process, make recommendations of how we can remediate those issues, and then start working on a plan so that in future engagements, we use a different tool, or we modify the existing tool to better serve our needs.
Now, at the end of this Lessons Learned meeting, we’re going to create what’s called a Lessons Learned Report or in some organizations they call this an After Action Report or AAR. Now the thing I want to remind you here with Lessons Learned Reports is that if nobody actually reads them and implements the changes, then they’re not actually lessons learned. They’re just lessons documented, and lessons documented don’t make us any better. Instead, you always want to take some time to go back and look at the Lessons Learned Reports from previous engagements.
For example, if we’re going to go back to the same organization 12 months from now and do a retest, the first thing I would want to do is pull up the Lessons Learned Report from the last engagement, and see what worked well and what didn’t. Then we can put some new processes in place to make sure we don’t run into those same issues with that same client again. That is the idea of Lessons Learned. It’s learning from your mistakes and continually improving your processes as a penetration tester, and an overall penetration testing team.
199. Retesting (OBJ 4.2)
In this lesson, we are going to talk about retesting, because usually, a penetration test is not a singular event but if something that’s going to be done on a regular basis. For example, if your target organization hired you as part of a PCIDSS compliance requirement, they’re going to need to do that again about once a year. And so you’re probably going to be coming back. Now, when you’re planning for your retest, you want to analyze any progress that’s been made by applying the mitigations from the attack vectors that you identified during the last penetration test. After all, this makes sense. At the end of your engagement you gave them a long list of things that were wrong with their network and they went and hopefully fixed all those things. But you never can be too sure. So you’re going to want to go and check the same ways you got in last time to see if they’re still going to be effective.
For example, in one large organization I used to do a lot of penetration tests with, every time we did a penetration test, we were hired to come back 30 days later, to go back after any vulnerabilities that we listed as critical. Then, we could tell them if their system administrators and their security staff have already patched and mitigated those critical vulnerabilities within 30 days of us leaving. That was the way this organization used to be able to recheck their people and verify they were actually doing their job and fixing the things that we found wrong with the network during the last penetration test.
Now how often you’re going to come back is going to be dependent on the organization, their budget and their regulatory needs. Additionally, you may be working with your client to do mini penetration tests after the big one. For example, once a year you might do a full scale penetration test but once a quarter, you might go after just a couple of specific areas that they want to verify they’re not vulnerable in. For example, maybe there is a forward facing web application that ties into their customer database and they want to check that at least once a quarter to make sure nobody can get into it by using your penetration testing services in addition to their own vulnerability management services.
Now, the whole reason that we have to do these retest at some interval is that, penetration tests are only a spot in time assessment. That means when you finish that penetration test, that network is now getting more secure or less secure depending on whether they took your recommendations and started applying them or they’ve ignored them and those networks are getting older and more vulnerable by the day because more exploits for older software are being found every single day. And this is why retesting in the penetration testing space is so important because networks are changing all the time and we need to make sure we understand what is the status of that particular network, if we’re the organization who owns that network.
