For this instalment of our Customer Stories, we sat down with the Information System Security Expert of a major European financial institution. Read on to learn how a heavily regulated industry implements Bug Bounty.
What made you decide to launch a Bug Bounty programme?
In all honesty, I wasn’t convinced at first. I had met with a few service providers who suggested I implement a Bug Bounty programme. Still, I had the overall impression that it was a “patchwork” of various missions carried out by their pentesters and not an established, focused activity.
Then I met with YesWeHack representatives who presented different advantages that met my expectations in terms of security. I started to see things in a different light and I took the plunge at the beginning of 2018. We launched an initial Bug Bounty campaign on a site we knew well and had already tested multiple times before; choosing a mature scope enabled me to better control the potential costs of the programme.
I wasn’t expecting to get any feedback as we were already on our eighth penetration test on the site since I’d been on the job, and the last test had been done just the week before. We were reasonably confident. But the result proved us wrong: we received multiple vulnerability reports, including one that was critical, just an hour after launching the programme. We now know that this vulnerability had existed for a year and a half. In other words, two penetration testing had taken place without ever noticing it. So, Bug Bounty instantly appealed to us. (Laughs)
In a broader sense, what do you see as Bug Bounty’s added value compared to penetration testing?
I see four main elements.
The first – and the one that appealed to me at the start – is the fact that you’re getting something close to the real thing: meaning, we’re not trying to get a comprehensive view of the vulnerabilities. We’re trying to hit right where it hurts, immediately. That’s very close to what a real attack would do.
Second, and crucial point, given the current agility trend, is continuity. We continually update our internet applications (at least once a month), so it becomes complicated and costly to pentest each deployment. Bug Bounty allows us to continuously monitor.
Third, and generally speaking, is the efficiency and quality of the work done by the hunters. On sites that we’d tested multiple times, we systematically found vulnerabilities, some of which were critical. The hunters are paid based on results, so, clearly, their goal is to find something concrete for us—and that’s my goal, too. One of the problems with penetration testing is that the results rely entirely on the pentester.
With Bug Bounty, we work with specialists. The hunters aren’t necessarily going to go looking for every type of vulnerability that may exist on a web application. Instead, they go where they perform the best—and where they can be swift and effective. And that’s what we’re looking for, too.
And the last, but definitely not least, is the ROI. Between the bonus budget and the cost of the platform—and given the results—over the year, it’s very profitable. I have twice as many vulnerabilities reported via my Bug Bounty programme than I did with penetration testing, and it costs me about 50% less.
I’ve been pentesting specific, critical sites for years and getting empty or nearly empty reports in return. We were paying because it was our policy, and we needed to meet our compliance requirements; but it brought us nothing, or almost nothing more, in terms of security.
Does Bug Bounty mean the end of the pentest, or are the two complementary?
They’re still complimentary, but it’s going to reduce the perimeter seriously: we’re going to try to keep the same number of penetration testing, but focus them on less critical areas.
Now, I’m wondering if it makes any sense to do annual penetration testing, on applications managed in agile mode. And for all the critical web-based applications, we’re moving towards full Bug Bounty.
How does Bug Bounty work in your company? Have you noticed any changes in your teams since launching your Bug Bounty programme?
We’re in a ‘managed’ programme, meaning we don’t do the vulnerability triage ourselves. But, internally, I’m in charge of overseeing the Bug Bounty programme. It’s a little time-consuming at first; you need the discipline to avoid the trap of going to look at every vulnerability, one by one, as soon as you get them. Once you establish your rhythm (for me, that’s once a week), it’s advantageous. Speaking of time, it’s close to that of managing a classic penetration testing. On the other hand, it’s a lot simpler to launch and monitor than a penetration testing.
We gave the Operational Security team direct access to our program. Thus, our colleagues can see the vulnerabilities as soon as they’re validated. Also, our developers can interact directly with the hunters. We’ve involved them right from the launch of the project, so, it wasn’t viewed as a ’punishment’ at all, but rather as a game of offence/defence. That’s exactly how I wanted it to work; it was like: ’Oh, they’re good, how do they do that?’. (Laughs)
It has made us a lot more effective. It helps them improve and, in general, helps us make progress in terms of security.
And the impact in terms of agility?
We were already aware that we needed to do things. Still, we couldn’t address the need for checking applications in production by performing penetration tests (monthly roll-outs). In essence, we were in the process of deploying without really respecting our security policy because we could only audit them over time. Therefore, we needed to find another solution, and that solution was Bug Bounty. I do feel like Bug Bounty *is* designed for agility.
We can’t be agile when we’re doing pentests; it doesn’t work: there are just too many projects to follow, and we can’t do one before each roll-out due to the lack of time, responsiveness and means. The deadlines are too tight, continually shifting, and the tests have to be scheduled several times a year.
For example, when a vulnerability is reported, validated and sent for processing, we’re told internally that the Application Firewall has been updated as a result. Then, we ask the hunters to check – and they always manage to exploit the vulnerability by going around the WAF in another way. This interplay is continuous improvement in action and lets us show that the code needs to be fixed. If necessary, we can put the hunters and developers in contact, to further discuss the vulnerability and its fix.
Everyone is empowered, equipped, and a lot more responsive—that’s the beauty of DevSecOps.