What Does a Successful Security Program Look Like?

A few months ago, I gave this talk at our customer summit, Altitude, in Denver. The purpose of the talk was to describe what I had seen as a security professional that made up a successful security program.

My background as a student (as a Red Team lead, I saw a lot of organizational problems schools deal with), as a security consultant (as a penetration tester at Bishop Fox, I saw a wide range of security problems, security organizations, and tech stacks), and at BetterCloud (as a security architect, I do defensive security and handle constraints and implement real security) together have given me a lot of insight into what success and failure look like in a security program.

The goal here is to share what I’ve seen work. Hopefully, you come away with two or three ideas you can implement right away, depending on how far along you are in your journey to success.

First, what does failure look like?

You will better understand success by being able to identify failure:

Doing security by yourself is failure. It doesn’t matter how small your organization is. If security is on your mind only, you will fail no matter how great you are or your team is.

What does success look like?

Success is, conversely, not doing security only by yourself.

Success is having leadership buy-in, lending authority to decisions that come from the security team. Success looks like employees who feel personal ownership — the ones who involve Security before making an architectural or purchasing decision with security implications, or the ones who lock their computer workstation before walking away.

And this is what success can also look like — these two are non-security employees at BetterCloud, talking about security in Slack because they’re interested.

What sorts of programs can help us get to success?

Few of us have achieved a truly successful security program. (Or if you have, you’re reading this post remembering how bad it used to be for you. Congratulations! You’ve made it in life.) This next section will detail the successful programs I’ve seen before or at BetterCloud.

Laptop bounty

This program had the most immediate culture change. Here’s how it worked. Any employee who found someone else’s computer unlocked could jump into Slack, send a message like ‘Hacked by Jim,’ and get rewards for it (we used Amazon gift cards, but anything from kudos to steak dinners will do, depending on your budget).

People really took to the gamified nature of it, especially the sales department. After a few months, people were actually excited when we rolled out a timed auto-lock of machines (when was the last time your employees were excited about a new security tool being rolled out?). With this program, we only had to focus our culture-changing efforts on a few key people who were repeat offenders.

If you’ve ever been to the hacking convention DEFCON, you’ve probably seen their “Wall of Sheep”: a list of people who were using unencrypted credentials over the WiFi. The resulting “naming and shaming” is usually a fairly effective deterrent against bad behavior. Once we identified the few remaining people who didn’t respond to the bounty, almost all responded to seeing our own custom board displaying those who were the best at catching others and those who were the worst at locking their laptops.

After that, we talked directly to the employee. After that, we talked to their manager (leadership buy-in!). After that, we would have talked to HR. Fortunately, it never got that far.

One last piece of the program that had a big effect was that the executives agreed that their standards should be higher (or really, our CEO David Politis agreed with himself that they should, and everyone else wisely agreed). Executives caught by this program paid out $100 personally, and as you can imagine, this had an immediate effect.

Phishing yourself

A scary event that happened not to us, but in close physical proximity to us, led us to start phishing ourselves regularly. At first, it was the whole company, and then it grew into bits of the company at a time. We’d send fake emails from executives, fake Google Drive emails, fake emails from journalists after funding rounds, fake wire transfer requests, etc.

Before each phishing engagement, we would look at phishing trends and create an educational campaign and present it to everybody. We’d then do a phishing campaign, collect metrics, and present on our findings. If necessary, we’d do a follow up educational campaign.

This was very successful — so successful that employees started questioning suspicious emails out loud, wondering if they were security tests. We created an email address to allow employees to report phishing attempts, giving us insight into phishing emails that weren’t directed at us.

Regular internal assessments

These assessments are on top of whatever automatic and SDLC-driven testing you have in place. This has helped us find vulnerabilities on a regular cadence and learn how they affect other pieces in the application. Certain classes of vulnerabilities just aren’t going to be found by anything other than someone who understands how the code flows between disparate pieces and data sources. Finding vulnerabilities here goes a long way in establishing credibility and rapport with engineering departments, and there’s the obvious benefit of finding vulnerabilities before they go out.

What does this look like at BetterCloud? SAST, quarterly manual reviews of our code, and Security being involved in feature/architecture discussions before code is written.

Top-down support

This is often seen as a “You got it or you don’t, and if you don’t, go work somewhere else” — not true! If you want it and don’t have it, come talk to our executive team about peer-to-peer conversations. Or, if you’re a BetterCloud customer, schedule a session with our Expert Advisory Group and invite your executive. We pretty much always find frightening things during these implementations with new customers.

At BetterCloud, our CEO frequently makes the point that security is everyone’s responsibility. As a SaaS Management Platform, we need to manage high levels of access to SaaS applications for our customers and do it securely. No easy feat! This manifests itself as top-level support we can fall back on to justify changes we need to implement, or as vulnerability fixes that get top priority over other important work.

SIEMer down now

In this age of multiple SaaS and on-prem products that are an integral part of our normal work, one thing you can do to make your life much easier is to start asking on vendor calls: “How easily will this product export logs into my existing SIEM?” Visibility is truly a prerequisite of security, so you must have visibility into what your applications are doing — or more to the point, what users are doing with your applications.

Start consolidating your logs and alerts, make them actionable, and get insight and patterns into behavior. We like to get start/finish notifications for security-related workflows, multi-factor authentication, application behavior, cloud infrastructure, etc.

#security

The #security Slack channel started out as our team channel, but it quickly evolved into a grab bag of employees from multiple offices who were just interested in security. We usually use it for discussion — the latest vulnerabilities, hacks, security tech news, etc. Often, the discussion isn’t driven by us; it’s driven by others. We get a pretty good mix of people who participate, or at least read. It’s even become so popular that people have used it in the past to let us know about vulnerability announcements.

CSP

CSP, for those new to it, is Content Security Policy, and it governs which resources your application front-end can load and what sites you can connect to. It’s most often used for setting rules on Javascript files — where they can come from and what dangerous directives they can have in them (here’s looking at you, eval). CSP is a program to do for the first six months or so, but it turns into a result of a successful security program a year out, if you still have it and it’s not neutered.

One of the first things we did with the new BetterCloud platform was set up our policy. Since then, I’ve never let go, even if it means telling people no, which has forced me to be able to explain myself really well and help find creative solutions. It’s also forced me to inject myself earlier in the SDLC process, to prevent wasted time when someone starts developing off a library that will break against our policy. This side effect has created great security hygiene in our development process because we are consulted much earlier in the process and can head off potentially disastrous decisions.

Weekly check-ins

Often with security, we don’t have the ability to do everything we want. We need engineering teams to code fixes, or Platform Services to modify infrastructure, or IT to enable security features. Even if we could do it ourselves, we’d likely be stepping codes or not adhering to their own internal practices.

Since that means we often have lots of requests (though we try to take on work where we can), we try to consolidate requests and status updates into adjacent team meetings. Adjacent teams in this case are Platform Services (DevOps), IT, Engineering, and Architecture. We ask if they have any tasks for us (usually support or research), and we go over them and prioritize them. The idea itself is really boring, but it’s been one of the most effective ways to build relationships and keep in step with other teams.

Since it is a two-way street of asking for and giving tasks, it does a lot to build rapport between teams that we work with often. Additionally, a little bit of “social engineering,” or inviting them to our security team outings, does a lot to foster strong relationships.

No gold star for participation

For dev teams, we’ve met with them in a different sort of way. We’d sit down with the architect and the product owner, present all the outstanding work they had remaining, and then follow up with a score for what they did the previous month:

On a scale of 1-5, how much work do you have left? How much work did you get done in the last sprint?

I thought this was really campy at first, but it has ended up really getting teams chasing after that 5, even people on the team who I thought wouldn’t care. Eventually, we got teams chasing the “6” — not needing to have this meeting anymore. Everyone involved knew what I expected well enough that I didn’t have to meet with them to remind them.

End result

The end result is worth all the work required to get here. I come into work empowered by leadership and enjoy working with my coworkers. By having good security programs, gamifying education and security tasks, and convincing everyone we are all on the same team, I get:

• Lots of trust from other employees
• Developers with security issues coming to me as a sounding board, sometimes just to hear me say no!
• Employees self-reporting security events (policy violations, vulnerabilities, etc.)

What specific security hygiene issues do you face? Where do you lack visibility? How can you get teams to be accountable and responsive? How can you get other teams to involve Security in their process? Hopefully, you can take some (or all) of the above projects back to your security team to look at ways to improve your team’s maturity and bring it closer to success.

If you want to contact me with ideas, feedback, or to ask for suggestions (but not to sell me products), don’t hesitate to email me at austin.whipple@bettercloud.com. I will respond in the following manner: