PODCAST: John Visneski from The Pokémon Company on Adopting Serverless
The second episode of The Serverless Show is available above on video as well as on SoundCloud, and summarized below.
For this episode, Hillel and Tal from Protego Labs were joined by John Visneski. John is currently the Director of Information Security & Data Protection Officer at The Pokémon Company International, where he is responsible for everything from corporate IT systems to customer-facing platforms.
John elaborated on his background, “Previously, I was a Cyberspace Control Officer in the United States Air Force, which included working in The Pentagon on requirements and acquisitions to help steer the Air Force in the direction of the cloud and modern cybersecurity techniques. It’s been an interesting and cool transition from the Department of Defense to Pokémon.”
“At least now you’re securing things that are really important, right?” quipped Hillel.
“You’d be surprised. The things that we see here are very similar to the things that I used to see at the Pentagon,” replied John.
Serverless at Pokémon Company
Hillel asked John to describe what’s going on at Pokémon with serverless.
“We’re in the same position as a lot of companies. We start with grassroots. AWS told us about Lambda over and over. Finally, we kicked the tires to see what it can do to help our operational architecture. The short-term strategy is to continue to use serverless in ways that augment our developers, without necessarily replacing architecture writ large. So, things like using Lambda to automate blacklisting and whitelisting, and speed up things in our analytics pipeline. Activities that a dev would have to spend a lot of time reconfiguring and changing, but now with serverless architecture they can concentrate on driving the business forward.
“Will that change in the future as we get more bang for our buck? Yes, it might. But for now, we’re using it to augment things we’re doing already.”
Hillel replied that it parallels what he’s seeing. “Solve a particular problem with the technology that makes the most sense. If, and when, you fall in love with it, see how you can scale it out to other parts of the organization.”
The Impact of Serverless to Security
John explained, “It dovetails nicely with what a good security program should be concerned with, regardless of what technology you’re using. Observability, accountability, and understanding what your actual attack surface looks like, whether it’s hardware, software, functions, EC2 instances, Lambda, you name it.
“The security issue that comes inherently with serverless is the same thing that makes it awesome for devs to use, that speed, can get away from you quickly from an accountability standpoint. So you start to lack a good understanding of what your attack surface looks like.
“Now is that a reason not to go with serverless? I don’t think so. The analogy I like to use is that you’re sitting in canoe that’s slowly filling with water because it has holes. A speed boat pulls up next to you, and you say, ‘I don’t want to get in the speedboat because it’s too fast.’ Yeah, get in the speed boat! But then you have to understand that you’ll have to concentrate on a little bit more than you used to concentrate on.
“If one of my biggest problems is understanding the baseline of what our architecture looks like at all times – the fact that serverless can make that baseline either wildly large or wildly small quickly is a big challenge for my team,” concluded John.
Tal commented that it’s consistent with what we’ve seen with our customers. “They don’t really know where their attack vectors are. Main concerns are observability and defense. When you get into this new technology, you need to understand where you stand and your weakest points.”
Serverless Bitcoin Mining Attacks
In discussing the latest headlines, Hillel complimented our friends at PureSec on some cool research. PureSec pointed out that in a serverless environment, you can attack a function, and if it’s got some code injection vulnerability that lets an attacker run something, an attacker can do some crypto mining. The key point of this attack isn’t theft of your data, but using your AWS or Azure bill to execute crypto-mining.
John said, “First, kudos, because anytime you combine serverless, bitcoin mining, and crypto in one article, you’re going to be able to drive a whole lot of clicks. But companies have varying levels of knowledge of what their AWS bill really looks like. Some don’t even consolidate bills for various company departments. Some have alerts if the bill gets too high. I would be worried about someone going low and slow in a way that they could bleed a company for years, if it goes unnoticed.
“It’s fascinating, and it’s another good example of new technology, regardless if people think it’s safe because it’s fast, game changing, or disruptive, at the end of the day, someone figures out a way to take advantage of it,” concluded John.
Hillel commented that an attacker stealing your data or customer’s data is probably a bigger problem than worrying about someone mining crypto on your dime.
What Should People Be Focused on for Serverless Security?
Tal explained, “If you have a code injection, an attacker could modify your code or use your functions to run malicious code, you have other issues to be concerned about. Eventually, you can lose whole resources or the whole account if one function has exceeded permissions to read from sensitive data, then you can actually lose your customer’s data in a second.
“The main things to be concerned about are:
- First, know what you have. Map out what you need and clear everything else out.
- Follow the principle of least privilege. Make your serverless app do what it needs and nothing else.
- Encrypt sensitive data.
- Eventually, it’s good to have full defense on your serverless apps, including something to monitor function behavior in runtime, with a tool like Protego.”
The Pervasive Vendor Lock-In Conundrum
There was a recent article in Tech Beacon about vendor lock-in, which seems to be an endless topic because there is no resolution. Hillel said, “There are two important points about vendor lock-in. The first I’ll call API-level Lock-in. You write your code for Lambda and it’s not that easily portable to Azure functions, for example. I find that argument a little less powerful. It’s not that difficult to refactor functions from one cloud provider to another.
“But service-level lock-in is a bigger problem, when you’ve built your app on top of the cloud-provider services, such as S3 or EMR. And those particular services have their own ways of being interacted with that aren’t as easily ported when you move to another platform.”
John weighed in, “Vendor lock-in not always a bad thing. At the end of the day, decide what’s best for the business. If a particular technology aligns well enough with the business, not just from a security perspective, but functionality, business intelligence, and financially, people shouldn’t be afraid of a little lock-in.
“The use case of companies changing out large parts of their architecture is compelling and scary, but in reality, it doesn’t really happen as much as people think it does. Fortune 500 don’t stop business for six months to jump to another. I’m hesitant to say that vendor lock-in is as big of a problem as people think it is.”
Open Source – A Solution?
Hillel asked, “Will open source solve a problem, or just shift it around?”
John replied, “To a large extent, it’s a great idea, and we’ll see how far we get into standardizing how we’re doing serverless. But I think it really shifts the problem a little bit. If you’re trying to administer your own architecture, the value-add of being able to move up the stack with serverless is a little bit lost. Don’t get me wrong, I’m sure there are some great DevOps teams that could seize that and build out their architecture using open source. But I’m not sure the juice is worth the squeeze in this case.”
Vendor Lock Out
If you shift to serverless, how much of a concern is locking out multiple security vendors you might have used previously?
Tal stated, “You’re definitely going to need to think about how you deploy security now on your serverless application. Things like antivirus and other agents installed on the OS, you don’t own that anymore. Same thing for containers. You don’t really care about what’s going on in the environment in terms of the files or configurations there because you don’t own it. But you have to find a solution that will protect your serverless apps. Where do you put that?
“You could run a firewall on an EC2 instance alongside your serverless application. But then you lose the meaningful advantages of serverless, and how do you scale that? So now you have to maintain that.
“You need to think differently. You can’t use old defense with a new technology.”
Hillel added that the big concerns are where to put security and how to deploy, along with how to keep up with the velocity of serverless. When devs make changes so rapidly, you have to either tell them they can’t roll out code until a security person reviews it and reconfigures the WAF, if necessary, or have a security person chase.
John added, “The security person’s job in 2018 and beyond is keeping up with DevOps and IT to be an enabler instead of an impediment.”
“Great idea, embrace it,” replied Hillel.
— chrismunns (@chrismunns) June 11, 2018
I took offense to this tweet from Chris Munns from AWS, a great and smart guy. The article I wrote was trying to get across that you have 1,000 security decisions to make when you give IAM roles to 1,000 functions, and it’s challenging to get it right. But if you get the roles correct, you’re doing a huge service for the security of your application. The least privilege you’re applying can prevent the vast majority of attacks on your platform. I somehow hit a nerve as another security company trying to panic everyone.”
John added, “Even before we had all these tools at our disposal, even before our environments changed so quickly and we were spinning up EC2 instances or serverless functions in a snap, people aren’t perfect at configurations, and doing least privilege every time out of the gate. I think it’s dangerous to make these knee-jerk reactions. We can never rest on simply hoping everyone is configuring things correctly and expecting the shared responsibility model to be perfect. That’s not the case. There’s still human at the end of that keyboard, and they have a lot going on. And expecting them to get it all configured correctly is unrealistic. Instead of being afraid of that particular problem set, people should feel challenge for ops and security teams to work together in a way that doesn’t slow down, and make sure things are being protected effectively and efficiently.”
— acloud.guru (@acloudguru) June 7, 2018
John selected a cartoon and said, “It made me chuckle because the caveman analogy is apropos. If you’re in the security or ops space, it’s dangerous to be in a tribe of any sort. I don’t think any of us should be completely married to containers or serverless. The threat landscape is going to change a lot faster than you can keep up with. If you’re married to a tribe, you’re kinda being a caveman, and you’re also not married to the business. And if you’re not married to the business, you’re apt to fail.”
Tal selected a screenshot from a presentation at a Java conference.
— João Carvalho (@johnkarva) June 19, 2018
Top 5 Reasons You Need Serverless
- Your Sys Admin quit last Friday
- You got a million free queries… have to use them up
- You are already doing microservices, and want even more complexity
- You have a short memory… and like your applications with minimal state
- Someone else is paying the bills – you just want to code 🙂
Hillel added, “One of the things I like about serverless, we’ve gotten through some of the hype early. Many have managed to peer through a lot of the fuzz and uncertainty and find value for their organization. It’s still worth poking fun at the fact that it’s a buzzword and people want to go serverless without thinking about why. I still think there’s a lot of value for particular uses cases, and I think we’re going to see more and more.”
Future Serverless Podcasts?
Please send us your feedback and suggestions. Would you like to be a guest on the podcast and share your insights on serverless? Please email me.