For this episode, Hillel and Tal from Protego were joined by Mike Atkins, a distributed systems engineer at LaunchDarkly. Mike explained, “LaunchDarkly makes it easy to put flags anywhere in your application. Normally, people think of flags as something they use to manage their features, so they can deliver new features to customers in a progressive manner. Release something to a subset of users and then be able to see how they respond to it and continue the rollout of that feature based on that feedback.
“LaunchDarkly can also be used for more ops kind of use cases. When you change a flag in our application, the update to your app is pretty much real time, so you can also use LaunchDarkly to turn things on and off. For example, if something is causing a problem in your system, you can turn it off and it will take effect in your system immediately.”
Mike is the creator of Edgar Facts: a software application made for the Amazon Echo to keep you informed about his dog, Edgar. “I did that when it first came out, in about 2014, when serverless was very nascent. You had to develop the app on AWS Lambda, and that was the thing that first piqued my interest in serverless. It really changed my thinking about how you design software in this serverless environment, and the power of serverless computing that led me to design the system at the startup I was working at then using serverless, and it was pretty effective.”
The group first discussed several recent articles addressing questions, as serverless begins to mature in the market, where does it make sense? Are we under- or over-hyping it? Hillel stated, “There was an article by our friend Tom McLaughlin, who talks a lot about startups, when he’s seeing startups use serverless and when they should. I think the point he was making was if you’re really strapped for funding, then serverless is going to be something you can get started on a very small budget. But if you have a little bit more money, moving right to serverless might not necessarily be the best investment for you. I’m not sure I completely agree. For small companies, does it make sense to embrace serverless or is the pain of serverless going to be not worth it at first?”
Mike responded, “I’ve actually been in exactly that scenario where I was at a very small company and we made the decision to use serverless. I think it worked very well for us, because when you have very little usage, serverless is incredibly cheap, and there’s very little you have to manage from an ops perspective. If we were running our own EC2 instances or something like this, you would need to have people that were monitoring those servers to make sure they don’t crash, and if they do, they get restarted in a timely manner. But by using serverless technologies, we were able to sidestep all those problems. We just have to worry about our function error rate and lower it. If the error rate is high, that’s either a problem with our software or maybe we haven’t allocated enough resources to each function. But that’s a pretty easy thing to fix, and it allows you to move very quickly.”
Hillel replied, “I tend to agree. I’ve had a conversation with Tom, and I think part of it was that he’s seeing a lot of startups that aren’t using serverless, and I think he was questioning, ‘Why is that?’ I think his point was if you’re very familiar with a different technology, like using containers and VMs to get what you want done, then maybe investing in figuring out how to do the thing right with serverless isn’t the right investment for you at first. Maybe do that later. I actually have seen quite a few startups similar to what you just described where serverless really helps them move faster and alleviates other pains like operations.”
From a security perspective, is a startup better off beginning with serverless?
Tal stated, “First, for larger organizations that could spare the money, I think it might be a little difficult to just go full serverless. But I see that on a few big enterprises and I think it’s a good approach to start small and then just go. If everything works well, move ahead and take more parts to serverless. For startups, regardless of security, it’s great if you can start writing serverless code from the beginning.
“Regarding security, it could also be a good solution for startups. First, because the security for the infrastructure, such as firewalls and antivirus, also costs a lot of money. Most startups don’t have security in the first years because they don’t have money to invest in security. With serverless, you must still take care of your security of your solution, but at least part of it is covered by the cloud providers.
Hillel added, “If I’m a startup and I can’t spend too much time on security, handing over a lot of the platform to AWS, Microsoft, or Google is inherently going to make me more secure because it’s more things which I wouldn’t have secured that they will secure for me. And maybe I can take my small budget for security and focus on the things I really need to care about.”
On the flipside of that, there was an article from Cory O’Daniel about his move away from serverless. He had taken a serverless service that he wrote API Gateway Lambda and moved it over to Elixir and running non-serverlessly, and that was very beneficial. I think a key piece there was cost, but it wasn’t the only piece. I think the interesting point, which I hear a lot, is that serverless isn’t necessarily cheaper. I don’t always view that as the right metric for why people should be using serverless technologies.”
Mike stated, “I think one thing to keep in mind about that article is that the author makes a point that he was working in a company that had a very good ops team already established. They had this advantage to not use serverless because they had a team that would do a great job maintaining his Elixir service. Many other organizations that have invested in serverless may not have that same ops capabilities. So, it’s not just a simple, ‘How big is the AWS check that I’m writing each month?’ Spinning up an ops team for less than 12 grand that would be able to handle this kind of load seems difficult. For this particular use at this company where they have a good ops team, I think his decision makes total sense. But at a different company that has a different spread of capabilities, the same decision might not make sense.”
Hillel replied, “If you can manage the sprawl and the complexity of serverless architectures, they can be much faster to update. So, you can have much higher feature velocity in your application where you can roll out new bug fixtures and new features more quickly. I think to a lot of organizations that’s a big value, even if they’re paying more somehow for it. Then, obviously, there’s a bunch of security improvements with serverless that are also values people should focus on.”
The group discussed this article by Matt Asay on the same topic of where to use serverless. Hillel commented, “Even as far back as a year and a half and two years ago, there were a lot of big, established companies talking about how they were using serverless, and I thought that was really interesting. I think that’s continuing. I also got a kick out of that whole ‘leapfrogging containers.’ Is that something, Mike, you think is going to happen a lot in big enterprises, not seeing any need for containers and VMs?”
Mike replied, “Whenever an enterprise makes some kind of transition like this, it’s an extreme investment in what they think technology is going to do in the future. I think maybe this is more of an indication of things that these enterprises are willing to bet on in the longer run, because if you don’t do containers and you just go straight to serverless, it seems like you’re basically betting the future of your company more on serverless.”
Hillel responded, “It’s an interesting gamble. Compared to containers, there are more people in large enterprises at least making noise about serverless. It’s still early days and enterprises are trying to figure out how to build, operate, and monitor large product servers.”
Our friend Tom McLaughlin wrote an overview on security and serverless from a DevOps perspective. Hillel found it to be a good high-level article to help people figure out the most critical areas to focus their efforts.
Tal stated, “It’s a good article, which covers the basic and most discussed topics of serverless security like shared responsibilities. For example, you’re not responsible for the operating system, so hacks like Spectre and Meltdown won’t happen. But on the other hand, you don’t control the infrastructure. If AWS, Azure, or Google want to patch or update it for a security reason or not, it might affect your application. In terms of security, it’s very important that your environment be monitored. You should use either the tools given by the providers or get your own tools to get an insight of what happens in your environment, and then fine-grain your security.
“Another thing that he mentioned is all the application security is probably the newest part here because things change. Attacks are not coming from the API call, but could come from different ways like queues or emails, etc. He summarizes with the fact that if you take monolithic operations teams and put them on serverless, they would probably be minimized with the delivery process with how fast things are going. This is probably going to happen to security professionals, also, in the organization. If you put monolithic security-oriented professionals on a serverless environment, they might get lost because you need to deal with so many functions and so many resources and the continuous delivery is much faster. My opinion in one word: you need automation for that.”
Hillel added that a lot of the paradigms and tools have to shift and adapt to this new world. “We know that, as with lots of other areas, it never happens as fast as it needs to happen, but it does eventually get there. And certainly, people shifting to serverless should be focused on what they can not do anymore, which is great, and how they can repurpose people doing that to things that they should be doing and what tools are out there to help them.
Mike added, “If you think about how software systems are built nowadays, they have many, many moving parts to them. In a serverless application, you could have hundreds of functions. If you realize that some aspect of your system is currently under attack by somebody and you want to shut it down, sometimes it’s quite difficult because you must identify all the functions that are affected and then shut them down. If you were using feature flags to turn on and off pieces of your application, say your event ingestion system is under attack and it’s composed of hundreds of functions and you have a feature flag that decides whether or not the event ingestion system is used. You could turn off the entire system with this one flag. It would allow you to react to some kind of attack.
“Similarly, you can use flags to target individual users of your system. You could even do it by session. If you realize that people that are connecting with some session attribute are doing something malicious, you can shut them out of your application, or out of part of the app with feature flags. Your application will see a flag update within 200 milliseconds.
“By carefully using feature flagging, I could actually make sure I can turn things on and off at the right granularity for me, as opposed to having to go chase little functions and do things to it. You’re able to turn off pieces of it without intimate knowledge of how the system is built.”
Congratulations to our friends at Serverless, Inc on their Series A funding. Hillel pointed out that companies like Serverless, Inc. are at the forefront of how you get your serverless software running, and operate and manage it, and they’re really important to the ecosystem. That’s a great vote of confidence for them, obviously, but also for the whole space.
X: “We’re concerned about our infrastructure spend, and think we want to migrate to Serverless.”
Me: “What’s your AWS bill?”
X: “About $500 a month.”
Me: “With the developer time it will cost, you’ll hit break-even in about 150 years. Your move.”
— Corey Quinn (@QuinnyPig) August 22, 2018
Mike commented, “His point is that if you’re only spending $500 a month on AWS, serverless probably isn’t a reasonable investment, because the amount of time it’ll take you to transition your system to serverless, as far as developer costs, will never match your operations savings. While that’s true, if somebody’s running a software company that is only spending $500 a month on AWS and they aren’t planning to grow at all, that’s probably not a good business anyway. I think if you were to reconsider this from a perspective of how much the business is planning to grow, then it’s a much more interesting conversation.
“If, two years from now, you’d be spending $50,000 a month, the investment in serverless probably does make sense, because as you scale a system up, you’re going to run into a lot of unexpected behaviors. I think you’re able to control those unexpected behaviors through growth in a serverless system a little bit better than you are in a more conventional system.”
— Dan Frommer (@df1) August 14, 2018
Tal’s favorite tweet is from Dan Frommer mentioning a talk that they’re going to give at OWASP Israel in September. Hillel is also speaking at the event. Tal said, “I’m happy to see many companies understand that this is where the technology is going, and you can see more talks about serverless and serverless security. Also, kudos to those guys from function 01. We’ll see them, too.”
One tweet from one person who built a Lambda function and API Gateway (without context) that cost them lots of money and suddenly “serverless” is terrible.
Ignores the years of success stories & great content explaining how to do it properly.
When will tech learn? #echoChamber
— Paul Johnston (@PaulDJohnston) August 17, 2018
Hillel concluded, “My favorite tweet came from Paul Johnston this week, who’s a friend of mine and I don’t deny it, but I liked his tweet. It was highlighting that sometimes we in security get accused of promoting fear and FUD. But this is the notion that one person has one bad experience with something like serverless because they get some sort of runaway function costing them thousands of dollars a month because they misconfigured something. And then suddenly serverless is terrible and dangerous for everyone. We need to learn how to take some of the noise that we’re getting on the fringes of things and take it into perspective when we hear about thousands of other teams using the technology successfully. I thought it was a good lesson. It does highlight that sometimes it’s more fun to hear about the horror stories, but there is quite a bit of good going on.”