For this episode of The Serverless Show, Tal was in the office with Hillel in Jerusalem, and they were joined by Slobodan Stojanovic, CTO of Cloud Horizon, and an official AWS Serverless Community Hero, via Zoom from Belgrade, Serbia.
Slobodan stated, “First, thank you for having me here because it’s a lovely podcast and I love serverless podcasts. We need more of them and this is definitely one of the best. I work at Cloud Horizon as CTO and a partner. We’re based in Montreal and Belgrade. We do services for other startups and larger companies, mostly American and Canadian companies, and we have a development team in Belgrade. We do a lot of things, some of which are serverless now. We are trying to move more things to serverless, but it depends on our customers.
“We also now offer a product called Vacation Tracker. When Cloud Horizon was really small, we had some issues with tracking vacations for our team. It’s hard to notify everyone that I’ll be on vacation next week and things like that, so we built a small Slack chatbot that to help. Now it’s a bit bigger. It’s not a chatbot anymore, but more like a web application, and it’s totally serverless.
“I’m also involved with some open source projects. I started with Node community and helping out with some node libraries a long time ago. Now I’m mostly doing serverless-related things. I joined the Claudia.js team a long time ago, almost at the beginning, and helped Gojko Adzic and Alexander Simovich to build Claudia.js. Claudia was and still is a deployment library for AWS Lambda and API gateway. At the beginning, it was really hard to deploy serverless applications. If you tried to do that manually, you need to zip everything, to set the permissions, and things like that. The idea of Claudia was to extend AWS CLI tools and to help users deploy serverless applications easier. We continued doing Claudia and a few other things. We contributed a bit to AWS SAM and we built some other applications that are open source. We’re trying to build tools that we need and that the serverless community needs.”
Hillel stated, “The first topic we want to talk about is a bit close to home. It’s a project we were very involved in setting up called DVSA under OWASP. DVSA stands for Damn Vulnerable Serverless Application. The high-level goals for us were, a lot of customers were coming to us saying, ‘Hey, how do I understand what the challenges are in serverless, in terms of security? What problems am I trying to solve? What are the solutions that I should be looking for? What’s the mindset? What’s changed? What’s the same?’
“We could talk about that all day long, but it’s really a lot easier sometimes to see this and play with it and touch it. The idea of this project was to have an open-source project that we could get involved with to help really put out a framework that people can see what are vulnerable serverless applications, how should security tools be best used to make them safe?”
Tal stated, “I think the most important thing for anyone that deals with serverless and wants to know about security is to understand what they are dealing with. Because they can learn from the Internet or from even AWS is very secure aware, but unless you actually see that and you can play with that, then you don’t see the implications of the security vulnerabilities or security issues that you have in your serverless applications. What we decided to do is to create an educational tool, this damn vulnerable serverless application. It’s an open-source tool that will allow both white hat hackers, so if a company has red or blue teams that evaluate security of the product, they can use the tool to understand better about how to interact with the serverless application, how you can exploit certain things. On the other side, developers can play with the tool and also see the code, so they can actually do the hack part first, but then go back to the code, fix it, and then see if they managed to make the application more secure. From my 10 years of experience in training developers for secure coding, this is the best part for a developer to really understand the security mindset, and then when you go to serverless, it’s new, so they don’t know it yet. This is a very good way to figure it out.
Tal continued, “The fact that we donated to OWASP is that we wanted it to grow, and we believe it should grow, so OWASP is a good community. It’s the best community for application security. We believe that donating to OWASP is important for this tool to grow. We already can see that. I’m leading the project, and I can see very high involvement. At this point, we have a lot of will to be involved in this tool and people are coming to me every day to join the tool and to support it, to make it better. I think that it can eventually get everyone that’s involved in serverless to understand security better, which is the goal.”
Hillel added, “The code is on GitHub, there’s also some lessons you can use to walk through on GitHub, so that’s also great. We’ll be deploying it under the serverless application repository as well, so if you want to just one click, get it installed in the accounts, you can start playing with it. It will be very easy to do that.”
Tal said, “Yeah, so you can write either on your account, or if you don’t have an account, if you’re a freelancer who wants to learn, you can just browse to serverless.fail which has the online version of it, and you can just play with it.”
Hillel added, “If you’re a security engineer, if you’re one of our competitors, if you’re one of our partners, if you’re just somebody who is interested, get involved. Take a look at the project. There are lots of things that can be added. There’s lots of work still to be done to make this tool really robust and make sure it’s the educational tool that can let everybody very quickly understand what they need to be doing and a way to evaluate security products. Definitely get involved. I think OWASP was a great umbrella for this, and so we reached out.”
Responding to an inquiry from Hillel, Slobodan stated, “Yes, I think open source is really important for serverless, but before we continue discussing that, I want to say that your tool is really awesome and security in serverless apps is really, really important. It’s important for enterprise users, of course, but it’s also important for users that are just starting with serverless to understand what are the new risks that they will face in serverless functions and serverless apps that they didn’t have in some other applications. Sometimes those risks are the same, but they can be used in a different way, and it’s really important to remind people that security is really important because with serverless everything will scale and infrastructure is managed, but your code and security is definitely not fully managed, so we definitely need to be more careful about that. Your tool is really awesome, and people should take a look at it and start playing with it even before they start really building something for production.”
(And no, dear reader, we did not prompt him to say that!)
Slobodan continued, “I think open source is really important, not just for serverless, basically for everything in technology, but for serverless especially because we talk about serverless a lot in the past few years, but serverless is still really new, the concept. Not everyone is using it and we saw more people on serverless conferences and more people on re:Invent than other big conferences talking about it, but I think most of the people just started using serverless. The platforms are still not really complete. Everything is evolving slowly, and it’s really important for us to show vendors what do we need and how can we build our applications better?
“In the early days, it was really, really hard to deploy serverless applications and serverless framework, a new way to deploy and organize your serverless application. We helped with Claudia and there were a few other really good frameworks at that moment or libraries because Claudia is not really a framework.
“So, at some point, AWS showed the need of reliable deployment tools, so they built AWS serverless application model and it was kind of influenced by existing frameworks at that moment. But they went a step further and they open sourced their tool, so people from the community can just help them to build the deployment tool. That’s exactly what we tried to do from Claudia.js because we do node.js deployments really well. We tried to contribute a bit and help them to build the SAM build command for Node.js. We sent a few pull requests and you can use some parts of the code that we had in Claudia.js now in AWS SAM. Yes, open source is really important and I think it will enable us to build applications in a new way because even vendors don’t know fully what we want to do, and serverless is new for everyone, so we definitely need to help a lot pushing that forward.”
Hillel agreed. “I think the fact that the vendors, themselves, are approaching us with more open source is a really good step forward. It really lets them move faster, lets us move faster. It lets them capture the ability of the community to lead in a lot of ways and not just follow.”
Next, the group discussed an article titled “Serverless is Awesome if You Overlook the Inflated Costs, Dislike Distributed Computing, and Love Vendor Lock-In.” Hillel stated, “It was research that I believe was done at the University of California – Berkeley, and it was really trying to look at if you try to build a true complex application, does serverless work all that well? The research contended it doesn’t work all that well. I think some of the challenges they highlighted were doing truly distributed computing that does require some amount of shared state is really challenging. It can be very slow with serverless. They talk a lot about network throughput and Lambda functions and how Amazon designs that. Those are things, I think, that are changing actually over the past couple months and the coming months, but some of the throughput for network-bound applications can be challenging, as well as lack of availability for GPUs and things like that.
“The upshot of the article, at least from the researcher’s side, was yeah, this is a real hype and trend and a buzzword, but when you try to build real applications, for a lot of common-use cases, it doesn’t work all that well. On the other hand, there are a lot of big companies and small startups and everything in the middle jumping on the serverless bandwagon and figuring out how to use this well for them and really benefiting a lot. I’m curious, in your view, which of these points are valid? How are people mitigating some of them? If these things are really challenging, why is it people are still embracing serverless and rewriting or creating new applications based on this technology?”
Slobodan replied, “In my opinion, I think people can really build a real-world application, production-ready application in serverless because you’re doing that. So far, we don’t have a big issue. Of course, we have some issues, but we had issues with any technology that we worked with, so that’s normal. But serverless enables us to build our product much faster and much cheaper because it’s not just cheaper like our infrastructure is really cheap for now. Maybe it’s different when we scale, but it’s not just the infrastructure. People can build and release some things much, much faster, so the total cost of everything is cheaper because we can release our features and things like that faster.”
Slobodan continued, “Of course, if you’re using a lot of tools that exist right now in the ecosystem, including some security tools, some monitoring tools, and many other things. There’s a great article that went out a few weeks ago, by Mark Schwartz. He’s now Enterprise Strategies at AWS, but he was working as CIO at U.S. Citizenship and Immigration Services. He’s talking about differences between vendor lock-in and switching costs, because he thinks that vendor lock-in term is misleading because it’s not really vendor lock-in. It’s more switching costs, because even if you go with the different programming language and then want to switch, for example, from Java to Node.js, that will have some cost assigned to it. Of course, before you implement something in your production-ready application, you need to analyze that tool really well to see what your switching cost will be and how likely it is for you to need to switch to some other tool in the next few years.
“When you put all those things on paper, you can decide if it makes sense for you to build an application with serverless or anything else. For us, it really makes sense to build things with serverless because I don’t think vendors will just lock us in, and we’ll discuss that later, but you can trigger Lambda functions even outside of AWS. If you use all the services from AWS or from Microsoft or from other vendors, you have the full potential of the platform, but you can build services with mixed things. I think a serverless framework has Event Gateway now, and they allow you to have some kind of external gateway that allows you to trigger functions by different vendors and things like that. But I don’t think that things like vendor lock-in are too big an issue right now.”
Slobodan continued, “I had a presentation three years ago with a few topics that are problems with serverless application. Right now I can cross off all those things, because vendors fixed that in just two years. So everything is changing very fast and serverless has some downsides, but I think upsides are much bigger, at least for me.”
Hillel stated, “I think, in the areas where serverless struggles, we’re also just seeing the vendors make up that gap very quickly. It could be that today you’re doing something where you say, ‘Hey, this isn’t ideal without a GPU,’ but you know the vendors are going to make GPUs available in serverless functions at some point in the near future. I think the same is true for a lot of things. Network bandwidth is the bottleneck and I’m not sure that’s really the bottleneck for most applications.”
Tal stated, “It’s not. If you look at Netflix, you can say that they probably have so much bandwidth that they require for outbound traffic, and they use serverless.”
Hillel said, “I think overall those things are going to work themselves out. The third thing I wanted to talk about is the TriggerMesh announcement. That was interesting because I think it ties into some of the things you’re talking about here, as well, Slobodan. I’m going to simplify it and there’s probably somebody from TriggerMesh who’s throwing up now as I speak, but it essentially allows you to run Lambda functions on top of Knative, which allows you to run Lambda functions on Google Cloud for now.
“On the one hand, this made a lot of buzz and maybe it’s exciting that I can write my Lambda functions and then someone is able to take those functions and allow them to run with their API somehow on top of Knative and do some of the mapping and the glue there. On the other hand, I’m wondering, how valuable is this, really? Part of me feels like this is just kind of a gimmick. The problem has never really been writing functions that can run on multiple platforms. That’s not really something that people need all that often, nor is it all that challenging. It’s really much more like you spoke about. The rest of the services that you’re integrating with. Maybe if you could channel Mark Schwartz, how would Mark Schwartz react to this announcement?”
Slobodan laughed and said, “I don’t think he would react to this. Maybe this is really important for someone, but for me, it’s not a big announcement because, for me, Lambda functions are just like a glue between other services that we are using. Our application uses a lot of Lambda functions, but most of them are just really small and they’re just converting data and passing to some other services. The idea of serverless, for me at least, is to focus on our business logic and outsource everything that we can. Building everything inside functions is definitely not that.
“You can trigger a Lambda function from wherever you want if you use AWS API. I don’t think this will create a big value for me, but maybe for someone else it will, but for me, it would be much better if we can use other services from other vendors. I don’t see a big value in multi-cloud, anyway, so I don’t think that I’m the best person to talk about upsides of these kind of announcements.”
Hillel said, “I wonder if the excitement or maybe even the announcement itself is seen positively in the sense that it’s a bridge from what people have done today, which is mostly write Lambda functions and write their workflows in Lambda, to allow them to kind of embrace a more open Knative-based environment. Someone was just saying to you, ‘Hey, if you think Knative is the goal, if you think Knative is the right way to do serverless on a platform that’s more flexible, here’s a way that you can get there without having to rewrite your code.’ Although I agree with you. I don’t think that’s the big challenge for most people. I don’t think that’s the big shift. I mean if you’ve written thousands and thousands of lines of code in each Lambda function, you’re probably not doing serverless right anyway, right?”
Tal added, “But maybe it would help in the transition to serverless.”
Slobodan stated, “Yes, definitely. You said that Knative is a goal, but I think if you think that Knative or serverless or any other technology is a goal, you’re doing that wrong because, for us, the goal is not to use some technology. For us, it’s to build some kind of application that people will use and to solve some real-world problems. Technology is just enabling us to do that, so whatever technology enables you to do that, well, I’m okay with that. For me, serverless is helping me to build some things faster and better. If for someone Knative is a better tool, I’m okay with that, too.” Hillel replied, “Yeah, we do tend to fall in love with our stacks rather than with our solutions. That’s a good point. Tal, when I think about all this kind of the multi-cloud question and the Knative versus more proprietary question and all of the ways that people are embracing serverless, bring us back to security there. What are the key pieces in terms of security that we talk about the benefits and the challenges? How are they influenced by some of these changes?”
Tal replied, “Yeah, I think that maybe the major downsides for security in serverless would be that it’s new. It’s new and that means that many people or a lot of practitioners don’t know what serverless security means, so they can’t deal with it. But this could be solved over time as more open-source projects that get out there will help people focus and know better.
“But there are many advantages. I think the main one for a startup, for example, would be that the serverless architecture comes with built-in security for the infrastructure. Only that allows them to start building an application that might not be the most secure, might have some security flaws in the application, but they don’t need to deal with operation systems or infrastructure security, which involves a lot of time and a lot of operations. This is a very good advantage of serverless, but it also has some other opportunities like microservices-type of architecture where everything is fine-grained and you can, especially on AWS, specify a specific permission or specific row for each of your functions and remove that from DevSecOps, when they deal with security and permissions in the infrastructure level.
“You can actually make your function do the only thing that it needs to be doing and reduce the attack surface of your application. With just a little effort, maybe some automation tools you can get, comparing serverless application with that amount of effort to a monolithic application or a traditional application, there’s a huge gap in security towards serverless.”
Hillel quipped, “I was worried how you were going to end up there. I agree.”
Slobodan commented, “We see that vendors are moving really, really fast, for example, on the AWS re:Invent this year, they announced a lot of services. It’s really hard to just read a list of the announcements, and other vendors are trying to follow that. Do you think that releasing all those things really fast affects security and how? Maybe when they release one service, they can open some new security holes into existing applications and things like that. Actually, do you think that has any effect on security of serverless applications?”
Hillel replied, “I would say there’s probably two levels of the problem. First of all, I agree it’s an issue. I think the one level is the vendors, themselves, and I think Amazon does a pretty good job of trying to make sure the things that they’re rolling out are pretty well thought through and secured. But obviously, any velocity increase is going to increase your risk of security. I think the bigger challenge is that Amazon has enabled a world where things change very quickly for people who don’t necessarily know how to use them.
“We address, very often, the issue of permissions. Part of the challenge with permissions is, I mean there are 280 new API calls in the past three or four months on the AWS API for new services and new functionality. Those are 280 more services you need to know how to provide the correct permissions for. What’s the right IAM role of those permissions? It’s not always well documented in Amazon because of the speed at which they move, so those challenges can really increase, whereas when you go to platforms that don’t move as quickly, and some of the other vendors don’t necessarily move as quickly as Amazon. One of the benefits is the documentation tends to be a bit more robust. There aren’t as many decisions to make. Things don’t change as quickly, so you can be an expert at some level, I think. That’s definitely a challenge.”
Hillel continued, “Now if you asked somebody, ‘Would you give up on that rapid velocity of new features?’ Obviously, the answer is no, and I think, in general, one thing serverless has taught me about security is security needs to be aligned with the business, and aligned with the business is figuring out how to embrace that velocity. For example, as a security company, we struggle with how can we, as the people trying to configure IAM roles as part of our solution, know what the correct permission is for each new service that gets out there? How do we keep ahead of that? That’s obviously a challenge.
“I think it definitely creates more challenges. I don’t think anybody wants to trade, necessarily, and go back to a world where things roll out once every three months. But yeah, you know what it’s like, Slobodan. You go to re:Invent and your head spins from all the new functionality. Security is one of those heads that spins.”
Hillel continued, “Let’s get to my favorite segment, where we outline our favorite tweets. mine. Mine is super biased. Tal is going to love me now. I picked Tal’s tweet.
— Tal Melamed (@_nu11p0inter) January 11, 2019
I think Tal’s point that people are running ahead with embracing serverless and what they can do with it and getting more and more interested is great. We’d like to see a similar climb of people being interested in learning about how to secure serverless and what serverless security means. Hopefully, we’ll see that red line climb up.”
Tal replied, “It will start. Like every other technology, it starts in the delay; too big of a delay.”
Slobodan commented on his favorite tweet, “It was really hard to pick just one tweet because there were a lot of really great tweets, but I picked the one from Stackery.
“[Chances are] anyone willing to challenge the status quo and embrace #serverless is open-minded and forward-thinking.” – @FarrahC32. Read her interview with @forrestbrazeal on her journey to Stackery and the opportunity for inclusion in this space: https://t.co/qFYAY3bPnw pic.twitter.com/0g9Cga9IFf
— Stackery.io (@stackeryio) January 10, 2019
“It’s about an interview with Forrest Brazeal about inclusivity and open-minded and forward-thinking community in serverless. I love serverless because it enables people to build the well-working products even if they don’t have two years of experience. I think this interview is really awesome because she’s talking about a lot of great things in this community. I think we still need to do many things to make this community even more open-minded and inclusive for everyone, but I think we have a good start. I definitely recommend that article.”
Tal stated, “My favorite tweet is complex. I don’t know if you know, but the DVSA has a Twitter account. But not only has a Twitter account, the credentials to the account are actually implemented into the insecure application. So that was part of the fun of it, that people can eventually, if they manage to, they can steal these tokens or API keys and tweet in the name of the DVSA application.
“At the beginning of DVSA, when we published it, there was a tweet that I sent from the DVSA that was very happy with a lot of people coming into the application and starting to attack it. It was quite a success and it was retweeted 400 times and even more likes. But eventually, probably someone managed to get the keys and deleted this tweet. It was sad, so I wrote another tweet that, “I’m sorry that my tweet was deleted because someone hacked it, hopeful to get these retweets again,” but I didn’t. It was just a cry for help.”
For #Hacks sake! I accidentally hacked myself and deleted my famous tweet (600 Likes / 380 RTs)
That what happens where you ask people to hack you….#PLEASE HELP me get another one 🙁
— DVSA (@DVSAowasp) January 11, 2019
Hillel replied, “I didn’t realize this story was going to have a sad ending. Now I’m sad. I think you win at the self-referential tweet award for this week.”
(A note from the editor: I’m embedding a tweet, but there’s no telling how long it will live! So here’s a link directly to the DVSA Twitter profile page.)
Slobodan said, “I wrote a book with my friend Alexander Simovic about starting with serverless application on Node.js., called Serverless Applications with Node.js. It’s published by Manning Publications and the promo code we’re sharing for a 40% discount is “claudia40”. If anyone wants to start with serverless with Node.js, I think we tried to write the book in a way that is welcoming for beginners, so it should be a big and friendly book.
Hillel stated, “It’s a great resource for anyone starting out. It’s the kind of resource that a developer wrote for developers. It’s not one of these things you’re going to go, ‘Now what do I do?’ It’s a really good place to start, so highly recommend it.”
Slobodan replied, “Yeah, and in the next version, we’ll probably try to add more things about security because it was really hard to fit everything into 300 pages, but we tried to keep everything that people need to start with. I hope that we’ll see a lot of books that talk about security soon.”