Software Engineering Daily

We are booking sponsorships for Q3, find more details at https://softwareengineeringdaily.com/sponsor/

CamelCamelCamel: Amazon Price Tracker with Daniel Green

Fri, 24 May 2019 09:00:47 -0000

CamelCamelCamel is a tool for tracking prices on items on Amazon.com. The company was launched eleven years ago and is built off of the Amazon Product Advertising API. Daniel Green is one of the founders of CamelCamelCamel, and he joins the show to describe his experience building the product.

Amazon and CamelCamelCamel (C3) have a complex relationship. C3 makes its money from referral listings. When a user tracks the price of an Amazon item using C3, that user will probably eventually click on the referral listing on C3. If the user purchases the item on Amazon, C3 gets a percentage of the purchase from Amazon.

C3 has millions of product listings where they are tracking the price of items on Amazon. They have created a directory with a large subset of Amazon’s items by leveraging an API that was originally meant for advertising. Whether or not this is a proper use of the API, C3 arguably leads to more purchasing volume on Amazon–which is ostensibly why Amazon lets the company continue to operate.

In today’s show, Daniel gives a history of CamelCamelCamel, including his own background as an engineer working in finance and how he wound up working on C3 for the past decade. We also talk through how C3 is architected.

RECENT UPDATES:

We are booking sponsorships for Q3, find more details at https://softwareengineeringdaily.com/sponsor/

Gaming with Eli Brown

Thu, 23 May 2019 09:00:45 -0000

Gaming is becoming mainstream.

Popular multiplayer games such as Fortnite and Minecraft present players with a massive virtual world to explore, build, and compete within. Turn-based games such as Hearthstone and Magic are breeding a new generation of board game and card game aficionados. Social media networks like Twitch and YouTube have turned gaming into a voyeuristic sport that is outcompeting many physical sports games for attention.

Guilded is a platform for managing gaming teams. On Guilded, there are teams for games like League of Legends, Fortnite, and World of Warcraft. These teams use Guilded to manage calendars, Discord bots, forum software, documents, statistics, and recruiting.

This might sound confusing–why does a gaming team need document management, calendars, and analytics? Are we talking about a video game team or a software company? To understand Guilded, you need to understand the rapidly changing modern gaming ecosystem.

Eli Brown is a founder of Guilded.gg. He joins the show to talk about the world of gaming, its intersection with social media, and the fascinating engineering problems involved in building a platform for gaming teams.

RECENT UPDATES:

We are booking sponsorships for Q3, find more details at https://softwareengineeringdaily.com/sponsor/

Monolithic Repositories with Ciera Jaspan

Wed, 22 May 2019 09:00:44 -0000

Google’s codebase is managed in a single monolithic repository. An engineer at Google can explore almost any area of the codebase within the entire company. In order to enable this, Google has built tooling to support the monolithic repo, including a virtual file system and a set of build tools.

A monolithic repository is not to be confused with a monolithic deployment. Google’s infrastructure consists of thousands of small services interacting over a network, and scaling individually. But all of the code for each of these different independent modules is in the same version control system.

Ciera Jaspan is a staff software engineer at Google working on developer infrastructure. She worked on an internal research project within Google to find out how engineers felt about the monolithic repository system and how it compared to a large number of small repositories.

Ciera joins the show to discuss repository management, internal tooling, and Google’s approach to researching developer productivity within the company.

RECENT UPDATES:

We are booking sponsorships for Q3, find more details at https://softwareengineeringdaily.com/sponsor/

Scaling Intuit with Alex Balazs

Tue, 21 May 2019 09:00:41 -0000

Alex Balazs is the Intuit Chief Architect and has been working at the company for almost twenty years.

Intuit’s products include QuickBooks, TurboTax, and Mint. These applications are used to file taxes, manage business invoices, conduct personal accounting, and other critical aspects of a user’s financial life. Because the applications are managing money for users, there is not much room for error.

When Intuit was started, the company made desktop software. In his time at Intuit, Alex played a key role in rearchitecting the monolithic desktop applications to be resilient, reliable web applications. Intuit originally managed this software on their own servers. Since then, Intuit has migrated to the cloud using AWS.

Alex joins the show to discuss his experience scaling Intuit, his strategy for cloud migration, and his evaluation criteria for questions of build versus buy.

RECENT UPDATES:

We are booking sponsorships for Q3, find more details at https://softwareengineeringdaily.com/sponsor/

EmergingMarkets: Kenya with Nelly Cheboi

Mon, 20 May 2019 09:00:27 -0000

Africa is rapidly adopting the same software and hardware technologies that have transformed the western world over the last few decades. But access to computers and technology education is still uneven. Where there is access to computers, smartphone adoption often comes before access to laptops or desktop computers.

Nelly Cheboi is the founder of TechLit Africa, an organization that works to connect schools and families in Africa with computers and software. Nelly studied computer science, and worked as a software engineer before leaving her career to focus full-time on building a scalable model to take refurbished computers and give them to Africans who can make good use of them.

TechLit Africa is also building a software stack to equip schools in Africa without an Internet connection with an internal subnet including Wikipedia and other educational resources, so that people in the school can get an Internet-like experience despite a lack of access to the full Internet.

RECENT UPDATES:

We are booking sponsorships for Q3, find more details at https://softwareengineeringdaily.com/sponsor/

Facebook Strategy with Mike Vernal

Fri, 17 May 2019 09:00:25 -0000

Facebook’s strategy is shaped by long term goals, short term requirements, and the available resources of the company.

Long term goals are necessary for thinking through big decisions such as acquisitions, hardware product investments, and open source software ecosystems. To implement long term goals, Facebook needs to communicate the vision of the company and foster an internal culture that supports that vision.

Short term requirements can affect how the company is thinking on a more immediate time horizon.

When Facebook realized the importance of mobile computing, the mentality in the company quickly shifted from looking at mobile as a tax on engineering resources to a long-term source of business value. When Google started to work on Google+, Facebook engineers focused their resources on the potential competitive threat.

Facebook’s strategy is implemented by the engineers, product managers, and other employees of the company. Facebook is unique in its ability to allow those employees to self-assemble into work that is meaningful to the individuals as well as to the company.

As the long term goals and short term requirements of Facebook change over time, company resources are shifted to focus the company on the correct set of priorities. Some of those priorities might be speculative investments in new technologies. Other priorities might include doubling down on areas of the company that are showing promise.

Mike Vernal worked as a VP of product and engineering at Facebook for 8 years. He left the company in 2016 and joined Sequoia Capital, where he now works as a partner. In his time at Facebook, he helped architect and implement strategies relating to product direction and engineering.

Mike joins the show for a discussion about his time at Facebook and the strategic lessons that he learned from his time at the company.

RECENT UPDATES:

We are booking sponsorships for Q3, find more details at https://softwareengineeringdaily.com/sponsor/

Facebook React with Dan Abramov

Thu, 16 May 2019 09:00:24 -0000

RECENT UPDATES:

We are booking sponsorships for Q3, find more details at https://softwareengineeringdaily.com/sponsor/

Facebook Management with Jocelyn Goldfein

Wed, 15 May 2019 09:00:00 -0000

Facebook leadership was able to recognize the importance of mobile computing in time to develop high quality mobile applications, but there were numerous challenges.

The Facebook desktop web app had been difficult enough to build due to the unprecedented data requirements and amount of interactivity. Mobile introduced the additional hurdles of limited bandwidth and distinct native operating systems in Android and iPhone.

Facebook’s early efforts to build a mobile application involved a cross-platform HTML5 solution. HTML5 had insufficient performance for Facebook’s needs, and the company needed to develop native apps in order to deliver the desired experience.

Facebook’s ability to pivot to mobile is comparable to the classic story of Intel pivoting from a memory company to a microprocessor company. To succeed at mobile application development, Facebook had to shift its focus dramatically, reallocating engineering resources and acqui-hiring small mobile companies in order to build up the domain expertise for mobile.

As a side effect of this transition to mobile, Facebook developed an understanding of how dramatically software engineering was changed by the introduction of smartphones and the high bandwidth requirements of social networking. The challenges of this new paradigm led to the development of open source tools such as GraphQL and React Native, which have allowed countless projects to build applications more easily.

Jocelyn Goldfein was an engineering director at Facebook for four years, from 2010 to 2014. She currently works as an investor at Zetta Venture Partners. In her time at Facebook, Jocelyn saw the shift to mobile firsthand. In today’s episode, she describes how Facebook management works, and gives her perspective on the distinguishing characteristics of the engineering organization as a whole.

Facebook Developers with Nick Schrock

Tue, 14 May 2019 09:00:00 -0000

Facebook is a case study in the ability for developers to self-organize into groups who are working on projects that are meaningful to the company and personally satisfying to the individual engineers. Many engineers in the software industry work under a less capable manager who has complete control over their creativity. This leads to employee churn, dissatisfaction, and burnout.

Facebook’s ability to move fast is predicated on its ability to match engineers with problems that are interesting to those particular individuals. Whether you want to work on newsfeed or developer productivity tools or machine learning research, there is a path within Facebook to finding a problem that is both important and fun.

Facebook’s unique set of engineering challenges required the company to develop a unique set of internal tools. Because Facebook had data and throughput requirements which were unprecedented, the available tools and best practices at the time did not satisfy Facebook’s requirements. Over the years, Facebook has developed its own databases, caching strategies, and JavaScript frameworks.

Nick Schrock worked at Facebook for eight years. He is best known as a co-creator of GraphQL, a tool for efficiently fetching data through a federated request language. GraphQL was the result of years of evolution of internal tooling within Facebook.

Nick has discussed the creation of GraphQL in other podcasts, and we will have a more dedicated episode around a retrospective of GraphQL in the near future. Today’s episode is about the process by which developers at Facebook self-organized, and Nick’s ideas around how to identify a need for an internal tool.

Since leaving Facebook, Nick has parlayed his experience in developer tools into Dagster, a programming model for data applications.

Facebook Engineering with Pete Hunt

Mon, 13 May 2019 09:00:32 -0000

Facebook engineering is commonly described by two words: move fast.

Building products quickly has been a differentiating characteristic of the company since its inception. From the longtime engineers to the summer interns, Facebook instills a sense of immediacy and opportunity in all of its employees.

The goal of Facebook is to make the world more open and transparent, with the intention of creating greater understanding and connection through Internet services. More than any other company in history, Facebook has enabled people to communicate with each other via simple user interfaces and real, authenticated human identity.

Facebook must move fast, because the vision for Facebook is without precedent. It may feel like the Facebook mission is already finished, because you can already use Facebook to connect with anyone across the world with an Internet connection.

But once you are connected to somebody on Facebook, there are only a small number of interactions you can take: sending a message, sharing a photo, broadcasting a video stream. There are so many more parts of our lives waiting to be digitized, and many of these require a real identity system to work properly.

More than any other company, Facebook is positioned to expand our system of real-world human trust onto the Internet. The depth and breadth of the engineering problems required to accomplish this demands that Facebook move fast. To move slower would cause all of us to pay the opportunity cost of having to wait longer to interconnect our global society.

Pete Hunt worked as an engineer at Facebook for three and a half years. At Facebook, he helped build React, a set of technologies that have significantly improved frontend application interface development. After the Instagram acquisition, Pete was the first engineer from Facebook to join the Instagram team to help bring the two companies together.

Pete left Facebook in 2014 to start Smyte, a company that made trust and safety tools for marketplaces and social networks. Smyte was acquired by Twitter, where Pete now works on engineering problems relating to trust, safety, health, and infrastructure.

Pete joins the show for the first of several episodes with Facebook engineers. In these episodes, we will explore the engineering practices of Facebook–from scaling Facebook’s PHP monolith to open sourcing React and GraphQL. Other topics will include management, onboarding, and product strategy.

Our goal is to present a holistic picture of how Facebook engineering works, so that other organizations can learn to adopt practices that will allow them to move faster. We hope you enjoy this series on Facebook engineering.

RECENT UPDATES:

We are booking sponsorships for Q3, find more details at https://softwareengineeringdaily.com/sponsor/

Airtable with Howie Liu

Fri, 10 May 2019 09:00:31 -0000

Software engineering is harder than it should be. There are many people who have an app idea that they are not sure how to build. Some of these people are highly technical professionals like real estate agents, scientists, and accountants.

These professionals learn to use spreadsheets in their day-to-day work. Spreadsheets are also used widely by young people such as students. Spreadsheet users vary in terms of how familiar they are with the programmability of a spreadsheet, but there are certainly more people who have built complex spreadsheets than there are people who have built complex web apps.

Airtable is a tool for making application development easier and more accessible. The Airtable interface is similar to a spreadsheet and can be used for most spreadsheet applications. It can also serve as a rich backend database system to improve the productivity of software developers who are fully capable of building web applications.

There are high-level programmable components called Blocks and integrations with developer APIs like Twilio and Stripe.

Airtable has a permissions and collaboration system that allows interaction between engineers who might be using Airtable as a programmatic transactional database and operations members who might need to read or edit specific parts of the data on an ad hoc basis.

Howie Liu is the CEO of Airtable and he joins the show to talk about his vision for the product and the engineering problems he is working on to realize that vision. Airtable has not been trivial to build, and has required its own custom database backend and its own JavaScript rendering system.

Special thanks to Gareth Pronovost who is a full-time Airtable expert that I found on YouTube and who was generous enough to take some time to have a call with me and describe his experience using Airtable. The fact that there is a full profession around creating Airtable applications speaks to how unique this platform is.

RECENT UPDATES:

We are booking sponsorships for Q3, find more details at https://softwareengineeringdaily.com/sponsor/

FindCollabs Hackathon #1 has ended! Congrats to ARhythm, Kitspace, and Rivaly for winning 1st, 2nd, and 3rd place ($4,000, $1000, and a set of SE Daily hoodies, respectively). The most valuable feedback award and the most helpful community member award both go to Vynce Montgomery, who will receive both the SE Daily Towel and the SE Daily Old School Bucket Hat

Podsheets is our open source set of tools for managing podcasts and podcast businesses

New version of Software Daily, our app and ad-free subscription service

Virtual Data with Sunil Kamath

Thu, 09 May 2019 09:00:29 -0000

Relational data systems have evolved from single node instances to complex distributed systems.

Almost any database can be accessed through a SQL statement, but the guarantees of these databases can vary in terms of consistency, availability, latency, durability, and financial cost. Relational database systems that explore these different sets of tradeoffs are sometimes categorized as “NewSQL”.

There are also a wide variety of data systems that are not categorized as databases. Kafka is a distributed queue. HDFS is a distributed file system. Spark provides a distributed in-memory working set to process data. Cloud providers offer hosted bucket storage for your data lake and fast processing in the form of a data warehouse.

Sunil Kamath is a principal PM with Microsoft. Sunil has worked on database systems for two decades, and he joins the show to give his perspective on the current data world and his predictions for how data platforms will become easier to use. Sunil is optimistic about the use of virtual data for unifying the access of data for a variety of operational use cases.

RECENT UPDATES:

We are booking sponsorships for Q3, find more details at https://softwareengineeringdaily.com/sponsor/

Podsheets is our open source set of tools for managing podcasts and podcast businesses

New version of Software Daily, our app and ad-free subscription service

Web Assembly Runtime with Tyler McMullen

Wed, 08 May 2019 09:00:05 -0000

WebAssembly is a binary instruction format for applications to run in a memory-constrained, stack-based virtual machine. The WebAssembly ecosystem consists of tools and projects that allow programs in a variety of languages to compile into WebAssembly and run in a safe, fast, sandboxed runtime environment.

WebAssembly is a transformative technology for the Internet. Most users will experience it as a set of gradual, incremental improvements to their online experiences. Pages will load faster and become more dynamic. Applications will become more secure. Infrastructure will become cheaper, and those cost savings will eventually reach the consumer.

For developers, WebAssembly opens a world of possibility. In today’s operating systems, the user can feel a big difference between applications that need a large client-side runtime (such as video editing tools, or render-heavy games such as Half Life) and applications that are more lightweight and can run entirely on the web (such as Twitter).

Tyler McMullen is the CTO at Fastly. He joins the show to talk about the compilation path, the runtime, and the opportunities of WebAssembly.

RECENT UPDATES:

We are booking sponsorships for Q3, find more details at https://softwareengineeringdaily.com/sponsor/

Podsheets is our open source set of tools for managing podcasts and podcast businesses

New version of Software Daily, our app and ad-free subscription service

Kubernetes Virtualization with Paul Czarkowski

Tue, 07 May 2019 09:00:04 -0000

Modern server infrastructure usually runs in a virtualized environment. Virtual servers can exist inside of a container or inside of a virtual machine. Containers can also run on virtual machines. Kubernetes has allowed developers to manage their multiple containers, whether those containers are running in VMs or on bare metal (servers without VMs).

As organizations expand their Kubernetes deployments, the overhead of those deployments is becoming a relevant concern. So-called “Kubesprawl” can occur within organizations due to a lack of best practices on when new clusters should be spun up or spun down, and when clusters should be shared by teams or shared by services.

Paul Czarkowski is a principal technologist with Pivotal. He joins the show to discuss virtualization, Kubernetes, and the state of the cloud native ecosystem.

RECENT UPDATES:

We are booking sponsorships for Q3, find more details at https://softwareengineeringdaily.com/sponsor/

Podsheets is our open source set of tools for managing podcasts and podcast businesses

New version of Software Daily, our app and ad-free subscription service

Cloud Database Workloads with Jon Daniel

Mon, 06 May 2019 09:00:29 -0000

Relational databases such as Postgres are often used for critical workloads, such as user account data. To run a relational database service in the cloud requires a cloud provider to set up a highly durable, highly available system.

Jon Daniel is an infrastructure engineer at Heroku. Jon joins the show to describe the engineering and operations required to build a managed relational database service. Full disclosure: Heroku is a sponsor of Software Engineering Daily.

RECENT UPDATES:

We are booking sponsorships for Q3, find more details at https://softwareengineeringdaily.com/sponsor/

Podsheets is our open source set of tools for managing podcasts and podcast businesses

New version of Software Daily, our app and ad-free subscription service

Satellite Data Platform with Tim Kelton

Fri, 03 May 2019 09:00:49 -0000

Satellite images contain vast quantities of data. By analyzing the contents of satellite images over time, we can identify trends in weather, soil, and agriculture. If we combine that data with ground-level sensors, we can gather a clearer understanding of how chemicals in the air or in the dirt map to how things look from above via satellite.

Descartes Labs is a company that gathers high dimensional data about our planet and turns it into machine learning models to be used by customers. In order to do this, the company has built out a data pipeline involving queueing systems, machine learning frameworks, and internal tools that are used to aggregate, clean, model, and measure data.

Tim Kelton is a co-founder of Descartes Labs and he joins the show to discuss the high volume of data and the distributed systems that make up the Descartes Labs infrastructure.

RECENT UPDATES:

We are booking sponsorships for Q3, find more details at https://softwareengineeringdaily.com/sponsor/

Podsheets is our open source set of tools for managing podcasts and podcast businesses

New version of Software Daily, our app and ad-free subscription service

Security Monitoring with Jeff Williams

Thu, 02 May 2019 09:00:01 -0000

The modern software supply chain contains many different points of distribution: JavaScript frameworks, npm modules, Docker containers, open source repositories, cloud providers, on-prem firmware, IoT, networking proxies, and so much more.

With so much attack surface, securing a large enterprise is an uphill battle. Jeff Williams is the CTO at Contrast Security, a company that makes infrastructure monitoring tools. Contrast Security works by intercepting network traffic at a low level and assessing whether that traffic maps to a common threat model.

Jeff joins the show to talk about different approaches to monitoring and securing large infrastructure deployments.

Contrast Community Edition

RECENT UPDATES:

We are booking sponsorships for Q3, find more details at https://softwareengineeringdaily.com/sponsor/

Podsheets is our open source set of tools for managing podcasts and podcast businesses

New version of Software Daily, our app and ad-free subscription service

Software Growth with Greg Kogan

Wed, 01 May 2019 09:00:19 -0000

Growing a software business requires an understanding of engineering, sales, and marketing.

As we learn software engineering, we also pick up some knowledge about how a business should operate. We know that there are customers, and that our product needs to be scalable to serve more customers. We know that some features are more important than others, and so we focus on building the features that matter the most.

But unless we make a deliberate focus, engineers do not learn how to sell and market a software product.

Learning how to sell and market software is an important skill to develop. It allows a software engineer to be self-sufficient. If you already know how to write software, sales and marketing are actually the only other pieces you need to be an “entrepreneur”. And the basics of sales and marketing are often easier and more fun to learn than the first painful days of learning basic programming.

Greg Kogan is an engineer who has shifted his focus to working as a consultant for companies that are trying to go to market with a technical product. Greg has helped grow companies such as Netlify, Scalyr, and Domino Data Lab. Much of his work is around products targeted toward developers.

Greg joins the show to describe his methodical approach to selling and marketing software.

RECENT UPDATES:

We are booking sponsorships for Q3, find more details at https://softwareengineeringdaily.com/sponsor/

Podsheets is our open source set of tools for managing podcasts and podcast businesses

New version of Software Daily, our app and ad-free subscription service

Container Platform Security with Maya Kaczorowski

Tue, 30 Apr 2019 09:00:01 -0000

A Kubernetes instance occupies a wide footprint of multiple servers, creating an appealing target to an attacker, due to its access to a large pool of compute resources. A common attack against an exposed Kubernetes cluster is to take it over for the purposes of mining cryptocurrency. Thus it is important to keep a cluster secure.

The importance of security is magnified for a cloud provider. A cloud provider runs a managed Kubernetes service, which might be running thousands of Kubernetes clusters. If the cloud provider’s chosen distribution of Kubernetes contains a vulnerability, or if the Kubernetes instances are misconfigured, all of these clusters could be exposed to the same vulnerability.

Maya Kaczorowski works on the security of Google’s managed Kubernetes service GKE. In today’s show we discuss the attack surface of a managed Kubernetes service. Maya was previously on the show to talk about container security. This episode is a good companion to that one, as well as a previous show with Liz Rice about container security.

RECENT UPDATES:

We are booking sponsorships for Q3, find more details at https://softwareengineeringdaily.com/sponsor/

Podsheets is our open source set of tools for managing podcasts and podcast businesses

New version of Software Daily, our app and ad-free subscription service

Lyft’s Data Platform with Li Gao

Mon, 29 Apr 2019 09:00:33 -0000

We are booking sponsorships for Q3, find more details at https://softwareengineeringdaily.com/sponsor/

Podsheets is our open source set of tools for managing podcasts and podcast businesses

New version of Software Daily, our app and ad-free subscription service

Lyft generates petabytes of data. Driver and rider behavior, pricing information, the movement of cars through space; all of this data is received by Lyft’s backend services, buffered into Kafka queues, and processed by various stream processing systems.

Lyft moves the high volumes of data into a data lake for different users throughout the company to use offline. Machine learning jobs, batch jobs, streaming jobs and materialized databases can be created on top of that data lake. Druid and Superset are used for operational analytics and dashboarding.

Li Gao is a data engineer at Lyft. He joins the show to explore the different aspects of Lyft’s data platform. We also talk about the tradeoffs of streaming frameworks, and how to manage machine learning infrastructure. This episode is a great companion to our show about Uber’s data platform, and illustrates some fundamental differences in how the two ridesharing companies operate.

Cloud with Eric Brewer

Fri, 26 Apr 2019 09:00:00 -0000

To the extent that I am a software engineering journalist, I feel inclined to scrutinize all of the cloud providers. But to the extent that I am an engineer and a business person, I feel only admiration and love for the cloud providers. Cloud computing has brought the cost of starting an Internet business down to zero.

Cloud computing has opened up my eyes to a world of creative possibilities that knows no boundaries, and for that I will always be a fan of all of the rivaling cloud companies because they all have played a role in creating the current software landscape.

Eric Brewer is a Google Fellow and VP Infrastructure. He is well-known for his work on the CAP theorem, a distributed systems concept that formalized the tradeoffs between consistency, availability, and partition tolerance in a distributed system.

At Google, Eric is as much a strategist and product creator as he is a theoretician. He has worked on database systems such as Spanner, machine learning systems such as TensorFlow, and container orchestration systems such as Kubernetes and GKE.

Eric joins the show to talk about Google’s philosophy as a cloud provider, and how his understanding of distributed systems has evolved since joining the company.

Intricately: Mapping the Internet with Fima Leshinsky

Thu, 25 Apr 2019 09:00:52 -0000

RECENT UPDATES:

We are booking sponsorships for Q3, find more details at https://softwareengineeringdaily.com/sponsor/

Podsheets is our open source set of tools for managing podcasts and podcast businesses

New version of Software Daily, our app and ad-free subscription service

Intricately is a company that maps the breadth and depth of cloud infrastructure usage. Using a combination of clever algorithms, data engineering, and web crawlers, Intricately derives information about how different companies spend money on infrastructure.

Fima Leshinsky is the CEO and co-founder at Intricately. In his previous job at Akamai, he began to study how a cloud provider such as Akamai could figure out how much its competitors were charging certain customers. Since CDN infrastructure is a commodity with reasonably low switching cost, a provider that can undercut its competitors significantly can have an edge in the marketplace.

From his work at Akamai, Fima felt there was a market opportunity to provide this kind of service to the broader market of cloud providers. There are more cloud providers than ever before, and the kind of data that Intricately aggregates is highly useful to this competitive marketplace.

Fima joins the show to talk about the modern landscape of cloud providers, and how to build a system that maps the Internet.

gVisor: Secure Container Sandbox with Yoshi Tamura

Wed, 24 Apr 2019 09:00:51 -0000

RECENT UPDATES:

Podsheets is our open source set of tools for managing podcasts and podcast businesses

New version of Software Daily, our app and ad-free subscription service

Podsheets is our open source set of tools for managing podcasts and podcast businesses

The Linux operating system includes user space and kernel space. In user space, the user can create and interact with a variety of applications directly. In kernel space, the Linux kernel provides a stable environment in which device drivers interact with hardware and manage low level resources.

A Linux container is a virtualized environment that runs within user space. To perform an operation, a process in a container in user space makes a syscall (system call) into kernel space. This allows the container to have access to resources like memory and disk.

Kernel space must be kept secure to ensure operating system integrity–but Linux includes hundreds of syscalls. Each syscall represents an interface between the user space and kernel space. Security vulnerabilities can emerge from this wide attack surface of different syscalls, and most applications only need a small number of syscalls to perform their required functionality.

gVisor is a project to restrict the number of syscalls that the kernel and user space need to communicate. gVisor is a runtime layer between the user space container and the kernel space. gVisor reduces the number of syscalls that can be made into kernel space.

The security properties of gVisor make it an exciting project today–but it is the portability features of gVisor that hint at a huge future opportunity. By inserting an interpreter interface between containers and the Linux kernel, gVisor presents the container world with the opportunity to run on operating systems other than Linux.

There are many reasons why it might be appealing to run containers on an operating system other than Linux.

Linux was built many years ago, before the explosion of small devices, smart phones, IoT hubs, voice assistants and smart cars. To be more speculative, Google is working on a secretive new operating system called Fuscia. gVisor could be a layer that allows workloads to be ported from Linux servers to Fuscia servers.

Yoshi Tamura is a product manager at Google with a background in containers and virtualization. He joins the show to talk about gVisor and the different kinds of virtualization.

Observability Engineering with James Burns

Tue, 23 Apr 2019 09:00:31 -0000

RECENT UPDATES:

New version of Software Daily, our app and ad-free subscription service

Podsheets is our open source set of tools for managing podcasts and podcast businesses

Twilio is a communications infrastructure company with thousands of internal services and thousands of request per second. Each request generates logs, metrics, and distributed traces which can be used to troubleshoot failures and improve latency.

Since Twilio is used for 2-factor authentication and text message relaying, Twilio is critical infrastructure for most applications that implement it. The service must remain highly available even in times of peak application traffic, or outages at a particular cloud provider.

When he was at Twilio, James Burns worked on platform infrastructure and observability. James was at Twilio from 2014 to 2017, a time in which the company experienced rapid scalability. His work encompassed site reliability, monitoring, cost management and incident response. He also led chaos engineering exercises called “game days”, in which the company deliberately caused infrastructure to fail in order to ensure the reliability of failover systems and to discover problematic dependencies.

James joins the show to talk about his time at Twilio and his perspectives on how to instrument and observe complex applications. Full disclosure: James now works at LightStep, which is a sponsor of Software Engineering Daily.

Serverless Runtimes with Steren Giannini

Mon, 22 Apr 2019 09:00:25 -0000

RECENT UPDATES:

New version of Software Daily, our app and ad-free subscription service