- Full Stack Express
- Posts
- Netflix's Smooth Launch of Basic with Ads
Netflix's Smooth Launch of Basic with Ads
Stephen Sun
October 04, 2023
🌙 Hello world ☀️
Two years ago today, Meta (formerly Facebook) had its infamous worldwide outage that lasted 7 hours and 11 minutes, affecting every one of its platforms. As we reflect on that day, it serves as a reminder of the importance of continuous learning and improvement. After all, it's our dedication to the craft, that truly puts us on the map.
In this week’s email:
Testing: Netflix simulates expected traffic to identify scalability issues and validate their Ads algorithms.
Security: PayPal implements graph database to combat fraud.
TypeScript: The saga of the Closure Compiler, and why TypeScript won.
Node.js: Connect, a universal Socket API for JavaScript runtimes.
Career Advice: How to influence and think like a Staff Engineer.
The best error message is the one that never shows up.
In November 2022, Netflix introduced a new tier called "Basic with ads". This tier added new backend components and a remote call to their ads partner during playback.
To ensure a smooth launch, Netflix simulated the expected traffic weeks in advance, using a traffic migration framework. This simulation helped identify scalability issues and validate their Ads algorithms.
Realistic Test Traffic
Netflix's traffic generally follows a predictable pattern, with occasional spikes due to new content releases or regional AWS data center shifts. To test their new ad system, Netflix decided to replay real traffic, making it appear as if it was "Basic with ads" traffic. This approach allowed for realistic testing of the new systems and algorithms.
Netflix’s Traffic Patterns
The Setup
Netflix aimed to ensure that the simulation did not impact actual customers. They used real member viewing habits for the simulation, but no ads were shown to these members.
The data science team projected the subscriber count for the new tier a month post-launch. This data was used to simulate a subscriber population. Traffic matching this simulated population was stored in a Mantis stream.
This stream was then replayed in a separate production environment designed for this purpose. The responses from this environment, which included ad metadata, were stored in a Keystone stream. This data was then used to simulate device playback and ad impression tracking.
Netflix’s Traffic Replay Setup
The Rollout
Netflix started by replaying a small percentage of traffic, gradually increasing it while monitoring metrics. They reached 100% replay and ran it continuously.
They also tested their system's response to traffic spikes caused by regional AWS evacuations. During this period, they set up and tested their ad monitoring and alerting system. They also conducted chaos experiments to test their systems under failure scenarios, ensuring resilience.
Key Takeaways
Simulating real traffic provides confidence in new systems and algorithms.
Large-scale testing with representative traffic can reveal unexpected issues.
Replay traffic can be used for more than just load testing, offering potential for new product and feature development at Netflix.
P.S. If you’re enjoying the content of this newsletter, please share it with your network: https://www.fullstackexpress.io/subscribe
Account takeover (ATO) poses a significant fraud risk in online payments. To address this, PayPal has implemented a real-time graph database, which uses edges and vertices to represent relationships between objects.
Asset Sharing
Assets such as IP addresses, physical addresses, and device IDs are connected to PayPal accounts. By representing these connections in a graph structure, it becomes evident how assets are shared among different accounts.
Typically, regular accounts maintain consistent asset usage over time. However, accounts that have been taken over by fraudsters often exhibit frequent and abrupt changes in their associated assets.
By analyzing the Asset-Account Graph, PayPal can detect unusual asset-sharing patterns and identify potentially fraudulent accounts.
Asset to Account Graph Highlighting Abnormal Linking Behaviors
Transaction Patterns
Users often engage in multiple transactions in a short time frame. While traditional relational databases struggle to efficiently represent complex transaction patterns, graph databases excel in this area.
For instance, a pattern named "ABABA" indicates two accounts sending money back and forth in quick succession.
Such patterns, which are prevalent in ATO incidents, can be swiftly detected using graph databases, thereby preventing potential fraudulent activities.
Example of Transaction Patterns
Graph Features
Graph features involve calculating specific metrics based on the graph's topology. One notable feature is the "connected community", which describes interconnected subgraphs within the primary graph.
These features are invaluable for identifying groups of closely connected accounts and their transaction behaviors, which can be instrumental in detecting anomalies.
By examining these graph features, PayPal can identify key nodes in the graph, offering insights that can help prevent large-scale fraudulent activities.
Account-Transaction Graph
Key Takeaways
Graph databases are essential tools in combating ATO fraud, offering a more efficient way to represent and analyze complex relationships.
Asset sharing patterns in the graph can highlight abnormal behaviors, signaling potential account takeovers.
Transaction patterns, like the "ABABA" pattern, can be more easily detected and analyzed using graph databases compared to traditional relational databases.
Graph features, such as the "connected community", provide a deeper understanding of transaction behaviors and can be crucial in anomaly detection.
DATA STRUCTURES & ALGORITHMS
Valid Parenthesis
Missed the solutions to this week’s coding workout?
Learn how to use the stack data structure to solve this problem here.
The article discusses the saga of Google's Closure Compiler and how TypeScript eventually became more dominant in the JavaScript ecosystem. The Closure Compiler focused on optimal JavaScript minification, but faced challenges adapting to the modern JavaScript landscape. On the other hand, TypeScript prioritized developer experience, type safety, and adaptability, leading to its widespread adoption among software engineers. Dive into this chapter of JavaScript history.
The useEffectEvent is an experimental React hook that streamlines the process of adding and removing event listeners in React components. This hook offers developers a cleaner and more efficient way to handle events in functional components. See it in action.
Node.js is a widely-used backend technology that's increasingly targeted by online attacks. This guide highlights 15 best practices, including not running Node.js with root privileges, regularly updating NPM libraries, and adopting an HTTPS-only policy. Implementing these measures ensures robust security for your Node.js applications in production. Integrate these techniques and tips.
This article features insights from Ron DeVera, a seasoned Staff Engineer and former Twitter Tech Lead, on influence, managing relationships, and differences between staff and senior engineers. Key takeaways include understanding situations from others' perspectives, avoiding pushing suggestions too forcefully, using the Situation-Behavior-Impact framework for feedback, and recognizing the workload of managers when discussing topics. Learn how to think like a staff engineer.
JAVASCRIPT ECOSYSTEM
JS Weekly Pulse
📢 NPM now supports publishing packages with provenance from GitHub Actions and GitLab CI/CD, linking them verifiably to their public source repositories.
📢 A new socket API that works across JavaScript runtimes is in the works.
📢 Updates from the 98th TC39 meeting include Resizable ArrayBuffers, ShadowReam, and RegExp escaping.
📢 Next.js Conf 2023 is set for October 26 at 10am Pacific.
🚀 Astro 3.2 - Enhanced view transitions and more seamless integrations.
🚀 Node.js 20.8.0 - Performance improvements to streams.
🚀 Babel 7.23.0 - Decorator metadata and new import features.
🚀 ESLint 9.0.0 - Breaking changes for rule authors.
🚀 pnpm 8.8.0 - New options for pnpm commands, improvements to error handling and node-linker.
RECOMMENDATIONS
To-Do List
✅ Explore: GitHub’s Innovation Graph compares metrics by economy as well as specific categories such as Git pushes, repositories, and programming languages.
✅ Learn: Understand the significance of securing container images, and how key tools like Sigstore Cosign, Notary v2, and Docker Content Trust are integrated.
✅ Watch: Discover key principles for scaling distributed systems, drawing from Amazon and eBay’s architecture, and explore the importance of simplicity, communication methods, and consistent feedback.
✅ Listen: Andrew Hoying from Google discusses the evolution of system hardening, challenges at Google's scale, the balance between security and productivity, adapting to regulations, and how lessons learned have influenced product improvements.
✅ Try it out: ElectricSQL offers a reactive real-time sync layer for Postgres, enabling instant local-first data synchronization seamlessly within your existing Postgres-driven apps.
✅ Register: P99 CONF is a premier tech conference featuring talks from engineers at leading companies like Netflix, Google, and TikTok, covering high-performance applications, distributed systems, and many more.
COMMUNITY SPOTLIGHT
Hot Picks in the Dev Community
N64js: An n64 emulator written in (mostly) pure ES6 JavaScript. It runs many ROMs at full frame rate.
Dropzone: An easy to use drag & drop JavaScript library. It supports image previews and shows nice progress bars.
Chatgpt.js: A powerful, open source client-side JavaScript library for ChatGPT.
Xterm.js: A terminal for the web.
LangChain.js: A framework for building apps with language models. Written in TypeScript.
MEME
First Day on the Job
Reply