Gusto's 81x Performance Improvement in Payments Packaging

🌙 Hello world ☀️ 

If you’re lucky enough to be in London today, don’t miss out on Google Cloud Next London. This is a free, in-person event featuring the latest innovations in AI, Data, and Security with keynotes from Google, Google Cloud and Google DeepMind executives.

In this week’s email:

• Architecture: Learn how Gusto revamped their payment system, achieving an 81x speed improvement.

• Infrastructure: Learn how Slack rebuilt their cron execution system for scalability and efficiency.

• JavaScript: Speeding up the JavaScript ecosystem, the barrel file debacle.

• React: Building accessible and typesafe modern web forms.

• Next.js/Remix: An in-depth case study comparing Next.js 13 and Remix.

• Developer Tools: Why successful tools must align with developers' practical needs rather than idealistic aspirations.

ARCHITECTURE
Gusto’s 81x Performance Improvement in Payments Packaging

Peter Harris, Simul Kadakia

In 2022, Gusto's Payments Engineering team faced an impending challenge: a predicted spike in payment volumes by year's end threatened to overwhelm their existing payment packaging system.

How It Started
The original process, which bundled money movements into ACH files for bank transfers, had a few challenges:

1. Ran in 5-minute cycles, gathering a maximum of 5,000 entries each time.

2. This single-threaded system presented multiple inefficiencies, such as often producing ACH files with minimal entries and mishandling cancellations.

Revamping For Efficiency
Addressing these challenges required a shift in architecture. Gusto adopted a map-reduce framework, enabling parallel processing of ACH entries using tools like Sidekiq and Redis.

This architectural change significantly leaned on tools like Sidekiq and Redis to manage parallel tasks without compromising data integrity.

The team also introduced the ACH Engine, a solution which:

• Enabled packaging closer to submission time.

• Optimized file sizes.

• Cut out redundant operations.

Rethinking Redlock and Smarter Cancellations
The Redlock system, initially implemented to prevent race conditions, became a notable bottleneck. However, by refining their interaction with Redis, the team markedly boosted efficiency.

Furthermore, Gusto implemented a better cancellation solution that prioritized such requests, eliminating potential process conflicts. Through targeted optimizations, including eliminating redundant checks and refining datastore queries, Gusto significantly amplified their processing capability.

Results
After nearly a year of effort, Gusto's Payment Engineering team achieved remarkable improvements in performance:

• Entry Count

  • Before: 149,418

  • After: 692,920

• Processing Time

  • Before: 55 minutes

  • After: 3 minutes 9 seconds

• Throughput

  • Before: 2,716 entries per minute

  • After: 219,974 entries per minute

• Overall Improvement:

  • 81x faster

P.S. If you’re enjoying the content of this newsletter, please share it with your network: https://www.fullstackexpress.io/subscribe

INFRASTRUCTURE
Rebuilding Slack’s Cron Execution System for Scalability and Efficiency

Claire Adams

Cron scripts are pivotal for Slack's operations such as reminders, email notifications, and database cleanups. As Slack grew, the number of scripts and the data they process expanded, leading to reliability concerns and scaling challenges.

This led to the need for a new, scalable, and more reliable system for executing cron scripts.

Original System and Shortcomings
In its infancy, Slack's cron infrastructure was fairly straightforward. It comprised a single node that housed all the scripts and one crontab file that spelled out the schedules for each script.

While this served its purpose initially, as the data and scripts grew, merely migrating to larger nodes became an untenable solution. The main problem lay in the fact that the entire system hinged on this one node, making any issues—be it provisioning, rotation, or configuration—a potential system-wide disruption.

New Cron Execution System
The new system consists of three core components:

1. Scheduled Job Conductor

   

   Scheduled Job Conductor

   • What: A Golang service for mimicking cron functionality, leveraging a Golang cron library.

   • Why: To preserve the original cron string format for easier migration.

   • How it Operates:

     • Utilizes Bedrock (Slack’s Kubernetes wrapper) for scalability.

     • One pod does the scheduling, while others wait to take over if needed to avoid downtime.

2. Slack's Job Queue

   

   Slack’s Job Queue

   • What: An asynchronous compute platform running around 9 billion jobs daily.

   • Composition: Jobs flow through Kafka, then to Redis, and finally to a job worker for execution.

   • Advantage: Offloads compute and memory needs to an existing robust system, reducing build and maintenance time.

3. Vitess Database Table

   

   Vitess Table

   • Purpose: Deduplication, job tracking, and reporting.

   • Functionality:

     • Ensures only one copy of a script runs at once.

     • Tracks the job state in the system.

     • Supports a user-interface for script execution information, aiding in error detection and state verification.

By utilizing existing solutions, the effort for both development and future maintenance is reduced. This approach ensures longevity while optimizing resource allocation.

DATA STRUCTURES & ALGORITHMS
Linked List Cycle

Missed the solutions to this week’s coding workout?

Learn the algorithm used to solve this problem here.

JAVASCRIPT
Speeding Up the JavaScript Ecosystem - The Barrel File Debacle

Marvin Hagemeister

Numerous projects are cluttered with files that simply re-export other files. These "barrel files" significantly contribute to the sluggish performance of JavaScript tools in larger projects. Marvin analyzes the effects of barrel files on performance.

REACT
Accessible, Typesafe, Progressively Enhanced Modern Web Forms

Kent C. Dodds

Building web forms is a complex task often requiring developers to override browser defaults. Modern frameworks like Remix simplify form handling by emulating default browser behaviors while adding enhancements. Tools like Zod and Conform aid in creating typesafe and accessible forms with schema-based validation and seamless integration between forms and underlying data structures.

NEXT.JS, REMIX
Next.js 13 vs Remix: An In-depth Case Study

Prateek Surana

Next.js and Remix are both popular frameworks for building React web applications. Prateek Surana provides a comparative analysis of their new routing and data-fetching techniques, using a Twitter clone project as an example. While Remix uses loaders for efficient parallel data fetching, Next.js 13 leverages React Server Components for more direct and streamlined data integration within React components.

DEVELOPER TOOLS
Why Aren’t There More Programming Languages Startups?

Jean Yang

Jean Yang, former professor and founder of Akita Software, examines why cutting-edge programming languages and tools from academia don't often translate to mainstream adoption in the tech industry. She argues that developers prioritize tools that enhance their workflows and address immediate needs rather than those offering perfect guarantees. For "deep tech" tools to gain traction, they must align more with developers' current workflows, be interoperable, and focus on incremental improvements that mesh with existing technologies.

JAVASCRIPT ECOSYSTEM
JS Weekly Pulse

• 📢 Security efforts to implement control-flow integrity (CFI) in V8 by preventing memory-corrupted exploits from executing arbitrary code.

• 📢 Japa offers a streamlined testing framework for Node.js without the transpilation overhead associated with Jest.

• 📢 NodeConf EU is set for November 6-8, 2023 in Ireland.

• 📢 GitHub Actions has restricted the use of GITHUB_ENV for setting the NODE_OPTIONS variable in workflows.

• 🚀 Bun 1.0.4 - Virtual modules support, server.requestIP, memory consumption reduction in Bun.serve(), and bug fixes.

• 🚀 TypeScript 5.3 Beta - Import attributes, enhanced narrowing, and various optimizations.

• 🚀 Redux Toolkit 1.9.7 - Enhancements to RTKQ hook TypeScript types for notably better performance.

• 🚀 pnpm 8.9.0 - Performance improvements for macOS and Windows Dev Drives

RECOMMENDATIONS
To-Do List

✅ Interesting: Firefox fixes a 22 year old bug.

✅ Join: GitHub is launching a certification program in 2024 to validate developers' skills and expertise, with an early exam opportunity at GitHub Universe on November 8-9. If you can’t make it, join the waitlist.

✅ Learn: Pinterest's rapid growth, highlighting the platform's ability to accommodate 11 million users while being supported by a lean team of just six engineers.

✅ Watch: Replay ViteConf 2023, including 44 talks about the Vite ecosystem.

✅ Listen: OWL is the world’s fastest JavaScript framework, inspired by Vue and React, designed specifically for Odoo's unique modular requirements and built with modern features like asynchronous rendering and a virtual DOM.

COMMUNITY SPOTLIGHT
Hot Picks in the Dev Community

Crank.js - A web framework that allows components to be defined using sync, async, and generator functions.

Gitify - GitHub notifications on your menu bar.

Kysely - Type-safe and autocompletion-friendly typescript SQL query builder. Inspired by knex.

Npkill - Easily find and remove old and heavy node_modules folders.

Wretch - Small wrapper around fetch that simplifies network requests and responses.

Docx - Easily generate and modify .docx files with JS/TS. Works for Node and on the Browser.

MEME
Friday Deployments Be Like…