Evolution of SoundCloud: Final Part

This is the final part of the SoundCloud series. If you haven’t read the previous parts, I encourage you to do so. You can read Part 1 here and Part 2 here.

Hi, I’m Stephen!

I spend my weekends researching, learning, and creating content for this newsletter.

It would mean the world to me, if you took a few seconds of your time to check out Bright Data.

At least see why they’re the world’s #1 web data platform. They even have their own dog walkers.

Go ahead and check them out. Now let’s get on with the show.

Web Intelligence, Unlocked

With Bright Data's cutting-edge proxy solutions, harness the full potential of web data for your business. Tap into our global proxy network to scale your data collection activities. Ecommerce platforms, travel agencies, financial institutions, and market researchers are all leveraging web data to gain a competitive edge.

Bright Data offers the scalability and flexibility necessary for gathering and analyzing web data. Take the first step towards data-driven excellence.

Previously, we learned why SoundCloud pivoted towards a microservices architecture.

The catalyst for this decision started with the engineering team’s inefficient development process.

In order to deliver more features, the team as a whole needed to be more productive.

As a result, a collaborative approach was implemented across the engineering, design, and product teams.

All working together in unison to deliver new features.

SoundCloud’s progression towards a microservice architecture proved to be challenging.

Even though new services were stood up, they still had to work in conjunction with the Mothership.

This is the evolution of SoundCloud’s architecture over the last 10 years.

Backends for Frontends

The combination of autonomous teams, new microservices, and an internal API led to a new concept: Backends for Frontends (BFFs).

Phil Calçado first introduced this architectural pattern at SoundCloud in 2013.

The pattern focuses on creating multiple, dedicated API gateways for each client platform (web, mobile). BFFs are also responsible for:

• Rate limiting

• Authentication

• Header sanitization

• Cache control

One advantage is autonomy. Separate APIs allow each specific client type to be optimized.

Another is resilience. If a BFF goes down in an availability zone, the rest of the platform would still be available.

Finally, sometimes ugly workarounds and mitigations are necessary. It’s much easier to implement these in the BFF versus a more complex application like the Mothership.

Today, SoundCloud handles dozens of BFFs, each one powering a dedicated API (Mobile, Web, Public, Partner).

These BFFs handle hundreds of millions of requests per hour.

With Great Power Comes Great Responsibility

Like any solution, there’s always tradeoffs.

For example, multiple BFFs can oftentimes lead to duplicate code. Even worse, inconsistent implementations.

Authorization rules that can only be applied at integration time can also be problematic. Imagine if private or unreleased tracks were leaked.

Two areas needing improvement were the Track and Playlist entities (more on these later).

As they grew, these entities were decomposed into multiple microservices.

How does one decide when to create a new BFF or when to reuse an existing one?

And how many BFFs is too many?

Both valid questions. Clearly, there’s a tradeoff between autonomy and operational overhead.

BFFs require extensive collaboration between frontend and backend engineers.

Value-Added Services

With the amount of business logic and authorization code living in the BFFs, they became difficult to maintain.

Keeping code synchronized and consistent was a big challenge.

To address these issues, the team needed to make some architecture improvements.

Enter Value-Added Services (VAS).

VAS are business services solely responsible for returning entities and associated value objects (aggregate).

Consolidating the business logic into its own service keeps the BFFs clean. In doing so, centralized point is created where metadata and authorization rules can be defined.

This is known as the separation of concerns principle.

The new architecture is divided into three core service layers:

1. Edge: BFFs live in this layer and maintain dedicated APIs. Each API is tailored to a specific client.

2. Value Added: Consumes and processes data from other services (VAS or Foundation) for user experience.

3. Foundation: Serves as the building blocks around different domains and contains low-level services.

Each VAS also consists of the following building blocks:

1. Domain: Represents a user or business concern. Used to draw boundaries around service integrations.

2. Entity: Represents an object with an independent identifier and lifecycle.

3. Value Objects: Contains an entity’s metadata. Also tied to the lifecycle of the entity.

4. Aggregate: Contains one or more relate entities.

Tracks and Playlists

In 2019, the team began applying the VAS pattern, starting with the Tracks VAS.

A track is an entity with associated value objects like metadata, transcodings, and authorization policies to determine who can see that track.

Since each track has an owning user (user is its own entity), the track object holds a user id as a reference.

Simultaneously, if another consuming service has a track id that needs to be resolved, that service can call the Tracks VAS.

The Tracks VAS will then determine whether the request is authorized and return the track aggregate accordingly.

This effort was successful and in 2020, the team started a major refactoring of the Public API.

Up first were the track-related endpoints.

With the new Tracks VAS, all track-related endpoints just had to be re-routed to the new service (Public API was calling the Mothership directly).

But the playlist-related endpoints were far more complex. Business logic and authorization rules were spread across multiple BFFs and services.

The logical solution was to stand up a new Playlists VAS to centralize all playlist-related logic.

By doing so, cross-platform feature development, refactoring and optimizations would be easier and faster.

This also solved the authorization inconsistencies across multiple services and reduced the risk of potential vulnerabilities.

SoundCloud’s Architecture Today

As SoundCloud’s platform grew, so did the scope of each VAS.

For example, SoundCloud provides music tracks to consumer applications.

But the platform also provides tools for creators to upload and distribute their music.

As both use cases grew, so did the complexity of the Tracks VAS. The engineering team determined that the VAS pattern needed to evolve.

A more scalable approach was needed.

What was originally one domain has now grown into two similar but different business domains.

As a result, using the same implementation for both was no longer the best solution.

It was time to create different implementations specific for each use case (same line of thinking as BFFs).

This led to the adoption of Domain Gateways and a Domain-Oriented Microservice Architecture (DOMA), an architecture also used by Uber.

Each business domain would have its own implementation, but still rely on the same foundational layer of services.

Overall, this façade provides both stability and an anti-corruption layer for each of the domains.

This pattern is still being implemented at SoundCloud today.

If you made it this far, thank you for reading! I hope you enjoyed it.

If I made a mistake, please let me know.

P.S. If you’re enjoying the content of this newsletter, please share it with your network and subscribe: https://www.fullstackexpress.io/subscribe

Resources

[1] “Evolution of SoundCloud’s Architecture,” developers.soundcloud.com.
https://developers.soundcloud.com/blog/evolution-of-soundclouds-architecture.

[2] “Building Products at SoundCloud,” developers.soundcloud.com.
https://developers.soundcloud.com/blog/building-products-at-soundcloud-part-1-dealing-with-the-monolith.

[3] “Service Architecture at SoundCloud,” developers.soundcloud.com.
https://developers.soundcloud.com/blog/service-architecture-1.

[4] “How SoundCloud Uses HAProxy with Kubernetes for User-Facing Traffic,” developers.soundcloud.com.
https://developers.soundcloud.com/blog/how-soundcloud-uses-haproxy-with-kubernetes-for-user-facing-traffic.

[5] “The End of the Public API Strangler,” developers.soundcloud.com.
https://developers.soundcloud.com/blog/end-of-the-strangler.

[6] “How We Develop New Features Using Offsites and Clean Architecture,” developers.soundcloud.com.
https://developers.soundcloud.com/blog/how-we-develop-new-features-using-offsites-and-clean-architecture.

[7] “How We Ended Up With Microservices,” philcalcado.com.
https://philcalcado.com/2015/09/08/how_we_ended_up_with_microservices.html.

[8] “BFF @ SoundCloud,” thoughtworks.com.
https://www.thoughtworks.com/insights/blog/bff-soundcloud.

[9] “Pattern: Backends For Frontends,” samnewman.io.
https://samnewman.io/patterns/architectural/bff/.

[10] “Pattern: Backends For Frontends,” developers.soundcloud.com.
https://developers.soundcloud.com/blog/announcing-twinagle.

[11] “Microservices and the Monolith,” developers.soundcloud.com.
https://developers.soundcloud.com/blog/microservices-and-the-monolith.