engblogs

Automatic stubbing of network requests to improve flakiness in automation testing

An open-source, scalable PageRank solution for SoundCloud search implemented in Spark, tracing the journey from in-memory Java and Hadoop MapReduce to GraphX and an RDD-based, iterative DiscoRank approach.

DiscoRank scales PageRank on Spark for SoundCloud's massive catalog, recounting the shift from in-memory Java to Hadoop MapReduce and Scalding, then a Spark-based iterative approach with RDDs to handle non-uniform edges, delivering 3-5 minute iterations on 700M nodes and 15B edges, and releasing as an open-source reusable library.

Applying PageRank algorithm using Spark to navigate and analyze vast music collection in SoundCloud.

A concise, one-line overview of SoundCloud's DeveloperBridge Trainee Program, detailing how junior engineers gain hands-on experience across frontend, backend, data processing, and machine learning-focused work through diverse team projects and mentorship, with outcomes including full-time roles.

An overview of SoundCloud's DeveloperBridge trainee program, a 12-month, paid junior-engineer initiative in Berlin that exposed participants to frontend, backend, and data processing work across multiple teams, with the aim of transitioning them to full-time roles and improving engineering diversity.

Achieving massive scalability with Kubernetes deployment

Insights from SoundCloud's DeveloperBridge Trainee Program focuses on the collaboration between SoundCloud's People Team and engineering management to bring in junior engineers.

Working on the OCaml compiler and toolchain at Jane Street to improve language features, type system, performance, profiling, debugging, documentation, and build automation.

SoundCloud tackles dynamic user traffic routing by replacing a brittle edge Ingress setup with a NodePort-based HAProxy deployment in Kubernetes, using rendezvous hashing to select a fixed set of nodes and a node agent for graceful maintenance.

Case study of SoundCloud routing user-facing traffic by migrating from a homegrown deployment platform to Kubernetes with an HAProxy edge, using rendezvous hashing and NodePort backends to keep traffic balanced as pods move.

Using HAProxy with Kubernetes for managing user-facing traffic in SoundCloud's deployment platform