Experimentation has become standard practice for companies, and one of the most important aspects is how to evaluate the results to make ship/no-ship decisions. Have you run into experiments where you don’t have enough data for statistically significant results or perhaps the performance of your primary metric seemingly disagrees with that of your secondary metrics? If so, leveraging existing features to perform variance reduction may help with coming to a conclusion. At Yelp, we have found that using features typically used in ML modeling, in particular, can help with measuring treatment effects better than solely using t-tests! Introduction Before deciding...

Continue reading

Our community of users will always come first, which is why Yelp takes significant measures to protect sensitive user information. In this spirit, the Database Reliability Engineering team implemented a data sanitization process long ago to prevent any sensitive information from leaving the production environment. The data sanitization process still enables developers to test new features and asynchronous jobs against a complete, real time dataset without complicated data imports. MySQL and other open source project innovations over the last decade have led us on a journey to Vitess, which is now responsible for over 1500 workflows across more than 100...

Continue reading

Yelp’s mission is to connect people with great local businesses. On the Recommendations & Discovery team, we sift through billions of users-business interactions to learn user preferences. Our solutions power several products across Yelp such as personalized push notifications, email engagement campaigns, the home feed, Collections and more. Here we discuss the generalized user to business recommendation model which is crucial to a lot of these applications. High level overview of our recommendation system. Our previous approach for user to business recommendation was based on Spark’s Alternating Least Squares (ALS) algorithm which factorized the user-business interaction matrix to user-vectors and...

Continue reading

In a previous post we detailed the architecture and motivation for developing our new PaaSTA-based deployment model. We’d now like to share our strategy for seamlessly migrating our existing Kafka clusters from EC2 to our Kubernetes-based internal compute platform. To help facilitate the migration, we built tooling which interfaced with various components of our cluster architecture to ensure that the process was automated and did not impair clients’ ability to read or write Kafka records. Migrating Kafka on EC2 to Kafka on PaaSTA Background In the status quo implementation, EC2-backed Kafka brokers within a cluster were associated with an auto...

Continue reading

At Yelp, we use Server Side Rendering (SSR) to improve the performance of our React-based frontend pages. After a string of production incidents in early 2021, we realized our existing SSR system was failing to scale as we migrated more pages from Python-based templates to React. Throughout the rest of the year, we worked to re-architect our SSR system in a way that increased stability, reduced costs, and improved observability for feature teams. Background What Is SSR? Server Side Rendering is a technique used to improve the performance of JavaScript templating systems (such as React). Rather than waiting for the...

Continue reading

Updating the ads experience for Yelp Advertisers by creating a new Native Ads Dashboard using Server-Driven UI. What is the Ads Dashboard? The Yelp Ads Dashboard is a tool that advertisers can use to update their ad settings and keep track of how their ad is performing. In 2020, we revamped the Ads Dashboard web experience to provide greater visibility into an ad’s performance and better access to control and customize options from a single page. Ads Dashboard on Desktop Recreating the Experience on Mobile In order to ensure consistency across platforms from both a visual and feature standpoint, we...

Continue reading

Yelp’s Kafka infrastructure ingests tens of billions of messages each day to facilitate data driven decisions and power business-critical pipelines and services. We have recently made some improvements to our Kafka deployment architecture by running some of our clusters on PaaSTA, Yelp’s own Platform as a Service. Our Kubernetes (k8s) based deployment leverages a custom Kubernetes operator for Kafka, as well as Cruise Control for lifecycle management. Kafka on PaaSTA on Kubernetes Architectural Motivations and Improvements In the past, all of our Kafka clusters ran on dedicated EC2 instances on AWS. Kafka was deployed directly on these hosts and configuration...

Continue reading

At Yelp we are always striving to optimize our user experience so we can help guide our customers to success. We aim to streamline the onboarding process for business owners by centralizing customer products into a single page. The Challenge Yelp offers an array of free and paid products that help local businesses connect with consumers. To set up these products on their Yelp page, business owners previously had to navigate through multiple tabs, which negatively impacted product setup rates (roughly 55%) and lowered overall user engagement. Previously, the only way businesses could set up Yelp advertising products was through...

Continue reading