Summary Here at Yelp, we generate business wins and a better platform by running A/B tests to measure the revenue impact of different user and business experience interventions. Accurately estimating key revenue indicators, such as the probability a customer retains at least \(n\)-days (\(n\)-day retention) or the expected dollar amount a customer spends over their first \(n\) days (\(n\)-day spend) is core to this experimentation process. Historically at Yelp, \(n\)-day customer or user retention was typically estimated as the proportion of customers/users we observed for more than \(n\) days who retained more than \(n\) days. Similarly, \(n\)-day spend was estimated...

Continue reading

Last October we sent a group of Yelpers to the 2019 Grace Hopper Celebration! Here are a few takeaways and reflections from some of our attendees. Who attended? Surashree K., software engineer on Semantic Business Information Clara M., product design lead on Content Anna F., machine learning engineer on Semantic Business Information Nikunja G., software engineer on Infrastructure Security Catlyn K., software engineer on Stream Processing What was your favorite session? Surashree: Honestly, it’s hard to choose, but the one that stuck with me was the talk by Jackie Tsay and Matthew Dierker on Google’s Smart Compose, the Gmail feature...

Continue reading

Varanus The monitor lizards are large lizards in the genus Varanus. Some time ago, our Android app got into a loop of sending data, due to some unlikely interactions between several different systems, which briefly overwhelmed our servers before we were able to turn it off. Fortunately, key code was behind an experiment. Otherwise, apps could have continued misbehaving for days, as there is no guarantee users would immediately update the app. It took an unusual combination of circumstances for this to happen, but this kind of problem seems to be a pervasive concern across the industry, and there are...

Continue reading

Yelp’s mission is to connect users with great local businesses. As part of that mission, we provide local businesses with an ads product to help them better reach out to users. This product strives to showcase the most relevant ads to the user without taking away from their overall search experience on Yelp. In this blog post, we’ll walk through the architecture of how this is made possible by using one of the largest machine learning systems at Yelp: Ads Targeting System. The Ads Targeting System is a machine learning (ML) system designed to serve only the most relevant ads...

Continue reading

We launched our very first Kafka cluster at Yelp more than five years ago. It was not monitored, did not expose any metrics, and we definitely did not have anyone on call for it. One year later, Kafka had already become one of the most important distributed systems running at Yelp, and today has become one of the core components of our infrastructure. Kafka has come a long way since the 0.8 version we were running back then, and our tooling (some of it open-source) has also significantly improved, increasing reliability and reducing the amount of operational work required to...

Continue reading

Scaling security coverage in a growing company is hard. The only way to do this effectively is to empower front-line developers to be able to easily discover, triage, and fix vulnerabilities before they make it to production servers. Today, we’re excited to announce that we’ll be open-sourcing fuzz-lightyear: a testing framework we’ve developed to identify Insecure Direct Object Reference (IDOR) vulnerabilities through stateful Swagger fuzzing, tailored to support an enterprise, microservice architecture. This integrates with our Continuous Integration (CI) pipeline to provide consistent, automatic test coverage as web applications evolve. The Problem As a class of vulnerabilities, IDOR is arguably...

Continue reading

The first half of this post covered the requirements and design choices of the Cassandra Source Connector and dove into the details of the CDC Publisher. As described, the CDC Publisher processes Cassandra CDC data and publishes it as loosely ordered PartitionUpdate objects into Kafka as intermediate keyed streams. The intermediate streams then serve as input for the DP Materializer. Data Pipeline Materializer The DP Materializer ingests the serialized PartitionUpdate objects published by the CDC Publisher, transforms them into fully formed Data Pipeline messages, and publishes them into the Data Pipeline. The DP Materializer is built on top of Apache...

Continue reading

Is there a restaurant you’ve always wanted to check out, but haven’t been able to because they don’t take reservations and the lines are out the door? Here at Yelp, we’re trying to solve problems just like these and delight consumers with streamlined dining experiences. Yelp Waitlist is part of the Yelp Restaurants product suite, and its mission is to take the mystery out of everyday dining experiences, enabling you to get in line at your favorite restaurant through just the tap of a button. For diners, in addition to joining an online waitlist, Yelp Waitlist provides live wait times...

Continue reading