Engineering Blog

Streaming MySQL tables in real-time to Kafka

This is the second post in a series covering Yelp's real-time streaming data infrastructure. Our series explores in-depth how we stream MySQL updates in real-time with an exactly-once guarantee, how we automatically track & migrate schemas, how we process and transform streams, and finally how we connect all of this into datastores like Redshift and Salesforce. Read the posts in the series: Billions of Messages a Day - Yelp's Real-time Data Pipeline Streaming MySQL tables in real-time to Kafka More Than Just a Schema Store PaaStorm: A Streaming Processor As our engineering team grew, we realized we had to move...

Continue reading

Yelp API v3 Developer Preview

For the past few months we’ve been working on revamping our API based off your feedback of wanting more Yelp data and functionality. Today, we’re excited to announce that the newest version of our API is entering developer preview. What’s new? We’re exposing two new features as part of the developer preview: autocomplete and transaction search. As a user performs a search, autocomplete will help them find what they want (some might even say we have the ability to read their minds). With autocomplete, a user’s search experience will feel much more intuitive. The API now exposes a search endpoint...

Continue reading

Billions of Messages a Day - Yelp's Real-time Data Pipeline

This is the first post in a series covering Yelp's real-time streaming data infrastructure. Our series explores in-depth how we stream MySQL updates in real-time with an exactly-once guarantee, how we automatically track & migrate schemas, how we process and transform streams, and finally how we connect all of this into datastores like Redshift and Salesforce. Read the posts in the series: Billions of Messages a Day - Yelp's Real-time Data Pipeline Streaming MySQL tables in real-time to Kafka More Than Just a Schema Store PaaStorm: A Streaming Processor Faced with the challenges of scaling out its engineering organization, Yelp...

Continue reading

Yelp Hackathon 19: Color Code

One of the values that we cherish at Yelp is to “Be Unboring”. It’s the quality of never accepting “standard” as okay and the guiding principle for creating new and remarkable things. One of the ways we foster that creative spirit at Yelp Engineering is through our internal hackathons. Many remarkable projects have come out of them - some of them open sourced, some of them revealing interesting demographic insights and some of them pushing the boundaries of science & technology. The 19th edition of our internal hackathon wasn’t any different. Close to 80 fantastic projects across our engineering offices...

Continue reading

Monitoring Cassandra at Scale

At Yelp we leverage Cassandra to fulfill a diverse workload that seems to combine every consistency and availability tradeoff imaginable. It is a fantastically versatile datastore, and a great complement for our developers to our MySQL and Elasticsearch offerings. However, our infrastructure is not done until it ships and is monitored. When we started deploying Cassandra we immediately started looking for ways to properly monitor the datastore so that we could alert developers and operators of issues with their clusters before cluster issues became site issues. Distributed datastores like Cassandra are built to deal with failure, but our monitoring solution...

Continue reading

Autoscaling PaaSTA Services

If you haven’t heard about PaaSTA before, feel free to check out the blog post introducing it. One step in creating a service is to decide how many compute resources it needs. From the inception of PaaSTA, changing a service’s resource allocation has required manually editing and pushing new configs, and service authors had to pore over graphs and alerts to determine the proper resource allocation for a service whenever load requirements changed. This changed earlier this month when autoscaling was introduced into PaaSTA. Why did we do this? Autoscaling was introduced into PaaSTA to make sure services are allocated...

Continue reading

Yelpicons: A Cross-Platform Icon Build System

It’s 10:00 AM and a designer just created a new icon for Yelp. At 10:05 AM iOS has received a newly versioned CocoaPod containing 1x, 2x, and 3x PNGs, Android has received an AAR Library containing mdpi, hdpi, xhdpi, and xxhdpi WebPs, and the Web CDN has a new SVG and PNG spritesheet, code to manage the PNG fallback for our IE8 users, and a new Python package to access the icon in a template. The workflow wasn’t always this streamlined and was only made possible through a cross-team collaboration with Android, iOS, Web, and Design at Yelp. With a...

Continue reading

Yelp Android App Went On A Diet

Whether it’s battery usage, network, or time, we care a lot about our users’ resources. A big app creates a barrier of entry for users on metered or low speed networks. Last Thanksgiving, our automated alerts let us know that our app size was getting larger than we’d like, and making it harder for low resource users to download our app. Here we saw an upward trend in app size as we added more features and built a more compelling app. In order to tackle app size, we first had to understand where that size was coming from. Breakdown of...

Continue reading