Since Yelp introduced its real-time streaming data infrastructure, “Data Pipeline”, it has grown in scope and matured vastly. It now supports some of Yelp’s most critical business requirements in its mission to connect people with great local businesses. Today, it has expanded into a diverse ecosystem of connectors sourcing data from Kafka and MySQL, and sinking data into Cassandra, Elasticsearch, Kafka, MySQL, Redshift, and S3. To ensure that the whole ecosystem is functioning correctly, Yelp’s Data Pipeline infrastructure is continually growing its repertoire of reliability techniques such as write-ahead logging, two-phase commit, fuzz testing, monkey testing, and black-box auditing to...

Continue reading

Peter Seibel wrote that to maximize engineering effectiveness, “Let a thousand flowers bloom. Then rip 999 of them out by the roots.” Flowers, in how the metaphor applies to us, are code patterns — the myriad different functions, classes, styles, and idioms that developers use when writing code. At first, new flowers are welcome — maybe the new pattern seems easier to use, more scalable, more efficient, or more suited to some particular task than the old. As a code base grows, and the flowers proliferate, however, it becomes clear which patterns work and which don’t. Suddenly, code patterns that...

Continue reading

If you haven’t heard about PaaSTA before, feel free to check out the blog post introducing it. One step in creating a service is to decide how many compute resources it needs. From the inception of PaaSTA, changing a service’s resource allocation has required manually editing and pushing new configs, and service authors had to pore over graphs and alerts to determine the proper resource allocation for a service whenever load requirements changed. This changed earlier this month when autoscaling was introduced into PaaSTA. Why did we do this? Autoscaling was introduced into PaaSTA to make sure services are allocated...

Continue reading

Website performance can be judged in a lot of ways, but perhaps the most important is user-perceived performance: the amount of time that is taken between clicking a link and having the desired page rendered on the screen. A big part of keeping things feeling snappy is understanding which bits of content are blocking the “critical rendering path,” and coming up with ways to shorten or unblock them. At Yelp we focused on shortening the process of loading our CSS stylesheets. Before the browser can begin rendering the page, it needs to have its HTML markup and CSS rules. Usually,...

Continue reading

We recently launched Yelp’s new Today Widget, so now you can search Yelp with a single tap from anywhere on your iPhone or iPad (iOS 8+). Today Widgets live in Notification Center, which can be accessed anywhere on the device by swiping on the top of the screen. You can search directly for a category near your current location, or open up the app directly to search input. App Extensions Today Widgets are a type of app extension (which include action, share, and photo editing extensions, custom keyboards, and document pickers). An app extension is a separate binary from the...

Continue reading

Over the years Yelp has built cool integrations with partners such as Apple, Microsoft, Yahoo and many others that have enabled users to interact with our rich dataset in unique ways. One of the methods we’ve used to achieve these integrations is by building a robust, scalable data feeds system, which contains information for over 50 million businesses. As we allow partners to have their own configurations, information included in these feeds may vary from one partner to another and we did not have an easy way to access the history of data feeds that were delivered to our partners....

Continue reading

One of the most engaging features of Yelp is our photos and videos gallery. When you visit a Yelp Business Page inside of the mobile app, there is a photo at the top of the page to provide visual context. It also serves as a compelling entry point to our photo viewer if you pull it down. We wanted to have this same effect on our mobile site, so we set out to develop a nice, smooth animation to pull down this photo and delight mobile web users with the same experience they’re used to on our mobile applications. The...

Continue reading

At Yelp, we often need to analyze data in real time. Right now most of our data is aggregated and logged using a system called Scribe. This means that real-time processes currently depend on tailing Scribe, reading messages as they are added to the end of the logs. Simply tailing the logs means that any time a tailer is not running, due to maintenance or an unexpected error, it misses all the information logged during that time. We want a more flexible system that allows processes to pick up where they left off, so our analyses account for all of...

Continue reading