Glossary

• ZTA: zero trust architecture
• SAML: security assertion markup language (an SSO facilitation protocol)
• Devbox: a remote server used to develop software

Zero Trust Access

Remote Future

Yelp is now a fully remote company, which means our employee base has become increasingly distributed across the world, making secure access to resources from anywhere a critical business function. Yelp historically used Ivanti Pulse Secure as the employee VPN, but due to the need for a more reliable solution, it became clear that a change was necessary to ensure secure and consistent access to internal resources. The Corporate Systems and Client Platform Engineering teams began looking for alternative connectivity options to replace Pulse in late 2023. In the early discussions, the question of what role a VPN should play in Yelp was asked. We knew that a large population of daily VPN users did not require full network access and that only select web applications were required to perform their job duties. Work was already underway to shift less sensitive applications to alternate access methods like our MTLS based Edge Gateway. However, this was not an immediate solution for all employees and required significant effort for widespread implementation. This led us to understand that we needed a solution that could support all Yelpers today, with the goal of reducing its use to more granular use cases in the future.

We also recognized that the vast majority of use cases that would be difficult to migrate off VPN were engineering oriented.Whether it was SSH access to a devbox or downloading files from internal servers, these activities involved complex and diverse access patterns. Additionally, they would benefit from improved throughput—something significantly constrained by the current Pulse solution.Additionally, we understood that Zero Trust Architecture (ZTA) was the future. ZTA was not only becoming an industry trend, but also aligned with our long term goals of reducing VPN utilization and creating more fine grained access control structures in the future, as opposed to broad, binary policies on huge subnets and network segments. Implementing a secure, modern high-availability solution was paramount to maintaining productivity for our employees.

Wireguard and Netbird

WireGuard debuted in 2015, and since then it has quickly become the protocol of choice for secure network access. When alternatives to Ivanti were being evaluated, we found ourselves gravitating towards WireGuard-based solutions. To say the least, there are plenty. To narrow down our options, we started compiling a list of must-haves that we quantified to be pillars of a great user experience.

The core pillars we identified as essential for a solution that met our operational and user needs are listed below. Netbird ultimately stood strong; supporting all 5 which made it our solution of choice:

1. Support for Okta as an identity and authentication provider
2. A simple and intuitive user interface
3. Open source and extensible
4. Capable of high throughput and low latency
5. Fault tolerant

Let’s expand over some items in this feature wishlist and how Yelp engineering worked towards realizing this using Netbird and Wireguard.

At Yelp, Okta is used to employ a Zero Trust Authentication model as part of our overall Zero Trust Access strategy. Our previous solution initially used LDAP for authentication, which lacked advanced user and device trust verification. Later, we transitioned to SAML, but the implementation within Ivanti’s product led to a suboptimal user experience due to a cumbersome browser-to-VPN client handoff for session authentication. To address these issues, we sought a solution that supported OpenID Connect (OIDC), specifically with Okta integration. This approach empowers us to enforce policies that ensure only users on managed devices with a secure security posture are granted access. In today’s environment, it’s not enough to simply verify the identity of the user.

While intuitive authentication alone was not our sole goal, OIDC was the first step to ensure a great user experience. Yelp is proud to employ a very diverse workforce with different levels of technical expertise and abilities. Therefore, we wanted an application that was simple and intuitive. Netbird’s implementation of their client largely met this goal but we found that simplicity is key when supporting less-technical users. We began to tailor the client experience for our specific needs, removing some of the more advanced options from the UI. Additionally, we added elements that empowered users to quickly and easily self repair, access user friendly documentation, and request assistance from our awesome helpdesk teams. We didn’t stop there though. We personalized the icons and added feedback for each specific stage of the connection process. All these modifications would not have been possible without a code base that was approachable, well thought out and open source.

Additionally, open source products would allow us to keep a finger on the pulse of a project by tracking its commit and issue histories. What’s more, if critical security issues ever arose, we would not be beholden to the maintainers alone - we ourselves could provide fixes if need be. Open source also means Yelp has the opportunity to contribute back to the community, enhancing the software for everyone’s benefit. To date, multiple changes have been pushed upstream to Netbird’s main branch from Yelpers working to solve issues we encountered, debugged, and ultimately solved.

While most users do not demand high throughput and low latency to complete their day to day business functions, improving these aspects was a clear quality-of-life enhancement we aimed to achieve with a new solution. Nothing drags on like downloading large logs at a snail’s pace, cloning big Git repos and watching the commits trickle in, or connecting to a terminal or remote desktop and feeling like you are moving in slow motion. Pulse, optimistically, had a peak download speed in the low tens of megabits per second for most users. With Netbird being plugged into a 10 gigabit backbone and supported by the blazing fast cryptography of Wireguard, testing showed users could achieve speeds upwards of 1 gigabit per second - mostly restricted by their home internet speed limits. Latency was also close to the pure cost of the wire users traversed with single digit milliseconds of overhead added by wrapping packets in the Wire Guard protocol. Simply put, it was fast and this is exactly what we wanted.

Finally, the worst user experience arises when things simply don’t work. We needed a solution that was robust and fault tolerant. Relying on users to connect to their optimal endpoint was a needless complication. Furthermore, outages regardless of their cause should not require user intervention to mitigate and ideally should be totally transparent to the end user’s experience. WireGuard’s mesh topology aspires to add this redundancy by creating multiple paths a user can take and be simultaneously connected to. Each of these paths or endpoints in Netbird are referred to as a router peer. All members of the mesh are peers but router peers serve the special role of being able to accept and egress traffic from other peers. Clients intrinsically have a one to many relationship with router peers they are permitted to use. This allows for maintenance or service interruption on one router peer without causing a user to reconnect to the network or experience noticeable degradation. Our testing with Netbird showed that a router peer that was actively handling traffic for a given peer could suddenly halt operation, and the client would experience a sub 2 second connectivity interruption while their traffic was rerouted to another host. This addressed the final pillar in our user experience aspirations as we could respond to incidents, security, operational, or otherwise with ease and confidence that Yelpers can continue working without disruption.

Implementation and Outlook

In summary, our desire to learn from our previous challenges with a legacy VPN solution led to a dramatic shift in corporate connectivity. We aimed to provide Yelpers with secure and consistent access to internal resources, ultimately enhancing the daily experiences of Yelpers who work tirelessly to connect people with great local businesses.

Despite encountering hurdles and bumps along the road of delivering this next generation in connectivity, the output shows the juice was well worth the squeeze. We look forward to our next post where we talk about the implementation, challenges, and initial architecture of Yelp’s deployment of Netbird.

Back to blog