原文地址

LinkedIn started in 2003 with the goal of connecting to your network for better job opportunities. It had only 2,700 members the first week. Fast forward many years, and LinkedIn’s product portfolio, member base, and server load has grown tremendously.

Today, LinkedIn operates globally with more than 350 million members. We serve tens of thousands of web pages every second of every day. We've hit our mobile moment where mobile accounts for more than 50 percent of all global traffic. All those requests are fetching data from our backend systems, which in turn handle millions of queries per second.

So, how did we get there?

The early years


Leo

LinkedIn started as many sites start today, as a single monolithic application doing it all. That single app was called Leo. It hosted web servlets for all the various pages, handled business logic, and connected to a handful of LinkedIn databases.

Ah, the good old days of website development - nice and simple

Member Graph

One of the first things to do as a social network is to manage member to member connections. We needed a system that queried connection data using graph traversals and lived in-memory for top efficiency and performance. With this different usage profile, it was clear it needed to scale independently of Leo, so a separate system for our member graph called Cloud was born - LinkedIn’s first service. To keep this graph service separate from Leo, we used Java RPC for communication.

It was around this time we needed search capabilities. Our member graph service started feeding data into a new search service running Lucene.

Replica read DBs

As the site grew, so did Leo, increasing its role and responsibility, and naturally increasing its complexity. Load balancing helped as multiple instances of Leo were spun up. But the added load was taxing LinkedIn’s most critical system - its member profile database.

An easy fix we did was classic vertical scaling - throw more CPUs and memory at it! While that bought some time, we needed to scale further. The profile database handled both read and write traffic, and so in order to scale, replica slave DBs were introduced. The replica DBs were a copy of the member database, staying in sync using the earliest version of databus (now open-sourced). They were set up to handle all read traffic and logic was built to know when it was safe (consistent) to read from a replica versus the main master DB.

* While the master-slave model worked as a medium-term solution, we’ve since moved to partitioned DBs

As the site began to see more and more traffic, our single monolithic app Leo was often going down in production, it was difficult to troubleshoot and recover, and difficult to release new code. High availability is critical to LinkedIn. It was clear we needed to “Kill Leo” and break it up into many small functional and stateless services.

“Kill Leo” was the mantra internally for many years...

Service Oriented Architecture

Engineering started to extract micro services to hold APIs and business logic like our search, profile, communications, and groups platforms. Later, our presentation layers were extracted for areas like our recruiter product or public profile. For new products, brand new services were created outside of Leo. Over time, vertical stacks emerged for each functional area.

We built frontend servers to fetch data models from different domains, handle presentation logic, and build the HTML (via JSPs). We built mid-tier services to provide API access to data models and backend data services to provide consistent access to its database(s). By 2010, we already had over 150 separate services. Today, we have over 750 services.

An example multi-tier service oriented architecture within LinkedIn

Being stateless, scaling could be achieved by spinning up new instances of any of the services and using hardware load balancers between them. We actively started to redline each service to know how much load it could take, and built out early provisioning and performance monitoring capabilities.

Caching

LinkedIn was seeing hypergrowth and needed to scale further. We knew we could reduce the load altogether by adding more layers of cache. Many applications started to introduce mid-tier caching layers like memcache or couchbase. We also added caches to our data layers and started to use Voldemort with precomputed results when appropriate.

Over time, we actually removed many mid-tier caches. Mid-tier caches were storing derived data from multiple domains. While caches appear to be a simple way to reduce load at first, the complexity around invalidation and the call graph was getting out of hand. Keeping the cache closest to the data store as possible keeps latencies low, allows us to scale horizontally, and reduces the cognitive load.

Kafka

To collect its growing amount of data, LinkedIn developed many custom data pipelines for streaming and queueing data. For example, we needed our data to flow into data warehouse, we needed to send batches of data into our Hadoop workflow for analytics, we collected and aggregated logs from every service, we collected tracking events like pageviews, we needed queueing for our inMail messaging system, and we needed to keep our people search system up to date whenever someone updated their profile.

As the site grew, more of these custom pipelines emerged. As the site needed to scale, each individual pipeline needed to scale. Something had to give. The result was the development of Kafka, our distributed pub-sub messaging platform. Kafka became a universal pipeline, built around the concept of a commit log, and was built with speed and scalability in mind. It enabled near realtime access to any data source, empowered our Hadoop jobs, allowed us to build realtime analytics, vastly improved our site monitoring and alerting capability, and enabled us to visualize and track our call graphs. Today, Kafka handles well over 500 billion events per day.

Kafka as the universal data stream broker

Inversion

Scaling can be measured across many dimensions, including organizational. In late 2011, LinkedIn kicked off an internal initiative called Inversion. This initiative put a pause on feature development and allowed the entire engineering organization to focus on improving tooling and deployment, infrastructure, and developer productivity. It was successful in enabling the engineering agility we need to build the scalable new products we have today.

The modern years


Rest.li

When we transformed from Leo to a service oriented architecture, the APIs we extracted assumed Java-based RPC, were inconsistent across teams, were tightly coupled with the presentation layer, and it was only getting worse. To address this, we built out a new API model called Rest.li. Rest.li was our move towards a data model centric architecture, which ensured a consistent stateless Restful API model across the company.

By using JSON over HTTP, our new APIs finally made it easy to have non-Java-based clients. LinkedIn today is still mainly a Java shop, but also has many clients utilizing Python, Ruby, Node.js, and C++ both developed in house as well as from tech stacks of our acquisitions. Moving away from RPC also freed us from high coupling with presentation tiers and many backwards compatibility problems. Plus, by using Dynamic Discovery (D2) with Rest.li, we got automated client based load balancing, discovery, and scalability of each service API.

Today, LinkedIn has over 975 Rest.li resources and over 100 billion Rest.li calls per day across all our datacenters.

Rest.li R2/D2 tech stack

Super Blocks

Service oriented architectures work well to decouple domains and scale services independently. But there are downsides. Many of our applications fetch many types of different data, in turn making hundreds of downstream calls. This is typically referred to as a “call graph”, or “fanout” when considering all the many downstream calls. For example, any Profile page request fetches much more beyond just profile data including photos, connections, groups, subscription info, following info, long form blog posts, connection degrees from our graph, recommendations, etc. This call graph can be difficult to manage and was only getting more and more unruly.

We introduced the concept of a super block - groupings of backend services with a single access API. This allows us to have a specific team optimize the block, while keeping our call graph in check for each client.

Multi-Data Center

Being a global company with a fast growing member population, we needed to scale beyond serving traffic from one data center. We began an effort years ago to address this, first by serving public profiles out of two data centers (Los Angeles and Chicago). Once proven, we embarked on enhancing all our services to handle data replication, callbacks from different origins, one-way data replication events, and pinning users to a geographically close data center.

Many of our databases run on Espresso (a new in-house multi-tenant datastore). Espresso was built with multi data centers in mind. It provides master / master support and handles much of the difficult replication.

Multiple data centers are incredibly important to maintain “site-up” and high availability. You need to avoid any single point of failure not just for each individual service, but the entire site. Today, LinkedIn runs out of three main data centers, with additional PoPs around the globe.

LinkedIn's operational setup as of 2015 (circles represent data centers, diamonds represent PoPs)

What else have we done?

Of course, our scaling story is never this simple. There’s a countless number of things we’ve done over the years across all engineering and operations teams, including some of these larger initiatives:

Many of our most critical systems have their own rich history and evolution to address scale over the years. This includes our member graph service (our first service outside from Leo), search (our second service), news feed, communications platform, and member profile backend.

We’ve built data infrastructure that enables long term growth. This was first evident with Databus and Kafka, and has continued with Samza for data streams, Espresso and Voldemort for storage solutions, Pinot for our analytics systems, as well as other custom solutions. Plus, our tooling has improved such that developers can provision this infra automatically.

We’ve developed a massive offline workflow using Hadoop and our Voldemort data store to precompute data insights like People You May Know, Similar profiles, Notable Alumni, and profile browse maps.

We’ve rethought our frontend approach, adding client templates into the mix (Profile page, University pages). This enables more interactive applications, requiring our servers to send only JSON or partial JSON. Plus, templates get cached in CDNs and the browser. We also started to use BigPipe and the Play framework, changing our model from a threaded web server to a non-blocking asynchronous one.

Beyond the application code, we’ve introduced multiple tiers of proxies using Apache Traffic Server and HAProxy to handle load balancing, data center pinning, security, intelligent routing, server side rendering, and more.

And finally, we continue to improve the performance of our servers with optimized hardware, advanced memory and system tuning, and utilizing newer Java runtimes.

What’s next


LinkedIn continues to grow quickly and there’s still a ton of work we can do to improve. We’re working on problems that very few ever get to solve - come join us!

A Brief History of Scaling LinkedIn的更多相关文章

  1. [转载] LinkedIn架构这十年

    原文: http://colobu.com/2015/07/24/brief-history-scaling-linkedin/ 原文: A Brief History of Scaling Link ...

  2. LinkedIn架构这十年

    原文: A Brief History of Scaling LinkedIn 2003年是LinkedIn元年,公司成立的目标是连接你的个人人脉以获得更好的的工作机会.上线第一周才有2700个会员注 ...

  3. Enhancing the Scalability of Memcached

    原文地址: https://software.intel.com/en-us/articles/enhancing-the-scalability-of-memcached-0 1 Introduct ...

  4. [转载] 首席工程师揭秘:LinkedIn大数据后台是如何运作的?(一)

    本文作者:Jay Kreps,linkedin公司首席工程师:文章来自于他在linkedin上的分享:原文标题:The Log: What every software engineer should ...

  5. History API与浏览器历史堆栈管理

    移动端开发在某些场景中有着特殊需求,如为了提高用户体验和加快响应速度,常常在部分工程采用SPA架构.传统的单页应用基于url的hash值进行路由,这种实现不存在兼容性问题,但是缺点也有--针对不支持o ...

  6. 使用backbone的history管理SPA应用的url

    本文介绍如何使用backbone的history模块实现SPA应用里面的URL管理.SPA应用的核心在于使用无刷新的方式更改url,从而引发页面内容的改变.从实现上来看,url的管理和页面内容的管理是 ...

  7. 使用h5的history改善ajax列表请求体验

    信息比较丰富的网站通常会以分页显示,在点“下一页”时,很多网站都采用了动态请求的方式,避免页面刷新.虽然大家都是ajax,但是从一些小的细节还是 可以区分优劣.一个小的细节是能否支持浏览器“后退”和“ ...

  8. HTML5学习笔记之History API

    这系列文章主要是学习Html5相关的知识点,以学习API知识点为入口,由浅入深的引入实例,让大家一步一步的体会"h5"能够做什么,以及在实际项目中如何去合理的运用达到使用自如,完美 ...

  9. Linux下history命令用法

    如果你经常使用 Linux 命令行,那么使用 history(历史)命令可以有效地提升你的效率.本文将通过实例的方式向你介绍 history 命令的 15 个用法. 使用 HISTTIMEFORMAT ...

随机推荐

  1. hdoj 1874 畅通工程续

    Problem Description 某省自从实行了很多年的畅通工程计划后,终于修建了很多路.不过路多了也不好,每次要从一个城镇到另一个城镇时,都有许多种道路方案可以选择,而某些方案要比另一些方案行 ...

  2. PHP 使用CURL

    CURL是一个非常强大的开源库,支持很多协议,包括HTTP.FTP.TELNET等,我们使用它来发送HTTP请求.它给我 们带来的好处是可以通过灵活的选项设置不同的HTTP协议参数,并且支持HTTPS ...

  3. 【动态规划】bzoj1664 [Usaco2006 Open]County Fair Events 参加节日庆祝

    将区间按左端点排序. f(i)=max{f(j)+1}(p[j].x+p[j].y<=p[i].x && j<i) #include<cstdio> #incl ...

  4. C++二级指针第三种内存模型

    #include "stdio.h" #include "stdlib.h" #include "string.h" void main() ...

  5. GridView获取当前行

    int row = ((GridViewRow)((DropDownList)sender).NamingContainer).RowIndex; //获取GridView里的DropDownList ...

  6. C2第四次作业解题报告

    看过题解后如果觉得还算有用,请帮忙加点我所在团队博客访问量 http://www.cnblogs.com/newbe/ http://www.cnblogs.com/newbe/p/4069834.h ...

  7. 在.sln文件中设置Visual Studio默认启动项目的简单方法

    昨天在一台电脑上用git新签出一个项目进行build,却出现一堆编译错误,而在原先的开发机上build无任何错误.对比分析后发现,开发机上VS的启动项目(startup project)与这台电脑上的 ...

  8. 困扰多日的C#调用Haskell问题竟然是Windows的一个坑

    最近一直被C#调用Haskell时的“尝试读取或写入受保护的内存”问题所困扰(详见C#调用haskell遭遇Attempted to read or write protected memory,C# ...

  9. 简洁清新的box样式

    小菜今天偶然打开了腾讯公益,本来是冲着公益活动去的,没想到腾讯公益界面做的还不错,索性把它的box样式剽窃了下来. 个人觉得腾讯的东西普遍比较精致,就拿页面来说,每一个细节都把握的很到位,例如取色,看 ...

  10. centos安装firefox flash插件

    centos下的firefox flash插件默认不是最新版的,安装过程如下: 将安装地址添加到repolist中 sudo yum -y install http://linuxdownload.a ...