原文地址

LinkedIn started in 2003 with the goal of connecting to your network for better job opportunities. It had only 2,700 members the first week. Fast forward many years, and LinkedIn’s product portfolio, member base, and server load has grown tremendously.

Today, LinkedIn operates globally with more than 350 million members. We serve tens of thousands of web pages every second of every day. We've hit our mobile moment where mobile accounts for more than 50 percent of all global traffic. All those requests are fetching data from our backend systems, which in turn handle millions of queries per second.

So, how did we get there?

The early years


Leo

LinkedIn started as many sites start today, as a single monolithic application doing it all. That single app was called Leo. It hosted web servlets for all the various pages, handled business logic, and connected to a handful of LinkedIn databases.

Ah, the good old days of website development - nice and simple

Member Graph

One of the first things to do as a social network is to manage member to member connections. We needed a system that queried connection data using graph traversals and lived in-memory for top efficiency and performance. With this different usage profile, it was clear it needed to scale independently of Leo, so a separate system for our member graph called Cloud was born - LinkedIn’s first service. To keep this graph service separate from Leo, we used Java RPC for communication.

It was around this time we needed search capabilities. Our member graph service started feeding data into a new search service running Lucene.

Replica read DBs

As the site grew, so did Leo, increasing its role and responsibility, and naturally increasing its complexity. Load balancing helped as multiple instances of Leo were spun up. But the added load was taxing LinkedIn’s most critical system - its member profile database.

An easy fix we did was classic vertical scaling - throw more CPUs and memory at it! While that bought some time, we needed to scale further. The profile database handled both read and write traffic, and so in order to scale, replica slave DBs were introduced. The replica DBs were a copy of the member database, staying in sync using the earliest version of databus (now open-sourced). They were set up to handle all read traffic and logic was built to know when it was safe (consistent) to read from a replica versus the main master DB.

* While the master-slave model worked as a medium-term solution, we’ve since moved to partitioned DBs

As the site began to see more and more traffic, our single monolithic app Leo was often going down in production, it was difficult to troubleshoot and recover, and difficult to release new code. High availability is critical to LinkedIn. It was clear we needed to “Kill Leo” and break it up into many small functional and stateless services.

“Kill Leo” was the mantra internally for many years...

Service Oriented Architecture

Engineering started to extract micro services to hold APIs and business logic like our search, profile, communications, and groups platforms. Later, our presentation layers were extracted for areas like our recruiter product or public profile. For new products, brand new services were created outside of Leo. Over time, vertical stacks emerged for each functional area.

We built frontend servers to fetch data models from different domains, handle presentation logic, and build the HTML (via JSPs). We built mid-tier services to provide API access to data models and backend data services to provide consistent access to its database(s). By 2010, we already had over 150 separate services. Today, we have over 750 services.

An example multi-tier service oriented architecture within LinkedIn

Being stateless, scaling could be achieved by spinning up new instances of any of the services and using hardware load balancers between them. We actively started to redline each service to know how much load it could take, and built out early provisioning and performance monitoring capabilities.

Caching

LinkedIn was seeing hypergrowth and needed to scale further. We knew we could reduce the load altogether by adding more layers of cache. Many applications started to introduce mid-tier caching layers like memcache or couchbase. We also added caches to our data layers and started to use Voldemort with precomputed results when appropriate.

Over time, we actually removed many mid-tier caches. Mid-tier caches were storing derived data from multiple domains. While caches appear to be a simple way to reduce load at first, the complexity around invalidation and the call graph was getting out of hand. Keeping the cache closest to the data store as possible keeps latencies low, allows us to scale horizontally, and reduces the cognitive load.

Kafka

To collect its growing amount of data, LinkedIn developed many custom data pipelines for streaming and queueing data. For example, we needed our data to flow into data warehouse, we needed to send batches of data into our Hadoop workflow for analytics, we collected and aggregated logs from every service, we collected tracking events like pageviews, we needed queueing for our inMail messaging system, and we needed to keep our people search system up to date whenever someone updated their profile.

As the site grew, more of these custom pipelines emerged. As the site needed to scale, each individual pipeline needed to scale. Something had to give. The result was the development of Kafka, our distributed pub-sub messaging platform. Kafka became a universal pipeline, built around the concept of a commit log, and was built with speed and scalability in mind. It enabled near realtime access to any data source, empowered our Hadoop jobs, allowed us to build realtime analytics, vastly improved our site monitoring and alerting capability, and enabled us to visualize and track our call graphs. Today, Kafka handles well over 500 billion events per day.

Kafka as the universal data stream broker

Inversion

Scaling can be measured across many dimensions, including organizational. In late 2011, LinkedIn kicked off an internal initiative called Inversion. This initiative put a pause on feature development and allowed the entire engineering organization to focus on improving tooling and deployment, infrastructure, and developer productivity. It was successful in enabling the engineering agility we need to build the scalable new products we have today.

The modern years


Rest.li

When we transformed from Leo to a service oriented architecture, the APIs we extracted assumed Java-based RPC, were inconsistent across teams, were tightly coupled with the presentation layer, and it was only getting worse. To address this, we built out a new API model called Rest.li. Rest.li was our move towards a data model centric architecture, which ensured a consistent stateless Restful API model across the company.

By using JSON over HTTP, our new APIs finally made it easy to have non-Java-based clients. LinkedIn today is still mainly a Java shop, but also has many clients utilizing Python, Ruby, Node.js, and C++ both developed in house as well as from tech stacks of our acquisitions. Moving away from RPC also freed us from high coupling with presentation tiers and many backwards compatibility problems. Plus, by using Dynamic Discovery (D2) with Rest.li, we got automated client based load balancing, discovery, and scalability of each service API.

Today, LinkedIn has over 975 Rest.li resources and over 100 billion Rest.li calls per day across all our datacenters.

Rest.li R2/D2 tech stack

Super Blocks

Service oriented architectures work well to decouple domains and scale services independently. But there are downsides. Many of our applications fetch many types of different data, in turn making hundreds of downstream calls. This is typically referred to as a “call graph”, or “fanout” when considering all the many downstream calls. For example, any Profile page request fetches much more beyond just profile data including photos, connections, groups, subscription info, following info, long form blog posts, connection degrees from our graph, recommendations, etc. This call graph can be difficult to manage and was only getting more and more unruly.

We introduced the concept of a super block - groupings of backend services with a single access API. This allows us to have a specific team optimize the block, while keeping our call graph in check for each client.

Multi-Data Center

Being a global company with a fast growing member population, we needed to scale beyond serving traffic from one data center. We began an effort years ago to address this, first by serving public profiles out of two data centers (Los Angeles and Chicago). Once proven, we embarked on enhancing all our services to handle data replication, callbacks from different origins, one-way data replication events, and pinning users to a geographically close data center.

Many of our databases run on Espresso (a new in-house multi-tenant datastore). Espresso was built with multi data centers in mind. It provides master / master support and handles much of the difficult replication.

Multiple data centers are incredibly important to maintain “site-up” and high availability. You need to avoid any single point of failure not just for each individual service, but the entire site. Today, LinkedIn runs out of three main data centers, with additional PoPs around the globe.

LinkedIn's operational setup as of 2015 (circles represent data centers, diamonds represent PoPs)

What else have we done?

Of course, our scaling story is never this simple. There’s a countless number of things we’ve done over the years across all engineering and operations teams, including some of these larger initiatives:

Many of our most critical systems have their own rich history and evolution to address scale over the years. This includes our member graph service (our first service outside from Leo), search (our second service), news feed, communications platform, and member profile backend.

We’ve built data infrastructure that enables long term growth. This was first evident with Databus and Kafka, and has continued with Samza for data streams, Espresso and Voldemort for storage solutions, Pinot for our analytics systems, as well as other custom solutions. Plus, our tooling has improved such that developers can provision this infra automatically.

We’ve developed a massive offline workflow using Hadoop and our Voldemort data store to precompute data insights like People You May Know, Similar profiles, Notable Alumni, and profile browse maps.

We’ve rethought our frontend approach, adding client templates into the mix (Profile page, University pages). This enables more interactive applications, requiring our servers to send only JSON or partial JSON. Plus, templates get cached in CDNs and the browser. We also started to use BigPipe and the Play framework, changing our model from a threaded web server to a non-blocking asynchronous one.

Beyond the application code, we’ve introduced multiple tiers of proxies using Apache Traffic Server and HAProxy to handle load balancing, data center pinning, security, intelligent routing, server side rendering, and more.

And finally, we continue to improve the performance of our servers with optimized hardware, advanced memory and system tuning, and utilizing newer Java runtimes.

What’s next


LinkedIn continues to grow quickly and there’s still a ton of work we can do to improve. We’re working on problems that very few ever get to solve - come join us!

A Brief History of Scaling LinkedIn的更多相关文章

  1. [转载] LinkedIn架构这十年

    原文: http://colobu.com/2015/07/24/brief-history-scaling-linkedin/ 原文: A Brief History of Scaling Link ...

  2. LinkedIn架构这十年

    原文: A Brief History of Scaling LinkedIn 2003年是LinkedIn元年,公司成立的目标是连接你的个人人脉以获得更好的的工作机会.上线第一周才有2700个会员注 ...

  3. Enhancing the Scalability of Memcached

    原文地址: https://software.intel.com/en-us/articles/enhancing-the-scalability-of-memcached-0 1 Introduct ...

  4. [转载] 首席工程师揭秘:LinkedIn大数据后台是如何运作的?(一)

    本文作者:Jay Kreps,linkedin公司首席工程师:文章来自于他在linkedin上的分享:原文标题:The Log: What every software engineer should ...

  5. History API与浏览器历史堆栈管理

    移动端开发在某些场景中有着特殊需求,如为了提高用户体验和加快响应速度,常常在部分工程采用SPA架构.传统的单页应用基于url的hash值进行路由,这种实现不存在兼容性问题,但是缺点也有--针对不支持o ...

  6. 使用backbone的history管理SPA应用的url

    本文介绍如何使用backbone的history模块实现SPA应用里面的URL管理.SPA应用的核心在于使用无刷新的方式更改url,从而引发页面内容的改变.从实现上来看,url的管理和页面内容的管理是 ...

  7. 使用h5的history改善ajax列表请求体验

    信息比较丰富的网站通常会以分页显示,在点“下一页”时,很多网站都采用了动态请求的方式,避免页面刷新.虽然大家都是ajax,但是从一些小的细节还是 可以区分优劣.一个小的细节是能否支持浏览器“后退”和“ ...

  8. HTML5学习笔记之History API

    这系列文章主要是学习Html5相关的知识点,以学习API知识点为入口,由浅入深的引入实例,让大家一步一步的体会"h5"能够做什么,以及在实际项目中如何去合理的运用达到使用自如,完美 ...

  9. Linux下history命令用法

    如果你经常使用 Linux 命令行,那么使用 history(历史)命令可以有效地提升你的效率.本文将通过实例的方式向你介绍 history 命令的 15 个用法. 使用 HISTTIMEFORMAT ...

随机推荐

  1. .net委托(转载)

    一.什么是委托? 委托就是定义一个 对于方法的引用,类似于c++中的函数指针.委托是用来将方法作为参数 传递进入其它方法的, 委托的格式? 修饰符1 delegate  修饰符2  委托类型名(变量n ...

  2. 贪心+构造( Codeforces Round #344 (Div. 2))

    题目:Report 题意:有两种操作: 1)t = 1,前r个数字按升序排列:   2)t = 2,前r个数字按降序排列: 求执行m次操作后的排列顺序. #include <iostream&g ...

  3. 斯坦福第十一课:机器学习系统的设计(Machine Learning System Design)

    11.1  首先要做什么 11.2  误差分析 11.3  类偏斜的误差度量 11.4  查全率和查准率之间的权衡 11.5  机器学习的数据 11.1  首先要做什么 在接下来的视频中,我将谈到机器 ...

  4. PYTHON第二天

    PYTHON之路 4.注释 Python开发规范:每一行不能超过80个字符 # : 代表注释当行 ‘’’  上下两行  ,  三个英文字符的引号,代表注释某一段代码 ‘’’ 知识点用户输入: PYth ...

  5. BZOJ3173 TJOI2013最长上升子序列(Treap+ZKW线段树)

    传送门 Description 给定一个序列,初始为空.现在我们将1到N的数字插入到序列中,每次将一个数字插入到一个特定的位置.每插入一个数字,我们都想知道此时最长上升子序列长度是多少? Input ...

  6. .android:allowTaskReparenting 等Activity 的task属性

    转自http://blog.csdn.net/javayinjaibo/article/details/8855678 1.android:allowTaskReparenting 这个属性用来标记一 ...

  7. 如何将jar包加入到Maven本地仓库

    原则上Maven的设计是不需要这么做的,因为pom.xml中依赖的jar包会自动实现从中央仓库下载到本地仓库.但是公司设计了一个setting,如果本地仓库没有,就去setting指定的url中下载j ...

  8. 502 Server dropped connection

    在本地电脑上开启了,全局VPN代理后,出现 502 报错. 502 Server dropped connection The following error occurred while tryin ...

  9. HTML 中禁用鼠标右键和不能选中文字

    在body中加这句就可以了. <body oncontextmenu="return false" onselectstart="return false" ...

  10. 在Linux CentOS上编译CoreCLR

    经过几天的努力,终于解决了在CentOS上编译CoreCLR的问题.最终发现问题是CMAKE_C_FLAGS的设置引起的. 只要在“src/pal/tools/clang-compiler-overr ...