原文地址

LinkedIn started in 2003 with the goal of connecting to your network for better job opportunities. It had only 2,700 members the first week. Fast forward many years, and LinkedIn’s product portfolio, member base, and server load has grown tremendously.

Today, LinkedIn operates globally with more than 350 million members. We serve tens of thousands of web pages every second of every day. We've hit our mobile moment where mobile accounts for more than 50 percent of all global traffic. All those requests are fetching data from our backend systems, which in turn handle millions of queries per second.

So, how did we get there?

The early years


Leo

LinkedIn started as many sites start today, as a single monolithic application doing it all. That single app was called Leo. It hosted web servlets for all the various pages, handled business logic, and connected to a handful of LinkedIn databases.

Ah, the good old days of website development - nice and simple

Member Graph

One of the first things to do as a social network is to manage member to member connections. We needed a system that queried connection data using graph traversals and lived in-memory for top efficiency and performance. With this different usage profile, it was clear it needed to scale independently of Leo, so a separate system for our member graph called Cloud was born - LinkedIn’s first service. To keep this graph service separate from Leo, we used Java RPC for communication.

It was around this time we needed search capabilities. Our member graph service started feeding data into a new search service running Lucene.

Replica read DBs

As the site grew, so did Leo, increasing its role and responsibility, and naturally increasing its complexity. Load balancing helped as multiple instances of Leo were spun up. But the added load was taxing LinkedIn’s most critical system - its member profile database.

An easy fix we did was classic vertical scaling - throw more CPUs and memory at it! While that bought some time, we needed to scale further. The profile database handled both read and write traffic, and so in order to scale, replica slave DBs were introduced. The replica DBs were a copy of the member database, staying in sync using the earliest version of databus (now open-sourced). They were set up to handle all read traffic and logic was built to know when it was safe (consistent) to read from a replica versus the main master DB.

* While the master-slave model worked as a medium-term solution, we’ve since moved to partitioned DBs

As the site began to see more and more traffic, our single monolithic app Leo was often going down in production, it was difficult to troubleshoot and recover, and difficult to release new code. High availability is critical to LinkedIn. It was clear we needed to “Kill Leo” and break it up into many small functional and stateless services.

“Kill Leo” was the mantra internally for many years...

Service Oriented Architecture

Engineering started to extract micro services to hold APIs and business logic like our search, profile, communications, and groups platforms. Later, our presentation layers were extracted for areas like our recruiter product or public profile. For new products, brand new services were created outside of Leo. Over time, vertical stacks emerged for each functional area.

We built frontend servers to fetch data models from different domains, handle presentation logic, and build the HTML (via JSPs). We built mid-tier services to provide API access to data models and backend data services to provide consistent access to its database(s). By 2010, we already had over 150 separate services. Today, we have over 750 services.

An example multi-tier service oriented architecture within LinkedIn

Being stateless, scaling could be achieved by spinning up new instances of any of the services and using hardware load balancers between them. We actively started to redline each service to know how much load it could take, and built out early provisioning and performance monitoring capabilities.

Caching

LinkedIn was seeing hypergrowth and needed to scale further. We knew we could reduce the load altogether by adding more layers of cache. Many applications started to introduce mid-tier caching layers like memcache or couchbase. We also added caches to our data layers and started to use Voldemort with precomputed results when appropriate.

Over time, we actually removed many mid-tier caches. Mid-tier caches were storing derived data from multiple domains. While caches appear to be a simple way to reduce load at first, the complexity around invalidation and the call graph was getting out of hand. Keeping the cache closest to the data store as possible keeps latencies low, allows us to scale horizontally, and reduces the cognitive load.

Kafka

To collect its growing amount of data, LinkedIn developed many custom data pipelines for streaming and queueing data. For example, we needed our data to flow into data warehouse, we needed to send batches of data into our Hadoop workflow for analytics, we collected and aggregated logs from every service, we collected tracking events like pageviews, we needed queueing for our inMail messaging system, and we needed to keep our people search system up to date whenever someone updated their profile.

As the site grew, more of these custom pipelines emerged. As the site needed to scale, each individual pipeline needed to scale. Something had to give. The result was the development of Kafka, our distributed pub-sub messaging platform. Kafka became a universal pipeline, built around the concept of a commit log, and was built with speed and scalability in mind. It enabled near realtime access to any data source, empowered our Hadoop jobs, allowed us to build realtime analytics, vastly improved our site monitoring and alerting capability, and enabled us to visualize and track our call graphs. Today, Kafka handles well over 500 billion events per day.

Kafka as the universal data stream broker

Inversion

Scaling can be measured across many dimensions, including organizational. In late 2011, LinkedIn kicked off an internal initiative called Inversion. This initiative put a pause on feature development and allowed the entire engineering organization to focus on improving tooling and deployment, infrastructure, and developer productivity. It was successful in enabling the engineering agility we need to build the scalable new products we have today.

The modern years


Rest.li

When we transformed from Leo to a service oriented architecture, the APIs we extracted assumed Java-based RPC, were inconsistent across teams, were tightly coupled with the presentation layer, and it was only getting worse. To address this, we built out a new API model called Rest.li. Rest.li was our move towards a data model centric architecture, which ensured a consistent stateless Restful API model across the company.

By using JSON over HTTP, our new APIs finally made it easy to have non-Java-based clients. LinkedIn today is still mainly a Java shop, but also has many clients utilizing Python, Ruby, Node.js, and C++ both developed in house as well as from tech stacks of our acquisitions. Moving away from RPC also freed us from high coupling with presentation tiers and many backwards compatibility problems. Plus, by using Dynamic Discovery (D2) with Rest.li, we got automated client based load balancing, discovery, and scalability of each service API.

Today, LinkedIn has over 975 Rest.li resources and over 100 billion Rest.li calls per day across all our datacenters.

Rest.li R2/D2 tech stack

Super Blocks

Service oriented architectures work well to decouple domains and scale services independently. But there are downsides. Many of our applications fetch many types of different data, in turn making hundreds of downstream calls. This is typically referred to as a “call graph”, or “fanout” when considering all the many downstream calls. For example, any Profile page request fetches much more beyond just profile data including photos, connections, groups, subscription info, following info, long form blog posts, connection degrees from our graph, recommendations, etc. This call graph can be difficult to manage and was only getting more and more unruly.

We introduced the concept of a super block - groupings of backend services with a single access API. This allows us to have a specific team optimize the block, while keeping our call graph in check for each client.

Multi-Data Center

Being a global company with a fast growing member population, we needed to scale beyond serving traffic from one data center. We began an effort years ago to address this, first by serving public profiles out of two data centers (Los Angeles and Chicago). Once proven, we embarked on enhancing all our services to handle data replication, callbacks from different origins, one-way data replication events, and pinning users to a geographically close data center.

Many of our databases run on Espresso (a new in-house multi-tenant datastore). Espresso was built with multi data centers in mind. It provides master / master support and handles much of the difficult replication.

Multiple data centers are incredibly important to maintain “site-up” and high availability. You need to avoid any single point of failure not just for each individual service, but the entire site. Today, LinkedIn runs out of three main data centers, with additional PoPs around the globe.

LinkedIn's operational setup as of 2015 (circles represent data centers, diamonds represent PoPs)

What else have we done?

Of course, our scaling story is never this simple. There’s a countless number of things we’ve done over the years across all engineering and operations teams, including some of these larger initiatives:

Many of our most critical systems have their own rich history and evolution to address scale over the years. This includes our member graph service (our first service outside from Leo), search (our second service), news feed, communications platform, and member profile backend.

We’ve built data infrastructure that enables long term growth. This was first evident with Databus and Kafka, and has continued with Samza for data streams, Espresso and Voldemort for storage solutions, Pinot for our analytics systems, as well as other custom solutions. Plus, our tooling has improved such that developers can provision this infra automatically.

We’ve developed a massive offline workflow using Hadoop and our Voldemort data store to precompute data insights like People You May Know, Similar profiles, Notable Alumni, and profile browse maps.

We’ve rethought our frontend approach, adding client templates into the mix (Profile page, University pages). This enables more interactive applications, requiring our servers to send only JSON or partial JSON. Plus, templates get cached in CDNs and the browser. We also started to use BigPipe and the Play framework, changing our model from a threaded web server to a non-blocking asynchronous one.

Beyond the application code, we’ve introduced multiple tiers of proxies using Apache Traffic Server and HAProxy to handle load balancing, data center pinning, security, intelligent routing, server side rendering, and more.

And finally, we continue to improve the performance of our servers with optimized hardware, advanced memory and system tuning, and utilizing newer Java runtimes.

What’s next


LinkedIn continues to grow quickly and there’s still a ton of work we can do to improve. We’re working on problems that very few ever get to solve - come join us!

A Brief History of Scaling LinkedIn的更多相关文章

  1. [转载] LinkedIn架构这十年

    原文: http://colobu.com/2015/07/24/brief-history-scaling-linkedin/ 原文: A Brief History of Scaling Link ...

  2. LinkedIn架构这十年

    原文: A Brief History of Scaling LinkedIn 2003年是LinkedIn元年,公司成立的目标是连接你的个人人脉以获得更好的的工作机会.上线第一周才有2700个会员注 ...

  3. Enhancing the Scalability of Memcached

    原文地址: https://software.intel.com/en-us/articles/enhancing-the-scalability-of-memcached-0 1 Introduct ...

  4. [转载] 首席工程师揭秘:LinkedIn大数据后台是如何运作的?(一)

    本文作者:Jay Kreps,linkedin公司首席工程师:文章来自于他在linkedin上的分享:原文标题:The Log: What every software engineer should ...

  5. History API与浏览器历史堆栈管理

    移动端开发在某些场景中有着特殊需求,如为了提高用户体验和加快响应速度,常常在部分工程采用SPA架构.传统的单页应用基于url的hash值进行路由,这种实现不存在兼容性问题,但是缺点也有--针对不支持o ...

  6. 使用backbone的history管理SPA应用的url

    本文介绍如何使用backbone的history模块实现SPA应用里面的URL管理.SPA应用的核心在于使用无刷新的方式更改url,从而引发页面内容的改变.从实现上来看,url的管理和页面内容的管理是 ...

  7. 使用h5的history改善ajax列表请求体验

    信息比较丰富的网站通常会以分页显示,在点“下一页”时,很多网站都采用了动态请求的方式,避免页面刷新.虽然大家都是ajax,但是从一些小的细节还是 可以区分优劣.一个小的细节是能否支持浏览器“后退”和“ ...

  8. HTML5学习笔记之History API

    这系列文章主要是学习Html5相关的知识点,以学习API知识点为入口,由浅入深的引入实例,让大家一步一步的体会"h5"能够做什么,以及在实际项目中如何去合理的运用达到使用自如,完美 ...

  9. Linux下history命令用法

    如果你经常使用 Linux 命令行,那么使用 history(历史)命令可以有效地提升你的效率.本文将通过实例的方式向你介绍 history 命令的 15 个用法. 使用 HISTTIMEFORMAT ...

随机推荐

  1. NOIP2012 题解

    Vigenère 密码 这个名字实在打不来... 题解:模拟 #include <cstdio> #include <cstring> +; bool cj; int cl, ...

  2. IIS7.5支持html页面包含(include)html页面

    前提条件: ServerSideIncludeModule的安装: 在安装iis的时候选择上该服务(“在服务端包含文件”,选项)即可,如下: 1:处理映射程序 添加模块映射 请求路径 *.html 模 ...

  3. my sql

    如果改了上面的4个配置文件,要让其立即生效,可以使用如下方法 source .bash_profile . .bash_profile 基于Apache+php+mysql的许愿墙网站的搭建 方案一: ...

  4. linux samba 服务器 简单配置

    1. rpm -qa|grep samba 查看是否有samba 2.使用yum -y install samba 安装samba 服务 进入/etc/samba/ 使用vi smb.conf 修改配 ...

  5. Yaf零基础学习总结3-Hello Yaf

    Yaf零基础学习总结3-Hello Yaf 上一次我们已经学习了如何安装yaf了,准备工作做好了之后我们来开始实际的编码了,码农都知道一个经典的语句就是“Hello World”了,今天我们开始入手Y ...

  6. SQL Server并行死锁案例解析

    并行执行作为提升查询响应时间,提高用户体验的一种有效手段被大家所熟知,感兴趣的朋友可以看我以前的博客SQL Server优化技巧之SQL Server中的"MapReduce", ...

  7. Magicodes.WeiChat——多租户的设计与实现

    概要 多租户(Multi Tenancy/Tenant)是一种软件架构,其定义是:在一台服务器上运行单个应用实例,它为多个租户提供服务. 本框架使用的是共享数据库.共享 Schema.共享数据表的数据 ...

  8. 鸟瞰Nodejs

    一,基础. 1,Node的包管理器:npm; 安装node环境时会自动安装. 本地模式获取一个包:npm install [package_name] 此时包被安装到当前木的node_modules子 ...

  9. Dynamic CRM 2013学习笔记(七)追踪、监控及性能优化

    本文将介绍CRM的三个内容追踪.监控及性能优化.追踪是CRM里一个很有用的功能,它能为我们的CRM调试或解决错误.警告提供有价值的信息:我们可以用window的性能监控工具来了解CRM的性能状况:最后 ...

  10. Ember.js系列文章

    JS前端框架之Ember.js系列文章 本文为文章索引,主要是罗列Ember.js的相关文章便于阅读. 相关演示代码:github for free. 基础篇 1. EmberJs之What|Why| ...