Building a Service Mesh with HAProxy and Consul
转自:https://www.haproxy.com/blog/building-a-service-mesh-with-haproxy-and-consul/
HashiCorp added a service mesh feature to Consul, its service-discovery and distributed storage tool. In this post, you’ll see how HAProxy is the perfect fit as a data plane for this architecture.
HAProxy is no stranger to the service mesh scene. Its high performance, low resource usage, and flexible design allows it to be embedded within various types of service-oriented architectures. For example, when Airbnb needed to scale its infrastructure to support a growing number of distributed services, it developed SmartStack. SmartStack is a service mesh solution that relies on instances of HAProxy relaying communication between services. Using HAProxy allowed SmartStack to take advantage of advanced load-balancing algorithms, traffic queuing, connection retries, and built-in health checking.
HAProxy Technologies is working with HashiCorp to bring you a Consul service mesh that utilizes HAProxy as a data plane. This will allow you to deploy the world’s fastest and most widely used software load balancer as a sidecar proxy, enabling secure and reliable communication between all of your services.
In Consul 1.2, HashiCorp released Connect, which is a feature that allows you to turn an existing Consul cluster into a service mesh. If you’re familiar with Consul, you’ll know it for its distributed key/value storage and dynamic service discovery. With the addition of Connect, you can register sidecar proxies that are colocated with each of your services and relay traffic between them, creating a service mesh architecture. Best of all, it’s a pluggable framework that allows you to choose the underlying proxy layer to pair with it.
With Connect, HAProxy can be combined with the Consul service mesh too. This provides you with many advantages when designing your architecture. First, let’s take a step back and cover what a service mesh is and why it benefits you to use one.
Why Use a Service Mesh?
Why would you use a service mesh anyway? It comes down to the challenges of operating distributed systems. When your backend services are distributed across servers, departments, and data centers, executing the steps of a business processes often involves plenty of network communication. However, if you’ve ever had the responsibility of managing such a network, L. Peter Deutsch’s Fallacies of Distributed Computing will ring true. The fallacies are:
- The network is reliable.
- Latency is zero.
- Bandwidth is infinite.
- The network is secure.
- Topology doesn’t change.
- There is one administrator.
- Transport cost is zero.
- The network is homogeneous.
Obviously, the network is not always reliable. Topology does change. The network is not secure by default. How can you manage these risks? Ultimately, you want to apply mechanisms that addresses these issues without adding tons of complexity into every service that you own. In essence, each application shouldn’t need to be aware that a network exists at all.
That is why service meshes are becoming so popular. They abstract away the network from the application code. A service mesh is a design pattern where, instead of having services communicate directly to one another or directly to a message bus, requests are first passed to intermediary proxies that relay messages from one service to another.
The benefit is that the proxies can support functions for dealing with an unruly network that all services need, but which are more easily handled outside of the services themselves. For example, a proxy could retry a connection if it fails the first time and secure the communication with TLS encryption. You end up separating network functionality from application functionality.
The network-related problems that we mentioned before are mitigated by capabilities within HAProxy. For example, the solutions include:
- The network is not reliable = Retry logic for connecting to services
- Bandwidth is not infinite = Rate limiting to prioritize certain traffic or prevent overuse
- Topology changes = Consistent endpoints, such as always communicating with localhost, service discovery, and DNS resolution at run time
- The network is not secure = Truly end-to-end encryption with mutual TLS authentication
- There is never just one administrator = Authorizing which services are allowed to connect to which other services
- Transport cost is not zero = Observability of service traffic
Let’s see how the pieces fit together.
Control Plane and Data Plane
How does Consul relate to HAProxy? What are the responsibilities of each? Think of the whole thing like a real-time strategy video game. One of my favorites is Starcraft. In this game, you, the player, control an army of workers whose sole mission in life is to await their marching orders from you and then execute them. For example, you could tell them to go mine for minerals or explore a new area of the map and they’ll happily go off and do what’s been asked of them.
Compared to a service mesh, you, the player, represent Consul. Consul gives the marching orders to all of the proxies under its influence. It provides each one with the configuration details it needs: information about where other services can be found, which port to listen on for incoming requests, which TLS certificate to use for encryption, and which upstream services the local service depends on. In service mesh terminology, Consul is the control plane.
HAProxy, the Advanced Proxy
The proxies take on a more drone-like approach to life. You don’t need to configure each instance of HAProxy individually. They are responsible for querying the central Consul registry to get their configuration information on their own. This proxy layer is called the data plane. Of course, they are just as important as your video game workers. The proxy technology that you choose determines the capabilities your service mesh will have. Will it be able to encrypt communication? Will it have logic for reconnecting to a service if the first attempt fails?
HAProxy gives you features that a distributed architecture requires. It is the product of the hard-working open-source community and has become known as the fastest and most widely used software load balancer in the world. You get TLS encryption, connection retry logic, server persistence, rate limiting, authorization between services, and observability all out-of-the-box and in a lightweight process.
The Components
The key pieces of your service mesh will include the following:
- Your service
- A Consul agent, running locally, that your service registers with
- The proxy, which is registered as a sidecar
- A quorum of Consul agents that are in server mode, responsible for managing the service registry and configuration information
Let’s cover these components in more detail.
Your Service (aka your business logic)
Your service is at the heart of the design. It exposes functionality over a TCP port and, optionally, relies on other distributed services. The goal is to minimize the ways in which you need to change it in order to fit into the service mesh. Ideally, it should continue on as it always has, oblivious to the topology of the outside network. Consul gives you this separation of concerns.
Like in our video game analogy, let’s say that we’re talking about a service that mines for minerals. However, maybe it needs to talk to the map service to find out where to start working. Ordinarily, you would need to configure it with a known address or DNS name of the map service. If these settings ever changed, your mining service’s configuration also had to change to point to the new endpoint.
With Consul, service discovery allows you to query a central registry to find out where services live. However, with Connect, it gets even better. A local instance of HAProxy is created next to the service so that it’s listening on localhost. Then, your service always queries it as if it were the remote map service. This is known as a sidecar proxy: Each service gets its own local proxy and, through the relaying of messages, can reach other remote services as if they too were local. Now, you can point your local service’s configuration at localhost and never need to change that endpoint again.
Consul Agent
Also local to your service is a Consul agent. An agent is the Consul executable running in regular agent mode, as opposed to server mode. You register the local service with this agent by adding a JSON file to the /etc/consul.d folder. This gets synced to the server-mode agents.
Think of it like a walkie-talkie to the agents that are running in server mode, which hold the source of truth. The registration gets sent up and saved to the global registry. Then, when HAProxy wants to discover where it can find services on the network, it asks its local Consul agent and that agent pulls the information back down. Afterwards, it gives the answer to your proxy.

A locally run Consul agent is one of the key components of the service mesh
The same goes for the upstream services you’re calling. They each get their own local Consul agent. They register with it so that others can find them and they also tell Consul about any services that they, in turn, depend on. All of this information is used to configure the proxies so that in the end, all services talk to localhost and all communication becomes proxy-to-proxy.
The HAProxy Sidecar
Next to each service and Consul agent is a sidecar proxy, which is an instance of HAProxy. You don’t configure it directly. Instead, you install it with a specialized handler that queries the Consul agent to know which upstream services your local service depends on. Then, HAProxy sets up listeners on localhost so that your application can talk to the remote endpoints that it needs to, but without needing to know exactly where those endpoints live. In essence, you’re abstracting away the network.
A benefit to routing traffic through a local proxy is that the proxy can enforce fine-grained authorization rules. Consul lets you define intentions, which are rules that govern whether one service can talk to another. At runtime, HAProxy queries the local Consul agent to check if an incoming connection is allowed.
To review, your service registers with its local Consul agent information about itself and the upstream services on which it depends. That agent sends that information up to the Consul servers, which maintain the central registry. The local instance of HAProxy then asks the local Consul agent for configuration information and the agent pulls back the data that the Consul servers have compiled.
HAProxy then configures itself, listening for incoming requests to the local service and also for outgoing requests that the service makes to other services. The HAProxy handler continues to check for changes to service registrations and updates itself when needed.

Each service and Consul agent have a sidecar proxy, an instance of HAProxy
In the end, all service-to-service communication ends up going through proxies. You can see why it’s important to choose a proxy implementation that meets your needs. When it comes to speed and the right mix of features, HAProxy has a lot of benefits.
Server-Mode Agents
You’ve probably got a good idea about how the Consul agents that are running in server mode fit into this architecture. They maintain the service registry, which is the source of truth about where each service endpoint can be found. They also store the settings that each proxy needs to self-configure itself, such as TLS certificates. The Consul servers host the Consul HTTP API that local agents use to send and receive information about the cluster.
Agents in server mode must elect a leader using a consensus protocol that is based on the Raft algorithm. For that reason, you should dedicate an odd number of nodes, such as three, to participate in the quorum. These agents should reside on separate machines, if possible, so that your cluster has resiliency.
The Implementation
Baptiste Assmann presented a solution for integrating HAProxy with Consul at this year’s HashiConf conference in San Francisco.
During his presentation, he demonstrated using HAProxy as a sidecar that’s configured using information from the local Consul agent. It uses a Golang binary as the handler for configuring HAProxy. That will be available in Q1 of 2019.
In the meantime, he has reproduced its behavior using Bash. The HAProxy/Consul Github repository uses a Bash script to configure local instances of HAProxy. You can spin up this demo environment using Docker Compose. To get started, download the repository and then run the following commands:
cd blog/building_service_mesh | |
docker-compose up -d consul-server | |
# Wait for consul-server to bootstrap itself | |
sleep 10 | |
# Create ACL token that agents use to connect to consul-server | |
docker-compose exec consul-server curl --request PUT --header "X-Consul-Token: mastertoken" --data '{ "ID": "agenttoken", "Name": "Agent Token", "Type": "client", "Rules": "node \"\" { policy = \"write\" } service \"\" { policy = \"write\" }" }' http://localhost:8500/v1/acl/create | |
sleep 1 | |
# Create agents | |
docker-compose up -d www redis |
You will then be able to open a browser and go to http://localhost:8500/ui to see the Consul dashboard. You’ll need to go to the ACL screen first and enter the master token secret: “mastertoken”, which we’ve set in consul-server/consul.d/basic_config.

The Consul Dashboard
Notice how a *-sidecar-proxy service has been generated for the two services we’re creating, redis and www. The www app is a Node.js application that connects to redis via the service mesh. It’s able to connect to Redis on localhost and the connection is routed to the right place.
For this example, each service is hosted inside of a Docker container. Within each container is the service, a local Consul agent, a running instance of HAProxy, and a script called controller.sh that configures HAProxy on the fly. There’s also a Lua file, authorize.lua, that validates the connections between services by checking the client certificate passed with each request. The shell script and Lua file are the same for both services.
They have different start.sh files though, which Docker executes on container startup. The start.sh script installs the service, which is the Node.js application for www and Redis for the redis container, and then registers it with the local Consul agent. Here is the JSON registration for www:
{ | |
"service": { | |
"name": "${SERVICENAME}", | |
"port": 8080, | |
"address": "${MYIP}", | |
"connect": { | |
"sidecar_service": { | |
"proxy": { | |
"upstreams": [ | |
{ | |
"destination_name": "redis", | |
"destination_type": "connect", | |
"local_bind_address": "127.0.0.1", | |
"local_bind_port": 6379 | |
}], | |
"config": { | |
"unsecured_bind_port": 21002, | |
"local_service_mode": "http" | |
} | |
} | |
} | |
} | |
}, | |
"acl_datacenter":"dc1", | |
"acl_default_policy":"deny", | |
"acl_down_policy":"extend-cache", | |
"acl_token":"agenttoken" | |
} |
The Node.js app listens on port 8080 locally, but is exposed through the service mesh on an automatically assigned port chosen by Consul. There’s a dependency on the upstream Redis service. HAProxy binds it locally to port 6379, but proxies it to the Redis container on a port chosen by Consul. In this way, the Node.js app can access Redis at 127.0.0.1:6379.
Also note that a configuration parameter called unsecured_bind_port allows you to access the app from outside of the service mesh. So, you can go to http://localhost:21002 on the machine where you’re running Docker. Here’s a screenshot of what it looks like:

Your app can be accessed from outside of the service mesh
In the example code, we’ve enabled Consul ACLs with a default deny rule. Before the www app can connect to Redis, you must add an Intention. Intentions are rules that allow or deny connections between services. You could allow the Node.js app to connect to Redis by adding a new Intention:

Adding new Intentions allows the Node.js app to connect to Redis
Now, the app will succeed when it tries to read from or write to Redis:

By adding new Intentions the app can successfully read from or write to Redis
After you remove the Intention, it will fail again due to the default deny set up by the ACL.
You can also see the HAProxy Stats page for the app by going to http://localhost:1936. This is great for troubleshooting connection problems.
To extend this example, add more containers, patterning them off of the given Dockerfiles. Then, update your start.sh file to install your application and register it with Consul. Last, add the new service to the docker-compose.yml file.
Conclusion
Consul’s Connect feature enables you to transform a Consul cluster into a service mesh. Connect is a pluggable framework that allows you to choose the proxy technology that fits your needs best.
In this blog post, you learned how HAProxy can be selected as the data plane, giving you access to features like TLS encryption, connection retry logic, server persistence, rate limiting, authorization between services, and observability. I hope you’re as excited as we are about the possibilities that this creates! In the coming months, we will be releasing more information about our integration with Consul and the new Golang implementation.
Building a Service Mesh with HAProxy and Consul的更多相关文章
- consul 1.2 支持service mesh
主要说明: This release supports a major new feature called Connect that automatically turns any existing ...
- Java进阶专题(二十八) Service Mesh初体验
前言 ⽬前,微服务的架构⽅式在企业中得到了极⼤的发展,主要原因是其解决了传统的单体架构中存在的问题.当单体架构拆分成微服务架构就可以⾼枕⽆忧了吗? 显然不是的.微服务架构体系中同样也存在很多的挑战 ...
- Istio入门实战与架构原理——使用Docker Compose搭建Service Mesh
本文将介绍如何使用Docker Compose搭建Istio.Istio号称支持多种平台(不仅仅Kubernetes).然而,官网上非基于Kubernetes的教程仿佛不是亲儿子,写得非常随便,不仅缺 ...
- Service Mesh服务网格新生代——Istio
Istio 是什么?使用云平台可以为组织提供丰富的好处.然而,不可否认的是,采用云可能会给 DevOps 团队带来压力.开发人员必须使用微服务已满足应用的可移植性,同时运营商管理了极其庞大的混合和多云 ...
- What’s a service mesh? And why do I need one?
https://buoyant.io/2017/04/25/whats-a-service-mesh-and-why-do-i-need-one/ Update 2018-02-06: Since t ...
- Qcon2017实录|Service Mesh:下一代微服务
https://zhuanlan.zhihu.com/p/30292372 数人云11月Meetup报名开启,看中西方大神如何论道云原生与微服务!本文作者敖小剑老师将在本次Meetup上继续分享Ser ...
- Service Mesh服务网格:是什么和为什么
Service Mesh服务网格:是什么和为什么 - 好雨云帮 CSDN 博客 - CSDN博客 https://blog.csdn.net/zyqduron/article/details/8043 ...
- Service Mesh服务网格新生代--Istio(转)
万字解读:Service Mesh服务网格新生代--Istio 官网地址:https://preliminary.istio.io/zh/docs/concepts/security/ Servic ...
- 什么是 Service Mesh
作者|敖小剑 微服务方兴未艾如火如荼之际,在 spring cloud 等经典框架之外,Service Mesh 技术正在悄然兴起.到底什么是 Service Mesh,它的出现能带来什么,又能改变什 ...
随机推荐
- Stack实现
栈的三种操作算法很简单 STACK-EMPTY(S) 1 if S.top == 0 2 return TRUE 3 else return FALSE PUSH(S, x) 1 S.top = ...
- php的json_encode第二个参数学习及应用
php5.4以上: json_encode($data, JSON_FORCE_OBJECT | JSON_UNESCAPED_SLASHES | JSON_UNESCAPED_UNICODE); p ...
- 【题解】Luogu P5400 [CTS2019]随机立方体
原题传送门 毒瘤计数题 我们设\(dp_i\)表示至少有\(i\)个极大数字的概率,\(ans_i\)表示恰好有\(i\)个极大数的概率,\(mi=Min(n,m,l)\) 易知: \[dp_i=\s ...
- ServletContextInitializer添加 servlet filter listener
ServletContextInitializer添加 servlet filter listener https://www.cnblogs.com/pomer-huang/p/9639322.ht ...
- .net Dapper 实践系列(2) ---事务添加(Layui+Ajax+Dapper+MySQL)
目录 写在前面 问题描述 解决方法 具体实现 写在前面 前面我们已经搭建好了项目,这一小节我们使用Dapper 中的事务实现一对多的添加操作. 问题描述 在做添加的时候很头疼需要从页面传递一组数据到后 ...
- StopWatch方法详解
namespace System.Diagnostics { // // 摘要: // 提供一组方法和属性,可用于准确地测量运行时间. public class Stopwatch { // // 摘 ...
- 【面试突击】-Redis常见面试题(一)
介绍:Redis 是一个开源的使用 ANSI C 语言编写.遵守 BSD 协议.支持网络.可基于内存亦可持久化的日志型.Key-Value 数据库,并提供多种语言的 API的非关系型数据库. 传统数据 ...
- 又一个js乱码的秘密alert放在js文件里中文乱码,可是放在HTML里显示中文就很好
用文本文档打开你的js文件,文件-另存为,编码更改为UTF-8保存. 回复 | PFly | 园豆:94 (初学一级) | 2017-07-17 21:32 显示结果中文乱码 支持(0)反对(0)回复 ...
- angular http interceptors 拦截器使用分享
拦截器 在开始创建拦截器之前,一定要了解 $q和延期承诺api 出于全局错误处理,身份验证或请求的任何同步或异步预处理或响应的后处理目的,希望能够在将请求移交给服务器之前拦截请求,并在将请求移交给服务 ...
- JavaWeb 之 EL表达式
EL 表达式 一.概述 1.概念 EL 表达式:Expression Language 表达式语言. 2.作用 替换和简化 jsp 页面中 java 代码的编写. 3.语法格式 ${表达式} 4.注意 ...