第十八章 dubbo-monitor计数监控

监控总体图：

红色：监控中心 -  dubbo-simple-monitor

黄色：provider

蓝色：consumer

统计总体流程：

• MonitorFilter向DubboMonitor发送数据
• DubboMonitor将数据进行聚合后（默认聚合1min中的统计数据）暂存到ConcurrentMap<Statistics, AtomicReference<long[]>> statisticsMap，然后使用一个含有3个线程（线程名字：DubboMonitorSendTimer）的线程池每隔1min钟，调用SimpleMonitorService遍历发送statisticsMap中的统计数据，每发送完毕一个，就重置当前的Statistics的AtomicReference<long[]>
• SimpleMonitorService将这些聚合数据塞入BlockingQueue<URL> queue中（队列大写为100000）
• SimpleMonitorService使用一个后台线程（线程名为：DubboMonitorAsyncWriteLogThread）将queue中的数据写入文件（该线程以死循环的形式来写）
• SimpleMonitorService还会使用一个含有1个线程（线程名字：DubboMonitorTimer）的线程池每隔5min钟，将文件中的统计数据画成图表

注意：

• SimpleMonitorService理解为一个服务提供者；而provider和consumer都是一个服务消费者，所以二者的DubboMonitor中的MonitorService实例都是一个代理实例。
• dubbo-monitor计数监控不支持异步调用下的数据监控

一、dubbo-monitor使用

在配置文件中添加：

 <dubbo:monitor address="10.211.55.5:9090" />

即开启了monitor监控，并且指定了监控中心服务器为“10.211.55.5:9090”。

9090端口是Prometheus的默认端口，dubbo提供的监控中心比较简陋，我们后续会使用Prometheus作为监控中心来存储监控数据。

二、服务端加载monitor配置

doExportUrlsFor1Protocol(ProtocolConfig protocolConfig, List<URL> registryURLs)中：

             if (!Constants.SCOPE_LOCAL.toString().equalsIgnoreCase(scope)) {
                 ...
                 if (registryURLs != null && registryURLs.size() > 0) {
                     for (URL registryURL : registryURLs) {
                         ...
                         URL monitorUrl = loadMonitor(registryURL);
                         if (monitorUrl != null) {
                             url = url.addParameterAndEncoded(Constants.MONITOR_KEY, monitorUrl.toFullString());
                         }
                        ...
                     }
                 } else {
                     ...
                 }
             }

其中loadMonitor(URL registryURL)方法主要用于创建MonitorConfig对象（如果monitor配置在dubbo.properties中的话），并且设置属性，之后设置到数据总线Url中。

     protected URL loadMonitor(URL registryURL) {
         if (monitor == null) {
             String monitorAddress = ConfigUtils.getProperty("dubbo.monitor.address");
             String monitorProtocol = ConfigUtils.getProperty("dubbo.monitor.protocol");
             if ((monitorAddress == null || monitorAddress.length() == 0) && (monitorProtocol == null || monitorProtocol.length() == 0)) {
                 return null;
             }

             monitor = new MonitorConfig();
             if (monitorAddress != null && monitorAddress.length() > 0) {
                 monitor.setAddress(monitorAddress);
             }
             if (monitorProtocol != null && monitorProtocol.length() > 0) {
                 monitor.setProtocol(monitorProtocol);
             }
         }
         appendProperties(monitor);
         ...
     }

三、消费端加载monitor配置

createProxy(Map<String, String> map)中：

                 List<URL> us = loadRegistries(false);
                 if (us != null && us.size() > 0) {
                     for (URL u : us) {
                         URL monitorUrl = loadMonitor(u);
                         if (monitorUrl != null) {
                             map.put(Constants.MONITOR_KEY, URL.encode(monitorUrl.toFullString()));
                         }
                         ...
                     }
                 }

四、MonitorFilter收集监控数据

consumer端在发起调用之前会先走filter链；provider端在接收到请求时也是先走filter链，然后才进行真正的业务逻辑处理。默认情况下，在consumer和provider的filter链中都会有Monitorfilter。

 /**
  * MonitorFilter. (SPI, Singleton, ThreadSafe)
  */
 @Activate(group = {Constants.PROVIDER, Constants.CONSUMER})
 public class MonitorFilter implements Filter {

     private static final Logger logger = LoggerFactory.getLogger(MonitorFilter.class);

     // key: 接口名.方法名 value: 当前的并发数
     private final ConcurrentMap<String, AtomicInteger> concurrents = new ConcurrentHashMap<String, AtomicInteger>();

     private MonitorFactory monitorFactory;// MonitorFactory$Adaptive

     public void setMonitorFactory(MonitorFactory monitorFactory) {
         this.monitorFactory = monitorFactory;
     }

     // intercepting invocation
     public Result invoke(Invoker<?> invoker, Invocation invocation) throws RpcException {
         if (invoker.getUrl().hasParameter(Constants.MONITOR_KEY)) {// 开启了monitor监控
             RpcContext context = RpcContext.getContext(); // provider must fetch context before invoke() gets called
             String remoteHost = context.getRemoteHost();
             long start = System.currentTimeMillis(); // record start timestamp
             getConcurrent(invoker, invocation).incrementAndGet(); // 并发数+1
             try {
                 Result result = invoker.invoke(invocation); // proceed invocation chain
                 collect(invoker, invocation, result, remoteHost, start, false);// 收集统计数据
                 return result;
             } catch (RpcException e) {
                 collect(invoker, invocation, null, remoteHost, start, true);// 发生异常时收集统计数据
                 throw e;
             } finally {
                 getConcurrent(invoker, invocation).decrementAndGet(); // 并发数-1
             }
         } else {
             return invoker.invoke(invocation);
         }
     }

     // collect info
     private void collect(Invoker<?> invoker, Invocation invocation, Result result, String remoteHost, long start, boolean error) {
         try {
             // ---- service statistics ----
             long elapsed = System.currentTimeMillis() - start; // 此次调用花费的时间
             int concurrent = getConcurrent(invoker, invocation).get(); // current concurrent count
             String application = invoker.getUrl().getParameter(Constants.APPLICATION_KEY);
             String service = invoker.getInterface().getName(); // service name
             String method = RpcUtils.getMethodName(invocation); // method name
             URL url = invoker.getUrl().getUrlParameter(Constants.MONITOR_KEY);
             Monitor monitor = monitorFactory.getMonitor(url);//根据monitorUrl获取Monitor实现（默认使用DubboMonitor）
             if (monitor == null) {
                 return;
             }
             int localPort;
             String remoteKey;
             String remoteValue;
             if (Constants.CONSUMER_SIDE.equals(invoker.getUrl().getParameter(Constants.SIDE_KEY))) {
                 // ---- for service consumer ----
                 localPort = 0;
                 remoteKey = MonitorService.PROVIDER;
                 remoteValue = invoker.getUrl().getAddress();
             } else {
                 // ---- for service provider ----
                 localPort = invoker.getUrl().getPort();
                 remoteKey = MonitorService.CONSUMER;
                 remoteValue = remoteHost;
             }
             String input = "", output = "";
             if (invocation.getAttachment(Constants.INPUT_KEY) != null) {
                 input = invocation.getAttachment(Constants.INPUT_KEY);
             }
             if (result != null && result.getAttachment(Constants.OUTPUT_KEY) != null) {
                 output = result.getAttachment(Constants.OUTPUT_KEY);
             }
             monitor.collect(new URL(Constants.COUNT_PROTOCOL,
                     NetUtils.getLocalHost(), localPort,
                     service + "/" + method,
                     MonitorService.APPLICATION, application,
                     MonitorService.INTERFACE, service,
                     MonitorService.METHOD, method,
                     remoteKey, remoteValue,
                     error ? MonitorService.FAILURE : MonitorService.SUCCESS, "1",// 成功失败数
                     MonitorService.ELAPSED, String.valueOf(elapsed),// 调用消耗的时间
                     MonitorService.CONCURRENT, String.valueOf(concurrent),// 并发数
                     Constants.INPUT_KEY, input,
                     Constants.OUTPUT_KEY, output));
         } catch (Throwable t) {
             logger.error("Failed to monitor count service " + invoker.getUrl() + ", cause: " + t.getMessage(), t);
         }
     }

     // concurrent counter
     private AtomicInteger getConcurrent(Invoker<?> invoker, Invocation invocation) {
         String key = invoker.getInterface().getName() + "." + invocation.getMethodName();
         AtomicInteger concurrent = concurrents.get(key);
         if (concurrent == null) {
             concurrents.putIfAbsent(key, new AtomicInteger());
             concurrent = concurrents.get(key);
         }
         return concurrent;
     }

 }

调用之前，记录调用开始时间、并发数，之后进行调用，最后进行统计数据收集：

• 获取计算各种统计数据（调用消耗时间、调用成功/错误数等）
• 使用MonitorFactory获取Monitor
• 将统计数据构造成url
• 使用Monitor收集这些统计数据

获取Monitor的源码后续再说。这里获取到的是DubboMonitor实例。

五、DubboMonitor聚合监控数据

     private static final int LENGTH = 10;
     private final ConcurrentMap<Statistics, AtomicReference<long[]>> statisticsMap = new ConcurrentHashMap<Statistics, AtomicReference<long[]>>();

     // 聚合统计数据
     public void collect(URL url) {
         // data to collect from url
         int success = url.getParameter(MonitorService.SUCCESS, 0);
         int failure = url.getParameter(MonitorService.FAILURE, 0);
         int input = url.getParameter(MonitorService.INPUT, 0);
         int output = url.getParameter(MonitorService.OUTPUT, 0);
         int elapsed = url.getParameter(MonitorService.ELAPSED, 0);
         int concurrent = url.getParameter(MonitorService.CONCURRENT, 0);
         // init atomic reference
         Statistics statistics = new Statistics(url);
         AtomicReference<long[]> reference = statisticsMap.get(statistics);
         if (reference == null) {
             statisticsMap.putIfAbsent(statistics, new AtomicReference<long[]>());
             reference = statisticsMap.get(statistics);
         }
         // use CompareAndSet to sum
         long[] current;
         long[] update = new long[LENGTH];
         do {
             current = reference.get();
             if (current == null) {
                 update[0] = success;
                 update[1] = failure;
                 update[2] = input;
                 update[3] = output;
                 update[4] = elapsed;
                 update[5] = concurrent;
                 update[6] = input;
                 update[7] = output;
                 update[8] = elapsed;
                 update[9] = concurrent;
             } else {
                 update[0] = current[0] + success;
                 update[1] = current[1] + failure;
                 update[2] = current[2] + input;
                 update[3] = current[3] + output;
                 update[4] = current[4] + elapsed;
                 update[5] = (current[5] + concurrent) / 2;
                 update[6] = current[6] > input ? current[6] : input;
                 update[7] = current[7] > output ? current[7] : output;
                 update[8] = current[8] > elapsed ? current[8] : elapsed;
                 update[9] = current[9] > concurrent ? current[9] : concurrent;
             }
         } while (!reference.compareAndSet(current, update));
     }

实际上这里聚合了1min钟的统计数据到statisticsMap中。

六、Monitor使用MonitorService存储数据到队列

     private final ScheduledExecutorService scheduledExecutorService = Executors.newScheduledThreadPool(3, new NamedThreadFactory("DubboMonitorSendTimer", true));
     private final ScheduledFuture<?> sendFuture;
     private final long monitorInterval;

     public DubboMonitor(Invoker<MonitorService> monitorInvoker, MonitorService monitorService) {
         this.monitorInvoker = monitorInvoker;
         this.monitorService = monitorService;
         this.monitorInterval = monitorInvoker.getUrl().getPositiveParameter("interval", );
         // collect timer for collecting statistics data
         sendFuture = scheduledExecutorService.scheduleWithFixedDelay(new Runnable() {
             public void run() {
                 // collect data
                 try {
                     send();
                 } catch (Throwable t) {
                     logger.error("Unexpected error occur at send statistic, cause: " + t.getMessage(), t);
                 }
             }
         }, monitorInterval, monitorInterval, TimeUnit.MILLISECONDS);
     }

     public void send() {
         if (logger.isInfoEnabled()) {
             logger.info("Send statistics to monitor " + getUrl());
         }
         String timestamp = String.valueOf(System.currentTimeMillis());
         for (Map.Entry<Statistics, AtomicReference<long[]>> entry : statisticsMap.entrySet()) {
             // get statistics data
             Statistics statistics = entry.getKey();
             AtomicReference<long[]> reference = entry.getValue();
             long[] numbers = reference.get();
             long success = numbers[0];
             long failure = numbers[1];
             long input = numbers[2];
             long output = numbers[3];
             long elapsed = numbers[4];
             long concurrent = numbers[5];
             long maxInput = numbers[6];
             long maxOutput = numbers[7];
             long maxElapsed = numbers[8];
             long maxConcurrent = numbers[9];

             // send statistics data
             URL url = statistics.getUrl()
                     .addParameters(MonitorService.TIMESTAMP, timestamp,
                             MonitorService.SUCCESS, String.valueOf(success),
                             MonitorService.FAILURE, String.valueOf(failure),
                             MonitorService.INPUT, String.valueOf(input),
                             MonitorService.OUTPUT, String.valueOf(output),
                             MonitorService.ELAPSED, String.valueOf(elapsed),
                             MonitorService.CONCURRENT, String.valueOf(concurrent),
                             MonitorService.MAX_INPUT, String.valueOf(maxInput),
                             MonitorService.MAX_OUTPUT, String.valueOf(maxOutput),
                             MonitorService.MAX_ELAPSED, String.valueOf(maxElapsed),
                             MonitorService.MAX_CONCURRENT, String.valueOf(maxConcurrent)
                     );
             monitorService.collect(url);

             // reset
             long[] current;
             long[] update = new long[LENGTH];
             do {
                 current = reference.get();
                 if (current == null) {
                     update[0] = 0;
                     update[1] = 0;
                     update[2] = 0;
                     update[3] = 0;
                     update[4] = 0;
                     update[5] = 0;
                 } else {
                     update[0] = current[0] - success;
                     update[1] = current[1] - failure;
                     update[2] = current[2] - input;
                     update[3] = current[3] - output;
                     update[4] = current[4] - elapsed;
                     update[5] = current[5] - concurrent;
                 }
             } while (!reference.compareAndSet(current, update));
         }
     }

• 首先从聚合数据存储器statisticsMap中获取相关统计数据并存储到数据总线Url中
• 之后调用MonitorService（这里是SimpleMonitorService），将统计数据存储到一个BlockingQueue中

注意：这里有一个改进点

• 由于monitorService.collect(url)是远程调用，这里在for循环体中执行远程调用，实际上是不合适的，我们可以将所有的url先暂存在一个List<URL>中，最后，使用一次monitorService.collect(urlList)即可 - 此时，可适当缩短数据发送时间。

SimpleMonitorService：

     private final BlockingQueue<URL> queue;

     public void collect(URL statistics) {
         queue.offer(statistics);
         if (logger.isInfoEnabled()) {
             logger.info("collect statistics: " + statistics);
         }
     }

七、MonitorService将数据写入本地文件

     private final ScheduledExecutorService scheduledExecutorService = Executors.newScheduledThreadPool(1, new NamedThreadFactory("DubboMonitorTimer", true));
     private final ScheduledFuture<?> chartFuture;
     private final Thread writeThread;
     private final BlockingQueue<URL> queue;
     private String statisticsDirectory = "statistics";
     private String chartsDirectory = "charts";
     private volatile boolean running = true;

     public SimpleMonitorService() {
         queue = new LinkedBlockingQueue<URL>(Integer.parseInt(ConfigUtils.getProperty("dubbo.monitor.queue", "")));
         writeThread = new Thread(new Runnable() {
             public void run() {
                 while (running) {
                     try {
                         write(); // write statistics
                     } catch (Throwable t) {
                         logger.error("Unexpected error occur at write stat log, cause: " + t.getMessage(), t);
                         try {
                             Thread.sleep(5000); // retry after 5 secs
                         } catch (Throwable t2) {
                         }
                     }
                 }
             }
         });
         writeThread.setDaemon(true);
         writeThread.setName("DubboMonitorAsyncWriteLogThread");
         writeThread.start();
         chartFuture = scheduledExecutorService.scheduleWithFixedDelay(new Runnable() {
             public void run() {
                 try {
                     draw(); // draw chart
                 } catch (Throwable t) {
                     logger.error("Unexpected error occur at draw stat chart, cause: " + t.getMessage(), t);
                 }
             }
         }, 1, , TimeUnit.SECONDS);
         statisticsDirectory = ConfigUtils.getProperty("dubbo.statistics.directory");
         chartsDirectory = ConfigUtils.getProperty("dubbo.charts.directory");
     }

write()将统计数据写入文件，draw()将统计数据画成图片。这两种方式在实际使用中都不会用到。

最后来看一下获取Monitor实例的过程（帮助我们开发自定义的Monitor）：

 Monitor monitor = monitorFactory.getMonitor(url);

MonitorFilter中的monitorFactory实例是：MonitorFactory$Adaptive。

 package com.alibaba.dubbo.monitor;

 import com.alibaba.dubbo.common.extension.ExtensionLoader;

 public class MonitorFactory$Adaptive implements com.alibaba.dubbo.monitor.MonitorFactory {
     public com.alibaba.dubbo.monitor.Monitor getMonitor(com.alibaba.dubbo.common.URL arg0) {
         if (arg0 == null) throw new IllegalArgumentException("url == null");
         com.alibaba.dubbo.common.URL url = arg0;
         String extName = (url.getProtocol() == null ? "dubbo" : url.getProtocol());
         if (extName == null)
             throw new IllegalStateException("Fail to get extension(com.alibaba.dubbo.monitor.MonitorFactory) name from url(" + url.toString() + ") use keys([protocol])");
         com.alibaba.dubbo.monitor.MonitorFactory extension = (com.alibaba.dubbo.monitor.MonitorFactory) ExtensionLoader.getExtensionLoader(com.alibaba.dubbo.monitor.MonitorFactory.class).getExtension(extName);
         return extension.getMonitor(arg0);
     }
 }

首先，根据输入的url中的protocol来获取相关的MonitorFactory（这里protocol默认是dubbo，所以获取的是DubboMonitorFactory，可以通过指定dubbo:monitor标签中的protocol属性来自定义获取XxxMonitorFactory），之后调用DubboMonitorFactory.getMonitor(arg0)。该方法在其父类AbstractMonitorFactory中：

 /**
  * AbstractMonitorFactory. (SPI, Singleton, ThreadSafe)
  */
 public abstract class AbstractMonitorFactory implements MonitorFactory {
     private static final Logger logger = LoggerFactory.getLogger(AbstractMonitorFactory.class);

     // lock for getting monitor center
     private static final ReentrantLock LOCK = new ReentrantLock();

     // monitor centers Map<RegistryAddress, Registry>
     private static final Map<String, Monitor> MONITORS = new ConcurrentHashMap<String, Monitor>();

     private static final Map<String, ListenableFuture<Monitor>> FUTURES = new ConcurrentHashMap<String, ListenableFuture<Monitor>>();

     private static final ExecutorService executor = new ThreadPoolExecutor(0, 10, 60L, TimeUnit.SECONDS, new SynchronousQueue<Runnable>(), new NamedThreadFactory("DubboMonitorCreator", true));

     public static Collection<Monitor> getMonitors() {
         return Collections.unmodifiableCollection(MONITORS.values());
     }

     public Monitor getMonitor(URL url) {
         url = url.setPath(MonitorService.class.getName()).addParameter(Constants.INTERFACE_KEY, MonitorService.class.getName());
         String key = url.toServiceStringWithoutResolving();// dubbo://10.211.55.5:9090/com.alibaba.dubbo.monitor.MonitorService
         Monitor monitor = MONITORS.get(key);
         Future<Monitor> future = FUTURES.get(key);
         if (monitor != null || future != null) {
             return monitor;
         }

         LOCK.lock();
         try {
             monitor = MONITORS.get(key);
             future = FUTURES.get(key);
             if (monitor != null || future != null) {
                 return monitor;
             }

             final URL monitorUrl = url;
             // 使用另外的线程MonitorCreator来创建Monitor实例（原因是：即使Monitor创建失败，也不会影响主流程）
             final ListenableFutureTask<Monitor> listenableFutureTask = ListenableFutureTask.create(new MonitorCreator(monitorUrl));
41             listenableFutureTask.addListener(new MonitorListener(key));
42             executor.execute(listenableFutureTask);
             FUTURES.put(key, listenableFutureTask);

             return null;
         } finally {
             // unlock
             LOCK.unlock();
         }
     }

     protected abstract Monitor createMonitor(URL url);

     // Callable和Runnable一样，也是创建一个线程去执行，只是Callable有返回值（T call()），而Runnable无返回值（void run()）
     class MonitorCreator implements Callable<Monitor> {

         private URL url;

         public MonitorCreator(URL url) {
             this.url = url;
         }

         @Override
         public Monitor call() throws Exception {
             Monitor monitor = AbstractMonitorFactory.this.createMonitor(url);//调用子类的createMonitor方法创建Monitor
             return monitor;
         }
     }

     class MonitorListener implements Runnable {

         private String key;

         public MonitorListener(String key) {
             this.key = key;
         }
         // listenableFutureTask一旦isDone()完成（正常完成、抛出异常、被中断等），就会立即执行该方法
         @Override
         public void run() {
             try {
                 ListenableFuture<Monitor> listenableFuture = AbstractMonitorFactory.FUTURES.get(key);
                 AbstractMonitorFactory.MONITORS.put(key, listenableFuture.get());
                 AbstractMonitorFactory.FUTURES.remove(key);
             } catch (InterruptedException e) {
                 logger.warn("Thread was interrupted unexpectedly, monitor will never be got.");
                 AbstractMonitorFactory.FUTURES.remove(key);
             } catch (ExecutionException e) {
                 logger.warn("Create monitor failed, monitor data will not be collected until you fix this problem. ", e);
             }
         }
     }
 }

来看DubboMonitorFactory.createMonitor(url)：

 /**
  * DefaultMonitorFactory
  */
 public class DubboMonitorFactory extends AbstractMonitorFactory {

     private Protocol protocol;

     private ProxyFactory proxyFactory;

     public void setProtocol(Protocol protocol) {
         this.protocol = protocol;
     }

     public void setProxyFactory(ProxyFactory proxyFactory) {
         this.proxyFactory = proxyFactory;
     }

     @Override
     protected Monitor createMonitor(URL url) {
         url = url.setProtocol(url.getParameter(Constants.PROTOCOL_KEY, "dubbo"));
         if (url.getPath() == null || url.getPath().length() == 0) {
             url = url.setPath(MonitorService.class.getName());
         }
         String filter = url.getParameter(Constants.REFERENCE_FILTER_KEY);
         if (filter == null || filter.length() == 0) {
             filter = "";
         } else {
             filter = filter + ",";
         }
         url = url.addParameters(Constants.CLUSTER_KEY, "failsafe", Constants.CHECK_KEY, String.valueOf(false),
                 Constants.REFERENCE_FILTER_KEY, filter + "-monitor");
         // 创建Invoker<MonitorService>，内部会构造与MonitorService实现类SimpleMonitorService所在的检测中心dubbo-simple-monitor的长连接
         Invoker<MonitorService> monitorInvoker = protocol.refer(MonitorService.class, url);
         // 获取MonitorService的代理
         MonitorService monitorService = proxyFactory.getProxy(monitorInvoker);
         return new DubboMonitor(monitorInvoker, monitorService);
     }

 }

注意：这里的SimpleMonitorService其实会部署在dubbo-simple-monitor中，被provider和consumer中的DubboMonitor调用。