源文地址

(September 2008)

For the last couple of years, I've been working on European Space Agency (ESA) projects - writing rather complex code generators. In the ESA project I am currently working on, I am also the technical lead; and I recently faced the need to (quickly) provide real-time plotting of streaming data. Being a firm believer in open-source, after a little Googling I found Gnuplot. From my (somewhat limited) viewpoint, Gnuplot appears to be the LaTEX equivalent in the world of graphs: amazing functionality that is also easily accessible. Equally important, Gnuplot follows the powerful paradigm that UNIX established: it comes with an easy to use scripting language, thus allowing its users to prescribe actions and "glue" Gnuplot together with other applications - and form powerful combinations.

To that end, I humbly submit a little creation of mine: a Perl script that spawns instances of Gnuplot and plots streaming data in real-time.


Plotting data in real-time

Interfacing over standard input

My coding experience has taught me to strive for minimal and complete interfaces: to that end, the script plots data that will arrive over the standard input, one sample per line. The samples are just numbers (integers / floating point numbers), and must be prefixed with the stream number ("0:", "1:", etc). Each plot window will also be configured to display a specific number of samples.

The resulting script is relatively simple - and easy to use:

bash ./driveGnuPlots.pl

Usage: ./driveGnuPlots.pl <options>
where options are (in order): NumberOfStreams How many streams to plot (windows)
Stream1_WindowSampleSize <Stream2...> This many window samples for each stream
Stream1_Title <Stream2_Title> ... Title used for each stream
(Optional) Stream1_geometry <...>. Sizes and positions in pixels The last parameters (the optionally provided geometries of the gnuplot windows)
are of the form:
WIDTHxHEIGHT+XOFF+YOFF

Note that the script uses the "autoscale" feature of GnuPlot, to automatically adapt to the incoming value ranges.

An example usage scenario: plotting sine and cosine

Let's say we want to see a sine and a cosine run side-by-side, in real-time. We also want to watch the cosine "zooming-in" by 10x (time-scale wise). The following code will print our test samples:

#!/usr/bin/perl -w
use strict; use Time::HiRes qw/sleep/; # First, set the standard output to auto-flush
select((select(STDOUT), $| = 1)[0]); # And loop 5000 times, printing values...
my $offset = 0.0;
while(1) {
print "0:".sin($offset)."\n";
print "1:".cos($offset)."\n";
$offset += 0.1;
if ($offset > 500) {
last;
}
sleep(0.02);
}

We'll use this code to test our plotting script: the data for two streams (sine and cosine) are printed in the expected format: one sample (one number) printed per line. To distinguish between the two streams, the sample is prefixed with "0:", "1:", etc. Notice that we explicitly set the autoflush flag for our standard output: we need the data output to be unbuffered, otherwise our plotting script will receive data in bursts (when the data are flushed from the producer), and the plots will "jerk" forward.

This is how we test the plotting script (assuming we saved the sample code above in sinuses.pl): <

bash$ ./sinuses.pl | ./driveGnuPlots.pl 2 50 500 "Sine" "Cosine"

To stop the plotting, use Ctrl-C on the terminal you spawned from.

The parameters we passed to driveGnuPlots.pl are:

  • 2 is the number of streams
  • The window for the first stream (sine) will be 50 samples wide
  • The window for the second stream (cosine) will be 500 samples wide (hence the different "zoom" factor)
  • The titles of the two streams follow

When executed, the script spawns one gnuplot per each stream, and displays the graphs in a clear, flicker-free manner. If you don't like the Gnuplot settings I used (e.g. the grid, or the colors, or...) feel free to change them: the setup code that defines the plotting parameters starts at line 82 of the script.

Executive summary: plotting streaming data is now as simple as selecting them out from your "producer" program (filtering its standard output through any means you wish: grep, sed, awk, etc), and outputing them, one number per line. Just remember to prefix with the stream number ("0:", "1:", etc, to allow for multiple streams), and make sure you flush your standard output, e.g.

For this kind of output:

    bash$ /path/to/programName
...(other stuff)
Measure: 7987.3
...(other stuff)
Measure: 8364.4
Measure: 8128.1
... You would do this: bash$ /path/to/programName | \
grep --line-buffered '^Measure:' | \
awk -F: '{printf("0:%f\n", $2); fflush();}' | \
driveGnuPlots.pl 1 50 "My data"

In the code above, grep filters out the lines that start with "Measure:", and awk selects the 2nd column ($2) and prefixes it with "0:" (since this is the 1st - and only, in this example - stream we will display). Notice that we used the proper options to force the standard output's flushing for both grep (--line-buffered) and awk (fflush() called).

Preparing for a demo

You don't want to move the GnuPlot windows after they are shown, do you? So you can just specify their placement, in "WIDTHxHEIGHT+XOFF+YOFF" format (in pixels):

bash$ ./sinus.pl | ./driveGnuPlots.pl 2 50 50 Sinus Cosinus 512x384+0+0 512x384+512+0

The provisioning of titles and GnuPlot window placement information, makes the script very well-suited for live demonstrations.

P.S. UNIX power in all its glory: it took me 30min to code this, and another 30 to debug it. Using pipes to spawned copies of gnuplots, we are able to do something that would require one or maybe two orders of magnitude more effort in any conventional programming language (yes, even accounting for custom graph libraries - you do have to learn their API and do your windows/interface handling...)

Update, November 30, 2009Andreas Bernauer has improved the script further, allowing multiple streams to be plotted in the same window. His work is here.
Update, December 20, 2009: Dima Kogan has done his own version, which detects the number of streams dynamically. He placed his code on GitHub.

Visualize real-time data streams with Gnuplot的更多相关文章

  1. FunDA(9)- Stream Source:reactive data streams

    上篇我们讨论了静态数据源(Static Source, snapshot).这种方式只能在预知数据规模有限的情况下使用,对于超大型的数据库表也可以说是不安全的资源使用方式.Slick3.x已经增加了支 ...

  2. NTFS格式下的Alternate Data Streams

    今天我写点NTFS的交换数据流以及其带来的安全问题(Alternate Data Stream/ADS) =============================================== ...

  3. Awesome Big Data List

    https://github.com/onurakpolat/awesome-bigdata A curated list of awesome big data frameworks, resour ...

  4. 翻译-In-Stream Big Data Processing 流式大数据处理

    相当长一段时间以来,大数据社区已经普遍认识到了批量数据处理的不足.很多应用都对实时查询和流式处理产生了迫切需求.最近几年,在这个理念的推动下,催生出了一系列解决方案,Twitter Storm,Yah ...

  5. Exploring the 7 Different Types of Data Stories

    Exploring the 7 Different Types of Data Stories What makes a story truly data-driven? For one, the n ...

  6. The difference between text mode and binary mode with file streams

    FIO14-C. Understand the difference between text mode and binary mode with file streams     Skip to e ...

  7. THE R QGRAPH PACKAGE: USING R TO VISUALIZE COMPLEX RELATIONSHIPS AMONG VARIABLES IN A LARGE DATASET, PART ONE

    The R qgraph Package: Using R to Visualize Complex Relationships Among Variables in a Large Dataset, ...

  8. Flink应用案例:How Trackunit leverages Flink to process real-time data from industrial IoT devices

    January 22, 2019Use Cases, Apache Flink Lasse Nedergaard     Recently there has been significant dis ...

  9. explore your hadoop data and get real-time results

    deep api integration makes getting value from your big data easy 深度api集成使你大数据訪问更加easy Elasticsearch ...

随机推荐

  1. 与众不同 windows phone (51) - 8.1 新增控件: DatePickerFlyout, TimePickerFlyout

    [源码下载] 与众不同 windows phone (51) - 8.1 新增控件: DatePickerFlyout, TimePickerFlyout 作者:webabcd 介绍与众不同 wind ...

  2. jsp中自定义Taglib案例

    一.使用TagSupport类案例解析 1.自定义Tag使用jdbc连接mysql数据库 1.1定义标签处理器类 package com.able.tag; import java.sql.Conne ...

  3. phpcms v9 升级视频云问题推荐位不能添加

    因为使用的是v9的早期版本,后来升级的时候没敢升级数据库,直接使用了老的数据库结构,造成[推荐位]添加不能使用,报告没有thumb列. 查看数据库果然没有,没办法要么添加相关的列,要么禁用上传缩略图. ...

  4. galera cluster DDL节点间状态不一致的问题

    近期某个系统中的galera cluseter环境发生A DDL操作后,B节点未同步的情况,同时B节点的errorlog中有如下警告信息: 2016-07-23 17:31:32 18920 [War ...

  5. 为什么不推崇复杂的ORM

    上一篇文章写完,回复的人很多,有的说的很中肯,有的貌似只是看到文章的标题就进来写评论的!还有人问为什么我要屏蔽掉[反对]按钮,因为谁写文章都是为了分享,都在说出自己的心得体会.不过由于大家遇到的项目, ...

  6. 开启Windows Server 2008 R2上帝模式

    TAG标签: 摘要:这个“God Mode” 应该大部分的网友都听过了,只是在 Windows Server 2008 R2 上也支持此一功能.启用方式非常简单,在桌面新建一个文件夹,命名为: God ...

  7. 用css伪类实现提示框效果

    题目要求用css实现下图效果: 很明显难点就在那个多出去的三角形上,下面代码是用一个div来实现的,用到了伪类 : befor和 : after,使用这两个伪类活生生的在div之前和之后多出了&quo ...

  8. sublimetext Emmet插件(Zen coding)

    1.省略div,插件会默认元素为div .container <div class="container"></div> 含糊标签名称,比如不需要指定li, ...

  9. UC如被百度控股,手机qq浏览器改如何进攻和防守

    很早以前在公司内部论坛里写的一篇文章,绯闻已经过过去了,现在已物事人物,UC已有阿里大靠山了. ----------------------------------------------- 据网络媒 ...

  10. 说说Python多线程中的daemon属性方法

    大家看多线程部分的时候肯定看到过daemon这个属性,当我在百度了一圈后也没发现有比较好的解释(或者大家对这个解释都非常清楚),于是自己通过代码和官方介绍了解它,进行了一些总结 给大家一些参考. 首先 ...