解读：计数器Counter

Counters: 44
File System Counters
        FILE: Number of bytes read=655771325
        FILE: Number of bytes written=984244425
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
        HDFS: Number of bytes read=260407668
        HDFS: Number of bytes written=17681802
        HDFS: Number of read operations=37
        HDFS: Number of large read operations=0
        HDFS: Number of write operations=10
Job Counters
        Launched map tasks=4
        Launched reduce tasks=5
        Other local map tasks=1
        Data-local map tasks=3
        Total time spent by all maps in occupied slots (ms)=60987
        Total time spent by all reduces in occupied slots (ms)=50362
Map-Reduce Framework
        Map input records=1152870
        Map output records=22472940
        Map output bytes=282888289
        Map output materialized bytes=327843405
        Input split bytes=1173
        Combine input records=0
        Combine output records=0
        Reduce input groups=579532
        Reduce shuffle bytes=327843405
        Reduce input records=22472940
        Reduce output records=579532
        Spilled Records=67418820
        Shuffled Maps =20
        Failed Shuffles=0
        Merged Map outputs=20
        GC time elapsed (ms)=2826
        CPU time spent (ms)=69670
        Physical memory (bytes) snapshot=2287190016
        Virtual memory (bytes) snapshot=7904223232
        Total committed heap usage (bytes)=1572864000
Shuffle Errors
        BAD_ID=0
        CONNECTION=0
        IO_ERROR=0
        WRONG_LENGTH=0
        WRONG_MAP=0
        WRONG_REDUCE=0
File Input Format Counters
        Bytes Read=0
File Output Format Counters
        Bytes Written=17681802

Counters: 44表示计数器总共44个，粉色表示计数器种类，即6类。

1). File System Counters：MR-Job执行依赖的数据来自不同的文件系统，这个group表示job与文件系统交互的读写统计

• HDFS: Number of bytes read=260407668　　//map从HDFS读取数据，包括源文件内容、split元数据。所以这个值比FileInputFormatCounters.BYTES_READ 要略大些。
• FILE: Number of bytes written=984244425　　//表示map task往本地磁盘中总共写了多少字节（其实，Reduce端的Merge也会写入本地File）
• FILE: Number of bytes read=655771325　　//reduce从本地文件系统读取数据（map结果保存在本地磁盘）
• HDFS: Number of bytes written=17681802　　//最终结果写入HDFS

2). Job Counters：MR子任务统计，即map tasks 和 reduce tasks

• Launched map tasks=4　　//启用map task的个数
• Launched reduce tasks=5　　//启用reduce task的个数

3). Map-Reduce Framework：MR框架计数器

• Map input records=1152870　　//map task从HDFS读取的文件总行数
• Reduce input groups=579532    //Reduce输入的分组个数，如<hello,{1,1}> <me,1> <you,1>。如果有Combiner的话，那么这里的数值就等于map端Combiner运算后的最后条数，如果没有，那么就应该等于map的输出条数
• Combine input records=0　　//Combiner输入 = map输出
• Spilled Records=67418820　　//spill过程在map和reduce端都会发生，这里统计在总共从内存往磁盘中spill了多少条数据

4). Shuffle Errors：

5). File Input Format Counters：文件输入格式化计数器

　　Bytes Read=0　　//map阶段，各个map task的map方法输入的所有value值字节数之和

6). File Output Format Counters：文件输出格式化计数器

　　Bytes Written=17681802　　//MR输出总的字节数，包括【单词】,【空格】,【单词个数】及每行的【换行符】

自定义计数器

//自定义计数器<Key , Value>的形式
 
Counter counter = context.getCounter("查找hello", "hello");
 
if(string.contains("hello")){
 
counter.increment(1l);//出现一次+1
 
}