streaming窗口操作

之前一直对窗口操作不太理解。认为spark streaming本身已经是分片计算，还需要窗口操作干啥。

窗口操作最为简单易懂的场景就是，在M时间间隔计算一次N时间内的热搜。当M=N的时候，就像上述所说，窗口操作本身没什么优势;但当在M!=N的时候，窗口计算优势就体现出来了。

之前用storm的窗口计算，实在是麻烦。而spark streaming则要简单许多。

借用官网提供的图以及例子：

简来说就是10秒钟计算30秒内的单词数。

两个参数
window length - The duration of the window ( in the figure). -> N
sliding interval - The interval at which the window operation is performed ( in the figure). -> M

在每一次十秒分别输入：

1：

sdf sdfsd sdf

2：

sdf sdfsd sdf

3：

sdf sdfsd sdf
sdf sdfsd sdf 234

输入

期望得到的结果：

1

----------------------------------------

sdf : 2
sdfsd : 1

----------------------------------------

2:

----------------------------------------

sdf : 4
sdfsd : 2

----------------------------------------

3:

----------------------------------------

sdf : 8
sdfsd : 4
234:1

----------------------------------------

4:

----------------------------------------

sdf : 6
sdfsd : 3
234:1

----------------------------------------

5:

----------------------------------------

sdf : 4
sdfsd : 2
234:1

----------------------------------------

6:

----------------------------------------

无

----------------------------------------

输出

代码如下：

 var maxRatePerPartition = "3700"
     if (args.length > 1) maxRatePerPartition = args(0)
     var master = "local"
     if (args.length > 2) master = args(1)
     var duration = 10
     if (args.length > 3) duration = args(2).toInt
     var group = "brokers"
     if (args.length > 4) group = args(3)
     var topic = "test"
     if (args.length > 5) topic = args(4)
     var brokerlist = "master:9092"
     if (args.length > 6) brokerlist = args(5)

     val sparkConf = new SparkConf()
       .setMaster(master)
       .setAppName("window")

     val ssc = new StreamingContext(sparkConf, Seconds(duration))
     val topicsSet = topic.split(",").toSet
     val kafkaParams = Map[String, String](
       "metadata.broker.list" -> brokerlist,
       "group.id" -> group,
       "enable.auto.commit" -> "false")
     val kc = new KafkaCluster(kafkaParams)
     val topicAndPartition2 = kc.getPartitions(topicsSet)
     var topicAndPartition: Set[TopicAndPartition] = Set[TopicAndPartition]()

     var fromOffsets: Map[TopicAndPartition, Long] = Map[TopicAndPartition, Long]()

     var bool = true

     if (topicAndPartition2.isRight) topicAndPartition = kc.getPartitions(topicsSet).right.get

     else {
       val lateOffsets = kc.getLatestLeaderOffsets(topicAndPartition)
       if (lateOffsets.isLeft) { topicAndPartition.foreach { x => fromOffsets += (x -> 0) } }
       else { lateOffsets.right.get.foreach(x => fromOffsets += (x._1 -> x._2.offset)) }
       bool = false
     }

     if (bool) {
       var temp = kc.getConsumerOffsets(group, topicAndPartition)
       if (temp.isRight) {
         fromOffsets = temp.right.get
       } else if (temp.isLeft) {
         val lateOffsets = kc.getLatestLeaderOffsets(topicAndPartition)
         if (lateOffsets.isLeft) { topicAndPartition.foreach { x => fromOffsets += (x -> 0) } }
         else { lateOffsets.right.get.foreach(x => fromOffsets += (x._1 -> x._2.offset)) }
       }
     }
     val messageHandler = (mmd: MessageAndMetadata[String, String]) => (mmd.key, mmd.message)

     val messages = KafkaUtils.createDirectStream[String, String, StringDecoder, StringDecoder, (String, String)](ssc, kafkaParams, fromOffsets, messageHandler)

     var offsetsList = Array[OffsetRange]()
     val msgContent = messages.transform(x => {
       offsetsList = x.asInstanceOf[HasOffsetRanges].offsetRanges
       x
     })

     val data = msgContent.flatMap(x => x._2.split(" "))

     val data2 = data.map(x => (x, 1)).reduceByKeyAndWindow((a: Int, b: Int) => (a + b), Seconds(30), Seconds(10))

     data2.foreachRDD(x => {
       x.foreach(x => {
         println(x._1 + " = " + x._2)
       })
     })

     for (offsets <- offsetsList) {
       val topicAndPartition = TopicAndPartition(topic, offsets.partition)
       val o = kc.setConsumerOffsetMetadata(group, Map[TopicAndPartition, OffsetAndMetadata](topicAndPartition -> OffsetAndMetadata(offsets.untilOffset)))
       if (o.isLeft) {
         println(s"Error updating the offset to Kafka cluster: ${o.left.get}")
       }
     }

     ssc.start()
     ssc.awaitTermination()

输出结果达到预期效果。

reduceByKeyAndWindow提供了两个重载函数。两个函数实现的功能是一样的，差别在于invReduceFunc。函数1直接计算窗口，函数2计算并集加差集，效率更高。

函数1
reduceByKeyAndWindow(reduceFunc, windowDuration, slideDuration, numPartitions, filterFunc)
reduceByKeyAndWindow((a: Int, b: Int) => (a + b), (a: Int, b: Int) => (a - b),Seconds(30), Seconds(10))

函数2
reduceByKeyAndWindow(reduceFunc, invReduceFunc, windowDuration, slideDuration, numPartitions, filterFunc)
reduceByKeyAndWindow((a: Int, b: Int) => (a + b),Seconds(30), Seconds(10))

还是以上图为例

根据zhouzhihubeyond的博客，实现原理如下：

函数1(全量统计)：
window3 : time1+time2+time3
window5 : time3+time4+time5

函数2（增量统计）：
window3 : （time1+time2）+time3   然后checkpoint -> (time1+time2)|window3
window5 : window3 +time4+time5 - (time1+time2)  只需计算 time4+time5，无需计算time3

但是基于这种解释，有一个问题，为了便于window7的计算，在window5的时候，必须计算time3+time4并缓存，其实还是需要计算time3;而且需要缓存的大小(如window3需要缓存的大小是：(time1+time2)+（window3 = time1+time2+time3） = 2*time1+2*time2+time3=5/3*window3)是window本身的5/3。计算方式其实比函数1更复杂，而且更占用资源。

先问是不是，再问为什么。

上源码：

 /*
  * Licensed to the Apache Software Foundation (ASF) under one or more
  * contributor license agreements.  See the NOTICE file distributed with
  * this work for additional information regarding copyright ownership.
  * The ASF licenses this file to You under the Apache License, Version 2.0
  * (the "License"); you may not use this file except in compliance with
  * the License.  You may obtain a copy of the License at
  *
  *    http://www.apache.org/licenses/LICENSE-2.0
  *
  * Unless required by applicable law or agreed to in writing, software
  * distributed under the License is distributed on an "AS IS" BASIS,
  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  * See the License for the specific language governing permissions and
  * limitations under the License.
  */

 package org.apache.spark.streaming.dstream

 import scala.reflect.ClassTag

 import org.apache.spark.rdd.RDD
 import org.apache.spark.storage.StorageLevel
 import org.apache.spark.streaming._
 import org.apache.spark.streaming.Duration

 private[streaming]
 class WindowedDStream[T: ClassTag](
     parent: DStream[T],
     _windowDuration: Duration,
     _slideDuration: Duration)
   extends DStream[T](parent.ssc) {

   if (!_windowDuration.isMultipleOf(parent.slideDuration)) {
     throw new Exception("The window duration of windowed DStream (" + _windowDuration + ") " +
     "must be a multiple of the slide duration of parent DStream (" + parent.slideDuration + ")")
   }

   if (!_slideDuration.isMultipleOf(parent.slideDuration)) {
     throw new Exception("The slide duration of windowed DStream (" + _slideDuration + ") " +
     "must be a multiple of the slide duration of parent DStream (" + parent.slideDuration + ")")
   }

   // Persist parent level by default, as those RDDs are going to be obviously reused.
   parent.persist(StorageLevel.MEMORY_ONLY_SER)

   def windowDuration: Duration = _windowDuration

   override def dependencies: List[DStream[_]] = List(parent)

   override def slideDuration: Duration = _slideDuration

   override def parentRememberDuration: Duration = rememberDuration + windowDuration

   override def persist(level: StorageLevel): DStream[T] = {
     // Do not let this windowed DStream be persisted as windowed (union-ed) RDDs share underlying
     // RDDs and persisting the windowed RDDs would store numerous copies of the underlying data.
     // Instead control the persistence of the parent DStream.
     parent.persist(level)
     this
   }

   override def compute(validTime: Time): Option[RDD[T]] = {
     val currentWindow = new Interval(validTime - windowDuration + parent.slideDuration, validTime)
     val rddsInWindow = parent.slice(currentWindow)
     Some(ssc.sc.union(rddsInWindow))
   }
 }

函数1关键源码

/*
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements.  See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to You under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License.  You may obtain a copy of the License at
 *
 *    http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package org.apache.spark.streaming.dstream

import scala.collection.mutable.ArrayBuffer
import scala.reflect.ClassTag

import org.apache.spark.Partitioner
import org.apache.spark.rdd.{CoGroupedRDD, RDD}
import org.apache.spark.storage.StorageLevel
import org.apache.spark.streaming.{Duration, Interval, Time}

private[streaming]
class ReducedWindowedDStream[K: ClassTag, V: ClassTag](
    parent: DStream[(K, V)],
    reduceFunc: (V, V) => V,
    invReduceFunc: (V, V) => V,
    filterFunc: Option[((K, V)) => Boolean],
    _windowDuration: Duration,
    _slideDuration: Duration,
    partitioner: Partitioner
  ) extends DStream[(K, V)](parent.ssc) {

  require(_windowDuration.isMultipleOf(parent.slideDuration),
    "The window duration of ReducedWindowedDStream (" + _windowDuration + ") " +
      "must be multiple of the slide duration of parent DStream (" + parent.slideDuration + ")"
  )

  require(_slideDuration.isMultipleOf(parent.slideDuration),
    "The slide duration of ReducedWindowedDStream (" + _slideDuration + ") " +
      "must be multiple of the slide duration of parent DStream (" + parent.slideDuration + ")"
  )

  // Reduce each batch of data using reduceByKey which will be further reduced by window
  // by ReducedWindowedDStream
  private val reducedStream = parent.reduceByKey(reduceFunc, partitioner)

  // Persist RDDs to memory by default as these RDDs are going to be reused.
  super.persist(StorageLevel.MEMORY_ONLY_SER)
  reducedStream.persist(StorageLevel.MEMORY_ONLY_SER)

  def windowDuration: Duration = _windowDuration

  override def dependencies: List[DStream[_]] = List(reducedStream)

  override def slideDuration: Duration = _slideDuration

  override val mustCheckpoint = true

  override def parentRememberDuration: Duration = rememberDuration + windowDuration

  override def persist(storageLevel: StorageLevel): DStream[(K, V)] = {
    super.persist(storageLevel)
    reducedStream.persist(storageLevel)
    this
  }

  override def checkpoint(interval: Duration): DStream[(K, V)] = {
    super.checkpoint(interval)
    // reducedStream.checkpoint(interval)
    this
  }

  override def compute(validTime: Time): Option[RDD[(K, V)]] = {
    val reduceF = reduceFunc
    val invReduceF = invReduceFunc

    val currentTime = validTime
    val currentWindow = new Interval(currentTime - windowDuration + parent.slideDuration,
      currentTime)
    val previousWindow = currentWindow - slideDuration

    logDebug("Window time = " + windowDuration)
    logDebug("Slide time = " + slideDuration)
    logDebug("Zero time = " + zeroTime)
    logDebug("Current window = " + currentWindow)
    logDebug("Previous window = " + previousWindow)

    //  _____________________________
    // |  previous window   _________|___________________
    // |___________________|       current window        |  --------------> Time
    //                     |_____________________________|
    //
    // |________ _________|          |________ _________|
    //          |                             |
    //          V                             V
    //       old RDDs                     new RDDs
    //

    // Get the RDDs of the reduced values in "old time steps"
    val oldRDDs =
      reducedStream.slice(previousWindow.beginTime, currentWindow.beginTime - parent.slideDuration)
    logDebug("# old RDDs = " + oldRDDs.size)

    // Get the RDDs of the reduced values in "new time steps"
    val newRDDs =
      reducedStream.slice(previousWindow.endTime + parent.slideDuration, currentWindow.endTime)
    logDebug("# new RDDs = " + newRDDs.size)

    // Get the RDD of the reduced value of the previous window
    val previousWindowRDD =
      getOrCompute(previousWindow.endTime).getOrElse(ssc.sc.makeRDD(Seq[(K, V)]()))

    // Make the list of RDDs that needs to cogrouped together for reducing their reduced values
    val allRDDs = new ArrayBuffer[RDD[(K, V)]]() += previousWindowRDD ++= oldRDDs ++= newRDDs

    // Cogroup the reduced RDDs and merge the reduced values
    val cogroupedRDD = new CoGroupedRDD[K](allRDDs.toSeq.asInstanceOf[Seq[RDD[(K, _)]]],
      partitioner)
    // val mergeValuesFunc = mergeValues(oldRDDs.size, newRDDs.size) _

    val numOldValues = oldRDDs.size
    val numNewValues = newRDDs.size

    val mergeValues = (arrayOfValues: Array[Iterable[V]]) => {
      if (arrayOfValues.length != 1 + numOldValues + numNewValues) {
        throw new Exception("Unexpected number of sequences of reduced values")
      }
      // Getting reduced values "old time steps" that will be removed from current window
      val oldValues = (1 to numOldValues).map(i => arrayOfValues(i)).filter(!_.isEmpty).map(_.head)
      // Getting reduced values "new time steps"
      val newValues =
        (1 to numNewValues).map(i => arrayOfValues(numOldValues + i)).filter(!_.isEmpty).map(_.head)

      if (arrayOfValues(0).isEmpty) {
        // If previous window's reduce value does not exist, then at least new values should exist
        if (newValues.isEmpty) {
          throw new Exception("Neither previous window has value for key, nor new values found. " +
            "Are you sure your key class hashes consistently?")
        }
        // Reduce the new values
        newValues.reduce(reduceF) // return
      } else {
        // Get the previous window's reduced value
        var tempValue = arrayOfValues(0).head
        // If old values exists, then inverse reduce then from previous value
        if (!oldValues.isEmpty) {
          tempValue = invReduceF(tempValue, oldValues.reduce(reduceF))
        }
        // If new values exists, then reduce them with previous value
        if (!newValues.isEmpty) {
          tempValue = reduceF(tempValue, newValues.reduce(reduceF))
        }
        tempValue // return
      }
    }

    val mergedValuesRDD = cogroupedRDD.asInstanceOf[RDD[(K, Array[Iterable[V]])]]
      .mapValues(mergeValues)

    if (filterFunc.isDefined) {
      Some(mergedValuesRDD.filter(filterFunc.get))
    } else {
      Some(mergedValuesRDD)
    }
  }
}
 

函数2的关键源码

可以看到，它们关键的区别就在compute函数。

函数1的compute函数,没有问题就根据两个时间点得到当前窗口时间并取得相应的值而已

 override def compute(validTime: Time): Option[RDD[T]] = {
     val currentWindow = new Interval(validTime - windowDuration + parent.slideDuration, validTime)
     val rddsInWindow = parent.slice(currentWindow)
     Some(ssc.sc.union(rddsInWindow))
   }

函数2的compute函数就要复杂多了

 override def compute(validTime: Time): Option[RDD[(K, V)]] = {
     val reduceF = reduceFunc
     val invReduceF = invReduceFunc

     val currentTime = validTime
     val currentWindow = new Interval(currentTime - windowDuration + parent.slideDuration,
       currentTime)
     val previousWindow = currentWindow - slideDuration

     logDebug("Window time = " + windowDuration)
     logDebug("Slide time = " + slideDuration)
     logDebug("Zero time = " + zeroTime)
     logDebug("Current window = " + currentWindow)
     logDebug("Previous window = " + previousWindow)

     //  _____________________________
     // |  previous window   _________|___________________
     // |___________________|       current window        |  --------------> Time
     //                     |_____________________________|
     //
     // |________ _________|          |________ _________|
     //          |                             |
     //          V                             V
     //       old RDDs                     new RDDs
     //

     // Get the RDDs of the reduced values in "old time steps"
     val oldRDDs =
       reducedStream.slice(previousWindow.beginTime, currentWindow.beginTime - parent.slideDuration)
     logDebug("# old RDDs = " + oldRDDs.size)

     // Get the RDDs of the reduced values in "new time steps"
     val newRDDs =
       reducedStream.slice(previousWindow.endTime + parent.slideDuration, currentWindow.endTime)
     logDebug("# new RDDs = " + newRDDs.size)

     // Get the RDD of the reduced value of the previous window
     val previousWindowRDD =
       getOrCompute(previousWindow.endTime).getOrElse(ssc.sc.makeRDD(Seq[(K, V)]()))

     // Make the list of RDDs that needs to cogrouped together for reducing their reduced values
     val allRDDs = new ArrayBuffer[RDD[(K, V)]]() += previousWindowRDD ++= oldRDDs ++= newRDDs

     // Cogroup the reduced RDDs and merge the reduced values
     val cogroupedRDD = new CoGroupedRDD[K](allRDDs.toSeq.asInstanceOf[Seq[RDD[(K, _)]]],
       partitioner)
     // val mergeValuesFunc = mergeValues(oldRDDs.size, newRDDs.size) _

     val numOldValues = oldRDDs.size
     val numNewValues = newRDDs.size

     val mergeValues = (arrayOfValues: Array[Iterable[V]]) => {
       if (arrayOfValues.length != 1 + numOldValues + numNewValues) {
         throw new Exception("Unexpected number of sequences of reduced values")
       }
       // Getting reduced values "old time steps" that will be removed from current window
       val oldValues = (1 to numOldValues).map(i => arrayOfValues(i)).filter(!_.isEmpty).map(_.head)
       // Getting reduced values "new time steps"
       val newValues =
         (1 to numNewValues).map(i => arrayOfValues(numOldValues + i)).filter(!_.isEmpty).map(_.head)

       if (arrayOfValues(0).isEmpty) {
         // If previous window's reduce value does not exist, then at least new values should exist
         if (newValues.isEmpty) {
           throw new Exception("Neither previous window has value for key, nor new values found. " +
             "Are you sure your key class hashes consistently?")
         }
         // Reduce the new values
         newValues.reduce(reduceF) // return
       } else {
         // Get the previous window's reduced value
         var tempValue = arrayOfValues(0).head
         // If old values exists, then inverse reduce then from previous value
         if (!oldValues.isEmpty) {
           tempValue = invReduceF(tempValue, oldValues.reduce(reduceF))
         }
         // If new values exists, then reduce them with previous value
         if (!newValues.isEmpty) {
           tempValue = reduceF(tempValue, newValues.reduce(reduceF))
         }
         tempValue // return
       }
     }

     val mergedValuesRDD = cogroupedRDD.asInstanceOf[RDD[(K, Array[Iterable[V]])]]
       .mapValues(mergeValues)

     if (filterFunc.isDefined) {
       Some(mergedValuesRDD.filter(filterFunc.get))
     } else {
       Some(mergedValuesRDD)
     }
   }

如图，我在官方的基础上加了一个keyRDDs

    //  _____________________________
    // |  previous window   _________|___________________
    // |___________________|       current window        |  --------------> Time
    //                     |_____________________________|
    //
    // |________ _________|     |     |________ _________|
    //          |               |              |
    //          V               V              V
    //       old RDDs        keyRDDs         new RDDs
    //

从代码中可以看出,做了如下操作：

previous window - old RDDS = key RDDS
key RDDs + new RDDs = current window (当前窗口)

        // Get the previous window's reduced value
         var tempValue = arrayOfValues(0).head
         if (!oldValues.isEmpty) {
            tempValue = invReduceF(tempValue, oldValues.reduce(reduceF)) //自定义函数 前减后
         }
         // If new values exists, then reduce them with previous value         if (!newValues.isEmpty) {
            tempValue = reduceF(tempValue, newValues.reduce(reduceF)) //自定义函数 前加后
          }

然后对前一个窗口previous window作为缓存

// Get the RDD of the reduced value of the previous window
    val previousWindowRDD =
    getOrCompute(previousWindow.endTime).getOrElse(ssc.sc.makeRDD(Seq[(K, V)]()))

所以，非首次的计算需要计算old RDDs 和new RDDs,其实看起来还是很繁琐，效率当真较高一点吗？待测试。