应用场景：实时仪表盘（即大屏），每个集团下有多个mall，每个mall下包含多家shop，需实时计算集团下各mall及其shop的实时销售分析（区域、业态、店铺TOP、总销售额等指标）并提供可视化展现，之前时候一直在Strom实现，现在改为Spark2.3.2实现。

1、数据源：首先数据源来自于MQ、Socket、Flume和DFS等，一般Kafka、RocketMQ等居多，此处示例代码用的是RocketMQ；

2、实时计算框架：Storm（实时计算，Spout发射Tuple到各个Bolt，来一条即处理一条，一百毫秒以内的延迟）、SparkStreaming（准实时计算，基于微批次的实时计算，即一定时间段内的micro batch，每个micro batch的结构为DStream，底层是RDD）；

3、此处Spark Streaming准实时处理应用流程：1、RocketMQ --> 2、SparkStreaming --> 3、SparkSQL（Parquet） --> 4、Redis（大屏使用） && HDFS（Hive数仓ODS层，ODS->DW[DWD-DWS]-DM）；

4、系统设计

5、代码如下（4.3是Spark Streaming实现，复制粘贴即可运行）：

  • 5.1、RocketMQConfig类（用于配置和构建MQ消费者）
  • 5.2、SparkStreaming自定义RocketMQ接收器（可靠的接收器）
  • 5.3、SparkStreaming销售分析实时计算
  • 5.4、账单实体类
  • 5.5、BCD即账单裁剪后的实体类（减少数据量传输即降低节点间的序列化和反序列化开销）
  • 5.6、SparkSQL销售分析业务实现类

5.1、RocketMQConfig类（用于配置和构建MQ消费者）

  1. package com.mengyao.graph.etl.apps.commons.datasource.mq.receiver;
  2.  
  3. import org.apache.commons.lang.Validate;
  4. import org.apache.rocketmq.client.ClientConfig;
  5. import org.apache.rocketmq.client.consumer.DefaultMQPushConsumer;
  6. import org.apache.rocketmq.client.exception.MQClientException;
  7. import org.apache.rocketmq.common.consumer.ConsumeFromWhere;
  8. import org.apache.rocketmq.remoting.common.RemotingUtil;
  9.  
  10. import java.util.HashMap;
  11. import java.util.UUID;
  12.  
  13. /**
  14.  * RocketMQConfig for Consumer
  15.  * @author mengyao
  16.  *
  17.  */
  18. public class RocketMQConfig {
  19.  
  20.     // ------- the following is for common usage -------
  21.     /**
  22.      * RocketMq name server address
  23.      */
  24.     public static final String NAME_SERVER_ADDR = "nameserver.addr"; // Required
  25.  
  26.     public static final String CLIENT_NAME = "client.name";
  27.  
  28.     public static final String CLIENT_IP = "client.ip";
  29.     public static final String DEFAULT_CLIENT_IP = RemotingUtil.getLocalAddress();
  30.  
  31.     public static final String CLIENT_CALLBACK_EXECUTOR_THREADS = "client.callback.executor.threads";
  32.     public static final int DEFAULT_CLIENT_CALLBACK_EXECUTOR_THREADS = Runtime.getRuntime().availableProcessors();;
  33.  
  34.     public static final String NAME_SERVER_POLL_INTERVAL = "nameserver.poll.interval";
  35.     public static final int DEFAULT_NAME_SERVER_POLL_INTERVAL = 30000; // 30 seconds
  36.  
  37.     public static final String BROKER_HEART_BEAT_INTERVAL = "brokerserver.heartbeat.interval";
  38.     public static final int DEFAULT_BROKER_HEART_BEAT_INTERVAL = 30000; // 30 seconds
  39.  
  40.     // ------- the following is for push consumer mode -------
  41.     /**
  42.      * RocketMq consumer group
  43.       */
  44.     public static final String CONSUMER_GROUP = "consumer.group"; // Required
  45.  
  46.     /**
  47.      * RocketMq consumer topic
  48.      */
  49.     public static final String CONSUMER_TOPIC = "consumer.topic"; // Required
  50.  
  51.     public static final String CONSUMER_TAG = "consumer.tag";
  52.     public static final String DEFAULT_TAG = "*";
  53.  
  54.     public static final String CONSUMER_OFFSET_RESET_TO = "consumer.offset.reset.to";
  55.     public static final String CONSUMER_OFFSET_LATEST = "latest";
  56.     public static final String CONSUMER_OFFSET_EARLIEST = "earliest";
  57.     public static final String CONSUMER_OFFSET_TIMESTAMP = "timestamp";
  58.  
  59.     public static final String CONSUMER_MESSAGES_ORDERLY = "consumer.messages.orderly";
  60.  
  61.     public static final String CONSUMER_OFFSET_PERSIST_INTERVAL = "consumer.offset.persist.interval";
  62.     public static final int DEFAULT_CONSUMER_OFFSET_PERSIST_INTERVAL = 5000; // 5 seconds
  63.  
  64.     public static final String CONSUMER_MIN_THREADS = "consumer.min.threads";
  65.     public static final int DEFAULT_CONSUMER_MIN_THREADS = 20;
  66.  
  67.     public static final String CONSUMER_MAX_THREADS = "consumer.max.threads";
  68.     public static final int DEFAULT_CONSUMER_MAX_THREADS = 64;
  69.  
  70.     // ------- the following is for reliable Receiver -------
  71.     public static final String QUEUE_SIZE = "spout.queue.size";
  72.     public static final int DEFAULT_QUEUE_SIZE = 500;
  73.  
  74.     public static final String MESSAGES_MAX_RETRY = "spout.messages.max.retry";
  75.     public static final int DEFAULT_MESSAGES_MAX_RETRY = 3;
  76.  
  77.     public static final String MESSAGES_TTL = "spout.messages.ttl";
  78.     public static final int DEFAULT_MESSAGES_TTL = 300000;  // 5min
  79.  
  80.     // ------- the following is for pull consumer mode -------
  81.  
  82.     /**
  83.      * Maximum rate (number of records per second) at which data will be read from each RocketMq partition ,
  84.      * and the default value is "-1", it means consumer can pull message from rocketmq as fast as the consumer can.
  85.      * Other that, you also enables or disables Spark Streaming's internal backpressure mechanism by the config
  86.      *  "spark.streaming.backpressure.enabled".
  87.      */
  88.     public static final String  MAX_PULL_SPEED_PER_PARTITION = "pull.max.speed.per.partition";
  89.  
  90.     /**
  91.      * To pick up the consume speed, the consumer can pull a batch of messages at a time. And the default
  92.      * value is "32"
  93.      */
  94.     public static final String PULL_MAX_BATCH_SIZE = "pull.max.batch.size";
  95.  
  96.     /**
  97.      * pull timeout for the consumer, and the default time is "3000".
  98.      */
  99.     public static final String PULL_TIMEOUT_MS = "pull.timeout.ms";
  100.  
  101.     // the following configs for consumer cache
  102.     public static final String PULL_CONSUMER_CACHE_INIT_CAPACITY = "pull.consumer.cache.initialCapacity";
  103.     public static final String PULL_CONSUMER_CACHE_MAX_CAPACITY = "pull.consumer.cache.maxCapacity";
  104.     public static final String PULL_CONSUMER_CACHE_LOAD_FACTOR = "pull.consumer.cache.loadFactor";
  105.  
  106.     public static void buildConsumerConfigs(HashMap<String, String> props, DefaultMQPushConsumer consumer) {
  107.         buildCommonConfigs(props, consumer);
  108.  
  109.         String group = props.get(CONSUMER_GROUP);
  110.         Validate.notEmpty(group);
  111.         consumer.setConsumerGroup(group);
  112.  
  113.         consumer.setPersistConsumerOffsetInterval(getInteger(props,
  114.                 CONSUMER_OFFSET_PERSIST_INTERVAL, DEFAULT_CONSUMER_OFFSET_PERSIST_INTERVAL));
  115.         consumer.setConsumeThreadMin(getInteger(props,
  116.                 CONSUMER_MIN_THREADS, DEFAULT_CONSUMER_MIN_THREADS));
  117.         consumer.setConsumeThreadMax(getInteger(props,
  118.                 CONSUMER_MAX_THREADS, DEFAULT_CONSUMER_MAX_THREADS));
  119.  
  120.         String initOffset = props.get(CONSUMER_OFFSET_RESET_TO) != null ? props.get(CONSUMER_OFFSET_RESET_TO) : CONSUMER_OFFSET_LATEST;
  121.         switch (initOffset) {
  122.             case CONSUMER_OFFSET_EARLIEST:
  123.                 consumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_FIRST_OFFSET);
  124.                 break;
  125.             case CONSUMER_OFFSET_LATEST:
  126.                 consumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_LAST_OFFSET);
  127.                 break;
  128.             case CONSUMER_OFFSET_TIMESTAMP:
  129.                 consumer.setConsumeTimestamp(initOffset);
  130.                 break;
  131.             default:
  132.                 consumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_LAST_OFFSET);
  133.         }
  134.  
  135.         String topic = props.get(CONSUMER_TOPIC);
  136.         Validate.notEmpty(topic);
  137.         try {
  138.             consumer.subscribe(topic, props.get(CONSUMER_TAG) != null ? props.get(CONSUMER_TAG) : DEFAULT_TAG);
  139.         } catch (MQClientException e) {
  140.             throw new IllegalArgumentException(e);
  141.         }
  142.     }
  143.  
  144.     public static void buildCommonConfigs(HashMap<String, String> props, ClientConfig client) {
  145.         String namesvr = props.get(NAME_SERVER_ADDR);
  146.         Validate.notEmpty(namesvr);
  147.         client.setNamesrvAddr(namesvr);
  148.  
  149.         client.setClientIP(props.get(CLIENT_IP) != null ? props.get(CLIENT_IP) : DEFAULT_CLIENT_IP);
  150.         // use UUID for client name by default
  151.         String defaultClientName = UUID.randomUUID().toString();
  152.         client.setInstanceName(props.get(CLIENT_NAME) != null ? props.get(CLIENT_NAME) : defaultClientName);
  153.  
  154.         client.setClientCallbackExecutorThreads(getInteger(props,
  155.                 CLIENT_CALLBACK_EXECUTOR_THREADS, DEFAULT_CLIENT_CALLBACK_EXECUTOR_THREADS));
  156.         client.setPollNameServerInterval(getInteger(props,
  157.                 NAME_SERVER_POLL_INTERVAL, DEFAULT_NAME_SERVER_POLL_INTERVAL));
  158.         client.setHeartbeatBrokerInterval(getInteger(props,
  159.                 BROKER_HEART_BEAT_INTERVAL, DEFAULT_BROKER_HEART_BEAT_INTERVAL));
  160.     }
  161.  
  162.     public static int getInteger(HashMap<String, String> props, String key, int defaultValue) {
  163.         return Integer.parseInt(props.get(key) != null ? props.get(key) : String.valueOf(defaultValue));
  164.     }
  165.  
  166.     public static boolean getBoolean(HashMap<String, String> props, String key, boolean defaultValue) {
  167.         return Boolean.parseBoolean(props.get(key) != null ? props.get(key) : String.valueOf(defaultValue));
  168.     }
  169.  
  170. }

4.2、SparkStreaming自定义RocketMQ接收器（可靠的接收器）

  1. package com.mengyao.graph.etl.apps.commons.datasource.mq.receiver;
  2.  
  3. import org.apache.activemq.util.ByteArrayInputStream;
  4. import org.apache.commons.lang.StringUtils;
  5. import org.apache.commons.lang.Validate;
  6. //import org.apache.commons.lang3.SerializationUtils;
  7. import org.apache.rocketmq.client.consumer.DefaultMQPushConsumer;
  8. import org.apache.rocketmq.client.consumer.MQPushConsumer;
  9. import org.apache.rocketmq.client.consumer.listener.ConsumeConcurrentlyContext;
  10. import org.apache.rocketmq.client.consumer.listener.ConsumeConcurrentlyStatus;
  11. import org.apache.rocketmq.client.consumer.listener.ConsumeOrderlyContext;
  12. import org.apache.rocketmq.client.consumer.listener.ConsumeOrderlyStatus;
  13. import org.apache.rocketmq.client.consumer.listener.MessageListenerConcurrently;
  14. import org.apache.rocketmq.client.consumer.listener.MessageListenerOrderly;
  15. import org.apache.rocketmq.client.exception.MQClientException;
  16. import org.apache.rocketmq.common.message.MessageExt;
  17. import org.apache.spark.storage.StorageLevel;
  18. import org.apache.spark.streaming.receiver.Receiver;
  19.  
  20. import com.gooagoo.entity.mq.bill.MQBillMessage;
  21. import com.mengyao.graph.etl.apps.commons.beans.ods.fmt.BillInfoFmt;
  22. import com.mengyao.graph.etl.apps.commons.beans.ods.fmt.ConvertTool;
  23. import com.mengyao.graph.etl.apps.dashboard.beans.BCD;
  24.  
  25. import java.io.IOException;
  26. import java.io.ObjectInputStream;
  27. import java.util.HashMap;
  28. import java.util.List;
  29.  
  30. /**
  31.  * RocketMQ Receiver
  32.  * @author mengyao
  33.  *
  34.  */
  35. public class RocketMQReceiver extends Receiver<BCD> {
  36.  
  37.     /**
  38.      *
  39.      */
  40.     private static final long serialVersionUID = 2274826339951693341L;
  41.     private MQPushConsumer consumer;
  42.     private boolean ordered;
  43.     private HashMap<String, String> conf;
  44.  
  45.     public RocketMQReceiver(HashMap<String, String> conf, StorageLevel storageLevel) {
  46.         super(storageLevel);
  47.         this.conf = conf;
  48.     }
  49.  
  50.     @Override
  51.     public void onStart() {
  52.         Validate.notEmpty(conf, "Consumer properties can not be empty");
  53.         ordered = RocketMQConfig.getBoolean(conf, RocketMQConfig.CONSUMER_MESSAGES_ORDERLY, true);
  54.         consumer = new DefaultMQPushConsumer();
  55.         RocketMQConfig.buildConsumerConfigs(conf, (DefaultMQPushConsumer)consumer);
  56.         if (ordered) {
  57.             consumer.registerMessageListener(new MessageListenerOrderly() {
  58.                 @Override
  59.                 public ConsumeOrderlyStatus consumeMessage(List<MessageExt> msgs, ConsumeOrderlyContext context) {
  60.                     if (process(msgs)) {
  61.                         return ConsumeOrderlyStatus.SUCCESS;
  62.                     } else {
  63.                         return ConsumeOrderlyStatus.SUSPEND_CURRENT_QUEUE_A_MOMENT;
  64.                     }
  65.                 }
  66.             });
  67.         } else {
  68.             consumer.registerMessageListener(new MessageListenerConcurrently() {
  69.                 @Override
  70.                 public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs, ConsumeConcurrentlyContext context) {
  71.                     if (process(msgs)) {
  72.                         return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
  73.                     } else {
  74.                         return ConsumeConcurrentlyStatus.RECONSUME_LATER;
  75.                     }
  76.                 }
  77.             });
  78.         }
  79.         try {
  80.             consumer.start();
  81.             System.out.println("==== RocketMQReceiver start ====");
  82.         } catch (MQClientException e) {
  83.             e.printStackTrace();
  84.             throw new RuntimeException(e);
  85.         }
  86.     }
  87.  
  88.     public boolean process(List<MessageExt> msgs) {
  89.         if (msgs.isEmpty()) {
  90.             System.out.println("==== Msgs is null! ====");
  91.             return true;
  92.         }
  93.         System.out.println("==== receiver msgs: "+msgs.size()+" record. ====");
  94.         try {
  95.             for (MessageExt messageExt : msgs) {
  96.                 //MQBillMessage message = SerializationUtils.deserialize(messageExt.getBody());
  97.                 MQBillMessage message = deserialize(messageExt.getBody());
  98.                 if (null != message) {
  99.                     BillInfoFmt billFmt = ConvertTool.convertGagBillToOdsBillFmt(message.getData());
  100.                     if (validBillType(billFmt)) {
    　　　　　　　　　　　　　　 /**
    　　　　　　　　　　　　　　　 * this.store(BCD) 简单的接收一条存储一条，缺点是没有确认机制，不具备容错保证，会出现数据丢失。优点则是效率更高。
    　　　　　　　　　　　　　　　 * this.store(Iterator<BCD>)阻塞调用，当接收到的记录都存储到Spark后才会确认成功，当接收方采用复制（默认存储级别为复制）则在复制完成后确认成功，但在缓冲中的数据不被保证，会被重新发送。具备容错保证，可确保数据0丢失。
    　　　　　　　　　　　　　　　 */
  101.                         this.store(new BCD(billFmt.getId(), billFmt.getReceivableAmount(), billFmt.getSaleTime(), billFmt.getBillType(), billFmt.getShopId(), billFmt.getShopEntityId()));
  102.                     }
  103.                     billFmt=null;
  104.                     message.setData(null);
  105.                     message = null;
  106.                 } else {
  107.                     System.out.println("==== receiver msg is:"+messageExt.getBody()+", deserialize faild. ====");
  108.                 }
  109.             }
  110.             return true;
  111.         } catch (Exception e) {
  112.             e.printStackTrace();
  113.             return false;
  114.         }
  115.     }
  116.  
  117.     /**
  118.      * 验证账单类型是否为1、3、6
  119.      * @param bean
  120.      * @return
  121.      */
  122.     static boolean validBillType(BillInfoFmt bean) {
  123.         if (null == bean) {
  124.             System.out.println("==== BillBean is null! ====");
  125.             return false;
  126.         }
  127.         String billType = bean.getBillType();
  128.         if (StringUtils.isEmpty(billType)) {
  129.             System.out.println("==== BillBean.billType is null! ====");
  130.             return false;
  131.         }
  132.         String billTypeTrim = billType.trim();
  133.         return billTypeTrim.equals("1") || billType.equals("3") || billType.equals("6");
  134.     }
  135.  
  136.     @Override
  137.     public void onStop() {
  138.         consumer.shutdown();
  139.         System.out.println("==== RocketMQReceiver stop ====");
  140.     }
  141.  
  142.     private static MQBillMessage deserialize(byte[] body) throws Exception {
  143.         if (body == null) {
  144.             throw new IllegalArgumentException("The byte array must not be null");
  145.         }
  146.         MQBillMessage message = null;
  147.         ObjectInputStream in = null;
  148.         ByteArrayInputStream bais = null;
  149.         try {
  150.             bais = new ByteArrayInputStream(body);
  151.             in = new ObjectInputStream(bais);
  152.             message = (MQBillMessage) in.readObject();
  153.         } catch (final ClassNotFoundException ex) {
  154.             ex.printStackTrace();
  155.             throw ex;
  156.         } catch (final IOException ex) {
  157.             ex.printStackTrace();
  158.             throw ex;
  159.         } finally {
  160.             try {
  161.                 if (bais != null) {
  162.                     bais.close();
  163.                 }
  164.                 if (in != null) {
  165.                     in.close();
  166.                 }
  167.             } catch (final IOException ex) {
  168.                 // ignore close exception
  169.             }
  170.         }
  171.         return message;
  172.     }
  173.  
  174. }

4.3、SparkStreaming销售分析实时计

  1. package com.mengyao.graph.etl.apps.commons.datasource.bill;
  2.  
  3. import java.io.IOException;
  4. import java.text.SimpleDateFormat;
  5. import java.util.ArrayList;
  6. import java.util.Date;
  7. import java.util.HashMap;
  8. import java.util.HashSet;
  9. import java.util.List;
  10. import java.util.Map;
  11. import java.util.Set;
  12.  
  13. import org.apache.commons.lang.Validate;
  14. import org.apache.hadoop.conf.Configuration;
  15. import org.apache.hadoop.fs.FileSystem;
  16. import org.apache.hadoop.fs.Path;
  17. import org.apache.spark.SparkConf;
  18. import org.apache.spark.api.java.JavaRDD;
  19. import org.apache.spark.api.java.JavaSparkContext;
  20. import org.apache.spark.api.java.StorageLevels;
  21. import org.apache.spark.api.java.function.Function0;
  22. import org.apache.spark.broadcast.Broadcast;
  23. import org.apache.spark.sql.Dataset;
  24. import org.apache.spark.sql.Row;
  25. import org.apache.spark.sql.SaveMode;
  26. import org.apache.spark.sql.SparkSession;
  27. import org.apache.spark.streaming.Durations;
  28. import org.apache.spark.streaming.api.java.JavaReceiverInputDStream;
  29. import org.apache.spark.streaming.api.java.JavaStreamingContext;
  30. import org.apache.spark.util.LongAccumulator;
  31. import org.apache.spark.util.SizeEstimator;
  32. import org.apache.spark.streaming.Time;
  33.  
  34. import com.mengyao.graph.etl.apps.commons.beans.dim.RuianMall;
  35. import com.mengyao.graph.etl.apps.commons.datasource.mq.receiver.RocketMQConfig;
  36. import com.mengyao.graph.etl.apps.commons.datasource.mq.receiver.RocketMQReceiver;
  37. import com.mengyao.graph.etl.apps.dashboard.beans.BCD;
  38. import com.mengyao.graph.etl.apps.dashboard.service.SaleAnalysisService;
  39.  
  40. /**
  41.  * BillConsumer 大屏实时计算
  42.  *         1、hdfs dfs -rm -r hdfs://bd001:8020/data/consumer/bill/checkpoint/*
  43.  *         2、hdfs dfs -rm -r hdfs://bd001:8020/data/dashboard/ruian/sdt=curTime
  44.  *         3、spark-submit --class com.mengyao.graph.etl.apps.commons.datasource.bill.BillConsumer --master yarn --deploy-mode cluster --driver-memory 4g --executor-cores 6 --executor-memory 5g --queue default --verbose data-graph-etl.jar
  45.  *
  46.  * @author mengyao
  47.  *
  48.  */
  49. public class BillConsumer {
  50.  
  51.     private static final ThreadLocal<SimpleDateFormat> FORMATTER_YMD = ThreadLocal.withInitial(() -> new SimpleDateFormat("yyyyMMdd"));
  52.     private static final ThreadLocal<SimpleDateFormat> FORMATTER_YMDHMS = ThreadLocal.withInitial(() -> new SimpleDateFormat("yyyyMMddHHmmss"));
  53.     private static final ThreadLocal<SimpleDateFormat> FORMATTER_YMDHMSS = ThreadLocal.withInitial(() -> new SimpleDateFormat("yyyyMMddHHmmssSSS"));
  54.     private static final String BASE_PATH = "hdfs://bd001:8020/data/dashboard/ruian/";
  55.     private static final String TMP_PATH = "/merge";
  56.     private static String appName = "BillConsumer";
  57.     private static String logLevel = "ERROR";
  58.  
  59.     public static void main(String[] args) {
  60.         args = new String[] {"hdfs://bd001:8020/data/consumer/bill/checkpoint", "10", "base.mq.goo.com:9877", "Bill", "bill_dev_group_00013"};
  61.         if (args.length < 5) {
  62.             System.err.println("Usage: "+appName+" <checkpointDir> <milliseconds> <namesrvAddr> <groupId> <topic>");
  63.             System.exit(1);
  64.         }
  65.  
  66.         String checkPointDir = args[0];
  67.         int second = Integer.parseInt(args[1]);
  68.         String namesrv = args[2];
  69.         Validate.notNull(namesrv, "RocketMQ namesrv not null!");
  70.         String topic = args[3];
  71.         Validate.notNull(topic, "RocketMQ topic not null!");
  72.         String group = args[4];
  73.         Validate.notNull(group, "RocketMQ group not null!");
  74.  
  75.         Function0<JavaStreamingContext> jscFunc = () -> createJavaSparkStreamingContext(checkPointDir, second, namesrv, topic, group);
  76.         JavaStreamingContext jssc = JavaStreamingContext.getOrCreate(checkPointDir, jscFunc, new Configuration());
  77.         jssc.sparkContext().setLogLevel(logLevel);
  78.  
  79.         try {
  80.             jssc.start();
  81.             jssc.awaitTermination();
  82.         } catch (InterruptedException e) {
  83.             e.printStackTrace();
  84.         } finally {
  85.             jssc.stop(false, true);//关闭StreamingContext但不关闭SparkContext，同时等待数据处理完成
  86.         }
  87.     }
  88.  
  89.     /**
  90.      * 获取当前时间yyyyMMdd，匹配账单数据saleTime
  91.      * @param curTime
  92.      * @return
  93.      */
  94.     static String getCurrentDate(long curTime) {
  95.         return FORMATTER_YMD.get().format(new Date(curTime));
  96.     }
  97.  
  98.     /**
  99.      * 打印bill RDD<BCD>的分区及占用空间大小，debug方法
  100.      * @param rdd
  101.      * @param time
  102.      */
  103.     static void printStream(JavaRDD<BCD> rdd, Time time) {rdd.id();
  104.         System.out.println("==== time: "+FORMATTER_YMDHMSS.get().format(new Date(time.milliseconds()))+", rdd: partitions="+rdd.getNumPartitions()+", space="+SizeEstimator.estimate(rdd)/1048576+"mb");
  105.     }
  106.  
  107.     /**
  108.      * 合并parquet小文件
  109.      * @param session
  110.      * @param dfsDir
  111.      */
  112.     static void mergeSmallFiles(Dataset<Row> fulls, SparkSession session, String dfsDir) {
  113.         fulls.coalesce(1).write().mode(SaveMode.Overwrite).parquet(BASE_PATH+TMP_PATH);
  114.         session.read().parquet(BASE_PATH+TMP_PATH).coalesce(1).write().mode(SaveMode.Overwrite).parquet(dfsDir);
  115.     }
  116.  
  117.     /**
  118.      * 创建DFS Dir
  119.      * @param session
  120.      * @param dfsDir
  121.      */
  122.     static void mkDfsDir(SparkSession session, String dfsDir) {
  123.         FileSystem fs = null;
  124.         try {
  125.             fs = FileSystem.get(session.sparkContext().hadoopConfiguration());
  126.             Path path = new Path(dfsDir);
  127.             if(!fs.exists(path)) {
  128.                 fs.mkdirs(path);
  129.             }
  130.         } catch (IOException e) {
  131.             e.printStackTrace();
  132.         } finally {
  133.             try {
  134.                 if (null != fs) {fs.close();}
  135.             } catch (IOException e) {
  136.                 e.printStackTrace();
  137.             }
  138.         }
  139.     }
  140.  
  141.     /**
  142.      * 容错Driver
  143.      * @param checkPointDir
  144.      * @param second
  145.      * @param namesrv
  146.      * @param topic
  147.      * @param group
  148.      * @return
  149.      */
  150.     static JavaStreamingContext createJavaSparkStreamingContext(String checkPointDir, int second, String namesrv, String topic, String group) {
  151.         try {
  152.             SparkConf conf = new SparkConf()
  153.                     //==== Enable Back Pressure
  154.                     .set("spark.streaming.backpressure.enabled", "true")//启用被压机制
  155.                     .set("spark.streaming.backpressure.initialRate", "50000")//初始接收数据条数，如该值为空时使用spark.streaming.backpressure.initialRate为默认值
  156.                     .set("spark.streaming.receiver.maxRate", "100")//每秒接收器可接收的最大记录数
  157.                     //==== Enable Dynamic Resource Allocation
  158.                     .set("spark.dynamicAllocation.enabled", "false")//禁用spark动态资源分配
  159.                     .set("spark.streaming.dynamicAllocation.enabled", "true")//启用SparkStreaming动态资源分配，该配置和spark动态资源分配存在冲突，只能使用一个
  160.                     .set("spark.streaming.dynamicAllocation.minExecutors", "2")//启用SparkStreaming动态资源分配后的给应用使用的最小executor数
  161.                     .set("spark.streaming.dynamicAllocation.maxExecutors", "3")//启用SparkStreaming动态资源分配后的给应用使用的最大executor数
  162.                     //==== Spark Streaming Parallelism and WAL
  163.                     .set("spark.streaming.concurrentJobs", "1")//并行job数量，默认1
  164.                     .set("spark.streaming.blockInterval", "5000")//SparkStreaming接收器接收数据后5000毫秒生成block，默认200毫秒
  165.                     .set("spark.streaming.receiver.writeAheadLog.enable", "true")//开启SparkStreaming接收器的WAL来确保接收器实现至少一次的容错语义
  166.                     .set("spark.streaming.driver.writeAheadLog.allowBatching", "true")//driver端WAL
  167.                     .set("spark.streaming.driver.writeAheadLog.batchingTimeout", "15000")
  168.                     //==== Hive on Spark
  169.                     .set("hive.execution.engine", "spark")//设置hive引擎为spark，hdp-2.6.1.0默认支持tez、mr，可通过应用级别配置使用Hive on Spark
  170.                     .set("hive.enable.spark.execution.engine", "true")//启用Hive on Spark
  171.                     .set("spark.driver.extraJavaOptions", "-Dhdp.version=2.6.1.0-129")//hdp-2.6.1.0中要求Hive on Spark必须指定driver的jvm参数
  172.                     .set("spark.yarn.am.extraJavaOptions", "-Dhdp.version=2.6.1.0-129")//hdp-2.6.1.0中要求Hive on Spark必须指定ApplicationMaster的jvm参数
  173.                     //==== Hive Merge Small Files
  174.                     .set("hive.metastore.uris", "thrift://bd001:9083")//hive ThriftServer
  175.                     .set("hive.merge.sparkfiles", "true")//合并spark小文件
  176.                     .set("hive.merge.mapfiles", "true")//在只有map任务的作业结束时合并小文件。
  177.                     .set("hive.merge.mapredfiles", "true")//在mapreduce作业结束时合并小文件。
  178.                     .set("hive.merge.size.per.task", "268435456")//作业结束时合并文件的大小。
  179.                     .set("hive.merge.smallfiles.avgsize", "100000")//当作业的平均输出文件大小小于此数量时，Hive将启动另一个map-reduce作业，以将输出文件合并为更大的文件。如果hive.merge.mapfiles为true，则仅对仅map作业执行此操作;对于hive.merge.mapredfiles为true，仅对map-reduce作业执行此操作。
  180.                     //==== Spark SQL Optimizer
  181.                     .set("spark.sql.warehouse.dir", "hdfs://bd001:8020/apps/hive/warehouse")//SparkSQL依赖的hive仓库地址
  182.                     .set("spark.sql.files.maxPartitionBytes", "134217728")//SparkSQL读取文件数据时打包到一个分区的最大字节数
  183.                     .set("spark.sql.files.openCostInBytes", "134217728")//当SparkSQL读取的文件中有大量小文件时，小于该值的文件将被合并处理，默认4M，此处设置为128M
  184.                     .set("spark.sql.shuffle.partitions", "600")//SparkSQL运行shuffle的并行度
  185.                     .set("spark.sql.autoBroadcastJoinThreshold", "67108864")//设置为64M，执行join时自动广播小于该值的表，默认10M
  186.                     //==== Spark Core Configure
  187.                     .set("spark.rdd.compress","true")//开启rdd压缩以节省内存
  188.                     .set("spark.default.parallelism", "600")//并行任务数
  189.                     .set("spark.rpc.askTimeout", "300")//spark rpc超时时间
  190.                     .set("spark.eventLog.enabled", "true")//开启eventLog
  191.                     //==== Application Configure
  192.                     .set("spark.app.name", appName)//Spark Application名称
  193.                     .set("spark.master", "yarn")//运行模式为Spark on YARN
  194.                     .set("spark.deploy.mode", "cluster")//部署模式为yarn-cluster
  195.                     .set("spark.driver.memory", "4g")//driver内存4g
  196.                     .set("spark.driver.cores", "1")//driver计算vcore数量为1
  197.                     .set("spark.executor.memory", "5g")//executor内存为4g
  198.                     .set("spark.executor.heartbeatInterval", "20000")//executor心跳间隔20秒，默认10秒
  199.                     .set("spark.yarn.archive", "hdfs://bd001:8020/hdp/apps/2.6.1.0-129/spark2")//spark依赖jar存档到hdfs指定位置
  200.                     .set("spark.executor.extraJavaOptions", "-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseConcMarkSweepGC")//打印GC详情和耗时
  201.                     .set("spark.jars", "/usr/hdp/2.6.1.0-129/sqoop/lib/mysql-connector-java.jar")//如果使用了数据库驱动，则通过此配置即可
  202.                     //==== Serialized Configure
  203.                     .set("spark.kryoserializer.buffer", "512k")//默认64k，设置为256k
  204.                     .set("spark.kryoserializer.buffer.max", "256m")//默认64m，设置为256m
  205.                     .set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")//使用kryo序列化库
  206.                     .registerKryoClasses(new Class[]{HashMap.class, BCD.class})
  207.                     ;
  208.  
  209.             //构建MQ配置
  210.             @SuppressWarnings("serial")
  211.             HashMap<String, String> mqConf = new HashMap<String, String>() {{
  212.                 put(RocketMQConfig.NAME_SERVER_ADDR, namesrv);
  213.                 put(RocketMQConfig.CONSUMER_TOPIC, topic);
  214.                 put(RocketMQConfig.CONSUMER_GROUP, group);
  215.             }};
  216.  
  217.             JavaStreamingContext jssc = new JavaStreamingContext(conf, Durations.seconds(second));
  218.             jssc.checkpoint(checkPointDir);
  219.             jssc.remember(Durations.minutes(1440));
  220.  
  221.             //接収RocketMQ账单
  222.             JavaReceiverInputDStream<BCD> billListRDD = jssc.receiverStream(new RocketMQReceiver(mqConf, StorageLevels.MEMORY_AND_DISK_SER));
  223.  
  224.             billListRDD
  225.                 .foreachRDD((rdd, time) -> {
  226.                     boolean isEmpty = rdd.partitions().isEmpty();
  227.                     if (!isEmpty) {
  228.                         printStream(rdd, time);
  229.  
  230.                         long start = System.currentTimeMillis();
  231.                         String curTime = getCurrentDate(start).intern();
  232.                         String dfsDir = (BASE_PATH+"sdt="+curTime+"/").intern();
  233.  
  234.                         //账单计数器
  235.                         //LongAccumulator billAccumulator = BillAccumulator.getInstance(JavaSparkContext.fromSparkContext(rdd.context()), appName);
  236.                         //billAccumulator.add(rdd.count());
  237.  
  238.                         //初始化SparkSession
  239.                         SparkSession session = HiveSession.getInstance(conf);
  240.  
  241.                         //维表等广播
  242.                         BroadcastDIM dim = BroadcastWrapper.getInstance(JavaSparkContext.fromSparkContext(rdd.context())).getValue();
  243.                         SaleAnalysisService service = dim.getService();
  244.                         Dataset<Row> shop = dim.getShop();
  245.                         Dataset<Row> type = dim.getType();
  246.                         Dataset<Row> ruian = dim.getRuian();
  247.                         String mallIdStr = dim.getMallIdStr();
  248.  
  249.                         Set<String> areaSet=dim.getAreaSet();
  250.                         Map<String, Set<String>> areaMall=dim.getAreaMall();
  251.                         Set<String> mallSet=dim.getMallSet();
  252.                         Set<String> typeSet=dim.getTypeSet();
  253.  
  254.                         //如果时间为00:00:00时则认为是新的一天，更新广播数据
  255.                         if ((curTime+"000000").equals(FORMATTER_YMDHMS.get().format(new Date(time.milliseconds())))) {
  256.                             BroadcastWrapper.update(session, JavaSparkContext.fromSparkContext(rdd.context()));//更新dim表数据
  257.                             mkDfsDir(session, dfsDir);//初始化dfsDir
  258.                         }
  259.  
  260.                         //先持久化本次接收到的账单（写入当日）
  261.                         session.createDataFrame(rdd, BCD.class)
  262.                             .filter("sdt = "+curTime)
  263.                             .write()
  264.                             .mode("append")
  265.                             .parquet(dfsDir);
  266.  
  267.                         //再读取全量账单（读取当日）
  268.                         Dataset<Row> bills = session.read().parquet(dfsDir)
  269.                             .filter("shopId in ("+mallIdStr+")")
  270.                             .dropDuplicates("billId")
  271.                             //.coalesce(1)
  272.                             .cache();
  273.  
  274.                         //计算大屏指标
  275.                         System.out.println("==== 计算指标：时间条件："+curTime+" ====");
  276.                         service.totalSale(bills);
  277.                         service.totalRefund(bills);
  278.                         service.peakTime(bills);
  279.                         service.areaSaleTrendForAll(bills, ruian,areaSet, curTime);
  280.                         service.projectContrast(bills, ruian,areaMall);
  281.                         service.saleForShopTop10(bills, shop, ruian);
  282.                         service.projectTypeSaleContrast(bills, shop, type, ruian,areaSet,mallSet,typeSet);
  283.                         long end = System.currentTimeMillis();
  284.                         System.out.println("==== 计算指标：耗时："+(end-start)+"/ms ====");
  285.                         bills.unpersist();
  286.                         //每天最多存6个文件
  287.                         if (bills.inputFiles().length > 6) {
  288.                             mergeSmallFiles(bills, session, dfsDir);
  289.                         }
  290.                     } else {//如果SparkStreaming接收的Batch为空，则不做处理
  291.                         System.out.println("==== rdd is null! ");
  292.                     }
  293.                 });
  294.             return jssc;
  295.         } catch (Exception e) {
  296.             e.printStackTrace();
  297.         }
  298.         return null;
  299.     }
  300.  
  301. }
  302.  
  303. /**
  304.  * 广播DIM相关数据
  305.  * @author mengyao
  306.  *
  307.  */
  308. class BroadcastWrapper {
  309.     private static volatile Broadcast<BroadcastDIM> instance = null;
  310.  
  311.     public static Broadcast<BroadcastDIM> getInstance(JavaSparkContext jsc) {
  312.         if (instance == null) {
  313.             synchronized (BroadcastWrapper.class) {
  314.                 if (instance == null) {
  315.                     SparkSession session = HiveSession.getInstance(jsc.getConf());
  316.                     BroadcastDIM dim = new BroadcastDIM(session);
  317.                     dim.assign();
  318.                     instance = jsc.broadcast(dim);
  319.                 }
  320.             }
  321.         }
  322.         return instance;
  323.     }
  324.  
  325.     /**
  326.      * 每日更新数据
  327.      * @param batchTime    batch发生时间
  328.      * @param dayES    每日开始时间
  329.      */
  330.     public static void update(SparkSession session, JavaSparkContext jsc) {
  331.         BroadcastDIM dim = instance.getValue();
  332.         if (null!=dim) {
  333.             dim.assign();
  334.             jsc.broadcast(dim);
  335.         }
  336.     }
  337. }
  338.  
  339. class BroadcastDIM {
  340.     private SparkSession session;
  341.     private SaleAnalysisService service = new SaleAnalysisService();
  342.     private Dataset<Row> shop;
  343.     private Dataset<Row> type;
  344.     private Dataset<Row> ruian;
  345.     private String mallIdStr;
  346.     private List<RuianMall> ruianList ;
  347.     private Set<String>  areaSet;
  348.     private Set<String>  typeSet;
  349.     private Set<String> mallSet;
  350.     private Map<String, Set<String>> areaMall;
  351.     public BroadcastDIM(SparkSession session) {
  352.         this.session = session;
  353.     }
  354.     public void assign() {
  355.         ruian = session.sql("select id,item,name,area_en,area_cn,channel,mid,rmid from tbl_dim_ruian").cache();
  356.         Row[] ruianRows = (Row[])ruian.collect();
  357.         setRuianList(ruianRows);
  358.         setAreaSet();
  359.         setMallIdStr();
  360.         setRuianMallAll();
  361.         setMallSet();
  362.         shop = session.sql("select id,shop_entity_id,shop_entity_name,shop_id,shop_entity_type_root,shop_entity_type_leaf,bill_match_mode,leasing_model,shop_entity_status,open_time,close_time,open_time_list,close_time_list,monite_begin_time_list,monite_end_time_list,marketing_copywriter,marketing_image,font_style,font_size,contract_area,province,city,area,billhint,is_del,brand,brand_code,classify,in_aera,storey,leasing_resource,shop_entity_source,create_time,c_time_stamp,shop_entity_img,business_area_id,source,status,logo,door_head_photo,business_license,certificate,coordinates,consume_per,brand_name,brand_log,alipay,process  "
  363.                 + "from tbl_ods_shop where shop_id in ("+mallIdStr+")").cache();
  364.         type = session.sql("select * from tbl_ods_type").cache();
  365.         setRuianType();
  366.     }
  367.     public void  setRuianList(Row[] rows) {
  368.         if(rows.length>0){
  369.             ruianList=new ArrayList<>();
  370.             for(Row row:rows){
  371.                 RuianMall rm=new RuianMall();
  372.                 if(!row.isNullAt(0)){//id
  373.                     rm.setId(row.getInt(0));
  374.                 }
  375.                 if(!row.isNullAt(1)){//item
  376.                     rm.setItem(row.getString(1));
  377.                 }
  378.                 if(!row.isNullAt(2)){//name
  379.                     rm.setName(row.getString(2));
  380.                 }
  381.                 if(!row.isNullAt(3)){//area_en
  382.                     rm.setAreaEn(row.getString(3));
  383.                 }
  384.                 if(!row.isNullAt(4)){//area_cn
  385.                     rm.setArenCn(row.getString(4));
  386.                 }
  387.                 if(!row.isNullAt(5)){//channel
  388.                     rm.setChannel(row.getString(5));
  389.                 }
  390.                 if(!row.isNullAt(6)){//mid
  391.                     rm.setMid(row.getInt(6));
  392.                 }
  393.                 if(!row.isNullAt(7)){//rmid
  394.                     rm.setRmid(row.getString(7));
  395.                 }
  396.                 ruianList.add(rm);
  397.             }
  398.         }
  399.     }
  400.     /**
  401.      * 提取rmid 拼接成字符串
  402.      * @param rows
  403.      */
  404.     public void setMallIdStr() {
  405.         if(ruianList.size()>0){
  406.             StringBuilder sbStr=new StringBuilder();
  407.             for(RuianMall row : ruianList){
  408.                 sbStr.append("'").append(row.getRmid()).append("',");
  409.             }
  410.             String tmpValue = sbStr.toString();
  411.             mallIdStr=tmpValue.substring(0, tmpValue.length()-1);
  412.         }
  413.     }
  414.     /**
  415.      * 提取非空唯一中文区域名称
  416.      * @return
  417.      */
  418.     public void setAreaSet() {
  419.         if(ruianList.size()>0){
  420.             areaSet=new java.util.HashSet<>();
  421.             for(RuianMall row:ruianList){
  422.                 areaSet.add(row.getArenCn());
  423.             }
  424.         }
  425.     }
  426.     /**
  427.      * 提取瑞安mall 14个机构中文名
  428.      *
  429.      * @return
  430.      * Map<String,Set<String>>
  431.      * key:areaCn value :  Set<mallName>
  432.      */
  433.     public void setRuianMallAll( ){
  434.         areaMall=new HashMap<>();
  435.         ruianList.forEach(rm->{
  436.             if(areaMall.containsKey(rm.getArenCn())){//存在，更新mall的列表
  437.                 areaMall.get(rm.getArenCn()).add(rm.getName());
  438.             }else{//新的区域，新的mall
  439.                 Set<String> mallSet=new HashSet<>();
  440.                 mallSet.add(rm.getName());
  441.                 areaMall.put(rm.getArenCn(),mallSet);
  442.             }
  443.         });
  444.     }
  445.     /**
  446.      * 提取瑞安mall 14个机构中文名
  447.      *
  448.      * @return
  449.      */
  450.     public void setMallSet(){
  451.         mallSet=new java.util.HashSet<>();
  452.         ruianList.forEach(rm->{
  453.             if(!mallSet.contains(rm.getName())){//存在，更新mall的列表
  454.                 mallSet.add(rm.getName());
  455.             }
  456.         });
  457.     }
  458.     /**
  459.      * 性能、性能、性能 考虑，先写死
  460.      * 如果用账单、店铺、业态关联查询，效率会很慢
  461.      *
  462.      * @return
  463.      */
  464.     public void setRuianType(){
  465.         //TODO 目前写死，需要改
  466.         typeSet=new HashSet<>();
  467.         typeSet.add("服务");
  468.         typeSet.add("零售");
  469.         typeSet.add("娱乐");
  470.         typeSet.add("主力店");
  471.         typeSet.add("餐饮");
  472.         typeSet.add("其它");
  473.     }
  474.     public SaleAnalysisService getService() {
  475.         return service;
  476.     }
  477.     public void setService(SaleAnalysisService service) {
  478.         this.service = service;
  479.     }
  480.     public Dataset<Row> getShop() {
  481.         return shop;
  482.     }
  483.     public void setShop(Dataset<Row> shop) {
  484.         this.shop = shop;
  485.     }
  486.     public Dataset<Row> getType() {
  487.         return type;
  488.     }
  489.     public void setType(Dataset<Row> type) {
  490.         this.type = type;
  491.     }
  492.     public Dataset<Row> getRuian() {
  493.         return ruian;
  494.     }
  495.     public void setRuian(Dataset<Row> ruian) {
  496.         this.ruian = ruian;
  497.     }
  498.     public String getMallIdStr() {
  499.         return mallIdStr;
  500.     }
  501.     public void setMallIdStr(String mallIdStr) {
  502.         this.mallIdStr = mallIdStr;
  503.     }
  504.     public List<RuianMall> getRuianList() {
  505.         return ruianList;
  506.     }
  507.     public Set<String> getAreaSet() {
  508.         return this.areaSet;
  509.     }
  510.     public Map<String, Set<String>> getAreaMall() {
  511.         return this.areaMall;
  512.     }
  513.     public Set<String> getMallSet() {
  514.         return this.mallSet;
  515.     }
  516.     public Set<String> getTypeSet() {
  517.         return this.typeSet;
  518.     }
  519. }
  520.  
  521. /**
  522.  * 账单累加器
  523.  * @author mengyao
  524.  *
  525.  */
  526. class BillAccumulator {
  527.     private static volatile LongAccumulator instance = null;
  528.  
  529.     public static LongAccumulator getInstance(JavaSparkContext jsc, String name) {
  530.         if (instance == null) {
  531.             synchronized (BillAccumulator.class) {
  532.                 if (instance == null) {
  533.                     instance = jsc.sc().longAccumulator(name);
  534.                 }
  535.             }
  536.         }
  537.         return instance;
  538.     }
  539. }
  540.  
  541. /**
  542.  * SparkSQL的Hive数据源
  543.  * @author mengyao
  544.  *
  545.  */
  546. class HiveSession {
  547.     private static transient SparkSession instance = null;
  548.  
  549.     public static SparkSession getInstance(SparkConf conf) {
  550.         if (instance == null) {
  551.             instance = SparkSession.builder()
  552.                     .config(conf)
  553.                     .enableHiveSupport()
  554.                     .getOrCreate();
  555.         }
  556.         return instance;
  557.     }
  558. }

4.4、账单实体类

  1. package com.mengyao.graph.etl.apps.commons.beans.ods.fmt;
  2.  
  3. import java.io.Serializable;
  4. import java.util.ArrayList;
  5. import java.util.List;
  6. import java.util.Map;
  7.  
  8. import org.apache.commons.lang.StringUtils;
  9.  
  10. import com.mengyao.graph.etl.apps.commons.beans.ods.BillInfo;
  11. import com.mengyao.graph.etl.apps.commons.beans.ods.DiscountDetailsInfo;
  12. import com.mengyao.graph.etl.apps.commons.beans.ods.GoodsDetailInfo;
  13. import com.mengyao.graph.etl.apps.commons.beans.ods.SettlementWayInfo;
  14. import com.mengyao.utils.DateTimeUtils;
  15.  
  16. /**
  17.  * 账单信息 gag_bill.bill_info
  18.  *
  19.  * @author jmb
  20.  * @update 2018-12-13 for mengyao
  21.  */
  22. public class BillInfoFmt implements Serializable {
  23.     private static final long serialVersionUID = 1L;
  24.  
  25.     /**
  26.      * 预结单
  27.      */
  28.     public static final String BILLTYPE_JUJIEDAN = "2";
  29.     /**
  30.      * 结账单
  31.      */
  32.     public static final String BILLTYPE_JIEZHANGDAN = "1";
  33.     /**
  34.      * 日结账单
  35.      */
  36.     public static final String BILLTYPE_RIJIEDAN = "3";
  37.     /**
  38.      * 点菜单
  39.      */
  40.     public static final String BILLTYPE_DIANCAIDAN = "7";
  41.  
  42.     /**
  43.      * 账单编号（系统产生），UUID
  44.      */
  45.     private String id;
  46.  
  47.     /**
  48.      * 账单编号（系统产生），HBase主键
  49.      */
  50.     private String rowKey;
  51.  
  52.     /**
  53.      * 商家编号（系统产生）
  54.      */
  55.     private String shopId;
  56.  
  57.     /**
  58.      * 商家名称（系统产生）
  59.      */
  60.     private String shopName;
  61.  
  62.     /**
  63.      * 实体店编号（系统产生）
  64.      */
  65.     private String shopEntityId;
  66.  
  67.     /**
  68.      * 实体店名称（系统产生）
  69.      */
  70.     private String shopEntityName;
  71.  
  72.     /**
  73.      * 创建时间（系统产生）
  74.      */
  75.     private String createTime;
  76.  
  77.     /**
  78.      * 最后一次修改时间（系统产生）
  79.      */
  80.     private String cTimeStamp;
  81.  
  82.     /**
  83.      * 信息上传时间（以header中创建时间为准yyyyMMddHHmmss）
  84.      */
  85.     private String hcTime;
  86.  
  87.     /**
  88.      * 流水号(header中的流水号)
  89.      */
  90.     private String hserial;
  91.  
  92.     /**
  93.      * 修正账单数据的设备编号（如果没有修正，则与原始上传账单的采集终端编号相同）
  94.      */
  95.     private String fixTerminal;
  96.  
  97.     /**
  98.      * 采集终端编号（截获）
  99.      */
  100.     private String terminalNumber;
  101.  
  102.     /**
  103.      * 账单文件名称，唯一（截获），12位MAC地址+17位时间
  104.      */
  105.     private String billfileName;
  106.  
  107.     /**
  108.      * 反扫支付时终端生成的终端流水号UUID（全球唯一）[先支付后打单必填]
  109.      */
  110.     private String tsuuid;
  111.  
  112.     /**
  113.      * 历史账单文件名称，记录合并过程中历次账单文件名称，全量，mongodb为Array
  114.      */
  115.     private List<String> billfileNameHis;
  116.  
  117.     /**
  118.      * 账单序号，不保证唯一（截获）
  119.      */
  120.     private String billNo;
  121.  
  122.     /**
  123.      * 截获时间（截获）
  124.      */
  125.     private String interceptTime;
  126.  
  127.     /**
  128.      * 店铺全名（截获）
  129.      */
  130.     private String shopEntityFullName;
  131.  
  132.     /**
  133.      * 店铺地址（截获）
  134.      */
  135.     private String shopEntityAddress;
  136.  
  137.     /**
  138.      * 电话（截获）
  139.      */
  140.     private String telephone;
  141.  
  142.     /**
  143.      * 售货员（截获）
  144.      */
  145.     private String saler;
  146.  
  147.     /**
  148.      * 收银台（截获）
  149.      */
  150.     private String checkstand;
  151.  
  152.     /**
  153.      * 收银员（截获）
  154.      */
  155.     private String cashier;
  156.  
  157.     /**
  158.      * 应收金额（截获）
  159.      */
  160.     private Double receivableAmount;
  161.  
  162.     /**
  163.      * 原始应收金额（截获）
  164.      */
  165.     private Double defaultReceivableAmount;
  166.  
  167.     /**
  168.      * 商品数量（截获）
  169.      */
  170.     private Double totalNum;
  171.  
  172.     /**
  173.      * 原始商品数量（截获）
  174.      */
  175.     private Double defaultTotalNum;
  176.  
  177.     /**
  178.      * 小票流水号（截获）
  179.      */
  180.     private String billSerialNumber;
  181.  
  182.     /**
  183.      * 总金额（截获）
  184.      */
  185.     private Double totalFee;
  186.  
  187.     /**
  188.      * 原始总金额（截获）
  189.      */
  190.     private Double defaultTotalFee;
  191.  
  192.     /**
  193.      * 实收金额（截获）
  194.      */
  195.     private Double paidAmount;
  196.  
  197.     /**
  198.      * 原始实收金额（截获）
  199.      */
  200.     private Double defaultPaidAmount;
  201.  
  202.     /**
  203.      * 折扣金额（截获）
  204.      */
  205.     private Double discountAmount;
  206.  
  207.     /**
  208.      * 原始折扣金额（截获）
  209.      */
  210.     private Double defaultDiscountAmount;
  211.  
  212.     /**
  213.      * 优惠金额（截获）
  214.      */
  215.     private Double couponAmount;
  216.  
  217.     /**
  218.      * 原始优惠金额（截获）
  219.      */
  220.     private Double defaultCouponAmount;
  221.  
  222.     /**
  223.      * 找零金额（截获）
  224.      */
  225.     private Double changeAmount;
  226.  
  227.     /**
  228.      * 原始找零金额（截获）
  229.      */
  230.     private Double defaultChangeAmount;
  231.  
  232.     /**
  233.      * 结算方式（支持多项）（截获）例：[{"a":5.0,"p":"现金"},{"a":10.01,"p":"书券"}],"a":结算金额，小数[截获，必填],
  234.      * "p":"结算方式[截获，必填]"
  235.      */
  236.     private List<String> settlementWay;
  237.  
  238.     /**
  239.      * 销售时间（截获）
  240.      */
  241.     private String saleTime;
  242.  
  243.     /**
  244.      * 会员卡号（截获）
  245.      */
  246.     private String memberCardNumber;
  247.  
  248.     /**
  249.      * 累计消费（截获）
  250.      */
  251.     private Double totalConsumption;
  252.  
  253.     /**
  254.      * 原始累计消费（截获）
  255.      */
  256.     private Double defaultTotalConsumption;
  257.  
  258.     /**
  259.      * 网址（截获）
  260.      */
  261.     private String website;
  262.  
  263.     /**
  264.      * 小票图片（截获）,只存url
  265.      */
  266.     private String billImage;
  267.  
  268.     /**
  269.      * 商品详情（支持多项）（截获）[{"name":"可乐","itemserial":"PUMU00123","price":5.01,"totalnum":5.0,"totalprice":25.05},{"name":"金枪鱼","itemserial":"FOOD02012","price":10.55,"totalnum":1.5,"totalprice":15.83}]
  270.      * ,"name":"商品名称[截获，选填]","itemserial":"条形码[截获，选填]","price":单价,小数[截获，选填],"totalnum":总数,小数[截获，选填],"totalprice":总价,小数[截获，选填]
  271.      */
  272.     private List<String> goodsDetails;
  273.  
  274.     /**
  275.      * 房间号（截获）（酒店特有）
  276.      */
  277.     private String roomNo;
  278.  
  279.     /**
  280.      * 入住姓名（截获）（酒店特有）
  281.      */
  282.     private String checkinName;
  283.  
  284.     /**
  285.      * 桌号（截获）（餐饮特有）
  286.      */
  287.     private String deskNo;
  288.  
  289.     /**
  290.      * 消费人数（截获）（一般用于餐饮）
  291.      */
  292.     private Double consumeNum;
  293.  
  294.     /**
  295.      * 原始消费人数（截获）（一般用于餐饮）
  296.      */
  297.     private Double defaultConsumeNum;
  298.  
  299.     /**
  300.      * 原始账单所有文本信息(截获)
  301.      */
  302.     private String billText;
  303.  
  304.     /**
  305.      * 入住时间(截获)(酒店特有)
  306.      */
  307.     private String inTime;
  308.  
  309.     /**
  310.      * 离店时间(截获)(钟点房特有)
  311.      */
  312.     private String outTime;
  313.     /**
  314.      * 默认打印时间
  315.      */
  316.     private String defaultPrintDate;
  317.     /**
  318.      * 默认入住时间
  319.      */
  320.     private String defaultInTime;
  321.     /**
  322.      * 默认离店时间
  323.      */
  324.     private String defaultOutTime;
  325.  
  326.     /**
  327.      * 上传类型 1：全单上传 2. 筛选账单上传
  328.      */
  329.     private String uploadType;
  330.  
  331.     /**
  332.      * 除了上面截获外的自定义数据（截获），json串（Map<String, Object>）,json串中的key,value由采集终端自定义。
  333.      */
  334.     private Map<String, String> customRecord;
  335.  
  336.     /**
  337.      * 账单类型1:结账单2:预结单 3:日结单 4:处方 5:预付押金单 6:退货单 7:点菜单 8:发票单 [选填]
  338.      */
  339.     private String billType;
  340.  
  341.     /**
  342.      * 账单修改方式 0:默认值 1:收银员重打 2:人工修改金额 3:自动重解析
  343.      */
  344.     private String modifyType = "0";
  345.  
  346.     /**
  347.      * 账单来源 1:设备采集 2:人工补录3、解析服务 4.第三方
  348.      */
  349.     private String billSource = "1";
  350.  
  351.     /**
  352.      * 服务端解析路径[选填，用于服务端解析分析问题]
  353.      */
  354.     private String analyzPath;
  355.  
  356.     /**
  357.      * 刷卡，钱包等匹配账单的key,格式BILL_MAC_金额_截获时间
  358.      */
  359.     private String billMatchKey;
  360.  
  361.     /**
  362.      * 数据匹配支付结果等成功后,此字段保存支付结果等的主键 [选填] 匹配成功后为必填,理论上应该保证此值为全局唯一
  363.      */
  364.     private String matchId;
  365.  
  366.     /**
  367.      * 默认销售时间;
  368.      */
  369.     private String defaultSaleTime;
  370.  
  371.     /**
  372.      * 客户名称[截获，选填]
  373.      */
  374.     private String customerName;
  375.     /**
  376.      * 会员编号[截获，选填]
  377.      */
  378.     private String membershipId;
  379.     /**
  380.      * 会员级别[截获，选填]
  381.      */
  382.     private String memberLevels;
  383.     /**
  384.      * 宠物名称[截获，选填]
  385.      */
  386.     private String petName;
  387.     /**
  388.      * 宠物编号[截获，选填]
  389.      */
  390.     private String petNumber;
  391.     /**
  392.      * 负责人（医生）[截获，选填]
  393.      */
  394.     private String principal;
  395.     /**
  396.      * 预交押金[截获，选填]
  397.      */
  398.     private String deposit;
  399.     /**
  400.      * 打印时间yyyyMMddHHmmss[截获，选填，如果截获位数不够，后面补0]
  401.      */
  402.     private String printDate;
  403.     /**
  404.      * 默认拦截时间
  405.      */
  406.     private String defaultInterceptTime;
  407.     /**
  408.      * 匹配模型 sk0001:先刷卡后打单,单次刷卡; dd0001:先打单后刷卡,单次刷卡
  409.      */
  410.     private String printMatchType;
  411.  
  412.     /**
  413.      * 账单合并时使用,唯一
  414.      */
  415.     private String billMergeKey;
  416.     /**
  417.      * 存重打单的应收金额
  418.      */
  419.     private Double modifyAmount;
  420.     /**
  421.      * 存重打单的应收金额List
  422.      */
  423.     private List<Double> modifyAmountList;
  424.     /**
  425.      * 存重打单的应收金额的销售时间List
  426.      */
  427.     private List<String> modifyAmountSaleTimeList;
  428.     /**
  429.      * 存重打单的应收金额的截获时间List
  430.      */
  431.     private List<String> modifyAmountInterceptTimeList;
  432.     /**
  433.      * 唯一标识[选填]（目前可用来做积分标识使用）
  434.      */
  435.     private String uniqueId;
  436.     /**
  437.      * 历史唯一标识，记录合并过程中历次唯一标识，全量，mongodb为Array（目前可用来做积分标识使用）
  438.      */
  439.     private List<String> uniqueIdHis;
  440.     /**
  441.      * 是否追加二维码 1：是 2：否 [必填]
  442.      */
  443.     private String ifqrcode;
  444.     /**
  445.      * 优惠券券码
  446.      */
  447.     private String couponNum;
  448.     /**
  449.      * ERP会员
  450.      */
  451.     private String erpMemberCard;
  452.     /**
  453.      * 凭证类型 11:水单（商户pos、ERP打印）12:收银凭证（大pos打印） 13:支付凭证（签购单）99：类型未知 [选填，默认写99，表示未知]
  454.      */
  455.     private String voucherType;
  456.     /**
  457.      * 小票二维码[选填]
  458.      */
  459.     private String qrCode;
  460.     /**
  461.      * 积分标识 0：本次积分字段没找到或没有配置 1：有本次积分字段[选填]
  462.      */
  463.     private String integralmark;
  464.     /**
  465.      * 本次积分[选填]
  466.      */
  467.     private String thisintegral;
  468.     /**
  469.      * 备注[选填]
  470.      */
  471.     private String remarks;
  472.     /**
  473.      * 账单子类型，庖丁用于配置文件分类
  474.      */
  475.     private String templateName;
  476.     /**
  477.      * 购货方名称[截获，选填]
  478.      */
  479.     private String custName;
  480.     /**
  481.      * 购货方税号[截获，选填]
  482.      */
  483.     private String custTaxNo;
  484.     /**
  485.      * 购货方地址、电话[截获，选填]
  486.      */
  487.     private String custAdress;
  488.     /**
  489.      * 购货方银行及账号[截获，选填]
  490.      */
  491.     private String custBankAccount;
  492.  
  493.     /**
  494.      * 第三方订单号(第三方系统唯一)
  495.      */
  496.     private String thirdPartyOrderNo;
  497.     /**
  498.      * 充值卡消费金额[截获，选填]
  499.      */
  500.     private Double rechargeableCardConsumeAmount;
  501.     /**
  502.      * 原始充值卡消费金额[截获，选填]
  503.      */
  504.     private Double defaultRechargeableCardConsumeAmount;
  505.     /**
  506.      * 实付金额[截获，选填]
  507.      */
  508.     private Double outOfPocketAmount;
  509.     /**
  510.      * 原始实付金额[截获，选填]
  511.      */
  512.     private Double defaultOutOfPocketAmount;
  513.     /**
  514.      * 充值金额[截获，选填]
  515.      */
  516.     private Double rechargeAmount;
  517.     /**
  518.      * 原始充值金额[截获，选填]
  519.      */
  520.     private Double defaultRechargeAmount;
  521.     /**
  522.      * 会员价[截获，选填]
  523.      */
  524.     private Double memberPrice;
  525.     /**
  526.      * 原始会员价[截获，选填]
  527.      */
  528.     private Double defaultMemberPrice;
  529.     /**
  530.      * 会员折扣率[截获，选填]
  531.      */
  532.     private Double memberDiscountrate;
  533.     /**
  534.      * 原始会员折扣率[截获，选填]
  535.      */
  536.     private Double defaultMemberDiscountrate;
  537.     /**
  538.      * 会员累计消费[截获，选填]
  539.      */
  540.     private Double memberTotalConsumption;
  541.     /**
  542.      * 原始会员累计消费[截获，选填]
  543.      */
  544.     private Double defaultMemberTotalConsumption;
  545.     /**
  546.      * 外卖单 1:外卖单 2:非外卖单[截获，选填]
  547.      */
  548.     private String takeout;
  549.  
  550.     /**
  551.      * 优惠商品详情（支持多项）（截获），{"name":"可乐","price":5.01}
  552.      */
  553.     private List<String> discountDetails;
  554.  
  555.     public String getThirdPartyOrderNo() {
  556.         return thirdPartyOrderNo;
  557.     }
  558.  
  559.     public void setThirdPartyOrderNo(String thirdPartyOrderNo) {
  560.         this.thirdPartyOrderNo = thirdPartyOrderNo;
  561.     }
  562.  
  563.     public String getRemarks() {
  564.         return remarks;
  565.     }
  566.  
  567.     public void setRemarks(String remarks) {
  568.         this.remarks = remarks;
  569.     }
  570.  
  571.     public String getTemplateName() {
  572.         return templateName;
  573.     }
  574.  
  575.     public void setTemplateName(String templateName) {
  576.         this.templateName = templateName;
  577.     }
  578.  
  579.     public String getDefaultPrintDate() {
  580.         return this.defaultPrintDate;
  581.     }
  582.  
  583.     public void setDefaultPrintDate(String defaultPrintDate) {
  584.         this.defaultPrintDate = defaultPrintDate;
  585.     }
  586.  
  587.     public String getDefaultInTime() {
  588.         return this.defaultInTime;
  589.     }
  590.  
  591.     public void setDefaultInTime(String defaultInTime) {
  592.         this.defaultInTime = defaultInTime;
  593.     }
  594.  
  595.     public String getDefaultOutTime() {
  596.         return this.defaultOutTime;
  597.     }
  598.  
  599.     public void setDefaultOutTime(String defaultOutTime) {
  600.         this.defaultOutTime = defaultOutTime;
  601.     }
  602.  
  603.     public String getCouponNum() {
  604.         return this.couponNum;
  605.     }
  606.  
  607.     public void setCouponNum(String couponNum) {
  608.         this.couponNum = couponNum;
  609.     }
  610.  
  611.     public String getErpMemberCard() {
  612.         return this.erpMemberCard;
  613.     }
  614.  
  615.     public void setErpMemberCard(String erpMemberCard) {
  616.         this.erpMemberCard = erpMemberCard;
  617.     }
  618.  
  619.     public String getVoucherType() {
  620.         return this.voucherType;
  621.     }
  622.  
  623.     public void setVoucherType(String voucherType) {
  624.         this.voucherType = voucherType;
  625.     }
  626.  
  627.     public String getAnalyzPath() {
  628.         return this.analyzPath;
  629.     }
  630.  
  631.     public void setAnalyzPath(String analyzPath) {
  632.         this.analyzPath = analyzPath;
  633.     }
  634.  
  635.     public List<Double> getModifyAmountList() {
  636.         return this.modifyAmountList;
  637.     }
  638.  
  639.     public String getUniqueId() {
  640.         return this.uniqueId;
  641.     }
  642.  
  643.     public void setUniqueId(String uniqueId) {
  644.         this.uniqueId = uniqueId;
  645.     }
  646.  
  647.     public String getIfqrcode() {
  648.         return this.ifqrcode;
  649.     }
  650.  
  651.     public void setIfqrcode(String ifqrcode) {
  652.         this.ifqrcode = ifqrcode;
  653.     }
  654.  
  655.     public void setModifyAmountList(List<Double> modifyAmountList) {
  656.         this.modifyAmountList = modifyAmountList;
  657.     }
  658.  
  659.     public Double getModifyAmount() {
  660.         return this.modifyAmount;
  661.     }
  662.  
  663.     public void setModifyAmount(Double modifyAmount) {
  664.         this.modifyAmount = modifyAmount;
  665.     }
  666.  
  667.     public String getDefaultSaleTime() {
  668.         return this.defaultSaleTime;
  669.     }
  670.  
  671.     public void setDefaultSaleTime(String defaultSaleTime) {
  672.         this.defaultSaleTime = defaultSaleTime;
  673.     }
  674.  
  675.     public String getBillMergeKey() {
  676.         return this.billMergeKey;
  677.     }
  678.  
  679.     public void setBillMergeKey(String billMergeKey) {
  680.         this.billMergeKey = billMergeKey;
  681.     }
  682.  
  683.     public String getCustName() {
  684.         return custName;
  685.     }
  686.  
  687.     public void setCustName(String custName) {
  688.         this.custName = custName;
  689.     }
  690.  
  691.     public String getCustTaxNo() {
  692.         return custTaxNo;
  693.     }
  694.  
  695.     public void setCustTaxNo(String custTaxNo) {
  696.         this.custTaxNo = custTaxNo;
  697.     }
  698.  
  699.     public String getCustAdress() {
  700.         return custAdress;
  701.     }
  702.  
  703.     public void setCustAdress(String custAdress) {
  704.         this.custAdress = custAdress;
  705.     }
  706.  
  707.     public String getCustBankAccount() {
  708.         return custBankAccount;
  709.     }
  710.  
  711.     public void setCustBankAccount(String custBankAccount) {
  712.         this.custBankAccount = custBankAccount;
  713.     }
  714.  
  715.     public List<String> getUniqueIdHis() {
  716.         return uniqueIdHis;
  717.     }
  718.  
  719.     public void setUniqueIdHis(List<String> uniqueIdHis) {
  720.         this.uniqueIdHis = uniqueIdHis;
  721.     }
  722.  
  723.     public Double getRechargeableCardConsumeAmount() {
  724.         return rechargeableCardConsumeAmount;
  725.     }
  726.  
  727.     public void setRechargeableCardConsumeAmount(Double rechargeableCardConsumeAmount) {
  728.         this.rechargeableCardConsumeAmount = rechargeableCardConsumeAmount;
  729.     }
  730.  
  731.     public Double getDefaultRechargeableCardConsumeAmount() {
  732.         return defaultRechargeableCardConsumeAmount;
  733.     }
  734.  
  735.     public void setDefaultRechargeableCardConsumeAmount(Double defaultRechargeableCardConsumeAmount) {
  736.         this.defaultRechargeableCardConsumeAmount = defaultRechargeableCardConsumeAmount;
  737.     }
  738.  
  739.     public Double getOutOfPocketAmount() {
  740.         return outOfPocketAmount;
  741.     }
  742.  
  743.     public void setOutOfPocketAmount(Double outOfPocketAmount) {
  744.         this.outOfPocketAmount = outOfPocketAmount;
  745.     }
  746.  
  747.     public Double getDefaultOutOfPocketAmount() {
  748.         return defaultOutOfPocketAmount;
  749.     }
  750.  
  751.     public void setDefaultOutOfPocketAmount(Double defaultOutOfPocketAmount) {
  752.         this.defaultOutOfPocketAmount = defaultOutOfPocketAmount;
  753.     }
  754.  
  755.     public Double getRechargeAmount() {
  756.         return rechargeAmount;
  757.     }
  758.  
  759.     public void setRechargeAmount(Double rechargeAmount) {
  760.         this.rechargeAmount = rechargeAmount;
  761.     }
  762.  
  763.     public Double getDefaultRechargeAmount() {
  764.         return defaultRechargeAmount;
  765.     }
  766.  
  767.     public void setDefaultRechargeAmount(Double defaultRechargeAmount) {
  768.         this.defaultRechargeAmount = defaultRechargeAmount;
  769.     }
  770.  
  771.     public Double getMemberPrice() {
  772.         return memberPrice;
  773.     }
  774.  
  775.     public void setMemberPrice(Double memberPrice) {
  776.         this.memberPrice = memberPrice;
  777.     }
  778.  
  779.     public Double getDefaultMemberPrice() {
  780.         return defaultMemberPrice;
  781.     }
  782.  
  783.     public void setDefaultMemberPrice(Double defaultMemberPrice) {
  784.         this.defaultMemberPrice = defaultMemberPrice;
  785.     }
  786.  
  787.     public Double getMemberDiscountrate() {
  788.         return memberDiscountrate;
  789.     }
  790.  
  791.     public void setMemberDiscountrate(Double memberDiscountrate) {
  792.         this.memberDiscountrate = memberDiscountrate;
  793.     }
  794.  
  795.     public Double getDefaultMemberDiscountrate() {
  796.         return defaultMemberDiscountrate;
  797.     }
  798.  
  799.     public void setDefaultMemberDiscountrate(Double defaultMemberDiscountrate) {
  800.         this.defaultMemberDiscountrate = defaultMemberDiscountrate;
  801.     }
  802.  
  803.     public Double getMemberTotalConsumption() {
  804.         return memberTotalConsumption;
  805.     }
  806.  
  807.     public void setMemberTotalConsumption(Double memberTotalConsumption) {
  808.         this.memberTotalConsumption = memberTotalConsumption;
  809.     }
  810.  
  811.     public Double getDefaultMemberTotalConsumption() {
  812.         return defaultMemberTotalConsumption;
  813.     }
  814.  
  815.     public void setDefaultMemberTotalConsumption(Double defaultMemberTotalConsumption) {
  816.         this.defaultMemberTotalConsumption = defaultMemberTotalConsumption;
  817.     }
  818.  
  819.     public String getTakeout() {
  820.         return takeout;
  821.     }
  822.  
  823.     public void setTakeout(String takeout) {
  824.         this.takeout = takeout;
  825.     }
  826.  
  827.     public List<String> getDiscountDetails() {
  828.         return discountDetails;
  829.     }
  830.  
  831.     public void setDiscountDetails(List<String> discountDetails) {
  832.         this.discountDetails = discountDetails;
  833.     }
  834.  
  835.     public String getId() {
  836.         return this.id;
  837.     }
  838.  
  839.     public void setId(String id) {
  840.         this.id = id;
  841.     }
  842.  
  843.     public String getShopId() {
  844.         return this.shopId;
  845.     }
  846.  
  847.     public void setShopId(String shopId) {
  848.         this.shopId = shopId;
  849.     }
  850.  
  851.     public String getShopName() {
  852.         return this.shopName;
  853.     }
  854.  
  855.     public void setShopName(String shopName) {
  856.         this.shopName = shopName;
  857.     }
  858.  
  859.     public String getShopEntityId() {
  860.         return this.shopEntityId;
  861.     }
  862.  
  863.     public void setShopEntityId(String shopEntityId) {
  864.         this.shopEntityId = shopEntityId;
  865.     }
  866.  
  867.     public String getShopEntityName() {
  868.         return this.shopEntityName;
  869.     }
  870.  
  871.     public void setShopEntityName(String shopEntityName) {
  872.         this.shopEntityName = shopEntityName;
  873.     }
  874.  
  875.     public String getCreateTime() {
  876.         return this.createTime;
  877.     }
  878.  
  879.     public void setCreateTime(String createTime) {
  880.         this.createTime = createTime;
  881.     }
  882.  
  883.     public String getCTimeStamp() {
  884.         return this.cTimeStamp;
  885.     }
  886.  
  887.     public void setCTimeStamp(String cTimeStamp) {
  888.         this.cTimeStamp = cTimeStamp;
  889.     }
  890.  
  891.     public String getHcTime() {
  892.         return this.hcTime;
  893.     }
  894.  
  895.     public void setHcTime(String hcTime) {
  896.         this.hcTime = hcTime;
  897.     }
  898.  
  899.     public String getHserial() {
  900.         return this.hserial;
  901.     }
  902.  
  903.     public void setHserial(String hserial) {
  904.         this.hserial = hserial;
  905.     }
  906.  
  907.     public String getFixTerminal() {
  908.         return this.fixTerminal;
  909.     }
  910.  
  911.     public void setFixTerminal(String fixTerminal) {
  912.         this.fixTerminal = fixTerminal;
  913.     }
  914.  
  915.     public String getTerminalNumber() {
  916.         return this.terminalNumber;
  917.     }
  918.  
  919.     public void setTerminalNumber(String terminalNumber) {
  920.         this.terminalNumber = terminalNumber;
  921.     }
  922.  
  923.     public String getBillfileName() {
  924.         return this.billfileName;
  925.     }
  926.  
  927.     public void setBillfileName(String billfileName) {
  928.         this.billfileName = billfileName;
  929.     }
  930.  
  931.     public List<String> getBillfileNameHis() {
  932.         return this.billfileNameHis;
  933.     }
  934.  
  935.     public void setBillfileNameHis(List<String> billfileNameHis) {
  936.         this.billfileNameHis = billfileNameHis;
  937.     }
  938.  
  939.     public String getBillNo() {
  940.         return this.billNo;
  941.     }
  942.  
  943.     public void setBillNo(String billNo) {
  944.         this.billNo = billNo;
  945.     }
  946.  
  947.     public String getInterceptTime() {
  948.         return this.interceptTime;
  949.     }
  950.  
  951.     public void setInterceptTime(String interceptTime) {
  952.         this.interceptTime = interceptTime;
  953.     }
  954.  
  955.     public String getShopEntityFullName() {
  956.         return this.shopEntityFullName;
  957.     }
  958.  
  959.     public void setShopEntityFullName(String shopEntityFullName) {
  960.         this.shopEntityFullName = shopEntityFullName;
  961.     }
  962.  
  963.     public String getShopEntityAddress() {
  964.         return this.shopEntityAddress;
  965.     }
  966.  
  967.     public void setShopEntityAddress(String shopEntityAddress) {
  968.         this.shopEntityAddress = shopEntityAddress;
  969.     }
  970.  
  971.     public String getTelephone() {
  972.         return this.telephone;
  973.     }
  974.  
  975.     public void setTelephone(String telephone) {
  976.         this.telephone = telephone;
  977.     }
  978.  
  979.     public String getSaler() {
  980.         return this.saler;
  981.     }
  982.  
  983.     public void setSaler(String saler) {
  984.         this.saler = saler;
  985.     }
  986.  
  987.     public String getCheckstand() {
  988.         return this.checkstand;
  989.     }
  990.  
  991.     public void setCheckstand(String checkstand) {
  992.         this.checkstand = checkstand;
  993.     }
  994.  
  995.     public String getCashier() {
  996.         return this.cashier;
  997.     }
  998.  
  999.     public void setCashier(String cashier) {
  1000.         this.cashier = cashier;
  1001.     }
  1002.  
  1003.     public Double getReceivableAmount() {
  1004.         return this.receivableAmount;
  1005.     }
  1006.  
  1007.     public void setReceivableAmount(Double receivableAmount) {
  1008.         this.receivableAmount = receivableAmount;
  1009.     }
  1010.  
  1011.     public Double getTotalNum() {
  1012.         return this.totalNum;
  1013.     }
  1014.  
  1015.     public void setTotalNum(Double totalNum) {
  1016.         this.totalNum = totalNum;
  1017.     }
  1018.  
  1019.     public String getBillSerialNumber() {
  1020.         return this.billSerialNumber;
  1021.     }
  1022.  
  1023.     public void setBillSerialNumber(String billSerialNumber) {
  1024.         this.billSerialNumber = billSerialNumber;
  1025.     }
  1026.  
  1027.     public Double getTotalFee() {
  1028.         return this.totalFee;
  1029.     }
  1030.  
  1031.     public void setTotalFee(Double totalFee) {
  1032.         this.totalFee = totalFee;
  1033.     }
  1034.  
  1035.     public Double getPaidAmount() {
  1036.         return this.paidAmount;
  1037.     }
  1038.  
  1039.     public void setPaidAmount(Double paidAmount) {
  1040.         this.paidAmount = paidAmount;
  1041.     }
  1042.  
  1043.     public Double getDiscountAmount() {
  1044.         return this.discountAmount;
  1045.     }
  1046.  
  1047.     public void setDiscountAmount(Double discountAmount) {
  1048.         this.discountAmount = discountAmount;
  1049.     }
  1050.  
  1051.     public Double getCouponAmount() {
  1052.         return this.couponAmount;
  1053.     }
  1054.  
  1055.     public void setCouponAmount(Double couponAmount) {
  1056.         this.couponAmount = couponAmount;
  1057.     }
  1058.  
  1059.     public Double getChangeAmount() {
  1060.         return this.changeAmount;
  1061.     }
  1062.  
  1063.     public void setChangeAmount(Double changeAmount) {
  1064.         this.changeAmount = changeAmount;
  1065.     }
  1066.  
  1067.     public List<String> getSettlementWay() {
  1068.         return this.settlementWay;
  1069.     }
  1070.  
  1071.     public void setSettlementWay(List<String> settlementWay) {
  1072.         this.settlementWay = settlementWay;
  1073.     }
  1074.  
  1075.     public String getSaleTime() {
  1076.         return saleTime;
  1077.     }
  1078.  
  1079.     public void setSaleTime(String saleTime) {
  1080.         this.saleTime = saleTime;
  1081.     }
  1082.  
  1083.     public String getMemberCardNumber() {
  1084.         return this.memberCardNumber;
  1085.     }
  1086.  
  1087.     public void setMemberCardNumber(String memberCardNumber) {
  1088.         this.memberCardNumber = memberCardNumber;
  1089.     }
  1090.  
  1091.     public Double getTotalConsumption() {
  1092.         return this.totalConsumption;
  1093.     }
  1094.  
  1095.     public void setTotalConsumption(Double totalConsumption) {
  1096.         this.totalConsumption = totalConsumption;
  1097.     }
  1098.  
  1099.     public String getWebsite() {
  1100.         return this.website;
  1101.     }
  1102.  
  1103.     public void setWebsite(String website) {
  1104.         this.website = website;
  1105.     }
  1106.  
  1107.     public String getBillImage() {
  1108.         return this.billImage;
  1109.     }
  1110.  
  1111.     public void setBillImage(String billImage) {
  1112.         this.billImage = billImage;
  1113.     }
  1114.  
  1115.     public List<String> getGoodsDetails() {
  1116.         return this.goodsDetails;
  1117.     }
  1118.  
  1119.     public void setGoodsDetails(List<String> goodsDetails) {
  1120.         this.goodsDetails = goodsDetails;
  1121.     }
  1122.  
  1123.     public String getRoomNo() {
  1124.         return this.roomNo;
  1125.     }
  1126.  
  1127.     public void setRoomNo(String roomNo) {
  1128.         this.roomNo = roomNo;
  1129.     }
  1130.  
  1131.     public String getCheckinName() {
  1132.         return this.checkinName;
  1133.     }
  1134.  
  1135.     public void setCheckinName(String checkinName) {
  1136.         this.checkinName = checkinName;
  1137.     }
  1138.  
  1139.     public String getDeskNo() {
  1140.         return this.deskNo;
  1141.     }
  1142.  
  1143.     public void setDeskNo(String deskNo) {
  1144.         this.deskNo = deskNo;
  1145.     }
  1146.  
  1147.     public Double getConsumeNum() {
  1148.         return this.consumeNum;
  1149.     }
  1150.  
  1151.     public void setConsumeNum(Double consumeNum) {
  1152.         this.consumeNum = consumeNum;
  1153.     }
  1154.  
  1155.     public String getBillText() {
  1156.         return this.billText;
  1157.     }
  1158.  
  1159.     public void setBillText(String billText) {
  1160.          this.billText = billText;
  1161.     }
  1162.  
  1163.     public Map<String, String> getCustomRecord() {
  1164.         return this.customRecord;
  1165.     }
  1166.  
  1167.     public void setCustomRecord(Map<String, String> customRecord) {
  1168.         this.customRecord = customRecord;
  1169.     }
  1170.  
  1171.     public String getBillType() {
  1172.         return this.billType;
  1173.     }
  1174.  
  1175.     public void setBillType(String billType) {
  1176.         this.billType = billType;
  1177.     }
  1178.  
  1179.     public String getInTime() {
  1180.         return this.inTime;
  1181.     }
  1182.  
  1183.     public String getOutTime() {
  1184.         return this.outTime;
  1185.     }
  1186.  
  1187.     public void setInTime(String inTime) {
  1188.         this.inTime = inTime;
  1189.     }
  1190.  
  1191.     public void setOutTime(String outTime) {
  1192.         this.outTime = outTime;
  1193.     }
  1194.  
  1195.     public String getBillMatchKey() {
  1196.         return this.billMatchKey;
  1197.     }
  1198.  
  1199.     public void setBillMatchKey(String billMatchKey) {
  1200.         this.billMatchKey = billMatchKey;
  1201.     }
  1202.  
  1203.     public String getMatchId() {
  1204.         return this.matchId;
  1205.     }
  1206.  
  1207.     public void setMatchId(String matchId) {
  1208.         this.matchId = matchId;
  1209.     }
  1210.  
  1211.     public String getPrintMatchType() {
  1212.         return this.printMatchType;
  1213.     }
  1214.  
  1215.     public void setPrintMatchType(String printMatchType) {
  1216.         this.printMatchType = printMatchType;
  1217.     }
  1218.  
  1219.     public String getTsuuid() {
  1220.         return this.tsuuid;
  1221.     }
  1222.  
  1223.     public void setTsuuid(String tsuuid) {
  1224.         this.tsuuid = tsuuid;
  1225.     }
  1226.  
  1227.     public String getUploadType() {
  1228.         return this.uploadType;
  1229.     }
  1230.  
  1231.     public void setUploadType(String uploadType) {
  1232.         this.uploadType = uploadType;
  1233.     }
  1234.  
  1235.     public String getCustomerName() {
  1236.         return this.customerName;
  1237.     }
  1238.  
  1239.     public void setCustomerName(String customerName) {
  1240.         this.customerName = customerName;
  1241.     }
  1242.  
  1243.     public String getMembershipId() {
  1244.         return this.membershipId;
  1245.     }
  1246.  
  1247.     public void setMembershipId(String membershipId) {
  1248.         this.membershipId = membershipId;
  1249.     }
  1250.  
  1251.     public String getMemberLevels() {
  1252.         return this.memberLevels;
  1253.     }
  1254.  
  1255.     public void setMemberLevels(String memberLevels) {
  1256.         this.memberLevels = memberLevels;
  1257.     }
  1258.  
  1259.     public String getPetName() {
  1260.         return this.petName;
  1261.     }
  1262.  
  1263.     public void setPetName(String petName) {
  1264.         this.petName = petName;
  1265.     }
  1266.  
  1267.     public String getPetNumber() {
  1268.         return this.petNumber;
  1269.     }
  1270.  
  1271.     public void setPetNumber(String petNumber) {
  1272.         this.petNumber = petNumber;
  1273.     }
  1274.  
  1275.     public String getPrincipal() {
  1276.         return this.principal;
  1277.     }
  1278.  
  1279.     public void setPrincipal(String principal) {
  1280.         this.principal = principal;
  1281.     }
  1282.  
  1283.     public String getDeposit() {
  1284.         return this.deposit;
  1285.     }
  1286.  
  1287.     public void setDeposit(String deposit) {
  1288.         this.deposit = deposit;
  1289.     }
  1290.  
  1291.     public String getPrintDate() {
  1292.         return this.printDate;
  1293.     }
  1294.  
  1295.     public void setPrintDate(String printDate) {
  1296.         this.printDate = printDate;
  1297.     }
  1298.  
  1299.     public String getDefaultInterceptTime() {
  1300.         return this.defaultInterceptTime;
  1301.     }
  1302.  
  1303.     public void setDefaultInterceptTime(String defaultInterceptTime) {
  1304.         this.defaultInterceptTime = defaultInterceptTime;
  1305.     }
  1306.  
  1307.     public String getModifyType() {
  1308.         return this.modifyType;
  1309.     }
  1310.  
  1311.     public void setModifyType(String modifyType) {
  1312.         this.modifyType = modifyType;
  1313.     }
  1314.  
  1315.     public String getBillSource() {
  1316.         return this.billSource;
  1317.     }
  1318.  
  1319.     public void setBillSource(String billSource) {
  1320.         this.billSource = billSource;
  1321.     }
  1322.  
  1323.     public String getQrCode() {
  1324.         return qrCode;
  1325.     }
  1326.  
  1327.     public void setQrCode(String qrCode) {
  1328.         this.qrCode = qrCode;
  1329.     }
  1330.  
  1331.     public String getIntegralmark() {
  1332.         return integralmark;
  1333.     }
  1334.  
  1335.     public void setIntegralmark(String integralmark) {
  1336.         this.integralmark = integralmark;
  1337.     }
  1338.  
  1339.     public String getThisintegral() {
  1340.         return thisintegral;
  1341.     }
  1342.  
  1343.     public void setThisintegral(String thisintegral) {
  1344.         this.thisintegral = thisintegral;
  1345.     }
  1346.  
  1347.     public List<String> getModifyAmountSaleTimeList() {
  1348.         return modifyAmountSaleTimeList;
  1349.     }
  1350.  
  1351.     public void setModifyAmountSaleTimeList(List<String> modifyAmountSaleTimeList) {
  1352.         this.modifyAmountSaleTimeList = modifyAmountSaleTimeList;
  1353.     }
  1354.  
  1355.     public List<String> getModifyAmountInterceptTimeList() {
  1356.         return modifyAmountInterceptTimeList;
  1357.     }
  1358.  
  1359.     public void setModifyAmountInterceptTimeList(List<String> modifyAmountInterceptTimeList) {
  1360.         this.modifyAmountInterceptTimeList = modifyAmountInterceptTimeList;
  1361.     }
  1362.  
  1363.     public Double getDefaultReceivableAmount() {
  1364.         return defaultReceivableAmount;
  1365.     }
  1366.  
  1367.     public void setDefaultReceivableAmount(Double defaultReceivableAmount) {
  1368.         this.defaultReceivableAmount = defaultReceivableAmount;
  1369.     }
  1370.  
  1371.     public Double getDefaultTotalNum() {
  1372.         return defaultTotalNum;
  1373.     }
  1374.  
  1375.     public void setDefaultTotalNum(Double defaultTotalNum) {
  1376.         this.defaultTotalNum = defaultTotalNum;
  1377.     }
  1378.  
  1379.     public Double getDefaultTotalFee() {
  1380.         return defaultTotalFee;
  1381.     }
  1382.  
  1383.     public void setDefaultTotalFee(Double defaultTotalFee) {
  1384.         this.defaultTotalFee = defaultTotalFee;
  1385.     }
  1386.  
  1387.     public Double getDefaultPaidAmount() {
  1388.         return defaultPaidAmount;
  1389.     }
  1390.  
  1391.     public void setDefaultPaidAmount(Double defaultPaidAmount) {
  1392.         this.defaultPaidAmount = defaultPaidAmount;
  1393.     }
  1394.  
  1395.     public Double getDefaultDiscountAmount() {
  1396.         return defaultDiscountAmount;
  1397.     }
  1398.  
  1399.     public void setDefaultDiscountAmount(Double defaultDiscountAmount) {
  1400.         this.defaultDiscountAmount = defaultDiscountAmount;
  1401.     }
  1402.  
  1403.     public Double getDefaultCouponAmount() {
  1404.         return defaultCouponAmount;
  1405.     }
  1406.  
  1407.     public void setDefaultCouponAmount(Double defaultCouponAmount) {
  1408.         this.defaultCouponAmount = defaultCouponAmount;
  1409.     }
  1410.  
  1411.     public Double getDefaultChangeAmount() {
  1412.         return defaultChangeAmount;
  1413.     }
  1414.  
  1415.     public void setDefaultChangeAmount(Double defaultChangeAmount) {
  1416.         this.defaultChangeAmount = defaultChangeAmount;
  1417.     }
  1418.  
  1419.     public Double getDefaultTotalConsumption() {
  1420.         return defaultTotalConsumption;
  1421.     }
  1422.  
  1423.     public void setDefaultTotalConsumption(Double defaultTotalConsumption) {
  1424.         this.defaultTotalConsumption = defaultTotalConsumption;
  1425.     }
  1426.  
  1427.     public Double getDefaultConsumeNum() {
  1428.         return defaultConsumeNum;
  1429.     }
  1430.  
  1431.     public void setDefaultConsumeNum(Double defaultConsumeNum) {
  1432.         this.defaultConsumeNum = defaultConsumeNum;
  1433.     }
  1434.  
  1435.     public String getRowKey() {
  1436.         return rowKey;
  1437.     }
  1438.  
  1439.     public void setRowKey(String rowKey) {
  1440.         this.rowKey = rowKey;
  1441.     }
  1442.  
  1443.     @Override
  1444.     public String toString() {
  1445.         return id + "\t" + rowKey + "\t" + shopId + "\t" + shopName + "\t" + shopEntityId + "\t" + shopEntityName + "\t"
  1446.                 + createTime + "\t" + cTimeStamp + "\t" + hcTime + "\t" + hserial + "\t" + fixTerminal + "\t"
  1447.                 + terminalNumber + "\t" + billfileName + "\t" + tsuuid + "\t" + billfileNameHis + "\t" + billNo + "\t"
  1448.                 + interceptTime + "\t" + shopEntityFullName + "\t" + shopEntityAddress + "\t" + telephone + "\t" + saler
  1449.                 + "\t" + checkstand + "\t" + cashier + "\t" + receivableAmount + "\t" + defaultReceivableAmount + "\t"
  1450.                 + totalNum + "\t" + defaultTotalNum + "\t" + billSerialNumber + "\t" + totalFee + "\t" + defaultTotalFee
  1451.                 + "\t" + paidAmount + "\t" + defaultPaidAmount + "\t" + discountAmount + "\t" + defaultDiscountAmount
  1452.                 + "\t" + couponAmount + "\t" + defaultCouponAmount + "\t" + changeAmount + "\t" + defaultChangeAmount
  1453.                 + "\t" + settlementWay + "\t" + saleTime + "\t" + memberCardNumber + "\t" + totalConsumption + "\t"
  1454.                 + defaultTotalConsumption + "\t" + website + "\t" + billImage + "\t" + goodsDetails + "\t" + roomNo
  1455.                 + "\t" + checkinName + "\t" + deskNo + "\t" + consumeNum + "\t" + defaultConsumeNum + "\t" + billText
  1456.                 + "\t" + inTime + "\t" + outTime + "\t" + defaultPrintDate + "\t" + defaultInTime + "\t"
  1457.                 + defaultOutTime + "\t" + uploadType + "\t" + customRecord + "\t" + billType + "\t" + modifyType + "\t"
  1458.                 + billSource + "\t" + analyzPath + "\t" + billMatchKey + "\t" + matchId + "\t" + defaultSaleTime + "\t"
  1459.                 + customerName + "\t" + membershipId + "\t" + memberLevels + "\t" + petName + "\t" + petNumber + "\t"
  1460.                 + principal + "\t" + deposit + "\t" + printDate + "\t" + defaultInterceptTime + "\t" + printMatchType
  1461.                 + "\t" + billMergeKey + "\t" + modifyAmount + "\t" + modifyAmountList + "\t" + modifyAmountSaleTimeList
  1462.                 + "\t" + modifyAmountInterceptTimeList + "\t" + uniqueId + "\t" + uniqueIdHis + "\t" + ifqrcode + "\t"
  1463.                 + couponNum + "\t" + erpMemberCard + "\t" + voucherType + "\t" + qrCode + "\t" + integralmark + "\t"
  1464.                 + thisintegral + "\t" + remarks + "\t" + templateName + "\t" + custName + "\t" + custTaxNo + "\t"
  1465.                 + custAdress + "\t" + custBankAccount + "\t" + thirdPartyOrderNo + "\t" + rechargeableCardConsumeAmount
  1466.                 + "\t" + defaultRechargeableCardConsumeAmount + "\t" + outOfPocketAmount + "\t"
  1467.                 + defaultOutOfPocketAmount + "\t" + rechargeAmount + "\t" + defaultRechargeAmount + "\t" + memberPrice
  1468.                 + "\t" + defaultMemberPrice + "\t" + memberDiscountrate + "\t" + defaultMemberDiscountrate + "\t"
  1469.                 + memberTotalConsumption + "\t" + defaultMemberTotalConsumption + "\t" + takeout + "\t"
  1470.                 + discountDetails;
  1471.     }
  1472.  
  1473.     public static BillInfoFmt cloneBill(BillInfo fromBean) {
  1474.         if (null == fromBean) {
  1475.             return null;
  1476.         }
  1477.         BillInfoFmt toBean = new BillInfoFmt();
  1478.         toBean.setId(stringRemove(fromBean.getId()));
  1479.         toBean.setRowKey(stringRemove(fromBean.getRowKey()));
  1480.         toBean.setShopId(stringRemove(fromBean.getShopId()));
  1481.         toBean.setShopName(stringRemove(fromBean.getShopName()));
  1482.         toBean.setShopEntityId(stringRemove(fromBean.getShopEntityId()));
  1483.         toBean.setShopEntityName(stringRemove(fromBean.getShopEntityName()));
  1484.         toBean.setCreateTime(DateTimeUtils.getYmdhmsForNo(fromBean.getCreateTime()));
  1485.         toBean.setCTimeStamp(DateTimeUtils.getYmdhmsForNo(fromBean.getCTimeStamp()));
  1486.         toBean.setHcTime(stringRemove(fromBean.getHcTime()));
  1487.         toBean.setHserial(stringRemove(fromBean.getHserial()));
  1488.         toBean.setFixTerminal(stringRemove(fromBean.getFixTerminal()));
  1489.         toBean.setTerminalNumber(stringRemove(fromBean.getTerminalNumber()));
  1490.         toBean.setBillfileName(stringRemove(fromBean.getBillfileName()));
  1491.         toBean.setTsuuid(stringRemove(fromBean.getTsuuid()));
  1492.         toBean.setBillfileNameHis(fromBean.getBillfileNameHis());
  1493.         toBean.setBillNo(stringRemove(fromBean.getBillNo()));
  1494.         toBean.setInterceptTime(DateTimeUtils.getYmdhmsForNo(fromBean.getInterceptTime()));
  1495.         toBean.setShopEntityFullName(stringRemove(fromBean.getShopEntityFullName()));
  1496.         toBean.setShopEntityAddress(stringRemove(fromBean.getShopEntityAddress()));
  1497.         toBean.setTelephone(stringRemove(fromBean.getTelephone()));
  1498.         toBean.setSaler(stringRemove(fromBean.getSaler()));
  1499.         toBean.setCheckstand(stringRemove(fromBean.getCheckstand()));
  1500.         toBean.setCashier(stringRemove(fromBean.getCashier()));
  1501.         toBean.setReceivableAmount(fromBean.getReceivableAmount());
  1502.         toBean.setDefaultReceivableAmount(fromBean.getDefaultReceivableAmount());
  1503.         toBean.setTotalNum(fromBean.getTotalNum());
  1504.         toBean.setDefaultTotalNum(fromBean.getDefaultTotalNum());
  1505.         toBean.setBillSerialNumber(stringRemove(fromBean.getBillSerialNumber()));
  1506.         toBean.setTotalFee(fromBean.getTotalFee());
  1507.         toBean.setDefaultTotalFee(fromBean.getDefaultTotalFee());
  1508.         toBean.setPaidAmount(fromBean.getPaidAmount());
  1509.         toBean.setDefaultPaidAmount(fromBean.getDefaultPaidAmount());
  1510.         toBean.setDiscountAmount(fromBean.getDiscountAmount());
  1511.         toBean.setDefaultDiscountAmount(fromBean.getDefaultDiscountAmount());
  1512.         toBean.setCouponAmount(fromBean.getCouponAmount());
  1513.         toBean.setDefaultCouponAmount(fromBean.getDefaultCouponAmount());
  1514.         toBean.setChangeAmount(fromBean.getChangeAmount());
  1515.         toBean.setDefaultChangeAmount(fromBean.getDefaultChangeAmount());
  1516.         if (null != fromBean.getSettlementWay()) {
  1517.             List<SettlementWayInfo> rawList = fromBean.getSettlementWay();
  1518.             List<String> list = new ArrayList<>();
  1519.             for (SettlementWayInfo raw : rawList) {
  1520.                 list.add(raw.toString());
  1521.             }
  1522.             toBean.setSettlementWay(list);
  1523.         }
  1524.         toBean.setSaleTime(fromBean.getSaleTime());
  1525.         toBean.setMemberCardNumber(stringRemove(fromBean.getMemberCardNumber()));
  1526.         toBean.setTotalConsumption(fromBean.getTotalConsumption());
  1527.         toBean.setDefaultTotalConsumption(fromBean.getDefaultTotalConsumption());
  1528.         toBean.setWebsite(stringRemove(fromBean.getWebsite()));
  1529.         toBean.setBillImage(stringRemove(fromBean.getBillImage()));
  1530.         if (null != fromBean.getGoodsDetails()) {
  1531.             List<GoodsDetailInfo> rawList = fromBean.getGoodsDetails();
  1532.             List<String> list = new ArrayList<>();
  1533.             for (GoodsDetailInfo raw : rawList) {
  1534.                 list.add(raw.toString());
  1535.             }
  1536.             toBean.setGoodsDetails(list);
  1537.         }
  1538.         toBean.setRoomNo(stringRemove(fromBean.getRoomNo()));
  1539.         toBean.setCheckinName(stringRemove(fromBean.getCheckinName()));
  1540.         toBean.setDeskNo(stringRemove(fromBean.getDeskNo()));
  1541.         toBean.setConsumeNum(fromBean.getConsumeNum());
  1542.         toBean.setDefaultConsumeNum(fromBean.getDefaultConsumeNum());
  1543.         toBean.setBillText(stringRemove(fromBean.getBillText()));
  1544.         toBean.setInTime(DateTimeUtils.getYmdhmsForNo(fromBean.getInTime()));
  1545.         toBean.setOutTime(DateTimeUtils.getYmdhmsForNo(fromBean.getOutTime()));
  1546.         toBean.setDefaultPrintDate(DateTimeUtils.getYmdhmsForNo(fromBean.getDefaultPrintDate()));
  1547.         toBean.setDefaultInTime(DateTimeUtils.getYmdhmsForNo(fromBean.getDefaultInTime()));
  1548.         toBean.setDefaultOutTime(DateTimeUtils.getYmdhmsForNo(fromBean.getDefaultOutTime()));
  1549.         toBean.setUploadType(stringRemove(fromBean.getUploadType()));
  1550.         toBean.setCustomRecord(fromBean.getCustomRecord());
  1551.         toBean.setBillType(stringRemove(fromBean.getBillType()));
  1552.         toBean.setModifyType(stringRemove(fromBean.getModifyType()));
  1553.         toBean.setBillSource(stringRemove(fromBean.getBillSource()));
  1554.         toBean.setAnalyzPath(stringRemove(fromBean.getAnalyzPath()));
  1555.         toBean.setBillMatchKey(stringRemove(fromBean.getBillMatchKey()));
  1556.         toBean.setMatchId(stringRemove(fromBean.getMatchId()));
  1557.         toBean.setDefaultSaleTime(DateTimeUtils.getYmdhmsForNo(fromBean.getDefaultSaleTime()));
  1558.         toBean.setCustomerName(stringRemove(fromBean.getCustomerName()));
  1559.         toBean.setMembershipId(stringRemove(fromBean.getMembershipId()));
  1560.         toBean.setMemberLevels(stringRemove(fromBean.getMemberLevels()));
  1561.         toBean.setPetName(stringRemove(fromBean.getPetName()));
  1562.         toBean.setPetNumber(stringRemove(fromBean.getPetNumber()));
  1563.         toBean.setPrincipal(stringRemove(fromBean.getPrincipal()));
  1564.         toBean.setDeposit(stringRemove(fromBean.getDeposit()));
  1565.         toBean.setPrintDate(DateTimeUtils.getYmdhmsForNo(fromBean.getPrintDate()));
  1566.         toBean.setDefaultInterceptTime(DateTimeUtils.getYmdhmsForNo(fromBean.getDefaultInterceptTime()));
  1567.         toBean.setPrintMatchType(stringRemove(fromBean.getPrintMatchType()));
  1568.         toBean.setBillMergeKey(stringRemove(fromBean.getBillMergeKey()));
  1569.         toBean.setModifyAmount(fromBean.getModifyAmount());
  1570.         toBean.setModifyAmountList(fromBean.getModifyAmountList());
  1571.         toBean.setModifyAmountSaleTimeList(DateTimeUtils.getYmdhmsForNo(fromBean.getModifyAmountSaleTimeList()));
  1572.         toBean.setModifyAmountInterceptTimeList(
  1573.                 DateTimeUtils.getYmdhmsForNo(fromBean.getModifyAmountInterceptTimeList()));
  1574.         toBean.setUniqueId(stringRemove(fromBean.getUniqueId()));
  1575.         toBean.setUniqueIdHis(fromBean.getUniqueIdHis());
  1576.         toBean.setIfqrcode(stringRemove(fromBean.getIfqrcode()));
  1577.         toBean.setCouponNum(stringRemove(fromBean.getCouponNum()));
  1578.         toBean.setErpMemberCard(stringRemove(fromBean.getErpMemberCard()));
  1579.         toBean.setVoucherType(stringRemove(fromBean.getVoucherType()));
  1580.         toBean.setQrCode(stringRemove(fromBean.getQrCode()));
  1581.         toBean.setIntegralmark(stringRemove(fromBean.getIntegralmark()));
  1582.         toBean.setThisintegral(stringRemove(fromBean.getThisintegral()));
  1583.         toBean.setRemarks(stringRemove(fromBean.getRemarks()));
  1584.         toBean.setTemplateName(stringRemove(fromBean.getTemplateName()));
  1585.         toBean.setCustName(stringRemove(fromBean.getCustName()));
  1586.         toBean.setCustName(stringRemove(fromBean.getCustTaxNo()));
  1587.         toBean.setCustAdress(stringRemove(fromBean.getCustAdress()));
  1588.         toBean.setCustBankAccount(stringRemove(fromBean.getCustBankAccount()));
  1589.         toBean.setThirdPartyOrderNo(stringRemove(fromBean.getThirdPartyOrderNo()));
  1590.         toBean.setRechargeableCardConsumeAmount(fromBean.getRechargeableCardConsumeAmount());
  1591.         toBean.setDefaultRechargeableCardConsumeAmount(fromBean.getDefaultRechargeableCardConsumeAmount());
  1592.         toBean.setOutOfPocketAmount(fromBean.getOutOfPocketAmount());
  1593.         toBean.setDefaultOutOfPocketAmount(fromBean.getDefaultOutOfPocketAmount());
  1594.         toBean.setRechargeAmount(fromBean.getRechargeAmount());
  1595.         toBean.setDefaultRechargeAmount(fromBean.getDefaultRechargeAmount());
  1596.         toBean.setMemberPrice(fromBean.getMemberPrice());
  1597.         toBean.setDefaultMemberPrice(fromBean.getDefaultMemberPrice());
  1598.         toBean.setMemberDiscountrate(fromBean.getMemberDiscountrate());
  1599.         toBean.setDefaultMemberDiscountrate(fromBean.getDefaultMemberDiscountrate());
  1600.         toBean.setMemberTotalConsumption(fromBean.getMemberTotalConsumption());
  1601.         toBean.setDefaultMemberTotalConsumption(fromBean.getDefaultMemberTotalConsumption());
  1602.         toBean.setTakeout(stringRemove(fromBean.getTakeout()));
  1603.         if (null != fromBean.getDiscountDetails()) {
  1604.             List<DiscountDetailsInfo> rawList = fromBean.getDiscountDetails();
  1605.             List<String> list = new ArrayList<>();
  1606.             for (DiscountDetailsInfo raw : rawList) {
  1607.                 list.add(raw.toString());
  1608.             }
  1609.             toBean.setDiscountDetails(list);
  1610.         }
  1611.         return toBean;
  1612.     }
  1613.  
  1614.     /**
  1615.      * 原始数据格式化处理，待完善
  1616.      *
  1617.      * @param jsonStr
  1618.      * @return
  1619.      */
  1620.     private static String stringRemove(String strVal) {
  1621.         if (StringUtils.isEmpty(strVal)) {
  1622.             return "";
  1623.         }
  1624.         strVal = strVal.replace("\t", "");
  1625.         strVal = strVal.replace("\n", "");
  1626.         strVal = strVal.replace("\r", "");
  1627.         return strVal;
  1628.     }
  1629. }

4.5、BCD裁剪实体类

  1. package com.mengyao.graph.etl.apps.dashboard.beans;
  2.  
  3. import java.io.Serializable;
  4.  
  5. /**
  6.  * Bill Consumer Dashboard Bean
  7.  * @author mengyao
  8.  *
  9.  */
  10. public class BCD implements Serializable {
  11.     private static final long serialVersionUID = 1749406742944513387L;
  12.     private String billId;
  13.     private double receivableAmount;
  14.     private String saleTime;
  15.     private String sdt;//yyyyMMdd
  16.     private int hour;
  17.     private String billType;
  18.     private String shopId;
  19.     private String shopEntityId;
  20.     public BCD() {
  21.         super();
  22.     }
  23.     public BCD(String billId, double receivableAmount, String saleTime, String billType, String shopId, String shopEntityId) {
  24.         super();
  25.         this.billId = billId;
  26.         this.receivableAmount = receivableAmount;
  27.         this.saleTime = saleTime;
  28.         setDateTime(saleTime);
  29.         this.billType = billType;
  30.         this.shopId = shopId;
  31.         this.shopEntityId = shopEntityId;
  32.     }
  33.     public String getBillId() {
  34.         return billId;
  35.     }
  36.     public void setBillId(String billId) {
  37.         this.billId = billId;
  38.     }
  39.     public double getReceivableAmount() {
  40.         return receivableAmount;
  41.     }
  42.     public void setReceivableAmount(double receivableAmount) {
  43.         this.receivableAmount = receivableAmount;
  44.     }
  45.     public String getSaleTime() {
  46.         return saleTime;
  47.     }
  48.     public void setSaleTime(String saleTime) {
  49.         this.saleTime = saleTime;
  50.         setDateTime(saleTime);
  51.     }
  52.     public String getSdt() {
  53.         return sdt;
  54.     }
  55.     public void setSdt(String sdt) {
  56.         this.sdt = sdt;
  57.     }
  58.     public int getHour() {
  59.         return hour;
  60.     }
  61.     public void setHour(int hour) {
  62.         this.hour = hour;
  63.     }
  64.     public String getBillType() {
  65.         return billType;
  66.     }
  67.     public void setBillType(String billType) {
  68.         this.billType = billType;
  69.     }
  70.     public String getShopId() {
  71.         return shopId;
  72.     }
  73.     public void setShopId(String shopId) {
  74.         this.shopId = shopId;
  75.     }
  76.     public String getShopEntityId() {
  77.         return shopEntityId;
  78.     }
  79.     public void setShopEntityId(String shopEntityId) {
  80.         this.shopEntityId = shopEntityId;
  81.     }
  82.     private void setDateTime(String saleTime) {
  83.         if (null!=saleTime&&saleTime.length()==17) {
  84.             this.sdt = saleTime.substring(0, 8);
  85.             this.hour = Integer.parseInt(saleTime.substring(8, 10));
  86.         }
  87.     }
  88.     @Override
  89.     public String toString() {
  90.         return  billId + "\t" + receivableAmount + "\t" + saleTime + "\t" + sdt + "\t" + billType + "\t" + shopId + "\t" + shopEntityId;
  91.     }
  92.  
  93. }

4.6、销售分析业务实现类

  1. package com.mengyao.graph.etl.apps.dashboard.service;
  2.  
  3. import java.io.Serializable;
  4. import java.util.ArrayList;
  5. import java.util.Collection;
  6. import java.util.HashMap;
  7. import java.util.HashSet;
  8. import java.util.LinkedList;
  9. import java.util.List;
  10. import java.util.Map;
  11. import java.util.Map.Entry;
  12. import java.util.Set;
  13.  
  14. import org.apache.spark.sql.Column;
  15. import org.apache.spark.sql.Dataset;
  16. import org.apache.spark.sql.Row;
  17. import org.apache.spark.sql.functions;
  18.  
  19. import com.alibaba.fastjson.JSON;
  20. import com.alibaba.fastjson.JSONObject;
  21. import com.mengyao.graph.etl.apps.dashboard.beans.AreaProjSale;
  22. import com.mengyao.graph.etl.apps.dashboard.beans.AreaSaleTrendAll;
  23. import com.mengyao.graph.etl.apps.dashboard.beans.PeakTime;
  24. import com.mengyao.graph.etl.apps.dashboard.beans.ProjectTypeShopNumber;
  25. import com.mengyao.graph.etl.apps.dashboard.beans.ShopSaleRank;
  26. import com.mengyao.graph.etl.apps.dashboard.beans.TotalRefund;
  27. import com.mengyao.graph.etl.apps.dashboard.beans.TotalSale;
  28. import com.mengyao.graph.etl.apps.dashboard.beans.TotalSettlementBillNumber;
  29. import com.mengyao.utils.RedisUtil;
  30.  
  31. /**
  32.  * 大屏指标分析
  33.  * @author mengyao
  34.  *
  35.  */
  36. public class SaleAnalysisService implements Serializable {
  37.  
  38.     private static final long serialVersionUID = 8289368096001689148L;
  39.     //解决区域名称hash重复问题
  40.     private static final String SALT="aAb12";
  41.  
  42.     /**
  43.      * 每日重置大屏指标值
  44.      */
  45.     @Deprecated
  46.     public void reset() {
  47.         RedisUtil.setObject("dtsbn_6", "{\"结账单数\":{\"val\":7270}}\"");
  48.         RedisUtil.setObject("dpt_7", "{\"高峰时段\":{\"val\":13}}\"");
  49.         RedisUtil.setObject("dtr_5", "{\"退款金额\":{\"val\":7301.8}}\"");
  50.         RedisUtil.setObject("dts_4", "{\"总销售额\":{\"val\":1300523.8000000005}}\"");
  51.         RedisUtil.setObject("curDay", "20190227\"");
  52.         RedisUtil.setObject("dastfa_2_20190227", "{\"各区域销售额发展趋势\":{\"重庆天地\":[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,9.5,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0],\"创智天地\":[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,43.3,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0],\"上海瑞虹\":[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,16.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0],\"上海新天地\":[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,553.75,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0]}}\"");
  53.         RedisUtil.setObject("dasp_1", "{\"各区域销售额占比\":{\"重庆天地\":[{\"bn\":1,\"n\":\"重庆天地\",\"sa\":9.5}],\"创智天地\":[{\"bn\":3,\"n\":\"壹方\",\"sa\":43.3}],\"上海新天地\":[{\"bn\":4,\"n\":\"新天地时尚购物中心\",\"sa\":152.0},{\"bn\":3,\"n\":\"上海新天地南里北里\",\"sa\":74.0},{\"bn\":2,\"n\":\"湖滨道购物中心\",\"sa\":100.0},{\"bn\":10,\"n\":\"新天地广场\",\"sa\":227.75}],\"上海瑞虹\":[{\"bn\":1,\"n\":\"瑞虹天地星星堂\",\"sa\":16.0}]}}\"");
  54.         RedisUtil.setObject("dpc_8", "{\"各项目销售额对比\":{\"重庆天地\":[{\"bn\":1,\"n\":\"重庆天地\",\"sa\":9.5}],\"创智天地\":[{\"bn\":3,\"n\":\"壹方\",\"sa\":43.3}],\"上海新天地\":[{\"bn\":4,\"n\":\"新天地时尚购物中心\",\"sa\":152.0},{\"bn\":3,\"n\":\"上海新天地南里北里\",\"sa\":74.0},{\"bn\":2,\"n\":\"湖滨道购物中心\",\"sa\":100.0},{\"bn\":10,\"n\":\"新天地广场\",\"sa\":227.75}],\"上海瑞虹\":[{\"bn\":1,\"n\":\"瑞虹天地星星堂\",\"sa\":16.0}]}}\"");
  55.         RedisUtil.setObject("dpac_10", "{\"各项目单均消费\":{\"重庆天地\":[{\"bn\":1,\"n\":\"重庆天地\",\"sa\":9.5}],\"创智天地\":[{\"bn\":3,\"n\":\"壹方\",\"sa\":43.3}],\"上海新天地\":[{\"bn\":4,\"n\":\"新天地时尚购物中心\",\"sa\":152.0},{\"bn\":3,\"n\":\"上海新天地南里北里\",\"sa\":74.0},{\"bn\":2,\"n\":\"湖滨道购物中心\",\"sa\":100.0},{\"bn\":10,\"n\":\"新天地广场\",\"sa\":227.75}],\"上海瑞虹\":[{\"bn\":1,\"n\":\"瑞虹天地星星堂\",\"sa\":16.0}]}}\"");
  56.         RedisUtil.setObject("dsfst_9", "{\"店铺销售排行\":[{\"pn\":\"新天地时尚购物中心\",\"sa\":152.0,\"sn\":\"GREYBOX COFFEE\"},{\"pn\":\"新天地广场\",\"sa\":114.75,\"sn\":\"Arabica\"},{\"pn\":\"湖滨道购物中心\",\"sa\":100.0,\"sn\":\"LOKAL\"},{\"pn\":\"上海新天地南里北里\",\"sa\":74.0,\"sn\":\"哈肯铺_手感烘焙\"},{\"pn\":\"壹方\",\"sa\":60.3,\"sn\":\"新一天便利\"},{\"pn\":\"新天地广场\",\"sa\":41.0,\"sn\":\"奈雪的茶\"},{\"pn\":\"新天地广场\",\"sa\":39.0,\"sn\":\"蒲石小点\"},{\"pn\":\"新天地广场\",\"sa\":18.0,\"sn\":\"Fresh_Every_Day\"},{\"pn\":\"瑞虹天地星星堂\",\"sa\":16.0,\"sn\":\"老盛昌\"},{\"pn\":\"新天地广场\",\"sa\":15.0,\"sn\":\"多几谷\"}]}\"");
  57.         RedisUtil.setObject("dptsc_11", "{\"各项目业态销售额对比\":{\"上海新天地南里北里\":[{\"n\":\"餐饮\",\"sa\":74.0,\"sn\":3}],\"新天地广场\":[{\"n\":\"餐饮\",\"sa\":227.75,\"sn\":10}],\"重庆天地\":[{\"n\":\"其它\",\"sa\":9.5,\"sn\":0}],\"湖滨道购物中心\":[{\"n\":\"餐饮\",\"sa\":100.0,\"sn\":2}],\"壹方\":[{\"n\":\"餐饮\",\"sa\":43.3,\"sn\":3}],\"瑞虹天地星星堂\":[{\"n\":\"餐饮\",\"sa\":16.0,\"sn\":1}],\"新天地时尚购物中心\":[{\"n\":\"餐饮\",\"sa\":152.0,\"sn\":4}]}}\"");
  58.         RedisUtil.setObject("dpsn_12", "{\"各项目店铺数量\":{\"上海新天地南里北里\":[{\"n\":\"餐饮\",\"sa\":74.0,\"sn\":3}],\"新天地广场\":[{\"n\":\"餐饮\",\"sa\":227.75,\"sn\":10}],\"重庆天地\":[{\"n\":\"其它\",\"sa\":9.5,\"sn\":0}],\"湖滨道购物中心\":[{\"n\":\"餐饮\",\"sa\":100.0,\"sn\":2}],\"壹方\":[{\"n\":\"餐饮\",\"sa\":43.3,\"sn\":3}],\"瑞虹天地星星堂\":[{\"n\":\"餐饮\",\"sa\":16.0,\"sn\":1}],\"新天地时尚购物中心\":[{\"n\":\"餐饮\",\"sa\":152.0,\"sn\":4}]}}\"");
  59.     }
  60.  
  61.     /**
  62.      * 总销售额
  63.      * @param ds
  64.      */
  65.     public void totalSale(Dataset<Row> bill) {
  66.         Row[] rows = (Row[])bill
  67.                 .filter("billType=1 and receivableAmount>=0")
  68.                 .agg(functions.sum(new Column("receivableAmount")).alias("totalSale"), functions.count("receivableAmount"))
  69.                 .head(1);
  70.         Map<String, TotalSale> map = new HashMap<String, TotalSale>();
  71.         double totalSale=0D;
  72.         if(rows.length>0){
  73.             Row row = rows[0];
  74.             if (!row.isNullAt(0)) {
  75.                 //销售额
  76.                 totalSale=row.getDouble(0);
  77.             }
  78.             if (!row.isNullAt(1)) {
  79.                 //结账单数
  80.                 totalSettlementBillNumber(new TotalSettlementBillNumber(row.getLong(1)));
  81.             }
  82.         }
  83.         map.put("总销售额", new TotalSale(totalSale));
  84.         String str = JSONObject.toJSONString(map);
  85.         System.out.println("====####--totalSale##" + str);
  86.         // 将总销售额存入redis
  87.         RedisUtil.setObject("dts_4", str);
  88.     }
  89.  
  90.     /**
  91.      * 退款金额
  92.      * @param ds
  93.      */
  94.     public void totalRefund(Dataset<Row> bill) {
  95.         Row[] rows = (Row[]) bill
  96.                 .filter("((billType=6) or (billType=1 and receivableAmount<0))")
  97.                 .agg(functions.sum(functions.abs(new Column("receivableAmount"))).alias("totalSale"))
  98.                 .head(1);
  99.         Map<String, TotalRefund> map=new HashMap<>();
  100.         map.put("退款金额", new TotalRefund(0));
  101.         if(rows.length>0){
  102.             Row row = rows[0];
  103.             if (!row.isNullAt(0)) {
  104.                 map.put("退款金额", new TotalRefund(row.getDouble(0)));
  105.             }
  106.         }
  107.         String str=JSONObject.toJSONString(map);
  108.         System.out.println("====####--totalRefund##"+str);
  109.         //将退款金额存入redis
  110.         RedisUtil.setObject("dtr_5",str);
  111.     }
  112.  
  113.     /**
  114.      * 高峰时段
  115.      *  1、时段：小时；
  116.      *    2、高峰：当日每小时结账单数累计最大；
  117.      *    3、高峰时段：所有mall累计每小时结账单数；
  118.      * @param ds
  119.      * @return {"高峰时段":{"val":12}}
  120.      */
  121.     public void peakTime(Dataset<Row> bill) {
  122.         //账单表本身有mallid(shopId) 因此无需关联店铺表
  123.         Map<String, PeakTime> map=new HashMap<>();
  124.         map.put("高峰时段", new PeakTime(8));
  125.         Row[] rows = (Row[])bill
  126.                 .filter("billType=1")
  127.                 .groupBy("hour")
  128.                 .agg(functions.sum("receivableAmount").alias("totalSale"))
  129.                 .orderBy(new Column("totalSale").desc())
  130.                 .limit(1)
  131.                 .head(1);
  132.         if(rows.length>0){
  133.             Row row = rows[0];
  134.             if (!row.isNullAt(0)) {
  135.                 map.put("高峰时段", new PeakTime(row.getAs(0)));
  136.             }
  137.         }
  138.         String str=JSONObject.toJSONString(map);
  139.         System.out.println("====####--peakTime##"+str);
  140.         //将高峰时段放入redis
  141.         RedisUtil.setObject("dpt_7", str);
  142.     }
  143.  
  144.     /**
  145.      * 各区域销售发展趋势-多个区域（每个区域下有多个mall）当日0点~24点的累计销售额
  146.      * @param bill
  147.      * @param ruian
  148.      * @param curDay
  149.      */
  150.     public void areaSaleTrendForAll(Dataset<Row> bill, Dataset<Row> ruian,Set<String> areaSet, String curDay) {
  151.         Row[] rows = (Row[])bill
  152.                 .filter("billType=1 and receivableAmount>0")
  153.                 .join(ruian, bill.col("shopId").equalTo(ruian.col("rmid")), "leftouter")
  154.                 .groupBy("area_cn", "hour")
  155.                 .agg(functions.sum("receivableAmount").alias("areaDayHourSale"))
  156.                 .orderBy("areaDayHourSale")
  157.                 .select("area_cn","areaDayHourSale","hour")
  158.                 .collect();
  159. //        Map<String, Map<String,Map<Integer, Double>>> map = new HashMap<>();
  160.         Map<String, Map<String,Collection<Double>>> map = new HashMap<>();
  161.         Map<String,AreaSaleTrendAll> maps=new HashMap<>();
  162.         if(rows.length>0){
  163.             for (Row row : rows) {
  164.                 String areaCn=null;
  165.                 double areaDayHourSale=0D;
  166.                 int saleHour=0;
  167.                 if(!row.isNullAt(0)){
  168.                     areaCn=row.getString(0);
  169.                 }
  170.                 if(!row.isNullAt(1)){
  171.                     areaDayHourSale=row.getDouble(1);
  172.                 }
  173.                 if(!row.isNullAt(2)){
  174.                     saleHour=row.getInt(2);
  175.                 }
  176.                 if(maps.containsKey(areaCn+SALT)){
  177.                     AreaSaleTrendAll ast=maps.get(areaCn+SALT);
  178.                     ast.getVals().put(saleHour, areaDayHourSale);
  179.                 }else{
  180.                     HashMap<Integer,Double> vals=new HashMap<Integer,Double>();
  181.                     vals.put(saleHour, areaDayHourSale);
  182.                     maps.put(areaCn+SALT, new AreaSaleTrendAll(areaCn, saleHour, areaDayHourSale));
  183.                 }
  184.             }
  185.         }
  186.         //填充没有销售额的区域记录，使数据更加完整。
  187.         areaSet.forEach(areaCn->{
  188.             if(!maps.containsKey(areaCn+SALT)){
  189.                 maps.put(areaCn+SALT, new AreaSaleTrendAll(areaCn));
  190.             }
  191.         });
  192.  
  193.         System.out.println("==####========填充hou====begin=====================");
  194.         for (String key:maps.keySet()) {
  195.             System.out.println("Key:"+key);
  196.         }
  197.         System.out.println("==####========填充hou=====end====================");
  198.  
  199. //        Map<String,Map<Integer, Double>> rs=new HashMap<>();
  200. //        for(AreaSaleTrendAll asta:maps.values()){
  201. //            rs.put(asta.getAreaCn(), asta.getVals());
  202. //        }
  203.         Map<String,Collection<Double>> rs=new HashMap<>();
  204.         for(AreaSaleTrendAll asta:maps.values()){
  205.             rs.put(asta.getAreaCn(), asta.getVals().values());
  206.         }
  207.         map.put("各区域销售额发展趋势", rs);
  208.         //转成json 字符串
  209.         String str=JSONObject.toJSONString(map);
  210.         System.out.println("====####--areaSaleTrendForAll##"+str);
  211.         //放入redis
  212.         RedisUtil.setObject("dastfa_2_"+curDay, str);
  213.         //维护redis中最新日期
  214.         if(!RedisUtil.existsObject("curDay")){//如果为空，说明是第一次运行，直接将当前日期设置到redis中
  215.             RedisUtil.setObject("curDay", curDay);
  216.         }else{
  217.             String curDayRedis=(String) RedisUtil.getObject("curDay");
  218.             if((curDayRedis.compareTo(curDay))<0){//说明当前日期大于redis中的日期，更新
  219.                 RedisUtil.setObject("curDay", curDay);
  220.             }
  221.         }
  222.     }
  223.  
  224.     /**
  225.      * 各项目销售额对比
  226.      *     1、各项目：各个mall；
  227.      *    2、单一项目销售额：mall的当日开始营业时间到当前时间累计销售额；
  228.      */
  229.     public void projectContrast(Dataset<Row> bill, Dataset<Row> ruian,Map<String, Set<String>> ruianMallAll) {
  230.         Row[] rows = (Row[])bill
  231.                 .filter("billType=1 and receivableAmount>0")
  232.                 .join(ruian, bill.col("shopId").equalTo(ruian.col("rmid")), "leftouter")
  233.                 .groupBy("name", "area_cn")
  234.                 .agg(functions.sum("receivableAmount").alias("mallDaySale"), functions.count("receivableAmount").alias("mallDayBillNum"))
  235. //                .orderBy("area_cn")
  236.                 .select("name","area_cn","mallDaySale","mallDayBillNum")
  237.                 .collect();
  238.         Map<String,List<AreaProjSale>> rs=new HashMap<>();
  239.         if(rows.length>0) {
  240.             for (Row row : rows) {
  241.                 String mallName=null;
  242.                 String areaCn=null;
  243.                 double mallDaySale=0D;
  244.                 long mallDayBillNum=0L;
  245.                 AreaProjSale obj=new AreaProjSale();
  246.                 if(!row.isNullAt(0)){//项目、mall的名称
  247.                     mallName=row.getString(0);
  248.                     obj.setN(mallName);
  249.                 }
  250.                 if(!row.isNullAt(1)){//区域名称
  251.                     areaCn=row.getString(1);
  252.                 }
  253.                 if(!row.isNullAt(2)){//mall的日销售额
  254.                     mallDaySale=row.getDouble(2);
  255.                     obj.setSa(mallDaySale);
  256.                 }
  257.                 if(!row.isNullAt(3)){//mall的日结账单数
  258.                     mallDayBillNum=row.getLong(3);
  259.                     obj.setBn(mallDayBillNum);
  260.                 }
  261.                 //判断是否有该区域
  262.                 if(rs.containsKey(areaCn)){
  263.                     rs.get(areaCn).add(obj);
  264.                 }else{//不存在该区域
  265.                     List<AreaProjSale> list=new ArrayList<>();
  266.                     list.add(obj);
  267.                     rs.put(areaCn, list);
  268.                 }
  269.             }
  270.         }
  271.         //填充未产生账单的数据，默认0
  272.         //获取全部瑞安的区域和mallName的集合
  273.         Set<Entry<String,Set<String>>> entrySet = ruianMallAll.entrySet();
  274.         for(Map.Entry<String, Set<String>> entry:entrySet){
  275.             String areaCn=entry.getKey();//获取区域名称
  276.             //获取每个区域的标准mall的集合
  277.             Set<String> mallSet=entry.getValue();
  278.             if(rs.containsKey(areaCn)){//实际数据中已经存在该区域相关数据
  279.                 //判断实际数据mall是否完整
  280.                 //用来存放实际数据中mallname的集合
  281.                 Set<String> mallSetCur=new HashSet<>();
  282.                 //用来存放没有账单的mall的集合
  283.                 Set<String> mallSetNew=new HashSet<>();
  284.                 //遍历该区域下实际数据集合 并填充set
  285.                 for(AreaProjSale obj:rs.get(areaCn)){
  286.                     mallSetCur.add(obj.getN());
  287.                 }
  288.                 //求两个set的差集合  将标准数据放入
  289.                 mallSetNew.addAll(mallSet);
  290.                 //求差集合  标准数据-实际数据 得到差集
  291.                 mallSetNew.removeAll(mallSetCur);
  292.                 //遍历差集合 ，填充默认值
  293.                 mallSetNew.forEach(mn->{
  294.                     rs.get(areaCn).add(new AreaProjSale(mn));
  295.                 });
  296.             }else{//该区域不存在
  297.                 //便利该区域下标准mall集合，逐个放入
  298.                 List<AreaProjSale> list=new ArrayList<>();
  299.                 mallSet.forEach(mn->{
  300.                     list.add(new AreaProjSale(mn));
  301.                 });
  302.                 rs.put(areaCn, list);
  303.             }
  304.         }
  305.         //各区域销售额占比
  306.         areaSaleProportion(rs);
  307.         //各区域单均消费
  308.         projectAvgConsumer(rs);
  309.         //转成json
  310.         Map<String, Map<String,List<AreaProjSale>>> pmap=new HashMap<>();
  311.         pmap.put("各项目销售额对比", rs);
  312.         //转成json
  313.         String str=JSONObject.toJSONString(pmap);
  314.         System.out.println("====####--projectContrast##"+str);
  315.         //1 8各项目销售额对比dpc_8
  316.         RedisUtil.setObject("dpc_8", str);
  317.     }
  318.  
  319.     /**
  320.      * 所有mall中销售额最高的top10店铺
  321.      * @param bill
  322.      * @param shop
  323.      * @param ruian
  324.      */
  325.     public void saleForShopTop10(Dataset<Row> bill, Dataset<Row> shop, Dataset<Row> ruian) {
  326.         Row[] rows = (Row[])bill
  327.                 .filter("billType=1 and receivableAmount>0")
  328.                 .join(shop, bill.col("shopEntityId").equalTo(shop.col("shop_entity_id")), "leftouter")
  329.                 .join(ruian, bill.col("shopId").equalTo(ruian.col("rmid")), "leftouter")
  330.                 .groupBy("shop_entity_name", "name")
  331.                 .agg(functions.sum("receivableAmount").alias("shopDaySale"))
  332.                 .orderBy(new Column("shopDaySale").desc())
  333.                 .select("shop_entity_name","name","shopDaySale")
  334.                 .limit(10)
  335.                 .head(10);
  336.         List<ShopSaleRank> ssrList=new LinkedList<>();
  337.         if(rows.length>0) {
  338.             for (Row row : rows) {
  339.                 ShopSaleRank ssr=new ShopSaleRank();
  340.                 if(!row.isNullAt(0)){//店铺名称
  341.                     ssr.setSn(row.getString(0));
  342.                 }
  343.                 if(!row.isNullAt(1)){//项目/mall名称
  344.                     ssr.setPn(row.getString(1));
  345.                 }
  346.                 if(!row.isNullAt(2)){//店铺日销售额
  347.                     ssr.setSa(row.getDouble(2));
  348.                 }
  349.                 ssrList.add(ssr);
  350.             }
  351.         }
  352.         //转成json
  353.         Map<String, List<ShopSaleRank>> pmap=new HashMap<>();
  354.         pmap.put("店铺销售排行", ssrList);
  355.         String str=JSON.toJSONString(pmap);
  356.         System.out.println("====####--saleForShopTop10##"+str);
  357.         //将10个店铺日销售额放入redis
  358.         RedisUtil.setObject("dsfst_9",str);
  359.     }
  360.  
  361.     /**
  362.      * 各项目业态销售额对比
  363.      *
  364.      * @param session
  365.      * @param beginYMDH
  366.      * @param endYMDH
  367.      */
  368.     public void projectTypeSaleContrast(Dataset<Row> bill, Dataset<Row> shop, Dataset<Row> type, Dataset<Row> ruian,
  369.             Set<String> areaSet,Set<String> mallSet,Set<String> ruianTypeAll) {
  370.         Row[] rows = (Row[])bill
  371.                 .filter("billType=1 and receivableAmount>0")
  372.                 .join(shop, shop.col("shop_entity_id").equalTo(bill.col("shopEntityId")), "leftouter")
  373.                 .join(type, type.col("id").equalTo(shop.col("shop_entity_type_root")), "leftouter")
  374.                 .join(ruian, ruian.col("rmid").equalTo(bill.col("shopId")))
  375.                 .groupBy("name", "shop_type_name")
  376.                 .agg(functions.sum("receivableAmount").alias("shopTypeDaySale"), functions.countDistinct("shop_entity_id").alias("shopNum"))
  377.                 .select("name","shop_type_name","shopTypeDaySale","shopNum")
  378.                 .collect();
  379.         Map<String,List<ProjectTypeShopNumber>> map=new HashMap<>();
  380.         if(rows.length>0) {
  381.             for (Row row : rows) {
  382.                 ProjectTypeShopNumber pac=new ProjectTypeShopNumber();
  383.                 String mallName=null;
  384.                 if(!row.isNullAt(0)){//项目、mall名称
  385.                     mallName=row.getString(0);
  386.                 }
  387.                 if(!row.isNullAt(1)){//业态名称
  388.                     pac.setN(row.getString(1));
  389.                 }else{
  390.                     pac.setN("其它");
  391.                 }
  392.                 if(!row.isNullAt(2)){//mall的业态日销售额
  393.                     pac.setSa(row.getDouble(2));
  394.                 }
  395.                 if(!row.isNullAt(3)){//mall的业态店铺数量
  396.                     pac.setSn(row.getLong(3));
  397.                 }
  398.                 if(map.containsKey(mallName)){//更新map中的list列表
  399.                     List<ProjectTypeShopNumber> pacList=map.get(mallName);
  400.                     pacList.add(pac);
  401.                 }else{//新的mall 新建列表放入map
  402.                     List<ProjectTypeShopNumber> pacList=new LinkedList<>();
  403.                     pacList.add(pac);
  404.                     map.put(mallName, pacList);
  405.                 }
  406.             }
  407.         }
  408.         //为没有产生账单的业态或mall填充默认数据，是数据看起来完整
  409.         mallSet.forEach(mallName->{
  410.             if(!map.containsKey(mallName)) {//没有账单的mall
  411.                 List<ProjectTypeShopNumber> list=new ArrayList<>();
  412.                 //遍历全量业态，进行填充
  413.                 ruianTypeAll.forEach(type_->{
  414.                     list.add(new ProjectTypeShopNumber(type_,0.0,0));
  415.                 });
  416.                 map.put(mallName,list);
  417.             }else{//有账单的mall
  418.                 //用来获取实际账单数据中已存在的业态
  419.                 Set<String> ruianTypeCur = new HashSet<>();
  420.                 //用来存放没有账单的业态
  421.                 Set<String> ruianTypeNew = new HashSet<>();
  422.                 //遍历数据，提取实际数据中的业态
  423.                 for(ProjectTypeShopNumber obj:map.get(mallName)){
  424.                     ruianTypeCur.add(obj.getN());
  425.                 }
  426.                 //将产生账单的业态和全量业态求差即
  427.                 ruianTypeNew.addAll(ruianTypeAll);
  428.                 ruianTypeNew.removeAll(ruianTypeCur);
  429.                 //将没有实际账单的业态数据填从（默认值填充）
  430.                 for(String type_:ruianTypeNew){
  431.                     map.get(mallName).add(new ProjectTypeShopNumber(type_,0.0,0));
  432.                 }
  433.             }
  434.         });
  435.         //为==各项目店铺数量==填充数据
  436.         projectShopNumber(map);
  437.         //转成json
  438.         Map<String, Map<String,List<ProjectTypeShopNumber>>> pmap=new HashMap<>();
  439.         pmap.put("各项目业态销售额对比", map);
  440.         String str=JSON.toJSONString(pmap);
  441.         System.out.println("====####--projectTypeSaleContrast##"+str);
  442.         RedisUtil.setObject("dptsc_11",str);
  443.     }
  444.  
  445.     //=============================================================================================================
  446.     /**
  447.      * 结账单数
  448.      *
  449.      * @param tsb
  450.      */
  451.     public void totalSettlementBillNumber(TotalSettlementBillNumber tsb) {
  452.          Map<String, TotalSettlementBillNumber> map=new HashMap<>();
  453.          map.put("结账单数", tsb);
  454.          String str=JSONObject.toJSONString(map);
  455.          System.out.println("====####--totalSettlementBillNumber##"+str);
  456.          //将结账单数放入redis
  457.          RedisUtil.setObject("dtsbn_6", str);
  458.     }
  459.  
  460.     /**
  461.      * 各区域销售占比
  462.      * 直接将数据封装成json,无需额外处理
  463.      *
  464.      * @param rs 已经封装好的数据，请参考{@link com.mengyao.graph.etl.apps.dashboard.service.SaleAnalysisService.projectContrast()}
  465.      */
  466.     private void areaSaleProportion(Map<String, List<AreaProjSale>> rs) {
  467.         Map<String, Map<String,List<AreaProjSale>>> pmap=new HashMap<>();
  468.         pmap.put("各区域销售额占比 ", rs);
  469.         //转成json
  470.         String str=JSONObject.toJSONString(pmap);
  471.         System.out.println("====####--areaSaleProportion##"+str);
  472.         //各区域销售占比dasp_1
  473.         RedisUtil.setObject("dasp_1", str);
  474.     }
  475.  
  476.     /**
  477.      *  各项目单均消费
  478.      *  版本二 营业时间24小时制
  479.      *    1、各项目：各个mall；
  480.      *    2、单均消费：mall的当日开始营业时间到当前时间累计销售额，应收额累计/结账单累计；
  481.      *  直接将数据封装成json，无需额外处理
  482.      *
  483.      *  @param rs 已经封装好的数据，请参考{@link com.mengyao.graph.etl.apps.dashboard.service.SaleAnalysisService.projectContrast()}
  484.      */
  485.     private void projectAvgConsumer(Map<String, List<AreaProjSale>> rs) {
  486.         Map<String, Map<String,List<AreaProjSale>>> pmap=new HashMap<>();
  487.         pmap.put("各项目单均消费", rs);
  488.         //转成json
  489.         String str=JSONObject.toJSONString(pmap);
  490.         System.out.println("====####--projectAvgConsumer##"+str);
  491.         //10各项目单均消费dpac_10
  492.         RedisUtil.setObject("dpac_10", str);
  493.     }
  494.  
  495.     /**
  496.      * 各项目店铺数量
  497.      * 数据填充自上面的方法
  498.      *
  499.      * {@link projectTypeSaleContrast(SparkSession session,String beginYMDH,String endYMDH)}
  500.      */
  501.     private void projectShopNumber(Map<String,List<ProjectTypeShopNumber>> map) {
  502.         //转成json
  503.         Map<String, Map<String,List<ProjectTypeShopNumber>>> pmap=new HashMap<>();
  504.         pmap.put("各项目店铺数量", map);
  505.         String str=JSON.toJSONString(pmap);
  506.         System.out.println("====####--projectShopNumber##"+str);
  507.         RedisUtil.setObject("dpsn_12",str);
  508.     }
  509.  
  510. }

Spark-2.3.2【SparkStreaming+SparkSQL-实时仪表盘应用】的更多相关文章

  1. spark复习笔记(7):sparkstreaming

    一.介绍 1.sparkStreaming是核心模块Spark API的扩展,具有可伸缩,高吞吐量以及容错的实时数据流处理等.数据可以从许多来源(如Kafka,Flume,Kinesis或TCP套接字 ...

  2. Spark(十六)【SparkStreaming基本使用】

    目录 一. SparkStreaming简介 1. 相关术语 2. SparkStreaming概念 3. SparkStreaming架构 4. 背压机制 二. Dstream入门 1. WordC ...

  3. Spark入门实战系列--6.SparkSQL（下）--Spark实战应用

    [注]该系列文章以及使用到安装包/测试数据 可以在<倾情大奉送--Spark入门实战系列>获取 .运行环境说明 1.1 硬软件环境 线程,主频2.2G,10G内存 l  虚拟软件:VMwa ...

  4. 使用Flume+Kafka+SparkStreaming进行实时日志分析

    每个公司想要进行数据分析或数据挖掘,收集日志.ETL都是第一步的,今天就讲一下如何实时地(准实时,每分钟分析一次)收集日志,处理日志,把处理后的记录存入Hive中,并附上完整实战代码 1. 整体架构 ...

  5. flume+sparkStreaming实例 实时监控文件demo

    1,flume所在的节点不和spark同一个集群  v50和 10-15节点 flume在v50里面 flume-agent.conf spark是开的work节点,就是单点计算节点,不涉及到mast ...

  6. Spark入门实战系列--6.SparkSQL（上）--SparkSQL简介

    [注]该系列文章以及使用到安装包/测试数据 可以在<倾情大奉送--Spark入门实战系列>获取 .SparkSQL的发展历程 1.1 Hive and Shark SparkSQL的前身是 ...

  7. Spark入门实战系列--6.SparkSQL（中）--深入了解SparkSQL运行计划及调优

    [注]该系列文章以及使用到安装包/测试数据 可以在<倾情大奉送--Spark入门实战系列>获取 1.1  运行环境说明 1.1.1 硬软件环境 线程,主频2.2G,10G内存 l  虚拟软 ...

  8. Spark RDDs vs DataFrames vs SparkSQL

    简介 Spark的 RDD.DataFrame 和 SparkSQL的性能比较. 2方面的比较 单条记录的随机查找 aggregation聚合并且sorting后输出 使用以下Spark的三种方式来解 ...

  9. Spark（十二）SparkSQL简单使用

    一.SparkSQL的进化之路 1.0以前:   Shark 1.1.x开始:SparkSQL(只是测试性的)  SQL 1.3.x:          SparkSQL(正式版本)+Datafram ...

随机推荐

  1. Github Atom汉化方式

    1.下载:Atom  https://atom.io/ 2.安装 3.菜单栏 -- Setting --- Install --- 搜索Chinese --安装汉化包 4.重启 生效.

  2. VS升级后的配置问题

    当vs升级到更新的版本后,运行原来无误的程序会出现一系列问题. 例如:打不开iostream文件,lib文件,系统找不到文件等等 出现这类问题的原因是,编译环境的include path和librar ...

  3. C运行时库

    原文地址:http://blog.csdn.net/wqvbjhc/article/details/6612099 在开发window程序是经常会遇到编译好好的程序拿到另一台机器上面无法运行的情况,这 ...

  4. 将下载到本地的JAR包手动添加到Maven仓库(转)

    常用Maven仓库网址:http://mvnrepository.com/http://search.maven.org/http://repository.sonatype.org/content/ ...

  5. opencv 和 parfor

    一次遇到两个不熟悉的,因此在一起记一下. OpenCV的全称是:Open Source Computer Vision Library. OpenCv是一个基于(开源)发行的跨平台计算机视觉库,可以运 ...

  6. bzoj3998-弦论

    给定一个长度为\(n(n\le 5\times 10^5)\)的字符串,求它的第\(k\)小字串.有两种模式: \(Type=0\),不同位置的相同字串只算一个 \(Type=1\),不同位置相同字串 ...

  7. openstack之keystone部署

    前言 openstack更新频率是挺快的,每六个月更新一次(命名是是以A-Z的方式,Austin,Bexar...Newton).博主建议大家先可一种版本研究,等某一版本研究透彻了,在去研究新的版本. ...

  8. BZOJ4028 HEOI2015公约数数列（分块）

    前缀gcd的变化次数是log的,考虑对每一种gcd查询,问题变为查询一段区间是否存在异或前缀和=x/gcd. 无修改的话显然可以可持久化trie,但这玩意实在没法支持修改.于是考虑分块. 对于每一块将 ...

  9. POJ2826：An Easy Problem?!——题解（配特殊情况图）

    http://poj.org/problem?id=2826 题目大意:给两条线,让它接竖直下的雨,问其能装多少横截面积的雨. ———————————————————————————— 水题,看题目即 ...

  10. BZOJ2659 [Beijing wc2012]算不出的算式 【数形结合】

    题目链接 BZOJ2659 题解 真没想到,, 观察式子 \[\sum\limits_{k = 1}^{\frac{p - 1}{2}} \lfloor \frac{kq}{p} \rfloor\] ...