一、UDF函数编写

1.步骤

  1. 1.继承UDF类
  2. 2.重写evalute方法
  1. 1、继承GenericUDF
  2. 2、实现initialize、evaluate、getDisplayString方法

2.案例

实现lower函数:

  1. package com.xxx.udf;
  2.  
  3. import org.apache.hadoop.hive.ql.exec.UDF;
  4. import org.apache.hadoop.io.Text;
  5.  
  6. public class LowerUDF extends UDF {
  7.     public Text evaluate(Text input){
  8.         if(null == input){
  9.             return null;
  10.         }
  11.         String inputValue = input.toString().trim() ;
  12.         if(null == inputValue){
  13.             return null ;
  14.         }
  15.         return new Text(inputValue.toLowerCase()) ;
  16.     }
  17. }
  1. package com.xxx.udf;
  2.  
  3. import org.apache.hadoop.hive.ql.exec.UDFArgumentException;
  4. import org.apache.hadoop.hive.ql.exec.UDFArgumentLengthException;
  5. import org.apache.hadoop.hive.ql.metadata.HiveException;
  6. import org.apache.hadoop.hive.ql.udf.generic.GenericUDF;
  7. import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
  8. import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorFactory;
  9. import org.apache.hadoop.hive.serde2.objectinspector.primitive.StringObjectInspector;
  10. import org.apache.hadoop.io.Text;
  11.  
  12. public class LowerUDF extends GenericUDF {
  13.  
  14.     StringObjectInspector str ;
  15.     @Override
  16.     public ObjectInspector initialize(ObjectInspector[] arguments) throws UDFArgumentException {
  17.         //判断输入参数个数是否合法
  18.         if (arguments.length != 1) {
  19.             throw new UDFArgumentLengthException("输入参数长度不合法,应该为一个参数");
  20.         }
  21.  
  22.         //判断输入参数类型是否合法
  23.         if (!(arguments[0] instanceof StringObjectInspector)) {
  24.             throw new UDFArgumentException("输入非法参数，应为字符串类型");
  25.         }
  26.  
  27.         str=(StringObjectInspector)arguments[0];
  28.         //确定返回值类型
  29.         return PrimitiveObjectInspectorFactory.javaStringObjectInspector;
  30.     }
  31.  
  32.     @Override
  33.     public Object  evaluate(DeferredObject[] arguments) throws HiveException {
  34.         String input = str.getPrimitiveJavaObject(arguments[0].get());
  35.         return new Text(input.toLowerCase());
  36.     }
  37.  
  38.     @Override
  39.     public String getDisplayString(String[] children) {
  40.         return "方法的描述信息";
  41.     }
  42. }

3.打成jar包上传

mvn clean package

4.在hive中创建临时函数

  1. add jar /home/xxx/yf/to_lower.jar;
  2. create temporary function to_lower as 'com.xxx.udf.LowerUDF';
  3. select to_lower("DSJIFASD") from dual;
  4. drop temporary function comparestringbysplit;

二、UDAF函数编写

1.步骤

  1. 1、继承AbstractGenericUDAFResolver
  2. 2、继承GenericUDAFEvaluator
  3. 3、Evaluator需要实现 init、iterate、terminatePartial、merge、terminate这几个函数
  4.    init初始化
  5.    iterate函数处理读入的行数据
  6.    terminatePartial返回iterate处理的中间结果
  7.    merge合并上述处理结果
  8.    terminate返回最终值

2.案例

实现avg

  1. package com.xxx.udf;
  2.  
  3. import org.apache.hadoop.hive.ql.exec.UDAF;
  4. import org.apache.hadoop.hive.ql.exec.UDAFEvaluator;
  5.  
  6. public class Avg extends UDAF {
  7.     public static class AvgState {
  8.         private long mCount;
  9.         private double mSum;
  10.     }
  11.  
  12.     public static class AvgEvaluator implements UDAFEvaluator {
  13.         AvgState state;
  14.  
  15.         public AvgEvaluator() {
  16.             super();
  17.             state = new AvgState();
  18.             init();
  19.         }
  20.  
  21.         /**
  22.          * init函数类似于构造函数，用于UDAF的初始化
  23.          */
  24.         public void init() {
  25.             state.mSum = 0;
  26.             state.mCount = 0;
  27.         }
  28.  
  29.         /**
  30.          * iterate接收传入的参数，并进行内部的轮转。其返回类型为boolean * * @param o * @return
  31.          */
  32.  
  33.         public boolean iterate(Double o) {
  34.             if (o != null) {
  35.                 state.mSum += o;
  36.                 state.mCount++;
  37.             }
  38.             return true;
  39.         }
  40.  
  41.         /**
  42.          * terminatePartial无参数，其为iterate函数遍历结束后，返回轮转数据， * terminatePartial类似于hadoop的Combiner * * @return
  43.          */
  44.  
  45.         public AvgState terminatePartial() {
  46.             // combiner
  47.             return state.mCount == 0 ? null : state;
  48.         }
  49.  
  50.         /**
  51.          * merge接收terminatePartial的返回结果，进行数据merge操作，其返回类型为boolean * * @param o * @return
  52.          */
  53.  
  54.         public boolean merge(AvgState avgState) {
  55.             if (avgState != null) {
  56.                 state.mCount += avgState.mCount;
  57.                 state.mSum += avgState.mSum;
  58.             }
  59.             return true;
  60.         }
  61.  
  62.         /**
  63.          * terminate返回最终的聚集函数结果 * * @return
  64.          */
  65.         public Double terminate() {
  66.             return state.mCount == 0 ? null : Double.valueOf(state.mSum / state.mCount);
  67.         }
  68.     }
  69. }

实现sum

  1. package com.xxx.udf;
  2.  
  3. import org.apache.hadoop.hive.ql.exec.UDFArgumentException;
  4. import org.apache.hadoop.hive.ql.exec.UDFArgumentLengthException;
  5. import org.apache.hadoop.hive.ql.metadata.HiveException;
  6. import org.apache.hadoop.hive.ql.parse.SemanticException;
  7. import org.apache.hadoop.hive.ql.udf.generic.AbstractGenericUDAFResolver;
  8. import org.apache.hadoop.hive.ql.udf.generic.GenericUDAFEvaluator;
  9. import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
  10. import org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector;
  11. import org.apache.hadoop.hive.serde2.objectinspector.primitive.DoubleObjectInspector;
  12. import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorFactory;
  13. import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorUtils;
  14. import org.apache.hadoop.hive.serde2.objectinspector.primitive.StringObjectInspector;
  15. import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo;
  16. import org.apache.hadoop.io.DoubleWritable;
  17. import org.apache.hadoop.io.LongWritable;
  18.  
  19. public class Test extends AbstractGenericUDAFResolver {
  20.  
  21.     /**
  22.      * 获取处理逻辑类
  23.      * @param info
  24.      * @return
  25.      * @throws SemanticException
  26.      */
  27.     @Override
  28.     public GenericUDAFEvaluator getEvaluator(TypeInfo[] info) throws SemanticException {
  29.         //判断输入参数是否合法,参数个数，参数类型
  30.         if (info.length != 1) {
  31.             throw new UDFArgumentLengthException("输入参数个数非法，一个参数");
  32.         }
  33.  
  34.         return new GenericEvaluate();
  35.     }
  36.  
  37.     //处理逻辑类
  38.     public static class GenericEvaluate extends GenericUDAFEvaluator {
  39.         private PrimitiveObjectInspector input;
  40.         private DoubleWritable result ;                   //保存最终结果
  41.         private MyAggregationBuffer myAggregationBuffer;  //自定义聚合列，保存临时结果
  42.  
  43.         //自定义AggregationBuffer
  44.         public static class MyAggregationBuffer implements AggregationBuffer {
  45.             Double sum;
  46.         }
  47.  
  48.         @Override  //指定返回类型
  49.         public ObjectInspector init(Mode m, ObjectInspector[] parameters) throws HiveException {
  50.             super.init(m, parameters);
  51.             result = new DoubleWritable(0);
  52.             input = (PrimitiveObjectInspector) parameters[];
  53.             // 指定返回结果类型
  54.             return PrimitiveObjectInspectorFactory.writableDoubleObjectInspector;
  55.         }
  56.  
  57.         @Override   //获得一个聚合的缓冲对象，每个map执行一次
  58.         public AggregationBuffer getNewAggregationBuffer() throws HiveException {
  59.             MyAggregationBuffer myAggregationBuffer = new MyAggregationBuffer();
  60.             reset(myAggregationBuffer);  // 重置聚合值
  61.             return myAggregationBuffer;
  62.         }
  63.  
  64.         @Override
  65.         public void reset(AggregationBuffer agg) throws HiveException {
  66.             MyAggregationBuffer newAgg = (MyAggregationBuffer) agg;
  67.             newAgg.sum = 0.0;
  68.         }
  69.  
  70.         @Override  // 传入参数值聚合
  71.         public void iterate(AggregationBuffer agg, Object[] parameters) throws HiveException {
  72.             MyAggregationBuffer myAgg = (MyAggregationBuffer) agg;
  73.             double inputNum = PrimitiveObjectInspectorUtils.getDouble(parameters[], input);
  74.             myAgg.sum += inputNum;
  75.         }
  76.  
  77.         @Override  //
  78.         public Object terminatePartial(AggregationBuffer agg) throws HiveException {
  79.             MyAggregationBuffer newAgg = (MyAggregationBuffer) agg;
  80.             result.set(newAgg.sum);
  81.             return result;
  82.         }
  83.  
  84.         @Override  // 合并
  85.         public void merge(AggregationBuffer agg, Object partial) throws HiveException {
  86.             double inputNum = PrimitiveObjectInspectorUtils.getDouble(partial, input);
  87.             MyAggregationBuffer newAgg = (MyAggregationBuffer) agg;
  88.             newAgg.sum += inputNum;
  89.         }
  90.  
  91.         @Override  //输出最终结果
  92.         public Object terminate(AggregationBuffer agg) throws HiveException {
  93.             MyAggregationBuffer aggregationBuffer = (MyAggregationBuffer) agg;
  94.             result.set(aggregationBuffer.sum);
  95.             return result;
  96.         }
  97.     }
  98. }

3.打包

mvn clean package

4.创建临时函数

  1. add jar /home/xxx/yf/my_avg.jar;
  2. create temporary function my_avg as 'com.xxx.udf.UDTFExplode';
  3. select my_avg() from dual;
  4. drop temporary function my_avg;

三、UDTF函数编写

1.步骤

  1. 1.继承GenericUDTF
  2. 2.重写initialize、process方法
  3.   initialize初始化校验参数是否正确、
  4.   process处理返回结果、
  5.   forward将结果返回

2.案例

将字符串按照元素索引分别输出，如：‘a,c,b’   -- > a,1    c,2  b,3

  1. package com.suning.udf;
  2.  
  3. import java.util.ArrayList;
  4.  
  5. import org.apache.hadoop.hive.ql.exec.UDFArgumentException;
  6. import org.apache.hadoop.hive.ql.exec.UDFArgumentLengthException;
  7. import org.apache.hadoop.hive.ql.metadata.HiveException;
  8. import org.apache.hadoop.hive.ql.udf.generic.GenericUDTF;
  9. import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
  10. import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory;
  11. import org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector;
  12. import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorFactory;
  13.  
  14. public class UDTFExplode extends GenericUDTF {
  15.  
  16.     @Override
  17.     public void close() throws HiveException {
  18.         // TODO Auto-generated method stub
  19.  
  20.     }
  21.  
  22.     @Override
  23.     public StructObjectInspector initialize(ObjectInspector[] args) throws UDFArgumentException {
  24.         if (args.length != 1) {
  25.             throw new UDFArgumentLengthException("ExplodeMap takes only one argument");
  26.         }
  27.         if (args[].getCategory() != ObjectInspector.Category.PRIMITIVE) {
  28.             throw new UDFArgumentException("ExplodeMap takes string as a parameter");
  29.         }
  30.  
  31.         ArrayList<String> fieldNames = new ArrayList<String>();
  32.         ArrayList<ObjectInspector> fieldOIs = new ArrayList<ObjectInspector>();
  33.         fieldNames.add("col1");
  34.         fieldOIs.add(PrimitiveObjectInspectorFactory.javaStringObjectInspector);
  35.         fieldNames.add("col2");
  36.         fieldOIs.add(PrimitiveObjectInspectorFactory.javaStringObjectInspector);
  37.  
  38.         return ObjectInspectorFactory.getStandardStructObjectInspector(fieldNames, fieldOIs);
  39.     }
  40.  
  41.     @Override
  42.     public void process(Object[] args) throws HiveException {
  43.         // TODO Auto-generated method stub
  44.         String input = args[].toString();
  45.         String[] test = input.split(",");
  46.         for (int i = 0; i < test.length; i++) {
  47.             try {
  48.                 String[] result = (test[i]+":"+String.valueOf(i+1)).split(":");
  49.                 forward(result);
  50.             } catch (Exception e) {
  51.                 continue;
  52.             }
  53.         }
  54.     }
  55. }

3.打包

mvn clean package

4.创建临时函数

  1. add jar /home/xxx/yf/str_index.jar;
  2. create temporary function str_index as 'com.xxx.udf.UDTFExplode';
  3. select str_index("a,c,b") from dual;
  4. drop temporary function str_index;

pom.xml

  1. <?xml version="1.0" encoding="UTF-8"?>
  2. <project xmlns="http://maven.apache.org/POM/4.0.0"
  3.          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  4.          xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  5.     <modelVersion>4.0.0</modelVersion>
  6.  
  7.     <groupId>spark-hive</groupId>
  8.     <artifactId>spark-hive</artifactId>
  9.     <version>1.0-SNAPSHOT</version>
  10.  
  11.     <properties>
  12.         <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
  13.         <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
  14.         <scala.version>2.11.8</scala.version>
  15.         <spark.version>2.1.0.9</spark.version>
  16.         <spark.artifactId.version>2.11</spark.artifactId.version>
  17.     </properties>
  18.     <dependencies>
  19.         <dependency>
  20.             <groupId>commons-logging</groupId>
  21.             <artifactId>commons-logging</artifactId>
  22.             <version>1.1.1</version>
  23.             <type>jar</type>
  24.         </dependency>
  25.  
  26.         <dependency>
  27.             <groupId>org.apache.commons</groupId>
  28.             <artifactId>commons-lang3</artifactId>
  29.             <version>3.1</version>
  30.         </dependency>
  31.  
  32.         <dependency>
  33.             <groupId>log4j</groupId>
  34.             <artifactId>log4j</artifactId>
  35.             <version>1.2.17</version>
  36.         </dependency>
  37.  
  38.         <dependency>
  39.             <groupId>org.apache.hadoop</groupId>
  40.             <artifactId>hadoop-common</artifactId>
  41.             <version>2.6.2</version>
  42.         </dependency>
  43.         <dependency>
  44.             <groupId>mysql</groupId>
  45.             <artifactId>mysql-connector-java</artifactId>
  46.             <version>5.1.21</version>
  47.         </dependency>
  48.         <dependency>
  49.             <groupId>org.apache.spark</groupId>
  50.             <artifactId>spark-core_2.11</artifactId>
  51.             <version>2.1.0</version>
  52.         </dependency>
  53.         <dependency>
  54.             <groupId>org.apache.spark</groupId>
  55.             <artifactId>spark-streaming_2.11</artifactId>
  56.             <version>2.1.0</version>
  57.         </dependency>
  58.  
  59.         <dependency>
  60.             <groupId>org.apache.spark</groupId>
  61.             <artifactId>spark-streaming-kafka-0-8_2.11</artifactId>
  62.             <version>2.1.0</version>
  63.         </dependency>
  64.         <dependency>
  65.             <groupId>com.google.code.gson</groupId>
  66.             <artifactId>gson</artifactId>
  67.             <version>2.8.2</version>
  68.         </dependency>
  69.  
  70.         <dependency>
  71.             <groupId>org.apache.spark</groupId>
  72.             <artifactId>spark-sql_2.11</artifactId>
  73.             <version>2.1.0</version>
  74.         </dependency>
  75.  
  76.         <dependency>
  77.             <groupId>com.alibaba</groupId>
  78.             <artifactId>fastjson</artifactId>
  79.             <version>1.2.29</version>
  80.         </dependency>
  81.         <dependency>
  82.             <groupId>org.apache.spark</groupId>
  83.             <artifactId>spark-hive_${spark.artifactId.version}</artifactId>
  84.             <version>${spark.version}</version>
  85.             <scope>provided</scope>
  86.         </dependency>
  87.         <!--flink dependency-->
  88.         <dependency>
  89.             <groupId>org.apache.flink</groupId>
  90.             <artifactId>flink-java</artifactId>
  91.             <version>1.5.0</version>
  92.         </dependency>
  93.         <dependency>
  94.             <groupId>org.apache.flink</groupId>
  95.             <artifactId>flink-streaming-java_2.11</artifactId>
  96.             <version>1.5.0</version>
  97.         </dependency>
  98.         <dependency>
  99.             <groupId>org.apache.flink</groupId>
  100.             <artifactId>flink-clients_2.11</artifactId>
  101.             <version>1.5.0</version>
  102.         </dependency>
  103.         <dependency>
  104.             <groupId>org.apache.flink</groupId>
  105.             <artifactId>flink-connector-wikiedits_2.11</artifactId>
  106.             <version>1.5.0</version>
  107.         </dependency>
  108.         <!--hbase dependency-->
  109.         <dependency>
  110.             <groupId>org.apache.hbase</groupId>
  111.             <artifactId>hbase</artifactId>
  112.             <version>0.98.8-hadoop2</version>
  113.             <type>pom</type>
  114.         </dependency>
  115.         <dependency>
  116.             <groupId>org.apache.hbase</groupId>
  117.             <artifactId>hbase-client</artifactId>
  118.             <version>0.98.8-hadoop2</version>
  119.         </dependency>
  120.         <dependency>
  121.             <groupId>org.apache.hbase</groupId>
  122.             <artifactId>hbase-common</artifactId>
  123.             <version>0.98.8-hadoop2</version>
  124.         </dependency>
  125.         <dependency>
  126.             <groupId>org.apache.hbase</groupId>
  127.             <artifactId>hbase-server</artifactId>
  128.             <version>0.98.8-hadoop2</version>
  129.         </dependency>
  130.     </dependencies>
  131.     <build>
  132.         <plugins>
  133.             <plugin>
  134.                 <artifactId>maven-assembly-plugin</artifactId>
  135.                 <configuration>
  136.                     <descriptorRefs>
  137.                         <descriptorRef>jar-with-dependencies</descriptorRef>
  138.                     </descriptorRefs>
  139.                 </configuration>
  140.             </plugin>
  141.             <plugin>
  142.                 <groupId>org.codehaus.mojo</groupId>
  143.                 <artifactId>build-helper-maven-plugin</artifactId>
  144.                 <version>1.8</version>
  145.                 <executions>
  146.                     <execution>
  147.                         <id>add-source</id>
  148.                         <phase>generate-sources</phase>
  149.                         <goals>
  150.                             <goal>add-source</goal>
  151.                         </goals>
  152.                         <configuration>
  153.                             <sources>
  154.                                 <source>src/main/scala</source>
  155.                                 <source>src/test/scala</source>
  156.                             </sources>
  157.                         </configuration>
  158.                     </execution>
  159.                     <execution>
  160.                         <id>add-test-source</id>
  161.                         <phase>generate-sources</phase>
  162.                         <goals>
  163.                             <goal>add-test-source</goal>
  164.                         </goals>
  165.                         <configuration>
  166.                             <sources>
  167.                                 <source>src/test/scala</source>
  168.                             </sources>
  169.                         </configuration>
  170.                     </execution>
  171.                 </executions>
  172.             </plugin>
  173.             <plugin>
  174.                 <groupId>org.apache.maven.plugins</groupId>
  175.                 <artifactId>maven-compiler-plugin</artifactId>
  176.                 <version>2.3.2</version>
  177.                 <configuration>
  178.                     <source>1.7</source>
  179.                     <target>1.7</target>
  180.                     <encoding>${project.build.sourceEncoding}</encoding>
  181.                 </configuration>
  182.             </plugin>
  183.             <plugin>
  184.                 <groupId>org.scala-tools</groupId>
  185.                 <artifactId>maven-scala-plugin</artifactId>
  186.                 <executions>
  187.                     <execution>
  188.                         <goals>
  189.                             <goal>compile</goal>
  190.                             <goal>add-source</goal>
  191.                             <goal>testCompile</goal>
  192.                         </goals>
  193.                     </execution>
  194.                 </executions>
  195.                 <configuration>
  196.                     <scalaVersion>2.11.8</scalaVersion>
  197.                     <sourceDir>src/main/scala</sourceDir>
  198.                     <jvmArgs>
  199.                         <jvmArg>-Xms64m</jvmArg>
  200.                         <jvmArg>-Xmx1024m</jvmArg>
  201.                     </jvmArgs>
  202.                 </configuration>
  203.             </plugin>
  204.             <plugin>
  205.                 <groupId>org.apache.maven.plugins</groupId>
  206.                 <artifactId>maven-release-plugin</artifactId>
  207.                 <version>2.5.3</version>
  208.             </plugin>
  209.             <plugin>
  210.                 <groupId>org.apache.maven.plugins</groupId>
  211.                 <artifactId>maven-deploy-plugin</artifactId>
  212.                 <configuration>
  213.                     <skip>false</skip>
  214.                 </configuration>
  215.             </plugin>
  216.             <plugin>
  217.                 <groupId>org.apache.maven.plugins</groupId>
  218.                 <artifactId>maven-shade-plugin</artifactId>
  219.                 <version>2.4.1</version>
  220.                 <executions>
  221.                     <execution>
  222.                         <phase>package</phase>
  223.                         <goals>
  224.                             <goal>shade</goal>
  225.                         </goals>
  226.                         <configuration>
  227.                             <filters>
  228.                                 <filter>
  229.                                     <artifact>*:*</artifact>
  230.                                     <excludes>
  231.                                         <exclude>META-INF/*.SF</exclude>
  232.                                         org.apache.hive                          <exclude>META-INF/*.DSA</exclude>
  233.                                         <exclude>META-INF/*.RSA</exclude>
  234.                                     </excludes>
  235.                                 </filter>
  236.                             </filters>
  237.                             <minimizeJar>false</minimizeJar>
  238.                         </configuration>
  239.                     </execution>
  240.                 </executions>
  241.             </plugin>
  242.         </plugins>
  243.         <resources>
  244.             <resource>
  245.                 <directory>src/main/resources</directory>
  246.                 <filtering>true</filtering>
  247.             </resource>
  248.             <resource>
  249.                 <directory>src/main/resources/${profiles.active}</directory>
  250.             </resource>
  251.         </resources>
  252.  
  253.         <!-- 修复 Plugin execution not covered by lifecycle configuration -->
  254.         <pluginManagement>
  255.             <plugins>
  256.                 <plugin>
  257.                     <groupId>org.eclipse.m2e</groupId>
  258.                     <artifactId>lifecycle-mapping</artifactId>
  259.                     <version>1.0.0</version>
  260.                     <configuration>
  261.                         <lifecycleMappingMetadata>
  262.                             <pluginExecutions>
  263.                                 <pluginExecution>
  264.                                     <pluginExecutionFilter>
  265.                                         <groupId>org.codehaus.mojo</groupId>
  266.                                         <artifactId>build-helper-maven-plugin</artifactId>
  267.                                         <versionRange>[1.8,)</versionRange>
  268.                                         <goals>
  269.                                             <goal>add-source</goal>
  270.                                             <goal>add-test-source</goal>
  271.                                         </goals>
  272.                                     </pluginExecutionFilter>
  273.                                     <action>
  274.                                         <ignore></ignore>
  275.                                     </action>
  276.                                 </pluginExecution>
  277.  
  278.                                 <pluginExecution>
  279.                                     <pluginExecutionFilter>
  280.                                         <groupId>org.scala-tools</groupId>
  281.                                         <artifactId>maven-scala-plugin</artifactId>
  282.                                         <versionRange>[1.8,)</versionRange>
  283.                                         <goals>
  284.                                             <goal>compile</goal>
  285.                                             <goal>add-source</goal>
  286.                                             <goal>testCompile</goal>
  287.                                         </goals>
  288.                                     </pluginExecutionFilter>
  289.                                     <action>
  290.                                         <ignore></ignore>
  291.                                     </action>
  292.                                 </pluginExecution>
  293.                             </pluginExecutions>
  294.                         </lifecycleMappingMetadata>
  295.                     </configuration>
  296.                 </plugin>
  297.             </plugins>
  298.         </pluginManagement>
  299.     </build>
  300. </project>

UDF、UDAF、UDTF函数编写的更多相关文章

  1. 简述UDF/UDAF/UDTF是什么，各自解决问题及应用场景

    UDF User-Defined-Function 自定义函数 .一进一出: 背景 系统内置函数无法解决实际的业务问题,需要开发者自己编写函数实现自身的业务实现诉求. 应用场景非常多,面临的业务不同导 ...

  2. Hive 自定义函数 UDF UDAF UDTF

    1.UDF:用户定义(普通)函数,只对单行数值产生作用: 继承UDF类,添加方法 evaluate() /** * @function 自定义UDF统计最小值 * @author John * */ ...

  3. [转]HIVE UDF/UDAF/UDTF的Map Reduce代码框架模板

    FROM : http://hugh-wangp.iteye.com/blog/1472371 自己写代码时候的利用到的模板   UDF步骤: 1.必须继承org.apache.hadoop.hive ...

  4. hive中 udf,udaf,udtf

    1.hive中基本操作: DDL,DML 2.hive中函数 User-Defined Functions : UDF(用户自定义函数,简称JDF函数)UDF: 一进一出  upper  lower ...

  5. 【转】HIVE UDF UDAF UDTF 区别 使用

    原博文出自于:http://blog.csdn.net/longzilong216/article/details/23921235(暂时) 感谢! 自己写代码时候的利用到的模板   UDF步骤: 1 ...

  6. FlinkSQL使用自定义UDTF函数行转列-IK分词器

    一.背景说明 本文基于IK分词器,自定义一个UDTF(Table Functions),实现类似Hive的explode行转列的效果,以此来简明开发过程. 如下图Flink三层API接口中,Table ...

  7. hive UDTF函数

    之前说过HIVE,UDF(User-Defined-Function)函数的编写和使用,现在来看看UDTF的编写和使用. 1. UDTF介绍 UDTF(User-Defined Table-Gener ...

  8. hive自定义UDTF函数叉分函数

    hive自定义UDTF函数叉分函数 1.介绍 从聚合体日志中需要拆解出来各子日志数据,然后单独插入到各日志子表中.通过表生成函数完成这一过程. 2.定义ForkLogUDTF 2.1 HiveUtil ...

  9. PHP代码重用与函数编写

    代码重用与函数编写 1.使用require()和include()函数 这两个函数的作用是将一个文件爱你载入到PHP脚本中,这样就可以直接调用这个文件中的方法.require()和include()几 ...

随机推荐

  1. 微信小程序商业级实战

    1.微信开放能力:微信支付.微信登录.二维码生成.分享

  2. 【进阶3-2期】JavaScript深入之重新认识箭头函数的this（转）

    这是我在公众号(高级前端进阶)看到的文章,现在做笔记 https://github.com/yygmind/blog/issues/21 上篇文章详细的分析了各种this的情况,看过之后对this的概 ...

  3. 信息摘要算法之六：HKDF算法分析与实现

    HKDF是一种特定的键衍生函数(KDF),即初始键控材料的功能,KDF从其中派生出一个或多个密码强大的密钥.在此我们想要描述的是基于HMAC的HKDF. 1.HKDF概述 密钥派生函数(KDF)是密码 ...

  4. STM32L476应用开发之四：触摸屏驱动与数据交互

    数据交互可以说是任何一台仪器都需要的功能.我们的便携式气体分析仪,需要人来操作和配置,所以触摸屏就是我们必然的一个选择.本次我们计划采用3.5寸显示屏,串口通讯. 1.硬件设计 前面我们实验了串行通讯 ...

  5. maven install 报错 No compiler is provided in this environment. Perhaps you are running on a JRE rather than a JDK?

    1.控制台打印信息 [INFO] Scanning for projects... [INFO] [INFO] ---------------------< org.cqupt.mauger:R ...

  6. Confluence 6 后台中的默认空间模板设置

    Confluence 6 后台中的默认空间模板设置界面的布局. https://www.cwiki.us/display/CONFLUENCEWIKI/Customizing+Default+Spac ...

  7. ionic3 点击输入 框弹出白色遮罩 并把 界面顶到上面

    这个蛋疼 问题 ,在android 上面遇到这种情况 ,键盘弹出的时候 总有一个白色的遮罩在后面出现,网上查了很久都没发现原因. 偶然发现一个方法 ,试下了下 问题解决了. 代码如下: IonicMo ...

  8. SpringData使用与整合

    SpringData 整合源码:链接: https://pan.baidu.com/s/1_dDEEJoqaBTfXs2ZWsvKvA 提取码: cp6s(jar包自行寻找) author:Simpl ...

  9. LeetCode（113）：路径总和 II

    Medium! 题目描述: 给定一个二叉树和一个目标和,找到所有从根节点到叶子节点路径总和等于给定目标和的路径. 说明: 叶子节点是指没有子节点的节点. 示例:给定如下二叉树,以及目标和 sum = ...

  10. SQLmap超详细文档和实例演示

    第一部分,使用文档的说明 Options(选项): -h, -–help 显示此帮助消息并退出 -hh 显示更多帮助信息并退出 –-version 显示程序的版本号并退出 -v VERBOSE 详细级 ...