<Using parquet with impala>

Operations upon Impala

Create table

• stored as parquet
• like parquet '/user/etl/datafile1' stored as parquet

Loading data

• shuffle / no shuffle to choose
• 使用insert ... select 而不是 insert ... values, 因为后者产生a separate tiny data file.
• impala decodes the column data in Parquet files based on the ordinal position of the columns.
• loading data into parquet tables is a memory-intensive operation --> buffered until it reaches one data block in size, then organized and compressed in memory before written out.

Query performance

• Query performance for Parquet tables depends on the number of columns needed to process the SELECT list and WHERE clauses of the query, the way data is divided into large data files with block size equal to file size, the reduction in I/O by reading the data for each column in compressed format, which data files can be skipped(for partitioned tables), and the CPU overhead of decompressing the data for each column. [查询性能取决于select & where子句中的列数目，数据切分成文件的方式，读取压缩形式的每一列的I/O，哪些数据文件可以被跳过(对于partitioned tables)，解压缩每一列的CPU负担]

• An efficient query for a Parquet table:

  • select avg(income) from census_data where state = 'CA';

  • 只处理两列；
  • 如果表是通过state列partitioned的，这将会更高效。因为该语句对每个数据文件只需要读取并解码一列，并且只需要读取在'CA'文件目录下的文件
• An inefficient query:

  • select * from census_data;

  • Impala have to read the entire contents, decompress each column for each row group;
  • Does not take advantage of unique strengths for Parquet data files

• Impala can optimize queries on Parquet tables, especially join queries, better when statistics are available for all the tables. See COMPUTE STATS Statement following.

• 以上，高效地查询可以通过这些方式：
  • 查询语句方面：select & where从句只操作必须的列；尽量使用分区的列来减少所需查找的文件。
  • 文件方面：合适的block size & 压缩方式；使用分区。

Partitioning for Parquet tables

• Partitioning: An important performance technique.

• The Parquet file format is ideal for tables containing many columns, where most queries only refer to a small subset of the columns.[Parquet文件格式适合含有很多列，并且大多数查询只涉及到部分列的场景。]

• By using WHERE clause that refer to the partition key columns, impala can skip the data files for certain partitions entirely.

• Inserting into a partitioned Parquet table can be a resource-intensive operation.

Snappy and GZip Compression

• using the INSERT statement with COMPRESSION_CODEC query option to control the underlying compression —> snappy (the default), gzip, and none.

• TBD...

For Impala Complex Types

• Impala supports the complex types ARRAY, STRUCT, and MAP; See Complex Types following;

How Parquet data files are organized

• although column-oriented file format, Parquet keeps all the data for a row within the same data file, to ensure that the columns for a row are always available on the same node for processing.[Parquet是面向列的文件格式，但是Parquet将一行中的所有数据保存在同一数据文件中，来确保一行中的所有列能在同一节点中被处理]。

• Within that data file, the data for a set of rows is rearranged so that all the values from the first column are organized in one contiguous block, then all the values from the second column, and so on. [在数据文件内部，就是按列式存储方式存储的啦]

RLE and Dictionary Encoding

• Paquet uses some automatic compression techniques, such as run-length encoding (RLE) and dictionary encoding, based on analysis of the actual data values.[Parquet会根据实际数据值使用一些自动的压缩技术，比如游程长度编码和字典编码]

• RLE: condenses sequences of repeated data values. For example, if many consecutive rows all contain the same value for a country code, those repeating values can be represented by the value followed by a count of how many times it appears consecutively. [RLE实际上就是将连续的相同值压缩表示成值+个数]

• Dictionary encoding:...

COMPUTE STATS Statement

• Gathers information about volume and distribution of data in a table and all associated columns and partitions. [通过compute语句收集表以及所有相关的列和分区的体积和分布信息。]

• The information is stored in the metastore database, and used by Impala to help optimize queries. [impala可以根据收集到的信息优化查询。]

  • For example, if Impala can determine that a table is large or small, or has many or few distinct values it can organize parallelize the work appropriately for a join query or insert operation. [比如，如果impala知道表的大小，或者是否有很多不同值，它就可以为join查询以及insert操作组织合适的并行度]

• COMPUTE STATS

• TBD...

Complext Types

• Complex types (also referred to as nested types) let you represent multiple data values within a single row/column position.

Benefits

• The reasons for using it:

  • already have data produced by Hive or other non-Impala component that uses the complex type column names;

  • Your data model originates with a non-SQL programming language or a NoSQL data management system. For example, if you are representing Python data expressed as nested lists, dictionaries, and tuples, those data structures correspond closely to Impala ARRAY, MAP, and STRUCT types;

  • Your analytic queries involving multiple tables could benefit from greater locality during join processing. By packing more related data items within each HDFS data block, complex types let join queries avoid the network overhead of the traditional Hadoop shuffle or broadcast join techniques. [包含多个表的分析查询可以受益于join操作中的局部性。通过将很多相关的数据项packing到一个HDFS数据块中，复杂类型减轻了join带来的网络负担。]

Overview

• Array, map, struct

  • The ARRAY and MAP types are closely related: they represent collections with arbitrary numbers of elements, where each element is the same type. 【key of map is not necessarily unique.】

  • STRUCT groups together a fixed number of items into a single element.

• The elements of an ARRAY or MAP, or the fields of a STRUCT, can also be other complex types. —> nested

• when visualizing ur data model in familiar SQL terms, u can think of each ARRAY or MAP as a miniature table, and each STRUCT as a row within such a table.

• By default, the table represented by an ARRAY has two columns, POS to represent ordering of elements, and ITEM representing the value of each element. Likewise, by default, the table represented by a MAP encodes key-value pairs, and therefore has two columns, KEY and VALUE.

Design Consideration

How complex types differ from traditional data warehouse schemas

• In traditional data warehousing, related values were typically arranged in one of two ways: [传统的数据仓库中，related values有两种组织方式]

  • Split across normalized tables, using foreign key columns. This arrangement avoided duplicate data and therefore the data wad compact. But join queries could be expensive because the related data had to be retrieved from separate locations. [分隔成多个多个normalized表，通过外键关联到一起。这种方式避免了重复数据，因为表是紧凑的。但是join查询是昂贵的，因为相关数据需要从不同的位置取得。]

  • Flattened into a single denormalized table. This removing the need for join queries, but values were repeated.The extra data volume could cause performance issues in other parts of the workflow, such as longer ETL cycles or more expensive full-table scans during queries.[扁平化成一个denormalized表。这种方式避免了join查询，但是数据重复。这种重复还会造成其他情况下的性能问题，比如更昂贵的全表扫描操作。]

• Complex types represent a middle ground that addresses these performance and volume concerns. [复杂类型是一种性能和体积之间的折中方式。]

• ...

Using complex types from SQL

DDL Statements

• create table contacts_array_of_phones
  (
      id BIGINT,
      name STRING,
      address STRING,
      phone_number ARRAY<STRING>
  ) stored as parquet;

• create table contacts_unlimited_phones
  (
      id BIGINT,
      name STRING,
      address STRING,
      phone_number MAP<STRING, STRING>
  ) stored as PARQUET;

SQL statements that support complex types

• Currently: CREATE TABLE, ALTER TABLE, DESCRIBE, LOAD DATA, and SELECT.

• The result set of an Impala query always contains all scalar types; the elements and fields within any complex type queries must be "unpacked" using join queries. —> A query cannot directly retrieve the entire value for a complex type column. [复杂类型内的字段必须先通过join查询被'unpacked'。也就是查询不能直接取得复杂类型的值。]

  • select c_orders from customer limit 1;
    
    ERROR: AnalysisException: Expr 'c_orders' in select list returns a complex type 'ARRAY<STRUCT<o_orderkey:BIGINT,o_orderstatus:STRING, ... l_receiptdate:STRING,l_shipinstruct:STRING,l_shipmode:STRING,l_comment:STRING>>>>'.
    Only scalar types are allowed in the select list.

  • --- only scalar in select, and add region.r_nations in from
    select r_name, r_nations.item.n_name from region, region.r_nations limit 7;

• select * only retrieves scalar columns. [select * 语句只会返回所有的scalar列]

• the following queries work equivalently. They each return customer and order data for customers that have at least one order. [以下两个查询是等价的。它们只会返回至少有一个订单的顾客]

  • select c.c_name, o.o_orderkey
    from customer c, c.c_orders o limit 5;
    +--------------------+------------+
    | c_name             | o_orderkey |
    +--------------------+------------+
    | Customer#000072578 | 558821     |
    | Customer#000072578 | 2079810    |
    | Customer#000072578 | 5768068    |
    | Customer#000072578 | 1805604    |
    | Customer#000072578 | 3436389    |
    +--------------------+------------+

  • select c.c_name, o.o_orderkey
    from customer c
    inner join c.c_orders o
    limit 5;
    
    +--------------------+------------+
    | c_name             | o_orderkey |
    +--------------------+------------+
    | Customer#000072578 | 558821     |
    | Customer#000072578 | 2079810    |
    | Customer#000072578 | 5768068    |
    | Customer#000072578 | 1805604    |
    | Customer#000072578 | 3436389    |
    +--------------------+------------+

• The following query using an outer join returns customers that have orders, plus customers with no orders (no entries in the C_ORDERS array): [下面的查询使用outer join，会返回有订单和没有订单(在C_ORDERS数组下没有entries)的顾客。]

  • select c.c_custkey, o.o_orderkey
    from customer c left outer join c.c_orders o
    limit 5;
    
    +-----------+------------+
    | c_custkey | o_orderkey |
    +-----------+------------+
    | 60210     | NULL       |
    | 147873    | NULL       |
    | 72578     | 558821     |
    | 72578     | 2079810    |
    | 72578     | 5768068    |
    +-----------+------------+

• Correlated subqueries. Note the correlated reference to the table alias C. The COUNT(*)operation applies to all the elements of the C_ORDERS array for the corresponding row, avoiding the need for a GROUP BY clause. [仔细看，聚集在c的每一行(也就是每个customer)] [相关子查询：count(*)操作作用在相应行的c_orders数组上，避免了额外的group by子句。]

  • select c_name, howmany
    from customer c,
        (select count(*) from c.c_orders) v
    limit 5;
    
    +--------------------+---------+
    | c_name             | howmany |
    +--------------------+---------+
    | Customer#000030065 | 15      |
    | Customer#000065455 | 18      |
    | Customer#000113644 | 21      |
    | Customer#000111078 | 0       |
    | Customer#000024621 | 0       |
    +--------------------+---------+