寻找hive视图

如何hive视图

1.mysql数据库

[centos@s201 ~]$ mysql -uroot -proot
 
mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| azkaban            |
| big12              |
| hive               |
| mysql              |
| performance_schema |
+--------------------+
6 rows in set (0.05 sec)

2.找hive库

 show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| azkaban            |
| big12              |
| hive               |
| mysql              |
| performance_schema |
+--------------------+
 
mysql> use hive;

3.观表

show tables;
+---------------------------+
| Tables_in_hive            |
+---------------------------+
| AUX_TABLE                 |
| BUCKETING_COLS            |
| CDS                       |
| COLUMNS_V2                |
| COMPACTION_QUEUE          |
| COMPLETED_COMPACTIONS     |
| COMPLETED_TXN_COMPONENTS  |
| DATABASE_PARAMS           |
| DBS                       |
| DB_PRIVS                  |
| DELEGATION_TOKENS         |
| FUNCS                     |
| FUNC_RU                   |
| GLOBAL_PRIVS              |
| HIVE_LOCKS                |
| IDXS                      |
| INDEX_PARAMS              |
| KEY_CONSTRAINTS           |
| MASTER_KEYS               |
| NEXT_COMPACTION_QUEUE_ID  |
| NEXT_LOCK_ID              |
| NEXT_TXN_ID               |
| NOTIFICATION_LOG          |
| NOTIFICATION_SEQUENCE     |
| NUCLEUS_TABLES            |
| PARTITIONS                |
| PARTITION_EVENTS          |
| PARTITION_KEYS            |
| PARTITION_KEY_VALS        |
| PARTITION_PARAMS          |
| PART_COL_PRIVS            |
| PART_COL_STATS            |
| PART_PRIVS                |
| ROLES                     |
| ROLE_MAP                  |
| SDS                       |
| SD_PARAMS                 |
| SEQUENCE_TABLE            |
| SERDES                    |
| SERDE_PARAMS              |
| SKEWED_COL_NAMES          |
| SKEWED_COL_VALUE_LOC_MAP  |
| SKEWED_STRING_LIST        |
| SKEWED_STRING_LIST_VALUES |
| SKEWED_VALUES             |
| SORT_COLS                 |
| TABLE_PARAMS              |
| TAB_COL_STATS             |
| TBLS                      |
| TBL_COL_PRIVS             |
| TBL_PRIVS                 |
| TXNS                      |
| TXN_COMPONENTS            |
| TYPES                     |
| TYPE_FIELDS               |
| VERSION                   |
| WRITE_SET                 |
+---------------------------+

4.TBLS表结构

mysql> desc TBLS;
+--------------------+--------------+------+-----+---------+-------+
| Field              | Type         | Null | Key | Default | Extra |
+--------------------+--------------+------+-----+---------+-------+
| TBL_ID             | bigint(20)   | NO   | PRI | NULL    |       |
| CREATE_TIME        | int(11)      | NO   |     | NULL    |       |
| DB_ID              | bigint(20)   | YES  | MUL | NULL    |       |
| LAST_ACCESS_TIME   | int(11)      | NO   |     | NULL    |       |
| OWNER              | varchar(767) | YES  |     | NULL    |       |
| RETENTION          | int(11)      | NO   |     | NULL    |       |
| SD_ID              | bigint(20)   | YES  | MUL | NULL    |       |
| TBL_NAME           | varchar(128) | YES  | MUL | NULL    |       |
| TBL_TYPE           | varchar(128) | YES  |     | NULL    |       |
| VIEW_EXPANDED_TEXT | mediumtext   | YES  |     | NULL    |       |
| VIEW_ORIGINAL_TEXT | mediumtext   | YES  |     | NULL    |       |
+--------------------+--------------+------+-----+---------+-------+

5.根据TBL_TYPE找到视图

mysql> select * from TBLS where tbl_type='VIRTUAL_VIEW';
+--------+-------------+-------+------------------+--------+-----------+-------+----------+--------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------+
| TBL_ID | CREATE_TIME | DB_ID | LAST_ACCESS_TIME | OWNER  | RETENTION | SD_ID | TBL_NAME | TBL_TYPE     | VIEW_EXPANDED_TEXT                                                                                                                                                                                                                    | VIEW_ORIGINAL_TEXT                                                                                                                        |
+--------+-------------+-------+------------------+--------+-----------+-------+----------+--------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------+
|    187 |  1544269997 |     6 |                0 | centos |         0 |   222 | a        | VIRTUAL_VIEW | select `a`.`id`, `a`.`tag`, count(*) as `count` from (select `temptags`.`id`,  `xx`.`tag` from `big12`.`temptags` lateral view explode(`parsejson`(`temptags`.`json`)) `xx` as `tag`) `a` group by `a`.`id`, `a`.`tag`                | select id, tag, count(*) as count from (select id,  tag from temptags lateral view explode(parsejson(json)) xx as tag) a group by id, tag |
|    193 |  1544336757 |     6 |                0 | centos |         0 |   229 | a1       | VIRTUAL_VIEW | select `logevent`.`deviceid`, `logevent`.`musicid`, sum(cast(`logevent`.`mark` as int)) as `sum` from `big12`.`logevent` where `logevent`.`musicid` is not null group by `logevent`.`deviceid`, `logevent`.`musicid`                  | select deviceid, musicid, sum(cast(mark as int)) as sum from logevent where musicId is not null group by deviceid, musicid                |
|    194 |  1544336831 |     6 |                0 | centos |         0 |   230 | a2       | VIRTUAL_VIEW | select
`a1`.`deviceid` ,
`a1`.`musicid`,
`a1`.`sum`,
max(`a1`.`sum`)over(partition by `a1`.`deviceid`) as `sum2`
from `big12`.`a1`                                                                                               | select
deviceid ,
musicid,
sum,
max(sum)over(partition by deviceid) as sum2
from a1                                                  |
|    227 |  1550817816 |     6 |                0 | centos |         0 |   262 | zz1      | VIRTUAL_VIEW | select `duowan_parquet`.`id`, `duowan_parquet`.`name`, `duowan_parquet`.`pass`, `duowan_parquet`.`email`, `duowan_parquet`.`nickname` from `big12`.`duowan_parquet` where substring(`duowan_parquet`.`id`,1,1) in (1,2,3,4,5,6,8,9,0) | select * from duowan_parquet where substring(id,1,1) in (1,2,3,4,5,6,8,9,0)                                                               |
+--------+-------------+-------+------------------+--------+-----------+-------+----------+--------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------+

6.查看存储库信息的DBS表

mysql> desc DBS;
+-----------------+---------------+------+-----+---------+-------+
| Field           | Type          | Null | Key | Default | Extra |
+-----------------+---------------+------+-----+---------+-------+
| DB_ID           | bigint(20)    | NO   | PRI | NULL    |       |
| DESC            | varchar(4000) | YES  |     | NULL    |       |
| DB_LOCATION_URI | varchar(4000) | NO   |     | NULL    |       |
| NAME            | varchar(128)  | YES  | UNI | NULL    |       |
| OWNER_NAME      | varchar(128)  | YES  |     | NULL    |       |
| OWNER_TYPE      | varchar(10)   | YES  |     | NULL    |       |
+-----------------+---------------+------+-----+---------+-------+

 select * from  DBS limit 10;
+-------+-----------------------+------------------------------------------------+-------------+------------+------------+
| DB_ID | DESC                  | DB_LOCATION_URI                                | NAME        | OWNER_NAME | OWNER_TYPE |
+-------+-----------------------+------------------------------------------------+-------------+------------+------------+
|     1 | Default Hive database | hdfs://s201/user/hive/warehouse                | default     | public     | ROLE       |
|     6 | NULL                  | hdfs://s201/user/hive/warehouse/big12.db       | big12       | centos     | USER       |
|    11 | NULL                  | hdfs://s201/user/hive/warehouse/music164.db    | music164    | centos     | USER       |
|    16 | NULL                  | hdfs://s201/user/hive/warehouse/big12_umeng.db | big12_umeng | centos     | USER       |
|    21 | NULL                  | hdfs://s201/user/hive/warehouse/big12_2.db     | big12_2     | centos     | USER       |
|    26 | NULL                  | hdfs://s201/user/hive/warehouse/iml.db         | iml         | centos     | USER       |
|    31 | NULL                  | hdfs://s201/user/hive/warehouse/wqbin.db       | wqbin       | centos     | USER       |
+-------+-----------------------+------------------------------------------------+-------------+------------+------------+

7.如何删除视图跑路的脚本如下

7.1连接mysql

import pymysql
 
conn = pymysql.connect(host='192.168.154.201', user='root', passwd='root', db='hive')
cur = conn.cursor()
 
# 查询
sql = "select NAME,TBL_NAME from TBLS a join DBS b on a.DB_ID=b.DB_ID "
reCount = cur.execute(sql)  # 返回受影响的行数
print(reCount)
data = cur.fetchall()  # 返回数据,返回的是tuple类型
print(data)
 
cur.close()
conn.close()

(('big12', 'a'), ('big12', 'a1'), ('big12', 'a2'), ('big12', 'zz1'))

7.2删除hive视图

使用pyhive连接hive删除所有视图

import pymysql
conn = pymysql.connect(host='192.168.154.201', user='root', passwd='root', db='hive')
cur = conn.cursor()
# 查询
sql = "select NAME,TBL_NAME from TBLS a join DBS b on a.DB_ID=b.DB_ID where a.TBL_TYPE='VIRTUAL_VIEW'"
reCount = cur.execute(sql)  # 返回受影响的行数
data = cur.fetchall()  # 返回数据,返回的是tuple类型
print(data)
cur.close()
conn.close()
 
from pyhive import hive
import thrift
import sasl
import thrift_sasl
conn = hive.Connection(host='192.168.154.201', port=10000, database='big12',auth='NOSASL')
cursor=conn.cursor()
for a,b in data:
    cursor.execute("drop view "+a+"."+b)
conn.close()