Apache Atlas 架构图

Atlas 支持多数据源接入:Hive、HBase、Storm等

Type System

Type

Atlas 中定义了一些元数据类型

── AtlasBaseTypeDef
│ ├── AtlasEnumDef
│ └── AtlasStructDef
│ ├── AtlasBusinessMetadataDef
│ ├── AtlasClassificationDef
│ ├── AtlasEntityDef
│ └── AtlasRelationshipDef
├── AtlasStructType
│ ├── AtlasBusinessMetadataType
│ ├── AtlasClassificationType
│ ├── AtlasRelationshipType
│ └── AtlasEntityType
│ └── AtlasRootEntityType
├── AtlasType
│ ├── AtlasArrayType
│ ├── AtlasBigDecimalType
│ ├── AtlasBigIntegerType
│ ├── AtlasByteType
│ ├── AtlasDateType
│ ├── AtlasDoubleType
│ ├── AtlasEnumType
│ ├── AtlasFloatType
│ ├── AtlasIntType
│ ├── AtlasLongType
│ ├── AtlasMapType
│ ├── AtlasObjectIdType
│ ├── AtlasShortType
│ ├── AtlasStringType
│ └── AtlasStructType
│ ├── AtlasBusinessMetadataType
│ ├── AtlasClassificationType
│ ├── AtlasEntityType
│ └── AtlasRelationshipType
├── AtlasTypeDefStore
│ └── AtlasTypeDefGraphStore
│ └── AtlasTypeDefGraphStoreV2
└── StructTypeDefinition
└── HierarchicalTypeDefinition
├── ClassTypeDefinition
└── TraitTypeDefinition

Entity

Entity 是基于类型的具体实现

AtlasEntity
├── AtlasEntityExtInfo
│ ├── AtlasEntitiesWithExtInfo
│ └── AtlasEntityWithExtInfo
├── AtlasEntityStore
│ └── AtlasEntityStoreV2
├── AtlasEntityStream
│ └── AtlasEntityStreamForImport
├── AtlasEntityType
│ └── AtlasRootEntityType
└── IAtlasEntityChangeNotifier
├── AtlasEntityChangeNotifier
└── EntityChangeNotifierNop

Attributes

针对模型定义属性

AtlasAttributeDef
└── AtlasRelationshipAttributeDef

AtlasAttributeDef 属性字段:

private String                   name;
private String typeName;
private boolean isOptional;
private Cardinality cardinality;
private int valuesMinCount;
private int valuesMaxCount;
private boolean isUnique;
private boolean isIndexable;
private boolean includeInNotification;
private String defaultValue;
private String description;
private int searchWeight = DEFAULT_SEARCHWEIGHT;
private IndexType indexType = null;
private List<AtlasConstraintDef> constraints;
private Map<String, String> options;
private String displayName; 具体实现: db:
"name": "db",
"typeName": "hive_db",
"isOptional": false,
"isIndexable": true,
"isUnique": false,
"cardinality": "SINGLE" columns:
"name": "columns",
"typeName": "array<hive_column>",
"isOptional": optional,
"isIndexable": true,
“isUnique": false,
"constraints": [ { "type": "ownedRef" } ]
  • isComposite - 是否复合
  • isIndexable - 是否索引
  • isUnique - 是否唯一
  • multiplicity - 指示此属性是(必需的/可选的/还是可以是多值)的

System specific types and their significance

Referenceable

This type represents all entities that can be searched for using a unique attribute called qualifiedName.

  ├── Referenceable
├── ReferenceableDeserializer
├── ReferenceableSerializer
└── V1SearchReferenceableSerializer

Hooks

以Hive元信息采集为例分析采集过程:

全量导入

import-hive.sh

"${JAVA_BIN}" ${JAVA_PROPERTIES} -cp "${CP}"
org.apache.atlas.hive.bridge.HiveMetaStoreBridge $IMPORT_ARGS
importTables
└── importDatabases [addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java +295]
└── importHiveMetadata [addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java +289]

上面是调用过程:

importTables -> importTable --> registerInstances

AtlasEntitiesWithExtInfo ret = null;
EntityMutationResponse response = atlasClientV2.createEntities(entities);
List<AtlasEntityHeader> createdEntities = response.getEntitiesByOperation(EntityMutations.EntityOperation.CREATE); if (CollectionUtils.isNotEmpty(createdEntities)) {
ret = new AtlasEntitiesWithExtInfo(); for (AtlasEntityHeader createdEntity : createdEntities) {
AtlasEntityWithExtInfo entity = atlasClientV2.getEntityByGuid(createdEntity.getGuid()); ret.addEntity(entity.getEntity()); if (MapUtils.isNotEmpty(entity.getReferredEntities())) {
for (Map.Entry<String, AtlasEntity> entry : entity.getReferredEntities().entrySet()) {
ret.addReferredEntity(entry.getKey(), entry.getValue());
}
} LOG.info("Created {} entity: name={}, guid={}", entity.getEntity().getTypeName(), entity.getEntity().getAttribute(ATTRIBUTE_QUALIFIED_NAME), entity.getEntity().getGuid());
}
}

通过Http Post 的请求将库表数据更新至Atlas

atlasClientV2有很多Http接口

Atlas HTTP 客户端API:

实时监听

HiveHook implements ExecuteWithHookContext

ExecuteWithHookContext is a new interface that the Pre/Post Execute Hook can run with the HookContext.

实现run()方法来对Hive 相关事件做处理

Hive相关事件:

BaseHiveEvent
├── AlterTableRename
├── CreateHiveProcess
├── DropDatabase
├── DropTable
├── CreateDatabase
│ └── AlterDatabase
└── CreateTable
└── AlterTable
└── AlterTableRenameCol

以create database 为例分析流程:

//处理Hook 上下文信息
AtlasHiveHookContext context =
new AtlasHiveHookContext(this, oper, hookContext, getKnownObjects(), isSkipTempTables()); //建库事件处理,提取相关库信息
event = new CreateDatabase(context); if (event != null) {
final UserGroupInformation ugi = hookContext.getUgi() == null ? Utils.getUGI() : hookContext.getUgi();
super.notifyEntities(ActiveEntityFilter.apply(event.getNotificationMessages()), ugi);
} public enum HookNotificationType {
TYPE_CREATE, TYPE_UPDATE, ENTITY_CREATE, ENTITY_PARTIAL_UPDATE, ENTITY_FULL_UPDATE, ENTITY_DELETE,
ENTITY_CREATE_V2, ENTITY_PARTIAL_UPDATE_V2, ENTITY_FULL_UPDATE_V2, ENTITY_DELETE_V2
} //操作用户获取
if (context.isMetastoreHook()) {
try {
ugi = SecurityUtils.getUGI();
} catch (Exception e) {
//do nothing
}
} else {
ret = getHiveUserName(); if (StringUtils.isEmpty(ret)) {
ugi = getUgi();
}
} if (ugi != null) {
ret = ugi.getShortUserName();
} if (StringUtils.isEmpty(ret)) {
try {
ret = UserGroupInformation.getCurrentUser().getShortUserName();
} catch (IOException e) {
LOG.warn("Failed for UserGroupInformation.getCurrentUser() ", e); ret = System.getProperty("user.name");
}
}

主要:

获取实体信息, 传递Hook message的类型、操作用户

notifyEntities 可以看出其他组件HBase、impala也会调用该方法进行消息的发送

public static void notifyEntities(List<HookNotification> messages, UserGroupInformation ugi, int maxRetries) {
if (executor == null) { // send synchronously
notifyEntitiesInternal(messages, maxRetries, ugi, notificationInterface, logFailedMessages, failedMessagesLogger);
} else {
executor.submit(new Runnable() {
@Override
public void run() {
notifyEntitiesInternal(messages, maxRetries, ugi, notificationInterface, logFailedMessages, failedMessagesLogger);
}
});
}
}

消息通知框架:

NotificationInterface
├── AtlasFileSpool
└── AbstractNotification
├── KafkaNotification
└── Spooler

数据写入Kafka中:

@Override
public void sendInternal(NotificationType notificationType, List<String> messages) throws NotificationException {
KafkaProducer producer = getOrCreateProducer(notificationType); sendInternalToProducer(producer, notificationType, messages);
}

根据NotificationType写入指定topic 中:

private static final Map<NotificationType, String> PRODUCER_TOPIC_MAP = new HashMap<NotificationType, String>() {
{
put(NotificationType.HOOK, ATLAS_HOOK_TOPIC);
put(NotificationType.ENTITIES, ATLAS_ENTITIES_TOPIC);
}
}; NOTIFICATION_HOOK_TOPIC_NAME("atlas.notification.hook.topic.name", "ATLAS_HOOK"),
NOTIFICATION_ENTITIES_TOPIC_NAME("atlas.notification.entities.topic.name", "ATLAS_ENTITIES"),

数据主要写入两个Topic中: ATLAS_ENTITIES、ATLAS_HOOK

ATLAS_HOOK是写入Hook事件消息, 创建库的事件元数据信息会写入该Topic中

如何唯一确定一个库:

public String getQualifiedName(Database db) {
return getDatabaseName(db) + QNAME_SEP_METADATA_NAMESPACE + getMetadataNamespace();
}

dbName@clusterName 确定唯一性

外延应用

一个基于Hive hook 实现Impala 元数据刷新的用例:

AutoRefreshImpala:https://github.com/Observe-secretly/AutoRefreshImpala

参考

[1] Apache Atlas – Data Governance and Metadata framework for Hadoop

[2] Apache Atlas 源码

[Apache Atlas] Atlas 架构设计及源代码简单分析的更多相关文章

  1. FFmpeg的HEVC解码器源代码简单分析:环路滤波(Loop Filter)

    ===================================================== HEVC源代码分析文章列表: [解码 -libavcodec HEVC 解码器] FFmpe ...

  2. FFmpeg的HEVC解码器源代码简单分析:CTU解码(CTU Decode)部分-TU

    ===================================================== HEVC源代码分析文章列表: [解码 -libavcodec HEVC 解码器] FFmpe ...

  3. FFmpeg的HEVC解码器源代码简单分析:CTU解码(CTU Decode)部分-PU

    ===================================================== HEVC源代码分析文章列表: [解码 -libavcodec HEVC 解码器] FFmpe ...

  4. FFmpeg源代码简单分析:libavdevice的gdigrab

    ===================================================== FFmpeg的库函数源代码分析文章列表: [架构图] FFmpeg源代码结构图 - 解码 F ...

  5. FFmpeg源代码简单分析:libavdevice的avdevice_register_all()

    ===================================================== FFmpeg的库函数源代码分析文章列表: [架构图] FFmpeg源代码结构图 - 解码 F ...

  6. FFmpeg源代码简单分析:configure

    ===================================================== FFmpeg的库函数源代码分析文章列表: [架构图] FFmpeg源代码结构图 - 解码 F ...

  7. FFmpeg源代码简单分析:makefile

    ===================================================== FFmpeg的库函数源代码分析文章列表: [架构图] FFmpeg源代码结构图 - 解码 F ...

  8. FFmpeg源代码简单分析:libswscale的sws_scale()

    ===================================================== FFmpeg的库函数源代码分析文章列表: [架构图] FFmpeg源代码结构图 - 解码 F ...

  9. FFmpeg源代码简单分析:libswscale的sws_getContext()

    ===================================================== FFmpeg的库函数源代码分析文章列表: [架构图] FFmpeg源代码结构图 - 解码 F ...

随机推荐

  1. 字节跳动Android面试凉凉,挥泪整理面筋,你不看看吗?

    想在金九银十找工作的现在可以开始准备了,这边给大家分享一下面试会遇到的问题. 找工作还是需要大家不要担心,由于我们干这一行的接触人本来就不多,难免看到面试官会紧张,主要是因为怕面试官问的答不上来,答不 ...

  2. 走心的中级Android工程师跳槽经验分享

    这些经验是我最近四个月,从准备面试到找到合适工作的汗水和泪水,希望对你们能有帮助! define 跳槽 跳槽前要思考的问题 钱不到位怎么办 心委屈怎么办 离职前的思考 确定要走时需要做的准备 行情怎么 ...

  3. 实战爬取Plati官网游戏实时最低价格-Python

    需要修改url中的id_r="这个",这个id需要从Battlefield V (plati.ru)中获取,其实也是这个链接中的#s24235. 配合了e-mail推送,其实这个e ...

  4. Linux之cat tail less常见用法

    1.cat 通常查找出错误日志 cat error.log | grep 'foo' , 这时候我们还有个需求就是输出当前这个日志的前后几行: cat error.log | grep -C 10 ' ...

  5. MySQL5.6忘记root密码

    第一步 修改 /etc/my.conf 文件 添加 skip-grant-tables 到 [mysqld] 下面就可以 这个参数的意思是设置为无需密码验证的登录 登录之后,可以添加用户,可以修改密码 ...

  6. 【笔记】F1 score

    F1 score 关于精准率和召回率 精准率和召回率可以很好的评价对于数据极度偏斜的二分类问题的算法,有个问题,毕竟是两个指标,有的时候这两个指标也会产生差异,对于不同的算法,精准率可能高一些,召回率 ...

  7. mongodb(docker-compose)

    version: '3.1' services: mongo: image: mongo restart: always environment: MONGO_INITDB_ROOT_USERNAME ...

  8. SpringBoot监听redis订阅监听和发布订阅

    前言 我们可以在redis中发布一条订阅到通道中,所有监听了这个通道的都可以收到这个发布的内容! redis订阅监听配置类 代码如下: RedisListenerConfig.java package ...

  9. com 组件的本知识

    (今日看到网络上关于"COM中GUID......"文章,写的好,故记录之.)当初微软设计com规范的时候,有两种选择来保证用户的设计的com组件可以全球唯一:第一种是采用和Int ...

  10. ANSI C说明了三个用于存储空间动态分配的函数

    1.1 malloc的全称是memory allocation,中文叫动态内存分配.原型:extern void *malloc(unsigned int num_bytes);说明:分配长度为num ...