CMDB资产管理系统开发【day26】:实现资产自动更新

1、需求分析

1、比对分析

比对的时候以那个数据源为主？

old [1,2,3 ] db数据库

new [2,3,4 ] 客户端汇报过来的

当然以客户端汇报过来的数据为主

2、更新分析

不同的表到底拿那个字段当做唯一值？

old nic {
	n1 10.0.0.3 --> 192.168.3.22  #源数据里要拿那个字段去找
	n2 172.168.10.2
	n3 192.165.52.33
}

1. sn：网卡没有sn ip一个客户端三个ip 都不靠谱，只有mac是唯一的地址，
2. 拿新ip对应的mac地址去到网卡表里搜。拿出来之后，对比ip地址、name
3. 如果改变了就更新，不同的表到底拿谁当做唯一值，所以要声明

3、伪代码：

def compile_component(table_name，
						compare_fields=[ip，mac，netmask]
						unique=mac，
						client_date=nic_data)
	for i in compare_fields:
		db_field = getattr(table_name,i)
		if db_field==nic_data[i]:
		else:
		table_name.db_field = nic_data[i]

1、很多字段，你咋知道那个变了？

• 任何一个字段变了都要记录下来，改完了之后还要记录下来

2、有什么好的办法没？

1. 没有什么好办法，你就必须每一个方法都一一对比，你可有N个字段，并且分布在不同的表
2. 比对的时候以那个数据源为主

2、update_asset函数解析

    def update_asset(self):
        func = getattr(self, '_update_%s' % self.clean_data['asset_type'])
        create_obj = func()
	#每一个表里面的数据都不一样,服务器端的字段是自己创建的，客户端没有，所以只能用客户端的比

3、_update_server函数解析

    def _update_server(self):
        nic = self.__update_asset_component(data_source=self.clean_data['nic'],
                                            fk='nic_set',
	'''
	fk是什么？
	网卡关联asset 反向查找如何查找 a.nic_set;nic_set.select_related()网卡就出来了看到几个网卡，三个网卡
	此处有截图
	为什么要把'nic_set'传进去？
	我通过asset反查我的网卡，内存，name就等于告诉去找那些表，
	'''
                                            update_fields=['name', 'sn', 'model', 'macaddress', 'ipaddress', 'netmask',
                                                           'bonding'],  #要查找的字段
                                            identify_field='macaddress' #唯一值
                                            )
        disk = self.__update_asset_component(data_source=self.clean_data['physical_disk_driver'],
                                             fk='disk_set',
                                             update_fields=['slot', 'sn', 'model', 'manufactory', 'capacity',
                                                            'iface_type'],
                                             identify_field='slot'
                                             )
        ram = self.__update_asset_component(data_source=self.clean_data['ram'],
                                            fk='ram_set',
                                            update_fields=['slot', 'sn', 'model', 'capacity'],
                                            identify_field='slot'
                                            )
        cpu = self.__update_cpu_component()
        manufactory = self.__update_manufactory_component()

        server = self.__update_server_component()

fk是什么？

• 网卡关联asset 反向查找如何查找 a.nic_set;nic_set.select_related()网卡就出来了看到几个网卡，三个网卡

• 为什么要把'nic_set'传进去？
• 我通过asset反查我的网卡，内存，name就等于告诉去找那些表

4、__update_asset_component函数解析

    def __update_asset_component(self, data_source, fk, update_fields, identify_field=None):
        '''
        data_source: the data source of this component from reporting data
        fk: which key to use to find the connection between main Asset obj and each asset component
        update_fields: what fields in DB will be compared and updated
        identify_field: use this field to identify each component of an Asset , if set to None,means only use asset id to identify
         '''
        print(data_source, update_fields, identify_field)
        try:
            component_obj = getattr(self.asset_obj, fk)
			#传进来的是字符串，要通过字符串反射成对象
            if hasattr(component_obj, 'select_related'):  # this component is reverse m2m relation with Asset model
                objects_from_db = component_obj.select_related() #把数据取出来，循环数据库里面的数据
                for obj in objects_from_db:
			#n1 10.0.0.3 --> 192.168.3.22  源数据里要拿那个字段去找
			#b.macaddress；getttr(b.'macaddress')； 8C-EC-4B-16-F9-9B
                    key_field_data = getattr(obj, identify_field)
                    # use this key_field_data to find the relative data source from reporting data
                    if type(data_source) is list:
                        for source_data_item in data_source:
			#到客户端的数据 源里去找到跟服务器数据库中key_field_data对应的条目
                            key_field_data_from_source_data = source_data_item.get(identify_field)
                            if key_field_data_from_source_data:
                                if key_field_data == key_field_data_from_source_data:  # find the matched source data for this component,then should compare each field in this component to see if there's any changes since last update
                                    self.__compare_componet(model_obj=obj, fields_from_db=update_fields,
                                                            data_source=source_data_item)
			'''
			拿着8C-EC-4B-16-F9-9B去客户端里找，如果找到就是一样的，循环是的是三个，
			拿着db的网卡，去客户端里循环，如果找不到要循环三次，
				第一没获取，不是这块网卡，
				循环第二次找到了，进行对比，就是这块网卡
				如果第二次循环找到，就不要往下摘了，如果我循环了三次还找不到就没,删除了，
			'''

                                    break #已经根据identify_field找到客户端中对应的数据条目，且对比完了，后面的loop没必要继续了
                            else:  # key field data from source data cannot be none
                                self.response_msg('warning', 'AssetUpdateWarning',
                                                  "Asset component [%s]'s key field [%s] is not provided in reporting data " % (
                                                      fk, identify_field))

                        else:  # couldn't find any matches, the asset component must be broken or changed manually
                            print(
                                '\033[33;1mError:cannot find any matches in source data by using key field val [%s],component data is missing in reporting data!\033[0m' % (
                                    key_field_data))
                            self.response_msg("error", "AssetUpdateWarning",
                                              "Cannot find any matches in source data by using key field val [%s],component data is missing in reporting data!" % (
                                                  key_field_data))

5、传进来的是字符串，要通过字符串反射成对象

6、删除一张网卡验证一下上面的函数

后台页面

报错代码如下比对一下

self.response_msg("error", "AssetUpdateWarning",
                                              "Cannot find any matches in source data by using key field val [%s],component data is missing in reporting data!" % (
                                                  key_field_data))

服务器端控制台（整个比对过程很清晰了）

1. 拿着db的网卡，去客户端里循环，如果找不到要循环三次，
2. 第一没获取，不是这块网卡，
3. 循环第二次找到了，进行对比，就是这块网卡
4. 如果第二次循环找到，就不要往下摘了，如果我循环了三次还找不到就没,删除了，

7、__compare_component函数刨析

源代码

    def __compare_component(self, model_obj, fields_from_db, data_source):
        print('---going to compare:[%s]' % model_obj, fields_from_db)
        print('---source data:', data_source)
        for field in fields_from_db:
            val_from_db = getattr(model_obj, field) 	#获取到客户端的数据
            val_from_data_source = data_source.get(field) #获取到数据库里的数据
            if val_from_data_source:
                # if type(val_from_db) is unicode:val_from_data_source = unicode(val_from_data_source)#no unicode in py3
                # if type(val_from_db) in (int,long):val_from_data_source = int(val_from_data_source) #no long in py3
                if type(val_from_db) in (int,):
                    val_from_data_source = int(val_from_data_source)
				#如果数据库里的数据类型是int，就会把客户端的数据转换过成int
                elif type(val_from_db) is float:
                    val_from_data_source = float(val_from_data_source)
                elif type(val_from_db) is str:
                    val_from_data_source = str(val_from_data_source).strip()
                if val_from_db == val_from_data_source:  # this field haven't changed since last update
                    pass
                    # print '\033[32;1m val_from_db[%s]  == val_from_data_source[%s]\033[0m' %(val_from_db,val_from_data_source)
                else:
                    print('\033[34;1m val_from_db[%s]  != val_from_data_source[%s]\033[0m' % (
                        val_from_db, val_from_data_source), type(val_from_db), type(val_from_data_source), field)
                    db_field = model_obj._meta.get_field(field)
                    db_field.save_form_data(model_obj, val_from_data_source)
                    model_obj.update_date = timezone.now()
                    model_obj.save()
                    log_msg = "Asset[%s] --> component[%s] --> field[%s] has changed from [%s] to [%s]" % (
                        self.asset_obj, model_obj, field, val_from_db, val_from_data_source)#记录日志
                    self.response_msg('info', 'FieldChanged', log_msg) #返回给客户端的日志
                    log_handler(self.asset_obj, 'FieldChanged', self.request.user, log_msg, model_obj) #真正要记录的日志
            else:
                self.response_msg('warning', 'AssetUpdateWarning',
                                  "Asset component [%s]'s field [%s] is not provided in reporting data " % (
                                      model_obj, field))

        model_obj.save()

cpu数据类型设计

虽然是int，但是是字符串形式的，我为了确保不出问题，数据格式按照数据库里的字段来

获取到数据库里的数据 就会把客户端的数据转换成int

 if val_from_data_source:
                # if type(val_from_db) is unicode:val_from_data_source = unicode(val_from_data_source)#no unicode in py3
                # if type(val_from_db) in (int,long):val_from_data_source = int(val_from_data_source) #no long in py3
                if type(val_from_db) in (int,):
                    val_from_data_source = int(val_from_data_source)
				#如果数据库里的数据类型是int，就会把客户端的数据转换过成int

之前的写法：

if type(val_from_db) is unicode:val_from_data_source = unicode(val_from_data_source)#no unicode in py3
if type(val_from_db) in (int,long):val_from_data_source = int(val_from_data_source) #no long in py3

记录日志

日志文件里的日志

8、如何赋一个新值

admin.py后台配置代码

admin.site.register(models.Asset,AssetAdmin)
admin.site.register(models.Server)
admin.site.register(models.NetworkDevice)
admin.site.register(models.IDC)
admin.site.register(models.BusinessUnit)
admin.site.register(models.Contract)
admin.site.register(models.CPU)
admin.site.register(models.Disk)
admin.site.register(models.NIC,NicAdmin)
admin.site.register(models.RAM)
admin.site.register(models.Manufactory)
admin.site.register(models.Tag)
admin.site.register(models.Software)
admin.site.register(models.EventLog,EventLogAdmin)
admin.site.register(models.NewAssetApprovalZone,NewAssetApprovalZoneAdmin)

最简单的方法（同学们的智慧）

　　

　　

alex的方法

代码

    db_field = model_obj._meta.get_field(field)
    db_field.save_form_data(model_obj, val_from_data_source)

客户端截图和后台截图

　　

9、__filter_add_or_deleted_components函数刨析

源代码

    def __filter_add_or_deleted_components(self, model_obj_name, data_from_db, data_source, identify_field):
        '''This function is filter out all  component data in db but missing in reporting data, and all the data in reporting data but not in DB'''
        print(data_from_db, data_source, identify_field)
        data_source_key_list = []  # save all the identified keys from client data,e.g: [macaddress1,macaddress2]
        if type(data_source) is list:
            for data in data_source:
                data_source_key_list.append(data.get(identify_field))

        elif type(data_source) is dict: #dprecated
            for key, data in data_source.items():
                if data.get(identify_field):
                    data_source_key_list.append(data.get(identify_field))
                else:  # workround for some component uses key as identified field e.g: ram
                    data_source_key_list.append(key)
        print('-->identify field [%s] from db  :', data_source_key_list)  #数据库里的数据
        print('-->identify[%s] from data source:', [getattr(obj, identify_field) for obj in data_from_db])

        data_source_key_list = set(data_source_key_list) #列表生成式
        data_identify_val_from_db = set([getattr(obj, identify_field) for obj in data_from_db])
        data_only_in_db = data_identify_val_from_db - data_source_key_list  # delete all this from db 只在数据库存在的数据
        data_only_in_data_source = data_source_key_list - data_identify_val_from_db  # add into db   #只在客户端里的数据
        print('\033[31;1mdata_only_in_db:\033[0m', data_only_in_db)
        print('\033[31;1mdata_only_in_data source:\033[0m', data_only_in_data_source)
        self.__delete_components(all_components=data_from_db, delete_list=data_only_in_db, #服务器端删除
                                 identify_field=identify_field) #以什么作为唯一值
        if data_only_in_data_source:
            self.__add_components(model_obj_name=model_obj_name, all_components=data_source,#服务器添加
                                  add_list=data_only_in_data_source, identify_field=identify_field)

10、服务器端后台改IP地址 无论怎么改，它永远和客户端一直

第一步：后台更改两次皮地址如下图

第一更改：

第二次更改

11、__delete_components函数刨析

源代码：

    def __delete_components(self, all_components, delete_list, identify_field):
        '''All the objects in delete list will be deleted from DB'''
        deleting_obj_list = []
        print('--deleting components', delete_list, identify_field)
        for obj in all_components:
            val = getattr(obj, identify_field)
            if val in delete_list:
                deleting_obj_list.append(obj)

        for i in deleting_obj_list:
            log_msg = "Asset[%s] --> component[%s] --> is lacking from reporting source data, assume it has been removed or replaced,will also delete it from DB" % (
                self.asset_obj, i)
            self.response_msg('info', 'HardwareChanges', log_msg)
            log_handler(self.asset_obj, 'HardwareChanges', self.request.user, log_msg, i)
            i.delete()

12、__add_components函数刨析

源代码

   def __add_components(self, model_obj_name, all_components, add_list, identify_field):
        model_class = getattr(models, model_obj_name)
        will_be_creating_list = []
        print('--add component list:', add_list)
        if type(all_components) is list:
            for data in all_components: #客户端的源数据
                if data[identify_field] in add_list:
                    # print data
                    will_be_creating_list.append(data)
        elif type(all_components) is dict: #deprecated
            for k, data in all_components.items():
                # workround for some components uses key as identified field ,e.g ram
                if data.get(identify_field):
                    if data[identify_field] in add_list:
                        # print k,data
                        will_be_creating_list.append(data)
                else:  # if the identified field cannot be found from data set,then try to compare the dict key
                    if k in add_list:
                        data[
                            identify_field] = k  # add this key into dict , because this dict will be used to create new component item in DB
                        will_be_creating_list.append(data)

        # creating components
        try:
            for component in will_be_creating_list:
                data_set = {}
                for field in model_class.auto_create_fields:
                    data_set[field] = component.get(field)
                data_set['asset_id'] = self.asset_obj.id
                obj = model_class(**data_set)
                obj.save()
                print('\033[32;1mCreated component with data:\033[0m', data_set)
                log_msg = "Asset[%s] --> component[%s] has justed added a new item [%s]" % (
                    self.asset_obj, model_obj_name, data_set)
                self.response_msg('info', 'NewComponentAdded', log_msg)
                log_handler(self.asset_obj, 'NewComponentAdded', self.request.user, log_msg, model_obj_name)

        except Exception as e:
            print("\033[31;1m %s \033[0m" % e)
            log_msg = "Asset[%s] --> component[%s] has error: %s" % (self.asset_obj, model_obj_name, str(e))
            self.response_msg('error', "AddingComponentException", log_msg)	

添加一张网卡

客户端截图

13、__filter_add_or_deleted_components函数刨析

源代码：

    def __filter_add_or_deleted_components(self, model_obj_name, data_from_db, data_source, identify_field):
        '''This function is filter out all  component data in db but missing in reporting data, and all the data in reporting data but not in DB'''
        print(data_from_db, data_source, identify_field)
        data_source_key_list = []  # save all the identified keys from client data,e.g: [macaddress1,macaddress2]
        if type(data_source) is list:
            for data in data_source:
                data_source_key_list.append(data.get(identify_field))

        elif type(data_source) is dict: #dprecated
            for key, data in data_source.items():
                if data.get(identify_field):
                    data_source_key_list.append(data.get(identify_field))
                else:  # workround for some component uses key as identified field e.g: ram
                    data_source_key_list.append(key)
        print('-->identify field [%s] from db  :', data_source_key_list)  #数据库里的数据
        print('-->identify[%s] from data source:', [getattr(obj, identify_field) for obj in data_from_db])

        data_source_key_list = set(data_source_key_list) #列表生成式
        data_identify_val_from_db = set([getattr(obj, identify_field) for obj in data_from_db])
        data_only_in_db = data_identify_val_from_db - data_source_key_list  # delete all this from db 只在数据库存在的数据
        data_only_in_data_source = data_source_key_list - data_identify_val_from_db  # add into db   #只在客户端里的数据
        print('\033[31;1mdata_only_in_db:\033[0m', data_only_in_db)
        print('\033[31;1mdata_only_in_data source:\033[0m', data_only_in_data_source)
        self.__delete_components(all_components=data_from_db, delete_list=data_only_in_db, #服务器端删除
                                 identify_field=identify_field) #以什么作为唯一值
        if data_only_in_data_source:
            self.__add_components(model_obj_name=model_obj_name, all_components=data_source,#服务器添加
                                  add_list=data_only_in_data_source, identify_field=identify_field)	

以数据库为主去找的思路，但是客户端加一块，服务器端就不知道

是因为你那着数据库里的两个网卡，去客户端里去匹配

总结：上面的方法肯定解决不了问题，就要想别的方法处理

数据库和客户端交集并集比对

创建资产组件以什么字段为准

1、不能以客户端为主，因为客户端，要是被篡改了，传过来500个字段

2、自动创建字段要提前定义好，代码看表结构

  硬盘：auto_create_fields = ['sn', 'slot', 'model', 'capacity']
  网卡：auto_create_fields = ['name', 'sn', 'model', 'macaddress', 'ipaddress', 'netmask', 'bonding']
  内存：auto_create_fields = ['sn', 'slot', 'manufactory', 'model', 'capacity', 'iface_type']

14、任意改内存 网卡硬盘数据（对自动更新的整体测试）

服务端给客户端返回资产id，但是客户端解析不出来，所以一直往待审批里面汇报

成功客户端截图如下

　　

修改成功后台截图