Fix multiple GPUs fails in training Mask_RCNN
Test with:
Keras: 2.2.4
Python: 3.6.9
Tensorflow: 1.12.0
==================
Problem:
Using code from https://github.com/matterport/Mask_RCNN
When setting GPU_COUNT > 1
enconter this error:
RuntimeError: It looks like you are subclassing `Model` and you forgot to call `super(YourClass, self).__init__()`. Always start with this line.
Traceback (most recent call last):
File "D:\Anaconda33\lib\site-packages\keras\engine\network.py", line 313, in __setattr__
is_graph_network = self._is_graph_network
File "parallel_model.py", line 46, in __getattribute__
return super(ParallelModel, self).__getattribute__(attrname)
AttributeError: 'ParallelModel' object has no attribute '_is_graph_network' During handling of the above exception, another exception occurred: Traceback (most recent call last):
File "parallel_model.py", line 159, in <module>
model = ParallelModel(model, GPU_COUNT)
File "parallel_model.py", line 35, in __init__
self.inner_model = keras_model
File "D:\Anaconda33\lib\site-packages\keras\engine\network.py", line 316, in __setattr__
'It looks like you are subclassing `Model` and you '
RuntimeError: It looks like you are subclassing `Model` and you forgot to call `super(YourClass, self).__init__()`. Always start with this line.
Solution 1:
changing code in mrcnn/parallel_model.py as the following:
class ParallelModel(KM.Model):
def __init__(self, keras_model, gpu_count):
"""Class constructor.
keras_model: The Keras model to parallelize
gpu_count: Number of GPUs. Must be > 1
"""
super(ParallelModel, self).__init__()
self.inner_model = keras_model
self.gpu_count = gpu_count
merged_outputs = self.make_parallel()
super(ParallelModel, self).__init__(inputs=self.inner_model.inputs,
outputs=merged_outputs)
When getting this error:
asking for two arguments: inputs and outputs
Just upgrade your Keras to 2.2.4
When getting this error:
No node-device colocations were active during op 'tower_1/mask_rcnn/anchors/Variable/anchors/Variable/read_tower_1/mask_rcnn/anchors/Variable_0' creation.
Device assignments active during op 'tower_1/mask_rcnn/anchors/Variable/anchors/Variable/read_tower_1/mask_rcnn/anchors/Variable_0' creation:
with tf.device(/gpu:1): <M:\new\mrcnn\parallel_model.py:70>
No node-device colocations were active during op 'anchors/Variable' creation.
No device assignments were active during op 'anchors/Variable' creation.
Traceback (most recent call last):
File "D:\Anaconda33\lib\site-packages\tensorflow\python\client\session.py", line 1334, in _do_call
return fn(*args)
File "D:\Anaconda33\lib\site-packages\tensorflow\python\client\session.py", line 1317, in _run_fn
self._extend_graph()
File "D:\Anaconda33\lib\site-packages\tensorflow\python\client\session.py", line 1352, in _extend_graph
tf_session.ExtendSession(self._session)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot colocate nodes {{colocation_node tower_1/mask_rcnn/anchors/Variable/anchors/Variable/read_tower_1/mask_rcnn/anchors/Variable_0}} and {{colocation_node anchors/Variable}}: Cannot merge devices with incompatible ids: '/device:GPU:0' and '/device:GPU:1'
[[{{node tower_1/mask_rcnn/anchors/Variable/anchors/Variable/read_tower_1/mask_rcnn/anchors/Variable_0}} = Identity[T=DT_FLOAT, _class=["loc:@anchors/Variable"], _device="/device:GPU:1"](tower_1/mask_rcnn/anchors/Variable/cond/Merge)]] During handling of the above exception, another exception occurred: Traceback (most recent call last):
File "train_mul.py", line 448, in <module>
"mrcnn_bbox", "mrcnn_mask"])
File "M:\new\mrcnn\model.py", line 2132, in load_weights
saving.load_weights_from_hdf5_group_by_name(f, layers)
File "D:\Anaconda33\lib\site-packages\keras\engine\saving.py", line 1022, in load_weights_from_hdf5_group_by_name
K.batch_set_value(weight_value_tuples)
File "D:\Anaconda33\lib\site-packages\keras\backend\tensorflow_backend.py", line 2440, in batch_set_value
get_session().run(assign_ops, feed_dict=feed_dict)
File "D:\Anaconda33\lib\site-packages\keras\backend\tensorflow_backend.py", line 197, in get_session
[tf.is_variable_initialized(v) for v in candidate_vars])
File "D:\Anaconda33\lib\site-packages\tensorflow\python\client\session.py", line 929, in run
run_metadata_ptr)
File "D:\Anaconda33\lib\site-packages\tensorflow\python\client\session.py", line 1152, in _run
feed_dict_tensor, options, run_metadata)
File "D:\Anaconda33\lib\site-packages\tensorflow\python\client\session.py", line 1328, in _do_run
run_metadata)
File "D:\Anaconda33\lib\site-packages\tensorflow\python\client\session.py", line 1348, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot colocate nodes node tower_1/mask_rcnn/anchors/Variable/anchors/Variable/read_tower_1/mask_rcnn/anchors/Variable_0 (defined at M:\new\mrcnn\model.py:1936) having device Device assignments active during op 'tower_1/mask_rcnn/anchors/Variable/anchors/Variable/read_tower_1/mask_rcnn/anchors/Variable_0' creation:
with tf.device(/gpu:1): <M:\new\mrcnn\parallel_model.py:70> and node anchors/Variable (defined at M:\new\mrcnn\model.py:1936) having device No device assignments were active during op 'anchors/Variable' creation. : Cannot merge devices with incompatible ids: '/device:GPU:0' and '/device:GPU:1'
[[node tower_1/mask_rcnn/anchors/Variable/anchors/Variable/read_tower_1/mask_rcnn/anchors/Variable_0 (defined at M:\new\mrcnn\model.py:1936) = Identity[T=DT_FLOAT, _class=["loc:@anchors/Variable"], _device="/device:GPU:1"](tower_1/mask_rcnn/anchors/Variable/cond/Merge)]] No node-device colocations were active during op 'tower_1/mask_rcnn/anchors/Variable/anchors/Variable/read_tower_1/mask_rcnn/anchors/Variable_0' creation.
Device assignments active during op 'tower_1/mask_rcnn/anchors/Variable/anchors/Variable/read_tower_1/mask_rcnn/anchors/Variable_0' creation:
with tf.device(/gpu:1): <M:\new\mrcnn\parallel_model.py:70> No node-device colocations were active during op 'anchors/Variable' creation.
No device assignments were active during op 'anchors/Variable' creation. Caused by op 'tower_1/mask_rcnn/anchors/Variable/anchors/Variable/read_tower_1/mask_rcnn/anchors/Variable_0', defined at:
File "train_mul.py", line 417, in <module>
model_dir=MODEL_DIR)
File "M:\new\mrcnn\model.py", line 1839, in __init__
self.keras_model = self.build(mode=mode, config=config)
File "M:\new\mrcnn\model.py", line 2064, in build
model = ParallelModel(model, config.GPU_COUNT)
File "M:\new\mrcnn\parallel_model.py", line 36, in __init__
merged_outputs = self.make_parallel()
File "M:\new\mrcnn\parallel_model.py", line 80, in make_parallel
outputs = self.inner_model(inputs)
File "D:\Anaconda33\lib\site-packages\keras\engine\base_layer.py", line 457, in __call__
output = self.call(inputs, **kwargs)
File "D:\Anaconda33\lib\site-packages\keras\engine\network.py", line 570, in call
output_tensors, _, _ = self.run_internal_graph(inputs, masks)
File "D:\Anaconda33\lib\site-packages\keras\engine\network.py", line 724, in run_internal_graph
output_tensors = to_list(layer.call(computed_tensor, **kwargs))
File "D:\Anaconda33\lib\site-packages\keras\layers\core.py", line 682, in call
return self.function(inputs, **arguments)
File "M:\new\mrcnn\model.py", line 1936, in <lambda>
anchors = KL.Lambda(lambda x: tf.Variable(anchors), name="anchors")(input_image)
File "D:\Anaconda33\lib\site-packages\tensorflow\python\ops\variables.py", line 183, in __call__
return cls._variable_v1_call(*args, **kwargs)
File "D:\Anaconda33\lib\site-packages\tensorflow\python\ops\variables.py", line 146, in _variable_v1_call
aggregation=aggregation)
File "D:\Anaconda33\lib\site-packages\tensorflow\python\ops\variables.py", line 125, in <lambda>
previous_getter = lambda **kwargs: default_variable_creator(None, **kwargs)
File "D:\Anaconda33\lib\site-packages\tensorflow\python\ops\variable_scope.py", line 2444, in default_variable_creator
expected_shape=expected_shape, import_scope=import_scope)
File "D:\Anaconda33\lib\site-packages\tensorflow\python\ops\variables.py", line 187, in __call__
return super(VariableMetaclass, cls).__call__(*args, **kwargs)
File "D:\Anaconda33\lib\site-packages\tensorflow\python\ops\variables.py", line 1329, in __init__
constraint=constraint)
File "D:\Anaconda33\lib\site-packages\tensorflow\python\ops\variables.py", line 1480, in _init_from_args
self._initial_value),
File "D:\Anaconda33\lib\site-packages\tensorflow\python\ops\variables.py", line 2177, in _try_guard_against_uninitialized_dependencies
return self._safe_initial_value_from_tensor(initial_value, op_cache={})
File "D:\Anaconda33\lib\site-packages\tensorflow\python\ops\variables.py", line 2195, in _safe_initial_value_from_tensor
new_op = self._safe_initial_value_from_op(op, op_cache)
File "D:\Anaconda33\lib\site-packages\tensorflow\python\ops\variables.py", line 2241, in _safe_initial_value_from_op
name=new_op_name, attrs=op.node_def.attr)
File "D:\Anaconda33\lib\site-packages\tensorflow\python\util\deprecation.py", line 488, in new_func
return func(*args, **kwargs)
File "D:\Anaconda33\lib\site-packages\tensorflow\python\framework\ops.py", line 3274, in create_op
op_def=op_def)
File "D:\Anaconda33\lib\site-packages\tensorflow\python\framework\ops.py", line 1770, in __init__
self._traceback = tf_stack.extract_stack() InvalidArgumentError (see above for traceback): Cannot colocate nodes node tower_1/mask_rcnn/anchors/Variable/anchors/Variable/read_tower_1/mask_rcnn/anchors/Variable_0 (defined at M:\new\mrcnn\model.py:1936) having device Device assignments active during op 'tower_1/mask_rcnn/anchors/Variable/anchors/Variable/read_tower_1/mask_rcnn/anchors/Variable_0' creation:
with tf.device(/gpu:1): <M:\new\mrcnn\parallel_model.py:70> and node anchors/Variable (defined at M:\new\mrcnn\model.py:1936) having device No device assignments were active during op 'anchors/Variable' creation. : Cannot merge devices with incompatible ids: '/device:GPU:0' and '/device:GPU:1'
[[node tower_1/mask_rcnn/anchors/Variable/anchors/Variable/read_tower_1/mask_rcnn/anchors/Variable_0 (defined at M:\new\mrcnn\model.py:1936) = Identity[T=DT_FLOAT, _class=["loc:@anchors/Variable"], _device="/device:GPU:1"](tower_1/mask_rcnn/anchors/Variable/cond/Merge)]] No node-device colocations were active during op 'tower_1/mask_rcnn/anchors/Variable/anchors/Variable/read_tower_1/mask_rcnn/anchors/Variable_0' creation.
Device assignments active during op 'tower_1/mask_rcnn/anchors/Variable/anchors/Variable/read_tower_1/mask_rcnn/anchors/Variable_0' creation:
with tf.device(/gpu:1): <M:\new\mrcnn\parallel_model.py:70> No node-device colocations were active during op 'anchors/Variable' creation.
No device assignments were active during op 'anchors/Variable' creation.
Adding this line:
import keras.backend.tensorflow_backend as KTF config = tf.ConfigProto()
config.allow_soft_placement=True
session = tf.Session(config=config)
KTF.set_session(session)
Solution 2:(not recommended)
downgrade Keras to 2.1.3:
conda install keras=2.1.3
(this works for someone but not works for me)
Reference:
https://github.com/matterport/Mask_RCNN/issues/921
https://github.com/tensorflow/tensorflow/issues/2285
Fix multiple GPUs fails in training Mask_RCNN的更多相关文章
- HDU 4913 Least common multiple(2014 Multi-University Training Contest 5)
题意:求所有自己的最小公倍数的和. 该集合是 2^ai * 3^bi 思路:线段树. 线段树中存的是 [3^b * f(b)] f(b)表示 因子3 的最小公倍数3的部分 为 3^b的个数 ...
- Stochastic Multiple Choice Learning for Training Diverse Deep Ensembles
作者提出的方法是Algotithm 2.简单来说就是,训练的时候,在几个模型中,选取预测最准确的(也就是loss最低的)模型进行权重更新.
- CatBoost使用GPU实现决策树的快速梯度提升CatBoost Enables Fast Gradient Boosting on Decision Trees Using GPUs
python机器学习-乳腺癌细胞挖掘(博主亲自录制视频)https://study.163.com/course/introduction.htm?courseId=1005269003&ut ...
- Training a classifier
你已经学习了如何定义神经网络,计算损失和执行网络权重的更新. 现在你或许在思考. What about data? 通常当你需要处理图像,文本,音频,视频数据,你能够使用标准的python包将数据加载 ...
- 用matlab训练数字分类的深度神经网络Training a Deep Neural Network for Digit Classification
This example shows how to use Neural Network Toolbox™ to train a deep neural network to classify ima ...
- PatentTips - Hierarchical RAID system including multiple RAIDs
BACKGROUND OF THE INVENTION The present invention relates to a storage system offering large capacit ...
- [C4] Andrew Ng - Improving Deep Neural Networks: Hyperparameter tuning, Regularization and Optimization
About this Course This course will teach you the "magic" of getting deep learning to work ...
- Deep Learning with Torch
原文地址:https://github.com/soumith/cvpr2015/blob/master/Deep%20Learning%20with%20Torch.ipynb Deep Learn ...
- VGGNet论文翻译-Very Deep Convolutional Networks for Large-Scale Image Recognition
Very Deep Convolutional Networks for Large-Scale Image Recognition Karen Simonyan[‡] & Andrew Zi ...
随机推荐
- witchcase
#include "stdafx.h" #include using namespace std; int _tmain(int argc, _TCHAR* argv[]) { i ...
- Lumen 使用事件需要注意的事项
Lumen 版本 5.2 参考手册 laravel event 需要注意的事项 如果是第一次在lumen下使用事件,需要修改bootstrap\app.php文件 添加对EventServicePro ...
- Mycat分布式数据库架构解决方案--搭建MySQL读写分离环境--一主多从
echo编辑整理,欢迎转载,转载请声明文章来源.欢迎添加echo微信(微信号:t2421499075)交流学习. 百战不败,依不自称常胜,百败不颓,依能奋力前行.--这才是真正的堪称强大!!! 本文主 ...
- iOS - 架构的认识过程,悬崖勒马。
16年的时候写过一篇代码讲解的,依旧是这三种架构,现在20年将近了,看到好的文章,是否增加新的认识. 16年链接 iOS - 架构模式 - 解密 MVC.MVP.MVVM.VIPER架构 新项目选择架 ...
- Java 数据类型 & 变量与常量 & 注释
一.数据类型 1.数据类型分类 Java 的数据类型分为两大类: 基本数据类型:整数.浮点数.字符型.布尔型 引用数据类型(对象类型):类.数组,字符串.接口等. 2.基本数据类型 四类八种基本数据类 ...
- 通过nginx部署前端代码实现前后端分离
实现前后端分离,可以让前后端独立开发.独立部署.独立单测,双方通过JSON进行数据交互. 对于前端开发人员来说,不用每次调试都需要启动或配置Java/Tomcat运行环境:对于后端开发人员来说 ,也不 ...
- MES应用案例|新宏泰电器乘上智能制造的东风
企业背景: 无锡新宏泰电器科技股份有限公司(下文简称:新宏泰电器)创立于1984年,公司主要生产断路器.微型电机.BMC/SMC材料.BMC/SMC模压制品及各类塑料模具的设计制造.已于2016年在沪 ...
- Excel 批量导入Mysql(创建表-追加数据)
之前弄数据库的时候, 测试excel导mysql, 中间用pandas 处理后再入库. 直接上代码, 此种有真意, 尽在不言中. #!/usr/bin/env python # coding: ut ...
- git使用.gitignore文件忽略相关文件上传
在使用git时,有些文件是不需要上传到仓库中的,比如idea/eclipse相关的文件,编译后的文件(target目录)等. 可以使用.gitignore文件进行配置.在git本地仓库创建.gitig ...
- 树莓派无显示屏连接wifi
在烧好Raspbian系统的TF卡boot分区新建 wpa_supplicant.conf 文件,内容如下(修改自己的WIFI名和密码,key_mgmt根据路由器配置),保存后启动树莓派即可自动连接W ...