Windows＋Anaconda＋DeepLabを低スペックGPUで動かす
結論からいうと、DeeplabでGPUを使うには潤沢なGPUメモリが必要。
Macbook Airで処理が厳しかったのでWindows機のGPUを利用したら早くなるかと思ったけど、そう甘い話ではなかった。
使用したGPUはGeForce GTX750Ti。メモリは2GB。
NVIDIA® GPU ドライバのインストール
下記サイトでGPU情報を入力してダウンロード。
Windows Driver Typeは合ってないとインストールできないらしいのでとりあえずStandardを選択。Download TypeについてはGRDしかサーチにヒットしなかった。
インストーラを実行して再起動。
Anacondaのインストール
下記サイトから3.7をダウンロード。
インストーラを実行。
Advanced Optionsの"Add Anaconda to the System PATH environment variable"にチェックを入れる。
Bashのインストール
コマンドプロンプトでそのまま動かすにはGit BashがよさげなのでGitをインストール。Windows Subsystem for Linuxなんていらなかった。
インストーラ実行時、“Use Git from the Windows Command Prompt”にチェックを入れる。その後“Use Windows’ default console window”もチェックを入れておく。
コマンドプロンプトで"bash"コマンドを実行してbashが起動することを確認する。
Tensorflowのインストール
Anaconda Navigatorを起動。
Environmentsで新規環境をCreate。（Python3.6を選択）
Environmentsでtensorflow-gpuのパッケージをインストール。
HomeでJupyter Notebookをインストール。
tensorflow-gpuをインストールするとCUDAとcuDNNもインストールされる。
CUDAとcuDNNのバージョンに拘る場合はCUDA→cuDNN→tensorflow-gpuの順に"Mark for specific version installation"でバージョンを指定。
バージョンについては以下サイトを参照。
DeepLabのインストール
EnvironmentsでPillowのパッケージをインストール。
EnvironmentsでJupyterのパッケージをインストール。
EnvironmentsでMatplotlibのパッケージをインストール。
環境の▶︎をクリックしてターミナルを起動して、以下を実行。
$ bash
$ cd
$ mkdir tensorflow
$ cd tensorflow
$ git clone https://github.com/tensorflow/models.git以下を実行。（ターミナル、Bashの起動は以降省略）
$ cd tensorflow/models/research/
$ export PYTHONPATH=$PYTHONPATH:`pwd`:`pwd`/slim
$ python deeplab/model_test.py自分の環境では以下のエラーが発生。
Traceback (most recent call last):
File "deeplab/model_test.py", line 20, in <module>
from deeplab import common
ModuleNotFoundError: No module named 'deeplab'
どうもPYTHONPATHをexportしたのではダメっぽい。おとなしく環境変数にPYTHONPATHを追加。
再起動して再実行。
WARNING: The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:
* https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
* https://github.com/tensorflow/addons
If you depend on functionality not listed there, please file an issue.
WARNING:tensorflow:From C:\Users\shin\.conda\envs\deeplab_mkl\lib\contextlib.py:60: TensorFlowTestCase.test_session (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version.
Instructions for updating:
Use `self.session()` or `self.cached_session()` instead.
2019-08-02 12:51:18.584982: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX AVX2
2019-08-02 12:51:18.593985: I tensorflow/core/common_runtime/process_util.cc:71] Creating new thread pool with default inter op setting: 4. Tune using inter_op_parallelism_threads for best performance.
C:\Users\shin\.conda\envs\deeplab_mkl\lib\site-packages\numpy\lib\type_check.py:546: DeprecationWarning: np.asscalar(a) is deprecated since NumPy v1.16, use a.item() instead
'a.item() instead', DeprecationWarning, stacklevel=1)
WARNING:tensorflow:From C:\Users\shin\tensorflow\models\research\deeplab\core\feature_extractor.py:196: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
WARNING:tensorflow:From C:\Users\shin\.conda\envs\deeplab_mkl\lib\site-packages\tensorflow\python\framework\op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
....s
----------------------------------------------------------------------
Ran 5 tests in 25.252s
OK (skipped=1)
とりあえずOK。
wgetをインストールして環境変数を設定。
（自分はここからセットアップファイルをDLした）
以下を実行。
$ cd tensorflow/models/research/deeplab
$ sh local_test.sh 以下のエラーが発生。
WARNING: The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:
* https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
* https://github.com/tensorflow/addons
If you depend on functionality not listed there, please file an issue.
Traceback (most recent call last):
File "C:/Users/shin/tensorflow/models/research/deeplab/model_test.py", line 21, in <module>
from deeplab import model
File "C:\Users\shin\tensorflow\models\research\deeplab\model.py", line 56, in <module>
from deeplab.core import feature_extractor
File "C:\Users\shin\tensorflow\models\research\deeplab\core\feature_extractor.py", line 24, in <module>
from nets.mobilenet import mobilenet_v2
ModuleNotFoundError: No module named 'nets'
先程実行したmodel_test.pyでまたimportに失敗している。確認するとnetsにPYTHONPATHは通っているようなので環境変数とexportが干渉している模様。
とりあえずlocal_test.shの33行目からexportをコメントアウト。
# export PYTHONPATH=$PYTHONPATH:`pwd`:`pwd`/slim
再度実行するとResourceExhaustedErrorが発生する。
GPUメモリに余裕がある環境では正常終了するはず。
以下はエラー内容。
2019-08-12 01:38:44.417028: W tensorflow/core/common_runtime/bfc_allocator.cc:271] ***********************************************************____**********________***********xxxxxxxx
2019-08-12 01:38:44.420560: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at pad_op.cc:122 : Resource exhausted: OOM when allocating tensor with shape[512,259,259] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
Traceback (most recent call last):
File "D:\conda\envs\deeplab_gpu\lib\site-packages\tensorflow\python\client\session.py", line 1334, in _do_call
return fn(*args)
File "D:\conda\envs\deeplab_gpu\lib\site-packages\tensorflow\python\client\session.py", line 1319, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "D:\conda\envs\deeplab_gpu\lib\site-packages\tensorflow\python\client\session.py", line 1407, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[512,259,259] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[{{node xception_65/entry_flow/block1/unit_1/xception_module/Pad}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
[[{{node gradients/xception_65/middle_flow/block1/unit_16/xception_module/separable_conv1_pointwise/BatchNorm/FusedBatchNorm_grad/FusedBatchNormGrad}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:/tensorflow/models/research/deeplab/train.py", line 513, in <module>
tf.app.run()
File "D:\conda\envs\deeplab_gpu\lib\site-packages\tensorflow\python\platform\app.py", line 125, in run
_sys.exit(main(argv))
File "C:/tensorflow/models/research/deeplab/train.py", line 507, in main
sess.run([train_tensor])
File "D:\conda\envs\deeplab_gpu\lib\site-packages\tensorflow\python\training\monitored_session.py", line 676, in run
run_metadata=run_metadata)
File "D:\conda\envs\deeplab_gpu\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1171, in run
run_metadata=run_metadata)
File "D:\conda\envs\deeplab_gpu\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1270, in run
raise six.reraise(*original_exc_info)
File "D:\conda\envs\deeplab_gpu\lib\site-packages\six.py", line 693, in reraise
raise value
File "D:\conda\envs\deeplab_gpu\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1255, in run
return self._sess.run(*args, **kwargs)
File "D:\conda\envs\deeplab_gpu\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1327, in run
run_metadata=run_metadata)
File "D:\conda\envs\deeplab_gpu\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1091, in run
return self._sess.run(*args, **kwargs)
File "D:\conda\envs\deeplab_gpu\lib\site-packages\tensorflow\python\client\session.py", line 929, in run
run_metadata_ptr)
File "D:\conda\envs\deeplab_gpu\lib\site-packages\tensorflow\python\client\session.py", line 1152, in _run
feed_dict_tensor, options, run_metadata)
File "D:\conda\envs\deeplab_gpu\lib\site-packages\tensorflow\python\client\session.py", line 1328, in _do_run
run_metadata)
File "D:\conda\envs\deeplab_gpu\lib\site-packages\tensorflow\python\client\session.py", line 1348, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[512,259,259] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[node xception_65/entry_flow/block1/unit_1/xception_module/Pad (defined at C:\tensorflow\models\research\deeplab\core\xception.py:90) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
[[node gradients/xception_65/middle_flow/block1/unit_16/xception_module/separable_conv1_pointwise/BatchNorm/FusedBatchNorm_grad/FusedBatchNormGrad (defined at C:/tensorflow/models/research/deeplab/train.py:386) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
Caused by op 'xception_65/entry_flow/block1/unit_1/xception_module/Pad', defined at:
File "C:/tensorflow/models/research/deeplab/train.py", line 513, in <module>
tf.app.run()
File "D:\conda\envs\deeplab_gpu\lib\site-packages\tensorflow\python\platform\app.py", line 125, in run
_sys.exit(main(argv))
File "C:/tensorflow/models/research/deeplab/train.py", line 464, in main
dataset.ignore_label)
File "C:/tensorflow/models/research/deeplab/train.py", line 373, in _train_deeplab_model
reuse_variable=(i != 0))
File "C:/tensorflow/models/research/deeplab/train.py", line 269, in _tower_loss
_build_deeplab(iterator, {common.OUTPUT_TYPE: num_of_classes}, ignore_label)
File "C:/tensorflow/models/research/deeplab/train.py", line 229, in _build_deeplab
'total_training_steps': FLAGS.training_number_of_steps,
File "C:\tensorflow\models\research\deeplab\model.py", line 313, in multi_scale_logits
nas_training_hyper_parameters=nas_training_hyper_parameters)
File "C:\tensorflow\models\research\deeplab\model.py", line 553, in _get_logits
nas_training_hyper_parameters=nas_training_hyper_parameters)
File "C:\tensorflow\models\research\deeplab\model.py", line 395, in extract_features
use_bounded_activation=model_options.use_bounded_activation)
File "C:\tensorflow\models\research\deeplab\core\feature_extractor.py", line 341, in extract_features
scope=name_scope[model_variant])
File "C:\tensorflow\models\research\deeplab\core\feature_extractor.py", line 408, in network_fn
*args, **kwargs)
File "C:\tensorflow\models\research\deeplab\core\xception.py", line 655, in xception_65
scope=scope)
File "C:\tensorflow\models\research\deeplab\core\xception.py", line 464, in xception
net = stack_blocks_dense(net, blocks, output_stride)
File "D:\conda\envs\deeplab_gpu\lib\site-packages\tensorflow\contrib\framework\python\ops\arg_scope.py", line 182, in func_with_args
return func(*args, **current_args)
File "C:\tensorflow\models\research\deeplab\core\xception.py", line 382, in stack_blocks_dense
net = block.unit_fn(net, rate=1, **unit)
File "D:\conda\envs\deeplab_gpu\lib\site-packages\tensorflow\contrib\framework\python\ops\arg_scope.py", line 182, in func_with_args
return func(*args, **current_args)
File "C:\tensorflow\models\research\deeplab\core\xception.py", line 293, in xception_module
scope='separable_conv' + str(i+1))
File "C:\tensorflow\models\research\deeplab\core\xception.py", line 284, in _separable_conv
scope=scope)
File "D:\conda\envs\deeplab_gpu\lib\site-packages\tensorflow\contrib\framework\python\ops\arg_scope.py", line 182, in func_with_args
return func(*args, **current_args)
File "C:\tensorflow\models\research\deeplab\core\xception.py", line 187, in separable_conv2d_same
inputs = fixed_padding(inputs, kernel_size, rate)
File "C:\tensorflow\models\research\deeplab\core\xception.py", line 90, in fixed_padding
[pad_beg, pad_end], [0, 0]])
File "D:\conda\envs\deeplab_gpu\lib\site-packages\tensorflow\python\ops\array_ops.py", line 2299, in pad
result = gen_array_ops.pad(tensor, paddings, name=name)
File "D:\conda\envs\deeplab_gpu\lib\site-packages\tensorflow\python\ops\gen_array_ops.py", line 5539, in pad
"Pad", input=input, paddings=paddings, name=name)
File "D:\conda\envs\deeplab_gpu\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 788, in _apply_op_helper
op_def=op_def)
File "D:\conda\envs\deeplab_gpu\lib\site-packages\tensorflow\python\util\deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "D:\conda\envs\deeplab_gpu\lib\site-packages\tensorflow\python\framework\ops.py", line 3300, in create_op
op_def=op_def)
File "D:\conda\envs\deeplab_gpu\lib\site-packages\tensorflow\python\framework\ops.py", line 1801, in __init__
self._traceback = tf_stack.extract_stack()
ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[512,259,259] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[node xception_65/entry_flow/block1/unit_1/xception_module/Pad (defined at C:\tensorflow\models\research\deeplab\core\xception.py:90) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
[[node gradients/xception_65/middle_flow/block1/unit_16/xception_module/separable_conv1_pointwise/BatchNorm/FusedBatchNorm_grad/FusedBatchNormGrad (defined at C:/tensorflow/models/research/deeplab/train.py:386) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
調べると「GPUのメモリが少ない時はバッチサイズを小さくせよ」と書いてあるのでtrain.pvをtrain_batch_size=1, fine_tune_batch_norm=falseに変更するが解決しない。
他に「learning rateを下げろ」とも書いてあるが、shapeのメモリ割当がエラーになってる状況で学習率を下げれば解決というのは直感的ではない。実際base_learning_rateを一桁下げても解決しなかった。
最終手段としてshapeのサイズを大きく削るためにtrain_crop_sizeを513から半分の257に落とすことでようやく動くようになった。
以下は実行結果。（tarファイルはwget済で解凍はコメントアウトしている）
$sh local_test.sh
WARNING: The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:
* https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
* https://github.com/tensorflow/addons
If you depend on functionality not listed there, please file an issue.
testBuildDeepLabWithDensePredictionCell (__main__.DeeplabModelTest) ... WARNING:tensorflow:From D:\conda\envs\deeplab_gpu_1131\lib\contextlib.py:60: TensorFlowTestCase.test_session (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version.
Instructions for updating:
Use `self.session()` or `self.cached_session()` instead.
2019-08-14 00:42:47.465318: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX AVX2
2019-08-14 00:42:47.962465: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: GeForce GTX 750 Ti major: 5 minor: 0 memoryClockRate(GHz): 1.0845
pciBusID: 0000:01:00.0
totalMemory: 2.00GiB freeMemory: 1.64GiB
2019-08-14 00:42:47.979726: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-08-14 00:47:42.772388: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-08-14 00:47:42.776306: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0
2019-08-14 00:47:42.778441: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N
2019-08-14 00:47:42.845737: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 614 MB memory) -> physical GPU (device: 0, name: GeForce GTX 750 Ti, pci bus id: 0000:01:00.0, compute capability: 5.0)
D:\conda\envs\deeplab_gpu_1131\lib\site-packages\numpy\lib\type_check.py:546: DeprecationWarning: np.asscalar(a) is deprecated since NumPy v1.16, use a.item() instead
'a.item() instead', DeprecationWarning, stacklevel=1)
WARNING:tensorflow:From C:\tensorflow\models\research\deeplab\core\feature_extractor.py:196: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
WARNING:tensorflow:From D:\conda\envs\deeplab_gpu_1131\lib\site-packages\tensorflow\python\framework\op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
ok
testBuildDeepLabv2 (__main__.DeeplabModelTest) ... 2019-08-14 00:47:45.789489: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-08-14 00:47:45.793316: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-08-14 00:47:45.796794: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0
2019-08-14 00:47:45.798929: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N
2019-08-14 00:47:45.801131: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 614 MB memory) -> physical GPU (device: 0, name: GeForce GTX 750 Ti, pci bus id: 0000:01:00.0, compute capability: 5.0)
2019-08-14 00:47:52.113547: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-08-14 00:47:52.117555: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-08-14 00:47:52.121403: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0
2019-08-14 00:47:52.124080: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N
2019-08-14 00:47:52.126899: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 614 MB memory) -> physical GPU (device: 0, name: GeForce GTX 750 Ti, pci bus id: 0000:01:00.0, compute capability: 5.0)
2019-08-14 00:47:59.469595: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-08-14 00:47:59.472214: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-08-14 00:47:59.475243: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0
2019-08-14 00:47:59.477378: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N
2019-08-14 00:47:59.479691: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 614 MB memory) -> physical GPU (device: 0, name: GeForce GTX 750 Ti, pci bus id: 0000:01:00.0, compute capability: 5.0)
2019-08-14 00:48:01.541404: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-08-14 00:48:01.544406: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-08-14 00:48:01.548149: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0
2019-08-14 00:48:01.550858: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N
2019-08-14 00:48:01.553651: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 614 MB memory) -> physical GPU (device: 0, name: GeForce GTX 750 Ti, pci bus id: 0000:01:00.0, compute capability: 5.0)
ok
testForwardpassDeepLabv3plus (__main__.DeeplabModelTest) ... 2019-08-14 00:48:04.755435: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-08-14 00:48:04.758365: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-08-14 00:48:04.761613: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0
2019-08-14 00:48:04.764360: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N
2019-08-14 00:48:04.766755: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 614 MB memory) -> physical GPU (device: 0, name: GeForce GTX 750 Ti, pci bus id: 0000:01:00.0, compute capability: 5.0)
ok
testWrongDeepLabVariant (__main__.DeeplabModelTest) ... ok
test_session (__main__.DeeplabModelTest)
Use cached_session instead. (deprecated) ... skipped 'Not a test.'
----------------------------------------------------------------------
Ran 5 tests in 322.189s
OK (skipped=1)
Uncompressing VOCtrainval_11-May-2012.tar
Removing the color map in ground truth annotations...
Converting PASCAL VOC 2012 dataset...
2019-08-14 00:48:42.821480: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX AVX2
2019-08-14 00:48:43.028672: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: GeForce GTX 750 Ti major: 5 minor: 0 memoryClockRate(GHz): 1.0845
pciBusID: 0000:01:00.0
totalMemory: 2.00GiB freeMemory: 1.64GiB
2019-08-14 00:48:43.036902: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-08-14 00:48:44.377312: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-08-14 00:48:44.381732: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0
2019-08-14 00:48:44.384612: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N
2019-08-14 00:48:44.387737: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 1384 MB memory) -> physical GPU (device: 0, name: GeForce GTX 750 Ti, pci bus id: 0000:01:00.0, compute capability: 5.0)
2019-08-14 00:48:44.400739: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-08-14 00:48:44.404001: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-08-14 00:48:44.408739: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0
2019-08-14 00:48:44.411680: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N
2019-08-14 00:48:44.414759: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 1384 MB memory) -> physical GPU (device: 0, name: GeForce GTX 750 Ti, pci bus id: 0000:01:00.0, compute capability: 5.0)
>> Converting image 1/1464 shard 0WARNING:tensorflow:From ./build_voc2012_data.py:116: FastGFile.__init__ (from tensorflow.python.platform.gfile) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.gfile.GFile.
>> Converting image 366/1464 shard 0
>> Converting image 732/1464 shard 1
>> Converting image 1098/1464 shard 2
>> Converting image 1464/1464 shard 3
2019-08-14 00:48:55.502984: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-08-14 00:48:55.506709: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-08-14 00:48:55.511331: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0
2019-08-14 00:48:55.514388: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N
2019-08-14 00:48:55.517934: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 1384 MB memory) -> physical GPU (device: 0, name: GeForce GTX 750 Ti, pci bus id: 0000:01:00.0, compute capability: 5.0)
2019-08-14 00:48:55.530578: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-08-14 00:48:55.534019: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-08-14 00:48:55.538623: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0
2019-08-14 00:48:55.541595: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N
2019-08-14 00:48:55.545672: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 1384 MB memory) -> physical GPU (device: 0, name: GeForce GTX 750 Ti, pci bus id: 0000:01:00.0, compute capability: 5.0)
>> Converting image 729/2913 shard 0
>> Converting image 1458/2913 shard 1
>> Converting image 2187/2913 shard 2
>> Converting image 2913/2913 shard 3
2019-08-14 00:49:15.413506: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-08-14 00:49:15.417567: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-08-14 00:49:15.422175: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0
2019-08-14 00:49:15.425160: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N
2019-08-14 00:49:15.428225: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 1384 MB memory) -> physical GPU (device: 0, name: GeForce GTX 750 Ti, pci bus id: 0000:01:00.0, compute capability: 5.0)
2019-08-14 00:49:15.440785: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-08-14 00:49:15.444147: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-08-14 00:49:15.449126: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0
2019-08-14 00:49:15.452633: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N
2019-08-14 00:49:15.455698: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 1384 MB memory) -> physical GPU (device: 0, name: GeForce GTX 750 Ti, pci bus id: 0000:01:00.0, compute capability: 5.0)
>> Converting image 363/1449 shard 0
>> Converting image 726/1449 shard 1
>> Converting image 1089/1449 shard 2
>> Converting image 1449/1449 shard 3
SYSTEM_WGETRC = c:/progra~1/wget/etc/wgetrc
syswgetrc = C:\Program Files (x86)\GnuWin32/etc/wgetrc
--2019-08-14 00:49:25-- http://download.tensorflow.org/models/deeplabv3_pascal_train_aug_2018_01_04.tar.gz
download.tensorflow.org をDNSに問いあわせています... 172.217.161.240, 2404:6800:400a:80c::2010
download.tensorflow.org|172.217.161.240|:80 に接続しています... 接続しました。
HTTP による接続要求を送信しました、応答を待っています... 416 Requested range not satisfiable
このファイルは既に全部取得しおわっています。何もすることはありません。
WARNING: The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:
* https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
* https://github.com/tensorflow/addons
If you depend on functionality not listed there, please file an issue.
INFO:tensorflow:Training on trainval set
WARNING:tensorflow:From D:\conda\envs\deeplab_gpu_1131\lib\site-packages\tensorflow\python\ops\control_flow_ops.py:423: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
WARNING:tensorflow:From C:\tensorflow\models\research\deeplab\core\preprocess_utils.py:203: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
WARNING:tensorflow:From D:\conda\envs\deeplab_gpu_1131\lib\site-packages\tensorflow\python\training\learning_rate_decay_v2.py:321: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Deprecated in favor of operator or tf.math.divide.
WARNING:tensorflow:From C:\tensorflow\models\research\deeplab\core\feature_extractor.py:196: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
WARNING:tensorflow:From D:\conda\envs\deeplab_gpu_1131\lib\site-packages\tensorflow\python\keras\layers\core.py:143: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.
Instructions for updating:
Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.
WARNING:tensorflow:From C:/tensorflow/models/research/deeplab/train.py:418: Print (from tensorflow.python.ops.logging_ops) is deprecated and will be removed after 2018-08-20.
Instructions for updating:
Use tf.print instead of tf.Print. Note that tf.print returns a no-output operator that directly prints the output. Outside of defuns or eager mode, this operator will not be executed unless it is directly specified in session.run or used as a control dependency for other operators. This is only a concern in graph mode. Below is an example of how to ensure tf.print executes in graph mode:
```python
sess = tf.Session()
with sess.as_default():
tensor = tf.range(10)
print_op = tf.print(tensor)
with tf.control_dependencies([print_op]):
out = tf.add(tensor, tensor)
sess.run(out)
```
Additionally, to use tf.print in python 2.7, users must make sure to import
the following:
`from __future__ import print_function`
INFO:tensorflow:Ignoring initialization; other checkpoint exists
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Graph was finalized.
2019-08-14 00:50:04.529118: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX AVX2
2019-08-14 00:50:04.788138: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: GeForce GTX 750 Ti major: 5 minor: 0 memoryClockRate(GHz): 1.0845
pciBusID: 0000:01:00.0
totalMemory: 2.00GiB freeMemory: 1.64GiB
2019-08-14 00:50:04.795740: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-08-14 00:50:06.046974: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-08-14 00:50:06.051474: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0
2019-08-14 00:50:06.054596: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N
2019-08-14 00:50:06.057693: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 1384 MB memory) -> physical GPU (device: 0, name: GeForce GTX 750 Ti, pci bus id: 0000:01:00.0, compute capability: 5.0)
WARNING:tensorflow:From D:\conda\envs\deeplab_gpu_1131\lib\site-packages\tensorflow\python\training\saver.py:1266: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
INFO:tensorflow:Restoring parameters from C:/tensorflow/models/research/deeplab/datasets/pascal_voc_seg/exp/train_on_trainval_set/train\model.ckpt-0
WARNING:tensorflow:From D:\conda\envs\deeplab_gpu_1131\lib\site-packages\tensorflow\python\training\saver.py:1070: get_checkpoint_mtimes (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file utilities to get mtimes.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Saving checkpoints for 0 into C:/tensorflow/models/research/deeplab/datasets/pascal_voc_seg/exp/train_on_trainval_set/train\model.ckpt.
2019-08-14 00:51:25.641196: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.02GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-08-14 00:51:25.780001: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.05GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-08-14 00:51:25.859173: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.55GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-08-14 00:51:25.868774: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.07GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-08-14 00:51:25.879919: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 781.71MiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-08-14 00:51:26.125294: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.13GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-08-14 00:51:26.174449: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.67GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-08-14 00:51:26.200191: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 715.38MiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
Total loss is :[0.491420537]
2019-08-14 00:51:26.513466: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 682.88MiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-08-14 00:51:26.521687: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 685.54MiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
INFO:tensorflow:Saving checkpoints for 10 into C:/tensorflow/models/research/deeplab/datasets/pascal_voc_seg/exp/train_on_trainval_set/train\model.ckpt.
WARNING: The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:
* https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
* https://github.com/tensorflow/addons
If you depend on functionality not listed there, please file an issue.
INFO:tensorflow:Evaluating on val set
WARNING:tensorflow:From D:\conda\envs\deeplab_gpu_1131\lib\site-packages\tensorflow\python\ops\control_flow_ops.py:423: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
INFO:tensorflow:Performing single-scale test.
WARNING:tensorflow:From C:\tensorflow\models\research\deeplab\core\feature_extractor.py:196: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
WARNING:tensorflow:From D:\conda\envs\deeplab_gpu_1131\lib\site-packages\tensorflow\python\ops\metrics_impl.py:259: to_int64 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
WARNING:tensorflow:From D:\conda\envs\deeplab_gpu_1131\lib\site-packages\tensorflow\python\ops\confusion_matrix.py:194: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
WARNING:tensorflow:From D:\conda\envs\deeplab_gpu_1131\lib\site-packages\tensorflow\python\ops\metrics_impl.py:1155: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Deprecated in favor of operator or tf.math.divide.
WARNING:tensorflow:From C:/tensorflow/models/research/deeplab/eval.py:170: print_model_analysis (from tensorflow.contrib.tfprof.model_analyzer) is deprecated and will be removed after 2018-01-01.
Instructions for updating:
Use `tf.profiler.profile(graph, run_meta, op_log, cmd, options)`. Build `options` with `tf.profiler.ProfileOptionBuilder`. See README.md for details
WARNING:tensorflow:From D:\conda\envs\deeplab_gpu_1131\lib\site-packages\tensorflow\python\profiler\internal\flops_registry.py:142: tensor_shape_from_node_def_name (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.graph_util.remove_training_nodes
4 ops no flops stats due to incomplete shapes.
Parsing Inputs...
=========================Options=============================
-max_depth 10000
-min_bytes 0
-min_peak_bytes 0
-min_residual_bytes 0
-min_output_bytes 0
-min_micros 0
-min_accelerator_micros 0
-min_cpu_micros 0
-min_params 0
-min_float_ops 0
-min_occurrence 0
-step -1
-order_by name
-account_type_regexes _trainable_variables
-start_name_regexes .*
-trim_name_regexes
-show_name_regexes .*
-hide_name_regexes
-account_displayed_op_only true
-select params
-output stdout:
==================Model Analysis Report======================
Doc:
scope: The nodes in the model graph are organized by their names, which is hierarchical like filesystem.
param: Number of parameters (in the Variable).
Profile:
node name | # parameters
_TFProfRoot (--/41.06m params)
aspp0 (--/524.80k params)
aspp0/BatchNorm (--/512 params)
（略）
xception_65/middle_flow/block1/unit_9/xception_module/separable_conv3_pointwise/BatchNorm/gamma (728, 728/728 params)
xception_65/middle_flow/block1/unit_9/xception_module/separable_conv3_pointwise/weights (1x1x728x728, 529.98k/529.98k params)
======================End of Report==========================
4 ops no flops stats due to incomplete shapes.
Parsing Inputs...
=========================Options=============================
-max_depth 10000
-min_bytes 0
-min_peak_bytes 0
-min_residual_bytes 0
-min_output_bytes 0
-min_micros 0
-min_accelerator_micros 0
-min_cpu_micros 0
-min_params 0
-min_float_ops 1
-min_occurrence 0
-step -1
-order_by float_ops
-account_type_regexes .*
-start_name_regexes .*
-trim_name_regexes
-show_name_regexes .*
-hide_name_regexes
-account_displayed_op_only true
-select float_ops
-output stdout:
==================Model Analysis Report======================
Doc:
scope: The nodes in the model graph are organized by their names, which is hierarchical like filesystem.
flops: Number of float operations. Note: Please read the implementation for the math behind it.
Profile:
node name | # float_ops
_TFProfRoot (--/108.33b flops)
xception_65/exit_flow/block2/unit_1/xception_module/separable_conv3_pointwise/Conv2D (6.85b/6.85b flops)
xception_65/exit_flow/block2/unit_1/xception_module/separable_conv2_pointwise/Conv2D (5.14b/5.14b flops)
（略）
xception_65/entry_flow/block1/unit_1/xception_module/separable_conv1_depthwise/depthwise_weights/Initializer/truncated_normal (576/1.15k flops)
xception_65/entry_flow/block1/unit_1/xception_module/separable_conv1_depthwise/depthwise_weights/Initializer/truncated_normal/mul (576/576 flops)
mean_iou/AssignAdd (441/441 flops)
mean_iou/Sum (420/420 flops)
mean_iou/Sum_1 (420/420 flops)
mean_iou/NotEqual (21/21 flops)
mean_iou/add (21/21 flops)
mean_iou/div (21/21 flops)
mean_iou/Greater (21/21 flops)
mean_iou/sub (21/21 flops)
mean_iou/Sum_2 (20/20 flops)
mean_iou/mean_iou (20/20 flops)
mean_iou/truediv (1/1 flops)
mean_iou/Greater_1 (1/1 flops)
======================End of Report==========================
INFO:tensorflow:Waiting for new checkpoint at C:/tensorflow/models/research/deeplab/datasets/pascal_voc_seg/exp/train_on_trainval_set/train
INFO:tensorflow:Found new checkpoint at C:/tensorflow/models/research/deeplab/datasets/pascal_voc_seg/exp/train_on_trainval_set/train\model.ckpt-10
INFO:tensorflow:Graph was finalized.
2019-08-14 00:52:16.250081: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX AVX2
2019-08-14 00:52:16.490306: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: GeForce GTX 750 Ti major: 5 minor: 0 memoryClockRate(GHz): 1.0845
pciBusID: 0000:01:00.0
totalMemory: 2.00GiB freeMemory: 1.64GiB
2019-08-14 00:52:16.495933: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-08-14 00:52:17.712482: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-08-14 00:52:17.715452: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0
2019-08-14 00:52:17.717487: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N
2019-08-14 00:52:17.719711: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 1384 MB memory) -> physical GPU (device: 0, name: GeForce GTX 750 Ti, pci bus id: 0000:01:00.0, compute capability: 5.0)
WARNING:tensorflow:From D:\conda\envs\deeplab_gpu_1131\lib\site-packages\tensorflow\python\training\saver.py:1266: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
INFO:tensorflow:Restoring parameters from C:/tensorflow/models/research/deeplab/datasets/pascal_voc_seg/exp/train_on_trainval_set/train\model.ckpt-10
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Starting evaluation at 2019-08-13-15:52:32
2019-08-14 00:52:36.191039: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.13GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-08-14 00:52:36.277546: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.77GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
WARNING: The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:
* https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
* https://github.com/tensorflow/addons
If you depend on functionality not listed there, please file an issue.
INFO:tensorflow:Visualizing on val set
WARNING:tensorflow:From D:\conda\envs\deeplab_gpu_1131\lib\site-packages\tensorflow\python\ops\control_flow_ops.py:423: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
INFO:tensorflow:Performing single-scale test.
WARNING:tensorflow:From C:\tensorflow\models\research\deeplab\core\feature_extractor.py:196: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
INFO:tensorflow:Waiting for new checkpoint at C:/tensorflow/models/research/deeplab/datasets/pascal_voc_seg/exp/train_on_trainval_set/train
INFO:tensorflow:Found new checkpoint at C:/tensorflow/models/research/deeplab/datasets/pascal_voc_seg/exp/train_on_trainval_set/train\model.ckpt-10
INFO:tensorflow:Starting visualization at 2019-08-13-15:59:03
INFO:tensorflow:Visualizing with model C:/tensorflow/models/research/deeplab/datasets/pascal_voc_seg/exp/train_on_trainval_set/train\model.ckpt-10
INFO:tensorflow:Graph was finalized.
2019-08-14 00:59:05.347459: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX AVX2
2019-08-14 00:59:05.601609: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: GeForce GTX 750 Ti major: 5 minor: 0 memoryClockRate(GHz): 1.0845
pciBusID: 0000:01:00.0
totalMemory: 2.00GiB freeMemory: 1.64GiB
2019-08-14 00:59:05.609238: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-08-14 00:59:06.847358: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-08-14 00:59:06.850280: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0
2019-08-14 00:59:06.852390: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N
2019-08-14 00:59:06.854572: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 1384 MB memory) -> physical GPU (device: 0, name: GeForce GTX 750 Ti, pci bus id: 0000:01:00.0, compute capability: 5.0)
WARNING:tensorflow:From D:\conda\envs\deeplab_gpu_1131\lib\site-packages\tensorflow\python\training\saver.py:1266: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
INFO:tensorflow:Restoring parameters from C:/tensorflow/models/research/deeplab/datasets/pascal_voc_seg/exp/train_on_trainval_set/train\model.ckpt-10
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Visualizing batch 1
2019-08-14 00:59:13.953295: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.13GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-08-14 00:59:14.038214: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.77GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
INFO:tensorflow:Visualizing batch 2
（略）
INFO:tensorflow:Visualizing batch 1450
INFO:tensorflow:Finished visualization at 2019-08-13-16:07:22
INFO:tensorflow:Prepare to export model to: C:/tensorflow/models/research/deeplab/datasets/pascal_voc_seg/exp/train_on_trainval_set/export/frozen_inference_graph.pb
INFO:tensorflow:Exported model performs single-scale inference.
WARNING:tensorflow:From C:\tensorflow\models\research\deeplab\core\feature_extractor.py:196: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
WARNING:tensorflow:From D:\conda\envs\deeplab_gpu_1131\lib\site-packages\tensorflow\python\framework\op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
WARNING:tensorflow:From C:/tensorflow/models/research/deeplab/export_model.py:163: all_variables (from tensorflow.python.ops.variables) is deprecated and will be removed after 2017-03-02.
Instructions for updating:
Please use tf.global_variables instead.
WARNING:tensorflow:From D:\conda\envs\deeplab_gpu_1131\lib\site-packages\tensorflow\python\tools\freeze_graph.py:127: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
2019-08-14 01:07:38.796015: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX AVX2
2019-08-14 01:07:38.985983: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: GeForce GTX 750 Ti major: 5 minor: 0 memoryClockRate(GHz): 1.0845
pciBusID: 0000:01:00.0
totalMemory: 2.00GiB freeMemory: 1.64GiB
2019-08-14 01:07:38.991631: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-08-14 01:07:40.196852: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-08-14 01:07:40.199853: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0
2019-08-14 01:07:40.201974: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N
2019-08-14 01:07:40.204284: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 1384 MB memory) -> physical GPU (device: 0, name: GeForce GTX 750 Ti, pci bus id: 0000:01:00.0, compute capability: 5.0)
INFO:tensorflow:Restoring parameters from C:/tensorflow/models/research/deeplab/datasets/pascal_voc_seg/exp/train_on_trainval_set/train/model.ckpt-10
WARNING:tensorflow:From D:\conda\envs\deeplab_gpu_1131\lib\site-packages\tensorflow\python\tools\freeze_graph.py:232: convert_variables_to_constants (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.graph_util.convert_variables_to_constants
WARNING:tensorflow:From D:\conda\envs\deeplab_gpu_1131\lib\site-packages\tensorflow\python\framework\graph_util_impl.py:245: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.graph_util.extract_sub_graph
INFO:tensorflow:Froze 732 variables.
INFO:tensorflow:Converted 732 variables to const ops.
$テスト～エクスポートまでの実行時間は26分。同パラメータをCPUに切り替えて実行した際の実行時間は101分だった。高速化自体はできている。
最終的に変更したtrain.pvのパラメータは以下の通り。
--train_crop_size="513,513" -> "257,257" \
--train_batch_size=4 -> 1 \
--fine_tune_batch_norm=true -> false \
注意事項
wgetのパッケージ
wgetをインストールする際にcondaコマンドでmenpo wgetをインストールすると、ダウンロード自体は正常に実行できるものの、キャリッジリターンが合ってないのか進捗表示が流れていく。
GitBashのプロンプト
Git Bashの戻りが遅いと感じたら以下のコマンドを実行。
PS1='$'
今後の方針
自分のWIndows環境でGPUを使うのは難しいとわかったので、GPU(TPU?)インスタンスの利用を検討する。GCP、できれば無料のGoogle Colaboratoryについて調査する。 ■
この記事が気に入ったらサポートをしてみませんか？