더러운 python 해결책 모음

• pip의 requirements.txt를 conda에서 설치할 때, 먼저 conda에서 찾아보고, 없으면 pip 에서 찾아보고, 없으면 무시하기
  • https://stackoverflow.com/questions/35802939/install-only-available-packages-using-conda-install-yes-file-requirements-t

while read requirement; do conda install --yes $requirement || pip install $requirement; done < requirements.txt

• gpu 메모리 먹는 아가 찾기

import torch
import gc
tensors = []
tensor_sizes = []
# torch tensor objectを全部集める
for obj in gc.get_objects():
   try:
       if torch.is_tensor(obj) or (hasattr(obj, 'data') and torch.is_tensor(obj.data)):
           if str(obj.device) != 'cpu':
               numel = obj.numel()
               if obj.dtype == torch.float32:
                   numel *= 4
               elif obj.dtype == torch.int64:
                   numel *= 8
               elif obj.dtype not in dtypes:
                   dtypes.append(obj.dtype)
                   print(obj.dtype)
                   numel *= 8
               tensor_sizes.append(numel)
               tensors.append(obj)
   except:
       pass
# 大きさ順にsort
size_index = sorted(range(len(tensor_sizes)), key=lambda k: tensor_sizes[k], reverse=True)
# 上位5個のtensorの大きさ
for i in range(0, 20):
   print(f"{tensors[size_index[i]].shape}, {tensor_sizes[size_index[i]] / 1024} KBytes")
   # print(tensors[size_index[i]])
# tensor 全体の大きさ(bytes)
total_sum = 0
for tensor_size in tensor_sizes:
   total_sum += tensor_size
print(f"{total_sum / (1024 * 1024)} MBytes")

• conda 관련
  • 환경 만들기
    • conda create -n myenv -c intel python=3.7 numpy scipy mkl
    • conda env create -f environment.yml -n myenv
  • 환경 업데이트하기
    • conda env update -f environment.yml
  • 환경 뽑아내기
    • conda env export --no-build > environment.yml
  • 환경 지우기
    • conda env remove -n myenv