Why the memory keeps increasing during training

[myalos]

I train nyu at night and find that it is killed the next day morning. I repeat the training and find the memory keep increasing. My total memory is 32G. I use NVIDIA tensorflow r1.15 https://github.com/NVIDIA/tensorflow/tree/r1.15.