T5base memory usage for interface

[Oxi84]

System Info

Ubuntu 18.04 RTX 2080

Who can help?

@patrickvonplaten

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Just load t5 base from any example.

Expected behavior

Is it normal that T5 base uses around 1000 MB of GPU memory in half precision (FP-16) for interface.

Many other models use roughly 50 percent of FP-32 model size for FP-16 which should be only 400-500 MB.

I read here that the model https://huggingface.co/google/t5-efficient-small-el16 is 350 mb in size so it uses roughly 50% of the size (175 MB) when used for interface.

Is there anything to do to reduce the memory usage. T5-large for example also in Fp16 uses around 2GB which is much better.

I expected memory usage to be 50% of model size for fp 16.

[patrickvonplaten]

@Oxi84,

Could you please provide us with a codesnippet that shows the error?

[Oxi84]

@patrickvonplaten

Here is it - it takes 1300 mb of GPU memory and the model size is just 850 mb. I tried all combinations, and 1300 mb is minimum. For example T5 large takes around 2000 MB, and ithe model size is 2.7 GB so it is more than 3X larger.

I expected T5 base would not use more than 600 MB of GPU memory.

from transformers import T5ForConditionalGeneration,T5Tokenizer,T5TokenizerFast  
model1b = T5ForConditionalGeneration.from_pretrained("iarfmoose/t5-base-question-generator",cache_dir="/root/Desktop/model_cache_tmp1/")
model1b.eval()
model1b.half()
model1b.to("cuda")  

[Oxi84]

Good thing is that when i load base and large model together it takes 2.7Gb of memory, while only base is 1.3Gb and only large is 2.3GB. Also adding additional base models seem to take just 200 MB. I suppose there is some kind of general cache you have implemented for all T5 models together.

from transformers import T5ForConditionalGeneration,T5Tokenizer,T5TokenizerFast  
model1ba = T5ForConditionalGeneration.from_pretrained("t5-base",cache_dir="/root/Desktop/model_cache_tmp1/")
model1ba.eval()
model1ba.half()
model1ba.to("cuda")  


import torch
from transformers import T5ForConditionalGeneration,T5Tokenizer,T5TokenizerFast  
model1b = T5ForConditionalGeneration.from_pretrained("t5-large",cache_dir="/root/Desktop/model_cache_tmp1/")
model1b.eval()
model1b.half()
model1b.to("cuda")  
input(11)

[patrickvonplaten]

Hey @Oxi84,

Note that some GPU memory is always taken up by PyTorch itself. See thread here: #16881 (comment)

[Oxi84]

Thanks.
It is awesome idea to merge all the models on one flask file/server, this saved me 1GB of memory.
i use gc collect and cuda empty cache to remove things from memory after each interface and a certain number of batches if not after each..

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.