How can one profile inference to know which layers/experts got activated during inference?

[segmond]

My idea is to gather enough data, then based on the most frequently utilized layers/experts, load those on the GPU. My workload is mostly programming, so I figure all layers/experts won't be frequently activated. If I can narrow down the ones that are more likely to be used for programming, I can have those loaded in GPU first. Doing this might yield some gains. Anyone know how to go about this?