In a real circumstance, we may have multiple nodes with varied performance of GPUs. Let's say we have
Nvidia Tesla V100(high),
Tesla M60(Medium) and
Tesla K80(Low) and we know there is a project using the instance type (cancer_prediction) which requires GPU Tesla M60 and above. In this case, we can taint nodes which have GPU Tesla K80.
kubectl taint nodes node1 gpu=low:NoSchedule
Then a project spawned from cancer_prediction is only scheduled on nodes which have Tesla V100(High) or Tesla M60(Medium).
Meanwhile, we have a project using the instance type (age_of_consumers) demanding much less GPU performance, in other words, it can accept/tolerate GPU of low performance, of course, it is much better if GPU of higher performance is available. In this case, we add a Toleration by editing Instance Type on Admin UI.
So the project is able to be scheduled not only on nodes with GPU(high/Medium) but also on nodes with GPU(low).