The thing is I have a large data inflow that needs to go through a seires of models for processing first being a classifier then depending on the output of that It will need to be segmented then go through some encoder which will then rejoin all outputs from the previous step after the classifier and go through another encoder. I need to batch as much as possible as gpus are more efficient that way. I am rn running a bunch of engines at the same time that all have their own processing queue and they pool the data until they get to some specified batch size then process and push their outputs to the other engines. This is probably very memory inefficient. Also this way I need to load all the models into vram at once which makes it only runnable on special machines with special gpus. it also restricts batch sizes. like encoder models vram usage typically increase linearly with the input batch size but stuff like segmentation or anything else that uses convolution can be exponential or anything in between. The worst thing is that execution time is important for this case
See this is why you need a help thread.