The concern is absolutely valid. For the purposes of practicality, this is a huge issue. You're probably going to need dozens, if not hundreds, of staff. It will probably have to be in ISED because it has to be coordinated across government.
I would highly recommend having an independent parliamentary office whose goal it is to oversee ISED to make sure it is not misusing the power.