Actually the movie (where you got the bit of information from) isn't canon with the Silver's powers. In fact, Norrin Rad tried seeing if he could live without the board (with Reed Richards help) and not only did he not need it, he found out he could leave it on planets as a sort of beacon for other heroes. Also, destroying the board was already thought of in the comics, he simply reconstructed it from thin air. You can't take his board, you can't break it, and its not the source of his power. The power cosmic is ethereal when it comes to those who obtain it, and by that I mean, those with the power cosmic don't recharge, or lose strength, the power cosmic can only be removed from them by entity that bestowed it upon them (Galactus)and even then....Galactus couldn't take back the power from Norrin...
In conclusion without the board, Goku gets that work, Vegeta is an after thought as he's the weaker of the two. Keep in mind that the power cosmic makes SS intangible and immortal, he can also steal power from entities like he did with the Hulk when he turned him back into Banner by zapping him of all Gamma radiation. Goku requires energy....game over man.