I'd say the tsuka is a write-off. To restore it you would need new shark/ray skin, and new ito. I would look into new fuchi/kashira as the ones on this piece look pretty bland. Its probably a JPY 50,000 restore job on the tsuka (excluding any new fittings). The restorers might just throw the whole tsuka away and make a new one from scratch, keeping the boar fittings. You would, however, end up with a nice display piece.
If its a Rai blade, or other decent blade, its most definitely worth getting nice fittings and restoring the saya (I quite like the saya with the pauwlonia crest... I hope that can be restored. I think the exposed wood bits can be repaired and re-lacquered, but it might not match the mottled lacquering of the rest of the saya. Still, I think it can be greatly improved from its current state). If its a junker blade, and you just want to restore it for sentimental reasons, it would be a labor-of-love kind of thing, wherein the cost of repairing the blade and the saya would end up being dearer than buying another antique sword altogether... but it could end up being a very nice family treasure. Maybe not a museum piece, but something to enjoy for a long time.