Hello Chansen,
It is difficult to draw conclusions about the owner of a sword by looking at the koshirae. The crest is used by many families (the site below lists no less than 10 families associated with crest, and it isn't an exhaustive list). The site also says the crest is known in several prefectures, but none of them are in Kyushu, where Bungo was located.
https://irohakamon.com/kamon/kashiwa/marunichuukashiwa.html
Another reason it is difficult to draw conclusions is because the parts on a koshirae are meant to be replaced as they are worn out, so its possible that some of the fittings on your koshirae have been replaced in the last 150 years or so by its various owners. Shopkeepers are also known to switch out the components, as some items are easier to sell individually. So a shop owner may break up the components of the koshirae and swap them out for other components. Sad, but true.
Anyway, the family crest is nice to have on a saya. The habaki also looks like it was something of a "special order" habaki due to the patterning on it.