They aren't so much of a hot mess, like say Superman 64, and this was the birth of 3D console gaming. So I give these two Castlevanias leeway for that.
As an aside, Mario came out roughly three years before, and 3D was still in its infancy. A game like Castlevania at that time didn't adapt well to 3D. It does now, as best as I can tell, though I haven't played any new 3D Castlevania games since Lament of Innocence on the PS2.
And it wasn't a bad game. Not great by any stretch of the word, but it controlled fine and was a fairly good entry in the series. So no, I don't think the N64 Castlevanias are awful. For the time, considering the hardware and the fact that it was new territory,
I think they're perfectly fine games, again, for what they are.