There's actually an official answer to this question. I remember an old interview with Miyamoto during the Wii era where he said that typically new Mario games are made because of advances in hardware and this was sort of a soft rule with the series. Super Mario Bros was for the Famicom, and Mario 2 was because of the disk system which opened up more possibilities. Mario 3 was because of how advanced the cartridges had become thanks to mappers, which allowed them to make a much bigger game. World was for SNES, and 64 for 64. Sunshine for the Gamecube, and Galaxy for the Wii.
I'm not sure if this soft rule is still in effect or not (maybe it stopped with Galaxy 2 and that was why he even mentioned it?), but based on that it would seem that the reason we never got a proper followup on the 64 is because there weren't any big breakthroughs in the tech that warranted a sequel like what had happened on Famicom. Perhaps if the 64DD had been successful than we might have seen a new Mario game for that.