Three come to mind:
The last boss of Dragon Quest Monsters: Joker for the DS. After grinding endlessly just to get there, the boss was such a huge difficultly spike that it required leveling up to max level just to win. As far as I recall (it's been a long time since I played that game), it's one of those games where leveling up your monsters is disgustingly long, because the game expects you to constantly trade your monsters up for better monsters (a lot like the SMT games--especially Persona), and nothing was anywhere near the last boss's level, which made the grind that much more painful. I threw my hands up and said, "Forget it" upon realizing the commitment.
Giygas got me the first time I played through Earthbound. I had no idea I had to use THAT command to win. It's almost completely worthless the entire game. I spent 30-40 minutes just fighting Giygas without making any progress. I finally accepted that beating him was too much of an effort if 30-40 minutes of fighting wasn't enough, then I put the game down. Some time later, I stumbled onto the solution, went out and rebought the game and finally saw the end credits roll. Needless to say, I was pretty irked to realize it was something so simple.
Not necessarily the last boss, but SMT: Strange Journey got me to surrender just by being (or feeling) incredibly long. I got to what I thought was the last dungeon, grinded my butt off to be strong enough for the "final fight," only to find out there was more to the game. I was already ready to roll the credits well before that fight, but it just didn't know when to say enough. Etrian Odyssey V did the same thing to me, and I loved that series to death. Knowing when to wrap things up in a game is just as important as knowing how to make a game long enough to keep it satisfying.
