The easy answer is always the Dreamcast since it had such a short lifespan and a slew of cancelled games like Half-Life and GTA3. While there is no doubt that it never would have competed with even the PS2 due to its paltry RAM count and inferior controller, it is undeniable that it definitely had more room to show off what it could really do. Shenmue 2 and Dead Or Alive 2 did show what is probably nearing that system's peak though. That alpha build of GTA3 is mighty impressive though and I'm curious just how functional and fully featured it can be if the developers go all the way with it.
I see people say Jaguar a lot when this topic comes up but I also always hear conflicting reports on what the hell the hardware was even capable of in terms of the Tom and Jerry architecture. Considering the homebrew scene that exists has failed to really do a ton of impressive things I'm more inclined to believe that it always was a middling mess of a machine. Battlesphere and the Doom port show that it wasn't a totally hopeless machine though.
3DO is another system that I do think was pushed about as far as it could have gone, but I still would have liked to see what a potential Need for Speed sequel or a Doom port that had actual time to get finished would have been like.
When the Wii U was first announced, there was this golf game they showed in the reveal trailer that blew my mind. You'd see the course on your TV, set your gamepad on the floor to show a ball on a tee, then swing your Wii remote like a golf club perpendicular to the gamepad to hit the ball "into" the TV. I nearly went insane – think of the possibilities! It was like a holodeck, and it could theoretically depict any environment for you to interact with!
God, even ports of some other games to the WiiU got such cool odds and ends that it made me wonder how much further certain concepts could be pushed with a second screen. A lot of it was nothing that we hadn't already seen on the DS, but having stuff like hacking minigames in Deus Ex: Human Revolution appear on a second touchscreen was novel. Even ZombiU's inventory being real-time and having the motion/feel of rummaging through a backpack was adorable and it even had that asymmetrical multiplayer mode where the gamepad player played a zombie RTS while trying to thwart a controller player who just played FPS.
Nintendoland is still one of the most enjoyable party games of all time and it only scratched the surface of what you could do with multiplayer on that kind of system. Mario Party 10 really felt like it did so little to take advantage of the gamepad for minigames, though it has been a while and I didn't see every minigame so maybe that's me being harsh. They just messed up the core board game too much for anyone to care about the minigames.