Good thread topic.
This question is actually pretty hard to answer. I'm pretty big into collecting so I own a lot of consoles/handhelds and surprisingly some of the unsuccessful ones are pretty fun. I guess if I had to limit my selection to one, I would say the Virtual Boy. Sure it was a failure, and sure the library was very small, but it is full of character and there are a few very good games in its library despite there being hardly any games for it at all. In particular, I feel like Mario Clash is extremely underrated - It's like a sequel to arcade Mario Bros, and it's so much fun to play. I have never particularly felt any attachment to arcade Mario Bros, but I was so hooked on Mario Clash that I couldn't stop playing until I had cleared every stage (And there are quite a few). It's a great game. Red Alarm is another great game, it's very similar to Star Fox and is very neat to look at. Unfortunately this game gets shit on by people who have tried it in an emulator and have never touched the actual hardware, and that's because without the stereoscopic 3d effect depth becomes completely imperceptible which makes this game completely unplayable. But if you have the chance to play in 3D it is fantastic. Honestly i could go on and on, but I've already written a lot so I will leave it there. In short, I love my Virtual Boy. I almost have the complete library for my region, I'm only missing 3 games and I hope that I have the chance to add them to my collection some day but those 3 games are very rare and expensive, and they're not particularly great, so perhaps my library will never be complete.
As for other devices, I have gotten a lot of enjoyment out of the Wonder Swan Color, Neo Geo Pocket Color, and Sega 32X.
Wonder Swan was definitely a unique concept, I do feel that the Game Boy was much better but I love my Wonder Swan. Unfortunately these devices didn't age very well and it is very hard to see the screens now. My Wonder Swan Color is modded with the front light of a Game Boy Advance SP and unfortunately that makes it basically impossible to see the screen unless I'm sitting in complete darkness (and even then, it's still hard to see lmao). I would like to do a mod to install a modern screen but unfortunately they're a little more than I'm looking to spend right now.
The Neo Geo Pocket Color is very neat, and it has aged very well. The screen is very clear and easy to see, the ergonomics aren't bad even for adult hands, and I love the joystick! I bought the NGPC specifically to play Sonic Pocket Adventure and it was worth it, I had a lot of fun with that game and trying to collect all of the photo shards was a nice challenge. I later picked up some other games that were also enjoyable, but unfortunately the system is expensive to collect for these days. I wish I could get more games for cheaper because it's a great system.
And as for the 32X, I bought it specifically to play Chaotix (Which I do not regret. The game isn't great by any means but it's not nearly as bad as it's reputation. It's a decent game but suffers from poor level design and a few odd decisions), but there are some great arcade ports available which might be the best home versions of those games. I have After Burner, Space Harrier, and Virtua Racing and those are my favorite versions of those games. I would like to pick up a few more games if I get the chance, but I would be satisfied even if my 32X collection never grows.
Someone else mentioned the Dreamcast. I often forget that this system was unsuccessful, so I would have to say my favorite is a close matchup between the Dreamcast and the Virtual Boy. The Dreamcast is the better device overall, the library is much better and I love the VMU, but damn do I love my Virtual Boy. It's a hard choice, and part of that is because they're so different. Dreamcast is Naomi at home with some big console adventures thrown into the mix, and Virtual Boy is a Game Boy on steroids that causes chronic back pain. I can't chose between those two.