The PSP is forever my goat. First time you really felt like you had a console in the palm of your hands, not to mention how ahead of its time it was with an online storefront, skype chat, texting, internet access and more. It was the perfect precursor to smart phones but with greater capabilities as a video game machine. I was fortunate to have several friends with a PSP who played various games I did so unlike with the Gameboy or DS lines I got to properly experience local and online multiplayer on a handheld!
Then there's the nostalgia with modding it. It's the first system most people ever modded and its wealth of homebrew opened my eyes to so much. It was the first time I ever emulated anything, first game I ever played homebrew games on, it felt like the ultimate piece of hardware for its era because it 100% was. I have so much love for the 3DS and Vita but the PSP was a once in a lifetime hardware event. It also helps that while console gaming was on a sort of decline in terms of quality the DS and PSP were still maintaining a constant stream of stellar software.
There will never be a system like the PSP again because it released at the perfect time and place in history to accomplish everything that it did. I truly feel sorry for people who didn't get to experience the PSP in its prime, there truly was nothing like it.