I started with these retrogaming handhelds way back with the Dingoo A320. That was the first time I saw something that was more than a cheap Famiclone or "PMP MP4 Handheld Game". You can put your own roms and emulators on it, and you can actually save your games. I was amazed just because of those basic features.
Around the time the pandemic hit, more of these handhelds started popping up. Devices like the RG 350M were running OpenDingux, which got my attention. I bought one, then two, then five, and now I have a small collection (mainly Anbernic ones).
I stuck with the budget handhelds though (around $80 or below). More recently, popular options would be anything from Anbernic's RG XX series of handhelds or the TrimUI Brick. Even within this tier, you can expect good performance up to PS1. Things get dicey with N64, PSP, DC and beyond. With custom firmware from an amazing community, these devices are awesome at retro emulation.