I've been using my Vita for retro gaming for years now, but recently got into a few dedicated handhelds over the last year.
I own the Miyoo Mini Plus, Miyoo A30, and the RGB30. I have also used the R36S, and returned a RG35XX H due to severe software issues at launch (to my understanding, this has been 100% fixed). Every one of them has some setbacks, there is not yet any perfect system. Note for ANY of these systems, custom firmware is a MUST
MM+ (30-35 USD) is what I use the most by a longshot. It fits nicely in the hands and pockets, plays enough systems, and is by far the most user friendly. Largest problem is while on it always emits a high pitched noise that doesn't seem to be fixable barring unplugging the speaker. I also had the charger port break off, and had to solder it back on. If you want a smaller size, I recommend the Miyoo Mini 4 instead, you just won't have wi-fi or bluetooth
MA30 (25-30 USD) is the newest and smallest, I wanted it for its extremely portable nature. It just a few days ago got a CFW software patch that fixed most problems, but not nearly all of them. Largest problems with the system would be its short battery life, it can overheat easily, and buttons are bound to keyboard functions, which can cause problems. I also still can't get the wi-fi to work. It worked fine when I originally bought it, then after one reboot, it hasn't worked since. It is also worth noting the system doesn't have a headphone jack, but you can use the USBC port
RGB30 (70-85 USD) is pretty neat for its 1:1 screen, and is powerful enough to play more games than most at its price, like Stardew Valley, Super Meat Boy, and Dreamcast. It has great battery life, is surprisingly pocketable (same height as the Vita) and nearly every basic feature you could want. Like nearly all of these devices with analogue sticks, they are useless for precision, as they love to snap to cardinal directions (like a D-pad). I find the system to be more difficult to hold for long periods than it has any business being. And the largest problem by far is that after a 2 foot drop onto a soft surface, the screen popped completely out. It works fine, but isn't actually designed to stay in place (I think I need to use some glue, which I really don't want to resort to)
R36S (30-35 USD) I bought for a kid friend (8yo), but I have spent some time with it. The shell and controls are the cheapest of the bunch by far, and the firmware sucks, but the system is weirdly powerful, even able to play low power PSP games. My largest problems would be that it has very loud buttons, and the triggers stick out way too far, making it not super pocketable
At the end of the day, I would recommend most people wither get a Miyoo Mini, or an RGXX system (any of the newer ones would do, they all look quite nice), or just stick to your phone and buy a slim controller to bring around. For a nice controller I would recommend the 8BitDo SN30 Pro (30-35 USD). For a super slim and cheap option I have a "ShanWan lightweight mini" on its way (6 USD), hopefully it will do nicely enough
I long for the day I can have a dedicated system that will scale GBA games well and allow for multiplayer with the system. I long to netbattle again
.....Maybe I should make a thread for dedicated reviews on my experience with emulation system options at some point