Emulators and AI

[jul]

Does anyone know where "we" are at when it comes to making an AI play emulated games? Is there some ground breaking, (relatively) easy to install, already trained or specifically designed stuff that works with mame or retroarch or any emulator?

What I have in mind is an OS dedicated to games (DIY arcade machine typically) that would launch random games and play them, like you get to watch AIvsAI on street fighter or king of fighter games or playing solo on any game, and then after a couple of minutes it quits and launches another random game and plays it, and so on, and it saves on quit so next time the game is launched again, it automatically loads the save state and continues to play where it was last time the AI played it. Sort of infinite loop of random video game cinema.

So?

Is there an existing/available AI project for MAME or retroarch that works well already?

 

[Xdqwerty]

Why don't you wanna play the game yourself?

 

[jul]

Why don't you wanna play the game yourself?

Because

 

[Xdqwerty]

Because

 

[Lainon]

Not sure what you mean specifically by AI? If you mean the an LLM, then sure perhaps having a multi-modal model that can view the current frame and be able to make inputs via an MCP or something would work, a la "Claude Plays Pokémon". That's the only way you could "do it" (roughly speaking) for what you're getting at, which is playing a lot of games just generally so you can have what you're aiming for.

Normally, something like this requires ML experience in which you need to have video of someone playing the game, categorize what items are etc. and then train your DL model, and have it learn from playing. An "OS" dedicated to this wouldn't change anything, it would just be some software you'd write that would interact with MAME or retroarch. There's plenty of writing on this online if you'd want to see how others are trying to do a similar thing.

 

[jul]

There's plenty of writing on this online if you'd want to see how others are trying to do a similar thing.

Well, precisely because there's plenty, going directly for the best and avoiding the rest, including reinventing the wheel, would be much appreciated. But maybe "we" aren't there yet and nobody made something worth the try.

a Python toolkit called MAME Toolkit has been developed to facilitate this training process, allowing AI algorithms to interact with the games by sending actions, receiving frame data, and tracking internal memory states of the games.

 

[magalee]

Does anyone know where "we" are at when it comes to making an AI play emulated games? Is there some ground breaking, (relatively) easy to install, already trained or specifically designed stuff that works with mame or retroarch or any emulator?

What I have in mind is an OS dedicated to games (DIY arcade machine typically) that would launch random games and play them, like you get to watch AIvsAI on street fighter or king of fighter games or playing solo on any game, and then after a couple of minutes it quits and launches another random game and plays it, and so on, and it saves on quit so next time the game is launched again, it automatically loads the save state and continues to play where it was last time the AI played it. Sort of infinite loop of random video game cinema.

So?

Is there an existing/available AI project for MAME or retroarch that works well already?

and for what tho..........

 

[Cawl canallia]

I wanted to see this AI was going to play Ace Attorney

 

[_oBSOLEte_]

IIRC There were a few video gameplays around with Google DeepMind playing Starcraft 2. Strangely enough, it wasn't something particularly enjoyable to watch.

Hyper-optimization of gameplay often looks far different than what the devs and players think "normal". I think about those tool assisted speedrun that wouldn't even make sense without pausing and having the commentary of some guy explaining the exploited glitches.

 

[jul]

and for what tho..........

For "infinite loop of random video game cinema.". When I don't play, I would like to watch something different from the default demo/attract mode, what's the problem with that?

 

[Duke_11037]

For "infinite loop of random video game cinema.". When I don't play, I would like to watch something different from the default demo/attract mode, what's the problem with that?

Couldn't you just watch YouTube or something?? There's plenty of places where you can find gameplay without commentary??

 

[jul]

Couldn't you just watch YouTube or something?? There's plenty of places where you can find gameplay without commentary??

No, that's not the same thing, at all, oranges and apples. I want the arcade machine to play vs itself while I don't play on it, I don't want to go on youtube with a web browser or even an app to watch videos of video games, with the youtube UI all over the place, the ads and such, or even having a video player playing video game videos, that's not what I want.

I want a fucken awesome multi system attract mode/demonstration mode.

 

[Evaccaneer Doom]

No, that's not the same thing, at all, oranges and apples. I want the arcade machine to play vs itself while I don't play on it, I don't want to go on youtube with a web browser or even an app to watch videos of video games, with the youtube UI all over the place, the ads and such, or even having a video player playing video game videos, that's not what I want.

I want a fucken awesome multi system attract mode/demonstration mode.

So you basically want pause menu games? Alright, that makes way more sense.
Unfortunately, that's an incredibly niche idea. I don't think people would try that even if AI was up to date.

 

[jul]

So you basically want pause menu games? Alright, that makes way more sense.
Unfortunately, that's an incredibly niche idea. I don't think people would try that even if AI was up to date.

"pause menu games" ? No, stop imagining stuffs. I want the arcade machine to play on its own, that's it. There's no pause menu story.

 

[_oBSOLEte_]

Depending on the randomness of the games loaded, I guess you could use prerecorded input files with Retroarch,

Basic guide to input recording with RetroArch (.bsv files) - shmups.system11.org

Either load other people's files and/or feed them to a LLM that would create new ones.

 

[Evaccaneer Doom]

"pause menu games" ? No, stop imagining stuffs.

That's not imagination. It's a real thing.
The ability to play games while paused was a feature that was patented a while ago and expired in 2015.
It's the closest thing to what you're saying

(Correction, it was called the "Loading Screen mini game," whoops)

 

[jul]

Depending on the randomness of the games loaded, I guess you could use prerecorded input files with Retroarch,

Basic guide to input recording with RetroArch (.bsv files) - shmups.system11.org

Either load other people's files and/or feed them to a LLM that would create new ones.

Indeed the record files would be a nice place to start, and it's not like I want an unbeatable AI, it just needs to be good enough to navigate itself through the game and improvise the shootings and the beatings, that's all. If it can work relatively well for most retro shooters and fighting games that's already that. I mean it's not as if there was a world of difference between ryu in the street fighter series and terry in fatal fury/kof etc in terms of gameplay. Same for most shooters, you have to insert a couple of coins, press start/A repeatedly until you reach the action and then let's roll. For RPGs it's an entire different story but for action games... There's probably a way.

 

[_oBSOLEte_]

Indeed the record files would be a nice place to start, and it's not like I want an unbeatable AI, it just needs to be good enough to navigate itself through the game and improvise the shootings and the beatings, that's all. If it can work relatively well for most retro shooters and fighting games that's already that. I mean it's as if there was a world of difference between ryu in the street fighter series and terry in fatal fury/kof etc in terms of gameplay. Same for most shooters, you have to insert a couple of coins, press start/A repeatedly until you reach the action and then let's roll. For RPGs it's an entire different story but for action games... There's probably a way.

I assume most of the randomness of the games may be reduced, as the file first load a save state then stream the input. Never really tested it, tho. I tried recording video files with Retroarch but it was easier to use ffmpeg via OBS, I'm still learning the ropes.

You might try your luck on the libretro forum. It's a process to get accepted on the board, they got their antibot tutorial who will run for a good 10 mins before letting you do anything. But once able to post you get an answer really fast.

 

[jul]

That's not imagination. It's a real thing.
The ability to play games while paused was a feature that was patented a while ago and expired in 2015.
It's the closest thing to what you're saying

(Correction, it was called the "Loading Screen mini game," whoops)

"The ability to play games while paused" way too complicated to implement, if it requires to modify the pause state of the original game, that's definitely not the closest thing to what I want.

 

[popckorn]

I once read there are Translation AI plugins in some emulators, but I have never dug into it.
Also read once Emulators might have AI texture upscaling plugins soon, as well, which could create interesting experiences.

 

[_oBSOLEte_]

I once read there are Translation AI plugins in some emulators, but I have never dug into it.
Also read once Emulators might have AI texture upscaling plugins soon, as well, which could create interesting experiences.
View attachment 56669

Wow, it looks like a Nightdive remaster!

 

[Lainon]

I once read there are Translation AI plugins in some emulators, but I have never dug into it.

It's available in RetroArch, since 2019 at least. Looks like it uses ZTranslate to work using ML to recognize characters and translate on the fly.

Also read once Emulators might have AI texture upscaling plugins soon, as well, which could create interesting experiences.

For textures you've been able to do this for a while using ESRGAN, or more recently diffusion models, however those aren't in realtime because to do so would take a lot of GPU power while the game is running and would vary based on game engine, system, and so on. Then there's RTX Remix which currently allows you (for select games right now on PC) to replace textures and objects in games in realtime by decompiling the games.

You can also do this for Resolution using FSR and DLSS implementation. It's actually not too complicated to get it working on engines that don't support those libraries currently.

 

[OpenVMS]

Does anyone know where "we" are at when it comes to making an AI play emulated games? Is there some ground breaking, (relatively) easy to install, already trained or specifically designed stuff that works with mame or retroarch or any emulator?

Google has made some strides with DeepMind:

Google AI learns to play open-world video games by watching them

An artificial intelligence from Google DeepMind can play nine open-world video games like a human, by watching video from a screen and controlling a keyboard and mouse

www.newscientist.com

This stuff is still firmly in the lab. As far as your want of some kind of cabinet that fires up, figures out how to play games and then maybe even does pretty good at them, we are far, far, far off. I'm talking decades at least.

AI has been big in the news lately, but it's good only at certain narrow things. It would take more of a generalized intelligence to be able to recognize the content in real time from the screen, figure out what it's looking at and send back game inputs quick enough to matter.

Cool product idea though, let me know if someone proves me wrong and shows up with this sooner than a few decades.

 

[jul]

but it's good only at certain narrow things.

I have 2D arcade/action games in mind, platformers, shooters and fighting games, nothing involving any understanding of any plot whatsoever.

3 examples

Those 3 archetypes would cover the needs for something around 70-80% of the entire collection I think

 

[OpenVMS]

I have 2D arcade/action games in mind, platformers, shooters and fighting games, nothing involving any understanding of any plot whatsoever.

Ah, that actually helps quite a bit! In theory, one could develop a genetic algorithm that say, takes screenshots of Galaga and learns which inputs results in the score moving up... Indeed, there's actually been a bit of work done in this area:

GitHub - Farama-Foundation/Arcade-Learning-Environment: The Arcade Learning Environment (ALE) -- a platform for AI research.

The Arcade Learning Environment (ALE) -- a platform for AI research. - Farama-Foundation/Arcade-Learning-Environment

github.com

So, I'll have to revise my estimate downwards a bit, but as of yet, the best hope you have is the DeepMind project above, and I'm not sure they make a "home edition" yet.