They have Wii's in Senior Centers in America for a reason. Specifically the Wii actually has been found to help with Alzheimer's and Dementia onset in the elderly, because it gives them a way to activate their brain in a fun, exciting way.
Certain games that ask the most out of their players, like Ninja Gaiden or Dark Souls, increase hand eye coordination, and can help increase your reflexes.
There's even educational games, like Brain Age, that have actually helped people do better on tests and such, because they are learning topics like Math with muscle memory, which my 4th grade teacher actually had me doing at the time.
Most people who say this are people who lived through the scares that news outlets put the public through because Video Games were at the time a new and misunderstood thing, it's kinda like the 90's with the Satanic Panic, where parents thought almost everything, mostly Heavy Metal Music, would make their kids follow Satan, which was bull.
Don't worry too much about all these labels that gets put on everything. Just enjoy what you enjoy and don't let naysayers ruin your mood.