I’d have to agree I do indeed find them stressful and not casual at all haha. 
I'm not fan of the ones that are visually too heavy personally.Hitting a note or its equivalent is pretty hard when you have a few milliseconds to do it and are being distracted by the visuals (since most of them have flsh or wild visuals too).
I guess there's two sides to this coin.I find rhythm games to be both casual AND social, though.
Half the fun of playing something like Guitar Hero, Boogie! or Dance Dance Revolution is having a grand time looking like an uncoordinated jackass in front of your friends ;D!
Wait... he has smashed the wall before?The Classic