I LOVE Street Fighter IV, but it was definitely an uphill battle. It took me YEARS to get decent at it.
At first I played Vanilla and Super and it just didn't feel right, most of the time it felt like controlling a sack of bricks, the game felt weirdly heavy for some reason. A few years later, I finally got my hands on Ultra and REALLY put some time into it. It's a game where you can either have a very fun match, or have your scrub of a friend wait for his Ultra meter to go up and beat you with the most random Ultra Combo ever, there's no middle ground with this game.

But even so, I began appreciating SF4 more as time went on. It certainly isn't the most balanced game out there, with characters ranging from annoying (Juri, Ken) to straight up bullshit (Elena).
It's a very execution heavy game if you're going for the 1-frame links, but I just found them to be too much work for too little reward. I have a friend who can put people through 53 hit combos with Seth but it barely does half their health and spends a ton of his resources, so what's the point? xD
I like that SF4 is a game that's easy to learn, but really hard to master. Lots of characters are viable to a point, and while I'm not a fan of the comeback mechanics of this game (and it did kinda ruin the experience early on but that's mostly due to the people I was playing against), I hold this game in high regard. Just like SF2, they didn't land a home run on the first hit, but it got a lot better as time went on.
As for SFV... hoo boy. My friend and I used to refer to it as "the dark ages".

Characters being a menace and staying a menace (Karin, Luke and Cammy), one super per character (at least 4 had a super AND an Ultra), bad characters either getting worse or getting better way too late in the game's life (Nash), some characters just getting some not really interesting revamps (Cody, Poison, that Mufasa lookin' ass Akuma) and some really bad newcomers too (Kolin, Menat).
The basics are here, but everything feels so monotone in SFV, if you see one pro player using a character, you've seen them all. It's such a one-note game at times, and the changes along the way sadly didn't do much to help it. There's no room for creativity in SFV, it feels.
There are things I like about SFV, Nash (ESPECIALLY Nash), Necalli, Urien being back, G, Balrog, Zeku, but for every step forward there's definitely a step back.
And oh yeah, THROW LOOPS.

Looks like Capcom didn't learn from their mistakes and they found a way to return in SF6. Oh dear.