This is the reason I've only beaten Silent Hill 4 once! The first half of the game is not perfect, but SH4 has a lot of really cool things that really shook up the formula at the time. Except the ghosts, they just suck. I don't feel like they are scary or anything, just annoying. The wall men too. Oh, most of the enemies have terrible stock sounds too. Despite those, so far so good. But then the second half of the game hits. I don't mind the infamous escort mission the rest of the game becomes, they are technically invincible (yes, I know why you don't want them to get hit though) and they help you out in fights. Having to wait for them to catch up at doors does suck though. I don't even mind the changes your room goes through and how hostile it becomes. It was done extremely well, making you feel safe for half the game and then it becomes one of your biggest concerns. ?
What I do mind is going through all of the exact same areas again, but with a couple new rooms added. The areas just become a bunch of shitty backtracking for new puzzles or just some new shitty parts in general (building world elevators). Repopulated monsters and more of them, ever present ghosts and the new super ghost for each area. Yes, Sword of Obedience solves the super ghost problem but the rest are still there to be irritating. There's also the villain that starts following you. As if ghosts weren't enough. Your escort's bloodthirsty AI has the issue of wailing on ghosts when you just want to run away. There's probably more I'm forgetting, but I feel like that's enough. x)
It's such a shame too because 4 has a really interesting story and backstory. Lots of cool ideas and the spiral staircase is one of my favorite parts in the series. Henry is a fart in the wind but at least you can learn a tiny bit about him by examining objects in your apartment. The Walter retconning from the... one newspaper article in 2 was a bit strange as well. What can you do? Still worth at least a playthough, it's a heavily flawed gem.