My age precludes me from having seen the worst football has to offer in terms of violence and hooliganism, so it's rare that I felt threatened at a football match.
Two games spring to mind - Went to see Liverpool play Leeds at Elland road about ten years ago (think we won 1-0?) and some of the behaviour of the Leeds fans outside the ground and in the town intimidated me, bearing in mind I was 9 at the time.
Went to see Stoke play at the Britannia against Cardiff around 00/01 or so when they were both in Division 2/League One, or the old third division. Was a Sunday game if I recall correctly, and we were seated in the Boothen end right next to the away fans, who were ripping up seats and chucking debris over at us. It didn't help that the Stoke fans then decided to respond in kind, and that the police had riled up the Cardiff support with a late escort from the station and then hording them all through one part of the ground. Then outside, there was a lot of fighting going on and the like.
I wouldn't say these were scary grounds, and the pale in comparison to some of the stories in this thread, but they remain the only times I've feared for my safety at a football match, although that possibly has as much to do with my young age during these encounters.