Unlike Parks & Rec, there isn't an obvious moment in the early seasons where they shift gears entirely and you see the quality jump almost immediately. Even adding Frank to the main cast wasn't a huge change, really.
That said, I found the early seasons a bit strange to watch - maybe it's the early 00's being in a liminal era of not yet feeling 'retro' but also being long enough ago not feel modern? Perhaps it's just the 4:3, non-HD filming?
Since then, it could well just be increased budgets, but as it goes on they do go to more places and get a lot more ambitious in what they're trying to do in an episode. So the variety increases and I've found it more enjoyable. A few characters start popping up in multiple episodes to flesh out the local area (beyond just the Waitress and the McPoyles).
Also, I wouldn't stop watching without the Season 4 finalé - The Nightman Cometh, and not just because other people need to have that song stuck in their head for days too.