Carell leaving might have been fine if it wasn't for Ed Helms filling the void, he becomes more insufferable with every re-watch.
I'd also recommend watching the blooper compilations on YouTube if you haven't, probably funnier than the show itself.
And that is why Fitzy is the true GOAT.