Calorie amounts are stressed far too much by a lot of diet programs. The amount of calories you take in or expend is indeed important, but calorie source rather than just amount is equally important. There's a lot of foods that are calorie dense but still good for you, so don't go out of your way to avoid stuff like peanut butter. Macronutrients are important. Fat, Protein and Carbohydrates. It's arguably more effective to keep track of these groups within a diet plan that it is to track specific caloric amounts.
Assuming that people eat a fairly varied diet (nobody's eating the same thing every day, or completely avoiding all meat/fruit/veg/fat/carbs completely - and anything that tells you it's healthy to overload on proteins or to cut out all fats or all carbs is therefore necessarily dangerous) then it all depends on your motivation/goal. If that goal is to lose weight, as it is with 99% of people in this thread, then how many calories you consume is by far and away more important than the groups you're eating.
You could in theory hit your daily calorie targets through eating biscuits or cucumber alone, but I don't think that anyone would suggest either is healthy. I think an assumption is made that people are eating a 'normal' diet when talking about calories, including fats, carbs, protein and a reasonable intake of fruit/veg.
If you're training or carrying out any regular form of exercise, or with any particular goals (long-distance running, weight-lifting, bulking-up muscles or targeting a specific sport) then that's when 'diet' comes into it more than calories, as certain food groups may be required for muscle build-up or repair, for instance, or energy loading before events.
