I think that there will be differences between sports, which is why a one size fits all approach doesn't work.
Similar to you I'm not an expert. But I would guess that the primary enduring physical characteristic post transition would be skeletal structure. That would be a reason why there might be more of a problem with trans people taking part in heavy contact sports. If you look at the contrast between male and female skulls you can intuitively see the risks that might come from head collisions.
With athletics I'd imagine that a heavier skeletal structure would actually be a disadvantage that might mitigate against other advantages that there might be in terms of residual muscular performance.
The thing about competitive sport is that all sorts of different things give different people an advantage. For instance, if you're a goal keeper, it obviously helps to be tall and agile. In some sports, upper body strength is vital, in others, lung capacity is key, or stamina, or strength. We don't consider it unfair that one team has a taller 'keeper than another. Michael Phelps, one of the greatest swimmers ever, has double jointed ankles which give him a better kick, and produces half the lactic acid, which causes fatigue, of other people. We don't try to level the playing field there by forcing him to have surgery on his ankles, or take drugs to increase his lactic acid, we just say how amazing it is that he has these natural quirks that give him the ability to be a great athlete.
There also seems to be a stereotype here that all trans women are 6 foot 6 tall with broad shoulders, hence they won, for instance, a women's contact sport such as rugby. That's not the case at all - and, as I've already mentioned, the taking of oestrogen causes a great deal of muscle wastage, regardless of the type of puberty that someone went through.
I'd be interested to know who the many people saying it would be dangerous to allow trans men to play rugby with cis men are - which I assume is what you're referring JustElmo? - are they experts or are they people who have just decided that based on their own feelings and prejudices without any actual evidence?