I fundamentally believe the public has a right to know who owns what, in terms of companies, land/property and other major assets. Indeed, the more people try to hide what they own, the more it is in the public interest to make that information public knowledge.
I agree that the public has a right to know who owns what and I doubt that regulations will be walked back in terms of knowing the actual beneficial owners and authorities/organisations/people having access to that data. The ECJ has made it pretty clear in my view, that while the Charter is very important, not all the Articles apply all the time, if there's a higher good or public interest to protect. So, no I don't think there's a way for companies or directors going up and being successful against declaring who the beneficial owner is.
And I can also kind of see why the ECJ might think the way the regulations were written and executed was an issue. Again, it's clear that it should be public knowledge who owns what, but having a publicly available online plattform everyone can search willy-nilly whenever they want with no real limit in terms of what data is put on there (including private addresses) does seem to be going a bit too far.
I think if the EU want to create transparency that way, they need to make clear rules what kind of data can be shown to people, i.e. name, Date of Birth and probably the current state/city of residence (not the exact address). That way it's clear who a person is as there's very little chance of people sharing the same name, date of birth and residence. I'd probably also include some kind of registration process to make sure it is known who's looking at the data and to maybe stop the mass collection of data by bots.
Another issue that also needs to be considered is that not everyone who has to register as a beneficial owner is a criminal tax dodger and money launderer or some rich billionaire from Russia trying to hide their fortune. On the contrary, most of the people will be regular business people who haven't broken any laws and are doing what's asked of them. I can kind of see why the ECJ thought making all their personal data public (again, possibly including stuff like their home address) is going a bit too far, when the reasoning by the EU was that they simply wanted to create an atmosphere of transparency as a deterrence for possible money launderers by making the data public.