Yeah, everyone has pretty much said Brand's lawyers were very procative with this kind of thing. The other part people need to keep in mind is that the victims almost certainly didn't know each other and would have probably worked in a similar industry or proximity to him, which is why they met him in the first place.
Basically, you can't underestimate how difficult it is for a victim to come forward in those circumstances. Maybe it was a one time thing and not a pattern of behaviour as far as they know, maybe everyone was drunk/high and the instance it gets to that point his lawyers paint you as a wreckhead who didn't know what was going on. That leads to no one in your industry wanting to hire you anymore - if his lawyers don't follow through on the threat to sue you into destitution.
It may not be by design but it is how the entertainment industry works in practice, and not only entertainment. Look how many MPs are 'under investigation' right now for cases we've heard absolutely nothing about.