Interesting, his character in the series doesn't seem that level gravitas, like he's some average sailor.
You don't get opportunity after opportunity to talk about his history. There is one event in the book which proved his gravitas, and it's also in the show, albeit with less capacity for explaining the reasoning. That's when Rodrigues is washed overboard and Blackthorne takes over and takes the ship to safety, when all the Japanese aboard were resigned to the ship being lost with all hands. Rodrigues later admits he'd have headed into the open sea and probably lost the ship, whereas Blackthorne, after taking over, went against Rodrigues's orders and aimed for and achieved the bay. After that, Yabu and Hiromatsu recognised Blackthorne's worth as a valuable figure in his own right.
One of the most underrated figures in the story is the priest in the prison, who was the reason why Toranaga sent Blackthorne there. Toranaga recognised Blackthorne's hunger for survival and his opposition to the Christian establishment in Japan, and sent Blackthorne to the prison with the implied threat of execution but with the opportunity to sweat information out of the priest, whom Toranaga probably viewed as the more valuable of the two. After the Catholics tried to murder Blackthorne in prison, Toranaga brought him out in his usual fashion, and later tried to bring the priest out as well. Unfortunately, the latter mistook his exit as the long awaited order for execution, and died from a heart attack, which left Blackthorne as Toranaga's only anti-Jesuit western asset on scene.