I think a group of nine would benefit from a few on each end of the relative spectrum and a solid group in the core. The justices seem to have migrated slowly to where we are - polarized to some extent.
I would say that it started earlier with the judges than with the voters. In fact the judges' polarization is one of the reasons why the voters are polarized.