Accidents happen in the evolution of species. Random mutations are the seeds of evolutionary change. But the accidents don't stick unless they serve some useful purpose.
The evolution of societies is not identical to the evolution of species. But the latter can be a guide to understanding the former. Social institutions that seem counterproductive or pernicious don't last unless they serve a purpose. If they had only negative effects, they’d be selected against and disappear. So when you see something that looks like a bad bargain for society and yet persists, look harder.
Political parties and the partisanship they engender are in ill repute. This is nothing new. When parties first formed in America in the 1790s, they were criticized even more sharply and consistently than they are today. But they've lasted over two centuries. They must serve some useful purpose.
Of course they do. The Constitution says nothing about how candidates should be recruited and selected. The political parties took on that task and still provide it. In the thousands of races for elective office every year, the parties pretty consistently whittle the myriads of contestants down to two per race. The whittling isn't perfect, but it's better than nothing. Without it or the equivalent, our politics would be utterly unmanageable.
Parties provide branding for candidates and thereby simplify the task of voters in making their choices. Very few voters have the time and patience to investigate the backgrounds and qualifications of all candidates for all offices. The two parties provide a shortcut. Republicans traditionally shared values distinct from the values shared by Democrats. The personal politics of Donald Trump have confused the issue, but Republicans are still identifiably different from Democrats. Knowing nothing besides party about candidates, voters can make a reasonable guess which of the two candidates in a general election aligns more closely with their own views.
These services by parties benefit the republic as a whole. Party activities also benefit particular groups. Service to party has long been a route to appointive office at the state and federal level. Presidents fill their cabinets with party loyalists, and the cabinet secretaries staff their departments with additional loyalists. The practice was reined in during the 1880s when a civil service law was passed, but presidents still have thousands of offices to bestow on their supporters. If rumor is to be believed, a second Trump administration will expand that number greatly. Add in the state offices filled similarly, and the people who depend on their parties for their livelihoods amounts to a modest-sized army.
Especially since the 1990s, lobbyists and law firms have associated themselves exclusively with one party or the other. When the Republicans win, business is good for the Republican support team. Conversely for the Democrats.
During that same period, certain media outlets have essentially identified as Republican or Democratic. Here the dynamic is different than for job seekers. Peddling outrage, Fox News battened on bashing Obama. MSNBC’s viewers applauded Biden's defeat of Trump, but the company's accountants lamented lost revenues. It was long the case in the newspaper business that war was good for circulation numbers. In modern media, the perpetual political war builds viewership, and the more partisan the war the better the building.
Lastly, partisanship makes party zealots feel good about themselves. And the zealots return the favor by intensifying partisanship. The zealots are the face of the parties, and the parties’ loudest voices. Their dominance is felt most clearly in primary elections, where they turn out in greater numbers than the rank and file. This being so, candidates cater to the zealots. Over time, Republican zealots drive the Republicans to the right, while Democratic zealots drive the Democrats to the left. Party moderates, though more numerous than the zealots, are marginalized.
Parties work for the zealots. And for the lobbyists and job-seekers and partisan media. And sometimes for all of us. Parties aren’t a bug of our system, but a feature.
I have said the same thing regarding the branding- the D or the R or the L or G provide information to the voter. There have been some (usually Republicans) who actually advocated eliminating party affiliation from ballots because of the branding.
As someone involved in the progressive side of the Democratic Party, progressives outside the party use party affiliation as an insult to denigrate Democrats because they don't get enough power under the coalition aspect of the parties. Since our Founders didn't create a parliamentary system, evolution to parties seems to have been inevitable. Parliamentary systems often rely on coalitions- we have ours within each party.
But there has been, IMO, one negative trend- the whole "I vote for the person not the party" twinned with the rise of "personality" candidates. In the past, as Mr Brands points out, the parties dictated a lot of the process for candidates on ballots. Now candidates for various positions cannibalize each other for volunteers and funds. I saw this first hand as a candidate for Congress in 2016.