Back in 1997, programmer and open-source activist Eric Raymond wrote an online essay, The Cathedral and the Bazaar, making the case for open-source software development, using Linux as an example:
Linux is subversive. Who would have thought even five years ago (1991) that a world-class operating system could coalesce as if by magic out of part-time hacking by several thousand developers scattered all over the planet, connected only by the tenuous strands of the Internet? . . . Linux overturned much of what I thought I knew. I had been preaching the Unix gospel of small tools, rapid prototyping and evolutionary programming for years. But I also believed there was a certain critical complexity above which a more centralized, a priori approach was required. I believed that the most important software (operating systems and really large tools like the Emacs programming editor) needed to be built like cathedrals, carefully crafted by individual wizards or small bands of mages working in splendid isolation, with no beta to be released before its time.
Linus Torvalds’s style of development–release early and often, delegate everything you can, be open to the point of promiscuity–came as a surprise. No quiet, reverent cathedral-building here–rather, the Linux community seemed to resemble a great babbling bazaar of differing agendas and approaches (aptly symbolized by the Linux archive sites, who’d take submissions from anyone) out of which a coherent and stable system could seemingly emerge only by a succession of miracles.
The fact that this bazaar style seemed to work, and work well, came as a distinct shock. . . .
Lots has happened in software workflow in the past quarter century, and it’s my impression that developers try to get the best of the cathedral and the bazaar. Ultimately, the two forms go together: any “bazaar” is built from individual “stalls” or mini-cathedrals, while any “cathedral” must ultimately compete in a larger “marketplace” or bazaar. So the question isn’t really “the cathedral or the bazaar,” but rather, where should development be more cathedral-like and where should it be bazaar-like. Raymond’s article was helpful in framing software projects in terms of these metaphors, and it connects closely to ideas in economics regarding the roles of firms within the larger economy.
The two software products that I use the most are R and Stan, and I’d say that R has suffered from being too bazaar-like and Stan has suffered from being too cathedral-like.
R code can be a mess because there’s no standard way that things are written, and also because it keeps adding layers and layers of complexity. Back in the day, if you had a matrix, it was a matrix and you could display it access its entries directly. Nowadays, I often need to do tricks such as as.numeric() or as.matrix() to strip away various structures in order to get at the actual numbers. I’m not saying this is always bad–I’m sure that the bazaar-keepers of R have good reasons for all this–it has just become more and more confusing to access attributes, objects, data, whatever.
When I say that Stan is too cathedral-like, I just mean that it seems to take a long time for improvements in the algorithms to show up in the program. I understand that this is important for quality control, I just find it frustrating to not be able to easily access the latest developments.
For both R and Stan, I’m not saying there’s some easy alternative. Things are what they are for a reason. I’m just using these as examples of how I sometimes think in terms of cathedrals and bazaars.
Statistical workflow
The most important difference between statistical workflow and mere statistical inference is that workflow is all about building trust. Steps such as simulation-based model checking, predictive model evaluation, and comparison of the results of multiple models are ways to stress-test our fitted models, to reveal their weaknesses and, sometimes, to lead to more confidence in the results.
It’s possible to fit models with all the workflow, but then you can end up with wrong answers, such as the ridiculous estimate that losing an election for governor costs 5-10 years of life.
One way to think of this is to consider medieval cathedrals. Still standing after all those centuries–that’s so impressive! But there’s selection. These cathedrals were built without good theory, and lots of them collapsed. As much as possible we would like to use science and engineering workflow to avoid building cathedrals that will immediately crumble. Science is a kind of cathedral–a collective human endeavor with large goals–and we would like to design it better.
But science is also a kind of bazaar . . . and, here’s the point: Bazaars, like cathedrals, are designed. They’re not designed in the same way as cathedrals, but they don’t just happen either. There have to be rules, to keep the bazaar from becoming a bloodbath. It’s fair enough to think of the current system of science, with its millions of scientists and hundreds of thousands of research labs, as being a sort of bazaar, and we’re trying to improve the workflow, which would make the bazaar more effective. I think we can do better than the current system of p-values, failed replications, deterministic claims, and waste of reviewer resources.
It’s been a while, so may be I’m fudging some of the technical details or my memory is incorrect, but I’m competent enough to make sure the product is correct, but…
WRT to Stan, there was some talk on the Stan development team years ago about, saving previous runs, so that you could update a model with additional data, or, when making predictions with out of sample data, or future forecasts, wasn’t there a conversation about being able to re-use the same model for different out of sample prediction data sets?
I made a forum account again, I can’t find the thread, this was from years ago.
From linkedin:
https://www.linkedin.com/feed/update/urn:li:activity:7404067738924556288?commentUrn=urn%3Ali%3Acomment%3A%28activity%3A7404067738924556288%2C7405613097299742723%29&dashCommentUrn=urn%3Ali%3Afsd_comment%3A%287405613097299742723%2Curn%3Ali%3Aactivity%3A7404067738924556288%29
Yes this is a problem with Bayesian as well. Some of the Stan developers talked about this, but it comes down to being able to save the estimates of a previous model run, and only have to compute the posterior predictive, for example, on new data. I’d have to be more theoretically rigourous in a serious explanation, but yes. It’s a lot of software dev. Saving and re-using pre-trained models. May be I am incorrect, or inaccurate, but using Stan, when making forecasts on different test sets I’d have to re-run a model on the same training sets, which is timely costly.
~~
End Linkedin.
But I’m available for incremental developments. I’ve been with SAS for a while.
We called this feature “standalone generated quantities” because that’s what it is from the Stan perspective. It lets you sample, save the draws, then plug them back into a new model with the same parameter declarations, new data declarations, and a new generated quantities block. So exactly for posterior predictive inference. Here’s the CmdStanR doc:
CmdStanR doc: https://mc-stan.org/cmdstanr/reference/model-method-generate-quantities.html
I couldn’t find the CmdStanPy doc, but here’s an example of its usage in Python:
CmdStanPy standalone generated quantities example: https://mc-stan.org/cmdstanpy/users-guide/examples/Run%20Generated%20Quantities.html
This is also available in Turing.jl as far as i know. if you have a chain of samples and a model, you can run generated quantities over the chain and get predictive results.
Presumably Turing.jl would do this like Stan and let you plug in multiple chains so that you can evaluate convergence diagnostics over any new expectations.
One of the reasons I like JAX is that this is just a vmap operation. I’m guessing Andrew’s not going to like this vmap because of the need to manipulate the extra index and write a lambda. That index manipulation is so endemic to ML programming that people learning JAX are used to it. Not having to fuss with indices was the motivation for rvar (in the R package posterior), which lets you manipulate draw-based random variables using arithmetic operations and still compute marginal expectations and variances.
I think of the analogy here as being one of how the software is designed and managed, and specifically whether there is a central controlling authority (by analogy, the church) and whether the design is top-down. While we have a governing body for Stan much like R does, the development of Stan is very open and bottom-up. That’s how someone like Tadej Ciglarič can come along and rebuild how we code matrices. Or how Mitzi Morris can build CmdStanPy. Nobody’s holding anyone back from doing any of this. We’re mainly constrained by not having enough developers, especially in C++.
If you look at R, its core is very tightly controlled. I have no idea what has changed in base R over the years, but it all feels very similar to me (but I’m not really a heavy R user). That’s why Radford Neal got so frustrated with pqR—there’s no way an established project like that can rebuild its core infrastructure. On the other hand, R allows people to contribute extensions to it in the form of packages. The form of these packages is highly constrained due to CRAN policy, hence the amazing headaches we’ve had particularly with RStan and why Jonah is so resistant to trying to put CmdStnaR onto CRAN. The huge number of add-on packages gives R a bazaar-like feeling if you were following the metaphor literally.
If you look at something like the Tidyverse, that’s not a change to the core of R, that’s a very popular set of packages built on top of R as extensions (I don’t know if any finagling of base R was done to support it—people reading this might know). In that sense, the Tidyverse is more like brms. brms is the most successful part of Stan in terms of number of users, but it was not generated from within the core of the project at Columbia—Paul Bürkner just started building it very soon after we launched Stan and he started commenting on our early message boards. And now I think he’s mainly moved on to focus on amortized inference, though he left brms in a state where other people can extend it. There’s no way anyone can keep up with the flood of brms-related messages on the Stan forums. All of this development is possible because of Stan’s open-source licensing, which R shares to a lesser extent (because of copyleft, distributing extensions to R brings more legal restrictions than extending Stan).
Although you may not realize it, there are dozens if not hundreds of packages that use core Stan. Another popular one in time-series analysis is Prophet, which is in wide use beyond Facebook, where it was developed. We only found out about Prophet when it launched. Then there’s Torsten for pharmacometrics, which was developed independently by a private company (albeit one with which we later shared a tech transfer grant—Metrum).
If you look at how other packages grew, like posterior or ShinyStan, it wasn’t through central control, though both of these projects are now governed by Stan. On the Python side, the work of posterior is being done in a package called ArviZ, which isn’t being governed by Stan. And ShinyStan is basically reimplemented in an online fashion for streaming data in MCMCMonitor, another non-Stan-governed project.
Medium-sized projects like Stan go through a lot of growing pains. For us, that meant moving from consensus-based decision making, to a kind of hierarchical leadership (sort of like Linux), which didn’t work at all for us. I absolute refused the title of “benevolent dictator for life” because I knew it’d cause a lot of strife. Linux isn’t exactly all fun and games with their leadership structure.
Stan eventually settled on the Apache model of voting. So there’s no controlling force behind Stan development technically, but I would say the folks who have been here longer do get listened to. But not enough to get my way on how our interfaces are designed and documented, for example (this is one of the main reasons I’m developing WALNUTS outside of Stan’s control, the other is so that it doesn’t take so long to code because Stan’s so picky about unit and functional testing). With something like BridgeStan (again, not centrally designed—Edward Roulades just built it as a convenience to help him build faster models in Julia), we have really opened up the use of Stan in algorithms. You see this in extensions like Nutpie, which are really wonderful sampling tools that can fit Stan models without being governed by Stan (that was an unexpected side benefit of BridgeStan—we were thinking of it mainly as a tool for developers to make it easy to do algorithm experimentation).
This is reminiscent of Coase’s “The Nature of the Firm”.
Firm : Market :: Cathedral : Bazaar
Which could imply the cathedral-bazaar trade off is driven by the degree to which distributed development can provide adequate coördination between components.
this analogy is highly dependent on our current dominant capitalist system. it is not an essential component of firms. for example Mondragon in Spain operates as a cooperative federation of individual worker owned coops… A similar thing was true during the spanish civil war more widely.
dominance hierarchies are so common we consider them facts of nature often. But they arent.
Jesus dude, Do you really have to shoehorn this ideologue shit into every topic?
Malnourished:
Given the topic of the post–different forms of coordination in group activity–I think the comments by Mark and Daniel are both relevant!
I think some skepticism of Daniel’s comment is warranted; even in Linux, the very paradigmatic example of the bazaar model of software development, Linus Torvalds’s role as a gatekeeper at the very top of the dominance hierarchy and his tight control over code submissions is crucial to its success.
https://linuxreviews.org/Linus_Torvalds#Kernel_development_model
Aaron, expertise and coordination are not the same as dominance. tomorrow if Torvalds started keeping things from going into the kernel that the bulk of kernel developers wanted, or started allowing all sorts of crap that people wanted out, Linux would fork. its been done thousands of times in free software development. for example Nextcloud vs Owncloud, Libreoffice vs Openoffice, Mariadb vs MySQL, SBCL vs CMUCL, LEDE vs OpenWrt (and then back to 0penWrt with different leadership). Openboard vs whatever it forked from…
Torvalds serves as a useful coordinator that the developers want, not a person with authority (a jargon term here) that the developers have no choice but to follow.
Contrast that to the majority stockholder of a typical corporation. if the workers dont do what that person says, they are fired and lose access to the capital owned by the company. In the Blair Mountain coal wars they then dropped bombs from airplanes on the workers.
in any case my comment merely pointed out that FIRMS dont HAVE to operate like the typical hierarchical dominance corporation structure and gave an example.
in this context hierarchical dominance has a particular technical meaning. it means that decisions made by the people at one level are binding on those at lower levels with no recourse and are enforceable by violence. Literally the only thing enforceable by violence in the Linux kernel is that you cant distribute binaries without transmitting the source code and legal rights to modify it. its an antihierarchy hack using the dominance hierarchy enforcement mechanism.
> this analogy is highly dependent on our current dominant capitalist system.
No, it isn’t. It depends only in the existence of a firm, or another organization, of people that work together to do something like building a cathedral or whatever. The fact that the firm is owned by an individual, or a foundation, or independent shareholders, or a subset of the employees like Mondragon seems highly irrelevant to the analogy.
the point i was making is that if you think of a firm as a cathedral organized top down and a market full of firms as a bazaar, thats because you think if a firm as organized from the top down “cathedral like” but that is not an essential component of what it means to be a firm. some firms are bazaars within the firm.
I’ve spent a number of years in the FOSS world, and I think that there is a lot of confusion. Raymond is clever, but there is a lot of magical thinking in his work. That’s why some of it is so charming. What’s not to love about “The mail must go through.”
Originally, his Cathedral/Bazaar framework was a contrast between the walled off proprietary model of the untouchable Cathedral where you can’t fix the bugs and the open source model of people fixing software on their own and sharing what their have done. The “best” software would win through market processes.
Of course, Raymond is also a libertarian and really was part of the early days that brought the right-libertarians together with the software world and gave us, ultimately, the Silicon Valley libertarian culture we see today. Because this software model often relies on the “benevolent dictator” which you see at the core of so many of their pseudo-philsophical musings.
In the Raymond idea the bazaar is the market. It’s CRAN with its tens of thousands of packages and let the best implementation win. Of course this ecosystem depends on there being a highly stable and extensible base R which is very much a cathedral, and thank goodness for that. GPL is great for encouraging extension ecosystems built around a stable base. This is a way that Raymond missed the boat. Of course, back in the middle ages the market towns were also cathedral towns, so the idea that they are in opposition misses the point.
Elin:
I agree with everything you write. Also, amusing point about the cathedral towns being the market towns. That’s a great encapsulation of the idea that the bazaar makes relies on the existence of the cathedral, or that the bazaar can be more effective when it is built up around a cathedral.
Regarding Raymond’s politics . . . calling him “libertarian” is the least of it! I did some quick googling and he’s writing things about people being thrown out of helicopters (“I am unsure whether I’ll help them do it, but I’m pretty sure I won’t try to stop them”), he writes that he believes the 2020 election was stolen (“I do increasingly wonder when the Democrats last won a national election by honest means. I am no longer prepared to bet that it has happened in my lifetime”), and he writes that in 1965 Trump “was a mainstream Democratic liberal then – got an award for anti-racism and promoting civil rights at least once.” At least once, huh?
This should not be taken to invalidate his cathedral/bazaar idea. “The cathedral and the bazaar” is a kind of literary art–not quite “poetry” or “fiction” as we would usually define these terms, but some sort of imaginative literature. Lots of great literature has been produced by people with crazy ideas. Indeed, one could argue that, to write great and innovative literature it could help to be open to wacky hypotheses. Common sense is a sort of filter that maybe can get in the way of flights of the imagination, and Raymond’s lack of common sense in the political realm might well have been a benefit in letting him write that essay so poetically. Or we could take it from the other direction and say that when he wrote the essay he had more of a grip on reality and since then has become just one more Fox News uncle, a victim of extreme politics and conspiracy theories. In any case, we can appreciate his essay just as we can appreciate the writings of various mid-twentieth-century authors who fell under the spell of communism or whatever.
I’m still amused/disturbed by Raymond’s skepticism that the Democrats have won any national election in his lifetime by honest means. Raymond was born in 1957. How does he think the Democrats stole the election of 1964? This was the year the 19-year-old Donald Trump apparently got that award for anti-racism and promoting civil rights!
Ugh, i always knew ESR was a bit nuts, but i definitely think he must have gotten worse under the right wing propaganda machine.
on the subject of literal cathedrals and bazaars though, its never a surprise when business builds up around a cathedral. A cathedral is a monument to wealth. Also, its a bit backwards probably. i remember watching a PBS show about African history, and bizarrely it mostly lionized the wealthy elites… in any case there were some consistent patterns described. basically some market would develop where a resource was being extracted and shipped somewhere. then a band of thugs would insert themselves in the port town or along the road and extract a fraction of that transported wealth in some kind of “tax” scheme, sometimes upwards of 50%. the thugs would then build up their town with grand monuments to themselves and walled cities and thrones and things… i guess later everyone would say how look, this thriving port and gold mine grew up next to this cathedral like structure.
cathedral like structures always grow up next to an economic wealth production system off an extraction mechanism. you dont get cathedrals without someone creating the wealth that fuels their construction.
The more you read, the more nutty he becomes. People think Stallman is awful–and he is, interpersonally–but Raymond is on a whole different level of disturbing.
Elin:
Yeah, the racism is horrible, but, hey, lots of people are racist. Racism is a readily available explanation for all sorts of things. The bit about throwing people out of helicopters is flat-out scary, but that could be exaggeration and he’s probably seen too many action movies or read too many combat-porn books. But the belief that the Democrats have not won a national election by honest means since 1957 . . . that’s tinfoil-hat territory.
I guess then the question is, which crazy beliefs are socially acceptable and which are not?
It depends. For example, there is no evidence of consequential fraud in the 2020 U.S. presidential election. But the determined effort of Ted Cruz and other political leaders have legitimized the claim of fraud, to the extent that to believe it, you don’t have to be a crazy person or a mischievous law professor, just a normal Republican. I’m pretty sure that Cruz himself is well enough informed to not think that election was fraudulent, but that doesn’t stop him from fanning the flames, I guess because he thinks there are positive political consequences from millions of people believing these baseless claims.
But the idea that the Democrats have not won a national election by honest means since 1957, that hasn’t been endorsed by anyone serious, so Raymond really doesn’t have any good excuse for believing that one.
To consider two less contentious examples, it’s apparently acceptable to state that scientific citations are worth $100,000 each or that the probability of an election being tied is 10^-90, even though, from a quantitative standpoint, these claims are even more ridiculous than the no-legitimate-Democratic-election-wins-since-1957 statement.
I also know people who say that there was this dude called Moses who led people out of Egypt . . . again, it’s socially acceptable to believe it!
Another one is the idea that UFOs are space aliens, a ridiculous notion that’s gained a foothold in the elite news media. I’ve contrasted this with ghosts, which many many people believe in (no surprise, as it’s intuitive that we should live on in some form after death, and there’s no end of phenomena of the creaking-in-the-attic variety that can be taken as ambiguous evidence for spiritual beings) but which don’t currently have a convenient way to plug into the media/scientific establishment.
Eric Raymond going off about elections on the internet represents to me some change regarding ridiculous beliefs that are becoming socially acceptable, for now on the fringe of the fringe but who knows, going forward? It’s scary. As someone put it once, it’s a very short jump from believing kale smoothies are a cure for cancer to denying the Holocaust happened.
A couple years ago I wrote about a friend of mine who “did an Eric Raymond” in that he moved from being someone with strong political views to going off the deep end, endorsing the claim that the covid vaccine was “the biggest crime ever committed on humanity.” Anyone can go crazy, but I think the problem here was that this particular insane belief was circulating on the internet in a way that legitimized it for him.
Both as a statistician and as a political scientist I’m horrified and fascinated by the ridiculous things people believe.
Andrew: FYI, did you know that the stuff “about people being thrown out of helicopters” is a well-known right-wing reference to fascist (I use the term advisely) death-squads executing people?
https://progressive.org/magazine/how-the-free-helicopter-rides-meme-went-viral-roberts-20230907/
“How does he think the Democrats stole the election of 1964?”
I assume this is a rhetorical question. While I wouldn’t want to even try to speak for him, I’ll speculate the answer would involve Communists using Democrats and African-Americans (my phrasing on that one, not his!) to intimidate election workers via something like snitches-get-stiches. Note: I am completely serious there (meaning, I conjecture he really would say something along those particular lines).
Seth:
There were also some other national elections won by the Democrats after 1964, including the presidential elections of 1976, 1992, 1996, 2008, 2012, and 2020, along with lots of national congressional elections that Democrats won decisively, including 1974, 1982, and several others. Also the Democrats would’ve won in 2000 had they counted all the votes. But, setting 2000 and 2020 aside, and also setting aside 1976 which was pretty close (although it was clearly won by the Democrats), there still remain the elections of 1992, 1996, 2008, and 2012. I guess he could say the Democrats won all these by “dishonest means” if that includes making campaign promises and not delivering . . . but, under that standard, very few presidential candidates, if any, have ever won an election by honest means, in which case it’s meaningless for him to single out the Democrats.
Andrew: No, I believe his answer would be the same for all them – “dishonest means” being racial violence, ballot fraud, voter intimidation, etc. And then he’d proceed to throw in some random inflammatory fact about corrupt Chicago elections, or about a left-wing activist who got convicted of polling place interference. One can say many things about him, but being unclear about his beliefs, is not one of them.
Seth:
People say things about Illinois in 1960, and of course there are those who claim that Bush got more votes than Gore in Florida in 2000, and at this point the 2020 election has been swallowed up into the noise machine, but it’s hard for me to see how violence, ballot fraud, voter intimidation, corrupt Chicago elections, and polling place interference could explain Democrats’ wide margins of victory in 1992, 1996, 2008, and 1964. Or even 1976 and 2012, for that matter: both those elections were close but I’ve never seen any serious claim that they were determined by any election irregularities.
I guess I could send him an email and ask!
Andrew: Yes, I do think emailing him would be helpful if you’re curious about the details of what he believes, and his reasons for those beliefs. He is not reticent!