Thanks for your patience.
It was some annoying firewall issue. Because of evil spammers we need the firewall, then the site is set up so that legitimate users can reach the it, but then problems arise, workarounds are needed, etc. The Columbia University information technology people have been very useful.
You do not have to thank us for our patience – it is a service you offer free of charge. So, thank YOU for bringing the blog back online.
This! Thanks!
For the definitive word regarding “spam”:
https://en.wikipedia.org/wiki/Spamming
“It [spam] later came to be used on Usenet to mean excessive multiple posting—the repeated posting of the same message. The unwanted message would appear in many, if not all newsgroups, just as Spam appeared in all the menu items in the Monty Python sketch.”
It was being reported as an SSL certificate issue to clients. You need to have SSL credentials that match your site’s to serve https. That’s more “paperwork” than “firewall”. You’ll see that your own personal blog isn’t https, so it comes up in web browsers with a “Not Secure” warning prepended to the address. Some security-conscious browser configs won’t let you open Andrew’s home page at all. But at least it won’t fail due to missing SSL certs. I didn’t realize anyone was still running non-https servers in 2024!
If Andrew wasn’t (quite reasonably!) worried about being sued, he could serve the blog much more cheaply and stably through WordPress. Instead he has to do it through his tertiary affiliation’s IT person (in ISERP), because neither Columbia as a whole nor his primary and secondary departments offer IT support. A single support person is a real bottleneck in terms of cost, expertise, and availability, as we saw over the holiday.
Bob:
It was a paperwork issue, but apparently the paperwork issue arose because of the firewall, which in turn was needed because of the spam attacks.
I access the blog from a static IP assigned to my office by my university (I am not in the US) and since a few weeks ago I have to do a cloudfare verification every time I access the blog. This issue is independent of the SSL error yesterday.
I’m at a university in the US, and I’ve been getting the Cloudfare verification requests for a few weeks as well; it’s still occurring today. I’ve received the requests when accessing the blog from multiple IP addresses.
It is very unfortunate that the lack of specificity of the spam filters makes it impossible to legitimately access the blog by non-standard methods.
I follow dozens of blogs, this being one of them. It would be very time-consuming if I had not created some infrastructure to help me cope with the 20-30 daily articles they post. What I do is use Python to download the text and an LLM to summarize it in a daily batch job. In most cases, this summary is enough to simply discard a post, typically when the subject is alien to my interests. Sometimes, the full article does not add anything relevant to the terse summary: in many cases, all the article says is something like “X thinks Y,” often embedded in hundreds of words. For some authors, the summary is good enough: they are just experts in turning a concise idea into a long-form essay. For others, there is a bit of God in every word of every sentence they write.
However, due to the firewall on this blog, I get plenty of statmodeling-related error lines in my logs…
Incidentally, the firewall is able to block (error 403) requests made with the requests package in Python, but it does not block curl requests or httr::GET requests in R, even if all of them have been carefully crafted to share the same headers, etc. Either by chance or by careful design, the firewall works in a very odd way.
If anyone is interested in a post-mortem, I am part of the A&S IT group that supports the Faculty of Arts & Sciences departments with their tech needs, thus through the Statistics Dept and Andrew Gelman, this blog. I’m leaving some stuff out, but here’s a background & overview of what happened:
— This blog has been running a long time. Certainly over a decade, as it was running when I joined Columbia, although I don’t know exactly how old. Andrew – when did you first start blogging?
— The site is powered by WordPress – super useful & powerful website software, (it runs 43% of all websites on the web) but also prone to attacks since it’s so widely used
— The site uses about a dozen or so themes and plugins, some custom written, others open source available on the WP plugin site. Many were not maintained. The original developer(s) are long gone, so the custom code remained as originally written, and certain plugins the original authors did not maintain. The wordpress core and popular plugins are updated regularly.
— A while back, some hackers found a security hole in one of the aforementioned unpatched themes/plugins and started injecting russian casino (and similar) links into the blog code. The injections were easily removed, but then they came right back. Numerous attempts to patch the security holes were unsuccessful, the attack vector was never found for sure. My best guess is that it came through an old caching plugin. I can’t be sure.
— The site was placed behind cloudflare Web Application Firewall, under Columbia University’s main cloudflare account. This cloudflare account handles hundreds of sites at Columbia. The standard set of rules were set in place to prevent these hacks and link injections. It worked great, no link injections since then.
— SSL certificates for the site are a combination of Cloudflare provided and managed by Columbia’s main IT group CUIT (from the Cloudflare content delivery network to the end users’ browsers) and WPEngine provided (from the web host on wpengine.com to the cloudflare servers)
— WPEngine uses Let’s Encrypt to sign their SSL certificates
— A few weeks ago, related to the protests on campus, Columbia University came under a sustained and wide-ranging DDoS attack, with hundreds of millions of attempted connections over a few days.
— As one part of DDoS mitigation, Cloudflare WAF rules were increased to “Zero Trust” (or similar settings), which successfully stopped the DDoS attacks on websites. This had the side effect of requiring new/unknown visitors to “prove you are human” as many readers have noticed which in turn blocked virtually all bots.
— One such block was related to RSS feeds, so we did place a whitelist on the RSS page, which helped readers using RSS to read the articles.
— This gets a bit complicated, but to put it simply, the way Let’s Encrypt verifies domain control is by placing a token on the server and using an HTTPS request to see if the token exists on the server. If the token is there, the requester is the owner, and the SSL cert is issued.
— Over memorial day weekend, the Let’s Encrypt SSL certificate (which encrypts between the WP host and CloudFlare) was set to expire.
— The automated renewal script ran as it should, but due to CloudFlare new Zero Trust rules, the verification from Let’s Encrypt’s servers was immediately blocked and the cert was not renewed. Thus, traffic from WPEngine to cloudflare was not authenticated, resulting in the “Invalid SSL Certificate – The origin web server does not have a valid SSL certificate” error shown in the browser on May 27 and 28.
— On Monday, I was able to re-issue the WPEngine Let’s Encrypt SSL certificate using domain verification, by using a DNS TXT record to verify the token over the domain name system instead of an HTTPS request.
— Longer term we are looking to put a cloudflare rule exception to allow Let’s Encrypt traffic through so that the automated renewals resume.
Eric:
Thanks for the information! The blog started in fall, 2004, and we’ve been posting every day since. I don’t think it was always on WordPress but I’m not sure.
Thank for posting this.
Bob76