Unscheduled Downtime
-
You may have noticed that we were having some display issues today. Unfortunately this was reported to me in the middle of my day-job workday and had to wait until this evening for resolution. Though the outage looked bad it was actually a really simple fix once identified. We should be back up and running without further issue. If you're still seeing a problem please refresh your browser. If it doesn't clear, please check incognito mode to be sure that there's an issue, then message me for help.
If you're curious about the root cause and how I figured it out, you can read on for a post-mortem:
Initially, I thought that something had gone wrong with data in the database. I figured that a post had been posted that threw a wrench into the works. I poked around to see if I could identify a culprit but nothing stood out. I also knew that I needed a control system so that I could isolate the issue and begin changing variables until something popped up.
I have a local developer instance of the forums which tends to lag a few weeks behind production. I really only update it if I need to do some work with data that's current in prod. In addition this database is totally independent of production so I knew this would serve as a good control.
I fired up the dev instance and sure enough it showed the exact same issue. This meant that it wasn't anything in the database that was causing our problem. Since this was a display issue AND it seemed to happen without warning, I figured it was probably CSS related, and likely something that was hosted in a CDN. One look at the source on the page shows that the bootswatch CSS is hosted in a CDN. In addition the path to the CSS read 'latest' as opposed to specifying a specific version. I also know that Bootstrap just released their long awaited 4.0 version. Everything started to line up.
I searched around a bit in the forum's template files to see if I could find a reference to the CDN, but there wasn't one. It had to be coming from somewhere else. I decided to head over to nodebb's official forum to see if somebody had any insight. Sure enough the top post was discussing this very issue.
Long story short the fix is in and it's really quite simple. Check out this pull request:
https://github.com/NodeBB/NodeBB/commit/d16667a5fb8f5bb9458c8e94dfb60cdb50a27011Even if you aren't a coder you can see that it identifies the places where the code referenced 'latest' and has been updated to reference a specific version '3.3.7.' A quick patch and rebuild of the UI and everything is back to normal.
Sorry again for the downtime. That wasn't fun. Big shoutout to @myanruth for reporting the issue to me. Remember guys, if you see something, say something.
