[uday]

The server has not been completely stable recently. The rest of this post explains why, and what we're doing about it.

- Over the past several weeks, we became aware of an obscure bug in the operating system used by our server ( Linux, if you care). This bug had the effect of occasionally (under certain conditions of high usage) making the server so slow as to seem unresponsive. Another effect of this bug was a series of short pauses throughout the day.

We installed an upgrade to the operating system last night. In theory, that should be the end of that.

- Crashes. The past week has seen a few crashes. Some of these have been caused by experimental code (ironically, code that is trying to make the system more stable) that I've been trying out. We're now running a 'stable' version of BBO again, and this should help. Some of these have been caused by old bugs that have been around 'forever' but are only triggered under rare circumstances. We still have a couple of these out there, and we're trying to track them down.

- Planned outages. Some of you will have noticed that we've been bouncing the server at about 1AM EST. Some of our code changes require that the server be bounced. I expect to continue doing this every night or two until the system stabilizes. Why 1AM? Usage is relatively light at this time. Further, it is early enough that we can stay up and monitor the system for a couple of hours after each bounce (typically, the bounce involves some dangerous new code).


Anyway. Thats what has been happening. We still have a couple of crash-capable bugs out there. We're working on them, to the exlusion of nearly all else, and I expect that things will be better within a week.

ui

[uday]

We installed a new version of the BBO server last night; this version is less bloated, and includes some checks that will prevent at least 2 of the recent crashes from recurring. Fingers crossed.