Speaking of panic...

It's a rare occurrence that I lose sleep over my job. I still enjoy it, and I care about it, and the work I do. But it feels... staid to me now. A shade too familiar, as though I can go about my business safe in the knowledge that the day will contain few surprises for me. In someways this is good. Predictability, if anathema to excitement, at least makes it easy to deliver work with some consistency. On the other hand it means that when things do go wrong it comes as a bit of a shock.

The project I've been hard at work at for the last couple of months is near completion and has gone swimmingly well thus far. The various components involved all behaved exactly as they should and I'd managed to throw in several neat additions that will delight at least a few people when they discover what I've done. Late last week I started to move the code onto a test set up which more closely parallels our production environment. And as expected everything went as well as I could have hoped for...

...until it all froze up after about an hour. The hackles on the back of my neck raised slightly at that point. I wasn't worried yet. After all, it wasn't necessarily my code that caused the problem - there could have been other forces at work. Dark, nefarious forces intent on bringing me to heel. Or something like that. Anyway, with the slightest trepidation, I reset everything, and kicked it all off once more. Again, everything went fine... until it all stopped about two hours later. The next iteration lasted only about 17 minutes. The only consistency was that it would invariably terminate, although at a random interval, and seemingly without rhyme or reason. My raised hackles were now joined a thin trickle of cold sweat.

I spent two days trying to discover what the problem might be, without much success. My best guess in the end was that a firewall, peculiar to this particular test environment - for arcane reasons that I shan't delve into - was somehow interfering, although I'm no network engineer and the network engineer we do have proved singularly unhelpful on the topic, being unable to confirm or deny much of anything, or to even look slightly interested in what was going on. With desperation setting in, I managed to concoct a firewall-less test environment, copied the code onto it, and set matters in motion once more. This was about a 4pm yesterday. By 5:30pm it was still running. In and of itself this meant little - on the old set up it had managed to run for three hours once. Since I could do little more than wait, I went home, there to sit around and obsess about what I would find when I went into work the following morn. After a while I decided that I'd sat around and obsessed enough without achieving much, and sloped off to bed. I ended up laying in bad, wide awake, obsessing about it instead. After several more hours of obsessing I eventually drifted off to sleep... only to wake up a couple of hours later to begin the obsessing all over again.

Unable to take the suspense any longer, I practically ran to the office this morning to find out what had happened. And it was still running. By god it was still running! I skipped around the office for a bit after that, trilling to those I passed in the corridor "it works, it works!". They had no idea what I was talking about, but really I didn't care. It worked! And it was still worked even as I left this evening.

This has not been a fun 24 hours! I'm immensely glad that things are once more behaving as I intended them to, but I'm feeling extremely testy to a particular network engineer. Razzin' frazzin' firewall. Tomorrow I'm going to have words with someone about that...