interandom

I had a tough time deciding if this should be in the “IT Rants” or “IT Joys” category. I really love my Blackberry and the functionality the Blackberry Enterprise Server provides is awesome, however this is the second it’s blown up. And anything that ever blows up should be classified under Rants.

So.

I reboot the server BES is installed on. A normal reboot, as in, start, shutdown, restart. I did this because I needed to move where the server was in my “virtual infrastructure.”

Server boots back up. BES is dead.

The long story:

All the work I had scheduled for last Saturday night, starting at 8PM, was done by 11PM. Only 3 hours seemed pretty good. But then I went to test BES and saw that it was dead. I’ve been through this before, so I tried everything I knew until I absolutely couldn’t think of anything else. After all the googling and browsing of RIM’s support site. By the time I was done with all of that, it was 5:30AM. Yup, I’d been at it for 6.5 hours. Then I say to myself, “okay.. let’s call RIM for support. I know our support contract is only for 8am to 5pm, M-F, and this is early Saturday morning, but they probably have some option where I can pay them $245 (like Microsoft) and get support right then and there, outside of my contract.” Well it turns out they don’t. Unless you have a 24×7 support contract (”T2 or better” in BES-speak) you ain’t gettin’ no help no matter what. No matter how much money you’re willing to give them. It ain’t happenin’.

One of the (very few) things I like about Microsoft is that no matter what time of day, and no matter what Microsoft product you’re using, and no matter which unsupported way you’re using it, there is ALWAYS a skilled living, breathing human being available to help you. The person may be on the other side of the world but they’re there to help you. I’ve used this service about 4 times now. I only use it when I’m absolutely out of options, and by then the $245 they charge is pretty acceptable. One time I was on the phone with the same rep for 9 hours (not exaggerating) fixing an Active Directory problem. 9 hours of Microsoft-inside tech support for $245? Yes, that’s good. Well, the cost was worth it. Having to work on a Microsoft problem for 9 hours is definitely not worth anything. It’s a pity my servers are all virtual now because there’s nothing physical I can punch, out of frustration. (I could kick the ESX server they’re running on, but that wouldn’t be fair!)

So then I went on RIM’s support site and saw that you can buy 24×7 support right now, online, with your web browser. I saw this and felt very happy - I could just upgrade my support account right now and be able to call them. I go through all the checkout steps. At the end, I’m told that they have received my order, and it will be processed in 3 to 5 business days! No where else in this process did they say I wouldn’t have instant access.

At that point, it was 6AM, I’d been at it since 8PM, I was totally wiped out and out of options, so I went to bed.

I woke up at 5pM the next day. The only thing good about an overnight IT battle is how you feel after your 11 hours of sleep.

But the feeling was short-lived; as I awoke, I remembered that BES is still broken. As is common for me, I had dreams about the problem, and I actually came up with a few more things to try.

I would build a totally new server, dedicated solely to BES. Thanks to my VMware Virtual Infrastructure, doing this, from home even, was extremely easy. I had a new Windows 2003 Server up and running at about 7:30pm, having started the work around 6:30. (I didn’t have a template. But I do now.)

Everything looked good. I coped the BES database over, attached it in MSDE, installed BES with the same SP and hotfix level I had before. Went to start up the Blackberry Manager. And I got the same error I had on the original machine:

“Unable to retrieve the server’s distinguished name or allocate a MAPI buffer for this server.”

All that work for nothing. Well, not entirely nothing. It was good I had it installed on a separate server now, because while fixing the problem with RIM support, I had to reboot it about 20 times. The machine it was originally on was a domain controller and the DHCP server, so having to reboot it that often would really stink.

I was on the phone with RIM support for 3 hours, 45 minutes. We went over every single possible thing they knew of and I knew of to try and fix it. Absolutely none of them worked. Then the RIM support guy suggested we make a new username for BES to run under. We were using BESAdmin before, this new one I named bbadmin at his suggestion. Soon as we switched everything over to that new user (which involves a lot of tweaks all over active directory, exchange, and the OS), the server ran perfectly.

So, in the end, the problem was some sort of corruption in the BESAdmin account. How a reboot could cause this, I don’t know. It was a normal reboot. No errors anywhere. Scandisk found no problems. The SAN move went fine.

This has been added to my list of why Windows sucks. There are things it’s good at, yes. But when Windows is bad, it’s way, way, way badder than anything else in existence. I ask you. Has there ever been a single problem in your life that WASN’T IT related and took 16 hours or more of constant labor to fix? I’ve lost so many weekends, so much of my life, to Windows.