interandom

I’m using a demo copy of a backup program named “NetVault” produced by a company known as “Bakbone”. It was suggested to me by my CDW representative as an alternative to Veritas. As you probably already know, I’m not that happy with Veritas.

To my surprise, they offer a Linux version of the NetVault server software that has all the features of its Windows counterpart, including things like Exchange mail store backups, MS SQL backups, etc. That right there is enough to get me to try the product: to not rely on Windows for backups anymore.

The GUI is pretty straight forward to use. In my case, the backup source consists entirely of network shares and the destination is an Apple Xserve RAID. In other words, I’m backing up to hard drives. Setting this up in Bakbone is slightly different than other products I’ve used… you have to create what they call a “virtual tape library”. You specify how many tapes you want (each tape = one file) and how big each tape is (the size of the resulting file). The drawback here is if you, say, create enough tapes to fill up a 2.6TB partition, it will take a long time for this operation to complete. It creates each file right away, filling it with zeros. 2.6 terabytes of zeros. It took a long, long time.

However, the pro to this is that as far as the rest of NetVault is concerned, you’ve got a real tape library. The virtualness of it is hidden from the rest of the system. As a test, I’m having it back up all 2.5TB of data on my NetApp. If it goes well I’ll probably try to buy it. The Veritas desktop agent is really slow and consumes a lot of resources, and I’d like to get rid of it.

Note to self: When you remove a NIC in ESX from a bond, and you don’t inform your switch of this change. Strange Things Occur.

This prompted me to write a little something describing how to set up NIC teaming in ESX Server. For the uninitiated, NIC teaming, also known as link aggregation or 802.3ad, is a way of grouping network interface cards to improve reliability (redundancy) and performance (transfer speed).

To make this work well, you need to do some configuring in ESX and also in your switch. Like always, my switch example will be for a Cisco IOS-based Catalyst, though the principles involved in 802.3ad are pretty simple and standardized so you should be able to apply it to any other switch with the 802.3ad capability.

Enable 802.3ad NIC teaming in VMware ESX Server

NOTE: This example involves using the ESX MUI; as of the current version of ESX and VirtualCenter, you are forced to use the MUI for these changes.

1) Login to the MUI and click Options. Click Network Connections.

2) Assuming you already have a virtual switch, add an unused outbound adapter to your virtual switch. This is pretty much all you need to do on the ESX end of things - you now have a bond. You can bond more than 2 NICs as well.

3) In your switch, you must configure a port channel (Cisco-speak for a 802.3ad team), and then assign specific hardware ethernet ports to it. You also need to set up trunking on the port channel, if you want to use VLANs in your VMs. Like so:


ZORAC# conf t
ZORAC(config)# int Port-channel1
ZORAC(config-if)# switchport trunk encapsulation dot1q
ZORAC(config-if)# switchport trunk allowed vlan 1,2
ZORAC(config-if)# switchport mode trunk


The above creates a port channel. Now we’ll assign ports GigabitEthernet0/1 and GigabitEthernet0/2 to the channel.


ZORAC# conf t
ZORAC(config)# int GigabitEthernet0/1
ZORAC(config-if)# switchport trunk encapsulation dot1q
ZORAC(config-if)# switchport trunk allowed vlan 1,2
ZORAC(config-if)# switchport mode trunk
ZORAC(config-if)# channel-group 1 mode on
ZORAC(config-if)# exit
ZORAC(config)# int GigabitEthernet0/2
ZORAC(config-if)# switchport trunk encapsulation dot1q
ZORAC(config-if)# switchport trunk allowed vlan 1,2
ZORAC(config-if)# switchport mode trunk
ZORAC(config-if)# channel-group 1 mode on


Now GigabitEthernet0/1 and 0/2 are in a 802.3ad team. You may also want to use the below snippet to configure how load balancing will work with your team:


ZORAC# conf t
ZORAC(config)# port-channel load-balance dst-ip


This will balance the traffic going into the server based on its destination IP address. Load balancing settings for traffic going out of the server are decided by ESX and are also configurable. The default is “out-mac”, where ESX load-balances based on the destination MAC address. Using out-ip instead can improve network performance for VMs that produce a lot of network traffic. Traffic gets distributed more evenly across all the links in a team. However, your network switch has to support this. If you’ve got a Catalyst configured as above, then you’ve got the support.

To change ESX’s load balancing to out-ip, do the following:

1) Determine what the name of your team, or, bond is. The easiest way I’ve found to do this is to run this in the service console:


[root@esx root]# cat /etc/vmware/hwconfig | grep bond


You’ll see a few lines appear, mentioning either bond0, bond1 or something similar. Remember which bond it is.

2) Add the following line to /etc/vmware/hwconfig. Check to see if you already have a similar line - I didn’t but you might if you’ve attempted something like this before:


nicteam.bond0.load_balance_mode = "out-ip"


Be sure to put in the correct value for “bond0″.

Now here’s the catch: changes to /etc/vmware/hwconfig are not read until you reboot, and no one wants to reboot an ESX Server. You can activate the change immediately by typing the following command into the service console. Again, replace bond0 with the name of your bond:


echo "nicteaming load-balance out-ip" > /proc/vmware/net/bond0/config


It took a lot of googling for me to figure out exactly what you had to echo into config to make the change immediately. Hopefully this post will make the answer easier to find :)

That’s all! You’ve now got an 802.3ad NIC team running with IP-based load balancing on incoming and outgoing traffic. VMware has published a white paper about this subject, which you can view for more information.

I just finished deploying “Trend Micro Client Server Messaging Suite for SMB”, aka Trend Micro’s networked antivirus product. Despite the convoluted name, it’s AWESOME. I can’t imagine how they could make deployment any easier… I login to the admin web site, select all the computers in my AD domain, and click install. It does the rest. It even uninstalls whatever AV product might be on a computer before it installs itself. (Goodbye McAfee!)

I’ve already had one report from someone saying they’ve noticed that their computer is running faster. That’s Trend’s other claim to fame: low resource utilization. It’s also already found viruses that the other two products have ignored. And the cost for licensing the server and 51 clients was very small, especially compared to what McAfee and Symantec charge for their products.

Trend’s admin tools are wonderful. It’s all done through your web browser (and you can use any browser you like, so long as it’s Internet Explorer….). You can set policies like “no one can uninstall the AV client without a password”, “no one can change the update settings”, etc. It will automatically download updates from the Trend server on your network, to save your outbound bandwidth. If it can’t find a Trend server (for example, a laptop that someone brought home), it will then download over the Internet from Trend directly.

So, in summary, Trend Micro’s AV product is easier to deploy, easier to manage, faster, more efficient, better at catching viruses, AND CHEAPER, than McAfee and Symantec.

I’m really, really happy that I don’t have to deal with Shitcaffe or Pissant-ec ViruScan anymore. The next P-o-S IT product I hope to eradicate from my life is VERITAS BACKUPEXEC, which I’ve ranted on before. I read an article where someone was complaining about the same stuff in BackupExec that I was, and then he went on to mention that he switched to Retrospect’s corporate product and was quite happy. So, today, I hope to try a Retrospect demo. The absolutely disgusting thing is that I can get a competitive upgrade to Retrospect, if I choose to go with the product, for $400, and that includes unlimited licensing for servers and clients. We paid way, way more than that for our Veritas licensing, and we’ll need to pay even more when we add more employees. So, if all goes well, Veritas can go the way of Mcafee.

I was looking through my referrers again and I noticed that VMware, Inc.’s VMTM Blog linked to me. Thanks a lot, guys.

The VMTN is a great resource - especially the white papers and discussion forums. I received a lot of early education from those forums, and they’ve helped me with some really strange problems. The way the forums are set up is pretty cool, too - there is a point system in place to encourage people answer questions as often and as best they can. The question-askers are the ones that award points or not, so the system is extremely fair.

I’m scheduled to take my VCP exam this Saturday at 1:15PM. I have no doubts about how well I’ll do. I’ll describe the experience here once I’ve finished it.

Today was the last day of the “Virtual Infrastructure with ESX Server” course I was taking. It was a wonderful course and the instructor couldn’t have been better. I guess I’m not accustomed to teachers knowing what they’re teaching. Must have something to do with where I went to high school… Anyway, I learned a lot and I love ESX Server even more now. I must get my hands on a fibre channel switch and a VirtualCenter license.

I’ve been looking through my access logs and noticed a few google searches for people looking for information on 802.1q VLAN trunking with VMware ESX Server and a Cisco switch. So I thought I’d post a mini-HOWTO:

Scenario: Trunk 2 VLANs to your ESX Server, and assign them to different VMs. One VLAN will be the native VLAN (1) and the other will be VLAN 50.

1) Define VLAN 50 on the switch:


ZORAC>en
Password:
ZORAC#conf t
Enter configuration commands, one per line. End with CNTL/Z.
ZORAC(config)#int vlan50
ZORAC(config-if)#exit
ZORAC(config)#


2) Set up native VLAN tagging so ESX Server VMs can talk to the native VM through a port-group:


ZORAC(config)#vlan dot1q tag native


3) Set up the trunk to your ESX Server. This snippet assumes that your ESX Server is connected to port GigabitEthernet0/1.


ZORAC(config)#int GigabitEthernet0/1
ZORAC(config-if)#switchport trunk encapsulation dot1q
ZORAC(config-if)#switchport trunk allowed vlan 1,50
ZORAC(config-if)#switchport mode trunk


That’s it for the switch configuration. You may want to do a ‘write mem’ …

4) Using the ESX MUI, define a port group for each VLAN on a virtual switch. Note that you have to make a port group for VLAN 1, not just VLAN 50:

a) Once logged into the ESX MUI, click the Options tab, then click Network Connections.
b) Click “Add…” next to “Port Groups” under “Properties” of your virtual switch.
c) Fill out the form and click Create Port Group.
d) Repeat steps a-c for each VLAN.

5) Open the properties for a VM that you would like to assign a VLAN to. Select the Hardware tab.

6) Click “edit” next to your Network Adapter.

7) Under “Network Connection”, choose the port group you want the VM to be connected to. Click OK. Note that now you’ve created a port group, you should verify that all of your VMs are connected to the port group you want.

That’s it. The cool thing is that your VM won’t even know it’s on a VLAN - after ESX receives a packet, it strips all tagging information before sending it to the VM. As far as the VM is concerned, it’s plugged into a regular switch port.

Some day soon I’ll cover trunking and port channels for NIC teaming and bandwidth load balancing.

Right now I’m at the San Diego Training and Conference Center, taking the “Virtual Infrastructure w/ ESX Server” course. The SDTCC is on the 5th floor of the Wells Fargo building in urban San Diego. I can see a lot of the city from this high up, though the building has many more floors. Anyway, the way they have the class set up is pretty cool… there are a bunch of low-end Windows machines here (provided by the SDTCC), and we all use Citrix to connect to more powerful Windows desktops held at VMware in Palo Alto. From those desktops, we configure and work with our ESX Servers, which are also located in Palo Alto. The setup is great because you can use it from anywhere in the world without having to lug servers around.

We have access to about 6 ESX boxes. Each ESX box is a dual Xeon 3.4GHz with only 2GB RAM, but we’re only making a few VMs on each box. The instructor, Brian Perry, is absolutely wonderful. Finally, someone that can explain something clearly AND concisely! He’s really nice, too. It’s obvious he knows the product inside and out. And he’s not another arrogant IT bastard (like me.)

I absolutely 100% fucking hate Veritas and all of their clunky, bloated products. Due to a stupid glitch in BackupExec, the Job History window now shows over 213,000 failed jobs, and it adds about 10,000 more a day. Why is this a problem? It slows down the whole system. Why can’t I delete them? It won’t let me! You have to select each job you want to delete, and click Delete. You can “Select All”, but after waiting 4 hours for it to actually select all 213,000 jobs, it never finished. So now I’m trying to do it 14,000 jobs at a time. Selecting 14,000 jobs takes about 20 minutes and then deleting them takes another 30 minutes. And I’ll still have almost 200k to go. And all while this is happening, Task Manager says Veritas is only using 25% CPU! I HAVE TWO XEONS FOR GOD’S SAKE! USE THEM!

And let’s not leave NetBackup out of this rant, either. I don’t use it at my current job but I have used it in the past and it sucks too. All Veritas products have this sort of nasty feeling to them, sort of like an Oracle-style bloat with some Microsoft style clunkiness, merged into one stinking shitpile. And their support web site is so, so horrible. I won’t even go into detail on this one, just please, for Jebus, don’t go there.

And now Symantec owns Veritas! What could be better? Now they can release Norton Utilities SystemWorks 2006 Mega Elite Edition with built-in Veritas Technology(tm). It will protect your computer from viruses and spyware and malware and trojans and hackers and spambots and script kiddies and IE exploits and ninjas and old Soviet nukes and so many other things, that you’ll have about 2% CPU power left over to get any work done. And that 2% CPU left will be used to back up your system, including all the crap it blocked. And when your system crashes from running out of resources, you can use some Norton Utilities piece of crap to fuck it up more.

If I didn’t have to get up at 6AM tomorrow for VMware training, I’d be completely uninstalling Veritas and reinstalling it right now, just to clear the fucking database. But I can’t, because it’ll keep me up too late. And if there’s one thing I hate more than anything else, it’s leaving ANY problem unresolved.

I’ve made a prioritized list of what certifications I want to pursue after VCP. MCSE and CCNP. Why the Microsoft one first?

It probably comes as a surprise to those who know me that I’d be pursuing a Microsoft certification in the first place. It still shocks me. What shocks me even more is that I use Outlook with an Exchange server… and I like it (!!!). I’m very much a pro-opensource-software person, and I prefer to use open code everywhere I can. But there is simply no open-source package out there that works as a drop-in replacement for Active Directory and Exchange. It needs to be a drop-in replacement because many people will keep using Outlook. It’s easy to change the server software; you do that on just a few machines. But changing the email client for every employee - with many employees not wanting to change - doesn’t work so well. Yes, Exchange has a bit going on behind closed doors that makes its functionality hard to replicate. But it’s not too hard - it uses IMAP with a few tweaks for just about everything, including the calendaring. And then there’s the Blackberry - a device I’ve hopelessly fallen in love with - that works very well if you have a Blackberry Enterprise Server, which only works with Exchange (or Notes).

So in other words, switching to a different mail server that isn’t exactly compatible with everything else that Exchange works with, is not an option. (Someone out there, PLEASE make a drop-in Exchange/Active Directory replacement - I worked 60 hours in 3 days just a few weeks ago recovering from a problem that I’d never have with a Linux-based solution. I’ll pay anything for such a product.)

Why get an MCSE then? Because as you can see, my business is invested so deeply into the functionality Exchange provides that I have no choice but to learn it and support it very well. And supporting it well requires knowing the underlying operating system well. For a long time I, like many other pro-*nix people, thought Windows was just an ugly graphical interface bolted on to a bunch of black magic, all hidden from anyone, including administrators. It turns out that’s not entirely true, at least not anymore … Microsoft provides tools for getting under the hood of just about every subsystem that makes up an Active Directory / Exchange deployment. But they’re all esoteric and don’t follow any preexisting nomenclature - they’re all Microsoft-specific. You can learn a lot through work experience (I learned a frightening amount about Active Directory when recovering from that disaster I mentioned earlier) but there’s still a lot that you have to go to the Knowledge Base for, or call Microsoft Support for (at $245 a call).

I could learn all of this on my own, but I’d rather learn it all now, and maybe make Windows a little less unreliable at the same time. I’m far more passionate about obtaining Cisco knowledge, but I realize I know more than enough to keep our Catalysts and Aironets going, and I don’t even support our router (our ISP does). So Windows is the weakest link in my support skills. I wouldn’t say that I’m a weak Windows admin, but it’s the least honed skill of all the IT stuff I deal with. It takes a lot for me to admit that. And it takes even more to actually want to do something about it.

A few months after the VCP thing is over, I plan on finding some online MCSE courses. CDW seems to have a pretty good deal. I’ve looked into the testing requirements and it’s pretty involved - 6 tests on different subjects, a lot more than the VMware certification, but then again the Windows world is a lot bigger.

One last plead: Open-source companies, hackers, I implore you: please save me from Windows. Save me before it’s too late. Save me before I have to run adsiedit again. PLEASE MAKE AN EXCHANGE REPLACEMENT!

Starting this Tuesday at 9AM, I’ll be in urban San Diego receiving training for my VMware Certified Professional certification. The specific class I’m taking is called “Virtual Infrastructure w/ ESX Server & VirtualCenter”. The course lasts 4 days, ending Friday at 5PM.

(Here’s a quick treatise on VMware and hardware virtualization: basically, the software lets you have a pretend computer, existing entirely in software, running inside your real computer. And inside the pretend computer, you can run any PC software you’d normally run. All of VMware’s products provide this functionality. They differ only in focus: ESX Server is meant for running lots and lots of VMs on a single beefy physical server; Workstation is focused more on bleeding edge features and desktop-level tasks).

The main focus of the course is VMware’s “datacenter-grade” hardware virtualization software, which they call ESX Server. I can say, without any doubt, that this is the most amazing product I’ve ever used. There are so many benefits to using it that I won’t even bother listing all of them here, but a few of them are:

• Server consolidation - I’ve already replaced 4 physical servers with 1 ESX Server, and that server has room for probably 5 more machines. This is all on a regular Dell PowerEdge 1850, dual 3GHz Xeon, 8GB RAM. The only thing exotic in the hardware configuration is a slightly high amount of RAM. My main ESX server has 2 Windows 2003 servers running (Exchange and AD) and two Linux servers (one is a DMZ email relay and the other is a general purpose Linux server on our internal LAN).
• Disaster recovery - Every night I have a script that runs that makes exact duplicates of all 4 servers, while they’re running. The whole process takes about half an hour. I could recreate all 4 servers in a matter of minutes from these backups, even if the entire data center was destroyed. An entire server, including its hard drive and system configuration, exists solely as a handful of files, which can be moved around like any other files.
• You can trunk 802.1q VLANs into the ESX Server and assign VMs (virtual machines) aribtrarily to any VLAN you like, without having to use port-based VLANing.
• You can give VMs CPU and RAM quotas; you can even over-commit CPU and RAM and the system uses a pretty nifty contention algorithm to decide who has to slow down. In addition, it can tell if two VMs are allocating memory for the same piece of code, and only actually physically allocate it once.
• There’s another feature called VMotion which I haven’t had the joy of using yet because I don’t have a full SAN (yet). It allows you to migrate one VM to a separate ESX Server, while the VM is running, without shutting it down, in a matter of seconds. This is great if you need to reboot a server to add more RAM, or if there is some sort of impending doom.
• I can access the console of any VM through the included VMware Remote Console program, available for Windows and Linux (no Mac support, unfortunately). This gives me free KVM-over-IP functionality.

I’ve been playing with VMware’s stuff since VMware Workstation 2.0, which came out quite a while ago. Up until I discovered ESX Server, I thought of VMware as a pretty handy tool for certain situations, but I was never totally obsessed with it. I am now. Taking this course is just another way of embracing my obsession :)

If you haven’t tried any of VMware’s products, check out their web site for more information. You can get 30-day free trials for VMware Workstation and ESX Server by filling out a simple form. The only “gotcha” with ESX Server is that it is picky about the hardware it will run on, and there is the minimum requirement of two CPUs. VMware Workstation will run on almost anything.

This is my fifth personal web site and my third blog. Like most every other blogger, I go through periods of blogging and not blogging. All three of my blogs have had a more specific focus than the generic “this is what I ate for breakfast” topic. In the past I’ve written about R/C helicopters and the different books I’ve read, but on this new blog I plan on writing about the amazing technology I get to work with on a daily basis at my job. I’ve also gone a bit more “out there” with the design and color scheme, as you’ve probably already noticed. It sort of reminds me of a BBS, which makes me happy. Also worth mentioning is the “Current Obsession(s)” section on the right, which is something I plan to update with the parts of my work I am currently obsessed with.