Ok, so it’s been a while, but I’ve been very busy building a few new products Honestly I don’t know how the other bloggers manage to get time to blog if they’re actually doing work as well, but I digress….
The task laid upon me was pretty simple in it’s definition:
“Provide one or more ways for a customers to replicate their on-premise VM’s to a cloud provider in a scalable and secure manner”
… simple, right? Not so much….
The problem is, pretty much none of the software vendors understand real multi-tenancy (let alone implement it in any sort of serious way). Every product we tested fell way short and after talking with SE’s at various vendors, it was pretty clear we’re on our own. Undeterred, it was time to get my hands dirty.
This post pretty much goes my thought process when designing this particular solution and some hints on implementing it yourself. Hopefully you find it interesting / useful.
We ended up settling on supporting Veeam as the first product to target. In the SME space, Veeam is already wildly popular; so it made good commercial sense to support it if we could. We’ve also had experience automating Veeam and just generally know the product pretty well…
The problems to overcome with Veeam
Veeam (like most other software…) pretty much assumes all infrastructure is owned /controlled by the same organization, it’s designed with that in mind and it’s very difficult to ‘tack on’ multi-tenancy later.
So with Veeam, the software can either be installed in one of two places:
- At the Source (customer site): means a “push” type architecture, Snapshots are pushed up to a remote ESXi host.
- At the Destination (Cloud-Provider): Means a “Pull“ type architecture, Snapshots are requested and pulled from the client site.
Problem is: Veeam needs a high-privilege level on the source VM’s, datastore and target ESXi host, datastore and resource pools. Things like ‘register VM’, delete VM, revert to snapshot @ the cluster or host level… Obviously, no cloud-provider in their right mind is going to give that sort of access to a third party (a customer) and just trust them to not delete someone elses VM on the target. Conversely, no customer would ever agree to giving a cloud-provider access to their internal VM infrastructure to ‘pull’ the backups, nor should they have to…..
The later a pproach also means the cloud-provider needs to securely store credentials to the customer-site which could have close to root-level access….. Across a large number of clients, that represents a massive risk and frankly, its probably just a disaster waiting to happen. Rather than try to build a secure place to store that sort of info… why not just avoid storing it at all.
In the end we decided on going with a “Push-Type” architecture for a number of reasons:
- Pulling customer data represented an unacceptable risk in credential storage.
- No clear demarcation point in the infrastructure.
- Realistically; Troubleshooting backup/snapshot etc issues in the customer environment would become our problem w/ a “pull-type” architecture.
So “Push” it is, then…
So it’s decided, we want to give the customer access to our SAN and prod ESXi server…. yeh, not so much!
That presents another problem;
Q: How do we give customers access to an ESXi API and some storage to replicate to? But not our production ESXi nodes and preferably not directly to our SAN either.
A: Nested ESXi !?
In this particular use-case, nested ESXi really makes a lot of sense:
- They can be rapidly provisioned on demand.
- Access / permissions can be limited to each tenant/customer individually.
- They can be easily ‘dual-homed’ onto the individual customer’s existing VLAN within our environment.
- We know how to manage ESXi servers.
- They’re obviously well integrated with vmware Orchestrator.
- We know how to use the vSphere API’s / have a library of workflows already….
Figure 1a (Above) shows how the infrastructure fits together and hopefully raises a few questions:
Q: What’s with the Static MACs?
A: Good question, Watson! There’s a few reasons, actually.
A1: We’re deploying from templates, w/ vmk0 using DHCP. Setting a static MAC on the primary vmnic allows us to preallocate the IP for the management interface and know what IP to connect to in the deployment workflows etc.
A2: Static MACs allow us to use arp-locking @ the switch level to prevent customers stealing other customer’s IPs (even if they gain root-privilege and SSH access to the nested host… RTSM is obviously disabled by policy, by the way).
A3: If the MAC of the adapter matches the VMK that’s bound to it, it’s not necessary to enable Promiscuous-mode on the portgroup(s). Further info about the ‘normal’ nested ESXi deployment available here.
Q: vmk0 is marked as “following”, what’s that mean?
A: vmk0′s MAC will match the first physical adapter (in this case, that’s actually a virtual adapter). This is achieved using the command: “esxcfg-advcfg -s 1 /Net/FollowHardwareMac” described in KB1031111. It’s worth noting, this is the default behaviour when installing ESXi. This setting allows us to deploy from a template w/ a Static MAC set on the primary ‘physical’ adapter and have vmk0 pickup the correct DHCP address.
Q: Why NFS?
A: for this use-case, we want everything to be ‘easily’ automated during deployment. NFS provisioned through a Nexenta appliance allows us to achieve this.
Q: So what happens when I want to power-on my VM on the nested host, surely the performance would be terrible?
A: Yes, it would…. luckily, we’ve thought of that…. read the next section
Q: Routing would be a problem wouldn’t it? Where’s the default gateway?
A: Gateway is set on the customer network and we use a couple of fairly tight static routes to route back to the infrastructure we need to (vcenter, AD, DNS etc).
So what happens when the customer needs to start the VMs…..
Well, that’s the million-dollar question isn’t it! Basically, that’s really the clever part (and where most of the effort in automating went) :)
In short, we start the VM’s on real ESX servers and give the customer access to them the same way we do with Virtual Private Server offering (VPS)….
We use web-front end, integrated into our portal, vmware orchestrator workflows to do it. The high-level goes something like this:
- Customer logs into ZG portal.
- List of VM’s registered on their Replication Target, list of available networks presented to customer dynamically.
- Customer selects VM’s to Failover (or test Failover), the destination VLAN (selected from a list of their services).
- Customer pushes ‘the big red button’
- Selected VM’s are unregistered from the Replication Target (noting down the storage path of the VMX etc).
- Customer’s NFS DataStore is mounted to a production host with sufficient resources.
- VM’s are registered to Prod vCenter.
- VM’s are reconfigured to the destination VLAN, permissions added for the customer’s production VPSUSERxxxx account.
- VM’s are powered on.
- Storage vMotion to Production LUNs (iSCSI) initiated.
- Unmount NFS DataStore from prod host when svMotion completes.
- FAILOVER Complete.
So hopefully, that gives you a little insight into what’s possible with nested ESXi hosts with a bit of lateral thinking. I found the project to be really rewarding (certainly challenging!) and its projects like these that makes me love my job.
I’d like to thank Nicki Pereira (ZettaGrid General Manager) for allowing me to post this and having the vision to allow the project in the first place.
Shameless plug: The Replication Target is one of ZettaGrid’s showcase products, part of the “ZettaGrid Replication Service” suite. The suite is launching into Public-Beta @ CeBIT next week (22-05-2012). I’ll be presenting an overview on Thursday in the Cloud Theatre and will be floating around talking to anyone who’s interested. Come say hello (world)!