Enterprise Backup Network with ANIRA

One of the most critical if not the most critical component of the IT infrastructure is the network, although many times taken for granted. In today’s Client-Server environment and even more with the Cloud Computing model, offices without connectivity to the network become useless in trying to carry out their daily business.

If your business or part of your business is disconnected from the others, it will impact your business in a significant way not including making your customers angry.

This post goes into what it takes to implement a cost-effective backup network, should the primary network link fail.

The scenario described includes multiple remote offices or field locations connected via bonded T1 circuits to an MPLS network. All major services are provided to these remote offices through a central location which is almost always the case, making an outage fatal to the remote office.

Despite redundant T1 circuits providing an aggregate of 3Mbps to the remote office, CRC errors or physical errors on one of the circuits will bring the bonded circuit down; so relying on the 2nd circuit active circuit as backup is a flawed approach.

The router performs only WAN functionality, leaving all other routing and VLAN based-network segmentation and security within the office to a layer-3 capable switch.

The routing protocol of choice is BGP as it is natively used by the MPLS network.

The backup link we are looking for would need to be cost effective, meaning it should not add to the bottom line significantly until it is needed. It would also require sufficient bandwidth for data and voice applications to be ran at an acceptable level from the remote office.

AT&T provides a product that fits this description called ANIRA (AT&T Netgate). There is a minimal monthly rate, a cap of 1Mpbs aggregate bandwidth and additional charge for usage.

This could be done with off-the-shelf equipment in lieu of the ANIRA product but this approach requires additional challenges such as creating the VPN tunnels to equipment at the main office and correct propagation of routes when the main circuit at the remote office goes down. This AT&T service provides the management of the backup devices as well as the connectivity through a VPN tunnel into the MPLS cloud.

The image above illustrates the network topology.

Should the remote office loose  network connectivity, traffic will start to flow through the Netgate which will trigger the device to connect and initiate a VPN tunnel advertising all routes belonging to that office into the MPLS network.

The routing protocol used to determine which path, traffic will take is VRRP or Virtual Router Redundancy Protocol. This will allow the default route used by the switch to float between the main router and the backup device.

Cisco configuration outlined below:

track 1 interface Multilink ip routing

interface FastEtherner0/0
description Internal Network
ip address
duplex auto
speed auto
vrrp 1 description LAN
vrrp 1 ip
vrrp 1 preempt delay minimum 60
vrrp 1 priority 110
vrrp 1 track 1 decrement 100
arp timeout 60

The Netgate device has an IP address of and a VRRP IP address of

A brief description of relevant configuration below:

The VRRP IP address floats between the routers (main router/Netgate) depending which one has the highest priority. The Netgate has a default priority or weight of 50 and an additional 25 when the VPN is connected. In a normal state we want to main router to handle traffic so we force a priority to anything higher than 75 which is the maximum for the Netgate.

vrrp 1 priority 110

To be in a position to decide if the default route should move to the Netgate, we need to know if the T1’s are down. In this example having a T1 down should not be a deciding factor because there is an additional T1 that can handle the traffic, so we chose to monitor the bonded interface at the IP layer.

track 1 interface Multilink ip routing

In the event of an outage the main router will need to lower its priority or weight, below the priority of the Netgate, so that it becomes the new default router with IP address

vrrp 1 track 1 decrement 100

This event will bring the main router’s priority to 10, well below the minimum for the Netgate.

When the main circuit comes back online we want to switch back to it and bring down the VPN tunnel. We accomplish this using the following command: vrrp 1 preempt

However when a T1 comes back up, its usually not a clean process and the telco might also be performing intrusive testing; so its important that we allow some time before we switch traffic back to the main circuit.

vrrp 1 preempt delay minimum 60

Using this configuration should be able to provide an automatic redundant backup network link for remote offices at an affordable price.

Educause 2008

This years Educause conference took place in Orlando, Florida.

Educause is a nonprofit association whose mission is to advance higher education by promoting the intelligent use of information technology. Membership is open to institutions of higher education, corporations serving the higher education information technology market, and other related associations and organizations.

The association provides a social networking Connect site that supports blogs, wikis, podcasts and other platforms for IT professionals to generate and find content and to engage their peers; professional development opportunities; print and electronic publications, including e-books, monographs, and the magazines Educause Quarterly (EQ) and Educause Review[1]; strategic policy advocacy; teaching and learning initiatives; applied research; special interest discussion groups; awards for leadership and transformative uses of information technology; and a Resource Center for IT professionals in higher education.

Major initiatives of Educause include the Core Data Service, the Educause Center for Applied Research (ECAR), the Educause Learning Initiative (ELI), Net@EDU (advanced networking), the Educause Policy Program, and the Educause/Internet2 Computer and Network Security Task Force. In addition, Educause manages the .edu Internet domain under a contract with the U.S. Department of Commerce.[1]

The current membership of Educause comprises more than 2,000 colleges, universities, and educational organizations, including 200 corporations, with 16,500 active members.

Below are pictures from the conference:

[slickr-flickr tag=”educause 2008″ id=”61116089@N00″ group=”n”]

My schedule at the conference:

Tuesday, October 28, 2008

Wednesday, October 29, 2008

Thursday, October 30, 2008

Friday, October 31, 2008

Overall I thought it was an excellent conference, there weren’t as many people this year as previous ones.

The exhibit hall was fun as always. Some exhibits were great and others sucked which brings up another subject. Marketing.

There were two exhibits that stood out amongst the crowd. The first one from Bradford Networks and the other from Trapeze Networks. These guys not only gathered leads, but engaged their prospective customers allowing them to deliver their sales pitch. Two companies that I will definitely be following up with.

Other companies that did well on their marketing pitch were Turning Technologies, Novell, CDW, Zimbra, Elluminate, and Microsoft. Although the only thing Microsoft had going for itself was as great demo on a smart-board of Image Composite Editor.

Microsoft Image Composite Editor is an advanced panoramic image stitcher. The application takes a set of overlapping photographs of a scene shot from a single camera location and creates a high-resolution panorama incorporating all the source images at full resolution. The stitched panorama can be saved in a wide variety of formats, from common formats like JPEG and TIFF to multi-resolution tiled formats like HD View and Silverlight Deep Zoom.

The things that characterized the good exhibits can be summarized in a few words. They were accessible, had an inviting environment, gave away free stuff (like free iTouch and laptops every hour) and had either professionals or very seasoned sales people giving the presentations.

On the other side of the coin, were the very big and expensive exhibits which just didn’t deliver.

Some that deserve mention are AT&T which has a very expensive three environment exhibit representing campus life and U-Verse all over the place. Alcatel-Lucent had a not very inviting exhibit and their staff sat down most of the time. Citrix was just offering a $5 Starbucks card for filling out a survey. Cognos had a closed exhibit that wasn’t inviting to anyone.

Its not that these companies were cheap, which they were; but they are spending a lot of money for lead generation when they could also be qualifying the leads and delivering their product demos to a captive audience.


Poor Man’s Disaster Recovery

Backups are probably the most tedious, time consuming jobs for a system admin and often regarded as a low priority until something goes wrong.

Hell breaks loose and you stumble around for tapes, building catalogs, restoring data, finding unusable tapes or corrupt data and looking for excuses or stories to tell management.

Last month I discussed personal backups and disaster recovery here.

I have added to my arsenal of tools an application called SyncBack which I run at the least every couple of days on all my data including the “My Documents” and “My Documents and Settings” folders making sure I have my data and settings backed up to an external USB drive.

I also use Mozy to have a historical backup of critical files, which has come very handy. Mozy provides 2 Gb for free of backup and have paid plans for additional storage. A client is installed on the computer and pretty much takes care of everything once its configured. Other players in this area include Carbonite, HP Upline, IDrive, SOS Online Backup, and Symantec Online Backup.

Disaster recovery is not about backups and what the quickest way to restore those files are, but rather to plan for the worst and how will you continue to operate if the unforseeable happens.

In a small business for example, its rare to have more than a server which serves as a print server, a file server, an e-mail server, a blackberry server, an application server, etc, etc. Even if there is another server or two they are all running several applications, so redundancy is not something that’s viable nor affordable for a small business.

OK. So backups are getting done. Whether they are being backed up online, to tape locally or to disk. You want a quick restore, then go for disk over tape.

Everything is Kosher….. not so fast.!

What would happen if the server had a major failure? Not something quickly addressed by ordering a replacement part. Could you put your clients on hold for a couple of weeks until a new server arrives?

What if there was a fire? What if someone broke in and stole the server?

That small business would most probably cease to exist if its operations depended heavily on the use of technology.

The same principles used in bigger businesses when it comes to disaster recovery, appear to be more critical to smaller businesses. Having a disaster recovery site where the server could be mirrored in the event of a loss.

What a better place than the small business owner’s home.?

So the challenge is to mirror a server located at the office with a server located at home. Sounds like something definitely out of reach for a small business, since it involves possibly duplicating licensing costs, software costs for mirroring and then there’s the issue of dealing to the caps for uploads on almost any broadband provider, which generally puts the bandwidth available at 512k or less.

rsync is a software application for Unix systems which synchronizes files and directories from one location to another while minimizing data transfer using delta encoding when appropriate. This program is ideal since it reduces the data transferred to a minimum over a limited link.

DeltaCopy is an open-source backup program port of rsync to Windows. It has several features which make it ideal for the task at hand including installs as a service, incremental backups, task scheduler, and e-mail notification.

DeltaCopy is installed on both the main server and the backup server. The backup server is configured with DeltaCopy running as a service and if encryption is required, a tunnel over ssh can be accomplished by installing an ssh-server using Cygwin for emulation.

The backup server will require DynDNS to make sure that the main server can reach the backup server by name. A couple of ports (873 (rsync) and 22 (ssh)) will also need to be forwarded on the DSL/Cable router on at the backup server side.

Then schedule and sleep well knowing you have a “Disaster Recovery” plan.


How to install a ssh server
Set up a personal, home SSH server


A laptop and disaster recovery

Over the last couple of years I have had several severe laptop failures, which have at least put me back a couple of weeks not to mention the hassle of re-installing the operating system and all the applications, which for me is in the 70s.

This time I took some action and looked into backing up all my data on a daily basis from my data partition and creating an image of my c partition on every major changes. (once a week or once every other week.)

Being able to boot my laptop with a CD or USB flash drive and plugin by FreeAgent Seagate to easily create an image was the goal. The gold standard in cloning drives is Symantec Ghost, which would not see my NTFS FreeAgent Seagate external usb drive no matter how much I tried. Several other commercial products including Acronis True Image, ImageCenter and Partition Saving and none of the above were up to the task.

Enter open source. SystemRescueCD is Linux system on a bootable CD-ROM for repairing your system and recovering your data after a crash.

Going a little bit further I was able to put SystemRescueCD which comes with a great number of drivers that will recognize literally anything on a flash drive.

After booting from the usb flash drive with SystemRescueCD, I started an Xwindows session by typing startx at the prompt.

Then the time came to mounting my external usb drive and imaging the C partition.

  • lsusb (to list all USB devices)
  • dmesg | grep -i “SCSI” (which would give you a list of devices attached to the laptop, in my case the Freeagent Seagate usb external drive – in my case sdb1)
  • mkdir /seagate (create a mount point for the usb external drive)
  • mount -t ntfs-3g /dev/sdb1 /seagate (mount the ntfs external usb drive with read/write permissions)
  • partimage (this launches the cloning application which allow you to create an image of the C partition)
  • umount -f /seagate (unmounting the external drive or any other drive for that matter in a linux/unix environment is extremelly important before unplugging)

Now to schedule my data backups on a daily basis with SyncBack.