Enterprise Backup Network with ANIRA

One of the most critical if not the most critical component of the IT infrastructure is the network, although many times taken for granted. In today’s Client-Server environment and even more with the Cloud Computing model, offices without connectivity to the network become useless in trying to carry out their daily business.

If your business or part of your business is disconnected from the others, it will impact your business in a significant way not including making your customers angry.

This post goes into what it takes to implement a cost-effective backup network, should the primary network link fail.

The scenario described includes multiple remote offices or field locations connected via bonded T1 circuits to an MPLS network. All major services are provided to these remote offices through a central location which is almost always the case, making an outage fatal to the remote office.

Despite redundant T1 circuits providing an aggregate of 3Mbps to the remote office, CRC errors or physical errors on one of the circuits will bring the bonded circuit down; so relying on the 2nd circuit active circuit as backup is a flawed approach.

The router performs only WAN functionality, leaving all other routing and VLAN based-network segmentation and security within the office to a layer-3 capable switch.

The routing protocol of choice is BGP as it is natively used by the MPLS network.

The backup link we are looking for would need to be cost effective, meaning it should not add to the bottom line significantly until it is needed. It would also require sufficient bandwidth for data and voice applications to be ran at an acceptable level from the remote office.

AT&T provides a product that fits this description called ANIRA (AT&T Netgate). There is a minimal monthly rate, a cap of 1Mpbs aggregate bandwidth and additional charge for usage.

This could be done with off-the-shelf equipment in lieu of the ANIRA product but this approach requires additional challenges such as creating the VPN tunnels to equipment at the main office and correct propagation of routes when the main circuit at the remote office goes down. This AT&T service provides the management of the backup devices as well as the connectivity through a VPN tunnel into the MPLS cloud.

The image above illustrates the network topology.

Should the remote office loose  network connectivity, traffic will start to flow through the Netgate which will trigger the device to connect and initiate a VPN tunnel advertising all routes belonging to that office into the MPLS network.

The routing protocol used to determine which path, traffic will take is VRRP or Virtual Router Redundancy Protocol. This will allow the default route used by the switch to float between the main router and the backup device.

Cisco configuration outlined below:

track 1 interface Multilink ip routing

interface FastEtherner0/0
description Internal Network
ip address 192.168.0.2 255.255.255.0
duplex auto
speed auto
vrrp 1 description LAN
vrrp 1 ip 192.168.0.1
vrrp 1 preempt delay minimum 60
vrrp 1 priority 110
vrrp 1 track 1 decrement 100
arp timeout 60

The Netgate device has an IP address of 192.168.0.3 and a VRRP IP address of 192.168.0.1.

A brief description of relevant configuration below:

The VRRP IP address 192.168.0.1 floats between the routers (main router/Netgate) depending which one has the highest priority. The Netgate has a default priority or weight of 50 and an additional 25 when the VPN is connected. In a normal state we want to main router to handle traffic so we force a priority to anything higher than 75 which is the maximum for the Netgate.

vrrp 1 priority 110

To be in a position to decide if the default route should move to the Netgate, we need to know if the T1’s are down. In this example having a T1 down should not be a deciding factor because there is an additional T1 that can handle the traffic, so we chose to monitor the bonded interface at the IP layer.

track 1 interface Multilink ip routing

In the event of an outage the main router will need to lower its priority or weight, below the priority of the Netgate, so that it becomes the new default router with IP address 192.168.0.1.

vrrp 1 track 1 decrement 100

This event will bring the main router’s priority to 10, well below the minimum for the Netgate.

When the main circuit comes back online we want to switch back to it and bring down the VPN tunnel. We accomplish this using the following command: vrrp 1 preempt

However when a T1 comes back up, its usually not a clean process and the telco might also be performing intrusive testing; so its important that we allow some time before we switch traffic back to the main circuit.

vrrp 1 preempt delay minimum 60

Using this configuration should be able to provide an automatic redundant backup network link for remote offices at an affordable price.

ISDN to IP Video Conferencing Migration

Over the last several months I have been hard at work on a migration which has been in the works for several years.

The goal is to move video conferencing transmissions from ISDN 128kbps bonded calls to 384kbps IP calls, in order to improve the quality of the video and cut long-distance phone costs.

Over 20 hours a week classes are transmitted from a lecture hall in Fort Worth, Texas to 20 sites across the United States. The equipment in place is a Polycom VS-4000 video conferencing unit which has input from multiple cameras, an Accord MGC-100 video conferencing bridge and 2 PRI lines coming into the bridge.

Tandberg and Polycom 128 units at the remote site dial bonded 64Kbps channels to achieve a 128kbps call.

The original plan called for fractional T1 circuits at every remote site all furbished by a single ISP in order to be able to assure quality of service from point to point. A fractional 512kbps T1 would provide sufficient bandwidth for a 384kbps call plus the overhead and the bridge would be connected to a fractional DS3 circuit (around 12Mbps).

The scope of the project grew and for one reason or another the remote site circuits became a full T1 (1.544 Mbps) circuit and the host became a full DS3 (45Mbps) circuit.

To complicate things further wireless network/Internet access, routed back to the hosting site would be provided for all the remote sites for future exam taking.

Network wise the host site will have a Cisco 7204VXR with a channelized DS3 card and each site would have a Cisco 1841 with T1-DSU card and a 4-port Ethernet card.

Quality of service would prioritize h323, rtp, rtsp and sip traffic over any other and wireless access points (Aruba Network AP-65) are every site would tunnel encrypted traffic back to a Aruba Network MMC-6000 Controller.

H323 traffic has always been tricky with firewalls and I anticipated that the problems encountered would be in that area as years of experience had taught me. I was pleasantly surprised this wasn’t the case.

The Aruba Wireless controller at the host site builds IPSec tunnels to all the network access points at the remote sites, allowing students to access resources at the host site securely while at the same time preventing ad-hoc users from having access.

Technical challenges actually came from this area of the project were the site routers provided the access points with DHCP options 60 with the value “ArubaAP” and option 43 with the value of the IP address belonging to the Aruba Controller.

In order for this communication to take place, several ports needed to be allowed from the remote site to the host site. TFTP (UDP 69) for downloading configuration files, PAPI (UDP 8211) Aruba Management protocol, GRE for the IPSec tunnel, syslog (UDP 514) for sending logs, ntp (UDP 123) for keeping time and FTP (tcp 21) for downloading firmware.

Routing was carefully examined and firewall rules were put in place but nothing happened. The access points would not connect successfully with the controller so it was time to crack out the sniffer and start looking at the packets sequence from a successful connection between the controller and an on-site access point and what the packets looked like from a remote site.

Lots of cups of coffee later I found that the Aruba Wireless Controller was receiving packets from the Access Points looking for its configuration, but the controller was answering on a different IP address to the AP.

An additional rule on the firewall allowing traffic from that second IP address on the controller (not the management IP) to the network the wireless access point was at using PAPI (udp 8211) fixed the issue.

Success! A very satisfying feeling.

Enter quality of service management which I am sure will be the next opportunity to excel.

Reblog this post [with Zemanta]

Web Conferencing With Dimdim

For a while I’ve been wanting to write several articles on the power of open source and its potential covering multiple software applications that I have run into and this is definitely on of those cases.

In this economical downturn, the use of open source will be more attractive than ever as a strategy to keep costs under control when being asked to do more with less.

This industry was defined and dominated by a company called Webex in the mid nineties which was later acquired by Cisco Systems. Although a very powerful application, it remained accessible to only those who could afford its high price tag.

Over the years several companies tried unsuccessfully to dethrone Webex, which remained intact most probably due to its reliability and stability.

In 2004, Citrix Systems brought the capability of performing web conferencing to the desktop cornering an untapped consumer/smb market and reigning king.

At the time GoToMeeting emerged, WebEx, LiveNote and others catered mostly to large corporations and sales divisions, entering in six-figure contracts. Citrix Online released GoToMeeting on an “all you can meet” basis, with one monthly (or annual charge) based on the number of authorized hosts. This pricing model was unique at the time, but has since been copied by competitors.

Late 2006 I started looking at open source alternatives to the Webex’s of the world and stumbled upon Dimdim while browsing through the goldmines of Freshmeat and Sourceforge.

The software at that point was still in alpha version 1.6. Installation was pretty straight forward once tomcat was installed and a plus was the possibility of integration with Moodle, an open source Course Management System (CMS).

Unfortunately the stability of the package was not there. Another package I looked at was Yugma which is a web based web conferencing service. Again it just wasn’t there.

Two years later and Dimdim has gone from Alpha to Beta and now Dimdim has exited Beta with version 4.5.

Dimdim‘s installation is far more complicated than earlier versions requiring several Python packages, and building and compiling other applications that support Dimdim. My first attempt at performing the installation was unsuccessful but a VM Appliance which is also provided under GPL3 license came up without a hitch.

The web service Dimdim works right out of the box and appears to be reliable and stable. Scalability will be my next test on this VMware appliance with 1Gb of RAM, to determine if it can handle 2-3 conferences and upward of 50 users.

Promising features include integration with other open source industry leaders.

Dimdim’s commitment to open source software development is supported by integrations with industry-leaders:

  • Zimbra: Dimdim now offers a free zimlet for Zimbra’s open source email system;
  • Moodle: Dimdim is integrated with version 1.9 of Moodle’s Course Management System;
  • SugarCRM: Dimdim is integrated with the leading open source customer relationship management system,
  • Claroline: Dimdim is embedded within with the collaborative learning environment.

[ad]

Hard SIP Phones – Cisco 7960 VoIP

Cisco 7960

Cisco 7960

Received my Cisco 7960 SIP phone and althought it was tough to get the firmware (7.4) [P0S3-07-4-00], it was pretty straight forward to get the phone running.

MORE…

Basically the configuration had to be erased, the phone unlocked and pointed to the asterisk server where the tftp daemon is and the firmware lies.

UPDATE:

Run the ‘setup-cisco’ command on the asterisk box to create the Default.cnf file.

The firmware here is essential. You MUST have this firmware in order for the Cisco phone to do SIP.

All the files needed *.bin, *.sbn, *.loads, and *.sb2 need to be located in the /tftpboot directory of the asterisk box and all files need to have 777 permissions.
The phone will need to be reset to factory defaults and set to pick up an IP address from the dhcp server. Once this is done the Alternate TFTP server will have to be keyed in pointing to the asterisk box. Doing a netstat -a on the asterisk box will show (udp 0 0 *:tftp) to make sure that the tftp daemon is listening.

Using the web GUI create a new phone configuration file. Use the MAC address of the phone which is located on the bottom of the phone. Edit the config file and update the Auth Name and password which you will use when setting up the extension later. Disconnect the phone and hopefully it will upload the firmware.

If you have problems then temporarily copy the files in /tftpboot/cisco_util/ /tftpboot/ and then once the phone has upgraded the firmware remove the the files.

At the end the /tftpboot directory should look like this:

drwxrwxrwx 2 root root 4096 Apr 29 2005 cisco_util
-rwxrwxrwx 1 root root 104 Jun 18 19:45 dialplan.xml
-rwxrwxrwx 1 root root 9570 Jun 18 19:45 merlin2.pcm
-rwxrwxrwx 1 root root 14 Jun 18 19:45 OS79XX.TXT
-rwxrwxrwx 1 root root 15 Jun 18 19:45 OS79XX.TXT.bk
-rwxrwxrwx 1 root root 129416 Jun 18 19:45 P003-07-3-00.bin
-rwxrwxrwx 1 root root 129820 Jun 18 19:45 P003-07-3-00.sbn
-rwxr-xr-x 1 root root 129472 Jun 18 19:45 P003-07-4-00.bin
-rwxr-xr-x 1 root root 129876 Jun 18 19:45 P003-07-4-00.sbn
-rwxrwxrwx 1 root root 459 Jun 18 19:45 P0S3-07-3-00.loads
-rwxrwxrwx 1 root root 592414 Jun 18 19:45 P0S3-07-3-00.sb2
-rwxrwxrwx 1 root root 582861 Jun 18 19:45 P0S3-07-3-00.zip
-rw-r–r– 1 root root 592222 Jun 18 19:45 P0S3-07-4-00.bin
-rwxr-xr-x 1 root root 461 Jun 18 19:45 P0S3-07-4-00.loads
-rwxr-xr-x 1 root root 592626 Jun 18 19:45 P0S3-07-4-00.sb2
-rwxrwxrwx 1 root root 26 Jun 18 19:45 RINGLIST.DAT
-rwxr-xr-x 1 root root 2335 Jun 18 19:45 SEP.cnf
-rwxr-xr-x 1 root root 2335 Jun 18 14:59 SEP.cnf.xml
-rwxrwxrwx 1 asterisk asterisk 2612 Jul 9 18:31 SIP.cnf
-rwxrwxrwx 1 root root 4074 Jul 9 18:29 SIPDefault.cnf
-rwxrwxrwx 1 root root 30 Jun 18 19:45 syncinfo.xml

UPDATE 2: (Thanks to Matthew Cochrane)

Upgrading factory SCCP load to current 7.4 SIP release:

1. Download 7.4 SIP firmware, and 7.1 (Cisco call manager) firmware (best I could ‘find’)
2. Unzip, and place both firmwares on your *@Home server in the tftpboot directory (copy 7.4 first, then 7.1) (winSCP did make it very easy). If you did not copy 7.4 first, change the OS79XX.TXT file to include “P003-07-1-00”
3. Run setup-cisco on *@Home server
4. Edit the image_version included in the SIPDefault.cnf to say “P003-07-1-00”
5. Ensure OS79XX.TXT has “P003-07-1-00” in it
6. Copy the contents of the cisco_utils directory to the foot tftpboot folder (cp /tftpboot/cisco_util/* /tftpboot)
7. Edit both new files (they start with xml and XML) so that the loadInformation7 and 8 have P003-07-1-00 in them.
8. Turn on phone
9. If you have DHCP configured correctly, then it should be pointed to your *@Home’s tftp server already, if not, manually setup the network configuration (see other guides online� key thing, press the settings button, go to network config, scroll down to “enable DHCP”, hit **#, and change to NO. Then type in your network settings, and have “alternate TFTP set to your asterisk server)
10. Your phone should now upgrade to 7.1, and reboot
11. If you phone says “Protocol Application Invalid”, your firmware flash failed, and your phone is now unrecoverable
12. Just kidding
13. Turn off your phone
14. Remove both xml files.
15. Edit the OS79XX.TXT and SIPDefault.cnf to now read “P0S3-07-4-00”
16. Turn your phone on.
17. It should now upgrade and reboot twice, finally saying that the phone is unprovisioned, but it should say SIP in the top right corner. Use the Cisco Config link on the maintenance page of *@Home and make a new phone configuration file.
18. Thank the internet, and whoever is hosting this guide for making your life oh-so-easy.

Avaya Office IP412 PBX

This particular job is for a University. This building is connected to the main campus via a private T1, but all traffic will remain local for now and be routed to the real world via a PRI/T1. The building has three floors, including a lower level, a first and second floor. The network is composed of all new Cisco switches including a 6509 with 6 x 10/100/1000 PoE blades, a 4507R with 4 10/100/1000 PoE blades and a couple of stacked 48 port 3750’s with the same charactetistics. The main idea is to have a voice and a data VLAN per floor, which will bring the sub-total to 6 VLANs.

MORE…

Additional VLANs will be provided for the Wireless Access Points, the servers, the voice server components (voice mail & PBX) and the managed switches. The VLANs are configured to forward all DHCP requests to the existing DHCP server, which will determine where the request came from and assign an IP address from the appropriate scope. A vendor that will remain unamed won the bid on this project and has been a nightmare from day one. I have never seen
so much incompetence in my life.

Another unique detail on this project was the need for offices to have both a computer and a phone on the same UTP ethernet cable, so an “expert” was brought in from Houston to configure this setup and QoS on the network.

Several approches were tried to address this issue, but the phones always kept getting an IP address from the same scope as the PCs. The vendor/”expert” solution was to make every port a trunk and thus allowing several VLAN to the port, but the problem still remained. How would the phone or PC distinguish from what DHCP scope it would pick up an address from or to be more exact how would the DHCP server decide if it was a PC or a phone asking for an IP address. We ran across an Avaya document that went into how to setup options on the DHCP server and decided to give it a try.

The TFTP server which was hosted on the Voice Mail server was defined on the DHCP server as option 66. A new option was created since it was not available on Windows NT 4.0. The option was 176 and the string on the Ayava document was as follows:

MCIPADD=XXX.XXX.XXX.XXX,MCPORT=1719,TFTPSRVR=YYY.YYY.YYY.YYY,L2QVLAN=ZZZ

were XXX.XXX.XXX.XXX is the IP address for the PBX, YYY.YYY.YYY.YYY is the IP address for the TFTP server and ZZZ is the VLAN that will be tagged to the phone.
After setting everything up, it worked like a charm but I was really worried about all these trunks since I have seen plenty of network nightmares with too many trunks.
Although we had a very thight dealine, I was stuburn enough and not believe the “expert” since I had seen on the all powerful google people using auxilary-vlans for voice on Cisco switches, but the “expert” said that tis woud only work on cisco switches.
After further testing I figured out that the key was in the DHCP server option settings. The switch running a recent version of IOS had the following configuration on each port.

switchport mode access
switchport access vlan
switchport voice vlan
spanning-tree portfast

The 6509 was running a recent CatOS, so an auxiliary vlan was used to emulate the phones vlan. The DHCP server took care of the rest, by tagging phone with a VLAN.

set vlan vlan# port_number
set port auxiliaryvlan port_number vlan#

QoS configuration for the each on the IOS switch was

service-policy output autoqos-voip-policy
qos trust device cisco-phone
qos trust cos
auto qos voip cisco-phone
tx-queue 3
priority high
shape percent 33

General configuration on the IOS switch is:

qos dbl
qos map dscp 24 25 26 27 28 29 30 31 to tx-queue 4
qos map dscp 32 33 34 35 36 37 38 39 to tx-queue 4
qos map cos 3 to dscp 26
qos map cos 5 to dscp 46
qos

Configuration for the CatOS QoS is as follows:

#qos – qos configuration via Autoqos
set qos enable
set qos map 2q2t tx 2 1 cos 1
set qos map 2q2t tx 2 1 cos 2
set qos map 2q2t tx 2 1 cos 3
set qos map 2q2t tx 2 2 cos 5
set qos drop-threshold 2q2t tx queue 1 100 100
set qos map 1p1q4t rx 1 3 cos 1
set qos map 1p1q4t rx 1 3 cos 2
set qos map 1p1q4t rx 1 3 cos 3
set qos map 1p1q4t rx 1 4 cos 6
set qos map 1p2q2t tx 2 1 cos 1
set qos map 1p2q2t tx 2 1 cos 2
set qos map 1p2q2t tx 2 1 cos 3
set qos map 1p2q2t tx 2 2 cos 6
set qos wrr 1p2q2t 50 255
set qos wred 1p2q2t tx queue 1 70:100 70:100
set qos wred 1p2q2t tx queue 2 70:90 100:100
set qos map 1p3q1t tx 2 1 cos 1
set qos map 1p3q1t tx 3 1 cos 3
set qos map 1p3q1t tx 3 1 cos 4
set qos map 1p3q1t tx 3 cos 6
set qos map 1p3q1t tx 3 cos 7
set qos wrr 1p3q1t 20 100 200
set qos wred 1p3q1t tx queue 3 70:90
set qos map 1p1q0t rx 2 cos 6
set qos map 1p1q0t rx 2 cos 7
set qos map 1p2q1t tx 2 1 cos 1
set qos map 1p2q1t tx 2 1 cos 2
set qos map 1p2q1t tx 2 1 cos 3
set qos map 1p2q1t tx 2 cos 6
set qos map 1p2q1t tx 2 cos 7
set qos wrr 1p2q1t 50 255
set qos txq-ratio 1p2q1t 70 15 15
set qos wred 1p2q1t tx queue 2 70:90
set qos map 1p1q8t rx 1 5 cos 1
set qos map 1p1q8t rx 1 5 cos 2
set qos map 1p1q8t rx 1 7 cos 3
set qos map 1p1q8t rx 1 7 cos 4
set qos map 1p1q8t rx 2 1 cos 6
set qos map 1p1q8t rx 2 1 cos 7
set qos cos-dscp-map 0 10 18 24 34 46 48 56
set qos ipprec-dscp-map 0 10 18 24 34 46 48 56
set qos policed-dscp-map 0,24,46:0
set qos policed-dscp-map 1:1
set qos policed-dscp-map 2:2
set qos policed-dscp-map 3:3
set qos policed-dscp-map 4:4
set qos policed-dscp-map 5:5
set qos policed-dscp-map 6:6
set qos policed-dscp-map 7:7
set qos policed-dscp-map 8:8
set qos policed-dscp-map 9:9
set qos policed-dscp-map 10:10
set qos policed-dscp-map 11:11
set qos policed-dscp-map 12:12
set qos policed-dscp-map 13:13
set qos policed-dscp-map 14:14
set qos policed-dscp-map 15:15
set qos policed-dscp-map 16:16
set qos policed-dscp-map 17:17
set qos policed-dscp-map 18:18
set qos policed-dscp-map 19:19
set qos policed-dscp-map 20:20
set qos policed-dscp-map 21:21
set qos policed-dscp-map 22:22
set qos policed-dscp-map 23:23
set qos policed-dscp-map 25:25
set qos policed-dscp-map 26:26
set qos policed-dscp-map 27:27
set qos policed-dscp-map 28:28
set qos policed-dscp-map 29:29
set qos policed-dscp-map 30:30
set qos policed-dscp-map 31:31
set qos policed-dscp-map 32:32
set qos policed-dscp-map 33:33
set qos policed-dscp-map 34:34
set qos policed-dscp-map 35:35
set qos policed-dscp-map 36:36
set qos policed-dscp-map 37:37
set qos policed-dscp-map 38:38
set qos policed-dscp-map 39:39
set qos policed-dscp-map 40:40
set qos policed-dscp-map 41:41
set qos policed-dscp-map 42:42
set qos policed-dscp-map 43:43
set qos policed-dscp-map 44:44
set qos policed-dscp-map 45:45
set qos policed-dscp-map 47:47
set qos policed-dscp-map 48:48
set qos policed-dscp-map 49:49
set qos policed-dscp-map 50:50
set qos policed-dscp-map 51:51
set qos policed-dscp-map 52:52
set qos policed-dscp-map 53:53
set qos policed-dscp-map 54:54
set qos policed-dscp-map 55:55
set qos policed-dscp-map 56:56
set qos policed-dscp-map 57:57
set qos policed-dscp-map 58:58
set qos policed-dscp-map 59:59
set qos policed-dscp-map 60:60
set qos policed-dscp-map 61:61
set qos policed-dscp-map 62:62
set qos policed-dscp-map 63:63

I will now try to do some soft phone, SIP Client and integration with Asterisk.