A Day in the life of Garrett Hildebrand, Network Wrangler
Jonathan E. Sisk

Garrett Hildebrand has one of the coolest jobs in the world.

He works for the University of California, Irvine in the Network & Academic Computing Services department (aka, NACS), in a group called Network Planning and Security, which Garrett and his team (John & Mike) refer to as NetPlanSec. For purposes of referring to he and his team collectively in this article, we will simply refer to them as GarJohMik.

His evocative title on his business card lists his job function as "Manager, Network Planning and Security, Network Wrangler"

When I first saw that title several years ago, I didn't realize how appropriate the "wrangler" designation was until I saw him in action. He cited an instance where a new building's communications budget was so low that after the layer 1 infrastructure (copper and fiber) was installed, only 89 thousand dollars remained for the entire layer 2/layer 3 (Ethernet switches and IP routing) budget, or data budget, equipment, tax, shipping and labor to install and set-up. He estimated a 200 thousand dollar requirement, and got it in the end. As he puts it, he identifies faculty requirements, comes up with a design, gets cost estimates, then identifies funding sources.

In this particular situation, he educated the faculty who would occupy the building on the cost to get them what they needed, and showed them what they'd actually get unless additional funding was identified. When the campus Design and Construction folks who had not allocated enough money for the data network went to the Dean of that school to request more money, he turned to the faculty for advice on whether it was really needed. Yes, they all said. Thus the situation was corrected, or "wrangled."

Ideally, Garrett said, he and another campus network planner--Todd Strand (Garrett is layer 2/3, while Todd is layer 1, and works in a different group called Campus Planning Unit) are contacted by the campus Design and Construction department early enough in the process to have adequate funds allocated for the network of a new building. But that isn't always the case, and when it is the plan can be foiled when changes in research or changes in plans for use of the building during the intervening years (between initial planning and actual construction) affect what the network must do to make everyone happy. Garrett has been doing this at UCI for 7 years now.

GarJohMik are also in charge of monitoring what they euphemistically refer to as "the border". Their job is to provide front-line security for the campus to all outside Internet access coming in from the OC-12 (622 megabits/second) connection to the regional CalREN-2 network which connects K-20 campuses around the state (see http://www.cenic.org/CalREN/index.html )

Utilizing the latest in monitoring and defense technology, they are constantly on the lookout for intruders, and their well-thought-out defensive perimeter keeps that part of their job to a minimum, allowing them to focus on the bandwidth-consumers inside their perimeter. You may wonder how Garrett manages to get these latest tools, when the California school system is cutting back on expenses.

Well, for starters, Garrett and Mike have been lobbying the campus for several years now to allocate funds to modify the UCInet (the name of the network at UCI) backbone to support a defensible campus border to the outside world, and to purchase certain tools to make such a thing possible. In addition, the Director of NACS worked with Garrett to apply for special state funds which were earmarked by Grey Davis for improving research through certain high-speed network initiatives, and they got some help there.

Mike, Garrett's main security guru, also created a number of home- brewed tools to aid in the security effort. One, for example, detects scans of UCInet-based computers originating from computers on the Internet, and automatically blocks them at the border in a timed "penalty box." They are released after the timer is up, and put right back in if they do it again. Mike says that after being put in the penalty box a certain number of times, they go into a black hole forever, unless manually removed.

But there is network wrangling and then there is security wrangling. Garrett has wrangled a partner relationship between UCI and Foundstone Technologies, an Orange-county based net security firm who may be best known for employing the authors of the popular book, "Hacking Exposed". They also develop software tools and teach courses on all aspects of hacking and prevention.

As a result of this partnership, Foundstone gets to test their network scanning technology on UCI's networks, and UCI gets to find out via these tools the vulnerabilities of UCI computers. GarJohMik then help folks on the campus to correct these vulnerabilities.

Each day, John uses a list created by NSP called "Top Talkers." Top Talkers are the names or IP addresses of computers which have used more than a certain amount of Internet bandwidth in a given period of time. These are singled out for examination to determine if the traffic is not exactly for research and education purposes--the mission of the University. In a word, non-academic, personal use. John also makes use of on-the-fly reports he creates using a NetPlanSec tool called argus. (Argus is at http://www.qosient.com/argus/ )

The very cool tools GarJohMik employ allow them to drill down to a specific IP of someone who is using an extraordinarily large amount of bandwidth and even pinpoint the service moving the data and the location of the remote host, which allows them to determine (much of the time) whether or not the bandwidth is being used for academic purposes. Rather than also being the force who shows up at the door of the errant bandwidth consumer, they simply notify the appropriate local department responsible for local action (think student housing here).

They used to send warnings out. Now, the repeated bandwidth suckers lose their net access privileges, that is, until they correct the problem or promise to curb their bandwidth appetite.

There is a very practical side to Garrett's job as well. The university has a 1-gigabit connection to CalREN-2's OC-12 connection point, and Internet access to all the universities on it share common Internet drains, which they are charged to use. The month-to-month costs are usage-based, and each year a one-year Internet budget is allocated to NACS to pay the bill with. If the usage begins to exceed the budget, the bill can't be paid.

In fact, in one month that Garrett recalls, the university was over-budget on the projected monthly allocation by close to $10,000. That's when the university set guidelines on personal use versus university business use of Internet access and put a limit on the amount of traffic which could be termed personal use. Garrett's group got the job of managing this and John is the main dude who manages this part of the ranch. "Pipe Wrangler" John says that he saves the university between 8 to 10 thousand dollars a month during the academic year (usage goes down during summer break because most students are gone).

Another practical side of this is that sometimes large spikes in bandwidth from computers on the campus to computers off-campus can signify that the computer has been hacked and is hosting copied DVDs or other copyrighted material using something called a Warez server placed there by hackers. In other cases, the hacked computer may being used to launch a scan on Internet-based computers, or even participate in a DDOS attack (Distributed Denial of Service).

Garrett's NetPlanSec group also serves as a containment and prevention crew for worms and certain virusi that are unleashed inside their net jurisdiction. Worms spread by propagating across the network from computer to computer, without the benefit of email. These worms usually scan other computers for known vulnerabilities, then exploit the vulnerability to get into that computer, which then continues to spread. Virii usually, but not always, spread via email. While NetPlanSec does not worry about personal computer virus programs, they do take particular notice when a virus propagates to the point where it starts affecting network bandwidth and availability.

For example, NetPlanSec warned the campus about Nimda before it got there, then worked with other groups such as NSP and other campus support groups to get Nimda cleaned-up when it infected campus systems. Nimda was both a virus and a worm. It could propagate BOTH ways.

While the campus has had its share of outages by both Code Red and Nimda, NetPlanSec has continued wrangle increasing control and order over the black hats out there. The most recent addition to their arsenal was a campus VPN (Virtual Private Network) capable of supporting off-campus connections from faculty and students from home which are tunneled and encrypted. This was the step that needed to be done before they could secure permission from the campus to close at the campus border all the ports that are routinely used to exploit Microsoft systems from the Internet.

Just the other night, Garrett related to me in a phone conversation how the so-called Slammer worm affected virtually every UC campus except UCI. Not one UCI system got infected in the attacks of January 25th and 26th by the Slammer worm which, as the New York Times puts it, "hindered the operations of hundreds of thousands of computers, slowed Internet traffic and even disrupted thousands of A.T.M. terminals." (see http://www.nytimes.com/2003/01/28/technology/28SOFT.html - registration required)

In fact, as the article reports, that worm even got to Microsoft Corporate computers.

Even cooler than having access to the latest and greatest in IP hardware (think Cisco here), Garrett has access to the entire campus, and it's complete infrastructure. As the campus consumes its parking lots to build more buildings, Garrett and Todd Strand are out there in advance laying out and anticipating the future wiring and, data networking, and access needs.

That infrastructure is Cisco powered. From the recent VPN concentrator, to the gigabit backbone, and from the edge switches to the "campus border." It would not surprise me if his Cisco Rep named a kid or two after Garrett.

Garrett tells me that when he got to UCI the campus was a hodge-podge of different network vendors equipment. Garrett and the current manager of Network Operations teamed up to convince campus management to allow them to sole-source with Cisco to simplify the training of support personnel (network buckaroos), ensure compatibility, improve uptime, and get better discounts. In effect, to establish a growth relationship with Cisco Systems, to benefit the campus mission and even establish some research relationships.

The first benefit of that relationship was the 1998-1999 backbone upgrade, of which Garrett was project manager. He worked with a fellow Cyberfolker - Ashan Willy - to come up with the initial design. Garrett then worked with Cisco Professional Services' Warren Chapman, and the UCI Network Operations group to design and build the backbone. Another Cyberfolker, Bob Sayle, has helped Garrett's NetPlanSec group in a number of its security initiatives, particularly firewalls and VPNs.

I want to once again thank Garrett for the fine lunch at the University Club and the gracious tours he provided to me. I hope this sheds a little light on a day in the life of Garrett for those of you too far away to possibly ever get the chance to see him in action.

Jonathan E. Sisk
On the digital coast of Southern California
January 30, 2003