|
Notes from the E-mail Town Hall Meeting
"We understand how terribly important e-mail is to all of you," said University Chief Information Officer H. David Lambert.
In an April 26 town hall forum attended by students, faculty, researchers, and staff from the main and medical campuses, Lambert and other UIS senior staff members spoke on the state of Georgetown's e-mail system. Lambert affirmed UIS's recognition of the importance of e-mail to university life and UIS's commitment to bring the system to stability.
Lambert, along with UIS director Matt McNally, sought to explain the instability—and how UIS will work to address it—by explaining how GUMail works. When a message is sent to a GUMail address, it is processed by servers called hubs. Installed on the hubs is an e-mail attachment filter. The filter removes any executable attachments, which often are viruses, and replaces them with a notification that an attachment has been removed from the message. Then the message is sent to an e-mail storage server running Sun Microsystems iPlanet software. Unlike most e-mail systems (which rely on the user's computer to download and store each message) GUMail is an IMAP system (one that allows users to view messages that remain on the server). When a GUMail user opens an e-mail message, iPlanet locates the message on the storage server and displays it in the user's e-mail program. E-mail sent out from GUMail accounts are processed by the same hubs that process incoming mail, and are also stripped of potentially virus-carrying attachments before being delivered.
Instability in the GUMail system has been caused by several factors, though the primary cause has been the alarming increase in the world's spam and virus traffic. Similarly, delivery delays have been caused by an overabundance of broadcast e-mail messages. The GUMail system was built in 2001 to process 150,000 to 350,000 e-mail messages a day, five times the daily average at the time. Now GUMail processes a daily average of 250,000 to 400,000 messages. While the hubs can accommodate normal e-mail traffic, spam, viruses, and broadcast e-mail are sent in bursts, disrupting regular e-mail traffic patterns and overwhelming the system.
Another factor is GUMail's configuration for e-mail forwarding. Though forwarding services make delivery of e-mail to multiple accounts through one address and lifetime e-mail addressing for alumni possible, it also creates an enormous need for processing power and storage. Approximately 35,000 faculty, staff, students, and alumni now use NetID routing and forwarding services to direct messages to other e-mail services. Whenever any of these other e-mail services cannot accept e-mail from Georgetown (because the end point mailbox is full or no longer exists) our e-mail service must shoulder the burden of continually retrying to send the mail for seven days. A high percentage of the e-mail processed by our servers is mail that is holding to be passed off to another organization's e-mail service.
Additional processing requirements were added to GUMail when attachment filtering began. The filter protected GUMail during the Sobig e-mail virus attacks last August and September, when many other institutions were forced to shut down their e-mail systems. Though the filter has not significantly slowed down e-mail delivery, one of April's e-mail outages occurred when the filtering software crashed, preventing all e-mail from reaching the storage server.
Other e-mail outages have been caused by incidents that could not be predicted and required fast work by engineers and operations personnel to recover service as quickly as possible. Once, a flaw in the IMAP software was identified as causing e-mail crashes and Sun Microsystems was called in to provide a patch. Another time, e-mail hardware was threatened by a failure in the air-conditioning system where e-mail servers are stored. The hardware had to be shut off to prevent permanent and expensive damage from occurring. There are also times when system outages completely unrelated to e-mail (such as network services providing connections to the e-mail servers) appear as if e-mail is failing. In all of these cases external factors impinge on Georgetown's ability to provide e-mail service and negatively influence the total quality of the service. UIS staff are working to make sure that all factors affecting the e-mail service are stable and secure.
Recent actions taken by UIS have reduced delivery delays and shortened resolution times for known problems. While there is a lessened probability of additional unscheduled outages system outages are still possible in the continually changing e-mail environment.
Recently UIS introduced new hubs with greater processing power and implemented a new queuing structure to better manage the high volume and volatility of e-mail traffic. Engineers then worked to tune the mail store to accept the increase in mail being delivered by the hubs.
Delivery of university-related e-mail has been facilitated by implementing queues on the hubs. In the past, all e-mail coming into or going out of the GUMail system was given equal priority. The queues stagger delivery of e-mail so that e-mail sent to or from an individual's GUMail addresses to other individuals' GUMail addresses is given higher priority than e-mail originating from outside the university or broadcast e-mail.
In addition, to alleviate traffic spikes caused by broadcast e-mail, broadcast messages, with the exception of public safety notices, will only be sent prior to 9:00 A.M. or after 6:00 P.M. (For more information, read about broadcast e-mail in this issue of E-Notes.) UIS will also begin working to reduce the number of undeliverable messages by removing e-mail routing for deactivated NetIDs (faculty and staff who have permanently left the university).
Combatting spam and virus is a priotity for UIS. Hackers, spammers, and virus writers will always find ways to overcome any means of preventing their actions, UIS will continue to search for new ways to minimize their effects. "Blacklist" software that blocks e-mail sent by known spammers may soon be installed on the hubs. According to Lambert, even a small blacklist can reduce the volume of e-mail by 20% to 25%.
UIS encourages GUMail users to take advantage of the built-in spam filters included in many newer e-mail programs, such as Mozilla, Outlook 2003, and Mac Mail (for Apple users). Instructions for using the filters can be found on UIS's E-mail Web site. There are also add-ons to Outlook to help manage spam and viruses.
UIS understands and shares the frustrations of the Georgetown University community when a GUMail outage occurs. We all depend on the GUMail system as an indispensable means of collaboration and communication. Any e-mail system carries some risk of an outage, but recent hardware and software upgrades have stabilized GUMail, and we continue to formulate strategies for meeting the challenges GUMail will face in the future. We thank those who attended the meeting and for a lively and honest dialogue and exchange of ideas. | 
