Wednesday, September 09, 2009

The Suspicious Case of the Email Attachments That Would Not Attach

We had our own little mystery at the Thomas Ford Memorial Library. Many investigators tried their hands at sleuthing before it was solved.

On a Friday a few weeks ago, several of our computer users reported that they could not attach email attachments. Word and WordPerfect documents would not attach to Yahoo mail or to Gmail. Photos could not be attached to Hotmail or uploaded to Flickr. Small PDFs attached but larger ones would not. Staff members who listened to our clients also reported attachment problems. Because major Internet services had recently been interrupted for various reasons, we suspected that the problem was out on the web and decided to just watch the situation. We posted a sign about the problem beside the computer sign-up sheet to alert our clients.

By the next Tuesday, however, the in-building reports escalated, though no pattern had emerged. Some staff reported that they could not attach any files in any email service, while others said they were having no problems. We called nearby libraries to see if they were also having problems, but they were not. We then suspected that we had an internal problem and called the consultants who manage our firewall. The tech guys checked the firewall and also called the Illinois Century Network, our Internet provider. No problems were found. Several techs suggested that our computers might have a widespread virus or other incompatibilities. This was the first time we heard "I've never seen a problem like this." It would not be the last.

With help from Versatile Computer Services, we began checking many of our computers for malware. A few days into our search, nothing had been discovered. The computers seemed clean. The situation, however, had changed. For two days, we could not attach any file to any email service from any computer in the building, except for Sandy's computer, which had no problems. We then discovered Sandy was using Thunderbird, composing her mail off the web and then sending. We tried Microsoft Outlook, which also worked. We had a workaround for staff, but the public computers were still in trouble.

As the search for malware continued, the pattern changed again. JPG, GIF, PDF, TXT, and many other files would now attach, but nothing from Microsoft Office would attach. We brought in wireless laptops. They attached emails easily at other wireless locations but not inside our library. The firewall again became the suspect. It was monitored, updated, and tested, but no problem was found. Then the pattern shifted again, and file attachment failure became nearly universal. "I've never seen a problem like this. It doesn't make sense," we heard from a tech professional for the third or fourth time.

A Versatile consultant then noticed that the dysfunction began before the feed came into the building, so the Internet provider became the suspect again. A tech from the Illinois Century Network assured us that our feed was not filtered or corrupted in any way. Our IP addresses were even changed temporarily to see if that would have an effect, but the problems persisted.

Just when there seemed to be no answer, the Illinois Century Network tech noticed that we had "a dirty T-1 line." When I asked what that was, I learned that it meant there was "noise or interference" (lay terms, no doubt) on the T-1 signal. AT&T was asked to clean the line. The company did, and all our problems went away.

As I look back on the period, I am astounded how tolerant and sympathetic our clients and staff were. We kept them informed about our efforts and arranged for some to visit nearby libraries. Several who asked how the troubleshooting was going offered their own theories. I listened, and later they listened with appreciation as I told them our service was restored. We all like a good mystery solved.

No comments: