I've published it below in its entirety.
Letter from LSAC:
February 28, 2013
As you know, for the last 48+ hours, many law schools and applicants have been unable to access LSAC’s websites, including ACES and LSAC.org. I will get into the details of what happened. First, I want to assure you that there was absolutely no hacking or malicious activity involved—you will be able to see that when you read the technical details below. All data stored on LSAC’s servers and coming in and going out from both ACES and LSAC.org is and has been completely safe.
Second, I want to express my concern and apologies to those of you who did not feel that you were adequately informed about the extent of the problem or how LSAC was addressing it as it was unfolding. We were anxious to communicate what we knew as soon as we knew it, but because the depth of the problem was not immediately apparent, and its impact was constantly changing, our messages may not have been as helpful as intended.
What we now know is that at approximately 6:00 am on February 26, LSAC’s Domain Name System (DNS) hosting provider (DBSi) caused a DNS outage that affected all of our websites. This outage was due entirely to a human error that DBSi made in renewing its domain name. This caused DBSi’s domain name, as well as the domain names of LSAC and 44 of their other clients, to be unreachable. DBSi resolved the problem by 9:00 am that morning. But, the error essentially caused the URLs for LSAC.org and ACES to disappear from the internet. Once DBSi’s domain name was fixed, the URL had to be replicated throughout the entire internet. There are tens of thousands of DNS servers worldwide, and they all replicate at random times, which explains why access to our sites was intermittent and sporadic, at best, until this morning. In addition, due to server settings, information stored on these DNS servers had to expire before the new information could become available to users.
We have been closely monitoring the replication process and are now confident that nearly 100 percent of the DNS replication has been completed. Virtually all users of LSAC.org and ACES are able to access the sites now. We are continuing to work individually with any schools or applicants who are still having problems. However, the impact of this could not have come at a worse time in the admission cycle. Services affected included ACES; law school applications for both candidates and law schools; registration for the LSAT and CAS; transcript, recommendation letter, and evaluation access and transmission; transmission of foreign transcript evaluations from AACRAO; registration for the MPRE; and e-mails. In addition, schools that were unable to access ACES may not have been able to communicate decisions or any other information to applicants. February LSAT score posting will not be adversely affected.
Communicating about this problem has been complicated by its technical nature, and by the fact that, although the technological “fix” was relatively easy to implement, it would not become effective for everyone at once, or even on a predictable schedule. Once the initial issue was corrected, there was literally nothing LSAC could do to create more access to its internet sites, even though the sites were not “down.” This was very frustrating for admission offices and applicants because we were saying there was nothing further that we could do to restore access for individual users and could not predict when any individual user would regain access. Now that the sites are accessible to nearly all users, we are looking at how we communicated to users about the problem and how we can do that more effectively in the future, including providing reliable workarounds. Potential workaround solutions were explored but were not viable due to potential risks in using them. We are also demanding that our vendor provide a plan for ensuring that this problem never happens again.
Following are the communication activities that did take place:
• E-mails to prelaw advisors on the morning the problem started. They had been e-mailing to us because they were hearing from applicants, and we wanted to reassure them that we were aware of the problems and doing everything possible, including working with approximately 50 law schools to adjust March 1 application deadlines.
• Twitter and Facebook messages to applicants and others who follow LSAC, who had also been calling and tweeting.
• Direct contact with law schools by Law School Support Services.
• E-mails to the Admit-L, Admit-LLM, and ACES listservs.
Obviously, there is always room for improvement when dealing with an emergency situation. We are reviewing our procedures for communicating about catastrophic problems that may arise and will provide more details in the future. Any suggestions you may have for how we can do better from your perspective,would be most welcome. However, no amount of downtime is acceptable when we are all so dependent on technology. I want to assure you that you can be confident that we have done and will continue to do everything possible to avoid problems and to communicate honestly with you about them if they do arise.
Sincerely,
Daniel O. Bernstine
No comments:
Post a Comment