danbricklin.com/log
Home	ISP Outage Explanation
	This is an email we received to explain outages in our connections to the Internet. Names, etc., have been changed to XXXX, etc. While they refer to "65 IP customers" each customer may be a company like us with many users. It's like saying "65 blocks in the city" not saying if they have single houses or apartment houses. From:      WWW, YYYY[SMTP:YYYY_WWW@XXXX.com] Sent:      Wednesday, November 03, 1999 4:48 PM To:      'zzzz@trellix.com' Subject:      POST MORTEM Dear Customer: This is a Post Mortem to inform you of the outage for RRRRR. POST MORTEM RRRRR  OUTAGE SUMMARY OF OUTAGE: 65 IP customers down due to bad hardware on RRRRR router Customers off of PPPPP were down on 2 separate occasions: October 29, 1999: 8:15am - 11:41am (3hours and 31 minutes) November 1, 1999: 3:47am - 11:53am (8 hours and 7 minutes) Customers off of QQQQQ were bouncing constantly starting on October 29, 1999 and continuing through October 31, 1999. DETAILED ACCOUNT OF TROUBLESHOOTING: On Friday, October 29, 1999 beginning at 8:40AM EST we began experiencing outages in New York.  The problem originally appeared to be due to a circuit outage but after several iterations of troubleshooting we discovered the problem to be with the XXXX router - RRRRR. Several organizations within XXXX were pulled together to attack the issue - Field Support, VVVVV, TTTTTT - as well as engineers from our first, second, and third level support.  SSSS, the manufacturer of our RRRRR router has a case open to investigate problems on this router also.  Several pieces of hardware were replaced and several hours of testing between 2 different POP sites and 2 operations centers were conducted over the course of 3 days. Service was temporarily restored at different intervals during the troubleshooting process, however, multiple problems were found and had to be corrected. The last outage occurred and was restored on Monday, November 1st .  We believe that all outstanding problems have been addressed and our second and third level engineers are continuing to closely monitor the router.  The case with SSSS is still open and under investigation.   We are sorry for any inconvenience we have caused.  If you have any further questions or concerns, feel free to contact your sales representative or contact us via e-mail at support@XXXX.net. XXXX IP Operations
	Home © Copyright 1999-2018 by Daniel Bricklin All Rights Reserved.