This weekend CNS staff spent time doing additional troubleshooting and testing of our virtual server and storage environment. After further analysis and testing, including replacing equipment in some servers, we were able to determine that a network switch in our storage environment was the source of the problem. As a troubleshooting measure, we have removed the defective switch from the storage configuration, and at this time are seeing indicators of greatly improved performance and stability. The next few days will be an indicator of whether the situation is in fact improved.
The issue is that our storage network is now in a less than fully redundant configuration, and we eventually need to replace the device. However, before we do that we must attempt to determine if the switch itself is defective, one of its interface modules, or if there was an error in the switch configuration or our storage topology. However, this testing can now be done in a controlled environment, and the switch restored when we're confident the underlying issue is resolved.
We again must thank everyone for their patience and consideration. Although we were able to avoid major downtime, we did have several outage events that were disruptive, and for that we apologize. The nature of this problem, however, was complicated and subtle, and much of our troubleshooting involved downing systems, making a change, and waiting for another failure. We hope those situations are no longer necessary and we can move forward.