Various services using the internal key-value store went down due to a disk issue leading to the complete loss of data for the KV service. All downstream services missing data misbehaved leading to widespread service failure. The issue has been resolved and a backup and more thoughtful design has been implemented.