Incident information

Misconfigured Load Balancer

Published · Apr 20, 2022

affected systems: ugrc api

severity: partial-outage

resolved: yes

Today at 5:00pm we migrated the infrastructure behind the UGRC API by updating the DNS to point to new infrastructure. This migration contained a misconfiguration in a load balancer that affected some API keys authorization; Specifically API keys using the Desktop Application style.

We will schedule this migration for a later date. Since we will no longer offer http traffic, an email communication explaining that all requests will require https and the date for when the change will be coming soon.

Rollback

The migration was rolled back with a DNS change routing requests to the original system.

Certificate issues

The hosting group modified the certificate on the new load balancer to test the misconfiguration. This caused clients who had not had successful DNS propagating to receive insecure https warnings.

Resolutions

The correct certificate was placed back on the load balancer while the DNS finished propagating.