Tell me about a time you failed or made a mistake — what did you learn?

STAR Framework Answer

Situation

I deployed a database migration that added an index to our 5M-row applications table during peak hours. The migration locked the table for 45 seconds, causing a partial outage.

Task

I needed to fix the immediate issue, prevent it from happening again, and rebuild trust with the team.

Action

1. Immediate fix: Killed the migration process, table became available again
2. Root cause analysis: I didn't check the table size or run the migration during off-hours
3. Documented the incident: Wrote a post-mortem (not a blame document)
4. Prevention:
   • Added a CI check that warns on migrations for tables > 100K rows
   • Created a runbook for safe migrations (use CREATE INDEX CONCURRENTLY in PostgreSQL)
   • Set up a migration review process requiring a second engineer's approval

Result

The CI check caught 3 potentially dangerous migrations in the following months. The team adopted the migration runbook, and I became the go-to person for database-related reviews.

Key Takeaways

Lesson	How I Apply It
Measure before migrating	Check table size, run EXPLAIN ANALYZE
Use safe migration tools	CONCURRENTLY, pt-online-schema-change
Automate prevention	CI checks > human memory
Write post-mortems	Share failures so others learn too
Own the mistake	"I caused this" builds trust, not "it was the system"