Standard Operating Procedure (SOP)

 

𝐒𝐭𝐚𝐧𝐝𝐚𝐫𝐝 𝐎𝐩𝐞𝐫𝐚𝐭𝐢𝐧𝐠 𝐏𝐫𝐨𝐜𝐞𝐝𝐮𝐫𝐞 (𝐒𝐎𝐏)

 

In some companies, every team is solely responsible for the design and development of their systems and what comes after that, such as deployments, monitoring, and troubleshooting of errors.

An SOP document is crucial to provide detailed step-by-step instructions to complete specific processes.

For example, when a failure occurs (Alarm A is triggered), a team member should find the steps to troubleshoot and mitigate this error in the SOP doc!

 

𝐆𝐨𝐨𝐝 𝐒𝐎𝐏

•  Go to logs for stage X (Link)
•  Check the dashboard for stage X (Link)
•  Update the ticket with what you found in the logs, and a link to the graphs
•  Ensure the Dead Letter Queue (DLQ) is empty; if not, apply the SOP section for retrying DLQ messages (Link)
•  Check if there is a customer impact (usually customer X, Y, and Z are impacted by such error) - (Link to clients dashboard)
•  Check if the error is because of a downstream dependency (Link to dependency H and R dashboards)
•  Update the ticket with your findings above, and engage on-calls for client/dependency teams if necessary.
•  ....
  

𝐖𝐡𝐞𝐧 𝐝𝐨 𝐭𝐞𝐚𝐦𝐬 𝐝𝐞𝐟𝐢𝐧𝐞 𝐒𝐎𝐏𝐬?

 

Unfortunately, most teams write the SOP sections for specific errors only after they occur, but in our team, we started to change our strategy and always define all possible points of failures during design and development, and add tasks to update the SOP doc with a section for each failure type!

For existing systems, I highly recommend reading about Pre-Mortem process.