When building modern applications, an astute observer will quickly see an issue. How do you give developers and operations engineers access to the application to build, fix, and operate the application without exposing customer data or violating business processes and systems? Do you know who has access to your customer's data? Do you know what a disgruntled employee is capable of doing to your application? On this episode, I discuss access, access to production systems, access to production data, access to sensitive business information, and access to sensitive customer data. Are you ready? Let's go. How do you give developers and operations engineers access to the application to build, fix, and operate the application, without exposing customer data or violating business processes and systems. After all, if your engineers have access to all of the data in the database, they have access to customer private information. and if they can modify data in the database or create or change privileged communications between services, they have complete control over your business and your business processes. Security best practices specify that engineers, both developers and IT operations, personnel, should have as little access as possible to the production application and its infrastructure. Sometimes business requirements make these restrictions even more critical, and some industry regulations such as, for instance, HIPAA can even involve legal requirements and restrictions. This security best practice is known as the principle of least privilege. However, this can be a problem. What happens if the site has a problem during the middle of the night and the on call engineer is called in. The engineer will need access to the application in production. Access they may not have because of the security access requirements in place. How can an on call engineer do the work they need to do without the additional permissions they need during the emergency? The answer is permission escalation. Permission escalation is a process of giving an on call engineer temporary increased system access during an emergency. This increased access typically comes with increased scrutiny. The engineer's activities while they have the increased access are typically logged, and sometimes they must be reviewed by a second set of eyes to ensure that only the necessary actions are actually being taken with the escalated permissions. Additionally, an engineer can typically only get escalated permissions during a registered site emergency. All of this ensures that a rogue engineer can't perform malicious activities on the site, including access customer private data inappropriately and cause damage to the system or the business. This process maintains the principle of least privilege. There are many ways to perform permission escalations, including Break the Glass, Logged Escalation, and Two Person Escalation. Another option is to create specialized, restricted, limited scope tooling to assist the engineer in performing specific critical operations, such as rebooting a server. Engineers often don't like the additional headaches and restrictions. that these rules require. They'd rather simply have all the permissions they need all the time. They argue that time is of the essence during the crisis. Any delay caused by excessive hurdles getting in the way during the crisis in their mind is too much of a restriction. However, site owners need to make sure their systems remain secure. And that means following all security and industry best practices. This includes abiding by all industry regulations. Giving engineers unrestricted access to everything in a production application, including and especially customer data, is just not safe and is just not good business practice.