Working in highly regulated environments data is classified in different ways depending on the sensitivity. In some cases, this is legally enforced and must be treated differently.
Examples of a data classification policy are:
- Highly Restricted Data, such as Passwords, Private Keys, SAML Tokens, and Credit Card Numbers.
- Restricted Data, such as Usernames and Customer IDs.
- Unrestricted Data, pretty much anything else.
This classification comes with certain obligations:
Any data that is Highly Restricted must never be made available in log file under any circumstances.
Restricted data could be made available in log files under specific conditions. For example, if an incident were to occur with a service then the on-call engineer would be able to enact a Break-Glass procedure to access this data to diagnose the problem. The Break-Glass procedure would, in turn, trigger a review, an audit, and possibly temporary revocation of privilege from that engineer.
What strategies can be employed to achieve this, particularly given that there are a broad range of logging, monitoring and instrumentation tools available on the market that don't provide a direct answer to this problem?
For example, both Splunk and AppDynamics have the ability to impose different access controls upon conditions of the telemetry being exposed; this means you can create a filter that masks out
<customerId>NNNNNNNNNNNN</customerId>. However, neither of these tools give you the capability automatically identify credit card numbers and mask them automatically.
Note: I believe this is related to DevOps because in a traditional tiered support model a relatively small group of people could have access to the data and filter it manually, by devolving responsibility for operating platforms to development teams this data is potentially exposed to a far wider audience.