What strategies can be employed to secure sensitive data in log files?

Question

Working in highly regulated environments data is classified in different ways depending on the sensitivity. In some cases, this is legally enforced and must be treated differently.

Examples of a data classification policy are:

• Highly Restricted Data, such as Passwords, Private Keys, SAML Tokens, and Credit Card Numbers.
• Restricted Data, such as Usernames and Customer IDs.
• Unrestricted Data, pretty much anything else.

This classification comes with certain obligations:

• Any data that is Highly Restricted must never be made available in log file under any circumstances.

• Restricted data could be made available in log files under specific conditions. For example, if an incident were to occur with a service then the on-call engineer would be able to enact a Break-Glass procedure to access this data to diagnose the problem. The Break-Glass procedure would, in turn, trigger a review, an audit, and possibly temporary revocation of privilege from that engineer.

What strategies can be employed to achieve this, particularly given that there are a broad range of logging, monitoring and instrumentation tools available on the market that don't provide a direct answer to this problem?

For example, both Splunk and AppDynamics have the ability to impose different access controls upon conditions of the telemetry being exposed; this means you can create a filter that masks out <customerId>NNNNNNNNNNNN</customerId>. However, neither of these tools give you the capability automatically identify credit card numbers and mask them automatically.

Note: I believe this is related to DevOps because in a traditional tiered support model a relatively small group of people could have access to the data and filter it manually, by devolving responsibility for operating platforms to development teams this data is potentially exposed to a far wider audience.

Answer 1

I think the solution comes down to a broad spectrum of approaches that ensures data protection:

• Data Classification: The most efficient technical strategy is to categorise the data at the point of creation rigorously. At its core, the developers are responsible for ensuring that all logged information is assigned a category. Categorization can, for example, be achieved through Splunk Metadata which, in turn, can be used to direct log entries to different buckets based upon their data categorization.

• Event Partitioning: There is often a desire to log sensitive information along side non-sensitive information. For example, if a new user was to sign up you may log:

  • Customer ID (Restricted)
  • Customer Type (Restricted)
  • Aquisition Source (Unrestricted)
  • Correlation ID (Unrestricted)

  It is possible to split this one "Event" into two parts, one containing the Restricted information and one containing th Unrestricted information. This aligns with the first point by allowing filtering rules to direct to different buckets.

• Data Masking: In specific circumstances it may not be possible to categorise the data at source, in my experience logging solutions do allow for Masking Rules to X out sensitive data. In the linked example a sed command is used to apply a regular expression to all data from a specific source. Once the restricted data has been masked out then the event can be considered to be Unrestricted. Care must be taken with the rule to ensure that information critical to an event such as a correlation ID doesn't match the Regular Expression used to match sensitive data.

• Event Filtering: As a last resort it may be necessary to filter all events of a particular type or source into a separate bucket if they contain sensitive data that cannot be categorised or masked. In this case the information in a restricted bucket could only be accessed through a Break-Glass mechanism to bypass access controls under the provisions of an incident.

With each of these solutions, testing is key to ensuring that nothing slips through the cracks. Logging and Event Management must be considered to be a first class Non-functional Requirement with the same level of development rigour applied to a solution - including peer review and tests to ensure that data is properly categorised and partitioned in the tool of choice.

Answer 2

It depends on what you mean by "log files" I supposed. If you mean the telemetry data you use to verify your system is operating correctly I would say "Don't log sensitive fields." You don't need that kind of information to alert you to high or low transaction rates, response times to your dependent services, etc.

If you mean data for billing or audit purposes I would suggest you establish a write-only pipeline where the data is written but can't be read by the writer. Then your billing and auditing pipeline kicks in and you have the controls there to audit who looked at individual records.

In the end security comes down to documented processes with their own secure log files, etc. At some point you have to trust someone or some process since all data is written so that it can later be read.