The key to effective Purview Data Loss Prevention (DLP) policy deployment with proven strategies to minimize false positives and protect sensitive data without disrupting business operations. This blog shares data-driven methodology for achieving accurate detections and building confidence in your organization's security policies.
Author: Nathan Berger,
Director of Security @ Cyclotron
With over 100 Purview deployments completed, our architects have learned invaluable lessons and developed proven best practices to ensure client success. In this blog, we’ll share how to prevent the most common issue with Data Loss Prevention (DLP) policies: Preventing false positives.
We often see clients deploy Purview on their own, discover a wealth of false positives, and turn off the policies or set them to audit mode. Policies never become useful, and later, a new set of tool owners ask, “Why do we even use Purview if it’s broken?”. This is a completely avoidable scenario, and one that takes a data-driven testing methodology to prevent. When managed well, Purview DLP policy rollout will not negatively affect business operations.
Two key outcomes should be achieved for each Purview DLP policy:
The policy protects our sensitive data as expected
The policy does not negatively impact our business
Let’s break this down. To protect sensitive data as expected, we need to consider two fundamental areas:
True positive rates: How often does our policy successfully trigger? We aim to say, “We can be confident that over 9 times out of 10, your policies will trigger correctly and intentionally.”
False negative rates: How often does our policy not prevent the data leakage events we meant to stop? We aim to say, “We’ve made some inferences about false negative rates and our best data says that at least 80% of the real data leakage events you want to prevent will be stopped by this policy.”
In addition, we want to prevent negative impact to business. This requires assessing false positive rates: How often does our policy accidentally block legitimate sharing? We aim to say, “For the remaining false positives (less than 1 in 10 blocked sharing events), the data blocked would not cause significant business impact.”
The Methodology: Data-Driven Testing
This diagram represents the key steps our Cyclotron engineers use when validating Purview policies in your environment. Almost every task here can be done in production environments without business impact.
To start, we work closely with your teams to finalize a policy design that meets your business requirements. Then we determine an acceptable True Positive Rate/False Positive Rate (clients usually choose 90/10% or 85/15%). If your organization’s risk tolerance is higher, you can use an 80/20% split. This testing criteria is important to determine before we review real impact, as it helps us establish a fair hypothesis and avoid biases.
The False Positive Rate (10-20%) is important to examine: Would it interrupt key revenue streams, or would it be benign? Our engineers will work with you to figure out your acceptable risk tolerance based on your business.
Once we’ve established good testing methodology, we break into two workflows: We scope a PoC, and we set up silent auditing to assess the org-wide impact of future rollout.
Executing the Proof of Concept (PoC)
The top row of steps in the methodology workflow include the following PoC activities:
Create the policy and scope it to test users only. Policies can take time to activate, so do this first. Give at least an hour for a policy to apply, and restart M365 Desktop apps if needed.
Validate your test data. Before testing, ensure your test data matches your policy. A surprising number of clients miss this. For example, Microsoft’s definitions of U.S. Social Security Number always include a keyword like “SSN” within 300 characters. Sending a list of Social Security Numbers without a keyword nearby will result in the policy failing. You can easily upload your sample data to the Purview admin center on the Sensitive Information Type (SIT) page to validate that it matches. One way to infer False Negative rates is to find real examples of past data violations and use them during sample detection. Alternatively, you can analyze real files that you aim to protect. This can massively aid your organization by ensuring your SITs match your desired use cases. Remember: Your business data might be formatted differently than what you would expect (e.g. the “SSN” keyword not normally found within 300 characters of the Social Security Numbers), which can ruin the DLP deployment by making it miss real data.
Validate your pre-requisites for each Location. Each data Location has different pre-requisites for targeting a policy. For example, scoping a Teams DLP policy to a distribution group will affect channel messages, but not chat messages. Scoping an Endpoint policy requires enabling Device Onboarding in the Purview admin center, and using an enrolled device with a Healthy connection status shown on the same page.
Validate user targeting. It’s a simple check: Make sure your users are added to the testing group.
Test the policy. Have a targeted user send the sample data over the location to the expected recipient (usually external).
By the end of these steps, we’ve established that our policies take action as expected in controlled conditions. In other parts of the project, we assess your pre-requisites org-wide to ensure those controlled conditions are met for the whole org by production rollout.
Assessing production impact before rollout
The second, parallel set of steps in the workflow is silent auditing, which shows all impacted data at rest across the organization. It also shows a time-limited sample of impacted data in motion (e.g. 2 weeks of emails that would have been encrypted, or Teams messages that would have been blocked).
The silent auditing steps include:
Create a copy of DLP policies, scoped to everyone, with no actions included and no alerts active. This results in simple logging in Activity Explorer that can be used for later analysis.
Why not use Simulation Mode? Cyclotron prefers reviewing full org-wide impact, not just a sample of impact. Simulation Mode is very helpful, but doesn’t include certain columns from Activity Explorer, and has less items that can be previewed directly from the portal. Even using Simulation Mode, we would create both the PoC policy and Audit policy anyway, so it doesn’t make much difference.
Wait for 1-2 weeks of normal business operations. For data at rest (OneDrive, SharePoint), all matching data will be audited within the first 24 hours of policy creation. For data in motion (emails, Teams messages, OneDrive/SharePoint new sharing events, etc.), events will be audited as they occur, so no past data is included. A larger time period allows for a better sample and a higher quality analysis. Note that 30 days is the absolute maximum sample due to Activity Explorer limits. You can work around this by exporting every 30 days.
Export the policy matches from Activity Explorer. Include any relevant columns such as Trainable Classifier, Sensitive Info Type, Sensitive Info Type Count, file path, and more. Any columns not selected in the GUI won’t appear in the export.
Classify the results. We like to create a new column, “Match”, and use TRUE, FALSE, or TBD to mark a random selection of 200-300 items as true or false positives. To expedite this arduous process, use context clues from file names or site names, which often give away the likelihood of sensitive data (e.g. Credit Card Number matched in a file “CC Authorization.docx”). For less clear items, use Activity Explorer alongside your export to examine the remaining Sensitive Information Types. Cyclotron will often conduct this work alongside you to avoid overexposing sensitive data, as real snippets of sensitive data will be viewed during this exercise.
Analyze the results. Did we exceed our True Positive rates? If the policy caused too many false positives, we can easily examine our false positives to tighten our detections.
If you tighten policy’s SIT detections after a testing round, ensure you compare the previous True Positive count with the new True Positive count. This helps you assess the likelihood of missing legitimate matches. False Negatives are notoriously difficult to assess, and this is the only step in the process you might see them.
At the end of this exercise, you should have confidence that your org-wide policy impact is acceptable. You’ll prevent data leakage, you won’t impact business operations in a significant way, and you can feel confident that you’re not missing data leaks.
Cyclotron helps organizations deploy DLP safely & effectively. We’re here to help both new & existing clients using this methodology during our Purview projects. Get in touch to learn more about how we can help your organization improve security and risk reduction with DLP.