Why a Government ID (SSN, National ID, Tax ID, Passport) Was Not Detected

Data type: Government & national identity numbers (SSN, driver's license, passport, national ID, tax ID, resident registration number)

Issue

A government or national identity number was present in scanned content but produced no finding, or produced a finding at lower confidence than expected. This is commonly reported during a POV or a detection validation exercise.

Environment

All content integrations and endpoint scan paths. Applies to native ID detectors. Causes 1, 2, 3, and 5 describe model behavior and do not apply to custom regex detectors, which match purely on pattern.

Before troubleshooting: test the sample in the Nightfall Playground

The Playground returns the model's confidence score for any text you paste, independent of your policies and detection rules. It is the fastest way to separate a detection question from a configuration question, and it is the first step for every report below.

→ playground.nightfall.ai

Paste the exact sample and record the result. There are three outcomes and they point in different directions:

Do this before changing any policy. If you want to evaluate the effect of a lower confidence threshold, test it in the Playground or in an isolated test policy rather than editing a production policy.

Sanity-check your test data with an LLM first

Before concluding anything is a miss, confirm your sample is a well-formed identifier. Paste it into Claude or ChatGPT and ask:

"Is this a structurally valid [US Social Security Number / Indonesian driver's license / Korean resident registration number]? Check the format, length, and any checksum or range rules. Is it a commonly used test, sample, or placeholder value?"

This catches invalid values and well-known test data in seconds, and resolves a large share of reported misses before a ticket is ever opened.

Cause and resolution

Most reported ID misses trace to one of the following. Work them in order — the first three explain the large majority.

1. No contextual clue near the value

ID detectors generally reach Very Likely confidence only when a contextual indicator appears within roughly 50–100 characters of the value, depending on the model. A bare identifier with no label is expected to score lower.

Context keyword coverage is language- and locale-specific, and includes abbreviations, informal variants, and romanized forms. Coverage is not identical across every detector — if a variant your organization uses is not recognized, tell us and we can add it.

주민등록번호 860903-4567890     → detected
주민번호 860903-4567890         → detected
Korea RRN 860903-4567890       → detected

Resolution 1: Add a contextual clue to your test sample. If it still does not trigger, share the sample with us — our model can learn your contextual clues.

Resolution 2: If the use case requires detecting unlabeled identifiers, set the detection rule minimum confidence to Likely rather than Very Likely. Validate the noise impact before rolling out broadly.

2. The value failed candidate identification before reaching the model

Detectors run a candidate-identification stage before any model scoring. If a value does not match the expected pattern or validation rules for that identifier, it never reaches the model, so no confidence score is produced at all. This is a hard filter, not a low score — adjusting the confidence threshold will not change the outcome.

Example — US Social Security Number. An SSN is structured as AAA-GG-SSSS (area, group, serial). The Social Security Administration has never issued numbers in certain ranges, so a nine-digit value falling in those ranges is not a possible SSN no matter how it is labeled:

Applied to samples:

Other identifier types apply their own format and validation rules, and several national IDs and driver's licenses include check digits that must reconcile before the value is treated as a candidate.

Resolution: Confirm the sample satisfies the documented format and validation rules for that country's identifier before treating the result as a miss. If your real-world format is legitimate but differs from the canonical one, that is a detector enhancement request rather than a tuning exercise — share samples with us and we will route them to our Detection team.

3. Well-known test values are suppressed

Sequential and repeated values may be suppressed as statistically common test and placeholder data. They are widely used as sample values, so treating them as real identifiers produces unacceptable noise.

Note that these values are structurally valid — they reach the model and are then suppressed, unlike the values in cause 2, which are never evaluated at all. The visible outcome is the same; the mechanism is not.

Resolution: Re-test with a non-sequential, format-valid value. This is the single most common cause of self-inflicted "misses" during validation.

4. Country mismatch, or the country is not supported

No detector exists for that country's identifier, so the miss is real and expected.

Country-specific passport number detectors currently exist for approximately 19 countries, plus generic passport/visa image detection covering all countries. Confirm the current supported list before responding; coverage changes.

Resolution: For unsupported countries, new detectors can be built on request. Please share the country, the identifier type (tax ID, driver's license, passport, national ID), and a few sanitized format examples with us. We can build an ID model(s) within 10 business days.

5. The ID was inside an image or a table

Extraction runs before detection, and it can remove the very thing the detector needs. Two variants, same symptom.

Identity documents as images. Image classification and text detection are separate capabilities. A passport or driver's license image may be classified as an identity document while the number printed on it produces no text finding — or the reverse. Track those as two issues, not one.

Extraction accuracy also matters at the character level. A single misread digit turns a valid identifier into one that fails format or checksum validation, at which point it is dropped at candidate identification (cause 2) and produces nothing at all. Low-resolution, compressed, skewed, or photographed images make this more likely.

IDs in tables. When an identifier sits in a table, the column header — SSN, Social Security Number, Passport No., NIK — is usually the only context the detector has. Whether it survives depends on the file type.

Header preserved: CSV, Excel, and other structured data files. Each value stays tied to its column, so the detector effectively reads:

Patient Name:            Lisa Samson
Phone Number:            303-781-8970
Member Identifier:       1EG4TE5MK73
Social Security Number:  523-36-8350

The label sits right next to the number. Scores Very Likely.

Header lost: images of tables and HTML tables. Screenshots, photos, PDFs, and HTML flatten into a single run of text:

Lisa Samson  303-781-8970  1EG4TE5MK73  523-36-8350  Alprazolam

The header row appears once, at the top. Every row after it is pushed further away:

Name  Phone  Member ID  SSN  Medication      ← the only place "SSN" appears
Row 1 …  …  …  523-36-8350  …                ← header is ~40 characters back
Row 2 …  …  …  489-36-8351  …                ← ~120 characters back
Row 40 …  …  …  478-36-8352  …               ← thousands of characters back

The detector looks for context within a short window around the value. Once the header falls outside that window, the identifier is just a nine-digit number with nothing around it — and an SSN that scores Very Likely as a structured file scores Possible, or produces no finding at all.

This is why partial detection down a long column is such a common report. The first rows are close enough to the header; the rest are not.

Resolution: Scan the source file directly where you can — a CSV or spreadsheet will always outperform a screenshot of the same data. Where image and HTML coverage matters, lower the rule to Likely and validate the noise impact. When reporting, send the original file or full-resolution image rather than a re-screenshot.

See also: Why Sensitive Data in an Image Was Not Detected · Why Detection Results Differ Between File Upload, Copy/Paste, and Screenshot

6. Coverage or configuration gap, not a detection failure

Frequently the detector never ran. Check, in this order:

• Is the detector deployed on the integration where the content lives? A common real-world pattern is SSN and driver's license coverage on chat but not on the file store where the identity documents actually reside.

• Is the detection rule scope set to Content (or Both), not File Name only?

• Is minimum confidence filtering the finding out?

• Is minimum number of findings suppressing a single occurrence?

• Are context rules or keyword gates excluding the content? Temporarily disable and re-test.

7. The identifier was split or fragmented

Values deliberately broken across characters, lines, or fields are not detected, and this is intentional. Detecting fragmented identifiers would require matching arbitrary character combinations and would generate an impractical volume of false positives.

What not to do

Do not lower confidence to Possible as a general fix. At Possible, a nine-digit ID detector will surface effectively every nine-digit number in the environment — expect 40–70% noise. Reserve Possible for narrow cases where exhaustive coverage is explicitly wanted and the volume is accepted, such as a one-time historical scan.

To evaluate the impact of a lower threshold safely, test in the Playground or an isolated test policy rather than changing a production policy.

Reporting a missed detection to Nightfall Support

Non-findings are not retained. For a missed detection there is no stored artifact to investigate after the fact — unlike a false positive, where the content is available in the event finding. Without a sample, the issue cannot be worked.

Contact Nightfall Support and include these three items at minimum:

Also helpful, in rough order of value:

1. The country and the identifier type

2. Your Playground result for the same sample — the confidence score, or "no finding"

3. Detection rule configuration: scope, minimum confidence, minimum findings, any context rules

4. Whether the same sample behaves differently in another tenant or another integration

Triage sequence

1. Paste the sample into the Playground and record the confidence score.

2. If the Playground scores it but the policy did not fire → configuration issue, work cause 6.

3. If the Playground scores it low → context issue, work cause 1. If the content came from an image or a table, work cause 5.

4. If the Playground produces nothing at all → candidate identification, test-value suppression, or missing coverage; work causes 2, 3, and 4.

5. Reproduce in a second tenant to rule out tenant-specific configuration.

6. Still unexplained → contact Nightfall Support with a sanitized sample, the time of the test, and the environment.