Data Discovery

Add a data source connection via the Integrations API. Credentials are encrypted using AES encryption before storage.

curl -X POST https://api.neostra.io/v1/integrations \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Production PostgreSQL",
    "type": "POSTGRESQL",
    "host": "db.example.com",
    "port": 5432,
    "database": "app_db",
    "username": "scanner_user",
    "password": "encrypted-credential"
  }'

Create a scan targeting one or more integrations. The orchestration service breaks the scan into individual scan tasks for each table, collection, or bucket.

curl -X POST https://api.neostra.io/v1/scans \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Q1 Full Discovery Scan",
    "integrationIds": ["int_abc123", "int_def456"],
    "scanType": "FULL"
  }'

The Python scanner worker continuously polls the scan_items table for pending tasks. For each task, it decrypts credentials via crypto.py, connects to the data source, samples data, and runs PII detection through pii_engine.py.

The PII engine uses Microsoft Presidio augmented with custom regex patterns to identify sensitive data. Detected entities are classified and written back as findings with confidence scores.

Access discovered findings through the Findings API or the Neostra dashboard. Findings are linked to data points, attributes, and can be mapped to processing activities for RoPA compliance.

Scan all configured data sources to identify where personal data resides. The scanner automatically classifies found data using the classification schema.

Use the ProcessingActivityController to link discovered data to declared processing purposes, legal bases, and retention periods.

Categorize discovered data by data subject type (customers, employees, partners) using the DataSubjectController.

The system aggregates all mapped data into a structured RoPA report that includes: purposes of processing, categories of personal data, categories of data subjects, recipients and transfers, retention periods, and technical/organizational security measures.