Querying and Importing Data

Common Data Provider Operations

• Query Providers Usage (common to all data sources)
  • Creating a Query Provider
  • Connecting to a Data Environment
  • Listing available queries
  • Searching for a query
  • Running a pre-defined query
  • Running an ad hoc query
  • Running a query across multiple connections
  • Splitting Query Execution into Chunks
  • Dynamically adding new queries

Individual Data Environments

• Microsoft Sentinel Provider
  • Changes from the previous implementation
  • Sentinel Configuration
  • Loading a QueryProvider for Microsoft Sentinel
  • Connecting to a MS Sentinel Workspace
  • The WorkspaceConfig class
  • Other MS Sentinel Documentation
• Microsoft 365 Defender Provider
  • M365 Defender Configuration
  • Loading a QueryProvider for M365 Defender
  • Specifying the Defender Cloud Instance to Connect to
  • Connecting to M365 Defender
  • Other M365 Defender Documentation
• Microsoft Graph API Provider
  • Microsoft Graph Configuration
  • Loading a QueryProvider for Microsoft Graph
  • Connecting to Microsoft Graph
  • Other Microsoft Graph Documentation
• The LocalData provider
  • LocalData Configuration
  • Loading a QueryProvider for LocalData
  • Connecting to LocalData
  • Example usage of LocalData driver
  • Other LocalData Documentation
• Splunk Provider
  • Splunk Configuration
  • Loading a QueryProvider for Splunk
  • Connecting to Splunk
  • Listing available queries
  • Running pre-defined queries
  • Running an ad hoc Splunk query
  • Other Splunk Documentation
• Azure Resource Graph Provider
  • Azure Resource Graph Configuration
  • Loading a Query Provider for Azure Resource Graph
  • Connecting to Azure Resource Graph
  • Listing available queries
  • Running pre-defined query
  • End-to-end Example
  • Other Azure Resource Graph Documentation
• Open Threat Research Security Datasets data provider and browser
  • Using the Data Provider to download datasets
  • Mordor Browser
• Sumologic Provider
  • Sumologic Configuration
  • Loading a QueryProvider for Sumologic
  • Connecting to Sumologic
  • Running a Sumologic query
  • Other Sumologic Documentation
• Azure Data Explorer/Kusto Provider
  • Changes from the previous implementation
  • Kusto Configuration
  • Loading a QueryProvider for Kusto
  • Connecting to a Kusto cluster
  • Kusto QueryProvider methods and properties
  • Running Ad Hoc queries
  • Writing query templates for Kusto clusters
  • Other Kusto Documentation
• Cybereason Provider
  • Cybereason Configuration
  • Loading and Connecting a QueryProvider for Cybereason
  • Listing available queries
  • Running pre-defined queries
  • Running an ad-hoc Cybereason query
  • Other Cybereason Documentation
• The OSQuery provider
  • OSQuery Configuration
  • Expected log file format
  • Using the OSQuery provider
  • Other OSQuery Provider Documentation
• The Velociraptor provider
  • Velociraptor Configuration
  • Expected log file format
  • Using the Velociraptor provider
• Microsoft Sentinel Provider - Legacy Version
  • Sentinel Configuration
  • Loading a QueryProvider for Microsoft Sentinel
  • Connecting to a MS Sentinel Workspace
  • Other MS Sentinel Documentation
• Azure Data Explorer/Kusto Provider - Legacy Version
  • Kusto Configuration
  • Data Query Format for Kusto clusters
  • Loading a QueryProvider for Kusto
  • Connecting to a Kusto cluster
  • Additional Kusto query parameters
  • Other Kusto Documentation

Built-in Data Queries

• Data Queries Reference
  • Queries for Microsoft Sentinel
  • Queries for Microsoft 365 Defender
  • Queries for Microsoft Graph
  • Queries for Splunk
  • Queries for Azure Resource Graph
  • Queries for Sumologic
  • Queries for Local Data

Other Data Modules and Functions

• Setting up Process Auditing for Linux in Azure Sentinel
  • Add your Linux VMs to the Log Analytics Workspace
  • Configure Auditing on your Linux VMs
  • Add Auditd as a Custom log in Log Analytics
  • Reading Audit Data from Log Analytics
• Data Uploaders
  • Uploading data to Azure Sentinel/Log Analytics
  • Uploading data to Splunk
• Data Masking Functions
  • Import the module
  • Individual Masking Functions
  • Masking DataFrames
  • Creating custom mappings
  • Using hash_item to preserve the structure/look of the hashed input
  • Checking Your Masking Results
• Reading from and writing to Azure Blob AzureBlobStorage
  • Description
  • Import the module
  • Initialize the class and connect
  • List Containers
  • Create a Container
  • List Blobs
  • Write to a Blob
  • Read from a Blob
  • Delete a Blob
  • Generate a SAS Token for a Blob
• SQL TO KQL Conversion (Experimental)
  • Caveat Emptor!
  • Simple SQL Query
  • SQL Joins
  • Table Renaming
  • Join with Aliases
  • Unions and Group By
  • Aliased and Calculated Select Columns

Contributing a Data Provider

• Writing and Contributing a Data Provider
  • Implementing a data provider
  • 1. Write the driver class
  • 2. Customize the driver
  • 3. Register the driver
  • 4. Add queries
  • 5. Add settings definition
  • 6. Add provider documentation
  • 7. Create driver unit tests