Microsoft Sentinel Provider
Th MS Sentinel QueryProvider uses the azure-monitor-query SDK to connect to Microsoft Sentinel workspaces.
Note
This provider replaces an earlier version,
which used KqlMagic as the underlying data connector.
The previous driver is still available but to use it you must
specify MSSentinel_Legacy
as the provider name when creating
the QueryProvider instance.
For more information about the previous driver see Microsoft Sentinel Provider - Legacy Version
Changes from the previous implementation
Uses the MSTICPy built-in Azure authentication by default - you do not have to specify parameters to enable this.
Supports simultaneous queries against multiple workspaces (see below).
Supports user-specified timeout for queries.
Supports proxies (via MSTICPy config or the
proxies
parameter to theconnect
method)The driver supports asynchronous execution of queries. This is used when you create a Query provider with multiple connections (e.g. to different clusters) and when you split queries into time chunks. See Running a query across multiple connections and Splitting Query Execution into Chunks for for more details. This is independent of the ability to specify multiple workspaces in a single connection as described above.
Some of the previous parameters have been deprecated:
mp_az_auth
is replaced byauth_types
(the former still works but will be removed in a future release).mp_az_auth_tenant_id
is replaced bytenant_id
(the former is no longer supported
Sentinel Configuration
You store configuration for your workspace (or workspaces) in
your msticpyconfig.yaml
.
For more information on using and configuring msticpyconfig.yaml see msticpy Package Configuration and MSTICPy Settings Editor
The MS Sentinel connection settings are stored in the
AzureSentinel\\Workspaces
section of the file.
Here is an example.
AzureSentinel:
Workspaces:
# Workspace used if you don't explicitly name a workspace when creating WorkspaceConfig
# Specifying values here overrides config.json settings unless you explicitly load
# WorkspaceConfig with config_file parameter (WorkspaceConfig(config_file="../config.json")
Default:
WorkspaceId: 271f17d3-5457-4237-9131-ae98a6f55c37
TenantId: 335b56ab-67a2-4118-ac14-6eb454f350af
ResourceGroup: soc
SubscriptionId: a5b24e23-a96a-4472-b729-9e5310c83e20
WorkspaceName: Workspace1
# To use these launch with an explicit name - WorkspaceConfig(workspace_name="Workspace2")
Workspace1:
WorkspaceId: "c88dd3c2-d657-4eb3-b913-58d58d811a41"
TenantId: "335b56ab-67a2-4118-ac14-6eb454f350af"
ResourceGroup: soc
SubscriptionId: a5b24e23-a96a-4472-b729-9e5310c83e20
WorkspaceName: Workspace1
TestWorkspace:
WorkspaceId: "17e64332-19c9-472e-afd7-3629f299300c"
TenantId: "4ea41beb-4546-4fba-890b-55553ce6003a"
ResourceGroup: soc
SubscriptionId: a5b24e23-a96a-4472-b729-9e5310c83e20
WorkspaceName: Workspace2
If you only use a single workspace, you only need to create a Default
entry and
add the values for your WorkspaceID and TenantID. You can add other entries here,
for example, SubscriptionID, ResourceGroup. These are not required for the data
queries but are recommended since they are used by other MSTICPy components.
If you use multiple workspaces, you can add further entries here. The key for
each entry (e.g. Workspace1
or TestWorkspace
in the example above)
is normally the name of the Azure Sentinel workspace but
you can use any name you prefer. You use this entry name when connecting
to a workspace.
Loading a QueryProvider for Microsoft Sentinel
qry_prov = QueryProvider(
data_environment="MSSentinel",
)
# or just
qry_prov = QueryProvider("MSSentinel")
Optional parameters for the Sentinel QueryProvider
timeout
: int (seconds)
Specify a timeout for queries. Default is 300 seconds,
the maximum is 600 seconds (10 minutes).
This parameter can be set here or in the connect
method
and overridden for individual queries.
proxies
: Dict[str, str]
Proxy settings for log analytics queries.
If proxies are configured in msticpyconfig.yaml this is used by default.
If specified as a parameter, specify proxies as a dictionary of the form
{protocol: proxy_url}
The only protocol supported by the driver is “https” (other protocols can be set in msticpyconfig.yaml but only https is used here). The proxy_url can contain optional authentication information in the format “https://username:password@proxy_host:port”
If you have a proxy configuration set in msticpyconfig.yaml and
you do not want to use it, set proxies
to None or an empty dictionary.
This parameter can be overridden in connect method.
Connecting to a MS Sentinel Workspace
Once you’ve created a QueryProvider you need to authenticate to Sentinel
Workspace. This is done by calling the connect() function of the Query
Provider. See
connect()
This function takes an initial parameter (called connection_str
for
historical reasons) that can be one of the following:
A WorkspaceConfig instance
A connection string (this is option is being deprecated)
None - in this case it will connect with the
Default
entry from your msticpyconfig.yaml file.
If you omit this parameter you use the workspace
parameter
to specify the workspace entry from msticpyconfig.yaml
to use.
Connecting to a Sentinel workspace
When connecting you can just pass the name of your workspace or
an instance of WorkspaceConfig to the query provider’s connect
method.
qry_prov.connect("Default")
qry_prov.connect(workspace="Default")
qry_prov.connect(workspace="MyOtherWorkspace")
# or, passing WorkspaceConfig
qry_prov.connect(WorkspaceConfig())
# or
qry_prov.connect(WorkspaceConfig(workspace="MyOtherWorkspace"))
MS Sentinel Authentication options
By default, the data provider will use Azure authentication
following the parameters defined in your msticpyconfig.yaml
file
(or the default values if you have not configured them in this file).
To read more about Azure authentication see Azure Authentication
You can override several authentication parameters including:
auth_types - a list of authentication types to try in order
tenant_id - the Azure tenant ID to use for authentication
If you are using a Sovereign cloud rather than the Azure global cloud, you should follow the guidance in Azure Authentication to configure the correct cloud.
Connecting to multiple Sentinel workspaces
There are two mechanisms for querying multiple MS Sentinel workspaces. One is a generic method common to all data providers. For more information on this see Running a query across multiple connections in the main Data Providers documentation.
The other is specific to the Sentinel data provider and is provided by the underlying Azure Monitor client. This latter capability is described in this section.
The Sentinel data provider supports connecting to multiple workspaces by
passing a list of workspace names or workspace IDs to the connect
method.
using the workspaces
or workspace_ids
parameters respectively.
workspace_ids
should be a list or tuple of workspace IDs.
workspaces
should be a list or tuple of workspace names. In order
to use this parameter you must have these workspaces configured in
your msticpyconfig.yaml.
These parameters override the workspace
parameter.
Connecting to multiple workspaces allows you to run queries across these workspaces and return the combined results as a single Pandas DataFrame. The workspaces must use common authentication credentials and are expected to have the same data schema.
qry_prov.connect(workspaces=["Default", "MyOtherWorkspace"])
qry_prov.SecurityAlert.list_alerts()
This will return a DataFrame containing the results of the query,
the results from each workspace will be indicated by the
TenantId
column, which will contain the workspace ID of
each workspace.
Note
This is a mechanism implemented by the underlying azure-monitor-query client library. It is independent of the MSTICPy capability to add multiple connections to a query provider (and run parallel queries against each workspace). You can use either of these but we recommended using one or the other and not both simultaneously.
Warning
Connecting to multiple workspaces like this means
that the schema
property will not return anything. This
only works if you connect to a single workspace. In this case,
it will return the schema of this workspace.
Other parameters for Sentinel connect()
method
For timeout
and proxies
see the section above.
Other parameters for query method
These parameters can be passed to either exec_query or to the built-in
template queries (e.g. list_alerts
).
timeout
: int (seconds) - override the default timeout (or timeout
specified for the driver) for this query.
fail_on_partial
: bool - if True, raise an exception if only
a partial result is returned. If False, return the partial result.
The default is False.
chunk_size
: int - split the query into chunks of this size and
The WorkspaceConfig class
You do not need to know the details of this class but it is used behind the scenes to provide workspace configuration information to the Sentinel data provider.
WorkspaceConfig
handles loading your workspace configuration
and generating a connection string from your configuration.
See WorkspaceConfig API documentation
WorkspaceConfig
works with workspace configuration stored in msticpyconfig.yaml.
To use WorkspaceConfig
, simple create an instance of it. It will automatically build
your connection string for use with the query provider library.
ws_config = WorkspaceConfig()
When called without parameters, WorkspaceConfig loads the “Default”
entry in your msticpyconfig.yaml. To specify a different workspace pass the workspace
parameter
with the name of your workspace entry. This value is the name of
the section in the msticpyconfig.yaml
Workspaces
section.
Note
the workspace
parameter value is the entry heading in
your msticpyconfig.yaml
. As mentioned above, this may
not necessarily be the same as your workspace name.
ws_config = WorkspaceConfig(workspace="TestWorkspace")
To see which workspaces are configured in your msticpyconfig.yaml use
the list_workspaces()
function.
Tip
list_workspaces
is a class function, so you do not need to
instantiate a WorkspaceConfig to call this function.
WorkspaceConfig.list_workspaces()
{'Default': {'WorkspaceId': '271f17d3-5457-4237-9131-ae98a6f55c37',
'TenantId': '335b56ab-67a2-4118-ac14-6eb454f350af'},
'Workspace1': {'WorkspaceId': 'c88dd3c2-d657-4eb3-b913-58d58d811a41',
'TenantId': '335b56ab-67a2-4118-ac14-6eb454f350af'},
'TestWorkspace': {'WorkspaceId': '17e64332-19c9-472e-afd7-3629f299300c',
'TenantId': '4ea41beb-4546-4fba-890b-55553ce6003a'}}
Other MS Sentinel Documentation
Built-in Queries for Microsoft Sentinel.
See also: Sentinel KQL driver API documentation