Microsoft Sentinel Dynamic Summaries
Dynamic Summaries are a Sentinel feature that allow you to persist results of query jobs in a summarized/serialized form. This might be useful for keeping results of daily watch jobs, for example. We will be using it in MSTICPy notebooks to publish more complex result sets from automated notebook runs.
Operations available include:
Retrieve list of current dynamic Summaries
Retrieve a full dynamic summary
Create a dynamic summary
Delete a dynamic summary
Update an existing dynamic summary
There is also a MSTICPy Python class
DynamicSummary
that lets you work with dynamic summary data in your notebook or Python application.
List Dynamic Summaries
list_dynamic_summaries
returns a list of all the dynamic summaries saved in the workspace.
This includes the name and ID of the summaries, who
created it, when and with what details.
Note
There is current no way to filter the list of results returned
sentinel.list_dynamic_summaries()
id |
name |
etag |
type |
systemData.createdAt |
systemData.createdBy |
systemData.createdByType |
systemData.lastModifiedAt |
systemData.lastModifiedBy |
systemData.lastModifiedByType |
properties.summaryName |
properties.sourceInfo.TI Records |
properties.tactics |
properties.techniques |
properties.summaryDescription |
properties.tenantId |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
/subscriptions/40dcc8bf.. |
cea27320-829c-4654-bbf0-b14367483418 |
“00002e10-0000-0a00-0000-639a7fa00000” |
Microsoft.SecurityInsights/dynamicsummaries |
2022-12-15T01:59:59.1574875Z |
User |
2022-12-15T01:59:59.1574875Z |
User |
test2 |
misc |
[] |
[] |
Test description |
72f988bf-86f1-41af-91ab-2d7cd011db47 |
||
/subscriptions/40dcc8bf.. |
5b574f4f-047c-4056-97aa-136b42b1bc5a |
“0000c010-0000-0a00-0000-639a836b0000” |
Microsoft.SecurityInsights/dynamicsummaries |
2022-12-15T02:16:10.7127404Z |
User |
2022-12-15T02:16:10.7127404Z |
User |
test3 |
misc |
[] |
[] |
Test description |
72f988bf-86f1-41af-91ab-2d7cd011db47 |
||
/subscriptions/40dcc8bf.. |
fba7525c-3e67-4b7f-9a78-36c7cf0f6423 |
“0000062d-0000-0a00-0000-639d0c900000” |
Microsoft.SecurityInsights/dynamicsummaries |
2022-12-15T02:29:27.5201639Z |
User |
2022-12-17T00:25:51.7860328Z |
User |
test4 |
misc |
[] |
[] |
A new description |
72f988bf-86f1-41af-91ab-2d7cd011db47 |
Create a Dynamic Summary
To create a dynamic summary, you may find it easier to work with the MSTICPy
DynamicSummary
class. You can read more about this in the DynamicSummary Class section below.
You can create a Dynamic Summary in MS Sentinel using the
create_dynamic_summary
.
This creates and uploads the summary object to the Sentinel workspace.
You can supply the properties of the dynamic summary as parameters to the function:
sentinel.connect()
sentinel.create_dynamic_summary(
name="My_XYZ_Summary",
description="Summarizing the running of the XYZ job.",
data=summary_df,
)
You can supply additional properties of the summary by adding additional
keyword parameters to the create_dynamic_summary
call. See
DynamicSummary
for a list of available properties.
sentinel.connect()
sentinel.create_dynamic_summary(
name="My_XYZ_Summary",
description="Summarizing the running of the XYZ job.",
data=summary_df,
tactics=["discovery", "exploitation"],
techniques=["T1064", "T1286"],
search_key="host.domain.dom",
)
You can also create a DynamicSummary
object and pass that as the
single parameter to create_dynamic_summary
.
dyn_summary = sentinel.new_dynamic_summary(
summary_name="My new summary",
summary_description="Description of summary",
source_info={"TI Records": "misc"},
summary_items=ti_summary_df,
)
sentinel.create_dynamic_summary(dyn_summary)
Get a Dynamic Summary
You can retrieve a DynamicSummary using the
get_dynamic_summary
method.
dyn_summary = sentinel.get_dynamic_summary(
summary_id="cea27320-829c-4654-bbf0-b14367483418"
)
dyn_summary
DynamicSummary(id=cea27320-829c-4654-bbf0-b14367483418, name=test2, items=0)
Note
The Sentinel API does not return any Summary Item records,
only the metadata properties associated with the DynamicSummary record.
Use the summary_items
parameter described next.
Supplying a summary_items=True
parameter will re-route the
request for Dynamic Summary data to the MS Sentinel DynamicSummary
table.
It will execute a query to retrieve the summary items along with the
summary metadata.
dyn_summary = sentinel.get_dynamic_summary(
summary_id="cea27320-829c-4654-bbf0-b14367483418",
summary_items=True
)
dyn_summary
Please wait. Loading Kqlmagic extension...done
Connecting...
popup schema 52b1ab41-869e-4138-9e40-2a4457f09bf0@loganalytics
connected
DynamicSummary(
summary_id='cea27320-829c-4654-bbf0-b14367483418'
summary_name='test2'
summary_description='Test description'
tenant_id='72f988bf-86f1-41af-91ab-2d7cd011db47'
tactics='[]'
techniques='[]'
ws_tenant_id='52b1ab41-869e-4138-9e40-2a4457f09bf0'
updated_time_utc='2022-12-15 01:59:59.157487500+00:00'
source_info='{'TI Records': 'misc'}'
created_by='user@microsoft.com'
summary_data_type='Summary'
time_generated='2022-12-15 02:00:03.152763400+00:00'
created_time_utc='2022-12-15 01:59:59.157487500+00:00'
summary_status='Active'
updated_by='user@microsoft.com'
summary_items=6
)
Note
Because this triggers a QueryProvider to load it may involve some additional initial delay.
You can display the summary items as a DataFrame.
dyn_summary.to_df()
index |
Ioc |
IocType |
QuerySubtype |
Provider |
Result |
Severity |
Details |
TimeGenerated |
---|---|---|---|---|---|---|---|---|
OTX |
hXXp://38[.]75[.]37[.]1/static/encrypt.min.js |
url |
OTX |
True |
2 |
{‘pulse_count’: 3, ‘names’: [‘Underminer EK’ |
2022-12-15 01:55:15.135136+00:00 |
|
VirusTotal |
hXXp://38[.]75[.]37[.]1/static/encrypt.min.js |
url |
VirusTotal |
False |
0 |
Request forbidden. Allowed query rate may ha |
2022-12-15 01:55:15.135136+00:00 |
|
XForce |
hXXp://38[.]75[.]37[.]1/static/encrypt.min.js |
url |
XForce |
False |
0 |
Not found. |
2022-12-15 01:55:15.135136+00:00 |
|
AzSTI |
hXXp://38[.]75[.]37[.]1/static/encrypt.min.js |
url |
AzSTI |
False |
0 |
Not found. |
2022-12-15 01:55:15.135136+00:00 |
|
OPR |
hXXp://38[.]75[.]37[.]1/static/encrypt.min.js |
url |
OPR |
False |
0 |
IoC type url not supported. |
2022-12-15 01:55:15.135136+00:00 |
|
Tor |
hXXp://38[.]75[.]37[.]1/static/encrypt.min.js |
url |
Tor |
True |
0 |
IoC type url not supported. |
2022-12-15 01:55:15.135136+00:00 |
Update a Dynamic Summary
You can add additional summary items to an existing Dynamic Summary
using the
update_dynamic_summary
method. You can also change existing properties of the DynamicSummary.
Warning
if the summary_id supplied does not exist a new DynamicSummary record will be created.
dyn_summary.summary_description = "A new description"
dyn_summary.summary_id = "fba7525c-3e67-4b7f-9a78-36c7cf0f6423"
sentinel.update_dynamic_summary(dyn_summary)
ds_upd = sentinel.get_dynamic_summary(dyn_summary.summary_id)
ds_upd.to_json()
Dynamic summary created/updated.
'{"summaryId": "122f7de3-7276-490b-9db0-11e9f07873d0", "summaryName": "test4",
"summaryDescription": "A new description", "tenantId": "72f988bf-86f1-41af-91ab-2d7cd011db47",
"tactics": [], "techniques": [], "rawContent": [], "sourceInfo": {"TI Records": "misc"}}'
Note
If you have summary items in the Dynamic Summary that you pass
to the update_dynamic_summary
method, the items will be appended
to any existing items.
Delete a Dynamic Summary
Dynamic Summaries can be deleted by calling
delete_dynamic_summary
and passing in the summary_id
.
Note
Since MS Sentinel/Log Analytics tables are append-only, the records
will not be removed from the DynamicSummary
table but the summary will
be deactivated and marked as Deleted.
sentinel.delete_dynamic_summary(summary_id="cea27320-829c-4654-bbf0-b14367483418")
DynamicSummary Class
API reference: DynamicSummary
This is Python class to encapsulate a Sentinel Dynamic Summary object. It is used only for local manipulation of the Summary object and does not affect the version stored in Sentinel unless you upload the changes using one of the APIs described earlier.
Using DynamicSummary
you can:
prepare a new summary object in your code or interactively before uploading to Sentinel
extract summary items as a data frame.
use it to amend/update an existing dynamic summary.
view a summary of the Dynamic summary
The most important methods are described below.
DynamicSummary initializer
You can create a DynamicSummary
object by supplying the required attributes as
parameters or create a “bare” class and add them as attributes.
import msticpy as mp
dyn_summary = mp.MicrosoftSentinel.new_dynamic_summary(
summary_name="My new summary",
summary_description="Description of summary",
source_info={"TI Records": "misc"},
)
dyn_summary.add_summary_items(data=ti_df)
dyn_summary
DynamicSummary(id=49f627af-1f05-42e6-9951-6a2bd7b9b233, name=My new summary, items=6)
You can also pass a DataFrame to new_summary
as summary_items
instead
of adding them with a separate call to
add_summary_items
.
Adding and appending Summary items
The add_summary_items
method takes one of:
A pandas DataFrame
A list/iterable of
DynamicSummaryItem
.A list/iterable of dictionaries, each of which contains the keys and values need for the summary item.
You can specify additional properties for the summary items by adding additional
parameters to add_summary_items
.
See add_summary_items
for a list of available properties.
dyn_summary.add_summary_items(
data=summary_df,
tactics=["discovery", "exploitation"],
techniques=["T1064", "T1286"],
observable_type="Account",
observable_value="user@some.dom",
)
If your source data is in a DataFrame you can also use the DataFrame
rows for some or all of the DynamicSummaryItem properties. Use the
summary_fields
parameter to specify which columns should be used
to populate the property value for that row.
dyn_summary.add_summary_items(
data=summary_df,
tactics=["discovery", "exploitation"],
techniques=["T1064", "T1286"],
observable_type="Account",
observable_value="user@some.dom",
summary_fields={
"user_name": "observable_value",
"user_name", "search_key",
}
)
add_summary_items
will remove any existing summary items and
replace with the new set specified.
append_summary_items
works in the same as add_summary_items
but will add to the current set without
erasing existing summary items. This is useful for updating an existing
summary with new rows.
dyn_summary = sentinel.get_dynamic_summary(summary_id="123123...")
dyn_summary.append_summary_items(data=new_items_df)
sentinel.update_dynamic_summary(dyn_summary)
Output SummaryItems as DataFrame
You can retrieve the summary items as a DataFrame using the
to_df
method.
dyn_summary.to_df()
index |
Ioc |
IocType |
QuerySubtype |
Provider |
|
---|---|---|---|---|---|
0 |
OTX |
hXXp://38[.]75[.]37[.]1/static/encrypt.min.js |
url |
OTX |
|
1 |
VirusTotal |
hXXp://38[.]75[.]37[.]1/static/encrypt.min.js |
url |
VirusTotal |
|
2 |
XForce |
hXXp://38[.]75[.]37[.]1/static/encrypt.min.js |
url |
XForce |
|
3 |
AzSTI |
hXXp://38[.]75[.]37[.]1/static/encrypt.min.js |
url |
AzSTI |
|
4 |
OPR |
hXXp://38[.]75[.]37[.]1/static/encrypt.min.js |
url |
OPR |
|
5 |
Tor |
hXXp://38[.]75[.]37[.]1/static/encrypt.min.js |
url |
Tor |
Convert Dynamic Summary to/from JSON
The instance method
to_json
returns the summary object
serialized to a JSON string. The
to_json_api
method also does this
but adds a wrapper layer that’s expected by the Sentinel API.
The class method
from_json
will return a DynamicSummary instance from
the JSON data. The input to this can either be the simple format (as returned
by to_json()
) or the API wrapped format (to_json_api()
)
View contents of the Dynamic Summary
You can view a text representation of a DynamicSummary object
by running it in a cell or printing it with the Python print
function
print(dyn_summary)
DynamicSummary(
summary_id='cea27320-829c-4654-bbf0-b14367483418'
summary_name='test2'
summary_description='Test description'
tenant_id='72f988bf-86f1-41af-91ab-2d7cd011db47'
tactics='[]'
techniques='[]'
ws_tenant_id='52b1ab41-869e-4138-9e40-2a4457f09bf0'
updated_time_utc='2022-12-15 01:59:59.157487500+00:00'
source_info='{'TI Records': 'misc'}'
created_by='user@microsoft.com'
summary_data_type='Summary'
time_generated='2022-12-15 02:00:03.152763400+00:00'
created_time_utc='2022-12-15 01:59:59.157487500+00:00'
summary_status='Active'
updated_by='user@microsoft.com'
summary_items=6
)
Using the fields
attribute for legal field names
When creating a DynamicSummary or DynamicSummaryItem you
frequently need to specify the field names as parameters.
To help prevent typos, both classes have a fields
attribute
that contains the names of all legal fields.
You can list all of the fields for each class just by running
(in a notebook/IPython) or printing the fields
attribute.
DynamicSummary.fields
Fields:
SUMMARY_ID='summary_id'
SUMMARY_NAME='summary_name'
SUMMARY_DESCRIPTION='summary_description'
TENANT_ID='tenant_id'
RELATION_NAME='relation_name'
RELATION_ID='relation_id'
SEARCH_KEY='search_key'
TACTICS='tactics'
TECHNIQUES='techniques'
SOURCE_INFO='source_info'
SUMMARY_ITEMS='summary_items'
This example shows how you might use the fields
attribute in
code.
dyn_summary.add_summary_items(
data=df,
summary_fields = {
DynamicSummaryItem.fields.EVENT_TIME_UTC: "TimeGenerated",
DynamicSummaryItem.fields.SEARCH_KEY: "UserPrincipalName",
}
)