Microsoft Sentinel Dynamic Summaries

Dynamic Summaries are a Sentinel feature that allow you to persist results of query jobs in a summarized/serialized form. This might be useful for keeping results of daily watch jobs, for example. We will be using it in MSTICPy notebooks to publish more complex result sets from automated notebook runs.

Operations available include:

• Retrieve list of current dynamic Summaries

• Retrieve a full dynamic summary

• Create a dynamic summary

• Delete a dynamic summary

• Update an existing dynamic summary

There is also a MSTICPy Python class DynamicSummary that lets you work with dynamic summary data in your notebook or Python application.

List Dynamic Summaries

list_dynamic_summaries returns a list of all the dynamic summaries saved in the workspace. This includes the name and ID of the summaries, who created it, when and with what details.

Note

There is current no way to filter the list of results returned

See list_dynamic_summaries

sentinel.list_dynamic_summaries()

id	name	etag	type	systemData.createdAt	systemData.createdBy	systemData.createdByType	systemData.lastModifiedAt	systemData.lastModifiedBy	systemData.lastModifiedByType	properties.summaryName	properties.sourceInfo.TI Records	properties.tactics	properties.techniques	properties.summaryDescription	properties.tenantId
/subscriptions/40dcc8bf..	cea27320-829c-4654-bbf0-b14367483418	“00002e10-0000-0a00-0000-639a7fa00000”	Microsoft.SecurityInsights/dynamicsummaries	2022-12-15T01:59:59.1574875Z	ianhelle@microsoft.com	User	2022-12-15T01:59:59.1574875Z	ianhelle@microsoft.com	User	test2	misc	[]	[]	Test description	72f988bf-86f1-41af-91ab-2d7cd011db47
/subscriptions/40dcc8bf..	5b574f4f-047c-4056-97aa-136b42b1bc5a	“0000c010-0000-0a00-0000-639a836b0000”	Microsoft.SecurityInsights/dynamicsummaries	2022-12-15T02:16:10.7127404Z	ianhelle@microsoft.com	User	2022-12-15T02:16:10.7127404Z	ianhelle@microsoft.com	User	test3	misc	[]	[]	Test description	72f988bf-86f1-41af-91ab-2d7cd011db47
/subscriptions/40dcc8bf..	fba7525c-3e67-4b7f-9a78-36c7cf0f6423	“0000062d-0000-0a00-0000-639d0c900000”	Microsoft.SecurityInsights/dynamicsummaries	2022-12-15T02:29:27.5201639Z	ianhelle@microsoft.com	User	2022-12-17T00:25:51.7860328Z	ianhelle@microsoft.com	User	test4	misc	[]	[]	A new description	72f988bf-86f1-41af-91ab-2d7cd011db47

Create a Dynamic Summary

To create a dynamic summary, you may find it easier to work with the MSTICPy DynamicSummary class. You can read more about this in the DynamicSummary Class section below.

You can create a Dynamic Summary in MS Sentinel using the create_dynamic_summary. This creates and uploads the summary object to the Sentinel workspace.

You can supply the properties of the dynamic summary as parameters to the function:

sentinel.connect()
sentinel.create_dynamic_summary(
    name="My_XYZ_Summary",
    description="Summarizing the running of the XYZ job.",
    data=summary_df,
)

You can supply additional properties of the summary by adding additional keyword parameters to the create_dynamic_summary call. See DynamicSummary for a list of available properties.

sentinel.connect()
sentinel.create_dynamic_summary(
    name="My_XYZ_Summary",
    description="Summarizing the running of the XYZ job.",
    data=summary_df,
    tactics=["discovery", "exploitation"],
    techniques=["T1064", "T1286"],
    search_key="host.domain.dom",
)

You can also create a DynamicSummary object and pass that as the single parameter to create_dynamic_summary.

dyn_summary = sentinel.new_dynamic_summary(
    summary_name="My new summary",
    summary_description="Description of summary",
    source_info={"TI Records": "misc"},
    summary_items=ti_summary_df,
)
sentinel.create_dynamic_summary(dyn_summary)

Get a Dynamic Summary

You can retrieve a DynamicSummary using the get_dynamic_summary method.

dyn_summary = sentinel.get_dynamic_summary(
    summary_id="cea27320-829c-4654-bbf0-b14367483418"
)
dyn_summary

DynamicSummary(id=cea27320-829c-4654-bbf0-b14367483418, name=test2, items=0)

Note

The Sentinel API does not return any Summary Item records, only the metadata properties associated with the DynamicSummary record. Use the summary_items parameter described next.

Supplying a summary_items=True parameter will re-route the request for Dynamic Summary data to the MS Sentinel DynamicSummary table. It will execute a query to retrieve the summary items along with the summary metadata.

dyn_summary = sentinel.get_dynamic_summary(
    summary_id="cea27320-829c-4654-bbf0-b14367483418",
    summary_items=True
)
dyn_summary

Please wait. Loading Kqlmagic extension...done
Connecting...
popup schema 52b1ab41-869e-4138-9e40-2a4457f09bf0@loganalytics
connected
DynamicSummary(
    summary_id='cea27320-829c-4654-bbf0-b14367483418'
    summary_name='test2'
    summary_description='Test description'
    tenant_id='72f988bf-86f1-41af-91ab-2d7cd011db47'
    tactics='[]'
    techniques='[]'
    ws_tenant_id='52b1ab41-869e-4138-9e40-2a4457f09bf0'
    updated_time_utc='2022-12-15 01:59:59.157487500+00:00'
    source_info='{'TI Records': 'misc'}'
    created_by='user@microsoft.com'
    summary_data_type='Summary'
    time_generated='2022-12-15 02:00:03.152763400+00:00'
    created_time_utc='2022-12-15 01:59:59.157487500+00:00'
    summary_status='Active'
    updated_by='user@microsoft.com'
    summary_items=6
)

Note

Because this triggers a QueryProvider to load it may involve some additional initial delay.

You can display the summary items as a DataFrame.

dyn_summary.to_df()

index	Ioc	IocType	QuerySubtype	Provider	Result	Severity	Details	TimeGenerated
OTX	hXXp://38[.]75[.]37[.]1/static/encrypt.min.js	url		OTX	True	2	{‘pulse_count’: 3, ‘names’: [‘Underminer EK’	2022-12-15 01:55:15.135136+00:00
VirusTotal	hXXp://38[.]75[.]37[.]1/static/encrypt.min.js	url		VirusTotal	False	0	Request forbidden. Allowed query rate may ha	2022-12-15 01:55:15.135136+00:00
XForce	hXXp://38[.]75[.]37[.]1/static/encrypt.min.js	url		XForce	False	0	Not found.	2022-12-15 01:55:15.135136+00:00
AzSTI	hXXp://38[.]75[.]37[.]1/static/encrypt.min.js	url		AzSTI	False	0	Not found.	2022-12-15 01:55:15.135136+00:00
OPR	hXXp://38[.]75[.]37[.]1/static/encrypt.min.js	url		OPR	False	0	IoC type url not supported.	2022-12-15 01:55:15.135136+00:00
Tor	hXXp://38[.]75[.]37[.]1/static/encrypt.min.js	url		Tor	True	0	IoC type url not supported.	2022-12-15 01:55:15.135136+00:00

Update a Dynamic Summary

You can add additional summary items to an existing Dynamic Summary using the update_dynamic_summary method. You can also change existing properties of the DynamicSummary.

Warning

if the summary_id supplied does not exist a new DynamicSummary record will be created.

dyn_summary.summary_description = "A new description"
dyn_summary.summary_id = "fba7525c-3e67-4b7f-9a78-36c7cf0f6423"
sentinel.update_dynamic_summary(dyn_summary)

ds_upd = sentinel.get_dynamic_summary(dyn_summary.summary_id)
ds_upd.to_json()

Dynamic summary created/updated.

'{"summaryId": "122f7de3-7276-490b-9db0-11e9f07873d0", "summaryName": "test4",
"summaryDescription": "A new description", "tenantId": "72f988bf-86f1-41af-91ab-2d7cd011db47",
"tactics": [], "techniques": [], "rawContent": [], "sourceInfo": {"TI Records": "misc"}}'

Note

If you have summary items in the Dynamic Summary that you pass to the update_dynamic_summary method, the items will be appended to any existing items.

Delete a Dynamic Summary

Dynamic Summaries can be deleted by calling delete_dynamic_summary and passing in the summary_id.

Note

Since MS Sentinel/Log Analytics tables are append-only, the records will not be removed from the DynamicSummary table but the summary will be deactivated and marked as Deleted.

sentinel.delete_dynamic_summary(summary_id="cea27320-829c-4654-bbf0-b14367483418")

DynamicSummary Class

API reference: DynamicSummary

This is Python class to encapsulate a Sentinel Dynamic Summary object. It is used only for local manipulation of the Summary object and does not affect the version stored in Sentinel unless you upload the changes using one of the APIs described earlier.

Using DynamicSummary you can:

• prepare a new summary object in your code or interactively before uploading to Sentinel

• extract summary items as a data frame.

• use it to amend/update an existing dynamic summary.

• view a summary of the Dynamic summary

The most important methods are described below.

DynamicSummary initializer

DynamicSummary

You can create a DynamicSummary object by supplying the required attributes as parameters or create a “bare” class and add them as attributes.

import msticpy as mp
dyn_summary = mp.MicrosoftSentinel.new_dynamic_summary(
    summary_name="My new summary",
    summary_description="Description of summary",
    source_info={"TI Records": "misc"},
)

dyn_summary.add_summary_items(data=ti_df)
dyn_summary

DynamicSummary(id=49f627af-1f05-42e6-9951-6a2bd7b9b233, name=My new summary, items=6)

You can also pass a DataFrame to new_summary as summary_items instead of adding them with a separate call to add_summary_items.

Adding and appending Summary items

The add_summary_items method takes one of:

• A pandas DataFrame

• A list/iterable of DynamicSummaryItem.

• A list/iterable of dictionaries, each of which contains the keys and values need for the summary item.

You can specify additional properties for the summary items by adding additional parameters to add_summary_items. See add_summary_items for a list of available properties.

dyn_summary.add_summary_items(
    data=summary_df,
    tactics=["discovery", "exploitation"],
    techniques=["T1064", "T1286"],
    observable_type="Account",
    observable_value="user@some.dom",
)

If your source data is in a DataFrame you can also use the DataFrame rows for some or all of the DynamicSummaryItem properties. Use the summary_fields parameter to specify which columns should be used to populate the property value for that row.

dyn_summary.add_summary_items(
    data=summary_df,
    tactics=["discovery", "exploitation"],
    techniques=["T1064", "T1286"],
    observable_type="Account",
    observable_value="user@some.dom",
    summary_fields={
         "user_name": "observable_value",
         "user_name", "search_key",
    }
)

add_summary_items will remove any existing summary items and replace with the new set specified.

append_summary_items works in the same as add_summary_items but will add to the current set without erasing existing summary items. This is useful for updating an existing summary with new rows.

dyn_summary = sentinel.get_dynamic_summary(summary_id="123123...")
dyn_summary.append_summary_items(data=new_items_df)
sentinel.update_dynamic_summary(dyn_summary)

Output SummaryItems as DataFrame

You can retrieve the summary items as a DataFrame using the to_df method.

dyn_summary.to_df()

	index	Ioc	IocType	QuerySubtype	Provider
0	OTX	hXXp://38[.]75[.]37[.]1/static/encrypt.min.js	url		OTX
1	VirusTotal	hXXp://38[.]75[.]37[.]1/static/encrypt.min.js	url		VirusTotal
2	XForce	hXXp://38[.]75[.]37[.]1/static/encrypt.min.js	url		XForce
3	AzSTI	hXXp://38[.]75[.]37[.]1/static/encrypt.min.js	url		AzSTI
4	OPR	hXXp://38[.]75[.]37[.]1/static/encrypt.min.js	url		OPR
5	Tor	hXXp://38[.]75[.]37[.]1/static/encrypt.min.js	url		Tor

Convert Dynamic Summary to/from JSON

The instance method to_json returns the summary object serialized to a JSON string. The to_json_api method also does this but adds a wrapper layer that’s expected by the Sentinel API.

The class method from_json will return a DynamicSummary instance from the JSON data. The input to this can either be the simple format (as returned by to_json()) or the API wrapped format (to_json_api())

View contents of the Dynamic Summary

You can view a text representation of a DynamicSummary object by running it in a cell or printing it with the Python print function

print(dyn_summary)

DynamicSummary(
    summary_id='cea27320-829c-4654-bbf0-b14367483418'
    summary_name='test2'
    summary_description='Test description'
    tenant_id='72f988bf-86f1-41af-91ab-2d7cd011db47'
    tactics='[]'
    techniques='[]'
    ws_tenant_id='52b1ab41-869e-4138-9e40-2a4457f09bf0'
    updated_time_utc='2022-12-15 01:59:59.157487500+00:00'
    source_info='{'TI Records': 'misc'}'
    created_by='user@microsoft.com'
    summary_data_type='Summary'
    time_generated='2022-12-15 02:00:03.152763400+00:00'
    created_time_utc='2022-12-15 01:59:59.157487500+00:00'
    summary_status='Active'
    updated_by='user@microsoft.com'
    summary_items=6
)

Using the fields attribute for legal field names

When creating a DynamicSummary or DynamicSummaryItem you frequently need to specify the field names as parameters. To help prevent typos, both classes have a fields attribute that contains the names of all legal fields.

You can list all of the fields for each class just by running (in a notebook/IPython) or printing the fields attribute.

DynamicSummary.fields

Fields:
    SUMMARY_ID='summary_id'
    SUMMARY_NAME='summary_name'
    SUMMARY_DESCRIPTION='summary_description'
    TENANT_ID='tenant_id'
    RELATION_NAME='relation_name'
    RELATION_ID='relation_id'
    SEARCH_KEY='search_key'
    TACTICS='tactics'
    TECHNIQUES='techniques'
    SOURCE_INFO='source_info'
    SUMMARY_ITEMS='summary_items'

This example shows how you might use the fields attribute in code.

dyn_summary.add_summary_items(
    data=df,
    summary_fields = {
        DynamicSummaryItem.fields.EVENT_TIME_UTC: "TimeGenerated",
        DynamicSummaryItem.fields.SEARCH_KEY: "UserPrincipalName",
    }
)