msticpy.nbtools package

msticpy.nbtools.azure_ml_tools module

Checker functions for Azure ML notebooks.

msticpy.nbtools.azure_ml_tools.check_mp_ver(min_msticpy_ver: Union[str, Tuple], extras: Optional[List[str]])

Check and optionally update the current version of msticpy.

Parameters

min_msticpy_ver (Tuple[int, int]) – Minimum MSTICPy version
extras (Optional[List[str]], optional) – A list of extras required for MSTICPy

Raises

ImportError – If MSTICPy version is insufficient and we need to upgrade

msticpy.nbtools.azure_ml_tools.check_python_ver(min_py_ver: Union[str, Tuple] = '3.6')

Check the current version of the Python kernel.

Parameters: min_py_ver (Tuple[int, int]) – Minimum Python version
Raises: RuntimeError – If the Python version does not support the notebook.

msticpy.nbtools.azure_ml_tools.check_versions(min_py_ver: Union[str, Tuple] = '3.6', min_mp_ver: Union[str, Tuple] = '1.8.2', extras: Optional[List[str]] = None, mp_release: Optional[str] = None, **kwargs)

Check the current versions of the Python kernel and MSTICPy.

Parameters

min_py_ver (Union[Tuple[int, int], str]) – Minimum Python version
min_mp_ver (Union[Tuple[int, int], str]) – Minimum MSTICPy version
extras (Optional[List[str]], optional) – A list of extras required for MSTICPy
mp_release (Optional[str], optional) – Override the MSTICPy release version. This can also be specified in the environment variable ‘MP_TEST_VER’

Raises

RuntimeError – If the Python version does not support the notebook. If the MSTICPy version does not support the notebook and the user chose not to upgrade

msticpy.nbtools.azure_ml_tools.get_aml_user_folder() → Optional[pathlib.Path]: Return the root of the user folder.

msticpy.nbtools.azure_ml_tools.is_in_aml(): Return True if running in Azure Machine Learning.

msticpy.nbtools.data_viewer module

Dataframe viewer.

class msticpy.nbtools.data_viewer.DataTableColumnChooser(data, selected_cols=None)

Bases: object

DataTableColumnChooser class.

Initialize the DataTableColumnChooser class.

property dataframe_columns: Return the selected set of DataFrame columns.

property datatable_columns: Return a list of Bokeh column definitions for the DataFrame.

display(): Display in IPython.

property selected_columns: Return the selected columns.

class msticpy.nbtools.data_viewer.DataTableFilter(data: pandas.core.frame.DataFrame)

Bases: object

Data filtering class.

Initialize the DataTableFilter class.

property bool_filters: Return current set of boolean filters.

property current_col: Return the currently selected column.

display(): Display in IPython.

property filtered_dataframe: pandas.core.frame.DataFrame: Return current filtered DataFrame.

import_filters(filters: Dict[str, msticpy.nbtools.data_viewer.FilterExpr])

Replace the current filters with filters.

Parameters: filters (Dict[str, FilterExpr]) – dict of filter name, FilterExpr FilterExpr is a tuple of: column [str], inv [bool], operator [str], expr [str]

class msticpy.nbtools.data_viewer.DataViewer(data: pandas.core.frame.DataFrame, selected_cols: Optional[List[str]] = None, debug=False)

Bases: object

Data viewer class.

Initialize the DataViewer class.

Parameters

data (pd.DataFrame) – The DataFrame to view
selected_cols (List[str], optional) – Initial subset of columns to show, by default None (all cols)
debug (bool) – Output additional debugging info to std out.

display(): Display the widget.

property filtered_data: pandas.core.frame.DataFrame: Return filtered dataframe.

property filters: Dict[str, msticpy.nbtools.data_viewer.FilterExpr]: Return current filters as a dict.

import_filters(filters: Dict[str, msticpy.nbtools.data_viewer.FilterExpr])

Import filter set replacing current filters.

Parameters: filters (Dict[str, FilterExpr]) – dict of filter name, FilterExpr FilterExpr is a tuple of: column [str], inv [bool], operator [str], expr [str]

show(): Display the data table control.

class msticpy.nbtools.data_viewer.FilterExpr(column, inv, operator, expr)

Bases: tuple

Create new instance of FilterExpr(column, inv, operator, expr)

property column: Alias for field number 0

count(value, /): Return number of occurrences of value.

property expr: Alias for field number 3

index(value, start=0, stop=9223372036854775807, /)

Return first index of value.

Raises ValueError if the value is not present.

property inv: Alias for field number 1

property operator: Alias for field number 2

msticpy.nbtools.foliummap module

Folium map class.

class msticpy.nbtools.foliummap.FoliumMap(title: str = 'layer1', zoom_start: float = 2.5, tiles=None, width: str = '100%', height: str = '100%', location: Optional[list] = None)

Bases: object

Wrapper class for Folium/Leaflet mapping.

Create an instance of the folium map.

Parameters

title (str, optional) – Name of the layer (the default is ‘layer1’)
zoom_start (int, optional) – The zoom level of the map (the default is 7)
tiles ([type], optional) – Custom set of tiles or tile URL (the default is None)
width (str, optional) – Map display width (the default is ‘100%’)
height (str, optional) – Map display height (the default is ‘100%’)
location (list, optional) – Location to center map on

folium_map

The map object.

Type: folium.Map

add_feature_sub_groups(subgroups: Iterable[folium.plugins.feature_group_sub_group.FeatureGroupSubGroup])

Add FeatureGroupSubGroups and to the map.

Parameters: subgroups (Iterable[FeatureGroupSubGroup]) – Iterable of FeatureGroupSubGroups

add_geo_hashes(geohashes: Iterable[str], **kwargs)

Add decoded geohashes to the map.

Parameters: geohashes (Iterable[str]) – Iterable of geolocation hashes

add_geoloc_cluster(geo_locations: Iterable[msticpy.datamodel.entities.geo_location.GeoLocation], **kwargs)

Add a collection of GeoLocation objects to the map.

Parameters: geo_locations (Iterable[GeoLocation]) – Iterable of GeoLocation entities.

add_ip_cluster(ip_entities: Iterable[msticpy.datamodel.entities.ip_address.IpAddress], **kwargs)

Add a collection of IP Entities to the map.

Parameters

ip_entities (Iterable[IpAddress]) – a iterable of IpAddress Entities
kwargs (icon properties to use for displaying this cluster) –

add_locations(locations: Iterable[Tuple[float, float]], **kwargs)

Add a collection of lat/long tuples to the map.

Parameters: locations (Iterable[Tuple[float, float]]) – Iterable of location tuples.

add_locations_to_feature_subgroup(locations: Iterable[Tuple[float, float]], subgroup: folium.plugins.feature_group_sub_group.FeatureGroupSubGroup, **kwargs)

Create markers from locations and add the FeatureGroupSubGroup.

Parameters

locations (Iterable[Tuple[float, float]]) – Collection of Latitude/Longitude coordinates to be added to the FeatureGroupSubGroup
subgroup (FeatureGroupSubGroup) – Subgroup to add locations to, then add to the map

add_locations_to_marker_cluster(locations: Iterable[Tuple[float, float]], cluster: folium.plugins.marker_cluster.MarkerCluster, **kwargs)

Create markers from locations and add to MarkerCluster.

Parameters

locations (Iterable[Tuple[float, float]]) – Collection of Latitude/Longitude coordinates to be added to the MarkerCluster
cluster (MarkerCluster) – Marker cluster to add locations to, then add to the map

add_marker_clusters(clusters: Iterable[folium.plugins.marker_cluster.MarkerCluster])

Add MarkerClusters and to the map.

Parameters: clusters (Iterable[MarkerCluster]) – Iterable of MarkerClusters

center_map(): Calculate and set map center based on current coordinates.

static create_feature_sub_group_of_marker_cluster(cluster: folium.plugins.marker_cluster.MarkerCluster, name: str) → folium.plugins.feature_group_sub_group.FeatureGroupSubGroup

Return a FeatureGroupSubGroup with name for a MarkerCluster.

Parameters

cluster (MarkerCluster) – Folium MarkerCluster to add FeatureGroupSubGroup to
name (str) – Desired name of the MarkerCluster

Returns

A Folium FeatureGroupSubGroup with the provided name as part of the given MarkerCluster

Return type

FeatureGroupSubGroup

static create_marker(location: Tuple[float, float], tooltip: Optional[str] = None, popup: Optional[str] = None, **kwargs) → folium.map.Marker

Create and return a Folium Marker at a given location.

Parameters

location (Tuple[float,float]) – Latitude/Longitude coordinates for the Marker
tooltip (str [Optional]) – Tooltip text for the Marker
popup (str [Optional]) – Popup text for the Marker

Returns

A Folium Marker at the given location coordinates

Return type

Marker

static create_marker_cluster(name: str)

Create and return a MarkerCluster with name.

Parameters: name (str) – Name of the MarkerCluster
Returns: A Folium MarkerCluster with the provided name
Return type: MarkerCluster

create_new_cluster_with_geohashes(geohashes: Iterable[str], name: str, **kwargs)

Create a MarkerCluster and add geohash locations.

Parameters

geohashes (Iterable[str]) – Collection of geohashes to be decoded and added to the MarkerCluster
name (str) – Name of Marker Cluster to create, add locations to, then add to the map

create_new_cluster_with_locations(locations: Iterable[Tuple[float, float]], name: str, **kwargs)

Create a MarkerCluster with locations.

Parameters

locations (Iterable[Tuple[float, float]]) – Collection of Latitude/Longitude coordinates to be added to the MarkerCluster
name (str) – Name of Marker Cluster to create, add locations to, then add to the map

create_new_subgroup_with_geohashes(geohashes: Iterable[str], subgroup_name: str, cluster_name: str, **kwargs)

Create a FeatureSubGroup with collection of geohash locations.

Parameters

geohashes (Iterable[str]) – Collection of geohashes to be decoded and added to the FeatureGroupSubGroup
subgroup_name (str) – Name of SubGroup to create, add locations to, then add to the map
cluster_name (str) – Name of the Marker Cluster to create and add the SubGroup to

create_new_subgroup_with_locations(locations: Iterable[Tuple[float, float]], subgroup_name: str, cluster_name: str, **kwargs)

Create subgroup of markers from locations.

Parameters

locations (Iterable[Tuple[float, float]]) – Collection of Latitude/Longitude coordinates to be added to the FeatureGroupSubGroup
subgroup_name (str) – Name of FeatureGroupSubGroup to create, add locations to, then add to the map
cluster_name (str) – Name of the cluster

Notes

This function creates a marker cluster and FeatureGroupSubGroup, then add the locations to the subgroup, then add the subgroup to the map.

enable_layer_control()

Enable Layer Control on the map.

Parameters: None –

save_map(path: str)

Save the map to path.

Parameters: path (str) – File path to save the current map

msticpy.nbtools.foliummap.decode_geo_hash(geohash: str) → Tuple[float, float, float, float]

Decode a geohash.

Parameters: geohash (str) – A string representation of a location
Returns: Tuple representation of a geohash, format of: (Latitude, Longitude, Latitude Error interval, Longitude Error Interval)
Return type: Tuple

msticpy.nbtools.foliummap.decode_geohash_collection(geohashes: Iterable[str])

Return collection of geohashes decoded into location coordinates.

Parameters: geohashes (Iterable[str]) – Collection of geohashes to be decoded
Returns: Collection of location coordinates in Latitude/Longitude
Return type: Iterable[Tuple[float, float]]

msticpy.nbtools.foliummap.get_center_geo_locs(loc_entities: Iterable[msticpy.datamodel.entities.geo_location.GeoLocation], mode: str = 'median') → Tuple[float, float]

Return the geographical center of the geo locations.

Parameters

loc_entities (Iterable[GeoLocation]) – GeoLocation entities with location information
mode (str, optional) – The averaging method to use, by default “median”. “median” and “mean” are the supported values.

Returns

Tuple of latitude, longitude

Return type

Tuple[Union[int, float], Union[int, float]]

msticpy.nbtools.foliummap.get_center_ip_entities(ip_entities: Iterable[msticpy.datamodel.entities.ip_address.IpAddress], mode: str = 'median') → Tuple[float, float]

Return the geographical center of the IP address locations.

Parameters

ip_entities (Iterable[IpAddress]) – IpAddress entities with location information
mode (str, optional) – The averaging method to us, by default “median”. “median” and “mean” are the supported values.

Returns

Tuple of latitude, longitude

Return type

Tuple[Union[int, float], Union[int, float]]

msticpy.nbtools.foliummap.get_map_center(entities: Iterable[msticpy.datamodel.entities.entity.Entity], mode: str = 'modal')

Calculate median point between Entity IP locations.

Parameters

entities (Iterable[Entity]) – An iterable of entities containing IpAddress geolocation information. The entities can be IpAddress entities or other entities that have IpAddress properties. The entities must all be of the same type.
mode (str, optional) – The averaging method to use, by default “median”. “median” and “mean” are the supported values.

Returns

The Latitude and Longitude calculated

Return type

Tuple

Notes

The function uses the first entity in the entities to determine how to process the collection. E.g. if the first entity has properties src_ip and dest_ip of type IpAddress, these are the only properties that will be processed for the remainder of the entities.

msticpy.nbtools.morph_charts module

Morph Charts class.

class msticpy.nbtools.morph_charts.MorphCharts

Bases: object

Create Morph Charts package data and render Morph Charts site.

Create object and populate charts container.

display(data: pandas.core.frame.DataFrame, chart_name: str) → IPython.lib.display.IFrame

Prepare package data and display MorphChart in an IFrame.

Parameters

data (pd.DataFrame:) – A DataFrame of data for the morphchart to plot.
chart_name (str:) – The name of the Morph Chart to plot.

get_chart_details(chart_name)

Get description for a chart.

Parameters: chart_name (str:) – The name of the chart you get description for.

list_charts(): Get a list of avaliable charts.

search_charts(keyword)

Search for charts that match a keyword.

Parameters: keyword (str:) – The keyword to search charts for.

msticpy.nbtools.nbdisplay module

Module for common display functions.

msticpy.nbtools.nbdisplay.display_alert(alert: Union[Mapping[str, Any], msticpy.nbtools.security_alert.SecurityAlert], show_entities: bool = False)

Display a Security Alert.

Parameters

alert (Union[Mapping[str, Any], SecurityAlert]) – The alert to display as Mapping (e.g. pd.Series) or SecurityAlert
show_entities (bool, optional) – Whether to display entities (the default is False)

msticpy.nbtools.nbdisplay.display_logon_data(logon_event: pandas.core.frame.DataFrame, alert: Optional[msticpy.nbtools.security_alert.SecurityAlert] = None, os_family: Optional[str] = None)

Display logon data for one or more events as HTML table.

Parameters

logon_event (pd.DataFrame) – Dataframe containing one or more logon events
alert (SecurityAlert, optional) – obtain os_family from the security alert (the default is None)
os_family (str, optional) – explicitly specify os_family (Linux or Windows) (the default is None)

Notes

Currently only Windows Logon events.

msticpy.nbtools.nbdisplay.display_process_tree(process_tree: pandas.core.frame.DataFrame)

Display process tree data frame. (Deprecated).

Parameters

process_tree (pd.DataFrame) – Process tree DataFrame
to (The display module expects the columns NodeRole and Level) –
of (be populated. NoteRole is one) –
'source' (or 'sibling'. Level indicates the 'hop' distance from the) –
node. –

msticpy.nbtools.nbdisplay.draw_alert_entity_graph(nx_graph: networkx.classes.graph.Graph, font_size: int = 12, height: int = 8, width: int = 8, margin: float = 0.3, scale: int = 1)

Draw networkX graph with matplotlib.

Parameters

nx_graph (nx.Graph) – The NetworkX graph to draw
font_size (int, optional) – base font size (the default is 12)
height (int, optional) – Image height (the default is 8)
width (int, optional) – Image width (the default is 8)
margin (float, optional) – Image margin (the default is 0.3)
scale (int, optional) – Position scale (the default is 1)
deprecated: (.) – 0.3.2: Matplotlib version ‘draw_alert_entity_graph’ no longer supported - use ‘plot_entity_graph’

msticpy.nbtools.nbdisplay.exec_remaining_cells(): Execute all cells below currently selected cell.

msticpy.nbtools.nbdisplay.format_alert(alert: Union[Mapping[str, Any], msticpy.nbtools.security_alert.SecurityAlert], show_entities: bool = False) → Union[IPython.core.display.HTML, Tuple[IPython.core.display.HTML, pandas.core.frame.DataFrame]]

Get IPython displayable Security Alert.

Parameters

alert (Union[Mapping[str, Any], SecurityAlert]) – The alert to display as Mapping (e.g. pd.Series) or SecurityAlert
show_entities (bool, optional) – Whether to display entities (the default is False)

Returns

Single or tuple of displayable IPython objects

Return type

Union[IPython.display.HTML, Tuple[IPython.display.HTML, pd.DataFrame]]

Raises

ValueError – If the alert object is in an unknown format

msticpy.nbtools.nbdisplay.format_logon(logon_event: Union[pandas.core.frame.DataFrame, pandas.core.series.Series], alert: Optional[msticpy.nbtools.security_alert.SecurityAlert] = None, os_family: Optional[str] = None) → IPython.core.display.HTML

Return logon data for one or more events as HTML table.

Parameters

logon_event (Union[pd.DataFrame, pd.Series]) – Dataframe containing one or more logon events or Series containing a single logon event.
alert (SecurityAlert, optional) – obtain os_family from the security alert (the default is None)
os_family (str, optional) – explicitly specify os_family (Linux or Windows) (the default is None)

Returns

HTML display object

Return type

IPython.display.HTML

msticpy.nbtools.nbdisplay.plot_entity_graph(entity_graph: networkx.classes.graph.Graph, node_size: int = 25, font_size: Union[int, str] = 10, height: int = 800, width: int = 800, scale: int = 2, hide: bool = False) → bokeh.plotting.figure.figure

Plot entity graph with Bokeh.

Parameters

entity_graph (nx.Graph) – The entity graph as a networkX graph
node_size (int, optional) – Size of the nodes in pixels, by default 25
font_size (int, optional) – Font size for node labels, by default 10 Can be an integer (point size) or a string (e.g. “10pt”)
width (int, optional) – Width in pixels, by default 800
height (int, optional) – Image height (the default is 800)
scale (int, optional) – Position scale (the default is 2)
hide (bool, optional) – Don’t show the plot, by default False. If True, just return the figure.

Returns

The network plot.

Return type

bokeh.plotting.figure

msticpy.nbtools.nbinit module

Initialization for Jupyter Notebooks.

msticpy.nbtools.nbinit.init_notebook(namespace: Dict[str, Any], def_imports: str = 'all', additional_packages: Optional[List[str]] = None, extra_imports: Optional[List[str]] = None, **kwargs) → bool

Initialize the notebook environment.

Parameters

namespace (Dict[str, Any]) – Namespace (usually globals()) into which imports are to be populated.
def_imports (str, optional) – Import default packages. By default “all”. Possible values are: - “all” - import all packages - “nb” - import common notebook packages - “msticpy” - import msticpy packages - “none” (or any other value) don’t load any default packages.
additional_packages (List[str], optional) – Additional packages to be pip installed, by default None. Packages are specified by name only or version specification (e.g. “pandas>=0.25”)
user_install (bool, optional) – Install packages in the “user” rather than system site-packages. Use this option if you cannot or do not want to update the system packages. You should usually avoid using this option with standard Conda environments.
extra_imports (List[str], optional) – Additional import definitions, by default None. Imports are specified as up to 3 comma-delimited values in a string: “{source_pkg}, [{import_tgt}], [{alias}]” source_pkg is mandatory - equivalent to a simple “import xyz” statement. {import_tgt} specifies an object to import from the package equivalent to “from source_pkg import import_tgt” alias allows renaming of the imported object - equivalent to the “as alias” part of the import statement. If you want to provide just source_pkg and alias include an additional placeholder comma: e.g. “pandas, , pd”
friendly_exceptions (Optional[bool]) – Setting this to True causes msticpy to hook the notebook exception hander. Any exceptions derived from MsticpyUserException are displayed but do not produce a stack trace, etc. Defaults to system/user settings if no value is supplied.
verbose (Union[int, bool], optional) – Controls amount if status output, by default 1 0 = No output 1 or False = Brief output (default) 2 or True = Detailed output
no_config_check (bool, optional) – Skip the check for valid configuration. Default is False.
verbosity (int, optional) –

Returns

True if successful

Return type

bool

Raises

MsticpyException – If extra_imports data format is incorrect. If package with required version check has no version information.

msticpy.nbtools.nbinit.list_default_imports(): List the default imports for init_notebook.

msticpy.nbtools.nbwidgets module

Widgets sub-package.

msticpy.nbtools.observationlist module

Observation summary collector.

class msticpy.nbtools.observationlist.Observation(caption: str, data: Any, description: Optional[str] = None, data_type: Optional[str] = None, link: Optional[str] = None, score: int = 0, tags: List[str] = NOTHING, additional_properties: Dict[str, Any] = NOTHING)

Bases: object

Observation definition.

Notes

captionstr: The title and index of the observation. Must be unique in the observation set.
descriptionOptional[str]: Text description of the observation. (default is None)
dataAny: The data to be stored for the observation (e.g. a pandas DataFrame). The object should implement a useable __repr__ to display correctly.
data_typeOptional[str]: The data type of the data property
linkOptional[str]: Link (usually a document-local link) to the originating section of the notebook. (default is None)
scoreint: The risk score associated with the observation. (default is 0)
tagsList[str]: Optional list of tags.
additional_properties Dict[str, Any]: Additional properties not covered by core properties.

Method generated by attrs for class Observation.

additional_properties: Dict[str, Any]

classmethod all_fields() → List[str]

Return all fields of Observation class.

Returns: List of all field names.
Return type: List[str]

caption: str

data: Any

data_type: Optional[str]

description: Optional[str]

display(): Display the observation.

link: Optional[str]

classmethod required_fields() → List[str]

Return required fields for Observation instance.

Returns: List of field names.
Return type: List[str]

score: int

tags: List[str]

class msticpy.nbtools.observationlist.Observations(observationlist: Optional[msticpy.nbtools.observationlist.Observations] = None)

Bases: object

Class to collect and display investigation observations.

Create an observation list.

Parameters: observationlist (Observations, optional) – Initialize from an existing Observations list (the default is None)

add_observation(observation: Optional[msticpy.nbtools.observationlist.Observation] = None, **kwargs)

Add an observation.

Add an observation as an Observation instance or as a set of keyword parameters (see Observation class for acceptable values). Any keyword parameters that are not properties of Observation will be stored in the Observation.additional_properties dictionary

Parameters

observation (Observation) – An observation instance.
kwargs (str, Any) – List of key value pairs of the property names and values of the Observation to be stored.

display_observations(): Display the current observations using IPython.display.

property observations: Mapping[str, msticpy.nbtools.observationlist.Observation]

Return the current list of Observations.

Returns: The current ordered dictionary of Observations
Return type: Mapping[str, Observation]

msticpy.nbtools.process_tree module

Process Tree Visualization.

class msticpy.nbtools.process_tree.ProcessTreeAccessor(pandas_obj)

Bases: object

Pandas api extension for Process Tree.

Instantiate pandas extension class.

build(schema: Optional[msticpy.sectools.proc_tree_schema.ProcSchema] = None, **kwargs) → pandas.core.frame.DataFrame

Build process trees from the process events.

Parameters

procs (pd.DataFrame) – Process events (Windows 4688 or Linux Auditd)
schema (ProcSchema, optional) – The column schema to use, by default None If None, then the schema is inferred
show_summary (bool) – Shows summary of the built tree, default is False. : bool
debug (bool) – If True produces extra debugging output, by default False

Returns

Process tree dataframe.

Return type

pd.DataFrame

Notes

It is not necessary to call this before plot. The process tree is built automatically. This is only needed if you want to return the processed tree data as a DataFrame

plot(**kwargs) → Tuple[bokeh.plotting.figure.figure, bokeh.models.layouts.LayoutDOM]

Build and plot a process tree.

Parameters

schema (ProcSchema, optional) – The data schema to use for the data set, by default None (if None the schema is inferred)
output_var (str, optional) – Output variable for selected items in the tree, by default None
legend_col (str, optional) – The column used to color the tree items, by default None
show_table (bool) – Set to True to show a data table, by default False.
height (int, optional) – The height of the plot figure (the default is 700)
width (int, optional) – The width of the plot figure (the default is 900)
title (str, optional) – Title to display (the default is None)
hide_legend (bool, optional) – Hide the legend box, even if legend_col is specified.
pid_fmt (str, optional) – Display Process ID as ‘dec’ (decimal) or ‘hex’ (hexadecimal), default is ‘hex’.

Returns

figure - The main bokeh.plotting.figure Layout - Bokeh layout structure.

Return type

Tuple[figure, LayoutDOM]

class msticpy.nbtools.process_tree.TreeResult(proc_tree, schema, levels, n_rows)

Bases: tuple

Create new instance of TreeResult(proc_tree, schema, levels, n_rows)

count(value, /): Return number of occurrences of value.

index(value, start=0, stop=9223372036854775807, /)

Return first index of value.

Raises ValueError if the value is not present.

property levels: Alias for field number 2

property n_rows: Alias for field number 3

property proc_tree: Alias for field number 0

property schema: Alias for field number 1

msticpy.nbtools.process_tree.build_and_show_process_tree(data: pandas.core.frame.DataFrame, schema: Optional[msticpy.sectools.proc_tree_schema.ProcSchema] = None, output_var: Optional[str] = None, legend_col: Optional[str] = None, **kwargs) → Tuple[bokeh.plotting.figure.figure, bokeh.models.layouts.LayoutDOM]

Build process tree from data and plot a tree.

Parameters

data (pd.DataFrame) – Window process creation or Linux Auditd events
schema (ProcSchema) – The data schema to use for the data set, by default None (if None the schema is inferred)
output_var (str, optional) – Output variable for selected items in the tree, by default None
legend_col (str, optional) – The column used to color the tree items, by default None
kwargs (Dict[str, Any]) – Additional arguments passed to plot_process_tree
height (int, optional) – The height of the plot figure (the default is 700)
width (int, optional) – The width of the plot figure (the default is 900)
title (str, optional) – Title to display (the default is None)
hide_legend (bool, optional) – Hide the legend box, even if legend_col is specified.
pid_fmt (str, optional) – Display Process ID as ‘dec’ (decimal) or ‘hex’ (hexadecimal), default is ‘hex’.

Returns

figure - The main bokeh.plotting.figure Layout - Bokeh layout structure.

Return type

Tuple[figure, LayoutDOM]

Notes

For full parameter set for process tree display see the help for plot_process_tree.

msticpy.nbtools.ti_browser module

Threat Intel Results Browser.

msticpy.nbtools.ti_browser.browse_results(data: pandas.core.frame.DataFrame, severities: Optional[Union[str, List[str]]] = None, **kwargs) → msticpy.nbtools.nbwidgets.select_item.SelectItem

Return TI Results list browser.

Parameters

data (pd.DataFrame) – TI Results data from TIProviders
severities (Union[List[str], str, None], optional) – A list of the severity classes to show or the string ‘all’. By default these are [‘warning’, ‘high’].
kwargs – passed to SelectItem constuctor.

Returns

SelectItem browser for TI Data.

Return type

SelectItem

msticpy.nbtools.ti_browser.get_ti_select_options(ti_data: pandas.core.frame.DataFrame, severities: Optional[Union[str, List[str]]] = None): Get SelectItem options for TI data.

msticpy.nbtools.ti_browser.raw_results(raw_result: str) → str: Create pre-formatted details for raw results.

msticpy.nbtools.ti_browser.ti_details_display(ti_data): Return TI Details display function.

msticpy.nbtools.timeseries module

Module for common display functions.

msticpy.nbtools.timeseries.display_timeseries_anomalies(data: pandas.core.frame.DataFrame, y: str = 'Total', time_column: str = 'TimeGenerated', anomalies_column: str = 'anomalies', source_columns: Optional[list] = None, period: int = 30, **kwargs) → bokeh.plotting.figure.figure

Display time series anomalies visualization.

Parameters

data (pd.DataFrame) – DataFrame as a time series data set retrieved from KQL time series functions. Dataframe must have columns specified in y, time_column and anomalies_column parameters
y (str, optional) – Name of column holding numeric values to plot against time series to determine anomalies (the default is ‘Total’)
time_column (str, optional) – Name of the timestamp column (the default is ‘TimeGenerated’)
anomalies_column (str, optional) – Name of the column holding binary status(1/0) for anomaly/benign (the default is ‘anomalies’)
source_columns (list, optional) – List of default source columns to use in tooltips (the default is None)
period (int, optional) – Period of the dataset for hourly-no of days, for daily-no of weeks. This is used to correctly calculate the plot height. (the default is 30)
ref_time (datetime, optional) – Input reference line to display (the default is None)
title (str, optional) – Title to display (the default is None)
legend (str, optional) – Where to position the legend None, left, right or inline (default is None)
yaxis (bool, optional) – Whether to show the yaxis and labels
range_tool (bool, optional) – Show the the range slider tool (default is True)
height (int, optional) – The height of the plot figure (the default is auto-calculated height)
width (int, optional) – The width of the plot figure (the default is 900)
xgrid (bool, optional) – Whether to show the xaxis grid (default is True)
ygrid (bool, optional) – Whether to show the yaxis grid (default is False)
color (list, optional) – List of colors to use in 3 plots as specified in order 3 plots- line(observed), circle(baseline), circle_x/user specified(anomalies). (the default is [“navy”, “green”, “firebrick”])

Returns

The bokeh plot figure.

Return type

figure

msticpy.nbtools.timeseries.display_timeseries_anomolies(data: pandas.core.frame.DataFrame, y: str = 'Total', time_column: str = 'TimeGenerated', anomalies_column: str = 'anomalies', source_columns: Optional[list] = None, period: int = 30, **kwargs) → bokeh.plotting.figure.figure

Display time series anomalies visualization.

Parameters

data (pd.DataFrame) – DataFrame as a time series data set retrieved from KQL time series functions. Dataframe must have columns specified in y, time_column and anomalies_column parameters
y (str, optional) – Name of column holding numeric values to plot against time series to determine anomalies (the default is ‘Total’)
time_column (str, optional) – Name of the timestamp column (the default is ‘TimeGenerated’)
anomalies_column (str, optional) – Name of the column holding binary status(1/0) for anomaly/benign (the default is ‘anomalies’)
source_columns (list, optional) – List of default source columns to use in tooltips (the default is None)
period (int, optional) – Period of the dataset for hourly-no of days, for daily-no of weeks. This is used to correctly calculate the plot height. (the default is 30)
ref_time (datetime, optional) – Input reference line to display (the default is None)
title (str, optional) – Title to display (the default is None)
legend (str, optional) – Where to position the legend None, left, right or inline (default is None)
yaxis (bool, optional) – Whether to show the yaxis and labels
range_tool (bool, optional) – Show the the range slider tool (default is True)
height (int, optional) – The height of the plot figure (the default is auto-calculated height)
width (int, optional) – The width of the plot figure (the default is 900)
xgrid (bool, optional) – Whether to show the xaxis grid (default is True)
ygrid (bool, optional) – Whether to show the yaxis grid (default is False)
color (list, optional) – List of colors to use in 3 plots as specified in order 3 plots- line(observed), circle(baseline), circle_x/user specified(anomalies). (the default is [“navy”, “green”, “firebrick”])

Returns

The bokeh plot figure.

Return type

figure

msticpy.nbtools.timeline module

Module for common display functions.

msticpy.nbtools.timeline.check_df_columns(data: pandas.core.frame.DataFrame, req_columns: List[str], help_uri: str, plot_type: str)

Check that specified columns are in the DataFrame.

Parameters

data (pd.DataFrame) – [description]
req_columns (List[str]) – [description]
help_uri (str) – [description]
plot_type (str) – [description]

Raises

MsticpyParameterError – If one or more columns not found in data

msticpy.nbtools.timeline.display_timeline(data: Union[pandas.core.frame.DataFrame, dict], time_column: str = 'TimeGenerated', source_columns: Optional[list] = None, **kwargs) → bokeh.models.layouts.LayoutDOM

Display a timeline of events.

Parameters

data (Union[dict, pd.DataFrame]) –

Either dict of data sets to plot on the timeline with the following structure:

Key (str) - Name of data set to be displayed in legend
Value (Dict[str, Any]) - containing:
    data (pd.DataFrame) - Data to plot
    time_column (str, optional) - Name of the timestamp column
    source_columns (list[str], optional) - source columns to use
        in tooltips
    color (str, optional) - color of datapoints for this data
If any of the last values are omitted, they default to the values
supplied as parameters to the function (see below)

Or DataFrame as a single data set or grouped into individual plot series using the group_by parameter

time_column (str, optional) – Name of the timestamp column (the default is ‘TimeGenerated’)
source_columns (list, optional) – List of default source columns to use in tooltips (the default is None)
title (str, optional) – Title to display (the default is None)
alert (SecurityAlert, optional) – Add a reference line/label using the alert time (the default is None)
ref_event (Any, optional) – Add a reference line/label using the alert time (the default is None)
ref_time (datetime, optional) – Add a reference line/label using ref_time (the default is None)
group_by (str) – (where data is a DataFrame) The column to group timelines on
legend (str, optional) – “left”, “right”, “inline” or “none” (the default is to show a legend when plotting multiple series and not to show one when plotting a single series)
yaxis (bool, optional) – Whether to show the yaxis and labels (default is False)
ygrid (bool, optional) – Whether to show the yaxis grid (default is False)
xgrid (bool, optional) – Whether to show the xaxis grid (default is True)
range_tool (bool, optional) – Show the the range slider tool (default is True)
height (int, optional) – The height of the plot figure (the default is auto-calculated height)
width (int, optional) – The width of the plot figure (the default is 900)
color (str) – Default series color (default is “navy”)
overlay_data (pd.DataFrame:) – A second dataframe to plot as a different series.
overlay_color (str) – Overlay series color (default is “green”)
hide (bool, optional) – If True, create but do not display the plot. By default, False.
ref_events (pd.DataFrame, optional) – Add references line/label using the event times in the dataframe. (the default is None)
ref_time_col (str, optional) – Add references line/label using the this column in ref_events for the time value (x-axis). (this defaults the value of the time_column parameter or ‘TimeGenerated’ time_column is None)
ref_col (str, optional) – The column name to use for the label from ref_events (the default is None)
ref_times (List[Tuple[datetime, str]], optional) – Add one or more reference line/label using (the default is None)

Returns

The bokeh plot figure.

Return type

LayoutDOM

msticpy.nbtools.timeline.display_timeline_values(data: pandas.core.frame.DataFrame, value_col: Optional[str] = None, time_column: str = 'TimeGenerated', source_columns: Optional[list] = None, **kwargs) → bokeh.models.layouts.LayoutDOM

Display a timeline of events.

Parameters

data (pd.DataFrame) – DataFrame as a single data set or grouped into individual plot series using the group_by parameter
time_column (str, optional) – Name of the timestamp column (the default is ‘TimeGenerated’)
value_col (str) – The column name holding the value to plot vertically
source_columns (list, optional) – List of default source columns to use in tooltips (the default is None)
x (str, optional) – alias of time_column
y (str) – an alias for value_col
title (str, optional) – Title to display (the default is None)
ref_event (Any, optional) – Add a reference line/label using the alert time (the default is None)
ref_time (datetime, optional) – Add a reference line/label using ref_time (the default is None)
ref_label (str, optional) – A label for the ref_event or ref_time reference item
group_by (str) – (where data is a DataFrame) The column to group timelines on
legend (str, optional) – “left”, “right”, “inline” or “none” (the default is to show a legend when plotting multiple series and not to show one when plotting a single series)
yaxis (bool, optional) – Whether to show the yaxis and labels
range_tool (bool, optional) – Show the the range slider tool (default is True)
height (int, optional) – The height of the plot figure (the default is auto-calculated height)
width (int, optional) – The width of the plot figure (the default is 900)
color (str) – Default series color (default is “navy”). This is overridden by automatic color assignments if plotting a grouped chart
kind (Union[str, List[str]]) – one or more glyph types to plot., optional Supported types are “circle”, “line” and “vbar” (default is “vbar”)
hide (bool, optional) – If True, create but do not display the plot. By default, False.
ref_events (pd.DataFrame, optional) – Add references line/label using the event times in the dataframe. (the default is None)
ref_time_col (str, optional) – Add references line/label using the this column in ref_events for the time value (x-axis). (this defaults the value of the time_column parameter or ‘TimeGenerated’ time_column is None)
ref_col (str, optional) – The column name to use for the label from ref_events (the default is None)
ref_times (List[Tuple[datetime, str]], optional) – Add one or more reference line/label using (the default is None)

Returns

The bokeh plot figure.

Return type

LayoutDOM

msticpy.nbtools.timeline_duration module

Timeline duration control.

class msticpy.nbtools.timeline_duration.PlotParams(height: Optional[int] = None, width: int = 900, title: Optional[str] = None, yaxis: bool = True, range_tool: bool = True, xgrid: bool = True, ygrid: bool = False, hide: bool = False, color: str = 'navy', ylabel_cols: Iterable[str] = NOTHING, ref_events: Optional[pandas.core.frame.DataFrame] = None, ref_col: Optional[str] = None, ref_times: Optional[List[Tuple[datetime.datetime, str]]] = None, source_columns: List = [])

Bases: object

Plot params for time_duration.

Method generated by attrs for class PlotParams.

color: str

classmethod field_list() → List[str]: Return field names as a list.

height: Optional[int]

hide: bool

range_tool: bool

ref_col: Optional[str]

ref_events: Optional[pandas.core.frame.DataFrame]

ref_times: Optional[List[Tuple[datetime.datetime, str]]]

source_columns: List

title: Optional[str]

width: int

xgrid: bool

yaxis: bool

ygrid: bool

ylabel_cols: Iterable[str]

msticpy.nbtools.timeline_duration.display_timeline_duration(data: pandas.core.frame.DataFrame, group_by: Union[Iterable[str], str], time_column: str = 'TimeGenerated', end_time_column: Optional[str] = None, **kwargs) → bokeh.models.layouts.LayoutDOM

Display a duration timeline of events grouped by one or more columns.

Parameters

data (pd.DataFrame) – Data to plot
group_by (Union[Iterable[str], str]) – The column name or iterable of column names to group the data by.
time_column (str) – Primary time column - will be used to calculate the start time of the duration for each group. If end_time_column is not specified it will also be used to calculate the end time.
end_time_column (Optional[str]) – If supplied, it will be used to calculate the end time of the duration for each group.
title (str, optional) – Title to display (the default is None)
ylabel_cols (Optional[Iterable[str]], optional) – The subset of the group columns to use for the y-axis labels.
yaxis (bool, optional) – Whether to show the yaxis and labels
range_tool (bool, optional) – Show the the range slider tool (default is True)
source_columns (list, optional) – List of default source columns to use in tooltips (the default is None)
height (int, optional) – The height of the plot figure (the default is auto-calculated height)
width (int, optional) – The width of the plot figure (the default is 900)
color (str) – Default series color (default is “navy”)
ref_events (pd.DataFrame, optional) – Add references line/label using the event times in the dataframe. (the default is None)
ref_col (str, optional) – The column name to use for the label from ref_events (the default is None)
ref_times (List[Tuple[datetime, str]], optional) – Add one or more reference line/label using (the default is None)

Returns

The bokeh plot figure.

Return type

LayoutDOM

msticpy.nbtools.timeline_pd_accessor module

Pandas accessor class for timeline functions.

class msticpy.nbtools.timeline_pd_accessor.TimeLineAccessor(pandas_obj)

Bases: object

Pandas api extension for Timeline.

Instantiate pandas extension class.

plot(**kwargs) → bokeh.models.layouts.LayoutDOM

Display a timeline of events.

Parameters

time_column (str, optional) – Name of the timestamp column (the default is ‘TimeGenerated’)
source_columns (list, optional) – List of default source columns to use in tooltips (the default is None)
title (str, optional) – Title to display (the default is None)
alert (SecurityAlert, optional) – Add a reference line/label using the alert time (the default is None)
ref_event (Any, optional) – Add a reference line/label using the alert time (the default is None)
ref_time (datetime, optional) – Add a reference line/label using ref_time (the default is None)
group_by (str) – The column to group timelines on.
legend (str, optional) – “left”, “right”, “inline” or “none” (the default is to show a legend when plotting multiple series and not to show one when plotting a single series)
yaxis (bool, optional) – Whether to show the yaxis and labels (default is False)
ygrid (bool, optional) – Whether to show the yaxis grid (default is False)
xgrid (bool, optional) – Whether to show the xaxis grid (default is True)
range_tool (bool, optional) – Show the the range slider tool (default is True)
height (int, optional) – The height of the plot figure (the default is auto-calculated height)
width (int, optional) – The width of the plot figure (the default is 900)
color (str) – Default series color (default is “navy”)
overlay_data (pd.DataFrame:) – A second dataframe to plot as a different series.
overlay_color (str) – Overlay series color (default is “green”)
ref_events (pd.DataFrame, optional) – Add references line/label using the event times in the dataframe. (the default is None)
ref_time_col (str, optional) – Add references line/label using the this column in ref_events for the time value (x-axis). (this defaults the value of the time_column parameter or ‘TimeGenerated’ time_column is None)
ref_col (str, optional) – The column name to use for the label from ref_events (the default is None)
ref_times (List[Tuple[datetime, str]], optional) – Add one or more reference line/label using (the default is None)

Returns

The bokeh plot figure.

Return type

LayoutDOM

plot_duration(group_by: Union[Iterable[str], str], time_column: str = 'TimeGenerated', end_time_column: Optional[str] = None, **kwargs) → bokeh.models.layouts.LayoutDOM

Display a duration timeline of events grouped by one or more columns.

Parameters

group_by (Union[Iterable[str], str]) – The column name or iterable of column names to group the data by.
time_column (str) – Primary time column - will be used to calculate the start time of the duration for each group. If end_time_column is not specified it will also be used to calculate the end time.
end_time_column (Optional[str]) – If supplied, it will be used to calculate the end time of the duration for each group.
title (str, optional) – Title to display (the default is None)
ylabel_cols (Optional[Iterable[str]], optional) – The subset of the group columns to use for the y-axis labels.
yaxis (bool, optional) – Whether to show the yaxis and labels
range_tool (bool, optional) – Show the the range slider tool (default is True)
source_columns (list, optional) – List of default source columns to use in tooltips (the default is None)
height (int, optional) – The height of the plot figure (the default is auto-calculated height)
width (int, optional) – The width of the plot figure (the default is 900)
color (str) – Default series color (default is “navy”)
ref_events (pd.DataFrame, optional) – Add references line/label using the event times in the dataframe. (the default is None)
ref_col (str, optional) – The column name to use for the label from ref_events (the default is None)
ref_times (List[Tuple[datetime, str]], optional) – Add one or more reference line/label using (the default is None)

Returns

The bokeh plot figure.

Return type

LayoutDOM

plot_values(value_col: Optional[str] = None, **kwargs) → bokeh.models.layouts.LayoutDOM

Display a timeline of events.

Parameters

time_column (str, optional) – Name of the timestamp column (the default is ‘TimeGenerated’)
value_col (str) – The column name holding the value to plot vertically
source_columns (list, optional) – List of default source columns to use in tooltips (the default is None)
x (str, optional) – alias of time_column
y (str, optional) – alias of value_col
title (str, optional) – Title to display (the default is None)
ref_event (Any, optional) – Add a reference line/label using the alert time (the default is None)
ref_time (datetime, optional) – Add a reference line/label using ref_time (the default is None)
ref_label (str, optional) – A label for the ref_event or ref_time reference item
group_by (str) – (where data is a DataFrame) The column to group timelines on
legend (str, optional) – “left”, “right”, “inline” or “none” (the default is to show a legend when plotting multiple series and not to show one when plotting a single series)
yaxis (bool, optional) – Whether to show the yaxis and labels
range_tool (bool, optional) – Show the the range slider tool (default is True)
height (int, optional) – The height of the plot figure (the default is auto-calculated height)
width (int, optional) – The width of the plot figure (the default is 900)
color (str) – Default series color (default is “navy”). This is overridden by automatic color assignments if plotting a grouped chart
kind (Union[str, List[str]]) – one or more glyph types to plot., optional Supported types are “circle”, “line” and “vbar” (default is “vbar”)
ref_events (pd.DataFrame, optional) – Add references line/label using the event times in the dataframe. (the default is None)
ref_time_col (str, optional) – Add references line/label using the this column in ref_events for the time value (x-axis). (this defaults the value of the time_column parameter or ‘TimeGenerated’ time_column is None)
ref_col (str, optional) – The column name to use for the label from ref_events (the default is None)
ref_times (List[Tuple[datetime, str]], optional) – Add one or more reference line/label using (the default is None)

Returns

The bokeh plot figure.

Return type

LayoutDOM

msticpy.nbtools.user_config module

User configuration functions.

Loads providers based on user_defaults section in msticpyconfig.yaml

UserDefaults:
# List of query providers to load
    QueryProviders:
        AzureSentinel:
        Default:          # name of the provider listed in AzureSentinel.Workspaces
            alias: azsent   # optional - create "qry_azsent" object in globals
        CyberSoc:
            alias: soc
            connect: False  # optional - do not connect on load
        Splunk:             # add non-sentinel providers like this
        connect: False
        LocalData: local

# List of other providers/components to load
LoadComponents:
    TILookup:           # No parameters
    GeoIpLookup:
    provider: GeoLiteLookup   # geoip provider to use
    Notebooklets:       # Load and intialize Notebooklets
    query_provider:   # Pass it this query provider at startup
        AzureSentinel:
        workspace: CyberSoc
    Pivot:              # No parameters
    AzureData:          # auth_methods passed as startup param
    auth_methods: ['cli','interactive']
    AzureSentinelAPI:
    auth_methods: ['env','interactive']
    connect: False   # Load but do not connect

Note: For components that require authentication the default is to connect after loading. You can skip the connect step by add connect: False to the entry.

msticpy.nbtools.user_config.load_user_defaults() → Dict[str, object]

Load providers from user defaults in msticpyconfig.yaml.

Returns: Dict of object name and provider instances.
Return type: Dict[str, object]