msticpy.nbtools package

msticpy.nbtools.azure_ml_tools module

Checker functions for Azure ML notebooks.

msticpy.nbtools.azure_ml_tools.check_mp_ver(min_msticpy_ver: Union[str, Tuple], extras: Optional[List[str]])

Check and optionally update the current version of msticpy.

Parameters
  • min_msticpy_ver (Tuple[int, int]) – Minimum MSTICPy version

  • extras (Optional[List[str]], optional) – A list of extras required for MSTICPy

Raises

ImportError – If MSTICPy version is insufficient and we need to upgrade

msticpy.nbtools.azure_ml_tools.check_python_ver(min_py_ver: Union[str, Tuple] = '3.6')

Check the current version of the Python kernel.

Parameters

min_py_ver (Tuple[int, int]) – Minimum Python version

Raises

RuntimeError – If the Python version does not support the notebook.

msticpy.nbtools.azure_ml_tools.check_versions(min_py_ver: Union[str, Tuple] = '3.6', min_mp_ver: Union[str, Tuple] = '1.5.2', extras: Optional[List[str]] = None, mp_release: Optional[str] = None, **kwargs)

Check the current versions of the Python kernel and MSTICPy.

Parameters
  • min_py_ver (Union[Tuple[int, int], str]) – Minimum Python version

  • min_mp_ver (Union[Tuple[int, int], str]) – Minimum MSTICPy version

  • extras (Optional[List[str]], optional) – A list of extras required for MSTICPy

  • mp_release (Optional[str], optional) – Override the MSTICPy release version. This can also be specified in the environment variable ‘MP_TEST_VER’

Raises

RuntimeError – If the Python version does not support the notebook. If the MSTICPy version does not support the notebook and the user chose not to upgrade

msticpy.nbtools.azure_ml_tools.get_aml_user_folder() Optional[pathlib.Path]

Return the root of the user folder.

msticpy.nbtools.azure_ml_tools.is_in_aml()

Return True if running in Azure Machine Learning.

msticpy.nbtools.data_viewer module

Dataframe viewer.

class msticpy.nbtools.data_viewer.DataTableColumnChooser(data, selected_cols=None)

Bases: object

DataTableColumnChooser class.

Initialize the DataTableColumnChooser class.

property dataframe_columns

Return the selected set of DataFrame columns.

property datatable_columns

Return a list of Bokeh column definitions for the DataFrame.

display()

Display in IPython.

property selected_columns

Return the selected columns.

class msticpy.nbtools.data_viewer.DataTableFilter(data: pandas.core.frame.DataFrame)

Bases: object

Data filtering class.

Initialize the DataTableFilter class.

property bool_filters

Return current set of boolean filters.

property current_col

Return the currently selected column.

display()

Display in IPython.

property filtered_dataframe: pandas.core.frame.DataFrame

Return current filtered DataFrame.

import_filters(filters: Dict[str, msticpy.nbtools.data_viewer.FilterExpr])

Replace the current filters with filters.

Parameters

filters (Dict[str, FilterExpr]) – dict of filter name, FilterExpr FilterExpr is a tuple of: column [str], inv [bool], operator [str], expr [str]

class msticpy.nbtools.data_viewer.DataViewer(data: pandas.core.frame.DataFrame, selected_cols: Optional[List[str]] = None, debug=False)

Bases: object

Data viewer class.

Initialize the DataViewer class.

Parameters
  • data (pd.DataFrame) – The DataFrame to view

  • selected_cols (List[str], optional) – Initial subset of columns to show, by default None (all cols)

  • debug (bool) – Output additional debugging info to std out.

display()

Display the widget.

property filtered_data: pandas.core.frame.DataFrame

Return filtered dataframe.

property filters: Dict[str, msticpy.nbtools.data_viewer.FilterExpr]

Return current filters as a dict.

import_filters(filters: Dict[str, msticpy.nbtools.data_viewer.FilterExpr])

Import filter set replacing current filters.

Parameters

filters (Dict[str, FilterExpr]) – dict of filter name, FilterExpr FilterExpr is a tuple of: column [str], inv [bool], operator [str], expr [str]

show()

Display the data table control.

class msticpy.nbtools.data_viewer.FilterExpr(column, inv, operator, expr)

Bases: tuple

Create new instance of FilterExpr(column, inv, operator, expr)

property column

Alias for field number 0

count(value, /)

Return number of occurrences of value.

property expr

Alias for field number 3

index(value, start=0, stop=9223372036854775807, /)

Return first index of value.

Raises ValueError if the value is not present.

property inv

Alias for field number 1

property operator

Alias for field number 2

msticpy.nbtools.foliummap module

Folium map class.

class msticpy.nbtools.foliummap.FoliumMap(title: str = 'layer1', zoom_start: float = 2.5, tiles=None, width: str = '100%', height: str = '100%', location: Optional[list] = None)

Bases: object

Wrapper class for Folium/Leaflet mapping.

Create an instance of the folium map.

Parameters
  • title (str, optional) – Name of the layer (the default is ‘layer1’)

  • zoom_start (int, optional) – The zoom level of the map (the default is 7)

  • tiles ([type], optional) – Custom set of tiles or tile URL (the default is None)

  • width (str, optional) – Map display width (the default is ‘100%’)

  • height (str, optional) – Map display height (the default is ‘100%’)

  • location (list, optional) – Location to center map on

folium_map

The map object.

Type

folium.Map

add_geoloc_cluster(geo_locations: Iterable[msticpy.datamodel.entities.geo_location.GeoLocation], **kwargs)

Add a collection of GeoLocation objects to the map.

Parameters

geo_locations (Iterable[GeoLocation]) – Iterable of GeoLocation entities.

add_ip_cluster(ip_entities: Iterable[msticpy.datamodel.entities.ip_address.IpAddress], **kwargs)

Add a collection of IP Entities to the map.

Parameters
  • ip_entities (Iterable[IpAddress]) – a iterable of IpAddress Entities

  • kwargs (icon properties to use for displaying this cluster) –

add_locations(locations: Iterable[Tuple[float, float]], **kwargs)

Add a collection of lat/long tuples to the map.

Parameters

locations (Iterable[Tuple[float, float]]) – Iterable of location tuples.

center_map()

Calculate and set map center based on current coordinates.

msticpy.nbtools.foliummap.get_center_geo_locs(loc_entities: Iterable[msticpy.datamodel.entities.geo_location.GeoLocation], mode: str = 'median') Tuple[float, float]

Return the geographical center of the geo locations.

Parameters
  • loc_entities (Iterable[GeoLocation]) – GeoLocation entities with location information

  • mode (str, optional) – The averaging method to use, by default “median”. “median” and “mean” are the supported values.

Returns

Tuple of latitude, longitude

Return type

Tuple[Union[int, float], Union[int, float]]

msticpy.nbtools.foliummap.get_center_ip_entities(ip_entities: Iterable[msticpy.datamodel.entities.ip_address.IpAddress], mode: str = 'median') Tuple[float, float]

Return the geographical center of the IP address locations.

Parameters
  • ip_entities (Iterable[IpAddress]) – IpAddress entities with location information

  • mode (str, optional) – The averaging method to us, by default “median”. “median” and “mean” are the supported values.

Returns

Tuple of latitude, longitude

Return type

Tuple[Union[int, float], Union[int, float]]

msticpy.nbtools.foliummap.get_map_center(entities: Iterable[msticpy.datamodel.entities.entity.Entity], mode: str = 'modal')

Calculate median point between Entity IP locations.

Parameters
  • entities (Iterable[Entity]) – An iterable of entities containing IpAddress geolocation information. The entities can be IpAddress entities or other entities that have IpAddress properties. The entities must all be of the same type.

  • mode (str, optional) – The averaging method to use, by default “median”. “median” and “mean” are the supported values.

Returns

The Lattitude and Longitude calculated

Return type

Tuple

Notes

The function uses the first entity in the entities to determine how to process the collection. E.g. if the first entity has properties src_ip and dest_ip of type IpAddress, these are the only properties that will be processed for the remainder of the entities.

msticpy.nbtools.morph_charts module

Morph Charts class.

class msticpy.nbtools.morph_charts.MorphCharts

Bases: object

Create Morph Charts package data and render Morph Charts site.

Create object and populate charts container.

display(data: pandas.core.frame.DataFrame, chart_name: str) IPython.lib.display.IFrame

Prepare package data and display MorphChart in an IFrame.

Parameters
  • data (pd.DataFrame:) – A DataFrame of data for the morphchart to plot.

  • chart_name (str:) – The name of the Morph Chart to plot.

get_chart_details(chart_name)

Get description for a chart.

Parameters

chart_name (str:) – The name of the chart you get description for.

list_charts()

Get a list of avaliable charts.

search_charts(keyword)

Search for charts that match a keyword.

Parameters

keyword (str:) – The keyword to search charts for.

msticpy.nbtools.nbdisplay module

Module for common display functions.

msticpy.nbtools.nbdisplay.display_alert(alert: Union[Mapping[str, Any], msticpy.nbtools.security_alert.SecurityAlert], show_entities: bool = False)

Display a Security Alert.

Parameters
  • alert (Union[Mapping[str, Any], SecurityAlert]) – The alert to display as Mapping (e.g. pd.Series) or SecurityAlert

  • show_entities (bool, optional) – Whether to display entities (the default is False)

msticpy.nbtools.nbdisplay.display_logon_data(logon_event: pandas.core.frame.DataFrame, alert: Optional[msticpy.nbtools.security_alert.SecurityAlert] = None, os_family: Optional[str] = None)

Display logon data for one or more events as HTML table.

Parameters
  • logon_event (pd.DataFrame) – Dataframe containing one or more logon events

  • alert (SecurityAlert, optional) – obtain os_family from the security alert (the default is None)

  • os_family (str, optional) – explicitly specify os_family (Linux or Windows) (the default is None)

Notes

Currently only Windows Logon events.

msticpy.nbtools.nbdisplay.display_process_tree(process_tree: pandas.core.frame.DataFrame)

Display process tree data frame. (Deprecated).

Parameters
  • process_tree (pd.DataFrame) – Process tree DataFrame

  • to (The display module expects the columns NodeRole and Level) –

  • of (be populated. NoteRole is one) –

  • 'source' (or 'sibling'. Level indicates the 'hop' distance from the) –

  • node.

msticpy.nbtools.nbdisplay.draw_alert_entity_graph(nx_graph: networkx.classes.graph.Graph, font_size: int = 12, height: int = 8, width: int = 8, margin: float = 0.3, scale: int = 1)

Draw networkX graph with matplotlib.

Parameters
  • nx_graph (nx.Graph) – The NetworkX graph to draw

  • font_size (int, optional) – base font size (the default is 12)

  • height (int, optional) – Image height (the default is 8)

  • width (int, optional) – Image width (the default is 8)

  • margin (float, optional) – Image margin (the default is 0.3)

  • scale (int, optional) – Position scale (the default is 1)

  • deprecated: (.) – 0.3.2: Matplotlib version ‘draw_alert_entity_graph’ no longer supported - use ‘plot_entity_graph’

msticpy.nbtools.nbdisplay.exec_remaining_cells()

Execute all cells below currently selected cell.

msticpy.nbtools.nbdisplay.format_alert(alert: Union[Mapping[str, Any], msticpy.nbtools.security_alert.SecurityAlert], show_entities: bool = False) Union[IPython.core.display.HTML, Tuple[IPython.core.display.HTML, pandas.core.frame.DataFrame]]

Get IPython displayable Security Alert.

Parameters
  • alert (Union[Mapping[str, Any], SecurityAlert]) – The alert to display as Mapping (e.g. pd.Series) or SecurityAlert

  • show_entities (bool, optional) – Whether to display entities (the default is False)

Returns

Single or tuple of displayable IPython objects

Return type

Union[IPython.display.HTML, Tuple[IPython.display.HTML, pd.DataFrame]]

Raises

ValueError – If the alert object is in an unknown format

msticpy.nbtools.nbdisplay.format_logon(logon_event: Union[pandas.core.frame.DataFrame, pandas.core.series.Series], alert: Optional[msticpy.nbtools.security_alert.SecurityAlert] = None, os_family: Optional[str] = None) IPython.core.display.HTML

Return logon data for one or more events as HTML table.

Parameters
  • logon_event (Union[pd.DataFrame, pd.Series]) – Dataframe containing one or more logon events or Series containing a single logon event.

  • alert (SecurityAlert, optional) – obtain os_family from the security alert (the default is None)

  • os_family (str, optional) – explicitly specify os_family (Linux or Windows) (the default is None)

Returns

HTML display object

Return type

IPython.display.HTML

msticpy.nbtools.nbdisplay.plot_entity_graph(entity_graph: networkx.classes.graph.Graph, node_size: int = 25, font_size: Union[int, str] = 10, height: int = 800, width: int = 800, scale: int = 2, hide: bool = False) bokeh.plotting.figure.figure

Plot entity graph with Bokeh.

Parameters
  • entity_graph (nx.Graph) – The entity graph as a networkX graph

  • node_size (int, optional) – Size of the nodes in pixels, by default 25

  • font_size (int, optional) – Font size for node labels, by default 10 Can be an integer (point size) or a string (e.g. “10pt”)

  • width (int, optional) – Width in pixels, by default 800

  • height (int, optional) – Image height (the default is 800)

  • scale (int, optional) – Position scale (the default is 2)

  • hide (bool, optional) – Don’t show the plot, by default False. If True, just return the figure.

Returns

The network plot.

Return type

bokeh.plotting.figure

msticpy.nbtools.nbinit module

Initialization for Jupyter Notebooks.

msticpy.nbtools.nbinit.init_notebook(namespace: Dict[str, Any], def_imports: str = 'all', additional_packages: Optional[List[str]] = None, extra_imports: Optional[List[str]] = None, **kwargs) bool

Initialize the notebook environment.

Parameters
  • namespace (Dict[str, Any]) – Namespace (usually globals()) into which imports are to be populated.

  • def_imports (str, optional) – Import default packages. By default “all”. Possible values are: - “all” - import all packages - “nb” - import common notebook packages - “msticpy” - import msticpy packages - “none” (or any other value) don’t load any default packages.

  • additional_packages (List[str], optional) – Additional packages to be pip installed, by default None. Packages are specified by name only or version specification (e.g. “pandas>=0.25”)

  • user_install (bool, optional) – Install packages in the “user” rather than system site-packages. Use this option if you cannot or do not want to update the system packages. You should usually avoid using this option with standard Conda environments.

  • extra_imports (List[str], optional) – Additional import definitions, by default None. Imports are specified as up to 3 comma-delimited values in a string: “{source_pkg}, [{import_tgt}], [{alias}]” source_pkg is mandatory - equivalent to a simple “import xyz” statement. {import_tgt} specifies an object to import from the package equivalent to “from source_pkg import import_tgt” alias allows renaming of the imported object - equivalent to the “as alias” part of the import statement. If you want to provide just source_pkg and alias include an additional placeholder comma: e.g. “pandas, , pd”

  • friendly_exceptions (Optional[bool]) – Setting this to True causes msticpy to hook the notebook exception hander. Any exceptions derived from MsticpyUserException are displayed but do not produce a stack trace, etc. Defaults to system/user settings if no value is supplied.

  • verbose (bool, optional) – Display more verbose status, by default False

  • no_config_check (bool, optional) – Skip the check for valid configuration. Default is False.

Returns

True if successful

Return type

bool

Raises

MsticpyException – If extra_imports data format is incorrect. If package with required version check has no version information.

msticpy.nbtools.nbinit.list_default_imports()

List the default imports for init_notebook.

msticpy.nbtools.nbwidgets module

Module for pre-defined widget layouts.

class msticpy.nbtools.nbwidgets.AlertSelector(*args, **kwargs)

Bases: msticpy.nbtools.nbwidgets.SelectAlert

AlertSelector.

View list of alerts and select one for investigation. Optionally provide and action to call with the selected alert as a parameter (typically used to display the alert.)

selected_alert

The selected alert

Type

SecurityAlert

alert_id

The SystemAlertId of the selected alert

Type

str

alerts

The current alert list (DataFrame)

Type

List[SecurityAlert]

action

The callback action to execute on selection of an alert.

Type

Callable[…, None]

.. deprecated:: 0.5.2

Superceded by SelectAlert. Please use that version

Create a new instance of AlertSelector.

Parameters
  • alerts (pd.DataFrame) – DataFrame of alerts.

  • action (Callable[..., None], optional) – Optional function to execute for each selected alert. (the default is None)

  • columns (List[str], optional) – Override the default column names to use from alerts (the default is [‘StartTimeUtc’, ‘AlertName’, ‘CompromisedEntity’, ‘SystemAlertId’])

  • auto_display (bool, optional) – Whether to display on instantiation (the default is False)

display()

Display the interactive widgets.

property query_params

Query parameters derived from alert.

Returns

Return type

dict(str, str) – Dictionary of parameter names

class msticpy.nbtools.nbwidgets.GetEnvironmentKey(env_var: str, help_str: Optional[str] = None, prompt: str = 'Enter the value: ', auto_display: bool = False, **kwargs)

Bases: msticpy.nbtools.nbwidgets.RegisteredWidget

GetEnvironmentKey.

Tries to retrieve an environment variable value. The value can be changed/set and optionally saved back to the system environment.

Create a new instance of GetEnvironmentKey.

Parameters
  • env_var (str) – Name of the environment variable.

  • help_str (str, optional) – Help to display if the environment variable is not set. (the default is None)

  • prompt (str, optional) – Prompt to display with the text box. (the default is “Enter the value: “)

  • auto_display (bool, optional) – Whether to display on instantiation (the default is False)

display()

Display the interactive widgets.

property layout

Return underlying widget collection.

property name

Get the current name of the key.

property value

Get the current value of the key.

class msticpy.nbtools.nbwidgets.GetText(default: Optional[str] = None, prompt: str = 'Enter the value: ', auto_display: bool = False, **kwargs)

Bases: msticpy.nbtools.nbwidgets.RegisteredWidget

GetEnvironmentKey.

Tries to retrieve an environment variable value. The value can be changed/set and optionally saved back to the system environment.

Create a new instance of GetEnvironmentKey.

Parameters
  • default (str) – Default value.

  • prompt (str, optional) – Prompt to display with the text box. (the default is “Enter the value: “)

  • auto_display (bool, optional) – Whether to display on instantiation (the default is False)

See also

RegisteredWidget

display()

Display the interactive widgets.

property layout

Return underlying widget collection.

property value

Get the current value of the key.

class msticpy.nbtools.nbwidgets.Lookback(default: Optional[int] = None, label: str = 'Select time ({units}) to look back', origin_time: Optional[datetime.datetime] = None, min_value: Optional[int] = None, max_value: Optional[int] = None, units: str = 'hour', auto_display: bool = False)

Bases: object

Time lookback slider.

Create an instance of the lookback slider widget.

Parameters
  • default (int, optional) – The default ‘lookback’ time (the default is 4)

  • label (str, optional) – The description to display (the default is ‘Select time ({units}) to look back’)

  • origin_time (datetime, optional) – The origin time (the default is datetime.utcnow())

  • min_value (int, optional) – Minimum value (the default is 1)

  • max_value (int, optional) – Maximum value (the default is 240)

  • units (str, optional) – Time unit (the default is ‘hour’) Permissable values are ‘day’, ‘hour’, ‘minute’, ‘second’, ‘week’ These can all be abbreviated down to initial characters (‘d’, ‘m’, etc.)

  • auto_display (bool, optional) – Whether to display on instantiation (the default is False)

display()

Display the interactive widgets.

property layout

Return underlying widget.

property lookback

Return current widget lookback value.

property value

Return current widget lookback value.

class msticpy.nbtools.nbwidgets.OptionButtons(description: Optional[str] = 'Select an option to continue', buttons: Optional[Iterable[str]] = None, default: Optional[str] = None, timeout: int = 0, debug: bool = False)

Bases: object

OptionButtons creates a sequence of buttons to choose from.

The widget can be run in synchronous mode as a simple option selector or in async mode with a timeout. In the latter mode, after the timeout has expired the widget value is set to the default option button value. To use the async mode you must call widget.display_async() with the async keyword.

value

The value of the option selected (case-normalized)

Type

str

Example

>>> opt = OptionButtons(description="Continue something?",
...  buttons=["Maybe", "Yes", "Cancel"], timeout=10)
>>> await opt.display_async()

Initialize the OptionButton widget.

Parameters
  • description (Optional[str], optional) – Description label displayed above the buttons, by default “Select an option to continue”

  • buttons (Optional[Iterable[str]], optional) – A list of button values, by default None. This will default to [“Yes”, “No”, “Cancel”]

  • default (Optional[str], optional) – The default value to use on timeout, by default the first value in the buttons list

  • timeout (int, optional) – Timeout in seconds, by default 0

  • debug (bool, optional) – Adds some debug information to an Output controle, by default False

display()

Display widget in simple sync mode.

async display_async(reset: bool = False)

Display widget with timeout.

Parameters

reset (bool, optional) – Resets any current value to None, by default False

property layout

Create layout for buttons.

class msticpy.nbtools.nbwidgets.Progress(completed_len: int, visible: bool = True)

Bases: object

UI Progress bar.

Instantiate new _Progress UI.

Parameters
  • completed_len (int) – The expected value that indicates 100% done.

  • visible (bool) – If True start the progress UI visible, by default True.

display()

Display the control.

hide()

Hide the controls.

property max: int

Return the current progress maximum value.

Returns

Max value

Return type

int

show()

Make the controls visible.

update_progress(new_total: int = 0, delta: int = 0)

Update progress UI by increment or new total.

Parameters
  • new_total (int, optional) – New total, by default 0

  • delta (int, optional) – Increment to update current total, by default 0

property value: int

Return the current progress value.

Returns

Progess value

Return type

int

class msticpy.nbtools.nbwidgets.QueryTime(**kwargs)

Bases: msticpy.nbtools.nbwidgets.RegisteredWidget

QueryTime.

Composite widget to capture date and time origin and set start and end times for queries.

See also

RegisteredWidget

Create new instance of QueryTime.

Parameters
  • origin_time (datetime, optional) – The origin time (the default is datetime.utcnow())

  • label (str, optional) – The description to display (the default is ‘Select time ({units}) to look back’)

  • before (int, optional) – The default number of units before the origin_time (the default varies based on the unit)

  • after (int, optional) – The default number of units after the origin_time (the default varies based on the unit)

  • start (Union[datetime, str]) – Start of query time - alternative to specifying origin, before, after

  • end (Union[datetime, str]) – End of query time - alternative to specifying origin, before, after

  • timespan (TimeSpan) – TimeSpan of query time - alternative to specifying origin, before, after

  • max_before (int, optional) – The largest value for before (the default varies based on the unit)

  • max_after (int, optional) – The largest value for after (the default varies based on the unit)

  • units (str, optional) – Time unit (the default is ‘hour’) Permissable values are ‘day’, ‘hour’, ‘minute’, ‘second’, ‘week’ These can all be abbreviated down to initial characters (‘d’, ‘m’, etc.)

  • auto_display (bool, optional) – Whether to display on instantiation (the default is False)

IDS_ATTRIBS = ['origin_time', 'before', 'after', '_query_start', '_query_end', '_label']
display()

Display the interactive widgets.

property end

Query end time.

property start

Query start time.

property timespan

Return the timespan as a TimeSpan object.

property units

Time units used by control.

property value

Return the timespan as a TimeSpan object.

class msticpy.nbtools.nbwidgets.RegisteredWidget(id_vals: Optional[List[Any]] = None, val_attrs: Optional[List[str]] = None, nb_params: Optional[Dict[str, str]] = None, name_space: Dict[str, Any] = {'ABC': <class 'abc.ABC'>, 'AlertSelector': <class 'msticpy.nbtools.nbwidgets.AlertSelector'>, 'Any': typing.Any, 'Callable': typing.Callable, 'Dict': typing.Dict, 'GetEnvironmentKey': <class 'msticpy.nbtools.nbwidgets.GetEnvironmentKey'>, 'GetText': <class 'msticpy.nbtools.nbwidgets.GetText'>, 'HTML': <class 'IPython.core.display.HTML'>, 'IntEnum': <enum 'IntEnum'>, 'Iterable': typing.Iterable, 'JSONDecodeError': <class 'json.decoder.JSONDecodeError'>, 'Layout': <class 'ipywidgets.widgets.widget_layout.Layout'>, 'List': typing.List, 'Lookback': <class 'msticpy.nbtools.nbwidgets.Lookback'>, 'Mapping': typing.Mapping, 'OptionButtons': <class 'msticpy.nbtools.nbwidgets.OptionButtons'>, 'Optional': typing.Optional, 'Progress': <class 'msticpy.nbtools.nbwidgets.Progress'>, 'QueryTime': <class 'msticpy.nbtools.nbwidgets.QueryTime'>, 'RegisteredWidget': <class 'msticpy.nbtools.nbwidgets.RegisteredWidget'>, 'SelectAlert': <class 'msticpy.nbtools.nbwidgets.SelectAlert'>, 'SelectItem': <class 'msticpy.nbtools.nbwidgets.SelectItem'>, 'SelectString': <class 'msticpy.nbtools.nbwidgets.SelectString'>, 'SelectSubset': <class 'msticpy.nbtools.nbwidgets.SelectSubset'>, 'TimeSpan': <class 'msticpy.common.timespan.TimeSpan'>, 'TimeUnit': <enum 'TimeUnit'>, 'Tuple': typing.Tuple, 'Union': typing.Union, 'VERSION': '1.5.2', 'WeakValueDictionary': <class 'weakref.WeakValueDictionary'>, '_WIDGET_REG': <WeakValueDictionary>, '__all__': ['Lookback', 'QueryTime', 'SelectAlert', 'AlertSelector', 'GetEnvironmentKey', 'GetText', 'SelectItem', 'SelectString', 'SelectSubset', 'Progress'], '__annotations__': {'_WIDGET_REG': <class 'weakref.WeakValueDictionary'>}, '__author__': 'Ian Hellen', '__builtins__': {'ArithmeticError': <class 'ArithmeticError'>, 'AssertionError': <class 'AssertionError'>, 'AttributeError': <class 'AttributeError'>, 'BaseException': <class 'BaseException'>, 'BlockingIOError': <class 'BlockingIOError'>, 'BrokenPipeError': <class 'BrokenPipeError'>, 'BufferError': <class 'BufferError'>, 'BytesWarning': <class 'BytesWarning'>, 'ChildProcessError': <class 'ChildProcessError'>, 'ConnectionAbortedError': <class 'ConnectionAbortedError'>, 'ConnectionError': <class 'ConnectionError'>, 'ConnectionRefusedError': <class 'ConnectionRefusedError'>, 'ConnectionResetError': <class 'ConnectionResetError'>, 'DeprecationWarning': <class 'DeprecationWarning'>, 'EOFError': <class 'EOFError'>, 'Ellipsis': Ellipsis, 'EnvironmentError': <class 'OSError'>, 'Exception': <class 'Exception'>, 'False': False, 'FileExistsError': <class 'FileExistsError'>, 'FileNotFoundError': <class 'FileNotFoundError'>, 'FloatingPointError': <class 'FloatingPointError'>, 'FutureWarning': <class 'FutureWarning'>, 'GeneratorExit': <class 'GeneratorExit'>, 'IOError': <class 'OSError'>, 'ImportError': <class 'ImportError'>, 'ImportWarning': <class 'ImportWarning'>, 'IndentationError': <class 'IndentationError'>, 'IndexError': <class 'IndexError'>, 'InterruptedError': <class 'InterruptedError'>, 'IsADirectoryError': <class 'IsADirectoryError'>, 'KeyError': <class 'KeyError'>, 'KeyboardInterrupt': <class 'KeyboardInterrupt'>, 'LookupError': <class 'LookupError'>, 'MemoryError': <class 'MemoryError'>, 'ModuleNotFoundError': <class 'ModuleNotFoundError'>, 'NameError': <class 'NameError'>, 'None': None, 'NotADirectoryError': <class 'NotADirectoryError'>, 'NotImplemented': NotImplemented, 'NotImplementedError': <class 'NotImplementedError'>, 'OSError': <class 'OSError'>, 'OverflowError': <class 'OverflowError'>, 'PendingDeprecationWarning': <class 'PendingDeprecationWarning'>, 'PermissionError': <class 'PermissionError'>, 'ProcessLookupError': <class 'ProcessLookupError'>, 'RecursionError': <class 'RecursionError'>, 'ReferenceError': <class 'ReferenceError'>, 'ResourceWarning': <class 'ResourceWarning'>, 'RuntimeError': <class 'RuntimeError'>, 'RuntimeWarning': <class 'RuntimeWarning'>, 'StopAsyncIteration': <class 'StopAsyncIteration'>, 'StopIteration': <class 'StopIteration'>, 'SyntaxError': <class 'SyntaxError'>, 'SyntaxWarning': <class 'SyntaxWarning'>, 'SystemError': <class 'SystemError'>, 'SystemExit': <class 'SystemExit'>, 'TabError': <class 'TabError'>, 'TimeoutError': <class 'TimeoutError'>, 'True': True, 'TypeError': <class 'TypeError'>, 'UnboundLocalError': <class 'UnboundLocalError'>, 'UnicodeDecodeError': <class 'UnicodeDecodeError'>, 'UnicodeEncodeError': <class 'UnicodeEncodeError'>, 'UnicodeError': <class 'UnicodeError'>, 'UnicodeTranslateError': <class 'UnicodeTranslateError'>, 'UnicodeWarning': <class 'UnicodeWarning'>, 'UserWarning': <class 'UserWarning'>, 'ValueError': <class 'ValueError'>, 'Warning': <class 'Warning'>, 'ZeroDivisionError': <class 'ZeroDivisionError'>, '__build_class__': <built-in function __build_class__>, '__debug__': True, '__doc__': "Built-in functions, exceptions, and other objects.\n\nNoteworthy: None is the `nil' object; Ellipsis represents `...' in slices.", '__import__': <built-in function __import__>, '__loader__': <class '_frozen_importlib.BuiltinImporter'>, '__name__': 'builtins', '__package__': '', '__spec__': ModuleSpec(name='builtins', loader=<class '_frozen_importlib.BuiltinImporter'>), 'abs': <built-in function abs>, 'all': <built-in function all>, 'any': <built-in function any>, 'ascii': <built-in function ascii>, 'bin': <built-in function bin>, 'bool': <class 'bool'>, 'breakpoint': <built-in function breakpoint>, 'bytearray': <class 'bytearray'>, 'bytes': <class 'bytes'>, 'callable': <built-in function callable>, 'chr': <built-in function chr>, 'classmethod': <class 'classmethod'>, 'compile': <built-in function compile>, 'complex': <class 'complex'>, 'copyright': Copyright (c) 2001-2020 Python Software Foundation. All Rights Reserved.  Copyright (c) 2000 BeOpen.com. All Rights Reserved.  Copyright (c) 1995-2001 Corporation for National Research Initiatives. All Rights Reserved.  Copyright (c) 1991-1995 Stichting Mathematisch Centrum, Amsterdam. All Rights Reserved., 'credits':     Thanks to CWI, CNRI, BeOpen.com, Zope Corporation and a cast of thousands     for supporting Python development.  See www.python.org for more information., 'delattr': <built-in function delattr>, 'dict': <class 'dict'>, 'dir': <built-in function dir>, 'divmod': <built-in function divmod>, 'enumerate': <class 'enumerate'>, 'eval': <built-in function eval>, 'exec': <built-in function exec>, 'exit': Use exit() or Ctrl-D (i.e. EOF) to exit, 'filter': <class 'filter'>, 'float': <class 'float'>, 'format': <built-in function format>, 'frozenset': <class 'frozenset'>, 'getattr': <built-in function getattr>, 'globals': <built-in function globals>, 'hasattr': <built-in function hasattr>, 'hash': <built-in function hash>, 'help': Type help() for interactive help, or help(object) for help about object., 'hex': <built-in function hex>, 'id': <built-in function id>, 'input': <built-in function input>, 'int': <class 'int'>, 'isinstance': <built-in function isinstance>, 'issubclass': <built-in function issubclass>, 'iter': <built-in function iter>, 'len': <built-in function len>, 'license': Type license() to see the full license text, 'list': <class 'list'>, 'locals': <built-in function locals>, 'map': <class 'map'>, 'max': <built-in function max>, 'memoryview': <class 'memoryview'>, 'min': <built-in function min>, 'next': <built-in function next>, 'object': <class 'object'>, 'oct': <built-in function oct>, 'open': <built-in function open>, 'ord': <built-in function ord>, 'pow': <built-in function pow>, 'print': <built-in function print>, 'property': <class 'property'>, 'quit': Use quit() or Ctrl-D (i.e. EOF) to exit, 'range': <class 'range'>, 'repr': <built-in function repr>, 'reversed': <class 'reversed'>, 'round': <built-in function round>, 'set': <class 'set'>, 'setattr': <built-in function setattr>, 'slice': <class 'slice'>, 'sorted': <built-in function sorted>, 'staticmethod': <class 'staticmethod'>, 'str': <class 'str'>, 'sum': <built-in function sum>, 'super': <class 'super'>, 'tuple': <class 'tuple'>, 'type': <class 'type'>, 'vars': <built-in function vars>, 'zip': <class 'zip'>}, '__cached__': '/home/docs/checkouts/readthedocs.org/user_builds/msticpy/envs/v1.5.0/lib/python3.7/site-packages/msticpy/nbtools/__pycache__/nbwidgets.cpython-37.pyc', '__doc__': 'Module for pre-defined widget layouts.', '__file__': '/home/docs/checkouts/readthedocs.org/user_builds/msticpy/envs/v1.5.0/lib/python3.7/site-packages/msticpy/nbtools/nbwidgets.py', '__loader__': <_frozen_importlib_external.SourceFileLoader object>, '__name__': 'msticpy.nbtools.nbwidgets', '__package__': 'msticpy.nbtools', '__spec__': ModuleSpec(name='msticpy.nbtools.nbwidgets', loader=<_frozen_importlib_external.SourceFileLoader object>, origin='/home/docs/checkouts/readthedocs.org/user_builds/msticpy/envs/v1.5.0/lib/python3.7/site-packages/msticpy/nbtools/nbwidgets.py'), '__version__': '1.5.2', '_default_before_after': <function _default_before_after>, '_default_max_buffer': <function _default_max_buffer>, '_parse_time_unit': <function _parse_time_unit>, 'asyncio': <module 'asyncio' from '/home/docs/.pyenv/versions/3.7.9/lib/python3.7/asyncio/__init__.py'>, 'check_kwargs': <function check_kwargs>, 'datetime': <class 'datetime.datetime'>, 'deprecated': <function deprecated>, 'display': <function display>, 'export': <function export>, 'json': <module 'json' from '/home/docs/.pyenv/versions/3.7.9/lib/python3.7/json/__init__.py'>, 'os': <module 'os' from '/home/docs/checkouts/readthedocs.org/user_builds/msticpy/envs/v1.5.0/lib/python3.7/os.py'>, 'pd': <module 'pandas' from '/home/docs/checkouts/readthedocs.org/user_builds/msticpy/envs/v1.5.0/lib/python3.7/site-packages/pandas/__init__.py'>, 'random': <module 'random' from '/home/docs/checkouts/readthedocs.org/user_builds/msticpy/envs/v1.5.0/lib/python3.7/random.py'>, 'timedelta': <class 'datetime.timedelta'>, 'widgets': <module 'ipywidgets' from '/home/docs/checkouts/readthedocs.org/user_builds/msticpy/envs/v1.5.0/lib/python3.7/site-packages/ipywidgets/__init__.py'>}, register: bool = True, **kwargs)

Bases: abc.ABC

Register widget in the widget registry.

Registered widgets will store their values in the register. Each widget has an ID that that is derived from one or more of the initialization parameters. If an instance of the same widget class is created with the same parameters, its previous value will be repopulated from the registry. This is especially useful in notebooks where people accidentally re-run the same cell after entering values.

Initialize a registered widget.

Parameters
  • id_vals (Optional[List[Any]], optional) – The list of parameter values to use to identify this widget instance, by default None

  • val_attrs (Optional[List[str]], optional) – The names of the attributes to persist in the registry and recall, by default [“value”]

  • nb_params (Optional[Dict[str, str]], optional) – A dictionary of attribute names and global variables. If the variable exists in the global namespace it will be used to populate the corresponding widget attribute. This is only done if the widget attribute currently has no value (i.e. restoring a value from the registry takes priority over this), by default None

  • name_space (Dict[str, Any], optional) – Namespace to look for global variables, by default None

  • register (bool) – Do not register the widget or retrieve values from previously- registered instance.

class msticpy.nbtools.nbwidgets.SelectAlert(alerts: pandas.core.frame.DataFrame, action: Optional[Callable[[...], Optional[Tuple]]] = None, columns: Optional[List[str]] = None, time_col: str = 'StartTimeUtc', id_col: str = 'SystemAlertId', auto_display: bool = False)

Bases: object

Alert Selector.

View list of alerts and select one for investigation. Optionally provide and action to call with the selected alert as a parameter (typically used to display the alert.)

selected_alert

The selected alert

Type

SecurityAlert

alert_id

The SystemAlertId of the selected alert

Type

str

alerts

The current alert list (DataFrame)

Type

List[SecurityAlert]

action

The callback action to execute on selection of an alert.

Type

Callable[…, Optional[Tuple[…]]]

Create a new instance of AlertSelector.

Parameters
  • alerts (pd.DataFrame) – DataFrame of alerts.

  • action (Callable[..., Optional[Tuple]], optional) – Optional function to execute for each selected alert. If the function returns one or a tuple of displayable objects these will be displayed.

  • columns (List[str], optional) – Override the default column names to use from alerts (the default is [‘StartTimeUtc’, ‘AlertName’, ‘CompromisedEntity’, ‘SystemAlertId’])

  • time_col (str, optional) – The column in your alerts that determines when it was created Default is ‘StartTimeUtc’.

  • id_col (str, optional) – The column in your data that determines the alert id Default is ‘SystemAlertId’.

  • auto_display (bool, optional) – Whether to display on instantiation (the default is False)

display()

Display the interactive widgets.

class msticpy.nbtools.nbwidgets.SelectItem(description: str = 'Select an item', item_list: Optional[List[str]] = None, action: Optional[Callable[[...], Optional[Tuple]]] = None, item_dict: Optional[Mapping[str, str]] = None, auto_display: bool = False, height: str = '100px', width: str = '50%', display_filter: bool = True, value: str = '')

Bases: object

Selection list from list or dict.

value

The selected value.

Type

Any

item_action

Action to call for each selection.

Type

Callable[…, Optional[Tuple]]

Select an item from a list or dict.

Parameters
  • description (str, optional) – The widget label to display. (the default is ‘Select an item’)

  • item_list (List[str], optional) – A list of items to select from (the default is None)

  • item_dict (Mapping[str, str], optional) – A dict of items to select from. When using item_dict the keys are displayed as the selectable items and value corresponding to the selected key is set as the value property. (the default is None)

  • action (Callable[..., Optional[Tuple[...]]], optional) – function to call when item selected (passed a single parameter - the value of the currently selected item) (the default is None). If the function returns one or a tuple of displayable objects these will be displayed.

  • auto_display (bool, optional) – Whether to display on instantiation (the default is False)

  • height (str, optional) – Selection list height (the default is ‘100px’)

  • width (str, optional) – Selection list width (the default is ‘50%’)

  • display_filter (bool, optional) – Whether to display item filter (the default is True)

  • value (str, optional) – A default value to pre-populate the filter with.

display()

Display the interactive widget.

property layout

Return underlying widget collection.

class msticpy.nbtools.nbwidgets.SelectString(*args, **kwargs)

Bases: msticpy.nbtools.nbwidgets.SelectItem

Selection list from list or dict.

value

The selected value.

Type

Any

item_action

Action to call for each selection.

Type

Callable[…, None]

.. deprecated:: 0.5.2

Superceded by SelectItem. Please use that version

Select an item from a list or dict.

Parameters
  • description (str, optional) – The widget label to display. (the default is ‘Select an item’)

  • item_list (List[str], optional) – A list of items to select from (the default is None)

  • item_dict (Mapping[str, str], optional) – A dict of items to select from. When using item_dict the keys are displayed as the selectable items and value corresponding to the selected key is set as the value property. (the default is None)

  • action (Callable[..., None], optional) – function to call when item selected (passed a single parameter - the value of the currently selected item) (the default is None)

  • auto_display (bool, optional) – Whether to display on instantiation (the default is False)

  • height (str, optional) – Selection list height (the default is ‘100px’)

  • width (str, optional) – Selection list width (the default is ‘50%’)

  • display_filter (bool, optional) – Whether to display item filter (the default is True)

display()

Display the interactive widget.

property layout

Return underlying widget collection.

class msticpy.nbtools.nbwidgets.SelectSubset(source_items: Union[Dict[str, str], List[Any]], default_selected: Optional[Union[Dict[str, str], List[Any]]] = None, display_filter: bool = True, auto_display: bool = True)

Bases: object

Class to select a subset from an input list.

Create instance of SelectSubset widget.

Parameters
  • source_items (Union[Dict[str, str], List[Any]]) – List of source items - either a dictionary(label, value), a simple list or a list of (label, value) tuples.

  • default_selected (Union[Dict[str, str], List[Any]]) – Populate the selected list with values - either a dictionary(label, value), a simple list or a list of (label, value) tuples.

  • display_filter (bool, optional) – Whether to display item filter (the default is True)

  • auto_display (bool, optional) – Whether to display on instantiation (the default is True)

display()

Display the control.

property selected_items: List[Any]

Return a list of the selected items.

If the input list is a list of tuples, this returns a list of the selected tuples.

Returns

List of items in the selected list.

Return type

List[Any]

property selected_values: List[Any]

Return list of selected values.

If the input list is a list of tuples, this returns a list of values of the items.

Returns

List of selected item values.

Return type

List[Any]

class msticpy.nbtools.nbwidgets.TimeUnit(value)

Bases: enum.IntEnum

Time unit enumeration and value.

day = 86400
hour = 3600
minute = 60
second = 1
week = 604800

msticpy.nbtools.observationlist module

Observation summary collector.

class msticpy.nbtools.observationlist.Observation(caption: str, data: Any, description: Optional[str] = None, data_type: Optional[str] = None, link: Optional[str] = None, score: int = 0, tags: List[str] = NOTHING, additional_properties: Dict[str, Any] = NOTHING)

Bases: object

Observation definition.

Notes

captionstr

The title and index of the observation. Must be unique in the observation set.

descriptionOptional[str]

Text description of the observation. (default is None)

dataAny

The data to be stored for the observation (e.g. a pandas DataFrame). The object should implement a useable __repr__ to display correctly.

data_typeOptional[str]

The data type of the data property

linkOptional[str]

Link (usually a document-local link) to the originating section of the notebook. (default is None)

scoreint

The risk score associated with the observation. (default is 0)

tagsList[str]

Optional list of tags.

additional_properties Dict[str, Any]

Additional properties not covered by core properties.

Method generated by attrs for class Observation.

additional_properties: Dict[str, Any]
classmethod all_fields() List[str]

Return all fields of Observation class.

Returns

List of all field names.

Return type

List[str]

caption: str
data: Any
data_type: Optional[str]
description: Optional[str]
display()

Display the observation.

classmethod required_fields() List[str]

Return required fields for Observation instance.

Returns

List of field names.

Return type

List[str]

score: int
tags: List[str]
class msticpy.nbtools.observationlist.Observations(observationlist: Optional[msticpy.nbtools.observationlist.Observations] = None)

Bases: object

Class to collect and display investigation observations.

Create an observation list.

Parameters

observationlist (Observations, optional) – Initialize from an existing Observations list (the default is None)

add_observation(observation: Optional[msticpy.nbtools.observationlist.Observation] = None, **kwargs)

Add an observation.

Add an observation as an Observation instance or as a set of keyword parameters (see Observation class for acceptable values). Any keyword parameters that are not properties of Observation will be stored in the Observation.additional_properties dictionary

Parameters
  • observation (Observation) – An observation instance.

  • kwargs (str, Any) – List of key value pairs of the property names and values of the Observation to be stored.

display_observations()

Display the current observations using IPython.display.

property observations: Mapping[str, msticpy.nbtools.observationlist.Observation]

Return the current list of Observations.

Returns

The current ordered dictionary of Observations

Return type

Mapping[str, Observation]

msticpy.nbtools.process_tree module

Process Tree Visualization.

class msticpy.nbtools.process_tree.ProcessTreeAccessor(pandas_obj)

Bases: object

Pandas api extension for Process Tree.

Instantiate pandas extension class.

build(schema: Optional[msticpy.sectools.proc_tree_schema.ProcSchema] = None, **kwargs) pandas.core.frame.DataFrame

Build process trees from the process events.

Parameters
  • procs (pd.DataFrame) – Process events (Windows 4688 or Linux Auditd)

  • schema (ProcSchema, optional) – The column schema to use, by default None If None, then the schema is inferred

  • show_summary (bool) – Shows summary of the built tree, default is False. : bool

  • debug (bool) – If True produces extra debugging output, by default False

Returns

Process tree dataframe.

Return type

pd.DataFrame

Notes

It is not necessary to call this before plot. The process tree is built automatically. This is only needed if you want to return the processed tree data as a DataFrame

plot(**kwargs) Tuple[bokeh.plotting.figure.figure, bokeh.models.layouts.LayoutDOM]

Build and plot a process tree.

Parameters
  • schema (ProcSchema, optional) – The data schema to use for the data set, by default None (if None the schema is inferred)

  • output_var (str, optional) – Output variable for selected items in the tree, by default None

  • legend_col (str, optional) – The column used to color the tree items, by default None

  • show_table (bool) – Set to True to show a data table, by default False.

  • height (int, optional) – The height of the plot figure (the default is 700)

  • width (int, optional) – The width of the plot figure (the default is 900)

  • title (str, optional) – Title to display (the default is None)

  • hide_legend (bool, optional) – Hide the legend box, even if legend_col is specified.

  • pid_fmt (str, optional) – Display Process ID as ‘dec’ (decimal) or ‘hex’ (hexadecimal), default is ‘hex’.

Returns

figure - The main bokeh.plotting.figure Layout - Bokeh layout structure.

Return type

Tuple[figure, LayoutDOM]

class msticpy.nbtools.process_tree.TreeResult(proc_tree, schema, levels, n_rows)

Bases: tuple

Create new instance of TreeResult(proc_tree, schema, levels, n_rows)

count(value, /)

Return number of occurrences of value.

index(value, start=0, stop=9223372036854775807, /)

Return first index of value.

Raises ValueError if the value is not present.

property levels

Alias for field number 2

property n_rows

Alias for field number 3

property proc_tree

Alias for field number 0

property schema

Alias for field number 1

msticpy.nbtools.process_tree.build_and_show_process_tree(data: pandas.core.frame.DataFrame, schema: Optional[msticpy.sectools.proc_tree_schema.ProcSchema] = None, output_var: Optional[str] = None, legend_col: Optional[str] = None, **kwargs) Tuple[bokeh.plotting.figure.figure, bokeh.models.layouts.LayoutDOM]

Build process tree from data and plot a tree.

Parameters
  • data (pd.DataFrame) – Window process creation or Linux Auditd events

  • schema (ProcSchema) – The data schema to use for the data set, by default None (if None the schema is inferred)

  • output_var (str, optional) – Output variable for selected items in the tree, by default None

  • legend_col (str, optional) – The column used to color the tree items, by default None

  • kwargs (Dict[str, Any]) – Additional arguments passed to plot_process_tree

  • height (int, optional) – The height of the plot figure (the default is 700)

  • width (int, optional) – The width of the plot figure (the default is 900)

  • title (str, optional) – Title to display (the default is None)

  • hide_legend (bool, optional) – Hide the legend box, even if legend_col is specified.

  • pid_fmt (str, optional) – Display Process ID as ‘dec’ (decimal) or ‘hex’ (hexadecimal), default is ‘hex’.

Returns

figure - The main bokeh.plotting.figure Layout - Bokeh layout structure.

Return type

Tuple[figure, LayoutDOM]

Notes

For full parameter set for process tree display see the help for plot_process_tree.

msticpy.nbtools.process_tree.plot_process_tree(data: pandas.core.frame.DataFrame, schema: Optional[msticpy.sectools.proc_tree_schema.ProcSchema] = None, output_var: Optional[str] = None, legend_col: Optional[str] = None, show_table: bool = False, **kwargs) Tuple[bokeh.plotting.figure.figure, bokeh.models.layouts.LayoutDOM]

Plot a Process Tree Visualization.

Parameters
  • data (pd.DataFrame) – DataFrame containing one or more Process Trees

  • schema (ProcSchema, optional) – The data schema to use for the data set, by default None (if None the schema is inferred)

  • output_var (str, optional) – Output variable for selected items in the tree, by default None

  • legend_col (str, optional) – The column used to color the tree items, by default None

  • show_table (bool) – Set to True to show a data table, by default False.

  • height (int, optional) – The height of the plot figure (the default is 700)

  • width (int, optional) – The width of the plot figure (the default is 900)

  • title (str, optional) – Title to display (the default is None)

  • hide_legend (bool, optional) – Hide the legend box, even if legend_col is specified.

  • pid_fmt (str, optional) – Display Process ID as ‘dec’ (decimal) or ‘hex’ (hexadecimal), default is ‘hex’.

Returns

figure - The main bokeh.plotting.figure Layout - Bokeh layout structure.

Return type

Tuple[figure, LayoutDOM]

Raises

ProcessTreeSchemaException – If the data set schema is not valid for the plot.

Notes

The output_var variable will be overwritten with any selected values.

msticpy.nbtools.ti_browser module

Threat Intel Results Browser.

msticpy.nbtools.ti_browser.browse_results(data: pandas.core.frame.DataFrame, severities: Optional[Union[str, List[str]]] = None, **kwargs) msticpy.nbtools.nbwidgets.SelectItem

Return TI Results list browser.

Parameters
  • data (pd.DataFrame) – TI Results data from TIProviders

  • severities (Union[List[str], str, None], optional) – A list of the severity classes to show or the string ‘all’. By default these are [‘warning’, ‘high’].

  • kwargs – passed to SelectItem constuctor.

Returns

SelectItem browser for TI Data.

Return type

SelectItem

msticpy.nbtools.ti_browser.get_ti_select_options(ti_data: pandas.core.frame.DataFrame, severities: Optional[Union[str, List[str]]] = None)

Get SelectItem options for TI data.

msticpy.nbtools.ti_browser.raw_results(raw_result: str) str

Create pre-formatted details for raw results.

msticpy.nbtools.ti_browser.ti_details_display(ti_data)

Return TI Details display function.

msticpy.nbtools.timeseries module

Module for common display functions.

msticpy.nbtools.timeseries.display_timeseries_anomolies(data: pandas.core.frame.DataFrame, y: str = 'Total', time_column: str = 'TimeGenerated', anomalies_column: str = 'anomalies', source_columns: Optional[list] = None, period: int = 30, **kwargs) bokeh.plotting.figure.figure

Display time series anomalies visualization.

Parameters
  • data (pd.DataFrame) – DataFrame as a time series data set retreived from KQL time series functions. Dataframe must have columns specified in y, time_column and anomalies_column parameters

  • y (str, optional) – Name of column holding numeric values to plot against time series to determine anomolies (the default is ‘Total’)

  • time_column (str, optional) – Name of the timestamp column (the default is ‘TimeGenerated’)

  • anomalies_column (str, optional) – Name of the column holding binary status(1/0) for anomaly/benign (the default is ‘anomalies’)

  • source_columns (list, optional) – List of default source columns to use in tooltips (the default is None)

  • period (int, optional) – Period of the dataset for hourly-no of days, for daily-no of weeks. This is used to correctly calculate the plot height. (the default is 30)

  • ref_time (datetime, optional) – Input reference line to display (the default is None)

  • title (str, optional) – Title to display (the default is None)

  • legend (str, optional) – Where to position the legend None, left, right or inline (default is None)

  • yaxis (bool, optional) – Whether to show the yaxis and labels

  • range_tool (bool, optional) – Show the the range slider tool (default is True)

  • height (int, optional) – The height of the plot figure (the default is auto-calculated height)

  • width (int, optional) – The width of the plot figure (the default is 900)

  • xgrid (bool, optional) – Whether to show the xaxis grid (default is True)

  • ygrid (bool, optional) – Whether to show the yaxis grid (default is False)

  • color (list, optional) – List of colors to use in 3 plots as specified in order 3 plots- line(observed), circle(baseline), circle_x/user specified(anomalies). (the default is [“navy”, “green”, “firebrick”])

Returns

The bokeh plot figure.

Return type

figure

msticpy.nbtools.timeline module

Module for common display functions.

msticpy.nbtools.timeline.check_df_columns(data: pandas.core.frame.DataFrame, req_columns: List[str], help_uri: str, plot_type: str)

Check that specified columns are in the DataFrame.

Parameters
  • data (pd.DataFrame) – [description]

  • req_columns (List[str]) – [description]

  • help_uri (str) – [description]

  • plot_type (str) – [description]

Raises

MsticpyParameterError – If one or more columns not found in data

msticpy.nbtools.timeline.display_timeline(data: Union[pandas.core.frame.DataFrame, dict], time_column: str = 'TimeGenerated', source_columns: Optional[list] = None, **kwargs) bokeh.models.layouts.LayoutDOM

Display a timeline of events.

Parameters
  • data (Union[dict, pd.DataFrame]) –

    Either dict of data sets to plot on the timeline with the following structure:

    Key (str) - Name of data set to be displayed in legend
    Value (Dict[str, Any]) - containing:
        data (pd.DataFrame) - Data to plot
        time_column (str, optional) - Name of the timestamp column
        source_columns (list[str], optional) - source columns to use
            in tooltips
        color (str, optional) - color of datapoints for this data
    If any of the last values are omitted, they default to the values
    supplied as parameters to the function (see below)
    

    Or DataFrame as a single data set or grouped into individual plot series using the group_by parameter

  • time_column (str, optional) – Name of the timestamp column (the default is ‘TimeGenerated’)

  • source_columns (list, optional) – List of default source columns to use in tooltips (the default is None)

  • title (str, optional) – Title to display (the default is None)

  • alert (SecurityAlert, optional) – Add a reference line/label using the alert time (the default is None)

  • ref_event (Any, optional) – Add a reference line/label using the alert time (the default is None)

  • ref_time (datetime, optional) – Add a reference line/label using ref_time (the default is None)

  • group_by (str) – (where data is a DataFrame) The column to group timelines on

  • legend (str, optional) – “left”, “right”, “inline” or “none” (the default is to show a legend when plotting multiple series and not to show one when plotting a single series)

  • yaxis (bool, optional) – Whether to show the yaxis and labels (default is False)

  • ygrid (bool, optional) – Whether to show the yaxis grid (default is False)

  • xgrid (bool, optional) – Whether to show the xaxis grid (default is True)

  • range_tool (bool, optional) – Show the the range slider tool (default is True)

  • height (int, optional) – The height of the plot figure (the default is auto-calculated height)

  • width (int, optional) – The width of the plot figure (the default is 900)

  • color (str) – Default series color (default is “navy”)

  • overlay_data (pd.DataFrame:) – A second dataframe to plot as a different series.

  • overlay_color (str) – Overlay series color (default is “green”)

  • hide (bool, optional) – If True, create but do not display the plot. By default, False.

  • ref_events (pd.DataFrame, optional) – Add references line/label using the event times in the dataframe. (the default is None)

  • ref_time_col (str, optional) – Add references line/label using the this column in ref_events for the time value (x-axis). (this defaults the value of the time_column parameter or ‘TimeGenerated’ time_column is None)

  • ref_col (str, optional) – The column name to use for the label from ref_events (the default is None)

  • ref_times (List[Tuple[datetime, str]], optional) – Add one or more reference line/label using (the default is None)

Returns

The bokeh plot figure.

Return type

LayoutDOM

msticpy.nbtools.timeline.display_timeline_values(data: pandas.core.frame.DataFrame, value_col: Optional[str] = None, time_column: str = 'TimeGenerated', source_columns: Optional[list] = None, **kwargs) bokeh.models.layouts.LayoutDOM

Display a timeline of events.

Parameters
  • data (pd.DataFrame) – DataFrame as a single data set or grouped into individual plot series using the group_by parameter

  • time_column (str, optional) – Name of the timestamp column (the default is ‘TimeGenerated’)

  • value_col (str) – The column name holding the value to plot vertically

  • source_columns (list, optional) – List of default source columns to use in tooltips (the default is None)

  • x (str, optional) – alias of time_column

  • y (str) – an alias for value_col

  • title (str, optional) – Title to display (the default is None)

  • ref_event (Any, optional) – Add a reference line/label using the alert time (the default is None)

  • ref_time (datetime, optional) – Add a reference line/label using ref_time (the default is None)

  • ref_label (str, optional) – A label for the ref_event or ref_time reference item

  • group_by (str) – (where data is a DataFrame) The column to group timelines on

  • legend (str, optional) – “left”, “right”, “inline” or “none” (the default is to show a legend when plotting multiple series and not to show one when plotting a single series)

  • yaxis (bool, optional) – Whether to show the yaxis and labels

  • range_tool (bool, optional) – Show the the range slider tool (default is True)

  • height (int, optional) – The height of the plot figure (the default is auto-calculated height)

  • width (int, optional) – The width of the plot figure (the default is 900)

  • color (str) – Default series color (default is “navy”). This is overridden by automatic color assignments if plotting a grouped chart

  • kind (Union[str, List[str]]) – one or more glyph types to plot., optional Supported types are “circle”, “line” and “vbar” (default is “vbar”)

  • hide (bool, optional) – If True, create but do not display the plot. By default, False.

  • ref_events (pd.DataFrame, optional) – Add references line/label using the event times in the dataframe. (the default is None)

  • ref_time_col (str, optional) – Add references line/label using the this column in ref_events for the time value (x-axis). (this defaults the value of the time_column parameter or ‘TimeGenerated’ time_column is None)

  • ref_col (str, optional) – The column name to use for the label from ref_events (the default is None)

  • ref_times (List[Tuple[datetime, str]], optional) – Add one or more reference line/label using (the default is None)

Returns

The bokeh plot figure.

Return type

LayoutDOM

msticpy.nbtools.timeline_duration module

Timeline duration control.

class msticpy.nbtools.timeline_duration.PlotParams(height: Optional[int] = None, width: int = 900, title: Optional[str] = None, yaxis: bool = True, range_tool: bool = True, xgrid: bool = True, ygrid: bool = False, hide: bool = False, color: str = 'navy', ylabel_cols: Iterable[str] = NOTHING, ref_events: Optional[pandas.core.frame.DataFrame] = None, ref_col: Optional[str] = None, ref_times: Optional[List[Tuple[datetime.datetime, str]]] = None, source_columns: List = [])

Bases: object

Plot params for time_duration.

Method generated by attrs for class PlotParams.

color: str
classmethod field_list() List[str]

Return field names as a list.

height: Optional[int]
hide: bool
range_tool: bool
ref_col: Optional[str]
ref_events: Optional[pandas.core.frame.DataFrame]
ref_times: Optional[List[Tuple[datetime.datetime, str]]]
source_columns: List
title: Optional[str]
width: int
xgrid: bool
yaxis: bool
ygrid: bool
ylabel_cols: Iterable[str]
msticpy.nbtools.timeline_duration.display_timeline_duration(data: pandas.core.frame.DataFrame, group_by: Union[Iterable[str], str], time_column: str = 'TimeGenerated', end_time_column: Optional[str] = None, **kwargs) bokeh.models.layouts.LayoutDOM

Display a duration timeline of events grouped by one or more columns.

Parameters
  • data (pd.DataFrame) – Data to plot

  • group_by (Union[Iterable[str], str]) – The column name or iterable of column names to group the data by.

  • time_column (str) – Primary time column - will be used to calculate the start time of the duration for each group. If end_time_column is not specified it will also be used to calculate the end time.

  • end_time_column (Optional[str]) – If supplied, it will be used to calculate the end time of the duration for each group.

  • title (str, optional) – Title to display (the default is None)

  • ylabel_cols (Optional[Iterable[str]], optional) – The subset of the group columns to use for the y-axis labels.

  • yaxis (bool, optional) – Whether to show the yaxis and labels

  • range_tool (bool, optional) – Show the the range slider tool (default is True)

  • source_columns (list, optional) – List of default source columns to use in tooltips (the default is None)

  • height (int, optional) – The height of the plot figure (the default is auto-calculated height)

  • width (int, optional) – The width of the plot figure (the default is 900)

  • color (str) – Default series color (default is “navy”)

  • ref_events (pd.DataFrame, optional) – Add references line/label using the event times in the dataframe. (the default is None)

  • ref_col (str, optional) – The column name to use for the label from ref_events (the default is None)

  • ref_times (List[Tuple[datetime, str]], optional) – Add one or more reference line/label using (the default is None)

Returns

The bokeh plot figure.

Return type

LayoutDOM

msticpy.nbtools.timeline_pd_accessor module

Pandas accessor class for timeline functions.

class msticpy.nbtools.timeline_pd_accessor.TimeLineAccessor(pandas_obj)

Bases: object

Pandas api extension for Timeline.

Instantiate pandas extension class.

plot(**kwargs) bokeh.models.layouts.LayoutDOM

Display a timeline of events.

Parameters
  • time_column (str, optional) – Name of the timestamp column (the default is ‘TimeGenerated’)

  • source_columns (list, optional) – List of default source columns to use in tooltips (the default is None)

  • title (str, optional) – Title to display (the default is None)

  • alert (SecurityAlert, optional) – Add a reference line/label using the alert time (the default is None)

  • ref_event (Any, optional) – Add a reference line/label using the alert time (the default is None)

  • ref_time (datetime, optional) – Add a reference line/label using ref_time (the default is None)

  • group_by (str) – The column to group timelines on.

  • legend (str, optional) – “left”, “right”, “inline” or “none” (the default is to show a legend when plotting multiple series and not to show one when plotting a single series)

  • yaxis (bool, optional) – Whether to show the yaxis and labels (default is False)

  • ygrid (bool, optional) – Whether to show the yaxis grid (default is False)

  • xgrid (bool, optional) – Whether to show the xaxis grid (default is True)

  • range_tool (bool, optional) – Show the the range slider tool (default is True)

  • height (int, optional) – The height of the plot figure (the default is auto-calculated height)

  • width (int, optional) – The width of the plot figure (the default is 900)

  • color (str) – Default series color (default is “navy”)

  • overlay_data (pd.DataFrame:) – A second dataframe to plot as a different series.

  • overlay_color (str) – Overlay series color (default is “green”)

  • ref_events (pd.DataFrame, optional) – Add references line/label using the event times in the dataframe. (the default is None)

  • ref_time_col (str, optional) – Add references line/label using the this column in ref_events for the time value (x-axis). (this defaults the value of the time_column parameter or ‘TimeGenerated’ time_column is None)

  • ref_col (str, optional) – The column name to use for the label from ref_events (the default is None)

  • ref_times (List[Tuple[datetime, str]], optional) – Add one or more reference line/label using (the default is None)

Returns

The bokeh plot figure.

Return type

LayoutDOM

plot_duration(group_by: Union[Iterable[str], str], time_column: str = 'TimeGenerated', end_time_column: Optional[str] = None, **kwargs) bokeh.models.layouts.LayoutDOM

Display a duration timeline of events grouped by one or more columns.

Parameters
  • group_by (Union[Iterable[str], str]) – The column name or iterable of column names to group the data by.

  • time_column (str) – Primary time column - will be used to calculate the start time of the duration for each group. If end_time_column is not specified it will also be used to calculate the end time.

  • end_time_column (Optional[str]) – If supplied, it will be used to calculate the end time of the duration for each group.

  • title (str, optional) – Title to display (the default is None)

  • ylabel_cols (Optional[Iterable[str]], optional) – The subset of the group columns to use for the y-axis labels.

  • yaxis (bool, optional) – Whether to show the yaxis and labels

  • range_tool (bool, optional) – Show the the range slider tool (default is True)

  • source_columns (list, optional) – List of default source columns to use in tooltips (the default is None)

  • height (int, optional) – The height of the plot figure (the default is auto-calculated height)

  • width (int, optional) – The width of the plot figure (the default is 900)

  • color (str) – Default series color (default is “navy”)

  • ref_events (pd.DataFrame, optional) – Add references line/label using the event times in the dataframe. (the default is None)

  • ref_col (str, optional) – The column name to use for the label from ref_events (the default is None)

  • ref_times (List[Tuple[datetime, str]], optional) – Add one or more reference line/label using (the default is None)

Returns

The bokeh plot figure.

Return type

LayoutDOM

plot_values(value_col: Optional[str] = None, **kwargs) bokeh.models.layouts.LayoutDOM

Display a timeline of events.

Parameters
  • time_column (str, optional) – Name of the timestamp column (the default is ‘TimeGenerated’)

  • value_col (str) – The column name holding the value to plot vertically

  • source_columns (list, optional) – List of default source columns to use in tooltips (the default is None)

  • x (str, optional) – alias of time_column

  • y (str, optional) – alias of value_col

  • title (str, optional) – Title to display (the default is None)

  • ref_event (Any, optional) – Add a reference line/label using the alert time (the default is None)

  • ref_time (datetime, optional) – Add a reference line/label using ref_time (the default is None)

  • ref_label (str, optional) – A label for the ref_event or ref_time reference item

  • group_by (str) – (where data is a DataFrame) The column to group timelines on

  • legend (str, optional) – “left”, “right”, “inline” or “none” (the default is to show a legend when plotting multiple series and not to show one when plotting a single series)

  • yaxis (bool, optional) – Whether to show the yaxis and labels

  • range_tool (bool, optional) – Show the the range slider tool (default is True)

  • height (int, optional) – The height of the plot figure (the default is auto-calculated height)

  • width (int, optional) – The width of the plot figure (the default is 900)

  • color (str) – Default series color (default is “navy”). This is overridden by automatic color assignments if plotting a grouped chart

  • kind (Union[str, List[str]]) – one or more glyph types to plot., optional Supported types are “circle”, “line” and “vbar” (default is “vbar”)

  • ref_events (pd.DataFrame, optional) – Add references line/label using the event times in the dataframe. (the default is None)

  • ref_time_col (str, optional) – Add references line/label using the this column in ref_events for the time value (x-axis). (this defaults the value of the time_column parameter or ‘TimeGenerated’ time_column is None)

  • ref_col (str, optional) – The column name to use for the label from ref_events (the default is None)

  • ref_times (List[Tuple[datetime, str]], optional) – Add one or more reference line/label using (the default is None)

Returns

The bokeh plot figure.

Return type

LayoutDOM

msticpy.nbtools.user_config module

User configuration functions.

Loads providers based on user_defaults section in msticpyconfig.yaml

UserDefaults:
# List of query providers to load
    QueryProviders:
        AzureSentinel:
        Default:          # name of the provider listed in AzureSentinel.Workspaces
            alias: azsent   # optional - create "qry_azsent" object in globals
        CyberSoc:
            alias: soc
            connect: False  # optional - do not connect on load
        Splunk:             # add non-sentinel providers like this
        connect: False
        LocalData: local

# List of other providers/components to load
LoadComponents:
    TILookup:           # No parameters
    GeoIpLookup:
    provider: GeoLiteLookup   # geoip provider to use
    Notebooklets:       # Load and intialize Notebooklets
    query_provider:   # Pass it this query provider at startup
        AzureSentinel:
        workspace: CyberSoc
    Pivot:              # No parameters
    AzureData:          # auth_methods passed as startup param
    auth_methods: ['cli','interactive']
    AzureSentinelAPI:
    auth_methods: ['env','interactive']
    connect: False   # Load but do not connect

Note: For components that require authentication the default is to connect after loading. You can skip the connect step by add connect: False to the entry.

msticpy.nbtools.user_config.load_user_defaults() Dict[str, object]

Load providers from user defaults in msticpyconfig.yaml.

Returns

Dict of object name and provider instances.

Return type

Dict[str, object]