msticpy.nbtools package
msticpy.nbtools.azure_ml_tools module
Checker functions for Azure ML notebooks.
- msticpy.nbtools.azure_ml_tools.check_mp_ver(min_msticpy_ver: Union[str, Tuple], extras: Optional[List[str]])
Check and optionally update the current version of msticpy.
- Parameters
min_msticpy_ver (Tuple[int, int]) – Minimum MSTICPy version
extras (Optional[List[str]], optional) – A list of extras required for MSTICPy
- Raises
ImportError – If MSTICPy version is insufficient and we need to upgrade
- msticpy.nbtools.azure_ml_tools.check_python_ver(min_py_ver: Union[str, Tuple] = '3.6')
Check the current version of the Python kernel.
- Parameters
min_py_ver (Tuple[int, int]) – Minimum Python version
- Raises
RuntimeError – If the Python version does not support the notebook.
- msticpy.nbtools.azure_ml_tools.check_versions(min_py_ver: Union[str, Tuple] = '3.6', min_mp_ver: Union[str, Tuple] = '1.5.2', extras: Optional[List[str]] = None, mp_release: Optional[str] = None, **kwargs)
Check the current versions of the Python kernel and MSTICPy.
- Parameters
min_py_ver (Union[Tuple[int, int], str]) – Minimum Python version
min_mp_ver (Union[Tuple[int, int], str]) – Minimum MSTICPy version
extras (Optional[List[str]], optional) – A list of extras required for MSTICPy
mp_release (Optional[str], optional) – Override the MSTICPy release version. This can also be specified in the environment variable ‘MP_TEST_VER’
- Raises
RuntimeError – If the Python version does not support the notebook. If the MSTICPy version does not support the notebook and the user chose not to upgrade
- msticpy.nbtools.azure_ml_tools.get_aml_user_folder() Optional[pathlib.Path]
Return the root of the user folder.
- msticpy.nbtools.azure_ml_tools.is_in_aml()
Return True if running in Azure Machine Learning.
msticpy.nbtools.data_viewer module
Dataframe viewer.
- class msticpy.nbtools.data_viewer.DataTableColumnChooser(data, selected_cols=None)
Bases:
object
DataTableColumnChooser class.
Initialize the DataTableColumnChooser class.
- property dataframe_columns
Return the selected set of DataFrame columns.
- property datatable_columns
Return a list of Bokeh column definitions for the DataFrame.
- display()
Display in IPython.
- property selected_columns
Return the selected columns.
- class msticpy.nbtools.data_viewer.DataTableFilter(data: pandas.core.frame.DataFrame)
Bases:
object
Data filtering class.
Initialize the DataTableFilter class.
- property bool_filters
Return current set of boolean filters.
- property current_col
Return the currently selected column.
- display()
Display in IPython.
- property filtered_dataframe: pandas.core.frame.DataFrame
Return current filtered DataFrame.
- import_filters(filters: Dict[str, msticpy.nbtools.data_viewer.FilterExpr])
Replace the current filters with filters.
- Parameters
filters (Dict[str, FilterExpr]) – dict of filter name, FilterExpr FilterExpr is a tuple of: column [str], inv [bool], operator [str], expr [str]
- class msticpy.nbtools.data_viewer.DataViewer(data: pandas.core.frame.DataFrame, selected_cols: Optional[List[str]] = None, debug=False)
Bases:
object
Data viewer class.
Initialize the DataViewer class.
- Parameters
data (pd.DataFrame) – The DataFrame to view
selected_cols (List[str], optional) – Initial subset of columns to show, by default None (all cols)
debug (bool) – Output additional debugging info to std out.
- display()
Display the widget.
- property filtered_data: pandas.core.frame.DataFrame
Return filtered dataframe.
- property filters: Dict[str, msticpy.nbtools.data_viewer.FilterExpr]
Return current filters as a dict.
- import_filters(filters: Dict[str, msticpy.nbtools.data_viewer.FilterExpr])
Import filter set replacing current filters.
- Parameters
filters (Dict[str, FilterExpr]) – dict of filter name, FilterExpr FilterExpr is a tuple of: column [str], inv [bool], operator [str], expr [str]
- show()
Display the data table control.
- class msticpy.nbtools.data_viewer.FilterExpr(column, inv, operator, expr)
Bases:
tuple
Create new instance of FilterExpr(column, inv, operator, expr)
- property column
Alias for field number 0
- count(value, /)
Return number of occurrences of value.
- property expr
Alias for field number 3
- index(value, start=0, stop=9223372036854775807, /)
Return first index of value.
Raises ValueError if the value is not present.
- property inv
Alias for field number 1
- property operator
Alias for field number 2
msticpy.nbtools.foliummap module
Folium map class.
- class msticpy.nbtools.foliummap.FoliumMap(title: str = 'layer1', zoom_start: float = 2.5, tiles=None, width: str = '100%', height: str = '100%', location: Optional[list] = None)
Bases:
object
Wrapper class for Folium/Leaflet mapping.
Create an instance of the folium map.
- Parameters
title (str, optional) – Name of the layer (the default is ‘layer1’)
zoom_start (int, optional) – The zoom level of the map (the default is 7)
tiles ([type], optional) – Custom set of tiles or tile URL (the default is None)
width (str, optional) – Map display width (the default is ‘100%’)
height (str, optional) – Map display height (the default is ‘100%’)
location (list, optional) – Location to center map on
- folium_map
The map object.
- Type
folium.Map
- add_geoloc_cluster(geo_locations: Iterable[msticpy.datamodel.entities.geo_location.GeoLocation], **kwargs)
Add a collection of GeoLocation objects to the map.
- Parameters
geo_locations (Iterable[GeoLocation]) – Iterable of GeoLocation entities.
- add_ip_cluster(ip_entities: Iterable[msticpy.datamodel.entities.ip_address.IpAddress], **kwargs)
Add a collection of IP Entities to the map.
- Parameters
ip_entities (Iterable[IpAddress]) – a iterable of IpAddress Entities
kwargs (icon properties to use for displaying this cluster) –
- add_locations(locations: Iterable[Tuple[float, float]], **kwargs)
Add a collection of lat/long tuples to the map.
- Parameters
locations (Iterable[Tuple[float, float]]) – Iterable of location tuples.
- center_map()
Calculate and set map center based on current coordinates.
- msticpy.nbtools.foliummap.get_center_geo_locs(loc_entities: Iterable[msticpy.datamodel.entities.geo_location.GeoLocation], mode: str = 'median') Tuple[float, float]
Return the geographical center of the geo locations.
- Parameters
loc_entities (Iterable[GeoLocation]) – GeoLocation entities with location information
mode (str, optional) – The averaging method to use, by default “median”. “median” and “mean” are the supported values.
- Returns
Tuple of latitude, longitude
- Return type
Tuple[Union[int, float], Union[int, float]]
- msticpy.nbtools.foliummap.get_center_ip_entities(ip_entities: Iterable[msticpy.datamodel.entities.ip_address.IpAddress], mode: str = 'median') Tuple[float, float]
Return the geographical center of the IP address locations.
- Parameters
ip_entities (Iterable[IpAddress]) – IpAddress entities with location information
mode (str, optional) – The averaging method to us, by default “median”. “median” and “mean” are the supported values.
- Returns
Tuple of latitude, longitude
- Return type
Tuple[Union[int, float], Union[int, float]]
- msticpy.nbtools.foliummap.get_map_center(entities: Iterable[msticpy.datamodel.entities.entity.Entity], mode: str = 'modal')
Calculate median point between Entity IP locations.
- Parameters
entities (Iterable[Entity]) – An iterable of entities containing IpAddress geolocation information. The entities can be IpAddress entities or other entities that have IpAddress properties. The entities must all be of the same type.
mode (str, optional) – The averaging method to use, by default “median”. “median” and “mean” are the supported values.
- Returns
The Lattitude and Longitude calculated
- Return type
Tuple
Notes
The function uses the first entity in the entities to determine how to process the collection. E.g. if the first entity has properties src_ip and dest_ip of type IpAddress, these are the only properties that will be processed for the remainder of the entities.
msticpy.nbtools.morph_charts module
Morph Charts class.
- class msticpy.nbtools.morph_charts.MorphCharts
Bases:
object
Create Morph Charts package data and render Morph Charts site.
Create object and populate charts container.
- display(data: pandas.core.frame.DataFrame, chart_name: str) IPython.lib.display.IFrame
Prepare package data and display MorphChart in an IFrame.
- Parameters
data (pd.DataFrame:) – A DataFrame of data for the morphchart to plot.
chart_name (str:) – The name of the Morph Chart to plot.
- get_chart_details(chart_name)
Get description for a chart.
- Parameters
chart_name (str:) – The name of the chart you get description for.
- list_charts()
Get a list of avaliable charts.
- search_charts(keyword)
Search for charts that match a keyword.
- Parameters
keyword (str:) – The keyword to search charts for.
msticpy.nbtools.nbdisplay module
Module for common display functions.
- msticpy.nbtools.nbdisplay.display_alert(alert: Union[Mapping[str, Any], msticpy.nbtools.security_alert.SecurityAlert], show_entities: bool = False)
Display a Security Alert.
- Parameters
alert (Union[Mapping[str, Any], SecurityAlert]) – The alert to display as Mapping (e.g. pd.Series) or SecurityAlert
show_entities (bool, optional) – Whether to display entities (the default is False)
- msticpy.nbtools.nbdisplay.display_logon_data(logon_event: pandas.core.frame.DataFrame, alert: Optional[msticpy.nbtools.security_alert.SecurityAlert] = None, os_family: Optional[str] = None)
Display logon data for one or more events as HTML table.
- Parameters
logon_event (pd.DataFrame) – Dataframe containing one or more logon events
alert (SecurityAlert, optional) – obtain os_family from the security alert (the default is None)
os_family (str, optional) – explicitly specify os_family (Linux or Windows) (the default is None)
Notes
Currently only Windows Logon events.
- msticpy.nbtools.nbdisplay.display_process_tree(process_tree: pandas.core.frame.DataFrame)
Display process tree data frame. (Deprecated).
- Parameters
process_tree (pd.DataFrame) – Process tree DataFrame
to (The display module expects the columns NodeRole and Level) –
of (be populated. NoteRole is one) –
'source' (or 'sibling'. Level indicates the 'hop' distance from the) –
node. –
- msticpy.nbtools.nbdisplay.draw_alert_entity_graph(nx_graph: networkx.classes.graph.Graph, font_size: int = 12, height: int = 8, width: int = 8, margin: float = 0.3, scale: int = 1)
Draw networkX graph with matplotlib.
- Parameters
nx_graph (nx.Graph) – The NetworkX graph to draw
font_size (int, optional) – base font size (the default is 12)
height (int, optional) – Image height (the default is 8)
width (int, optional) – Image width (the default is 8)
margin (float, optional) – Image margin (the default is 0.3)
scale (int, optional) – Position scale (the default is 1)
deprecated: (.) – 0.3.2: Matplotlib version ‘draw_alert_entity_graph’ no longer supported - use ‘plot_entity_graph’
- msticpy.nbtools.nbdisplay.exec_remaining_cells()
Execute all cells below currently selected cell.
- msticpy.nbtools.nbdisplay.format_alert(alert: Union[Mapping[str, Any], msticpy.nbtools.security_alert.SecurityAlert], show_entities: bool = False) Union[IPython.core.display.HTML, Tuple[IPython.core.display.HTML, pandas.core.frame.DataFrame]]
Get IPython displayable Security Alert.
- Parameters
alert (Union[Mapping[str, Any], SecurityAlert]) – The alert to display as Mapping (e.g. pd.Series) or SecurityAlert
show_entities (bool, optional) – Whether to display entities (the default is False)
- Returns
Single or tuple of displayable IPython objects
- Return type
Union[IPython.display.HTML, Tuple[IPython.display.HTML, pd.DataFrame]]
- Raises
ValueError – If the alert object is in an unknown format
- msticpy.nbtools.nbdisplay.format_logon(logon_event: Union[pandas.core.frame.DataFrame, pandas.core.series.Series], alert: Optional[msticpy.nbtools.security_alert.SecurityAlert] = None, os_family: Optional[str] = None) IPython.core.display.HTML
Return logon data for one or more events as HTML table.
- Parameters
logon_event (Union[pd.DataFrame, pd.Series]) – Dataframe containing one or more logon events or Series containing a single logon event.
alert (SecurityAlert, optional) – obtain os_family from the security alert (the default is None)
os_family (str, optional) – explicitly specify os_family (Linux or Windows) (the default is None)
- Returns
HTML display object
- Return type
IPython.display.HTML
- msticpy.nbtools.nbdisplay.plot_entity_graph(entity_graph: networkx.classes.graph.Graph, node_size: int = 25, font_size: Union[int, str] = 10, height: int = 800, width: int = 800, scale: int = 2, hide: bool = False) bokeh.plotting.figure.figure
Plot entity graph with Bokeh.
- Parameters
entity_graph (nx.Graph) – The entity graph as a networkX graph
node_size (int, optional) – Size of the nodes in pixels, by default 25
font_size (int, optional) – Font size for node labels, by default 10 Can be an integer (point size) or a string (e.g. “10pt”)
width (int, optional) – Width in pixels, by default 800
height (int, optional) – Image height (the default is 800)
scale (int, optional) – Position scale (the default is 2)
hide (bool, optional) – Don’t show the plot, by default False. If True, just return the figure.
- Returns
The network plot.
- Return type
bokeh.plotting.figure
msticpy.nbtools.nbinit module
Initialization for Jupyter Notebooks.
- msticpy.nbtools.nbinit.init_notebook(namespace: Dict[str, Any], def_imports: str = 'all', additional_packages: Optional[List[str]] = None, extra_imports: Optional[List[str]] = None, **kwargs) bool
Initialize the notebook environment.
- Parameters
namespace (Dict[str, Any]) – Namespace (usually globals()) into which imports are to be populated.
def_imports (str, optional) – Import default packages. By default “all”. Possible values are: - “all” - import all packages - “nb” - import common notebook packages - “msticpy” - import msticpy packages - “none” (or any other value) don’t load any default packages.
additional_packages (List[str], optional) – Additional packages to be pip installed, by default None. Packages are specified by name only or version specification (e.g. “pandas>=0.25”)
user_install (bool, optional) – Install packages in the “user” rather than system site-packages. Use this option if you cannot or do not want to update the system packages. You should usually avoid using this option with standard Conda environments.
extra_imports (List[str], optional) – Additional import definitions, by default None. Imports are specified as up to 3 comma-delimited values in a string: “{source_pkg}, [{import_tgt}], [{alias}]” source_pkg is mandatory - equivalent to a simple “import xyz” statement. {import_tgt} specifies an object to import from the package equivalent to “from source_pkg import import_tgt” alias allows renaming of the imported object - equivalent to the “as alias” part of the import statement. If you want to provide just source_pkg and alias include an additional placeholder comma: e.g. “pandas, , pd”
friendly_exceptions (Optional[bool]) – Setting this to True causes msticpy to hook the notebook exception hander. Any exceptions derived from MsticpyUserException are displayed but do not produce a stack trace, etc. Defaults to system/user settings if no value is supplied.
verbose (bool, optional) – Display more verbose status, by default False
no_config_check (bool, optional) – Skip the check for valid configuration. Default is False.
- Returns
True if successful
- Return type
bool
- Raises
MsticpyException – If extra_imports data format is incorrect. If package with required version check has no version information.
- msticpy.nbtools.nbinit.list_default_imports()
List the default imports for init_notebook.
msticpy.nbtools.nbwidgets module
Module for pre-defined widget layouts.
- class msticpy.nbtools.nbwidgets.AlertSelector(*args, **kwargs)
Bases:
msticpy.nbtools.nbwidgets.SelectAlert
AlertSelector.
View list of alerts and select one for investigation. Optionally provide and action to call with the selected alert as a parameter (typically used to display the alert.)
- selected_alert
The selected alert
- Type
- alert_id
The SystemAlertId of the selected alert
- Type
str
- alerts
The current alert list (DataFrame)
- Type
List[SecurityAlert]
- action
The callback action to execute on selection of an alert.
- Type
Callable[…, None]
- .. deprecated:: 0.5.2
Superceded by SelectAlert. Please use that version
Create a new instance of AlertSelector.
- Parameters
alerts (pd.DataFrame) – DataFrame of alerts.
action (Callable[..., None], optional) – Optional function to execute for each selected alert. (the default is None)
columns (List[str], optional) – Override the default column names to use from alerts (the default is [‘StartTimeUtc’, ‘AlertName’, ‘CompromisedEntity’, ‘SystemAlertId’])
auto_display (bool, optional) – Whether to display on instantiation (the default is False)
- display()
Display the interactive widgets.
- property query_params
Query parameters derived from alert.
- Returns
- Return type
dict(str, str) – Dictionary of parameter names
- class msticpy.nbtools.nbwidgets.GetEnvironmentKey(env_var: str, help_str: Optional[str] = None, prompt: str = 'Enter the value: ', auto_display: bool = False, **kwargs)
Bases:
msticpy.nbtools.nbwidgets.RegisteredWidget
GetEnvironmentKey.
Tries to retrieve an environment variable value. The value can be changed/set and optionally saved back to the system environment.
Create a new instance of GetEnvironmentKey.
- Parameters
env_var (str) – Name of the environment variable.
help_str (str, optional) – Help to display if the environment variable is not set. (the default is None)
prompt (str, optional) – Prompt to display with the text box. (the default is “Enter the value: “)
auto_display (bool, optional) – Whether to display on instantiation (the default is False)
- display()
Display the interactive widgets.
- property layout
Return underlying widget collection.
- property name
Get the current name of the key.
- property value
Get the current value of the key.
- class msticpy.nbtools.nbwidgets.GetText(default: Optional[str] = None, prompt: str = 'Enter the value: ', auto_display: bool = False, **kwargs)
Bases:
msticpy.nbtools.nbwidgets.RegisteredWidget
GetEnvironmentKey.
Tries to retrieve an environment variable value. The value can be changed/set and optionally saved back to the system environment.
Create a new instance of GetEnvironmentKey.
- Parameters
default (str) – Default value.
prompt (str, optional) – Prompt to display with the text box. (the default is “Enter the value: “)
auto_display (bool, optional) – Whether to display on instantiation (the default is False)
See also
- display()
Display the interactive widgets.
- property layout
Return underlying widget collection.
- property value
Get the current value of the key.
- class msticpy.nbtools.nbwidgets.Lookback(default: Optional[int] = None, label: str = 'Select time ({units}) to look back', origin_time: Optional[datetime.datetime] = None, min_value: Optional[int] = None, max_value: Optional[int] = None, units: str = 'hour', auto_display: bool = False)
Bases:
object
Time lookback slider.
Create an instance of the lookback slider widget.
- Parameters
default (int, optional) – The default ‘lookback’ time (the default is 4)
label (str, optional) – The description to display (the default is ‘Select time ({units}) to look back’)
origin_time (datetime, optional) – The origin time (the default is datetime.utcnow())
min_value (int, optional) – Minimum value (the default is 1)
max_value (int, optional) – Maximum value (the default is 240)
units (str, optional) – Time unit (the default is ‘hour’) Permissable values are ‘day’, ‘hour’, ‘minute’, ‘second’, ‘week’ These can all be abbreviated down to initial characters (‘d’, ‘m’, etc.)
auto_display (bool, optional) – Whether to display on instantiation (the default is False)
- display()
Display the interactive widgets.
- property layout
Return underlying widget.
- property lookback
Return current widget lookback value.
- property value
Return current widget lookback value.
- class msticpy.nbtools.nbwidgets.OptionButtons(description: Optional[str] = 'Select an option to continue', buttons: Optional[Iterable[str]] = None, default: Optional[str] = None, timeout: int = 0, debug: bool = False)
Bases:
object
OptionButtons creates a sequence of buttons to choose from.
The widget can be run in synchronous mode as a simple option selector or in async mode with a timeout. In the latter mode, after the timeout has expired the widget value is set to the default option button value. To use the async mode you must call widget.display_async() with the async keyword.
- value
The value of the option selected (case-normalized)
- Type
str
Example
>>> opt = OptionButtons(description="Continue something?", ... buttons=["Maybe", "Yes", "Cancel"], timeout=10) >>> await opt.display_async()
Initialize the OptionButton widget.
- Parameters
description (Optional[str], optional) – Description label displayed above the buttons, by default “Select an option to continue”
buttons (Optional[Iterable[str]], optional) – A list of button values, by default None. This will default to [“Yes”, “No”, “Cancel”]
default (Optional[str], optional) – The default value to use on timeout, by default the first value in the buttons list
timeout (int, optional) – Timeout in seconds, by default 0
debug (bool, optional) – Adds some debug information to an Output controle, by default False
- display()
Display widget in simple sync mode.
- async display_async(reset: bool = False)
Display widget with timeout.
- Parameters
reset (bool, optional) – Resets any current value to None, by default False
- property layout
Create layout for buttons.
- class msticpy.nbtools.nbwidgets.Progress(completed_len: int, visible: bool = True)
Bases:
object
UI Progress bar.
Instantiate new _Progress UI.
- Parameters
completed_len (int) – The expected value that indicates 100% done.
visible (bool) – If True start the progress UI visible, by default True.
- display()
Display the control.
- hide()
Hide the controls.
- property max: int
Return the current progress maximum value.
- Returns
Max value
- Return type
int
- show()
Make the controls visible.
- update_progress(new_total: int = 0, delta: int = 0)
Update progress UI by increment or new total.
- Parameters
new_total (int, optional) – New total, by default 0
delta (int, optional) – Increment to update current total, by default 0
- property value: int
Return the current progress value.
- Returns
Progess value
- Return type
int
- class msticpy.nbtools.nbwidgets.QueryTime(**kwargs)
Bases:
msticpy.nbtools.nbwidgets.RegisteredWidget
QueryTime.
Composite widget to capture date and time origin and set start and end times for queries.
See also
Create new instance of QueryTime.
- Parameters
origin_time (datetime, optional) – The origin time (the default is datetime.utcnow())
label (str, optional) – The description to display (the default is ‘Select time ({units}) to look back’)
before (int, optional) – The default number of units before the origin_time (the default varies based on the unit)
after (int, optional) – The default number of units after the origin_time (the default varies based on the unit)
start (Union[datetime, str]) – Start of query time - alternative to specifying origin, before, after
end (Union[datetime, str]) – End of query time - alternative to specifying origin, before, after
timespan (TimeSpan) – TimeSpan of query time - alternative to specifying origin, before, after
max_before (int, optional) – The largest value for before (the default varies based on the unit)
max_after (int, optional) – The largest value for after (the default varies based on the unit)
units (str, optional) – Time unit (the default is ‘hour’) Permissable values are ‘day’, ‘hour’, ‘minute’, ‘second’, ‘week’ These can all be abbreviated down to initial characters (‘d’, ‘m’, etc.)
auto_display (bool, optional) – Whether to display on instantiation (the default is False)
- IDS_ATTRIBS = ['origin_time', 'before', 'after', '_query_start', '_query_end', '_label']
- display()
Display the interactive widgets.
- property end
Query end time.
- property start
Query start time.
- property timespan
Return the timespan as a TimeSpan object.
- property units
Time units used by control.
- property value
Return the timespan as a TimeSpan object.
- class msticpy.nbtools.nbwidgets.RegisteredWidget(id_vals: Optional[List[Any]] = None, val_attrs: Optional[List[str]] = None, nb_params: Optional[Dict[str, str]] = None, name_space: Dict[str, Any] = {'ABC': <class 'abc.ABC'>, 'AlertSelector': <class 'msticpy.nbtools.nbwidgets.AlertSelector'>, 'Any': typing.Any, 'Callable': typing.Callable, 'Dict': typing.Dict, 'GetEnvironmentKey': <class 'msticpy.nbtools.nbwidgets.GetEnvironmentKey'>, 'GetText': <class 'msticpy.nbtools.nbwidgets.GetText'>, 'HTML': <class 'IPython.core.display.HTML'>, 'IntEnum': <enum 'IntEnum'>, 'Iterable': typing.Iterable, 'JSONDecodeError': <class 'json.decoder.JSONDecodeError'>, 'Layout': <class 'ipywidgets.widgets.widget_layout.Layout'>, 'List': typing.List, 'Lookback': <class 'msticpy.nbtools.nbwidgets.Lookback'>, 'Mapping': typing.Mapping, 'OptionButtons': <class 'msticpy.nbtools.nbwidgets.OptionButtons'>, 'Optional': typing.Optional, 'Progress': <class 'msticpy.nbtools.nbwidgets.Progress'>, 'QueryTime': <class 'msticpy.nbtools.nbwidgets.QueryTime'>, 'RegisteredWidget': <class 'msticpy.nbtools.nbwidgets.RegisteredWidget'>, 'SelectAlert': <class 'msticpy.nbtools.nbwidgets.SelectAlert'>, 'SelectItem': <class 'msticpy.nbtools.nbwidgets.SelectItem'>, 'SelectString': <class 'msticpy.nbtools.nbwidgets.SelectString'>, 'SelectSubset': <class 'msticpy.nbtools.nbwidgets.SelectSubset'>, 'TimeSpan': <class 'msticpy.common.timespan.TimeSpan'>, 'TimeUnit': <enum 'TimeUnit'>, 'Tuple': typing.Tuple, 'Union': typing.Union, 'VERSION': '1.5.2', 'WeakValueDictionary': <class 'weakref.WeakValueDictionary'>, '_WIDGET_REG': <WeakValueDictionary>, '__all__': ['Lookback', 'QueryTime', 'SelectAlert', 'AlertSelector', 'GetEnvironmentKey', 'GetText', 'SelectItem', 'SelectString', 'SelectSubset', 'Progress'], '__annotations__': {'_WIDGET_REG': <class 'weakref.WeakValueDictionary'>}, '__author__': 'Ian Hellen', '__builtins__': {'ArithmeticError': <class 'ArithmeticError'>, 'AssertionError': <class 'AssertionError'>, 'AttributeError': <class 'AttributeError'>, 'BaseException': <class 'BaseException'>, 'BlockingIOError': <class 'BlockingIOError'>, 'BrokenPipeError': <class 'BrokenPipeError'>, 'BufferError': <class 'BufferError'>, 'BytesWarning': <class 'BytesWarning'>, 'ChildProcessError': <class 'ChildProcessError'>, 'ConnectionAbortedError': <class 'ConnectionAbortedError'>, 'ConnectionError': <class 'ConnectionError'>, 'ConnectionRefusedError': <class 'ConnectionRefusedError'>, 'ConnectionResetError': <class 'ConnectionResetError'>, 'DeprecationWarning': <class 'DeprecationWarning'>, 'EOFError': <class 'EOFError'>, 'Ellipsis': Ellipsis, 'EnvironmentError': <class 'OSError'>, 'Exception': <class 'Exception'>, 'False': False, 'FileExistsError': <class 'FileExistsError'>, 'FileNotFoundError': <class 'FileNotFoundError'>, 'FloatingPointError': <class 'FloatingPointError'>, 'FutureWarning': <class 'FutureWarning'>, 'GeneratorExit': <class 'GeneratorExit'>, 'IOError': <class 'OSError'>, 'ImportError': <class 'ImportError'>, 'ImportWarning': <class 'ImportWarning'>, 'IndentationError': <class 'IndentationError'>, 'IndexError': <class 'IndexError'>, 'InterruptedError': <class 'InterruptedError'>, 'IsADirectoryError': <class 'IsADirectoryError'>, 'KeyError': <class 'KeyError'>, 'KeyboardInterrupt': <class 'KeyboardInterrupt'>, 'LookupError': <class 'LookupError'>, 'MemoryError': <class 'MemoryError'>, 'ModuleNotFoundError': <class 'ModuleNotFoundError'>, 'NameError': <class 'NameError'>, 'None': None, 'NotADirectoryError': <class 'NotADirectoryError'>, 'NotImplemented': NotImplemented, 'NotImplementedError': <class 'NotImplementedError'>, 'OSError': <class 'OSError'>, 'OverflowError': <class 'OverflowError'>, 'PendingDeprecationWarning': <class 'PendingDeprecationWarning'>, 'PermissionError': <class 'PermissionError'>, 'ProcessLookupError': <class 'ProcessLookupError'>, 'RecursionError': <class 'RecursionError'>, 'ReferenceError': <class 'ReferenceError'>, 'ResourceWarning': <class 'ResourceWarning'>, 'RuntimeError': <class 'RuntimeError'>, 'RuntimeWarning': <class 'RuntimeWarning'>, 'StopAsyncIteration': <class 'StopAsyncIteration'>, 'StopIteration': <class 'StopIteration'>, 'SyntaxError': <class 'SyntaxError'>, 'SyntaxWarning': <class 'SyntaxWarning'>, 'SystemError': <class 'SystemError'>, 'SystemExit': <class 'SystemExit'>, 'TabError': <class 'TabError'>, 'TimeoutError': <class 'TimeoutError'>, 'True': True, 'TypeError': <class 'TypeError'>, 'UnboundLocalError': <class 'UnboundLocalError'>, 'UnicodeDecodeError': <class 'UnicodeDecodeError'>, 'UnicodeEncodeError': <class 'UnicodeEncodeError'>, 'UnicodeError': <class 'UnicodeError'>, 'UnicodeTranslateError': <class 'UnicodeTranslateError'>, 'UnicodeWarning': <class 'UnicodeWarning'>, 'UserWarning': <class 'UserWarning'>, 'ValueError': <class 'ValueError'>, 'Warning': <class 'Warning'>, 'ZeroDivisionError': <class 'ZeroDivisionError'>, '__build_class__': <built-in function __build_class__>, '__debug__': True, '__doc__': "Built-in functions, exceptions, and other objects.\n\nNoteworthy: None is the `nil' object; Ellipsis represents `...' in slices.", '__import__': <built-in function __import__>, '__loader__': <class '_frozen_importlib.BuiltinImporter'>, '__name__': 'builtins', '__package__': '', '__spec__': ModuleSpec(name='builtins', loader=<class '_frozen_importlib.BuiltinImporter'>), 'abs': <built-in function abs>, 'all': <built-in function all>, 'any': <built-in function any>, 'ascii': <built-in function ascii>, 'bin': <built-in function bin>, 'bool': <class 'bool'>, 'breakpoint': <built-in function breakpoint>, 'bytearray': <class 'bytearray'>, 'bytes': <class 'bytes'>, 'callable': <built-in function callable>, 'chr': <built-in function chr>, 'classmethod': <class 'classmethod'>, 'compile': <built-in function compile>, 'complex': <class 'complex'>, 'copyright': Copyright (c) 2001-2020 Python Software Foundation. All Rights Reserved. Copyright (c) 2000 BeOpen.com. All Rights Reserved. Copyright (c) 1995-2001 Corporation for National Research Initiatives. All Rights Reserved. Copyright (c) 1991-1995 Stichting Mathematisch Centrum, Amsterdam. All Rights Reserved., 'credits': Thanks to CWI, CNRI, BeOpen.com, Zope Corporation and a cast of thousands for supporting Python development. See www.python.org for more information., 'delattr': <built-in function delattr>, 'dict': <class 'dict'>, 'dir': <built-in function dir>, 'divmod': <built-in function divmod>, 'enumerate': <class 'enumerate'>, 'eval': <built-in function eval>, 'exec': <built-in function exec>, 'exit': Use exit() or Ctrl-D (i.e. EOF) to exit, 'filter': <class 'filter'>, 'float': <class 'float'>, 'format': <built-in function format>, 'frozenset': <class 'frozenset'>, 'getattr': <built-in function getattr>, 'globals': <built-in function globals>, 'hasattr': <built-in function hasattr>, 'hash': <built-in function hash>, 'help': Type help() for interactive help, or help(object) for help about object., 'hex': <built-in function hex>, 'id': <built-in function id>, 'input': <built-in function input>, 'int': <class 'int'>, 'isinstance': <built-in function isinstance>, 'issubclass': <built-in function issubclass>, 'iter': <built-in function iter>, 'len': <built-in function len>, 'license': Type license() to see the full license text, 'list': <class 'list'>, 'locals': <built-in function locals>, 'map': <class 'map'>, 'max': <built-in function max>, 'memoryview': <class 'memoryview'>, 'min': <built-in function min>, 'next': <built-in function next>, 'object': <class 'object'>, 'oct': <built-in function oct>, 'open': <built-in function open>, 'ord': <built-in function ord>, 'pow': <built-in function pow>, 'print': <built-in function print>, 'property': <class 'property'>, 'quit': Use quit() or Ctrl-D (i.e. EOF) to exit, 'range': <class 'range'>, 'repr': <built-in function repr>, 'reversed': <class 'reversed'>, 'round': <built-in function round>, 'set': <class 'set'>, 'setattr': <built-in function setattr>, 'slice': <class 'slice'>, 'sorted': <built-in function sorted>, 'staticmethod': <class 'staticmethod'>, 'str': <class 'str'>, 'sum': <built-in function sum>, 'super': <class 'super'>, 'tuple': <class 'tuple'>, 'type': <class 'type'>, 'vars': <built-in function vars>, 'zip': <class 'zip'>}, '__cached__': '/home/docs/checkouts/readthedocs.org/user_builds/msticpy/envs/v1.5.0/lib/python3.7/site-packages/msticpy/nbtools/__pycache__/nbwidgets.cpython-37.pyc', '__doc__': 'Module for pre-defined widget layouts.', '__file__': '/home/docs/checkouts/readthedocs.org/user_builds/msticpy/envs/v1.5.0/lib/python3.7/site-packages/msticpy/nbtools/nbwidgets.py', '__loader__': <_frozen_importlib_external.SourceFileLoader object>, '__name__': 'msticpy.nbtools.nbwidgets', '__package__': 'msticpy.nbtools', '__spec__': ModuleSpec(name='msticpy.nbtools.nbwidgets', loader=<_frozen_importlib_external.SourceFileLoader object>, origin='/home/docs/checkouts/readthedocs.org/user_builds/msticpy/envs/v1.5.0/lib/python3.7/site-packages/msticpy/nbtools/nbwidgets.py'), '__version__': '1.5.2', '_default_before_after': <function _default_before_after>, '_default_max_buffer': <function _default_max_buffer>, '_parse_time_unit': <function _parse_time_unit>, 'asyncio': <module 'asyncio' from '/home/docs/.pyenv/versions/3.7.9/lib/python3.7/asyncio/__init__.py'>, 'check_kwargs': <function check_kwargs>, 'datetime': <class 'datetime.datetime'>, 'deprecated': <function deprecated>, 'display': <function display>, 'export': <function export>, 'json': <module 'json' from '/home/docs/.pyenv/versions/3.7.9/lib/python3.7/json/__init__.py'>, 'os': <module 'os' from '/home/docs/checkouts/readthedocs.org/user_builds/msticpy/envs/v1.5.0/lib/python3.7/os.py'>, 'pd': <module 'pandas' from '/home/docs/checkouts/readthedocs.org/user_builds/msticpy/envs/v1.5.0/lib/python3.7/site-packages/pandas/__init__.py'>, 'random': <module 'random' from '/home/docs/checkouts/readthedocs.org/user_builds/msticpy/envs/v1.5.0/lib/python3.7/random.py'>, 'timedelta': <class 'datetime.timedelta'>, 'widgets': <module 'ipywidgets' from '/home/docs/checkouts/readthedocs.org/user_builds/msticpy/envs/v1.5.0/lib/python3.7/site-packages/ipywidgets/__init__.py'>}, register: bool = True, **kwargs)
Bases:
abc.ABC
Register widget in the widget registry.
Registered widgets will store their values in the register. Each widget has an ID that that is derived from one or more of the initialization parameters. If an instance of the same widget class is created with the same parameters, its previous value will be repopulated from the registry. This is especially useful in notebooks where people accidentally re-run the same cell after entering values.
Initialize a registered widget.
- Parameters
id_vals (Optional[List[Any]], optional) – The list of parameter values to use to identify this widget instance, by default None
val_attrs (Optional[List[str]], optional) – The names of the attributes to persist in the registry and recall, by default [“value”]
nb_params (Optional[Dict[str, str]], optional) – A dictionary of attribute names and global variables. If the variable exists in the global namespace it will be used to populate the corresponding widget attribute. This is only done if the widget attribute currently has no value (i.e. restoring a value from the registry takes priority over this), by default None
name_space (Dict[str, Any], optional) – Namespace to look for global variables, by default None
register (bool) – Do not register the widget or retrieve values from previously- registered instance.
- class msticpy.nbtools.nbwidgets.SelectAlert(alerts: pandas.core.frame.DataFrame, action: Optional[Callable[[...], Optional[Tuple]]] = None, columns: Optional[List[str]] = None, time_col: str = 'StartTimeUtc', id_col: str = 'SystemAlertId', auto_display: bool = False)
Bases:
object
Alert Selector.
View list of alerts and select one for investigation. Optionally provide and action to call with the selected alert as a parameter (typically used to display the alert.)
- selected_alert
The selected alert
- Type
- alert_id
The SystemAlertId of the selected alert
- Type
str
- alerts
The current alert list (DataFrame)
- Type
List[SecurityAlert]
- action
The callback action to execute on selection of an alert.
- Type
Callable[…, Optional[Tuple[…]]]
Create a new instance of AlertSelector.
- Parameters
alerts (pd.DataFrame) – DataFrame of alerts.
action (Callable[..., Optional[Tuple]], optional) – Optional function to execute for each selected alert. If the function returns one or a tuple of displayable objects these will be displayed.
columns (List[str], optional) – Override the default column names to use from alerts (the default is [‘StartTimeUtc’, ‘AlertName’, ‘CompromisedEntity’, ‘SystemAlertId’])
time_col (str, optional) – The column in your alerts that determines when it was created Default is ‘StartTimeUtc’.
id_col (str, optional) – The column in your data that determines the alert id Default is ‘SystemAlertId’.
auto_display (bool, optional) – Whether to display on instantiation (the default is False)
- display()
Display the interactive widgets.
- class msticpy.nbtools.nbwidgets.SelectItem(description: str = 'Select an item', item_list: Optional[List[str]] = None, action: Optional[Callable[[...], Optional[Tuple]]] = None, item_dict: Optional[Mapping[str, str]] = None, auto_display: bool = False, height: str = '100px', width: str = '50%', display_filter: bool = True, value: str = '')
Bases:
object
Selection list from list or dict.
- value
The selected value.
- Type
Any
- item_action
Action to call for each selection.
- Type
Callable[…, Optional[Tuple]]
Select an item from a list or dict.
- Parameters
description (str, optional) – The widget label to display. (the default is ‘Select an item’)
item_list (List[str], optional) – A list of items to select from (the default is None)
item_dict (Mapping[str, str], optional) – A dict of items to select from. When using item_dict the keys are displayed as the selectable items and value corresponding to the selected key is set as the value property. (the default is None)
action (Callable[..., Optional[Tuple[...]]], optional) – function to call when item selected (passed a single parameter - the value of the currently selected item) (the default is None). If the function returns one or a tuple of displayable objects these will be displayed.
auto_display (bool, optional) – Whether to display on instantiation (the default is False)
height (str, optional) – Selection list height (the default is ‘100px’)
width (str, optional) – Selection list width (the default is ‘50%’)
display_filter (bool, optional) – Whether to display item filter (the default is True)
value (str, optional) – A default value to pre-populate the filter with.
- display()
Display the interactive widget.
- property layout
Return underlying widget collection.
- class msticpy.nbtools.nbwidgets.SelectString(*args, **kwargs)
Bases:
msticpy.nbtools.nbwidgets.SelectItem
Selection list from list or dict.
- value
The selected value.
- Type
Any
- item_action
Action to call for each selection.
- Type
Callable[…, None]
- .. deprecated:: 0.5.2
Superceded by SelectItem. Please use that version
Select an item from a list or dict.
- Parameters
description (str, optional) – The widget label to display. (the default is ‘Select an item’)
item_list (List[str], optional) – A list of items to select from (the default is None)
item_dict (Mapping[str, str], optional) – A dict of items to select from. When using item_dict the keys are displayed as the selectable items and value corresponding to the selected key is set as the value property. (the default is None)
action (Callable[..., None], optional) – function to call when item selected (passed a single parameter - the value of the currently selected item) (the default is None)
auto_display (bool, optional) – Whether to display on instantiation (the default is False)
height (str, optional) – Selection list height (the default is ‘100px’)
width (str, optional) – Selection list width (the default is ‘50%’)
display_filter (bool, optional) – Whether to display item filter (the default is True)
- display()
Display the interactive widget.
- property layout
Return underlying widget collection.
- class msticpy.nbtools.nbwidgets.SelectSubset(source_items: Union[Dict[str, str], List[Any]], default_selected: Optional[Union[Dict[str, str], List[Any]]] = None, display_filter: bool = True, auto_display: bool = True)
Bases:
object
Class to select a subset from an input list.
Create instance of SelectSubset widget.
- Parameters
source_items (Union[Dict[str, str], List[Any]]) – List of source items - either a dictionary(label, value), a simple list or a list of (label, value) tuples.
default_selected (Union[Dict[str, str], List[Any]]) – Populate the selected list with values - either a dictionary(label, value), a simple list or a list of (label, value) tuples.
display_filter (bool, optional) – Whether to display item filter (the default is True)
auto_display (bool, optional) – Whether to display on instantiation (the default is True)
- display()
Display the control.
- property selected_items: List[Any]
Return a list of the selected items.
If the input list is a list of tuples, this returns a list of the selected tuples.
- Returns
List of items in the selected list.
- Return type
List[Any]
- property selected_values: List[Any]
Return list of selected values.
If the input list is a list of tuples, this returns a list of values of the items.
- Returns
List of selected item values.
- Return type
List[Any]
msticpy.nbtools.observationlist module
Observation summary collector.
- class msticpy.nbtools.observationlist.Observation(caption: str, data: Any, description: Optional[str] = None, data_type: Optional[str] = None, link: Optional[str] = None, score: int = 0, tags: List[str] = NOTHING, additional_properties: Dict[str, Any] = NOTHING)
Bases:
object
Observation definition.
Notes
- captionstr
The title and index of the observation. Must be unique in the observation set.
- descriptionOptional[str]
Text description of the observation. (default is None)
- dataAny
The data to be stored for the observation (e.g. a pandas DataFrame). The object should implement a useable __repr__ to display correctly.
- data_typeOptional[str]
The data type of the data property
- linkOptional[str]
Link (usually a document-local link) to the originating section of the notebook. (default is None)
- scoreint
The risk score associated with the observation. (default is 0)
- tagsList[str]
Optional list of tags.
- additional_properties Dict[str, Any]
Additional properties not covered by core properties.
Method generated by attrs for class Observation.
- additional_properties: Dict[str, Any]
- classmethod all_fields() List[str]
Return all fields of Observation class.
- Returns
List of all field names.
- Return type
List[str]
- data: Any
- data_type: Optional[str]
- description: Optional[str]
- display()
Display the observation.
- link: Optional[str]
- classmethod required_fields() List[str]
Return required fields for Observation instance.
- Returns
List of field names.
- Return type
List[str]
- score: int
- tags: List[str]
- class msticpy.nbtools.observationlist.Observations(observationlist: Optional[msticpy.nbtools.observationlist.Observations] = None)
Bases:
object
Class to collect and display investigation observations.
Create an observation list.
- Parameters
observationlist (Observations, optional) – Initialize from an existing Observations list (the default is None)
- add_observation(observation: Optional[msticpy.nbtools.observationlist.Observation] = None, **kwargs)
Add an observation.
Add an observation as an Observation instance or as a set of keyword parameters (see Observation class for acceptable values). Any keyword parameters that are not properties of Observation will be stored in the Observation.additional_properties dictionary
- Parameters
observation (Observation) – An observation instance.
kwargs (str, Any) – List of key value pairs of the property names and values of the Observation to be stored.
- display_observations()
Display the current observations using IPython.display.
- property observations: Mapping[str, msticpy.nbtools.observationlist.Observation]
Return the current list of Observations.
- Returns
The current ordered dictionary of Observations
- Return type
Mapping[str, Observation]
msticpy.nbtools.process_tree module
Process Tree Visualization.
- class msticpy.nbtools.process_tree.ProcessTreeAccessor(pandas_obj)
Bases:
object
Pandas api extension for Process Tree.
Instantiate pandas extension class.
- build(schema: Optional[msticpy.sectools.proc_tree_schema.ProcSchema] = None, **kwargs) pandas.core.frame.DataFrame
Build process trees from the process events.
- Parameters
procs (pd.DataFrame) – Process events (Windows 4688 or Linux Auditd)
schema (ProcSchema, optional) – The column schema to use, by default None If None, then the schema is inferred
show_summary (bool) – Shows summary of the built tree, default is False. : bool
debug (bool) – If True produces extra debugging output, by default False
- Returns
Process tree dataframe.
- Return type
pd.DataFrame
Notes
It is not necessary to call this before plot. The process tree is built automatically. This is only needed if you want to return the processed tree data as a DataFrame
- plot(**kwargs) Tuple[bokeh.plotting.figure.figure, bokeh.models.layouts.LayoutDOM]
Build and plot a process tree.
- Parameters
schema (ProcSchema, optional) – The data schema to use for the data set, by default None (if None the schema is inferred)
output_var (str, optional) – Output variable for selected items in the tree, by default None
legend_col (str, optional) – The column used to color the tree items, by default None
show_table (bool) – Set to True to show a data table, by default False.
height (int, optional) – The height of the plot figure (the default is 700)
width (int, optional) – The width of the plot figure (the default is 900)
title (str, optional) – Title to display (the default is None)
hide_legend (bool, optional) – Hide the legend box, even if legend_col is specified.
pid_fmt (str, optional) – Display Process ID as ‘dec’ (decimal) or ‘hex’ (hexadecimal), default is ‘hex’.
- Returns
figure - The main bokeh.plotting.figure Layout - Bokeh layout structure.
- Return type
Tuple[figure, LayoutDOM]
- class msticpy.nbtools.process_tree.TreeResult(proc_tree, schema, levels, n_rows)
Bases:
tuple
Create new instance of TreeResult(proc_tree, schema, levels, n_rows)
- count(value, /)
Return number of occurrences of value.
- index(value, start=0, stop=9223372036854775807, /)
Return first index of value.
Raises ValueError if the value is not present.
- property levels
Alias for field number 2
- property n_rows
Alias for field number 3
- property proc_tree
Alias for field number 0
- property schema
Alias for field number 1
- msticpy.nbtools.process_tree.build_and_show_process_tree(data: pandas.core.frame.DataFrame, schema: Optional[msticpy.sectools.proc_tree_schema.ProcSchema] = None, output_var: Optional[str] = None, legend_col: Optional[str] = None, **kwargs) Tuple[bokeh.plotting.figure.figure, bokeh.models.layouts.LayoutDOM]
Build process tree from data and plot a tree.
- Parameters
data (pd.DataFrame) – Window process creation or Linux Auditd events
schema (ProcSchema) – The data schema to use for the data set, by default None (if None the schema is inferred)
output_var (str, optional) – Output variable for selected items in the tree, by default None
legend_col (str, optional) – The column used to color the tree items, by default None
kwargs (Dict[str, Any]) – Additional arguments passed to plot_process_tree
height (int, optional) – The height of the plot figure (the default is 700)
width (int, optional) – The width of the plot figure (the default is 900)
title (str, optional) – Title to display (the default is None)
hide_legend (bool, optional) – Hide the legend box, even if legend_col is specified.
pid_fmt (str, optional) – Display Process ID as ‘dec’ (decimal) or ‘hex’ (hexadecimal), default is ‘hex’.
- Returns
figure - The main bokeh.plotting.figure Layout - Bokeh layout structure.
- Return type
Tuple[figure, LayoutDOM]
Notes
For full parameter set for process tree display see the help for plot_process_tree.
See also
- msticpy.nbtools.process_tree.plot_process_tree(data: pandas.core.frame.DataFrame, schema: Optional[msticpy.sectools.proc_tree_schema.ProcSchema] = None, output_var: Optional[str] = None, legend_col: Optional[str] = None, show_table: bool = False, **kwargs) Tuple[bokeh.plotting.figure.figure, bokeh.models.layouts.LayoutDOM]
Plot a Process Tree Visualization.
- Parameters
data (pd.DataFrame) – DataFrame containing one or more Process Trees
schema (ProcSchema, optional) – The data schema to use for the data set, by default None (if None the schema is inferred)
output_var (str, optional) – Output variable for selected items in the tree, by default None
legend_col (str, optional) – The column used to color the tree items, by default None
show_table (bool) – Set to True to show a data table, by default False.
height (int, optional) – The height of the plot figure (the default is 700)
width (int, optional) – The width of the plot figure (the default is 900)
title (str, optional) – Title to display (the default is None)
hide_legend (bool, optional) – Hide the legend box, even if legend_col is specified.
pid_fmt (str, optional) – Display Process ID as ‘dec’ (decimal) or ‘hex’ (hexadecimal), default is ‘hex’.
- Returns
figure - The main bokeh.plotting.figure Layout - Bokeh layout structure.
- Return type
Tuple[figure, LayoutDOM]
- Raises
ProcessTreeSchemaException – If the data set schema is not valid for the plot.
Notes
The output_var variable will be overwritten with any selected values.
msticpy.nbtools.ti_browser module
Threat Intel Results Browser.
- msticpy.nbtools.ti_browser.browse_results(data: pandas.core.frame.DataFrame, severities: Optional[Union[str, List[str]]] = None, **kwargs) msticpy.nbtools.nbwidgets.SelectItem
Return TI Results list browser.
- Parameters
data (pd.DataFrame) – TI Results data from TIProviders
severities (Union[List[str], str, None], optional) – A list of the severity classes to show or the string ‘all’. By default these are [‘warning’, ‘high’].
kwargs – passed to SelectItem constuctor.
- Returns
SelectItem browser for TI Data.
- Return type
- msticpy.nbtools.ti_browser.get_ti_select_options(ti_data: pandas.core.frame.DataFrame, severities: Optional[Union[str, List[str]]] = None)
Get SelectItem options for TI data.
- msticpy.nbtools.ti_browser.raw_results(raw_result: str) str
Create pre-formatted details for raw results.
- msticpy.nbtools.ti_browser.ti_details_display(ti_data)
Return TI Details display function.
msticpy.nbtools.timeseries module
Module for common display functions.
- msticpy.nbtools.timeseries.display_timeseries_anomolies(data: pandas.core.frame.DataFrame, y: str = 'Total', time_column: str = 'TimeGenerated', anomalies_column: str = 'anomalies', source_columns: Optional[list] = None, period: int = 30, **kwargs) bokeh.plotting.figure.figure
Display time series anomalies visualization.
- Parameters
data (pd.DataFrame) – DataFrame as a time series data set retreived from KQL time series functions. Dataframe must have columns specified in y, time_column and anomalies_column parameters
y (str, optional) – Name of column holding numeric values to plot against time series to determine anomolies (the default is ‘Total’)
time_column (str, optional) – Name of the timestamp column (the default is ‘TimeGenerated’)
anomalies_column (str, optional) – Name of the column holding binary status(1/0) for anomaly/benign (the default is ‘anomalies’)
source_columns (list, optional) – List of default source columns to use in tooltips (the default is None)
period (int, optional) – Period of the dataset for hourly-no of days, for daily-no of weeks. This is used to correctly calculate the plot height. (the default is 30)
ref_time (datetime, optional) – Input reference line to display (the default is None)
title (str, optional) – Title to display (the default is None)
legend (str, optional) – Where to position the legend None, left, right or inline (default is None)
yaxis (bool, optional) – Whether to show the yaxis and labels
range_tool (bool, optional) – Show the the range slider tool (default is True)
height (int, optional) – The height of the plot figure (the default is auto-calculated height)
width (int, optional) – The width of the plot figure (the default is 900)
xgrid (bool, optional) – Whether to show the xaxis grid (default is True)
ygrid (bool, optional) – Whether to show the yaxis grid (default is False)
color (list, optional) – List of colors to use in 3 plots as specified in order 3 plots- line(observed), circle(baseline), circle_x/user specified(anomalies). (the default is [“navy”, “green”, “firebrick”])
- Returns
The bokeh plot figure.
- Return type
figure
msticpy.nbtools.timeline module
Module for common display functions.
- msticpy.nbtools.timeline.check_df_columns(data: pandas.core.frame.DataFrame, req_columns: List[str], help_uri: str, plot_type: str)
Check that specified columns are in the DataFrame.
- Parameters
data (pd.DataFrame) – [description]
req_columns (List[str]) – [description]
help_uri (str) – [description]
plot_type (str) – [description]
- Raises
MsticpyParameterError – If one or more columns not found in data
- msticpy.nbtools.timeline.display_timeline(data: Union[pandas.core.frame.DataFrame, dict], time_column: str = 'TimeGenerated', source_columns: Optional[list] = None, **kwargs) bokeh.models.layouts.LayoutDOM
Display a timeline of events.
- Parameters
data (Union[dict, pd.DataFrame]) –
Either dict of data sets to plot on the timeline with the following structure:
Key (str) - Name of data set to be displayed in legend Value (Dict[str, Any]) - containing: data (pd.DataFrame) - Data to plot time_column (str, optional) - Name of the timestamp column source_columns (list[str], optional) - source columns to use in tooltips color (str, optional) - color of datapoints for this data If any of the last values are omitted, they default to the values supplied as parameters to the function (see below)
Or DataFrame as a single data set or grouped into individual plot series using the group_by parameter
time_column (str, optional) – Name of the timestamp column (the default is ‘TimeGenerated’)
source_columns (list, optional) – List of default source columns to use in tooltips (the default is None)
title (str, optional) – Title to display (the default is None)
alert (SecurityAlert, optional) – Add a reference line/label using the alert time (the default is None)
ref_event (Any, optional) – Add a reference line/label using the alert time (the default is None)
ref_time (datetime, optional) – Add a reference line/label using ref_time (the default is None)
group_by (str) – (where data is a DataFrame) The column to group timelines on
legend (str, optional) – “left”, “right”, “inline” or “none” (the default is to show a legend when plotting multiple series and not to show one when plotting a single series)
yaxis (bool, optional) – Whether to show the yaxis and labels (default is False)
ygrid (bool, optional) – Whether to show the yaxis grid (default is False)
xgrid (bool, optional) – Whether to show the xaxis grid (default is True)
range_tool (bool, optional) – Show the the range slider tool (default is True)
height (int, optional) – The height of the plot figure (the default is auto-calculated height)
width (int, optional) – The width of the plot figure (the default is 900)
color (str) – Default series color (default is “navy”)
overlay_data (pd.DataFrame:) – A second dataframe to plot as a different series.
overlay_color (str) – Overlay series color (default is “green”)
hide (bool, optional) – If True, create but do not display the plot. By default, False.
ref_events (pd.DataFrame, optional) – Add references line/label using the event times in the dataframe. (the default is None)
ref_time_col (str, optional) – Add references line/label using the this column in ref_events for the time value (x-axis). (this defaults the value of the time_column parameter or ‘TimeGenerated’ time_column is None)
ref_col (str, optional) – The column name to use for the label from ref_events (the default is None)
ref_times (List[Tuple[datetime, str]], optional) – Add one or more reference line/label using (the default is None)
- Returns
The bokeh plot figure.
- Return type
LayoutDOM
- msticpy.nbtools.timeline.display_timeline_values(data: pandas.core.frame.DataFrame, value_col: Optional[str] = None, time_column: str = 'TimeGenerated', source_columns: Optional[list] = None, **kwargs) bokeh.models.layouts.LayoutDOM
Display a timeline of events.
- Parameters
data (pd.DataFrame) – DataFrame as a single data set or grouped into individual plot series using the group_by parameter
time_column (str, optional) – Name of the timestamp column (the default is ‘TimeGenerated’)
value_col (str) – The column name holding the value to plot vertically
source_columns (list, optional) – List of default source columns to use in tooltips (the default is None)
x (str, optional) – alias of time_column
y (str) – an alias for value_col
title (str, optional) – Title to display (the default is None)
ref_event (Any, optional) – Add a reference line/label using the alert time (the default is None)
ref_time (datetime, optional) – Add a reference line/label using ref_time (the default is None)
ref_label (str, optional) – A label for the ref_event or ref_time reference item
group_by (str) – (where data is a DataFrame) The column to group timelines on
legend (str, optional) – “left”, “right”, “inline” or “none” (the default is to show a legend when plotting multiple series and not to show one when plotting a single series)
yaxis (bool, optional) – Whether to show the yaxis and labels
range_tool (bool, optional) – Show the the range slider tool (default is True)
height (int, optional) – The height of the plot figure (the default is auto-calculated height)
width (int, optional) – The width of the plot figure (the default is 900)
color (str) – Default series color (default is “navy”). This is overridden by automatic color assignments if plotting a grouped chart
kind (Union[str, List[str]]) – one or more glyph types to plot., optional Supported types are “circle”, “line” and “vbar” (default is “vbar”)
hide (bool, optional) – If True, create but do not display the plot. By default, False.
ref_events (pd.DataFrame, optional) – Add references line/label using the event times in the dataframe. (the default is None)
ref_time_col (str, optional) – Add references line/label using the this column in ref_events for the time value (x-axis). (this defaults the value of the time_column parameter or ‘TimeGenerated’ time_column is None)
ref_col (str, optional) – The column name to use for the label from ref_events (the default is None)
ref_times (List[Tuple[datetime, str]], optional) – Add one or more reference line/label using (the default is None)
- Returns
The bokeh plot figure.
- Return type
LayoutDOM
msticpy.nbtools.timeline_duration module
Timeline duration control.
- class msticpy.nbtools.timeline_duration.PlotParams(height: Optional[int] = None, width: int = 900, title: Optional[str] = None, yaxis: bool = True, range_tool: bool = True, xgrid: bool = True, ygrid: bool = False, hide: bool = False, color: str = 'navy', ylabel_cols: Iterable[str] = NOTHING, ref_events: Optional[pandas.core.frame.DataFrame] = None, ref_col: Optional[str] = None, ref_times: Optional[List[Tuple[datetime.datetime, str]]] = None, source_columns: List = [])
Bases:
object
Plot params for time_duration.
Method generated by attrs for class PlotParams.
- color: str
- classmethod field_list() List[str]
Return field names as a list.
- height: Optional[int]
- hide: bool
- range_tool: bool
- ref_col: Optional[str]
- ref_events: Optional[pandas.core.frame.DataFrame]
- ref_times: Optional[List[Tuple[datetime.datetime, str]]]
- source_columns: List
- title: Optional[str]
- width: int
- xgrid: bool
- yaxis: bool
- ygrid: bool
- ylabel_cols: Iterable[str]
- msticpy.nbtools.timeline_duration.display_timeline_duration(data: pandas.core.frame.DataFrame, group_by: Union[Iterable[str], str], time_column: str = 'TimeGenerated', end_time_column: Optional[str] = None, **kwargs) bokeh.models.layouts.LayoutDOM
Display a duration timeline of events grouped by one or more columns.
- Parameters
data (pd.DataFrame) – Data to plot
group_by (Union[Iterable[str], str]) – The column name or iterable of column names to group the data by.
time_column (str) – Primary time column - will be used to calculate the start time of the duration for each group. If end_time_column is not specified it will also be used to calculate the end time.
end_time_column (Optional[str]) – If supplied, it will be used to calculate the end time of the duration for each group.
title (str, optional) – Title to display (the default is None)
ylabel_cols (Optional[Iterable[str]], optional) – The subset of the group columns to use for the y-axis labels.
yaxis (bool, optional) – Whether to show the yaxis and labels
range_tool (bool, optional) – Show the the range slider tool (default is True)
source_columns (list, optional) – List of default source columns to use in tooltips (the default is None)
height (int, optional) – The height of the plot figure (the default is auto-calculated height)
width (int, optional) – The width of the plot figure (the default is 900)
color (str) – Default series color (default is “navy”)
ref_events (pd.DataFrame, optional) – Add references line/label using the event times in the dataframe. (the default is None)
ref_col (str, optional) – The column name to use for the label from ref_events (the default is None)
ref_times (List[Tuple[datetime, str]], optional) – Add one or more reference line/label using (the default is None)
- Returns
The bokeh plot figure.
- Return type
LayoutDOM
msticpy.nbtools.timeline_pd_accessor module
Pandas accessor class for timeline functions.
- class msticpy.nbtools.timeline_pd_accessor.TimeLineAccessor(pandas_obj)
Bases:
object
Pandas api extension for Timeline.
Instantiate pandas extension class.
- plot(**kwargs) bokeh.models.layouts.LayoutDOM
Display a timeline of events.
- Parameters
time_column (str, optional) – Name of the timestamp column (the default is ‘TimeGenerated’)
source_columns (list, optional) – List of default source columns to use in tooltips (the default is None)
title (str, optional) – Title to display (the default is None)
alert (SecurityAlert, optional) – Add a reference line/label using the alert time (the default is None)
ref_event (Any, optional) – Add a reference line/label using the alert time (the default is None)
ref_time (datetime, optional) – Add a reference line/label using ref_time (the default is None)
group_by (str) – The column to group timelines on.
legend (str, optional) – “left”, “right”, “inline” or “none” (the default is to show a legend when plotting multiple series and not to show one when plotting a single series)
yaxis (bool, optional) – Whether to show the yaxis and labels (default is False)
ygrid (bool, optional) – Whether to show the yaxis grid (default is False)
xgrid (bool, optional) – Whether to show the xaxis grid (default is True)
range_tool (bool, optional) – Show the the range slider tool (default is True)
height (int, optional) – The height of the plot figure (the default is auto-calculated height)
width (int, optional) – The width of the plot figure (the default is 900)
color (str) – Default series color (default is “navy”)
overlay_data (pd.DataFrame:) – A second dataframe to plot as a different series.
overlay_color (str) – Overlay series color (default is “green”)
ref_events (pd.DataFrame, optional) – Add references line/label using the event times in the dataframe. (the default is None)
ref_time_col (str, optional) – Add references line/label using the this column in ref_events for the time value (x-axis). (this defaults the value of the time_column parameter or ‘TimeGenerated’ time_column is None)
ref_col (str, optional) – The column name to use for the label from ref_events (the default is None)
ref_times (List[Tuple[datetime, str]], optional) – Add one or more reference line/label using (the default is None)
- Returns
The bokeh plot figure.
- Return type
LayoutDOM
- plot_duration(group_by: Union[Iterable[str], str], time_column: str = 'TimeGenerated', end_time_column: Optional[str] = None, **kwargs) bokeh.models.layouts.LayoutDOM
Display a duration timeline of events grouped by one or more columns.
- Parameters
group_by (Union[Iterable[str], str]) – The column name or iterable of column names to group the data by.
time_column (str) – Primary time column - will be used to calculate the start time of the duration for each group. If end_time_column is not specified it will also be used to calculate the end time.
end_time_column (Optional[str]) – If supplied, it will be used to calculate the end time of the duration for each group.
title (str, optional) – Title to display (the default is None)
ylabel_cols (Optional[Iterable[str]], optional) – The subset of the group columns to use for the y-axis labels.
yaxis (bool, optional) – Whether to show the yaxis and labels
range_tool (bool, optional) – Show the the range slider tool (default is True)
source_columns (list, optional) – List of default source columns to use in tooltips (the default is None)
height (int, optional) – The height of the plot figure (the default is auto-calculated height)
width (int, optional) – The width of the plot figure (the default is 900)
color (str) – Default series color (default is “navy”)
ref_events (pd.DataFrame, optional) – Add references line/label using the event times in the dataframe. (the default is None)
ref_col (str, optional) – The column name to use for the label from ref_events (the default is None)
ref_times (List[Tuple[datetime, str]], optional) – Add one or more reference line/label using (the default is None)
- Returns
The bokeh plot figure.
- Return type
LayoutDOM
- plot_values(value_col: Optional[str] = None, **kwargs) bokeh.models.layouts.LayoutDOM
Display a timeline of events.
- Parameters
time_column (str, optional) – Name of the timestamp column (the default is ‘TimeGenerated’)
value_col (str) – The column name holding the value to plot vertically
source_columns (list, optional) – List of default source columns to use in tooltips (the default is None)
x (str, optional) – alias of time_column
y (str, optional) – alias of value_col
title (str, optional) – Title to display (the default is None)
ref_event (Any, optional) – Add a reference line/label using the alert time (the default is None)
ref_time (datetime, optional) – Add a reference line/label using ref_time (the default is None)
ref_label (str, optional) – A label for the ref_event or ref_time reference item
group_by (str) – (where data is a DataFrame) The column to group timelines on
legend (str, optional) – “left”, “right”, “inline” or “none” (the default is to show a legend when plotting multiple series and not to show one when plotting a single series)
yaxis (bool, optional) – Whether to show the yaxis and labels
range_tool (bool, optional) – Show the the range slider tool (default is True)
height (int, optional) – The height of the plot figure (the default is auto-calculated height)
width (int, optional) – The width of the plot figure (the default is 900)
color (str) – Default series color (default is “navy”). This is overridden by automatic color assignments if plotting a grouped chart
kind (Union[str, List[str]]) – one or more glyph types to plot., optional Supported types are “circle”, “line” and “vbar” (default is “vbar”)
ref_events (pd.DataFrame, optional) – Add references line/label using the event times in the dataframe. (the default is None)
ref_time_col (str, optional) – Add references line/label using the this column in ref_events for the time value (x-axis). (this defaults the value of the time_column parameter or ‘TimeGenerated’ time_column is None)
ref_col (str, optional) – The column name to use for the label from ref_events (the default is None)
ref_times (List[Tuple[datetime, str]], optional) – Add one or more reference line/label using (the default is None)
- Returns
The bokeh plot figure.
- Return type
LayoutDOM
msticpy.nbtools.user_config module
User configuration functions.
Loads providers based on user_defaults section in msticpyconfig.yaml
UserDefaults:
# List of query providers to load
QueryProviders:
AzureSentinel:
Default: # name of the provider listed in AzureSentinel.Workspaces
alias: azsent # optional - create "qry_azsent" object in globals
CyberSoc:
alias: soc
connect: False # optional - do not connect on load
Splunk: # add non-sentinel providers like this
connect: False
LocalData: local
# List of other providers/components to load
LoadComponents:
TILookup: # No parameters
GeoIpLookup:
provider: GeoLiteLookup # geoip provider to use
Notebooklets: # Load and intialize Notebooklets
query_provider: # Pass it this query provider at startup
AzureSentinel:
workspace: CyberSoc
Pivot: # No parameters
AzureData: # auth_methods passed as startup param
auth_methods: ['cli','interactive']
AzureSentinelAPI:
auth_methods: ['env','interactive']
connect: False # Load but do not connect
Note: For components that require authentication the default is to connect after loading. You can skip the connect step by add connect: False to the entry.
- msticpy.nbtools.user_config.load_user_defaults() Dict[str, object]
Load providers from user defaults in msticpyconfig.yaml.
- Returns
Dict of object name and provider instances.
- Return type
Dict[str, object]