msticpy.common package

msticpy.common.utility module

Miscellaneous helper methods for Jupyter Notebooks.

msticpy.common.utility.check_and_install_missing_packages(required_packages: List[str], force_notebook: bool = False, user: bool = True, upgrade: bool = False) → bool

Check and install missing packages from provided list of packages.

Parameters:
  • required_packages (List[str]) – List of packages to check and install in a current environment Note you can add package version constraints by appending them to the package name, e.g. pandas>=1.01
  • force_notebook (bool, optional) – Boolean value to force notebook version of progress bar, by default False (autodetect)
  • user (bool, optional) – Boolean value to toggle user flag while installing pip packages, by default True
  • upgrade (bool, option) – If true supply –upgrade flag to pip to install the latest version (applies to all package in required_packages)
Returns:

True if successful, else False

Return type:

bool

msticpy.common.utility.check_kwarg(arg_name: str, legal_args: List[str])

Check argument names against a list.

Parameters:
  • arg_name (str) – Argument to check
  • legal_args (List[str]) – List of possible arguments.
Raises:

NameError – If the argument is not legal. If the arg_name is a close match to one or more, legal_args these are returned in the exception.

msticpy.common.utility.check_kwargs(supplied_args: Dict[str, Any], legal_args: List[str])

Check all kwargs names against a list.

Parameters:
  • supplied_args (Dict[str, Any]) – Arguments to check
  • legal_args (List[str]) – List of possible arguments.
Raises:

NameError – If any of the arguments are not legal. If the an arg is a close match to one or more legal_args, these are returned in the exception.

msticpy.common.utility.check_py_version(min_ver: Tuple = (3, 6))

Check that the current python version is not less than min_ver.

Parameters:min_ver (Tuple, optional) – Minimum required version, by default (3,6)
msticpy.common.utility.enable_toggle_code()

Load JS Function to enable code toggle button.

msticpy.common.utility.escape_windows_path(str_path: str) → str

Escape backslash characters in a string.

msticpy.common.utility.export(func: Callable)

Decorate function or class to export to __all__.

msticpy.common.utility.get_nb_query_param(nb_url_search: str, param: str) → Optional[str]

Get a url query parameter from the search string.

Parameters:
  • nb_url_search (str) – The URL search string
  • param (str) – The parameter name to search for
Returns:

value of the query string parameter or None if not found.

Return type:

Optional[str]

Deprecated since version 0.3.2: Inline Javascript no longer supported

msticpy.common.utility.get_nb_query_params(nb_url_search: str) → dict

Get the url query parameters from the search string.

Parameters:nb_url_search (str) – The URL search string
Returns:dictionary of the query string parameters.
Return type:dict

Deprecated since version 0.3.2: Inline Javascript no longer supported

msticpy.common.utility.get_notebook_query_string()

Execute javascript to publish notebook query string as python variable.

Deprecated since version 0.3.2: Inline Javascript no longer supported

msticpy.common.utility.is_ipython() → bool

Return True if running in IPython environment.

Returns:True if running in IPython environment, otherwise False
Return type:bool
msticpy.common.utility.is_not_empty(test_object: Any) → bool

Return True if the test_object is not None or empty.

msticpy.common.utility.is_valid_uuid(uuid_str: Any) → bool

Return true if uuid_str is a value GUID/UUID.

Parameters:uuid_str (Any) – String to test
Returns:True if valid GUID/UUID.
Return type:bool
msticpy.common.utility.md(string: str, styles: Union[str, Iterable[str]] = None)

Return string as Markdown with optional style.

Parameters:
  • string (str) – The string to display
  • styles (Union[str, Iterable[str]], optional) – A style mnemonic or collection of styles. If multiple styles, these can be supplied as an interable of strings or a comma-separated string, by default None
msticpy.common.utility.md_error(string: str)

Return string as an error - red text prefixed by “Error”.

Parameters:string (str) – The error message.
msticpy.common.utility.md_warn(string: str)

Return string as a warning - orange text prefixed by “Warning”.

Parameters:string (str) – The warning message.
msticpy.common.utility.resolve_pkg_path(part_path: str)

Resolve a path relative to the package.

Parameters:part_path (str) – Absolute or relative path to resolve.
msticpy.common.utility.set_unit_testing(on: bool = True)

Set flag env var to indicated that code is being unit-tested.

Parameters:on (bool, optional) – Turn unit testing flag on or off, by default True
msticpy.common.utility.string_empty(string: str) → bool

Return True if the input string is None or whitespace.

msticpy.common.utility.toggle_code()

Display a toggle button to hide/reveal code cell.

msticpy.common.utility.unescape_windows_path(str_path: str) → str

Remove escaping from backslash characters in a string.

msticpy.common.utility.unit_testing() → bool

Return True if in unit testing.

Returns:True if in unit testing
Return type:bool

msticpy.common.wsconfig module

Module for Log Analytics-related configuration.

class msticpy.common.wsconfig.WorkspaceConfig(config_file: Optional[str] = None, workspace: str = None)

Bases: object

Workspace configuration class.

Load current Azure Notebooks configuration for Log Analytics.

Parameters:
  • config_file (Optional[str], optional) – path to a configuration file, Defaults to msticpyconfig.yaml if settings are configured there. If not, looks for a config.json in current folder
  • workspace (str, optional) – Workspace name (where multiple workspaces are configured), by default the Default workspace will be used.
CONF_RES_GROUP_KEY = 'resource_group'
CONF_SUB_ID_KEY = 'subscription_id'
CONF_TENANT_ID_KEY = 'tenant_id'
CONF_WS_ID_KEY = 'workspace_id'
CONF_WS_NAME_KEY = 'workspace_name'
PKG_CONF_TENANT_KEY = 'TenantId'
PKG_CONF_WS_KEY = 'WorkspaceId'
RESOURCE_GROUP = '{{cookiecutter.resource_group}}'
SUBSCRIPTION_ID = '{{cookiecutter.subscription_id}}'
TENANT_ID = '{{cookiecutter.tenant_id}}'
WORKSPACE_ID = '{{cookiecutter.workspace_id}}'
WORKSPACE_NAME = '{{cookiecutter.workspace_name}}'
code_connect_str

Return the Log Analytics connection string for dev code auth.

Returns:Connection string
Return type:str
config_loaded

Return True if workspace id and tenant id have values.

Returns:True if configuration loaded.
Return type:bool
classmethod list_workspaces() → Dict[KT, VT]

Return list of available workspaces.

Returns:Dictionary of workspaces with workspace and tenantIds.
Return type:Dict

msticpy.common.keyvault_client module

Keyvault client - adapted from Bluehound code.

class msticpy.common.keyvault_client.AuthClient(tenant_id: str, client_id: str, client_uri: str, name: str = None, **kwargs)

Bases: object

Authentication class base.

Initialize base authentication client for credential caching.

Parameters:
  • tenant_id (str) – Tenant ID of Azure User
  • client_id (str) – Client ID of application client
  • client_uri (str) – [description]
  • name (str, optional) – Name of the secret store, by default None
  • authority (str, optional) – The AAD authority - one of ‘global’, ‘usgov’, ‘de’ or ‘chi’
  • authority_uri (str, optional) – The AAD authority URI - overrides authority
  • debug (bool, optional) – Output debug information if True, by default False

Notes

The parameter values can also be obtained from the KeyVault section of msticpyconfig.yaml.

auth_id

Return name or ID of client.

token

Return the access token.

Returns:Access Token
Return type:str
user_oid

Return the user Object ID.

Returns:User OID.
Return type:str
class msticpy.common.keyvault_client.BHKeyVaultClient(tenant_id: str = None, vault_uri: str = None, vault_name: str = None, settings: msticpy.common.keyvault_client.KeyVaultSettings = None, **kwargs)

Bases: object

Core KeyVault client.

Initialize the BHKeyVault client.

Parameters:
  • tenant_id (str) – The tenant ID of the service
  • vault_uri (str, optional) – The full URI of the keyvault, by default None
  • vault_name (str, optional) – The name of the keyvault in the public cloud, by default None
  • authn_type (str, optional) – Authentication mode, by default ‘interactive’ Supported options are: - ‘device’ for device code authentication - ‘interactive’ for interactive browser authentication
  • authority (str, optional) – The AAD authority - one of ‘global’, ‘usgov’, ‘de’ or ‘chi’
  • authority_uri (str, optional) – The AAD authority URI - overrides authority
  • settings (KeyVaultSettings) – An instance of KeyVaultSettings containing KV parameters.
  • debug (bool, optional) – [description], by default False
Raises:

KeyVaultMissingVaultException – No Vault name or URI supplied.

Notes

The parameter values can also be obtained from the KeyVault section of msticpyconfig.yaml.

get_secret(secret_name: str) → Any

Retrieve a secret from the Vault.

Parameters:secret_name (str) – Name of the secret
Returns:The secret value
Return type:Any
Raises:KeyVaultMissingSecretException – Secret not found in the Vault.
secrets

Return the list of secret names from the vault.

set_secret(secret_name: str, value: Any) → azure.keyvault.secrets._models.KeyVaultSecret

Set a secret in the Vault.

Parameters:
  • secret_name (str) – Name of the secret
  • value (Any) – Secret value
Returns:

The secrets bundle for the secret

Return type:

KeyVaultSecret

class msticpy.common.keyvault_client.BHKeyVaultMgmtClient(tenant_id: str = None, subscription_id: str = None, resource_group: str = None, azure_region: str = None, settings: msticpy.common.keyvault_client.KeyVaultSettings = None, **kwargs)

Bases: object

Core KeyVault Management client.

Initialize BH KeyVault Management Client.

Parameters:
  • tenant_id (str, Optional) – Tenant ID
  • subscription_id (str, Optional) – Subscription ID
  • resource_group (str, Optional) – Resource Group name
  • azure_region (str, Optional) – Azure region - needed to create a new vault. By default, None
  • settings (KeyVaultSettings) – An instance of KeyVaultSettings containing KV parameters.
  • mgmt_uri (str, Optional) – The URI for Azure management endpoints.

Notes

The parameter values can also be obtained from the KeyVault section of msticpyconfig.yaml.

create_vault(vault_name: str) → azure.mgmt.keyvault.v2019_09_01.models._models_py3.Vault

Create new or update existing vault.

Parameters:vault_name (str) – Name of the Vault
Returns:The Vault object.
Return type:Vault
get_vault_uri(vault_name: str) → str

Return the URI for a vault name.

Parameters:vault_name (str) – The Vault name.
Returns:Vault URI.
Return type:str
list_vaults() → List[str]

Return a list of vaults for the subscription.

Returns:Vault names
Return type:List[str]
class msticpy.common.keyvault_client.KeyVaultSettings

Bases: object

KeyVaultSettings class - reads settings from msticpyconfig.

Notes

The KeyVault section in msticpyconfig.yaml can contain the following:

KeyVault:
    TenantId: {tenantid-to-use-for-authentication}
    SubscriptionId: {subscriptionid-containing-vault}
    ResourceGroup: {resource-group-containing-vault}
    AzureRegion: {region-for-vault}
    VaultName: {vault-name}
    UseKeyring: True
    Authority: global

SubscriptionId, ResourceGroup and AzureRegion are only used when creating new vaults. UseKeyring instructs the SecretsClient to cache Keyvault secrets locally using Python keyring. Authority is one of ‘global’, ‘usgov’, ‘de’, ‘chi’ Alternatively, you can specify AuthorityURI with the value pointing to the URI for logon requests.

Initialize new instance of KeyVault Settings.

AAD_AUTHORITIES = {'chi': 'https://login.chinacloudapi.cn', 'de': 'https://login.microsoftonline.de', 'global': 'https://login.microsoftonline.com/', 'usgov': 'https://login.microsoftonline.us'}
CLIENT_ID = '04b07795-8ddb-461a-bbee-02f9e1bf7b46'
KV_URIS = {'chi': None, 'de': None, 'global': 'https://{vault}.vault.azure.net/', 'usgov': 'https://{vault}.vault.usgovcloudapi.net/'}
MGMT_URIS = {'chi': None, 'de': None, 'global': 'https://management.azure.com/', 'usgov': 'https://management.usgovcloudapi.net/'}
authority_uri

Return authority URI for cloud.

Returns:Authority URI
Return type:str
cloud

Return the cloud for the settings.

get(key: str, default: Any = None) → Any

Return dict value or default.

get_tenant_authority_host(authority_uri: str = None, tenant: str = None) → str

Return tenant authority URI with no leading scheme.

Parameters:
  • authority_uri (str, optional) – The authority URI - otherwise try to retrieve from settings
  • tenant (str, optional) – TenantID or name, by default None. If not passed as a parameter try to get tenant from KeyVault configuration in msticpyconfig.yaml
Returns:

Tenant Authority

Return type:

str

Raises:

KeyVaultConfigException – If tenant is not defined.

get_tenant_authority_uri(authority_uri: str = None, tenant: str = None) → str

Return authority URI for tenant.

Parameters:
  • authority_uri (str, optional) – The authority URI - otherwise try to retrieve from settings
  • tenant (str, optional) – TenantID or name, by default None. If not passed as a parameter try to get tenant from KeyVault configuration in msticpyconfig.yaml
Returns:

Tenant Authority

Return type:

str

Raises:

KeyVaultConfigException – If tenant is not defined.

keyvault_uri

Return KeyVault URI template for current cloud.

mgmt_uri

Return Azure management URI template for current cloud.

class msticpy.common.keyvault_client.KeyringAuthClient(tenant_id: str, client_id: str, client_url: str, name: str = None, debug: bool = False)

Bases: msticpy.common.keyvault_client.AuthClient

Key Authentication Client.

Handles management of authentication and refresh tokens via keyring

Initialize KeyringAuthClient.

Parameters:
  • tenant_id (str) – Tenant ID of Azure User
  • client_id (str) – Client ID of application client
  • client_url (str) – [description]
  • name (str, optional) – Name of the secret store, by default None
  • debug (bool, optional) – Output debug information if True, by default False
auth_id

Return name or ID of client.

token

Return the access token.

Returns:Access Token
Return type:str
user_oid

Return the user Object ID.

Returns:User OID.
Return type:str

msticpy.common.secret_settings module

Settings provider for secrets.

class msticpy.common.secret_settings.KeyringClient(name: str = 'key-cache', debug: bool = False)

Bases: object

Keyring client wrapper.

Initialize the keyring client.

Parameters:
  • name (str, optional) – Name of the credential group, by default “system”
  • debug (bool, optional) – Output debug info, by default False
get_secret(secret_name: str) → Any

Retrieve a secret from the keyring.

Parameters:secret_name (str) – Secret name.
Returns:Secret value.
Return type:Any
set_secret(secret_name: str, secret_value: Any)

Set a secret in the keyring group.

Parameters:
  • secret_name (str) – Name of the secret
  • secret_value (Any) – Secret value
class msticpy.common.secret_settings.SecretsClient(tenant_id: str = None, use_keyring: bool = True)

Bases: object

Secrets client - manages keyvault and keyring secrets.

Initialize SecretsClient instance.

Parameters:
  • tenant_id (str, optional) – TenantID, by default None
  • use_keyring (bool, optional) – If True use keyring to cache secrets, by default True
Raises:

MsticpyKeyVaultConfigError – Missing or invalid configuration settings.

Notes

Requires KeyVault settings to be defined in msticpyconfig.yaml

static format_kv_name(setting_path)

Return normalized name for use as a KeyVault secret name.

get_secret_accessor(setting_path: str) → Callable[[], Any]

Return accessor function for a secret.

Parameters:setting_path (str) – The msticpy configuration path (dot-separated)
Returns:Accessor function for the secret value.
Return type:Callable[[None], Any]
static read_secret(secret_object: Any) → Any

Return the secret value.

Parameters:secret_object (Any) – If it is a func, call and return the return value of that func. Otherwise just return the object.
Returns:The secret value
Return type:Any

msticpy.sectools.provider_settings module

Helper functions for configuration settings.

class msticpy.common.provider_settings.ProviderArgs(**kwargs)

Bases: collections.UserDict

ProviderArgs dictionary.

clear() → None. Remove all items from D.
copy()
classmethod fromkeys(iterable, value=None)
get(k[, d]) → D[k] if k in D, else d. d defaults to None.
items() → a set-like object providing a view on D's items
keys() → a set-like object providing a view on D's keys
pop(k[, d]) → v, remove specified key and return the corresponding value.

If key is not found, d is returned if given, otherwise KeyError is raised.

popitem() → (k, v), remove and return some (key, value) pair

as a 2-tuple; but raise KeyError if D is empty.

setdefault(k[, d]) → D.get(k,d), also set D[k]=d if k not in D
update([E, ]**F) → None. Update D from mapping/iterable E and F.

If E present and has a .keys() method, does: for k in E: D[k] = E[k] If E present and lacks .keys() method, does: for (k, v) in E: D[k] = v In either case, this is followed by: for k, v in F.items(): D[k] = v

values() → an object providing a view on D's values
class msticpy.common.provider_settings.ProviderSettings(name: str, description: str, provider: Optional[str] = None, args: msticpy.common.provider_settings.ProviderArgs = NOTHING, primary: bool = False)

Bases: object

Provider settings.

msticpy.common.provider_settings.get_provider_settings(config_section='TIProviders') → Dict[str, msticpy.common.provider_settings.ProviderSettings]

Read Provider settings from package config.

Parameters:config_section (str, optional) – [description], by default “TIProviders”
Returns:Provider settings indexed by provider name.
Return type:Dict[str, ProviderSettings]
msticpy.common.provider_settings.reload_settings()

Reload settings from config files.

Parameters:clear_keyring (bool, optional) – Clears any secrets cached in keyring, by default False

msticpy.sectools.exceptions module

Miscellaneous helper methods for Jupyter Notebooks.

exception msticpy.common.exceptions.MsticpyAzureConfigError(*args, help_uri: Union[Tuple[str, str], str, None] = None, **kwargs)

Bases: msticpy.common.exceptions.MsticpyUserConfigError

Exception class for AzureData.

Create Azure data missing configuration exception.

Parameters:help_uri (Union[Tuple[str, str], str, None], optional) – Override the default help URI.
DEF_HELP_URI = ('Using the Azure API connector', 'https://msticpy.readthedocs.io/data_acquisition/AzureData.html#instantiating-and-connecting-with-an-azure-data-connector')
args
help_uri

Get the default help URI.

with_traceback()

Exception.with_traceback(tb) – set self.__traceback__ to tb and return self.

exception msticpy.common.exceptions.MsticpyConfigException

Bases: msticpy.common.exceptions.MsticpyException

Configuration exception class for msticpy.

args
with_traceback()

Exception.with_traceback(tb) – set self.__traceback__ to tb and return self.

exception msticpy.common.exceptions.MsticpyException

Bases: Exception

Default exception class for msticpy.

args
with_traceback()

Exception.with_traceback(tb) – set self.__traceback__ to tb and return self.

exception msticpy.common.exceptions.MsticpyKeyVaultConfigError(*args, help_uri: Union[Tuple[str, str], str, None] = None, **kwargs)

Bases: msticpy.common.exceptions.MsticpyUserConfigError

Key Vault configuration exception.

Create Key Vault configuration exception.

Parameters:help_uri (Union[Tuple[str, str], str, None], optional) – Override the default help URI.
DEF_HELP_URI = ('Using keyvault to store msticpy secrets', 'https://msticpy.readthedocs.io/getting_started/msticpyconfig.html#specifying-secrets-as-key-vault-secrets')
args
help_uri

Get the default help URI.

with_traceback()

Exception.with_traceback(tb) – set self.__traceback__ to tb and return self.

exception msticpy.common.exceptions.MsticpyKeyVaultMissingSecretError(*args, help_uri: Union[Tuple[str, str], str, None] = None, **kwargs)

Bases: msticpy.common.exceptions.MsticpyKeyVaultConfigError

Missing secret exception.

Create Key Vault missing key exception.

Parameters:help_uri (Union[Tuple[str, str], str, None], optional) – Override the default help URI.
DEF_HELP_URI = ('Using keyvault to store msticpy secrets', 'https://msticpy.readthedocs.io/getting_started/msticpyconfig.html#specifying-secrets-as-key-vault-secrets')
args
help_uri

Get the default help URI.

with_traceback()

Exception.with_traceback(tb) – set self.__traceback__ to tb and return self.

exception msticpy.common.exceptions.MsticpyKqlConnectionError(*args, help_uri: Union[Tuple[str, str], str, None] = None, **kwargs)

Bases: msticpy.common.exceptions.MsticpyUserError

Exception class for KqlConnection errors.

Create MsticpyKqlConnectionError exception.

Parameters:help_uri (Union[Tuple[str, str], str, None], optional) – Override the default help URI.
DEF_HELP_URI = ('Connecting to Azure Sentinel', 'https://msticpy.readthedocs.io/DataProviders.html#connecting-to-an-azure-sentinel-workspace')
args
help_uri

Get the default help URI.

with_traceback()

Exception.with_traceback(tb) – set self.__traceback__ to tb and return self.

exception msticpy.common.exceptions.MsticpyNoDataSourceError(*args, help_uri: Union[Tuple[str, str], str, None] = None, **kwargs)

Bases: msticpy.common.exceptions.MsticpyUserError

Exception class for missing data source errors.

Create MsticpyNoDataSourceError exception.

Parameters:help_uri (Union[Tuple[str, str], str, None], optional) – Override the default help URI.
DEF_HELP_URI = ('Querying and importing data', 'https://msticpy.readthedocs.io/DataAcquisition.html#querying-and-importing-data')
args
help_uri

Get the default help URI.

with_traceback()

Exception.with_traceback(tb) – set self.__traceback__ to tb and return self.

exception msticpy.common.exceptions.MsticpyNotConnectedError(*args, help_uri: Union[Tuple[str, str], str, None] = None, **kwargs)

Bases: msticpy.common.exceptions.MsticpyUserError

Exception class for NotConnected errors.

Create Not connected exception.

Parameters:help_uri (Union[Tuple[str, str], str, None], optional) – Override the default help URI.
DEF_HELP_URI = ('Querying and importing data', 'https://msticpy.readthedocs.io/DataAcquisition.html#querying-and-importing-data')
args
help_uri

Get the default help URI.

with_traceback()

Exception.with_traceback(tb) – set self.__traceback__ to tb and return self.

exception msticpy.common.exceptions.MsticpyResourceException

Bases: msticpy.common.exceptions.MsticpyException

Exception class for resource errors.

args
with_traceback()

Exception.with_traceback(tb) – set self.__traceback__ to tb and return self.

exception msticpy.common.exceptions.MsticpyUserConfigError(*args, help_uri: Union[Tuple[str, str], str, None] = None, **kwargs)

Bases: msticpy.common.exceptions.MsticpyUserError

Configuration user exception class for msticpy.

Create generic user configuration exception.

Parameters:help_uri (Union[Tuple[str, str], str, None], optional) – Override the default help URI.
DEF_HELP_URI = ('Configuring msticpy', 'https://msticpy.readthedocs.io/getting_started/msticpyconfig.html')
args
help_uri

Get the default help URI.

with_traceback()

Exception.with_traceback(tb) – set self.__traceback__ to tb and return self.

exception msticpy.common.exceptions.MsticpyUserError(*args, help_uri: Union[Tuple[str, str], str, None] = None, **kwargs)

Bases: msticpy.common.exceptions.MsticpyException

Msticpy User exception displaying friendly message.

Create an instance of the MsticpyUserError class.

Parameters:
  • args (Iterable of strings) – Args will be printed as text of the exception.
  • help_uri (Union[Tuple[str, str], str, None], optional) – Primary URL, by default “https://msticpy.readthedocs.org
Other Parameters:
 
  • title (str) – If a title keyword argument is supplied it will be used to create the title line.
  • *_uri (str) – Additional keyword arguments who’s names end in “_uri” will be used to create a list of references in addition to the primary help_uri

Notes

The exception text is displayed when the exception is created and not when it is raised. We recommend creating the exception within the raise statement. E.g.

raise MsticpyUserException(arg1, arg2…)

Developer note: Any classes derived from MsticpyUserError should be named with an “Error” suffix to distinguish these from standard exception types.

DEF_HELP_URI = ('msticpy documentation', 'https://msticpy.readthedocs.org')
args
help_uri

Get the default help URI.

with_traceback()

Exception.with_traceback(tb) – set self.__traceback__ to tb and return self.

msticpy.common.exceptions.is_ipython() → bool

Return True if running in IPython environment.

Returns:True if running in IPython environment, otherwise False
Return type:bool