msticpy.context.tilookup module

Module for TILookup classes.

Input can be a single IoC observable or a pandas DataFrame containing multiple observables. Processing may require a an API key and processing performance may be limited to a specific number of requests per minute for the account type that you have.

class msticpy.context.tilookup.TILookup(providers: List[str] | None = None, **kwargs)

Bases: Lookup

Threat Intel observable lookup from providers.

Initialize TILookup instance.

Parameters:

  • primary_providers (Optional[List[Provider]], optional) – Primary Providers, by default None

  • secondary_providers (Optional[List[Provider]], optional) – Secondary Providers, by default None

  • providers (Optional[List[str]], optional) – List of provider names to load, by default all available providers are loaded. To see the list of available providers call TILookup.list_available_providers(). Note: if primary_provides or secondary_providers is specified This will override the providers list.

CUSTOM_PROVIDERS: Dict[str, Provider] = {}
PACKAGE: str = 'tiproviders'
PROVIDERS: Dict[str, Tuple[str, str]] = {'AzSTI': ('azure_sent_byoti', 'AzSTI'), 'GreyNoise': ('greynoise', 'GreyNoise'), 'IntSights': ('intsights', 'IntSights'), 'OPR': ('open_page_rank', 'OPR'), 'OTX': ('alienvault_otx', 'OTX'), 'Pulsedive': ('pulsedive', 'Pulsedive'), 'RiskIQ': ('riskiq', 'RiskIQ'), 'Tor': ('tor_exit_nodes', 'Tor'), 'VirusTotal': ('virustotal', 'VirusTotal'), 'XForce': ('ibm_xforce', 'XForce')}
add_provider(provider: Provider, name: str | None = None, primary: bool = True)

Add a provider to the current collection.

Parameters:

  • provider (TIProvider) – Provider instance

  • name (str, optional) – The name to use for the provider (overrides the class name of provider)

  • primary (bool, optional) – “primary” or “secondary” if False, by default “primary”

property available_providers: List[str]

Return a list of builtin and plugin providers.

Returns:

List of TI Provider classes.

Return type:

List[str]

classmethod browse(data: DataFrame, severities: List[str] | None = None, **kwargs)

Return TI Results list browser.

Parameters:

  • data (pd.DataFrame) – TI Results data from TIProviders

  • severities (Optional[List[str]], optional) – A list of the severity classes to show. By default these are [‘warning’, ‘high’]. Pass [‘information’, ‘warning’, ‘high’] to see all results.

  • kwargs – passed to SelectItem constructor.

Returns:

SelectItem browser for TI Data.

Return type:

SelectItem

classmethod browse_results(data: DataFrame, severities: List[str] | None = None, **kwargs)

Return TI Results list browser.

Parameters:

  • data (pd.DataFrame) – TI Results data from TIProviders

  • severities (Optional[List[str]], optional) – A list of the severity classes to show. By default these are [‘warning’, ‘high’]. Pass [‘information’, ‘warning’, ‘high’] to see all results.

  • kwargs – passed to SelectItem constructor.

Returns:

SelectItem browser for TI Data.

Return type:

SelectItem

property configured_providers: List[str]

Return a list of available providers that have configuration details present.

Returns:

List of TI Provider classes.

Return type:

List[str]

disable_provider(providers: str | Iterable[str])

Set the provider as secondary (not used by default).

Parameters:

providers (Union[str, Iterable[str]) – Provider name or list of names. Use list_available_providers() to see the list of loaded providers.

Raises:

ValueError – If the provider name is not recognized.

enable_provider(providers: str | Iterable[str])

Set the provider(s) as primary (used by default).

Parameters:

providers (Union[str, Iterable[str]) – Provider name or list of names. Use list_available_providers() to see the list of loaded providers.

Raises:

ValueError – If the provider name is not recognized.

classmethod import_provider(provider: str) → Provider

Import provider class.

classmethod list_available_providers(show_query_types=False, as_list: bool = False) → List[str] | None

Print a list of builtin providers with optional usage.

Parameters:

  • show_query_types (bool, optional) – Show query types supported by providers, by default False

  • as_list (bool, optional) – Return list of providers instead of printing to stdout. Note: if you specify show_query_types this will be printed irrespective of this parameter setting.

Returns:

A list of provider names (if return_list=True)

Return type:

Optional[List[str]]

property loaded_providers: Dict[str, Provider]

Return dictionary of loaded providers.

Returns:

[description]

Return type:

Dict[str, TIProvider]

lookup_ioc(ioc: str | None = None, ioc_type: str | None = None, ioc_query_type: str | None = None, providers: List[str] | None = None, default_providers: List[str] | None = None, prov_scope: str = 'primary', **kwargs) → DataFrame

Lookup single IoC in active providers.

Parameters:

  • ioc (str) – IoC observable

  • observable (str) – alias for ioc

  • ioc_type (str, optional) – One of IoCExtract.IoCType, by default None If none, the IoC type will be inferred

  • ioc_query_type (str, optional) – The ioc query type (e.g. rep, info, malware)

  • providers (List[str]) – Explicit list of providers to use

  • default_providers (Optional[List[str]], optional) – Used by pivot functions as a fallback to providers. If providers is specified, it will override this parameter.

  • prov_scope (str, optional) – Use “primary”, “secondary” or “all” providers, by default “primary”

  • kwargs – Additional arguments passed to the underlying provider(s)

Returns:

The result returned as a tuple(bool, list): bool indicates whether a TI record was found in any provider list has an entry for each provider result

Return type:

pd.DataFrame

lookup_iocs(data: DataFrame | Mapping[str, str] | Iterable[str], ioc_col: str | None = None, ioc_type_col: str | None = None, ioc_query_type: str | None = None, providers: List[str] | None = None, default_providers: List[str] | None = None, prov_scope: str = 'primary', **kwargs) → DataFrame

Lookup a collection of IoCs.

Parameters:

  • data (Union[pd.DataFrame, Mapping[str, str], Iterable[str]]) – Data input in one of three formats: 1. Pandas dataframe (you must supply the column name in ioc_col parameter) 2. Mapping (e.g. a dict) of [observable, IoCType] 3. Iterable of observables - IoCTypes will be inferred

  • ioc_col (str, optional) – DataFrame column to use for observables, by default None (“col” and “column” are also aliases for this parameter)

  • ioc_type_col (str, optional) – DataFrame column to use for IoCTypes, by default None

  • ioc_query_type (str, optional) – The ioc query type (e.g. rep, info, malware)

  • providers (List[str]) – Explicit list of providers to use

  • default_providers (Optional[List[str]], optional) – Used by pivot functions as a fallback to providers. If providers is specified, it will override this parameter.

  • prov_scope (str, optional) – Use “primary”, “secondary” or “all” providers, by default “primary”

  • kwargs – Additional arguments passed to the underlying provider(s)

Returns:

DataFrame of results

Return type:

pd.DataFrame

lookup_iocs_sync(data: DataFrame | Mapping[str, str] | Iterable[str], ioc_col: str | None = None, ioc_type_col: str | None = None, ioc_query_type: str | None = None, providers: List[str] | None = None, default_providers: List[str] | None = None, prov_scope: str = 'primary', **kwargs) → DataFrame

Lookup a collection of IoCs.

Parameters:

  • data (Union[pd.DataFrame, Mapping[str, str], Iterable[str]]) – Data input in one of three formats: 1. Pandas dataframe (you must supply the column name in ioc_col parameter) 2. Mapping (e.g. a dict) of [observable, IoCType] 3. Iterable of observables - IoCTypes will be inferred

  • ioc_col (str, optional) – DataFrame column to use for observables, by default None (“col” and “column” are also aliases for this parameter)

  • ioc_type_col (str, optional) – DataFrame column to use for IoCTypes, by default None

  • ioc_query_type (str, optional) – The ioc query type (e.g. rep, info, malware)

  • providers (List[str]) – Explicit list of providers to use

  • default_providers (Optional[List[str]], optional) – Used by pivot functions as a fallback to providers. If providers is specified, it will override this parameter.

  • prov_scope (str, optional) – Use “primary”, “secondary” or “all” providers, by default “primary”

  • kwargs – Additional arguments passed to the underlying provider(s)

Returns:

DataFrame of results

Return type:

pd.DataFrame

lookup_item(item: str, item_type: str | None = None, query_type: str | None = None, providers: List[str] | None = None, default_providers: List[str] | None = None, prov_scope: str = 'primary', **kwargs) → DataFrame

Lookup single item in active providers.

Parameters:

  • item (str) – item to lookup

  • item_type (str, optional) – One of ItemExtract.ItemType, by default None If none, the Item type will be inferred

  • query_type (str, optional) – The query type (e.g. rep, info, malware)

  • providers (List[str]) – Explicit list of providers to use

  • default_providers (Optional[List[str]], optional) – Used by pivot functions as a fallback to providers. If providers is specified, it will override this parameter.

  • prov_scope (str, optional) – Use “primary”, “secondary” or “all” providers, by default “primary”

  • kwargs – Additional arguments passed to the underlying provider(s)

Returns:

DataFrame of results

Return type:

pd.DataFrame

lookup_items(data: DataFrame | Mapping[str, str] | Iterable[str], item_col: str | None = None, item_type_col: str | None = None, query_type: str | None = None, providers: List[str] | None = None, default_providers: List[str] | None = None, prov_scope: str = 'primary', **kwargs) → DataFrame

Lookup a collection of items.

Parameters:

  • data (Union[pd.DataFrame, Mapping[str, str], Iterable[str]]) – Data input in one of three formats: 1. Pandas dataframe (you must supply the column name in item_col parameter) 2. Mapping (e.g. a dict) of [item, ItemType] 3. Iterable of items - ItemTypes will be inferred

  • item_col (str, optional) – DataFrame column to use for items, by default None (“col” and “column” are also aliases for this parameter)

  • item_type_col (str, optional) – DataFrame column to use for ItemTypes, by default None

  • query_type (str, optional) – The item query type (e.g. rep, info, malware)

  • providers (List[str]) – Explicit list of providers to use

  • default_providers (Optional[List[str]], optional) – Used by pivot functions as a fallback to providers. If providers is specified, it will override this parameter.

  • prov_scope (str, optional) – Use “primary”, “secondary” or “all” providers, by default “primary”

  • progress (bool) – Use progress bar to track completion, by default True

  • kwargs – Additional arguments passed to the underlying provider(s)

Returns:

DataFrame of results

Return type:

pd.DataFrame

lookup_items_sync(data: DataFrame | Mapping[str, str] | Iterable[str], item_col: str | None = None, item_type_col: str | None = None, query_type: str | None = None, providers: List[str] | None = None, default_providers: List[str] | None = None, prov_scope: str = 'primary', **kwargs) → DataFrame

Lookup a collection of items.

Parameters:

  • data (Union[pd.DataFrame, Mapping[str, str], Iterable[str]]) – Data input in one of three formats: 1. Pandas dataframe (you must supply the column name in item_col parameter) 2. Mapping (e.g. a dict) of [item, ItemType] 3. Iterable of items - ItemTypes will be inferred

  • item_col (str, optional) – DataFrame column to use for items, by default None (“col” and “column” are also aliases for this parameter)

  • item_type_col (str, optional) – DataFrame column to use for ItemTypes, by default None

  • query_type (str, optional) – The item query type (e.g. rep, info, malware)

  • providers (List[str]) – Explicit list of providers to use

  • default_providers (Optional[List[str]], optional) – Used by pivot functions as a fallback to providers. If providers is specified, it will override this parameter.

  • prov_scope (str, optional) – Use “primary”, “secondary” or “all” providers, by default “primary”

  • kwargs – Additional arguments passed to the underlying provider(s)

Returns:

DataFrame of results

Return type:

pd.DataFrame

property provider_status: Iterable[str]

Return loaded provider status.

Returns:

List of providers and descriptions.

Return type:

Iterable[str]

provider_usage()

Print usage of loaded providers.

classmethod reload_provider_settings()

Reload provider settings from config.

reload_providers()

Reload settings and provider classes.

static result_to_df(item_lookup: DataFrame) → DataFrame

Return DataFrame representation of Lookup response.

Parameters:

item_lookup (pd.DataFrame) – Output from lookup_item

Returns:

The response as a DataFrame with a row for each provider response.

Return type:

pd.DataFrame

set_provider_state(prov_dict: Dict[str, bool])

Set a dict of providers to primary/secondary.

Parameters:

prov_dict (Dict[str, bool]) – Dictionary of provider name and bool - True if enabled/primary, False if disabled/secondary.