MSTICPy Package Configuration

Some elements of MSTICPy require configuration parameters. An example is the Threat Intelligence providers. Values for these and other parameters can be set in the msticpyconfig.yaml file.

The package has a default configuration file, which is stored in the package directory. You should not need to edit this file directly. Instead you can create a custom file with your own parameters - these settings will combine with or override the settings in the default file.

By default, the custom msticpyconfig.yaml is read from the current directory. You can specify an explicit location using an environment variable MSTICPYCONFIG.

You should also read the MSTICPy Settings Editor document to see how to configure settings using and interactive User Interface from a Jupyter notebook.

Configuration sections

AzureSentinel

Here you can specify your default workspace and tenant IDs and add additional workspaces if needed.

QueryDefinitions

This allows you to specify paths to additional yaml query template files.

TIProviders

This allows you to configure which providers are run by default and to supply any authorization keys needed to access the service.

OtherProviders

This section is similar to the TIProviders section, allowing you specify configuration options for specialist data providers.

DataProviders

This section is similar to the previous two sections, allowing you specify configuration options for other data providers.

Key Vault

This section contains Azure Key Vault settings. This is only used if you choose to store secrets (e.g. API keys) in Key Vault.

User Defaults

This section controls loading of default providers when using the package in a notebook. The settings here are loaded by the init_notebook function.

Specifying secrets as Environment Variables

Some configuration values can be references to environment Variables rather than have a value explicitly stored within the configuration file. You might want secrets such as API keys to be supplied this way. The Args subsection of TIProvider and OtherProvider entries supports storing values as simple strings or as references to named environment variables. You can see examples of both in the sample file below.

Specifying secrets as Key Vault secrets

MSTICPy can read secret values from Key Vault for use with TI and other providers. To use this you need to specify settings for your keyvault.

KeyVault:
  TenantId: 5d6a50cf-b1b6-4bfd-ad54-b9822b06ff92
  SubscriptionId: 40dcc8bf-0478-4f3b-b275-ed0a94f2c013
  ResourceGroup: YourResourceGroup
  AzureRegion: RegionToCreateKV
  VaultName: "myvault"
  UseKeyring: True
  Authority: global
  AuthnType: device

Under the top level KeyVault section the following entries are accepted. Some of these are only required if you plan to use MSTICPy to create a new Key Vault vault.

Required Settings

TenantId

the Identifier of your Azure tenant

VaultName

the name of the vault to use (note this can be overridden in the individual secret specifications

Authority

this specifies the Azure cloud instance to use.

For most users Authority is “global” (default). Other values are: - usgov: Azure US Government - de: German cloud - chi: China cloud

Required to Create a Key Vault

SubscriptionId

the Azure subscription holding the Key Vault

ResourceGroup

the Azure resource group in which to create the vault

AzureRegion

the Azure region in which to create the vault

Optional Settings

UseKeyring

if True (default) uses the Python keyring package to securely cache Key Vault secrets in your client session.

VaultName

the name of the vault to use (note this can be overridden in the individual secret specifications)

AuthnType

this governs the authentication type used by the KeyVault client (to read and write secrets). The choices are “interactive” for interactive browser authentication or “device” for authentication using a user/device code. The default is “interactive”.

Note

The AuthnType does not affect the authentication used by the Key Vault Management client (the creation and enumeration of vaults). This always uses device code authentication.

Specifying Key Vault Secrets in Provider Settings

There are three ways to specify the Key Vault vault and secret names to use for a given setting.

TIProviders:

  OpenPageRank:
    Args:
      AuthKey:
        KeyVault:

Adding an empty subkey named KeyVault will cause MSTICPy to generate a name for the secret, built from the path of the setting. This is the default usage. In the example below, the secret name will be “TIProviders-OpenPageRank-Args-AuthKey”. The vault name is taken from the setting in the KeyVault settings section.

OtherProviders:
  IPStack:
    Args:
      AuthKey:
        KeyVault: my_secret

This example specifies “my_secret” as the secret name. The vault name is taken from the setting in the KeyVault settings section.

OtherProviders:
  Contoso-GeopIp:
    Args:
      AuthKey:
        KeyVault: my_vault/my_secret

The final example specifies both a vault name and a secret name. The VaultName setting in the KeyVault section is ignored for this setting.

Populating Key Vault secrets from an existing msticpyconfig.yaml

You can use the MSTICPy settings editor to upload secrets to a Key Vault one-by-one. This is described in in the in the Key Vault Secrets section of MSTICPy Settings Editor document.

There is also a command line tool to move your secrets to Key Vault - config2ky.py. This tool is available in the MSTICPy GitHub repo. You can find it in the tools folder.

Running config2kv.py --help shows the usage of this utility.

The simplest way to use this tool is to populate your existing secrets as strings in your msticpyconfig.yaml. (as shown in some of the provider settings in the example at the end of this page).

You can specify this as the input file using the --path parameter. Alternatively, the tool will look for a msticpyconfig.yaml in the location specified by the MSTICPYCONFIG environment variable.

Create a KeyVault configuration section in the file, supplying the values described earlier. If you already have a vault that you want to use, put the name of the vault in the VaultName setting and run config2kv.py with the --exists parameter. This will bypass the Key Vault Management client section and the extra authentication step that this requires. If you do not have a vault or wish to create a new one, omit the --exists parameter and you will be prompted to create one.

The tool will read secrets and create secret names based on the path of the secret (as described above).

Warning

config2ky will only read and convert items in the provider Args sections. Currently, only ApiID and AuthKey values will be used.

The tool will then write the secret values to the vault. Finally a replacement msticpyconfig.yaml is written to the location specified in the --path argument. You can then delete or securely store your old configuration file and replace it with the one output by config2kv.

Tip

you can run config2ky with the --show parameter to perform a rehearsal. This will show you the Key Vault secrets that will be created and show the text of the msticpyconfig.yaml file that would have been created.

Using keyring to cache secrets

keyring is available on most Python platforms: Windows, Linux and MacOS. On Linux it requires the installation of optional components - either KWallet or Secret Service. See the Keyring Readme for more information.

Warning

keyring caching is enabled by default. If you are working in an environment that does not have one of the supported keyring backends installed you should disable keyring caching by adding Keyring: false to you configuration settings.

The advantage of using keyring is that you do not need to re-authenticate to Key Vault for each notebook that you use in each session. If you have UseKeyring: true in your msticpyconfig.yaml file, the first time that you access a Key Vault secret the secret value is stored as a keyring password with the same name as the Key Vault secret.

Unfortunately, the keyring package provides no way to list or delete stored secrets. If you need to remove the locally-stored secrets use the platform utility for the appropriate backend. For example, on Windows, cmdkey lets you list and manipulate local stored credentials.

Warning

keyring secrets are not automatically synchronized with the Key Vault secret values. If you change the value of a secret in Key Vault you must delete the keyring secret so that the new value will be re-read from Key Vault.

Manually managing your Key Vault secrets

You can use the Azure portal to create and manage your secrets. If you prefer to do this, simply enter the name of the secret in the corresponding section for the AuthKey or ApiID of your providers.

You can also use powershell or Python to manage these programmatically. MSTICPy has some convenience wrappers around the Azure SDK functions.

The documentation for these is available here: keyvault_client secrets_settings

User Defaults Section

This section specifies the query and other providers that you want to load by default. It is triggered from the init_notebook although you can call the load_user_defaults function to do this manually.

If you do not have this section in your configuration init_notebook will bypass auto-loading any components.

QueryProviders

This is a list of query providers that you want to load. Most of the providers have a single namespace/environment associated with them but in the case of Azure Sentinel, you can load multiple copies of the query provider for different workspaces. The example below shows three different formats that you can used. Each workspace name under QueryProviders must exist as a workspace definition in the AzureSentinel section of this file (see Commented configuration file sample below)

Note

Single-string entries in this and other sections (e.g. MyWorkspace: below) must be specified as empty dictionaries. This is done by adding a trailing “:” to the entry but no value on the other side of the colon. This is simply to make the settings parsing code a little easier. This is only when you are specifying a setting key - i.e. the first item on a line. The key values (“azsent”, “sco” and “False” in this example) should be entered without a trailing colon.

Note also that False is a boolean value, not a string. You should always enter True and False with proper capitalization.

UserDefaults
  QueryProviders:
    AzureSentinel:
      MyWorkspace:
      Default:
        alias: azsent
      CyberSoc:
        alias: soc
        connect: False

MyWorkspace is loaded as-is - equivalent to calling:

from msticpy.data import QueryProvider
from msticpy.common.wsconfig import WorkspaceConfig

qry_myworkspace = QueryProvider("AzureSentinel")
ws_config = WorkspaceConfig(workspace="MyWorkspace")
qry_myworkspace.connect(ws_config.code_connect_str)

The Default entry has a few differences. The name “Default” refers to the default workspace definition in the AzureSentinel section of the msticpyconfig file. The alias: azsent element is an alias that will be used to rename the provider. It is equivalent to the following code:

from msticpy.data import QueryProvider
from msticpy.common.wsconfig import WorkspaceConfig

qry_azsent = QueryProvider("AzureSentinel")
ws_config = WorkspaceConfig()
qry_azsent.connect(ws_config.code_connect_str)

The final CyberSoc entry has multiple key-value pairs under it. The “alias” entry works exactly the same as the previous example. The “connect” item tells the code not to automatically connect (authenticate) to Azure Sentinel. It is equivalent to the following code:

from msticpy.data import QueryProvider

qry_soc = QueryProvider("AzureSentinel")

In all three cases the query provider object (qry_soc in the last example) is stored in the global namespace of the notebook so you can always refer to it using this variable name.

Query providers for non-Azure Sentinel data sources use the same syntax for aliasing and suppressing connect/authenticate. For example:

UserDefaults
  QueryProviders:
    AzureSentinel:
      ...
    Splunk:
      connect: false
    LocalData:
      alias: local

LoadComponents

This section controls the loading and instantiation of a number of other data providers and components.

UserDefaults
  ...
  LoadComponents:
    TILookup:
    GeoIpLookup:
      provider: GeoLiteLookup
    Notebooklets:
      query_provider:
        AzureSentinel:
          workspace: CyberSoc
    Pivot:
    AzureData:
      auth_methods=['cli','interactive']
    AzureSentinelAPI:

Some of these accept additional parameters and some do not. Most of the configuration parameters for GeoIP providers, for example, are loaded from other sections of the configuration file.

GeoIpLookup - requires one parameter - the name of the provider that you want to use for GeoIP location resolution.

TILookup - no parameters, simply creates an instance of TILookup using the settings in the TIProviders section.

Notebooklets - to use this you must have MSTIC Notebooklets (msticnb see MSTICNB documentation). This has a required configuration setting, which MSTICPy passes to the notebooklets init function as the query_provider parameter. Other key/pair values included under the “query_provider” key are passed to the notebooklets initialization. Each parameter name is prefixed with the provider name so that it knows which parameters to send to which provider. In the example above notebooklets nbinit would be passed the following parameters:

nbinit(query_provider="AzureSentinel", AzureSentinel_workspace="CyberSoc")

The notebooklets package is loaded after most of the other providers (but before Pivot if that is included in the list) and is also sent the names of other providers (query and others such as TILookip) as its providers parameter. For more details see data_providers.init.

Pivot loads the Pivot library to add pivot functions to MSTICPy entities. It requires other providers to be loaded before itself (in order to harvest the pivot functions from them) so it is loaded last.

AzureData and AzureSentinel load the Azure resource API and Azure Sentinel API libraries respectively. Any key/pair values defined under either of these entries are passed to the provider connect method. In the AzureData example above this is equivalent to the following code.

from msticpy.data.azure_data import AzureData
az_data = AzureData()
az_data.connect(auth_methods=['cli','interactive'])

The components in the LoadProviders section have built-in friendly names for each component. These currently cannot be overridden from the configuration settings:

  • geoip

  • ti_lookup

  • nb

  • pivot

  • az_data

  • azs_api

MSTICPy current_providers Attribute

If you have loaded providers using the UserDefaults configuration the provider instances created are also stored in an attribute of the msticpy top level module.

>>> msticpy.current_providers

{'qry_azsent': <msticpy.data.data_providers.QueryProvider at 0x21604110ac8>,
'qry_myworkspace': <msticpy.data.data_providers.QueryProvider at 0x216041459c8>,
'qry_cybersoc': <msticpy.data.data_providers.QueryProvider at 0x21660d41308>,
'qry_splunk': <msticpy.data.data_providers.QueryProvider at 0x21661127208>,
'qry_local': <msticpy.data.data_providers.QueryProvider at 0x216605a7c48>,
'ti_lookup': <msticpy.sectools.tilookup.TILookup at 0x216611c7908>,
'geoip': <msticpy.sectools.geoip.GeoLiteLookup at 0x21660659c88>,
'pivot': <msticpy.datamodel.pivot.Pivot at 0x216602d8e88>,
'az_data': <msticpy.data.azure_data.AzureData at 0x21668aaf708>,
'azs_api': <msticpy.data.azure_sentinel.AzureSentinel at 0x21603f42388>,
'nb': <module 'msticnb' from 'e:\\src\\msticnb\\msticnb\\__init__.py'>}

You can use this to reference any of these loaded components. Although these values are normally also populated in the notebook global namespace you can re-populate them if needed. To write them back into the notebook namespace execute the following:

>>> globals().update(msticpy.current_providers)

Warning

This will overwrite any global variable with the same name as any of the items in the current_providers dictionary.

Extending msticpyconfig.yaml

You can also extend msticpyconfig to include additional sections to support other authentication and configuration options such as MDATP API connections. Refer to documentation on these features for required structures.

Settings are read by the refresh_config module. Combined settings are available as the settings attribute of this module. Default settings and custom settings (the settings that you specify in your own msticpyconfig.yaml) also available separately in the default_settings and custom_settngs attributes, respectively.

To force settings to be re-read after the package has been imported, call refresh_config.

The settings exposed in these attributes are python dictionaries that reflect the underlying YAML data in the configuration file.

Note

the wsconfig module, TIProviders, OtherProviders and the data libraries use additional functionality to provide higher-level views of the configuration data. An example of this is the using environment variable references to replace the actual configuration value with the secret stored in the environment variables.

Commented configuration file sample

AzureSentinel:
  Workspaces:
    # Workspace used if you don't explicitly name a workspace when creating WorkspaceConfig
    # Specifying values here overrides config.json settings unless you explictly load
    # WorkspaceConfig with config_file parameter (WorkspaceConfig(config_file="../config.json")
    Default:
      WorkspaceId: "d973e3d2-28e6-458e-b2cf-d38876fb1ba4"
      TenantId: "4cdf87a8-f0fc-40bb-9d85-68bcf4ac8e61"
    # To use these launch with an explicit name - WorkspaceConfig(workspace_name="Workspace2")
    Workspace2:
      WorkspaceId: "c88dd3c2-d657-4eb3-b913-58d58d811a41"
      TenantId: "f1f64e65-ff7c-4d71-ad5b-091b6ab39d51"
    Workspace3:
      WorkspaceId: "17e64332-19c9-472e-afd7-3629f299300c"
      TenantId: "4ea41beb-4546-4fba-890b-55553ce6003a"
UserDefaults:
  # List of query providers to load
  QueryProviders:
    - AzureSentinel:
      - Default: asi
      - CyberSoc:
        alias: soc
        connect: false
    - Splunk:
        connect: false
    - LocalData: local
  # List of other providers/components to load
  LoadComponents:
    - TILookup
    - GeoIpLookup: GeoLiteLookup
    - Notebooklets:
        query_provider:
          AzureSentinel: CyberSoc
    - Pivot
    - AzureData:
      auth_methods=['cli','interactive']
    - AzureSentinelAPI
QueryDefinitions:
  # Add paths to folders containing custom query definitions here
  Custom:
    - /var/global-queries
    - /home/myuser/queries
    - c:/users/myuser/documents
TIProviders:
  # If a provider has Primary: True it will be run by default on IoC lookups
  # Secondary providers can be
  OTX:
    Args:
      AuthKey: "4ea41beb-4546-4fba-890b-55553ce6003a"
    Primary: True
    Provider: "OTX" # WARNING - Do not change Provider values!
  VirusTotal:
    Args:
      AuthKey: "4ea41beb-4546-4fba-890b-55553ce6003a"
    Primary: False
    Provider: "VirusTotal"
  XForce:
    # You can store items in an environment variable using this syntax
    Args:
      ApiID:
        EnvironmentVar: "XFORCE_ID"
      AuthKey:
        EnvironmentVar: "XFORCE_KEY"
    Primary: True
    Provider: "XForce"
  AzureSentinel:
    # Note this can be a different workspace/tenant from your main workspace
    # This only controls where the Azure Sentinel TI provider looks for the
    # ThreatIndicator table.
    Args:
      WorkspaceID: "c88dd3c2-d657-4eb3-b913-58d58d811a41"
      TenantID: "f1f64e65-ff7c-4d71-ad5b-091b6ab39d51"
    Primary: True
    Provider: "AzSTI"
  OpenPageRank:
    Args:
      AuthKey:
        KeyVault:
    Primary: False
    Provider: "OPR"
  TorExitNodes:
    Primary: True
    Provider: "Tor"
  RiskIQ:
    Args:
      ApiID: "user@host.com"
      AuthKey: "aaaa-bbbb-cccc-dddd-eeee"
    Provider: "RiskIQ"
    Primary: True
OtherProviders:
  GeoIPLite:
    Args:
      AuthKey:
        EnvironmentVar: "MAXMIND_AUTH"
      DBFolder: "~/.msticpy"
    Provider: "GeoLiteLookup"
  IPStack:
    Args:
      AuthKey:
        KeyVault: my_secret
    Provider: "IPStackLookup"
  Contoso-GeopIp:
    Args:
      AuthKey:
        KeyVault: my_vault/my_secret
    Provider: "ContosoLookup"
DataProviders:
  AzureCLI:
    Args:
      clientId: "69d28fd7-42a5-48bc-a619-af56397b1111"
      tenantId: "69d28fd7-42a5-48bc-a619-af56397b2222"
      clientSecret: "69d28fd7-42a5-48bc-a619-af56397b3333"

See also

The Threat Intelligence Providers documention

wsconfig provider_settings wsconfig