msticpy Package Configuration

Some elements of msticpy require configuration parameters. An example is the Threat Intelligence providers. Values for these and other parameters can be set in the msticpyconfig.yaml file.

The package has a default configuration file, which is stored in the package directory. You should not need to edit this file directly. Instead you can create a custom file with your own parameters - these settings will combine with or override the settings in the default file.

By default, the custom msticpyconfig.yaml is read from the current directory. You can specify an explicit location using an environment variable MSTICPYCONFIG.

Configuration sections

AzureSentinel

Here you can specify your default workspace and tenant IDs and add additional workspaces if needed.

QueryDefinitions

This allows you to specify paths to additional yaml query template files.

TIProviders

This allows you to configure which providers are run by default and to supply any authorization keys needed to access the service.

OtherProviders

This section is similar to the TIProviders section, allowing you specify configuration options for specialist data providers.

DataProviders

This section is similar to the previous two sections, allowing you specify configuration options for other data providers.

Key Vault

This section contains Azure Key Vault settings. This is only used if you choose to store secrets (e.g. API keys) in Key Vault.

Specifying secrets as Environment Variables

Some configuration values can be references to environment Variables rather than have a value explicitly stored within the configuration file. You might want secrets such as API keys to be supplied this way. The Args subsection of TIProvider and OtherProvider entries supports storing values as simple strings or as references to named environment variables. You can see examples of both in the sample file below.

Specifying secrets as Key Vault secrets

msticpy can read secret values from Key Vault for use with TI and other providers. To use this you need to specify settings for your keyvault.

KeyVault:
  TenantId: 5d6a50cf-b1b6-4bfd-ad54-b9822b06ff92
  SubscriptionId: 40dcc8bf-0478-4f3b-b275-ed0a94f2c013
  ResourceGroup: YourResourceGroup
  AzureRegion: RegionToCreateKV
  VaultName: "myvault"
  UseKeyring: True
  Authority: global
  AuthnType: device

Under the top level KeyVault section the following entries are accepted. Some of these are only required if you plan to use msticpy to create a new Key Vault vault.

Required Settings

TenantId the Identifier of your Azure tenant
VaultName the name of the vault to use (note this can be overridden in the individual secret specifications
Authority this specifies the Azure cloud instance to use.

For most users Authority is “global” (default). Other values are: - usgov: Azure US Government - de: German cloud - chi: China cloud

Required to Create a Key Vault

SubscriptionId the Azure subscription holding the Key Vault
ResourceGroup the Azure resource group in which to create the vault
AzureRegion the Azure region in which to create the vault

Optional Settings

UseKeyring if True (default) uses the Python keyring package to securely cache Key Vault secrets in your client session.
VaultName the name of the vault to use (note this can be overridden in the individual secret specifications)
AuthnType this governs the authentication type used by the KeyVault client (to read and write secrets). The choices are “interactive” for interactive browser authentication or “device” for authentication using a user/device code. The default is “interactive”.

Note

The AuthnType does not affect the authentication used by the Key Vault Management client (the creation and enumeration of vaults). This always uses device code authentication.

Specifying Key Vault Secrets in Provider Settings

There are three ways to specify the Key Vault vault and secret names to use for a given setting.

TIProviders:

  OpenPageRank:
    Args:
      AuthKey:
        KeyVault:

Adding an empty subkey named KeyVault will cause msticpy to use a secret name build from the path of the setting. This is the default usage. In this example, the secret name will be “TIProviders-OpenPageRank-Args-AuthKey”. The vault name is taken from the setting in the KeyVault settings section.

OtherProviders:
  IPStack:
    Args:
      AuthKey:
        KeyVault: my_secret

This example specifies “my_secret” as the secret name. The vault name is taken from the setting in the KeyVault settings section.

OtherProviders:
  Contoso-GeopIp:
    Args:
      AuthKey:
        KeyVault: my_vault/my_secret

The final example specifies both a vault name and a secret name. The VaultName setting in the KeyVault section is ignored for this setting.

Populating Key Vault secrets from an existing msticpyconfig.yaml

The easiest way to move your secrets to Key Vault is to use the config2ky.py tool available in the msticpy GitHub repo. You can find it in the tools folder.

Running config2kv.py --help shows the usage of this utility.

The simplest way to use this tool is to populate your existing secrets as strings in your msticpyconfig.yaml. (as shown in some of the provider settings in the example at the end of this page).

You can specify this as the input file using the --path parameter. Alternatively, the tool will look for a msticpyconfig.yaml in the location specified by the MSTICPYCONFIG environment variable.

Create a KeyVault configuration section in the file, supplying the values described earlier. If you already have a vault that you want to use, put the name of the vault in the VaultName setting and run config2kv.py with the --exists parameter. This will bypass the Key Vault Management client section and the extra authentication step that this requires. If you do not have a vault or wish to create a new one, omit the --exists parameter and you will be prompted to create one.

The tool will read secrets and create secret names based on the path of the secret (as described above).

Warning

config2ky will only read and convert items in the provider Args sections. Currently, only ApiID and AuthKey values will be used.

The tool will then write the secret values to the vault. Finally a replacement msticpyconfig.yaml is written to the location specified in the --path argument. You can then delete or securely store your old configuration file and replace it with the one output by config2kv.

Tip

you can run config2ky with the --show parameter to perform a rehearsal. This will show you the Key Vault secrets that will be created and show the text of the msticpyconfig.yaml file that would have been created.

Using keyring to cache secrets

keyring is available on most Python platforms: Windows, Linux and MacOS. On Linux it requires the installation of optional components - either KWallet or Secret Service. See the Keyring Readme for more information.

Warning

keyring caching is enabled by default. If you are working in an environment that does not have one of the supported keyring backends installed you should disable keyring caching by adding Keyring: false to you configuration settings.

The advantage of using keyring is that you do not need to re-authenticate to Key Vault for each notebook that you use in each session. If you have UseKeyring: true in your msticpyconfig.yaml file, the first time that you access a Key Vault secret the secret value is stored as a keyring password with the same name as the Key Vault secret.

Unfortunately keyring provides no way to list or delete stored secrets. If you need to remove the locally-stored secrets use the platform utility for the appropriate backend. For example, on Windows, cmdkey lets you list and manipulate local stored credentials.

Warning

keyring secrets are not automatically synchronized with the Key Vault secret values. If you change the value of a secret in Key Vault you must delete the keyring secret so that the new value will be re-read from Key Vault.

Manually managing your Key Vault secrets

You can use the Azure portal to create and manage your secrets. If you prefer to do this, simply enter the name of the secret in the corresponding section for the AuthKey or ApiID of your providers.

You can also use powershell or Python to manage these programmatically. msticpy has some convenience wrappers around the Azure SDK functions.

The documentation for these is available here: keyvault_client secrets_settings

Extending msticpyconfig.yaml

You can also extend msticpyconfig to include additional sections to support other authentication and configuration options such as MDATP API connections. Refer to documentation on these features for required structures.

Settings are read by the refresh_config module. Combined settings are available as the settings attribute of this module. Default settings and custom settings (the settings that you specify in your own msticpyconfig.yaml) also available separately in the default_settings and custom_settngs attributes, respectively.

To force settings to be re-read after the package has been imported, call refresh_config.

The settings exposed in these attributes are python dictionaries that reflect the underlying YAML data in the configuration file.

Note

the wsconfig module, TIProviders, OtherProviders and the data libraries use additional functionality to provide higher-level views of the configuration data. An example of this is the using environment variable references to replace the actual configuration value with the secret stored in the environment variables.

Commented configuration file sample

AzureSentinel:
  Workspaces:
    # Workspace used if you don't explicitly name a workspace when creating WorkspaceConfig
    # Specifying values here overrides config.json settings unless you explictly load
    # WorkspaceConfig with config_file parameter (WorkspaceConfig(config_file="../config.json")
    Default:
      WorkspaceId: "d973e3d2-28e6-458e-b2cf-d38876fb1ba4"
      TenantId: "4cdf87a8-f0fc-40bb-9d85-68bcf4ac8e61"
    # To use these launch with an explicit name - WorkspaceConfig(workspace_name="Workspace2")
    Workspace2:
      WorkspaceId: "c88dd3c2-d657-4eb3-b913-58d58d811a41"
      TenantId: "f1f64e65-ff7c-4d71-ad5b-091b6ab39d51"
    Workspace3:
      WorkspaceId: "17e64332-19c9-472e-afd7-3629f299300c"
      TenantId: "4ea41beb-4546-4fba-890b-55553ce6003a"
QueryDefinitions:
  # Add paths to folders containing custom query definitions here
  Custom:
    - /var/global-queries
    - /home/myuser/queries
    - c:/users/myuser/documents
TIProviders:
  # If a provider has Primary: True it will be run by default on IoC lookups
  # Secondary providers can be
  OTX:
    Args:
      AuthKey: "4ea41beb-4546-4fba-890b-55553ce6003a"
    Primary: True
    Provider: "OTX" # WARNING - Do not change Provider values!
  VirusTotal:
    Args:
      AuthKey: "4ea41beb-4546-4fba-890b-55553ce6003a"
    Primary: False
    Provider: "VirusTotal"
  XForce:
    # You can store items in an environment variable using this syntax
    Args:
      ApiID:
        EnvironmentVar: "XFORCE_ID"
      AuthKey:
        EnvironmentVar: "XFORCE_KEY"
    Primary: True
    Provider: "XForce"
  AzureSentinel:
    # Note this can be a different workspace/tenant from your main workspace
    # This only controls where the Azure Sentinel TI provider looks for the
    # ThreatIndicator table.
    Args:
      WorkspaceID: "c88dd3c2-d657-4eb3-b913-58d58d811a41"
      TenantID: "f1f64e65-ff7c-4d71-ad5b-091b6ab39d51"
    Primary: True
    Provider: "AzSTI"
  OpenPageRank:
    Args:
      AuthKey:
        KeyVault:
    Primary: False
    Provider: "OPR"
  TorExitNodes:
    Primary: True
    Provider: "Tor"
OtherProviders:
  GeoIPLite:
    Args:
      AuthKey:
        EnvironmentVar: "MAXMIND_AUTH"
      DBFolder: "~/.msticpy"
    Provider: "GeoLiteLookup"
  IPStack:
    Args:
      AuthKey:
        KeyVault: my_secret
    Provider: "IPStackLookup"
  Contoso-GeopIp:
    Args:
      AuthKey:
        KeyVault: my_vault/my_secret
    Provider: "ContosoLookup"
DataProviders:
  AzureCLI:
    Args:
      clientId: "69d28fd7-42a5-48bc-a619-af56397b1111"
      tenantId: "69d28fd7-42a5-48bc-a619-af56397b2222"
      clientSecret: "69d28fd7-42a5-48bc-a619-af56397b3333"

See also

The Threat Intelligence Providers documention

wsconfig provider_settings wsconfig