Azure Data Enrichment

Description

This package contains functionality for enriching data regarding Azure host details with additional host details exposed via the Azure API. By providing an Azure Resource or Azure Subscription the package returns key contextual information regarding the Subscription or Resource. This package is primarily aimed at Azure IaaS resources but will work with any Azure Resource type. This feature is currently a work in progress and additional data enrichment capabilities will be added over time.

Azure Data API documentation

Instantiating and Connecting with an Azure Data Connector

See Azure Data

In order to connect to the Azure API and retrieve the required data we need to instantiate an Azure Data Connector and connect to the API. Authentication to the Azure API is handled via the azure_auth package. By default this package will attempt to use a prioritized list of authentication options. Available options are:

  • ‘env’ - This checks for credentials stored as environment variables. If this option is selected valid credentials in msticpyconfig.yaml will be written as environment variable values and used.

  • ‘cli’ - This attempts to use credentials generated by logging in via the Azure CLI on the host running the notebook kernel.

  • ‘msi’ - This attempts to use an Azure Managed Identity.

  • ‘interactive’ - This prompts the browser to interactively login using the device’s browser.

By default ['cli', 'msi', 'devicecode'] is used but you can provide an alternative list to .connect via the auth_methods parameter.

az = AzureData()
az.connect(auth_methods=['cli','interactive'])

For more details on Azure authentication see Azure Authentication in MSTICPy.

Get Azure Subscription Details

See get_subscriptions

Details about the subscription a resource is a member of can provide vital context to a security analyst when conducting an investigation. This package contains 2 functions to support this.

AZURE_DATA_CONNECTOR.list_subscriptions() returns a pandas DataFrame with details of all the subscriptions within the tenant.

az.get_subscriptions()
Subscription ID Display Name State
0 c690adc4-ec82-41fa-ad36-80f3c8899bd0 Visual Studio Enterprise SubscriptionState.enabled

See get_subscription_info

AZURE_DATA_CONNECTOR.get_subscription_info() gets information on a specific subscription ID.

az.get_subscription_info("c690adc4-ec82-41fa-ad36-80f3c8899bd0")
{'Subscription ID': 'c690adc4-ec82-41fa-ad36-80f3c8899bd0',
 'Display Name': 'Visual Studio Enterprise',
 'State': 'SubscriptionState.enabled',
 'Subscription Location Limits': 'Public_2014-09-01',
 'Subscription Quota': 'MSDN_2014-09-01',
 'Spending Limit': <SpendingLimit.on: 'On'>}

Get Azure Resource Details

See get_resources

As well as subscriptions we can return details on a specific Azure resource. AZURE_DATA_CONNECTOR.get_resources() returns a pandas DataFrame with details on all resources within a Subscription or Resource Group. In addition, you can request full properties on each Resource with the get_props = True parameter. However, this can take some time to return results.

resources = az.get_resources(sub_id="bca22c36-a158-44ff-8cbb-23fa92236a55")
resources.head()

resource_id

name

resource_type

location

tags

plan

properties

kind

managed_by

sku

identity

0

/subscriptions/3b701f84-d04b-4479-89b1-fa8827e…

cloud-shell-storage-westeurope-vnet

Microsoft.Network/virtualNetworks

centralus

{}

None

None

None

None

None

None

1

/subscriptions/3b701f84-d04b-4479-89b1-fa8827e…

csb3b701f84d04bx4479x89b

Microsoft.Storage/storageAccounts

westeurope

{‘ms-resource-usage’: ‘azure-cloud-shell’}

None

None

Storage

None

{‘additional_properties’: {}, ‘name’: ‘Standar…

None

2

/subscriptions/3b701f84-d04b-4479-89b1-fa8827e…

bluepot-01_OsDisk_1_ad7a7c0383444f02830ba46418…

Microsoft.Compute/disks

westus

None

None

None

None

/subscriptions/3b701f84-d04b-4479-89b1-fa8827e…

None

None

3

/subscriptions/3b701f84-d04b-4479-89b1-fa8827e…

bluepot-02_OsDisk_1_dce988e082e54617ae3622eca0…

Microsoft.Compute/disks

westus

None

None

None

None

/subscriptions/3b701f84-d04b-4479-89b1-fa8827e…

None

None

4

/subscriptions/3b701f84-d04b-4479-89b1-fa8827e…

CentOS-Test_OsDisk_1_7ee38d36b893481e8a68405c0…

Microsoft.Compute/disks

westus

None

None

None

None

/subscriptions/3b701f84-d04b-4479-89b1-fa8827e…

{‘additional_properties’: {}, ‘name’: ‘Premium…

None

See get_resource_details

You can return full details on a single resource with AZURE_DATA_CONNECTOR.get_resource_details() and passing a Resource ID.

az.get_resource_details(resource_id="/subscriptions/bca22c36-a158-44ff-8cbb-23fa92236a55/resourceGroups/Contoso/providers/Microsoft.Compute/virtualMachines/UbuntuDevEnv")
{'resource_id': '/subscriptions/bca22c36-a158-44ff-8cbb-23fa92236a55/resourceGroups/Contoso/providers/Microsoft.Compute/virtualMachines/UbuntuDevEnv',
 'name': 'UbuntuDevEnv',
 'resource_type': 'Microsoft.Compute/virtualMachines',
 'location': 'northeurope',
 'tags': {},
 'plan': None,
 'properties': {'vmId': 'f557c9da-309f-4ab9-93ec-b29d7c21be87',
  'hardwareProfile': {'vmSize': 'Standard_B1s'},
  'storageProfile': {'imageReference': {'publisher': 'Canonical',
    'offer': 'UbuntuServer',
    'sku': '18.04-LTS',
    'version': 'latest',
    'exactVersion': '18.04.201812040'},
   'osDisk': {'osType': 'Linux',
    'name': 'UbuntuDevEnv_OsDisk_1_fc3690fe9f2248a1b441c0a1616833c5',
    'createOption': 'FromImage',
    'caching': 'ReadWrite',
    'managedDisk': {'id': '/subscriptions/bca22c36-a158-44ff-8cbb-23fa92236a55/resourceGroups/CONTOSO/providers/Microsoft.Compute/disks/UbuntuDevEnv_OsDisk_1_fc3690fe9f2248a1b441c0a1616833c5'}},
   'dataDisks': [{'lun': 0,
     'name': 'UbuntuDevEnv_DataDisk_0',
     'createOption': 'Attach',
     'caching': 'None',
     'writeAcceleratorEnabled': False,
     'managedDisk': {'id': '/subscriptions/bca22c36-a158-44ff-8cbb-23fa92236a55/resourceGroups/Contoso/providers/Microsoft.Compute/disks/UbuntuDevEnv_DataDisk_0'},
     'toBeDetached': False}]},
  'osProfile': {'computerName': 'UbuntuDevEnv',
   'adminUsername': 'user',
   'linuxConfiguration': {'disablePasswordAuthentication': True,
    'ssh': {'publicKeys': [{'path': '/home/user/.ssh/authorized_keys',
       'keyData': ''}]},
    'provisionVMAgent': True},
   'secrets': [],
   'allowExtensionOperations': True},
  'networkProfile': {'networkInterfaces': [{'id': '/subscriptions/bca22c36-a158-44ff-8cbb-23fa92236a55/resourceGroups/Contoso/providers/Microsoft.Network/networkInterfaces/ubuntudevenv3'}]},
  'provisioningState': 'Succeeded'},
 'kind': None,
 'managed_by': None,
 'sku': None,
 'identity': None}

Note

You can also provide a dictionary of resource details if you don’t have a complete Resource ID. The details dictionary must contain: * resource_group_name * resource_provider_namespace * parent_resource_path (if there isn’t one leave as a empty string). * resource_type * resource_name

resource_details = {"resource_group_name":"Contoso",
                   "resource_provider_namespace":"Microsoft.Compute",
                   "parent_resource_path":"",
                   "resource_type":"virtualMachines",
                   "resource_name":"UbuntuDevEnv"}
az.get_resource_details(resource_details=resource_details)
{'resource_id': '/subscriptions/bca22c36-a158-44ff-8cbb-23fa92236a55/resourceGroups/Contoso/providers/Microsoft.Compute/virtualMachines/UbuntuDevEnv',
 'name': 'UbuntuDevEnv',
 'resource_type': 'Microsoft.Compute/virtualMachines',
 'location': 'northeurope',
 'tags': {},
 'plan': None,
 'properties': {'vmId': 'f557c9da-309f-4ab9-93ec-b29d7c21be87',
  'hardwareProfile': {'vmSize': 'Standard_B1s'},
  'storageProfile': {'imageReference': {'publisher': 'Canonical',
    'offer': 'UbuntuServer',
    'sku': '18.04-LTS',
    'version': 'latest',
    'exactVersion': '18.04.201812040'},
   'osDisk': {'osType': 'Linux',
    'name': 'UbuntuDevEnv_OsDisk_1_fc3690fe9f2248a1b441c0a1616833c5',
    'createOption': 'FromImage',
    'caching': 'ReadWrite',
    'managedDisk': {'id': '/subscriptions/bca22c36-a158-44ff-8cbb-23fa92236a55/resourceGroups/CONTOSO/providers/Microsoft.Compute/disks/UbuntuDevEnv_OsDisk_1_fc3690fe9f2248a1b441c0a1616833c5'}},
   'dataDisks': [{'lun': 0,
     'name': 'UbuntuDevEnv_DataDisk_0',
     'createOption': 'Attach',
     'caching': 'None',
     'writeAcceleratorEnabled': False,
     'managedDisk': {'id': '/subscriptions/bca22c36-a158-44ff-8cbb-23fa92236a55/resourceGroups/Contoso/providers/Microsoft.Compute/disks/UbuntuDevEnv_DataDisk_0'},
     'toBeDetached': False}]},
  'osProfile': {'computerName': 'UbuntuDevEnv',
   'adminUsername': 'user',
   'linuxConfiguration': {'disablePasswordAuthentication': True,
    'ssh': {'publicKeys': [{'path': '/home/user/.ssh/authorized_keys',
       'keyData': ''}]},
    'provisionVMAgent': True},
   'secrets': [],
   'allowExtensionOperations': True},
  'networkProfile': {'networkInterfaces': [{'id': '/subscriptions/bca22c36-a158-44ff-8cbb-23fa92236a55/resourceGroups/Contoso/providers/Microsoft.Network/networkInterfaces/ubuntudevenv3'}]},
  'provisioningState': 'Succeeded'},
 'kind': None,
 'managed_by': None,
 'sku': None,
 'identity': None}

Get Azure Network Details

See get_network_details

If your Azure resources has a network interface associated with it (for example a VM) you can return details on the interface as associated Network Security Group (NSG). Calling this function is very similar to getting resource details however instead of passing it a resource ID you provide the network interface ID for the network device you want details for.

az.get_network_details(networkID=NETWORK_INTERFACE_ID, sub_id=SUBSCRIPTION_ID)

Note

If youa are looking for a VM network interface ID you can use get_resource_details to get details on the VM. The network interface will be under properties > networkProfile > networkInterfaces > id

This will return a DataFrame containing details of all IP addresses and subnets associated with the network interface.

Get Azure Metrics

See get_metrics

Azure provides a range of metrics for resources. The types of metrics available depends on the Azure resource in question, a full list of metrics can be found here.

You can return all of these metrics with get_metrics.

In order to call this function you need to provide the metrics you want to retrieve in a comma separated string e.g. “”Percentage CPU,Disk Read Bytes,Disk Write Bytes”, along with the resource ID of the item you wish to retrieve the metrics for, and the subscription ID that resource is part of. You can also choose to get the metrics sampled at either the minute or the hour interval, and for how many days preceding you want metrics for. By default the function returns hourly metrics for the last 30 days.

az.get_metrics(metrics="Percentage CPU", resource_id=resource_details['resource_id'], sub_id=sub_details['Subscription ID'], sample_time="hour", start_time=15)

This returns a dictionary of items with the metric name as they key and a DataFrame of the metrics as the value.

Note

get_metrics is resource specific, so if you want to get metrics from more than one resource you will need separate function calls.