Azure Data Enrichment
Description
This package contains functionality for enriching data regarding Azure host details with additional host details exposed via the Azure API. By providing an Azure Resource or Azure Subscription the package returns key contextual information regarding the Subscription or Resource. This package is primarily aimed at Azure IaaS resources but will work with any Azure Resource type. This feature is currently a work in progress and additional data enrichment capabilities will be added over time.
Azure Data API documentation
Instantiating and Connecting with an Azure Data Connector
See Azure Data
In order to connect to the Azure API and retrieve the required data we need to instantiate an Azure Data Connector and connect to the API. Authentication to the Azure API is handled via the azure_auth package. By default this package will attempt to use a prioritized list of authentication options. Available options are:
‘env’ - This checks for credentials stored as environment variables. If this option is selected valid credentials in msticpyconfig.yaml will be written as environment variable values and used.
‘cli’ - This attempts to use credentials generated by logging in via the Azure CLI on the host running the notebook kernel.
‘msi’ - This attempts to use an Azure Managed Identity.
‘interactive’ - This prompts the browser to interactively login using the device’s browser.
By default ['cli', 'msi', 'devicecode']
is used but you can provide an alternative
list to .connect
via the auth_methods parameter.
az = AzureData()
az.connect(auth_methods=['cli','interactive'])
For more details on Azure authentication see Azure Authentication in MSTICPy.
Get Azure Subscription Details
See get_subscriptions
Details about the subscription a resource is a member of can provide vital context to a security analyst when conducting an investigation. This package contains 2 functions to support this.
AZURE_DATA_CONNECTOR.list_subscriptions() returns a pandas DataFrame with details of all the subscriptions within the tenant.
az.get_subscriptions()
Subscription ID | Display Name | State | |
---|---|---|---|
0 | c690adc4-ec82-41fa-ad36-80f3c8899bd0 | Visual Studio Enterprise | SubscriptionState.enabled |
See get_subscription_info
AZURE_DATA_CONNECTOR.get_subscription_info() gets information on a specific subscription ID.
az.get_subscription_info("c690adc4-ec82-41fa-ad36-80f3c8899bd0")
{'Subscription ID': 'c690adc4-ec82-41fa-ad36-80f3c8899bd0',
'Display Name': 'Visual Studio Enterprise',
'State': 'SubscriptionState.enabled',
'Subscription Location Limits': 'Public_2014-09-01',
'Subscription Quota': 'MSDN_2014-09-01',
'Spending Limit': <SpendingLimit.on: 'On'>}
Get Azure Resource Details
See get_resources
As well as subscriptions we can return details on a specific Azure resource. AZURE_DATA_CONNECTOR.get_resources() returns a pandas DataFrame with details on all resources within a Subscription or Resource Group. In addition, you can request full properties on each Resource with the get_props = True parameter. However, this can take some time to return results.
resources = az.get_resources(sub_id="bca22c36-a158-44ff-8cbb-23fa92236a55")
resources.head()
resource_id |
name |
resource_type |
location |
tags |
plan |
properties |
kind |
managed_by |
sku |
identity |
|
---|---|---|---|---|---|---|---|---|---|---|---|
0 |
/subscriptions/3b701f84-d04b-4479-89b1-fa8827e… |
cloud-shell-storage-westeurope-vnet |
Microsoft.Network/virtualNetworks |
centralus |
{} |
None |
None |
None |
None |
None |
None |
1 |
/subscriptions/3b701f84-d04b-4479-89b1-fa8827e… |
csb3b701f84d04bx4479x89b |
Microsoft.Storage/storageAccounts |
westeurope |
{‘ms-resource-usage’: ‘azure-cloud-shell’} |
None |
None |
Storage |
None |
{‘additional_properties’: {}, ‘name’: ‘Standar… |
None |
2 |
/subscriptions/3b701f84-d04b-4479-89b1-fa8827e… |
bluepot-01_OsDisk_1_ad7a7c0383444f02830ba46418… |
Microsoft.Compute/disks |
westus |
None |
None |
None |
None |
/subscriptions/3b701f84-d04b-4479-89b1-fa8827e… |
None |
None |
3 |
/subscriptions/3b701f84-d04b-4479-89b1-fa8827e… |
bluepot-02_OsDisk_1_dce988e082e54617ae3622eca0… |
Microsoft.Compute/disks |
westus |
None |
None |
None |
None |
/subscriptions/3b701f84-d04b-4479-89b1-fa8827e… |
None |
None |
4 |
/subscriptions/3b701f84-d04b-4479-89b1-fa8827e… |
CentOS-Test_OsDisk_1_7ee38d36b893481e8a68405c0… |
Microsoft.Compute/disks |
westus |
None |
None |
None |
None |
/subscriptions/3b701f84-d04b-4479-89b1-fa8827e… |
{‘additional_properties’: {}, ‘name’: ‘Premium… |
None |
See get_resource_details
You can return full details on a single resource with AZURE_DATA_CONNECTOR.get_resource_details() and passing a Resource ID.
az.get_resource_details(resource_id="/subscriptions/bca22c36-a158-44ff-8cbb-23fa92236a55/resourceGroups/Contoso/providers/Microsoft.Compute/virtualMachines/UbuntuDevEnv")
{'resource_id': '/subscriptions/bca22c36-a158-44ff-8cbb-23fa92236a55/resourceGroups/Contoso/providers/Microsoft.Compute/virtualMachines/UbuntuDevEnv',
'name': 'UbuntuDevEnv',
'resource_type': 'Microsoft.Compute/virtualMachines',
'location': 'northeurope',
'tags': {},
'plan': None,
'properties': {'vmId': 'f557c9da-309f-4ab9-93ec-b29d7c21be87',
'hardwareProfile': {'vmSize': 'Standard_B1s'},
'storageProfile': {'imageReference': {'publisher': 'Canonical',
'offer': 'UbuntuServer',
'sku': '18.04-LTS',
'version': 'latest',
'exactVersion': '18.04.201812040'},
'osDisk': {'osType': 'Linux',
'name': 'UbuntuDevEnv_OsDisk_1_fc3690fe9f2248a1b441c0a1616833c5',
'createOption': 'FromImage',
'caching': 'ReadWrite',
'managedDisk': {'id': '/subscriptions/bca22c36-a158-44ff-8cbb-23fa92236a55/resourceGroups/CONTOSO/providers/Microsoft.Compute/disks/UbuntuDevEnv_OsDisk_1_fc3690fe9f2248a1b441c0a1616833c5'}},
'dataDisks': [{'lun': 0,
'name': 'UbuntuDevEnv_DataDisk_0',
'createOption': 'Attach',
'caching': 'None',
'writeAcceleratorEnabled': False,
'managedDisk': {'id': '/subscriptions/bca22c36-a158-44ff-8cbb-23fa92236a55/resourceGroups/Contoso/providers/Microsoft.Compute/disks/UbuntuDevEnv_DataDisk_0'},
'toBeDetached': False}]},
'osProfile': {'computerName': 'UbuntuDevEnv',
'adminUsername': 'user',
'linuxConfiguration': {'disablePasswordAuthentication': True,
'ssh': {'publicKeys': [{'path': '/home/user/.ssh/authorized_keys',
'keyData': ''}]},
'provisionVMAgent': True},
'secrets': [],
'allowExtensionOperations': True},
'networkProfile': {'networkInterfaces': [{'id': '/subscriptions/bca22c36-a158-44ff-8cbb-23fa92236a55/resourceGroups/Contoso/providers/Microsoft.Network/networkInterfaces/ubuntudevenv3'}]},
'provisioningState': 'Succeeded'},
'kind': None,
'managed_by': None,
'sku': None,
'identity': None}
Note
You can also provide a dictionary of resource details if you don’t have a complete Resource ID. The details dictionary must contain: * resource_group_name * resource_provider_namespace * parent_resource_path (if there isn’t one leave as a empty string). * resource_type * resource_name
resource_details = {"resource_group_name":"Contoso",
"resource_provider_namespace":"Microsoft.Compute",
"parent_resource_path":"",
"resource_type":"virtualMachines",
"resource_name":"UbuntuDevEnv"}
az.get_resource_details(resource_details=resource_details)
{'resource_id': '/subscriptions/bca22c36-a158-44ff-8cbb-23fa92236a55/resourceGroups/Contoso/providers/Microsoft.Compute/virtualMachines/UbuntuDevEnv',
'name': 'UbuntuDevEnv',
'resource_type': 'Microsoft.Compute/virtualMachines',
'location': 'northeurope',
'tags': {},
'plan': None,
'properties': {'vmId': 'f557c9da-309f-4ab9-93ec-b29d7c21be87',
'hardwareProfile': {'vmSize': 'Standard_B1s'},
'storageProfile': {'imageReference': {'publisher': 'Canonical',
'offer': 'UbuntuServer',
'sku': '18.04-LTS',
'version': 'latest',
'exactVersion': '18.04.201812040'},
'osDisk': {'osType': 'Linux',
'name': 'UbuntuDevEnv_OsDisk_1_fc3690fe9f2248a1b441c0a1616833c5',
'createOption': 'FromImage',
'caching': 'ReadWrite',
'managedDisk': {'id': '/subscriptions/bca22c36-a158-44ff-8cbb-23fa92236a55/resourceGroups/CONTOSO/providers/Microsoft.Compute/disks/UbuntuDevEnv_OsDisk_1_fc3690fe9f2248a1b441c0a1616833c5'}},
'dataDisks': [{'lun': 0,
'name': 'UbuntuDevEnv_DataDisk_0',
'createOption': 'Attach',
'caching': 'None',
'writeAcceleratorEnabled': False,
'managedDisk': {'id': '/subscriptions/bca22c36-a158-44ff-8cbb-23fa92236a55/resourceGroups/Contoso/providers/Microsoft.Compute/disks/UbuntuDevEnv_DataDisk_0'},
'toBeDetached': False}]},
'osProfile': {'computerName': 'UbuntuDevEnv',
'adminUsername': 'user',
'linuxConfiguration': {'disablePasswordAuthentication': True,
'ssh': {'publicKeys': [{'path': '/home/user/.ssh/authorized_keys',
'keyData': ''}]},
'provisionVMAgent': True},
'secrets': [],
'allowExtensionOperations': True},
'networkProfile': {'networkInterfaces': [{'id': '/subscriptions/bca22c36-a158-44ff-8cbb-23fa92236a55/resourceGroups/Contoso/providers/Microsoft.Network/networkInterfaces/ubuntudevenv3'}]},
'provisioningState': 'Succeeded'},
'kind': None,
'managed_by': None,
'sku': None,
'identity': None}
Get Azure Network Details
See get_network_details
If your Azure resources has a network interface associated with it (for example a VM) you can return details on the interface as associated Network Security Group (NSG). Calling this function is very similar to getting resource details however instead of passing it a resource ID you provide the network interface ID for the network device you want details for.
az.get_network_details(networkID=NETWORK_INTERFACE_ID, sub_id=SUBSCRIPTION_ID)
Note
If youa are looking for a VM network interface ID you can use get_resource_details to get details on the VM. The network interface will be under properties > networkProfile > networkInterfaces > id
This will return a DataFrame containing details of all IP addresses and subnets associated with the network interface.
Get Azure Metrics
See get_metrics
Azure provides a range of metrics for resources. The types of metrics available depends on the Azure resource in question, a full list of metrics can be found here.
You can return all of these metrics with get_metrics.
In order to call this function you need to provide the metrics you want to retrieve in a comma separated string e.g. “”Percentage CPU,Disk Read Bytes,Disk Write Bytes”, along with the resource ID of the item you wish to retrieve the metrics for, and the subscription ID that resource is part of. You can also choose to get the metrics sampled at either the minute or the hour interval, and for how many days preceding you want metrics for. By default the function returns hourly metrics for the last 30 days.
az.get_metrics(metrics="Percentage CPU", resource_id=resource_details['resource_id'], sub_id=sub_details['Subscription ID'], sample_time="hour", start_time=15)
This returns a dictionary of items with the metric name as they key and a DataFrame of the metrics as the value.
Note
get_metrics is resource specific, so if you want to get metrics from more than one resource you will need separate function calls.