Skip to content

Commit

Permalink
Enable Flow Logs and Traffic Analytics (#24)
Browse files Browse the repository at this point in the history 
* Change how network watchers are being managed (depend on auto deployment feature)
* Introduce secured storage account for flow logs
* Every NSG (other than the cluster's NSG ... pending yet) has flow logs (v2) enabled, with traffic analytics.
  • Loading branch information
@ckittel
ckittel authored Jul 13, 2021
1 parent 6ee76c7 commit ed7c22d
Show file tree
Hide file tree
Showing 10 changed files with 728 additions and 64 deletions.
17 changes: 9 additions & 8 deletions docs/deploy/04-subscription.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ The following three resource groups will be created in the steps below.
| rg-enterprise-networking-hubs | Contains all of your organization's regional hubs. A regional hub resources in this implementation include an the hub Virtual Network, egress firewall, Azure Bastion, and Log Analytics for network logging. They may also contain your VPN Gateways, which are not addressed in this implementation. |
| rg-enterprise-networking-spokes | Contains all of your organization's regional spokes and related networking resources. All spokes will peer with their regional hub and subnets will egress through the regional firewall in the hub. |
| rg-bu0001a0005 | Contains the regulated cluster resources. |
| networkWatcherRG | Contains regional Network Watchers. _(This is only created if your subscription doesn't already have Network Watchers in place.)_ |
| networkWatcherRG | Contains regional Network Watchers. _(Most subscriptions already have this.)_ |

Both Azure Kubernetes Service and Azure Image Builder Service use a concept of a dynamically-created _infrastructure_ resource group. So in addition to the four resource groups mentioned above, as you follow these instructions, you'll end up with six resource groups; two of which are automatically created and their lifecycle tied to their owning service. You will not see these two infrastructure resource groups get created until later in the walkthrough when their owning service is created.

Expand All @@ -28,13 +28,12 @@ To help govern our resources, there are policies we apply over the scope of thes
| Policy Name | Scope | Purpose |
|--------------------------------|---------------------------------|---------------------------------------------------------------------------------------------------|
| Enable Azure Defender Standard | Subscription | Ensures that Azure Defender for Kubernetes, Container Service, and Key Vault are always enabled. |
| Deploy Network Watcher | Subscription | A _Deploy if not exists_ policy to ensure there is a regional network watcher for your virtual networks. _(This is only created if your subscription doesn't already have Network Watchers in place.)_ |
| Allowed resource types | rg-enterprise-networking-hubs | Restricts the hub resource group to just relevant networking resources. |
| VNet must have Network Watcher | rg-enterprise-networking-hubs | Audit policy that will trigger if a network is deployed to a region that doesn't have a Network Watcher. _(This is only created if your subscription doesn't already have Network Watchers in place.)_ |
| Allowed resource types | rg-enterprise-networking-spokes | Restricts the spokes resource group to just relevant networking resources. |
| VNet must have Network Watcher | rg-enterprise-networking-spokes | Audit policy that will trigger if a network is deployed to a region that doesn't have a Network Watcher. _(This is only created if your subscription doesn't already have Network Watchers in place.)_ |
| Allowed resource types | rg-bu0001a0005 | Restricts the workload resource group to just resources necessary for this specific architecture. |
| Allowed resource types | networkWatcherRG | Restricts the Network Watcher resource group to just Network Watcher resources. _(This is only created if your subscription doesn't already have Network Watchers in place.)_ |
| Allowed resource types | networkWatcherRG | Restricts the Network Watcher resource group to just Network Watcher resources. _(Audit only mode to prevent conflict with any existing policy that manages this common resource group.)_ |
| No public AKS clusters | rg-bu0001a0005 | Restricts the creation of AKS clusters to only those with private Kubernetes API server. |
| No out-of-date AKS clusters | rg-bu0001a0005 | Restricts the creation of AKS clusters to only recent versions. |
| No AKS clusters without RBAC | rg-bu0001a0005 | Restricts the creation of AKS clusters to only those that are Azure AD RBAC enabled. |
Expand Down Expand Up @@ -81,28 +80,30 @@ Not only do we enable them in the steps below by default, but also set up an Azu
TENANTID_AZURERBAC=$(az account show --query tenantId -o tsv)
```

1. Check for existing Network Watchers.
1. Check for a pre-existing resource group with the name `networkWatcherRG`.

```bash
[ $(az network watcher list --query 'length([])' -o tsv) -eq 0 ] && ENABLE_NETWORK_WATCHERS=true || ENABLE_NETWORK_WATCHERS=false
NETWORK_WATCHER_RG_REGION=$(az group list --query "[?name=='networkWatcherRG'].location" -o tsv)
```

> Azure Network Watchers are regional singletons in your subscription and should always be handled external to any specific workload; at the subscription level. Because your subscription may already have configuration (existing instances and/or a corresponding _deploy if not exists_ policy applied) around Network Watcher, it's hard to deliver a "one size fits all" solution in this isolated walkthrough. If the subscription you're deploying into doesn't have _any_ Network Watchers deployed, you'll have them set up as part of this deployment. If however, you already have Network Watcher resources, we'll leave those alone and won't deploy any additional resources related to them. If you have any conflicts in this walkthrough related to Network Watchers (existing management group policies, etc.), simply set `ENABLE_NETWORK_WATCHERS=false` and all related attempts to set up Network Watchers will be skipped.
If your subscription is managed in such a way that Azure Network Watcher resources are found in a resource group other than the Azure default of `networkWatcherRG` or they do not use the Azure default `NetworkWatcher_<region>` naming convention, you will need to adjust the various ARM templates to compensate. Network Watchers are singletons (per region) in subscriptions, and organizations often manage them (and Flow Logs) via Azure Policy. This walkthrough assumes default naming conventions as set by Azure's [automatic deployment feature of Network Watchers](https://docs.microsoft.com/azure/network-watcher/network-watcher-create#network-watcher-is-automatically-enabled).

If at any time during the deployment you get an error stating "**resource 'NetworkWatcher_\<region>' not found**", you will need to skip flow log creation by passing `false` to that ARM template's `deployFlowLogResources` parameter or you can manually create the required Network Watcher with that name.

1. Perform subscription-level deployment.

This will deploy the resource groups, Azure Policies, and Azure Security Center configuration all as identified above.

```bash
# [This may take up to six minutes to run.]
az deployment sub create -f subscription.json -l centralus -p enableNetworkWatchers=${ENABLE_NETWORK_WATCHERS}
az deployment sub create -f subscription.json -l centralus -p networkWatcherRGRegion="${NETWORK_WATCHER_RG_REGION}"
```

If you do not have permissions on your subscription to enable Azure Defender (which requires the Azure RBAC role of _Subscription Owner_ or _Security Admin_), then instead execute the following variation of the same command. This will not enable Azure Defender services nor will Azure Policy attempt to enable the same (the policy will still be created, but in audit-only mode). Your final implementation should be to a subscription with these security services activated.

```bash
# [This may take up to five minutes to run.]
az deployment sub create -f subscription.json -l centralus -p enableAzureDefender=false enforceAzureDefenderAutoDeployPolicies=false enableNetworkWatchers=${ENABLE_NETWORK_WATCHERS}
az deployment sub create -f subscription.json -l centralus -p enableAzureDefender=false enforceAzureDefenderAutoDeployPolicies=false networkWatcherRGRegion="${NETWORK_WATCHER_RG_REGION}"
```

## Azure Security Benchmark
Expand Down
2 changes: 1 addition & 1 deletion docs/deploy/05-networking-hub.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ Egressing your spoke traffic through a hub network (following the hub-spoke mode

## Expected results

After executing these steps you'll have the `rg-enterprise-networking-hubs` resource group populated with a regional virtual network (vnet), Azure Firewall, Azure Bastion, and Azure Monitor for network observability. No spokes will have been created yet, so the default firewall rules are maximally restrictive, as there is no expected outflow of traffic so none is allowed. We'll open up access on an as-needed bases throughout this walk through.
After executing these steps you'll have the `rg-enterprise-networking-hubs` resource group populated with a regional virtual network (vnet), Azure Firewall, Azure Bastion, and Azure Monitor & Flow Log storage for network observability. No spokes will have been created yet, so the default firewall rules are maximally restrictive, as there is no expected outflow of traffic so none is allowed. We'll open up access on an as-needed bases throughout this walk through.

Specifically, you'll see networking/hub-region.v​_n_.json referenced a couple times in this process. Think of this as an evolution of a _single_ ARM template as the number and needs of the connected spokes change over time. You can diff the v​_n_ and v​_n+1_ ARM templates to see this progression over time. Typically your network team would have encapsulated this hub in a file named something like `hub-eastus2.json` and updated it in their source control as dependencies/requirements dictate. It likely would have not taken as many parameters as either, as those would be constants that could be easily defined directly in the template as the file would be specific to the region's spokes. To keep this reference implementation more flexible on what region you select, you'll be asked to provide deployment parameters and the filename can remain the generic name of hub-​_region_.

Expand Down
8 changes: 8 additions & 0 deletions docs/deploy/13-validation-logs.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -209,6 +209,14 @@ Data from Log Analytic on the networking and cluster infrastructure are being de

From here you can view Alerts, Incidents, start a Hunting session, view/add related Workbooks (such as the **Azure Kubernetes Service (AKS) Security** workbook), etc.

## Traffic Analytics

NSG Flow Logs were configured as part of the default install of this reference implementation. Along with the raw logs being stored in the storage account in the hub resource group, you can access Azure Traffic Analytics for a dashboard of network flows.

1. Open the [Azure Traffic Analytics hub](https://portal.azure.com/#blade/Microsoft_Azure_Network/NetworkWatcherMenuBlade/trafficAnalytics) on the Azure Portal.
1. Adjust filters as needed at the top.
1. You can then see Application Ports, NSG Hits, App Gateway, and other network flows.

### Next step

:arrow_forward: [Clean Up Azure Resources](./14-cleanup.md)
4 changes: 4 additions & 0 deletions docs/deploy/14-cleanup.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,10 @@ After you are done exploring your deployed [AKS Baseline Cluster for Regulated W

## Steps

1. Delete flow logs

The `networkWatcherRG` resource group is where flow log definitions were created for this reference implementation. All of the flow logs that were created were prefixed with `fl` and were followed by a GUID, targeting a virtual network either in the hub or spokes resource group.

1. Delete the resource groups as a way to delete all contained Azure resources.

> To delete all Azure resources associated with this reference implementation, you'll need to delete the three resource groups created.
Expand Down
132 changes: 132 additions & 0 deletions networking/hub-region.v0.json
View file
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,13 @@
"metadata": {
"description": "A /27 under the VNet Address Space for regional Azure Bastion"
}
},
"deployFlowLogResources": {
"defaultValue": true,
"type": "bool",
"metadata": {
"description": "Flow Logs are enabled by default, if for some reason they cause conflicts with flow log policies already in place in your subscription, you can disable them by passing 'false' to this parameter."
}
}
},
"variables": {
Expand All @@ -75,6 +82,7 @@
"bastionPipName": "[concat('pip-ab-', parameters('location'))]",
"bastionServiceName": "[concat('ab-', parameters('location'))]",
"hubLaName": "[concat('la-hub-', parameters('location'), '-', uniqueString(resourceId('Microsoft.Network/virtualNetworks', variables('hubVnetName'))))]",
"regionFlowLowStorageAccountName": "[take(concat('stnfl', parameters('location'), uniqueString(resourceGroup().id)), 24)]",
"azureSentialLaSolutionName": "[concat('SecurityInsights(', variables('hubLaName'), ')')]"
},
"resources": [
Expand Down Expand Up @@ -447,6 +455,81 @@
}
]
},
{
"type": "Microsoft.Storage/storageAccounts",
"apiVersion": "2021-02-01",
"name": "[variables('regionFlowLowStorageAccountName')]",
"location": "[parameters('location')]",
"sku": {
"name": "standard_LRS"
},
"kind": "StorageV2",
"properties": {
"accessTier": "Hot",
"minimumTlsVersion": "TLS1_2",
"supportsHttpsTrafficOnly": true,
"allowBlobPublicAccess": false,
"allowSharedKeyAccess": false,
"networkAcls": {
"bypass": "AzureServices",
"defaultAction": "Deny",
"ipRules": []
}
},
"resources": [
{
"type": "providers/diagnosticSettings",
"apiVersion": "2017-05-01-preview",
"name": "Microsoft.Insights/default",
"dependsOn": [
"[resourceId('Microsoft.Storage/storageAccounts', variables('regionFlowLowStorageAccountName'))]",
"[resourceId('Microsoft.OperationalInsights/workspaces', variables('hubLaName'))]"
],
"properties": {
"workspaceId": "[resourceId('Microsoft.OperationalInsights/workspaces', variables('hubLaName'))]",
"logs": [],
"metrics": [
{
"category": "Transaction",
"enabled": true
}
]
}
}
]
},
{
"type": "Microsoft.Storage/storageAccounts/blobServices/providers/diagnosticsettings",
"apiVersion": "2017-05-01-preview",
"name": "[concat(variables('regionFlowLowStorageAccountName'), '/default/Microsoft.Insights/default')]",
"dependsOn": [
"[resourceId('Microsoft.Storage/storageAccounts', variables('regionFlowLowStorageAccountName'))]",
"[resourceId('Microsoft.OperationalInsights/workspaces', variables('hubLaName'))]"
],
"properties": {
"workspaceId": "[resourceId('Microsoft.OperationalInsights/workspaces', variables('hubLaName'))]",
"logs": [
{
"category": "StorageRead",
"enabled": true
},
{
"category": "StorageWrite",
"enabled": true
},
{
"category": "StorageDelete",
"enabled": true
}
],
"metrics": [
{
"category": "Transaction",
"enabled": true
}
]
}
},
{
"type": "Microsoft.Network/azureFirewalls",
"apiVersion": "2020-05-01",
Expand Down Expand Up @@ -538,6 +621,55 @@
}
}
]
},
{
"condition": "[parameters('deployFlowLogResources')]",
"name": "connect-hub-regional-flowlogs",
"type": "Microsoft.Resources/deployments",
"resourceGroup": "networkWatcherRG",
"apiVersion": "2020-10-01",
"dependsOn": [
"[resourceId('Microsoft.Network/networkSecurityGroups', variables('bastionNetworkNsgName'))]",
"[resourceId('Microsoft.Storage/storageAccounts', variables('regionFlowLowStorageAccountName'))]",
"[resourceId('Microsoft.Network/virtualNetworks', variables('hubVnetName'))]",
"[resourceId('Microsoft.OperationalInsights/workspaces', variables('hubLaName'))]",
"[resourceId('Microsoft.Network/azureFirewalls', variables('hubFwName'))]" // This doesn't depend on the FW, but because network watchers are auto-deployed, this helps prevents a race condition between the vnet creation triggering a NetworkWatcher auto provisioning, and the referencing of it here.
],
"properties": {
"mode": "Incremental",
"template": {
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"variables": {},
"resources": [
{
"name": "[concat('NetworkWatcher_', parameters('location'), '/fl', guid(resourceId(resourceGroup().name, 'Microsoft.Network/networkSecurityGroups', variables('bastionNetworkNsgName'))))]",
"type": "Microsoft.Network/networkWatchers/flowLogs",
"apiVersion": "2020-05-01",
"location": "[parameters('location')]",
"properties": {
"targetResourceId": "[resourceId(resourceGroup().name, 'Microsoft.Network/networkSecurityGroups', variables('bastionNetworkNsgName'))]",
"storageId": "[resourceId(resourceGroup().name, 'Microsoft.Storage/storageAccounts', variables('regionFlowLowStorageAccountName'))]",
"enabled": true,
"format": {
"version": 2
},
"flowAnalyticsConfiguration": {
"networkWatcherFlowAnalyticsConfiguration": {
"enabled": true,
"workspaceResourceId": "[resourceId(resourceGroup().name, 'Microsoft.OperationalInsights/workspaces', variables('hubLaName'))]",
"trafficAnalyticsInterval": 10
}
},
"retentionPolicy": {
"days": 365,
"enabled": true
}
}
}
]
}
}
}
],
"outputs": {
Expand Down
Loading

0 comments on commit ed7c22d

Please sign in to comment.