-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Can not create data-collection rule with a custom stream #7399
Comments
Thank you for opening this issue, we will look into it. |
@Joris-van-der-Wel thanks Joris for this difference. Not sure how azure portal deal with this but azure cli generates code based on service api doc, which can be referred here: https://github.com/Azure/azure-rest-api-specs/blob/42869ad83518d47ab2a24c07d888cb29fdeb43fd/specification/monitor/resource-manager/Microsoft.Insights/stable/2022-06-01/dataCollectionRules_API.json#L1135-L1148 . If that's the common user case, we'll let service team know the gap and whether they need to remove this enum validations. |
I think that jsonschema definition is indeed wrong. I think the proper schema would be something like: {
"streams": {
"description": "List of streams for this data flow.",
"type": "array",
"items": {
"pattern": "^(Microsoft-Event|Microsoft-InsightsMetrics|Microsoft-Perf|Microsoft-Syslog|Microsoft-WindowsEvent|Custom-[\\w-]+)$",
"type": "string"
}
}
} Also, it seems to me that the actual API server uses a different schema to validate with? Would it not be possible to copy that schema? For example if I pass a wrong value to the "streams" property, the server will respond with: {
"error": {
"code": "InvalidPayload",
"message": "Data collection rule is invalid",
"details": [
{
"code": "InvalidStream",
"message": "'Streams' stream 'Foo' must be a custom stream or one of the allowed streams.",
"target": "Properties.DataFlows[0].Streams[0]"
}
]
}
} Similarly, if I use an invalid property name for the stream declaration, I get this error: {
"error": {
"code": "InvalidPayload",
"message": "Data collection rule is invalid",
"details": [
{
"code": "InvalidStreamDeclaration",
"message": "'' stream 'Foo' must be a custom stream.",
"target": "Properties.StreamDeclarations[0].Key"
},
{
"code": "InvalidStreamDeclaration",
"message": "'' is not a valid custom stream name. It should start with 'Custom-' followed by a letter and then any combination of alphanumeric characters and '-', '_'",
"target": "Properties.StreamDeclarations[0].Key"
}
]
}
} Custom streams are supported by Azure in all of the other ways that I have tried: Azure Portal
This will look something like this: {
"id": "/subscriptions/XXX/resourceGroups/XXX/providers/Microsoft.Insights/dataCollectionRules/example-dcr",
"location": "westeurope",
"type": "Microsoft.Insights/dataCollectionRules",
"name": "example-dcr",
"properties": {
"dataSources": {},
"destinations": {
"logAnalytics": [{
"workspaceResourceId": "/subscriptions/XXX/resourcegroups/joris/providers/microsoft.operationalinsights/workspaces/XXX",
"workspaceId": "XXX",
"name": "123456789abcdef123456789abcdef12"
}]
},
"dataFlows": [{
"streams": ["Custom-Example_CL"],
"transformKql": "source",
"destinations": ["123456789abcdef123456789abcdef12"],
"outputStream": "Custom-Example_CL"
}],
"streamDeclarations": {
"Custom-Example_CL": {
"columns": [
{
"name": "TimeGenerated",
"type": "datetime"
},
{
"name": "hello",
"type": "string"
}
]
}
},
"dataCollectionEndpointId": "/subscriptions/XXX/resourceGroups/XXX/providers/Microsoft.Insights/dataCollectionEndpoints/XXX"
}
} Azure CLI (list)Next, try to use: [
{
"dataCollectionEndpointId": "/subscriptions/XXX/resourceGroups/XXX/providers/Microsoft.Insights/dataCollectionEndpoints/XXX",
"dataFlows": [{
"destinations": ["123456789abcdef123456789abcdef12"],
"outputStream": "Custom-Example_CL",
"streams": ["Custom-Example_CL"],
"transformKql": "source"
}],
"dataSources": {},
"destinations": {
"logAnalytics": [{
"name": "123456789abcdef123456789abcdef12",
"workspaceId": "XXX",
"workspaceResourceId": "/subscriptions/XXX/resourcegroups/XXX/providers/microsoft.operationalinsights/workspaces/XXX"
}]
},
"etag": "\"XXX\"",
"id": "/subscriptions/XXX/resourceGroups/XXX/providers/Microsoft.Insights/dataCollectionRules/example-dcr",
"immutableId": "dcr-3456789abcdef123456789abcdef1245",
"location": "westeurope",
"name": "example-dcr",
"provisioningState": "Succeeded",
"resourceGroup": "XXX",
"streamDeclarations": {
"Custom-Example_CL": {
"columns": [
{
"name": "TimeGenerated",
"type": "datetime"
},
{
"name": "hello",
"type": "string"
}
]
}
},
"systemData": {
"createdAt": "2024-03-28T11:00:00Z",
"createdBy": "jvanderwel@XXX",
"createdByType": "User",
"lastModifiedAt": "2024-03-28T11:00:00Z",
"lastModifiedBy": "jvanderwel@XXX",
"lastModifiedByType": "User"
},
"type": "Microsoft.Insights/dataCollectionRules"
}
] Azure Resource ManagerI can deploy a bicep template that uses a custom stream without issue: resource workspace 'Microsoft.OperationalInsights/workspaces@2022-10-01' = {
name: 'myworkspace'
location: 'westeurope'
}
resource dce 'Microsoft.Insights/dataCollectionEndpoints@2022-06-01' = {
name: 'mydce'
location: 'westeurope'
}
resource table 'Microsoft.OperationalInsights/workspaces/tables@2022-10-01' = {
parent: workspace
name: 'Example_CL'
properties: {
schema: {
name: 'Example_CL'
columns: [
{name: 'TimeGenerated', type: 'datetime'}
{name: 'hello', type: 'string'}
]
}
}
}
resource dcr 'Microsoft.Insights/dataCollectionRules@2022-06-01' = {
name: 'mydcr'
location: 'westeurope'
properties: {
dataCollectionEndpointId: dce.id
dataFlows: [
{
destinations: ['log-analytics-dest']
outputStream: 'Custom-Example_CL'
streams: ['Custom-Example_CL']
transformKql: 'source'
}
]
destinations: {
logAnalytics: [
{
name: 'log-analytics-dest'
workspaceResourceId: workspace.id
}
]
}
streamDeclarations: {
'Custom-Example_CL': {
columns: [
{name: 'TimeGenerated', type: 'datetime'}
{name: 'hello', type: 'string'}
]
}
}
}
} |
@Joris-van-der-Wel we will add support for enumeration arguments which skips validation recently. Please follow the linked pr for future development and releases. |
Describe the bug
I am attempting to create a data collection rule in azure monitor, that is intended to be used with a custom application that calls the Logs Ingestion API. I have created a logs-analytics workspace containing a custom table. My data collection rule should be configured without any data sources, instead a custom stream for my table is declared.
This all works fine when I configure this via the Azure Portal GUI. However the CLI refuses to create the data-collection rule because the validation for the
--data-flows
option is too strict. Specifically, the validation for the"streams"
property: https://github.com/Azure/azure-cli-extensions/blob/e2dad8c97f70c6b002749a7864f9dda2485498dd/src/monitor-control-service/azext_amcs/aaz/latest/monitor/data_collection/rule/_create.py#L157C9-L159C10If I comment out line 158 (containing
enum=...
):nano ~/.azure/cliextensions/monitor-control-service/azext_amcs/aaz/latest/monitor/data_collection/rule/_create.py
.Then my rule is created without issue.
Related command
Errors
unrecognized value 'Custom-MyTable_CL' from choices '['Microsoft-Event', 'Microsoft-InsightsMetrics', 'Microsoft-Perf', 'Microsoft-Syslog', 'Microsoft-WindowsEvent']'
Issue script & Debug output
Expected behavior
The rule should be created without error
Environment Summary
Additional context
No response
The text was updated successfully, but these errors were encountered: