JSONschema support for custom DevSkim rules #666
Comments
|
Thanks for the suggestion. I think this is a great idea and also appreciate your initial stab at it here. |
|
Thank you. If I have time, I'll try to work some more on it later. If this could be included in some kind of unit test or similar (in CI/CD) then consistency is practically enforced, mistakes directly spotted and all potential use cases de facto documented in the JSONschema specification. In some TS and GoLang projects with similarities I read a bunch of JSON files (like custom DevSkim rules) and validate them against a strictly defined JSONschema. This way there are no unhandled exceptions anymore. The library handles them and tells exactly what failed and what it expected. For example the library Ajv and https://github.com/xeipuuv/gojsonschema. |
|
After analyzing 145 scripts (mostly samples from this and the ApplicationInspector repo) and locally fixing all these mentioned here: microsoft/ApplicationInspector#595 in my test folder. I came up with the following JSON schema. I'm not sure it is entirely complete, probably not, but it passes all 145 scripts. I tried to use ./test\cryptography\extended.json: '' should be non-empty
./test\cryptography\weakssl.json: '' should be non-empty
./test\frameworks\build.json: '' should be non-empty
./test\frameworks\c.json: '' should be non-empty
./test\os\dynamic_execution.json: '' should be non-empty
./test\security\api\dangerous_api.json: '' should be non-empty
./test\security\attack_surface\outbound_network.json: '' should be non-empty
./test\security\control_flow\dynamic_execution.json: '' should be non-empty
./test\security\control_flow\format_string.json: '' should be non-empty
./test\security\control_flow\permission_evelation.json: '' should be non-empty
./test\security\cryptography\general.json: '' should be non-empty
./test\security\cryptography\hardcoded_tls.json: '' should be non-empty
./test\security\cryptography\hash_algorithm.json: '' should be non-empty
./test\security\cryptography\protocol.json: '' should be non-empty
./test\security\cryptography\random.json: '' should be non-empty
./test\security\frameworks\dotnet_framework.json: '' should be non-empty
./test\security\hygiene\localhost.json: '' should be non-empty
./test\security\hygiene\todo.json: '' should be non-empty
./test\security\manualreview\dynamiccode.json: '' should be non-empty
./test\security\privacy\secrets.json: '' should be non-emptyAny suggestions are welcome, I like to see this adapted into this project and actively worked on as a ultimate syntax reference for writing DevSkim rules. Perpahs this can be included as a first step to validate when using the Here is the new JSONschema: {
"$schema": "https://json-schema.org/draft-07/schema#",
"title": "Schema for validating DevSkim rules",
"type": "array",
"items": {
"type": "object",
"properties": {
"name": {
"type": "string",
"maxLength": 128,
"minLength": 1
},
"id": {
"type": "string",
"pattern": "^[A-Za-z0-9_-]+$",
"maxLength": 32,
"minLength": 1
},
"description": {
"type": "string",
"maxLength": 256,
"minLength": 1
},
"recommendation": {
"type": "string",
"maxLength": 256,
"minLength": 0,
"$comment": "TODO: This should be minLength: 1 - https://github.com/microsoft/ApplicationInspector/issues/595"
},
"applies_to": {
"type": "array",
"items": {
"type": "string",
"pattern": "^[a-zA-Z0-9._/ -]*$",
"maxLength": 96,
"minLength": 1
},
"additionalProperties": false
},
"tags": {
"type": "array",
"items": {
"type": "string",
"pattern": "^[\\w.:/-]+$",
"maxLength": 96,
"minLength": 1
},
"additionalProperties": false
},
"confidence": {
"type": "string",
"enum": [
"low",
"medium",
"high",
"unspecified"
]
},
"severity": {
"type": "string",
"enum": [
"low",
"medium",
"high",
"critical",
"moderate",
"important",
"manualreview",
"bestpractice",
"unspecified",
"Critical",
"ManualReview",
"BestPractice"
]
},
"rule_info": {
"type": "string",
"maxLength": 32,
"minLength": 1,
"pattern": "^([a-zA-Z0-9_]+\\.md)$"
},
"patterns": {
"type": "array",
"items": {
"type": "object",
"properties": {
"pattern": {
"type": "string",
"maxLength": 1024,
"minLength": 1
},
"xpaths": {
"type": "array",
"items": {
"type": "string",
"maxLength": 256,
"minLength": 1
}
},
"xpathnamespaces": {
"type": "object",
"properties": {
"default": {
"type": "string",
"maxLength": 128,
"minLength": 1
},
"android": {
"type": "string",
"maxLength": 128,
"minLength": 1
}
},
"additionalProperties": false
},
"jsonpaths": {
"type": "array",
"items": {
"type": "string",
"maxLength": 256,
"minLength": 1
}
},
"ymlpaths": {
"type": "array",
"items": {
"type": "string",
"maxLength": 256,
"minLength": 1
}
},
"type": {
"type": "string",
"enum": ["regex", "RegexWord", "regexword", "string", "substring"]
},
"scopes": {
"type": "array",
"items": {
"type": "string",
"maxLength": 64,
"minLength": 1
}
},
"modifiers": {
"type": "array",
"items": {
"type": "string",
"enum": ["i", "m"],
"maxLength": 1,
"minLength": 1
}
},
"confidence": {
"type": "string",
"enum": ["high", "High", "medium", "low"]
},
"_comment": {
"type": "string",
"maxLength": 128,
"minLength": 1
}
},
"required": ["pattern", "type"],
"additionalProperties": false
},
"additionalProperties": false
},
"fix_its": {
"type": "array",
"items": {
"type": "object",
"properties": {
"name": {
"type": "string",
"maxLength": 64,
"minLength": 1
},
"type": {
"type": "string",
"enum": ["RegexReplace"]
},
"replacement": {
"type": "string",
"maxLength": 64,
"minLength": 1
},
"pattern": {
"type": "object",
"properties": {
"pattern": {
"type": "string",
"maxLength": 128,
"minLength": 1
},
"type": {
"type": "string",
"enum": ["regex", "substring"]
},
"scopes": {
"type": "array",
"items": {
"type": "string",
"maxLength": 32,
"minLength": 1
}
},
"_comment": {
"type": "string",
"maxLength": 128,
"minLength": 1
},
"modifiers": {
"type": "array",
"items": {
"type": "string",
"enum": ["i", "m"],
"maxLength": 1,
"minLength": 1
}
}
},
"required": ["pattern", "type", "scopes"],
"additionalProperties": false
}
},
"required": ["name", "type", "replacement", "pattern"],
"additionalProperties": false
},
"additionalProperties": false
},
"must-match": {
"type": "array",
"items": {
"type": "string",
"maxLength": 1024,
"minLength": 1
},
"additionalProperties": false
},
"must-not-match": {
"type": "array",
"items": {
"type": "string",
"maxLength": 1024,
"minLength": 1
},
"additionalProperties": false
},
"_comment": {
"type": "string",
"maxLength": 128,
"minLength": 1
},
"does_not_apply_to": {
"type": [
"array"
],
"items": {
"type": "string",
"maxLength": 32,
"minLength": 1
},
"additionalProperties": false
},
"depends_on_tags": {
"type": "array",
"items": {
"type": "string",
"maxLength": 32,
"minLength": 1
},
"additionalProperties": false
},
"applies_to_file_regex": {
"type": "array",
"items": {
"type": "string",
"maxLength": 32,
"minLength": 1
},
"additionalProperties": false
},
"overrides": {
"type": "array",
"items": {
"type": "string",
"maxLength": 32,
"minLength": 1
},
"additionalProperties": false
},
"conditions": {
"type": "array",
"items": {
"type": "object",
"properties": {
"pattern": {
"type": "object",
"properties": {
"pattern": {
"type": "string",
"maxLength": 128,
"minLength": 1
},
"type": {
"type": "string",
"enum": ["regex", "regexword", "string", "substring"]
},
"scopes": {
"type": "array",
"items": {
"type": "string",
"maxLength": 32,
"minLength": 1
}
},
"modifiers": {
"type": "array",
"items": {
"type": "string",
"enum": ["i", "m"],
"maxLength": 1,
"minLength": 1
}
},
"xpaths": {
"type": "array",
"items": {
"type": "string",
"maxLength": 256,
"minLength": 1
}
},
"_comment": {
"type": "string",
"maxLength": 128,
"minLength": 1
}
},
"required": ["type", "scopes"],
"additionalProperties": false
},
"search_in": {
"type": "string",
"maxLength": 128,
"minLength": 1
},
"negate_finding": {
"type": "boolean"
},
"_comment": {
"type": "string",
"maxLength": 128,
"minLength": 1
}
},
"required": ["pattern", "search_in"],
"additionalProperties": false
},
"additionalProperties": false
}
},
"required": [
"patterns",
"tags",
"description",
"name",
"id"
],
"additionalProperties": false
},
"additionalProperties": false
}Here is a quick script I used to validate all rules I had against the JSONschema. Use it after running import os
import json
from jsonschema import validate, ValidationError
# Load schema
with open('schema.json') as schema_file:
schema = json.load(schema_file)
# Directory to search for JSON files
root_dir = './test'
# Validate JSON files
errors = []
for dirpath, _, filenames in os.walk(root_dir): # Recursively walk through the test folder
for file in filenames:
if file.endswith('.json'):
file_path = os.path.join(dirpath, file)
try:
with open(file_path) as json_file:
data = json.load(json_file)
validate(instance=data, schema=schema)
except ValidationError as e:
# Append the error with full file path
errors.append(f"{file_path}: {e.message}")
except json.JSONDecodeError as e:
errors.append(f"{file_path}: Invalid JSON - {e.msg}")
# Output results
if errors:
print("\n".join(errors))
else:
print("All files are valid.")Later as JSONSchema draft 7 allows I'd like to see more $comment values added with actual documentation on what it does and how to use it. |
|
Another pro-tip in VSCode I setup in my user settings.json I renamed the file extension of my custom devskim rules from
As well as autocomplete:
In fact, if we standardize this schema and encourage the use of |
|
Thanks for doing all this @JaneX8! I'll review this second iteration and try to fill in anything that might be missing. |
Is your feature request related to a problem? Please describe.
Yes, the documentation is really lacking and I'm having a very hard time getting things to work. A JSONSchema for custom rules would not only help validate the formatting it would be the ultimate documentation and test (that could be embedded even) and really help feedback for custom rule development.
Describe the solution you'd like
A complete JSONSchema description for the Custom Rule format for DevSkim.
Describe alternatives you've considered
XST.
Additional context
Here is a start, very incomplete but I still wouldn't know how to use all advanced features of DevSkim (such as ymlpath):
{ "$schema": "http://json-schema.org/draft-07/schema#", "title": "Generated schema for Root", "type": "array", "items": { "type": "object", "properties": { "name": { "type": "string" }, "id": { "type": "string" }, "description": { "type": "string" }, "recommendation": { "type": "string" }, "applies_to": { "type": "array", "items": { "type": "string" } }, "tags": { "type": "array", "items": { "type": "string" } }, "confidence": { "type": "string" }, "severity": { "type": "string" }, "rule_info": { "type": "string" }, "patterns": { "type": "array", "items": { "type": "object", "properties": { "pattern": { "type": "string" }, "type": { "type": "string" }, "scopes": { "type": "array", "items": { "type": "string" } } }, "required": [ "pattern", "type", "scopes" ] } }, "fix_its": { "type": "array", "items": { "type": "object", "properties": { "name": { "type": "string" }, "type": { "type": "string" }, "replacement": { "type": "string" }, "pattern": { "type": "object", "properties": { "pattern": { "type": "string" }, "type": { "type": "string" }, "scopes": { "type": "array", "items": { "type": "string" } } }, "required": [ "pattern", "type", "scopes" ] } }, "required": [ "name", "type", "replacement", "pattern" ] } }, "must-match": { "type": "array", "items": { "type": "string" } }, "must-not-match": { "type": "array", "items": { "type": "string" } } }, "required": [ "name", "id", "description", "recommendation", "applies_to", "tags", "confidence", "severity", "rule_info", "patterns", "fix_its", "must-match", "must-not-match" ] } }Also helpful:
The text was updated successfully, but these errors were encountered: