forked from vectordotdev/vector
-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathtemplating.md.erb
135 lines (98 loc) · 3.15 KB
/
templating.md.erb
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
---
title: Templating
description: Learn Vector's templating syntax used to creat dynamic values in Vector's configuration.
status: beta
---
Vector supports a template syntax for select configuration options. This allows
you to use event fields when configuring Vector:
```text
field = "application_id={{ application_id }}/date=%F/"
```
## Example
For example, let's partition data on AWS S3 by `application_id` and `date`.
We can accomplish this with the
[`key_prefix` option][docs.sinks.aws_s3#key_prefix] in the
[`aws_s3` sink][docs.sinks.aws_s3]:
```toml
[sinks.backup]
type = "aws_s3"
bucket = "all_application_logs"
key_prefix = "application_id={{ application_id }}/date=%F/"
```
Notice that Vector allows direct field references as well as `strftime`
specifiers. If we were to run the following [`log` event][docs.data-model.log]
though Vector:
```json
{
"timestamp": "2020-02-14T01:22:23.223Z",
"application_id": 1,
"message": "Hello world"
}
```
The value of the `key_prefix` option would equal:
```text
application_id=1/date=2020-02-14
```
This effectively enables dynamic partitioning, something fundamental to storing
log data.
## Syntax
### Event Fields
Individual [`log` event][docs.data-model.log] fields can be access with the
`{{<field-path-notation>}}` syntax:
```toml
option = "{{ field_path_notation }}"
```
Vector's [field notation][docs.reference.field-path-notation] is simple. It uses
`.` to target nested fields and `[<index>]` to target array values. Learn more
in the [field notation docs][docs.reference.field-path-notation].
### Strftime Specifiers
In addition to directly accessing fields, Vector offers a shortcut for injecting
[strftime specifiers][urls.strptime_specifiers]:
```toml
options = "year=%Y/month=%m/day=%d/"
```
The value is based off of the [`timestamp` field][docs.data-model.log#timestamp],
and the name of this field can be changed via the
[global `timestamp_key` option][docs.global-options#timestamp_key].
### Escaping
You can escape this syntax by prefixing the character with a `\`. For example,
you can escape the event field syntax like so:
```toml
option = "\{{ field_name }}"
```
And strptime specified like so:
```toml
options = "year=\%Y/month=\%m/day=\%d/"
```
Each value above will be treated _literally_.
## How It Works
### Array Values
Array values can be accessed using Vector's
[field notation syntax][docs.reference.field-path-notation]:
```text
option = "{{ parent.child[0] }}"
```
### Fallback Values
Vector currently does not support fallback values. [Issue 1692][urls.issue_1692]
is open to add this functionality. In the interim, you can use the
[`lua` transform][docs.transforms.lua] to set a default value:
```toml
[transforms.set_defaults]
# REQUIRED
type = "lua"
inputs = ["my-source-id"]
source = """
if event["my_field"] == nil then
event["my_field"] = "default"
end
"""
```
### Missing Fields
If a field is missing, a blank string will be inserted in it's place. Vector
will not error, drop the event, or log anything.
### Nested Fields
Nested fields can be accessed using Vector's
[field notation syntax][docs.reference.field-path-notation]:
```text
option = "{{ parent.child }}"
```