Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE]Support Vega Overlay aggregations #138

Open
YANG-DB opened this issue Mar 8, 2024 · 0 comments
Open

[FEATURE]Support Vega Overlay aggregations #138

YANG-DB opened this issue Mar 8, 2024 · 0 comments
Assignees
Labels
content integration / getting-started content enhancement New feature or request visualization a visual widget for a specific purpose

Comments

@YANG-DB
Copy link
Member

YANG-DB commented Mar 8, 2024

Is your feature request related to a problem?
For many vega components showing data-point graphs or time-series based aggregation, its interesting to compare the selected time range with previously time bucket -

  • week over week
  • day over day
  • hour over hour
Screenshot 2024-03-08 at 3 46 26 PM

Do you have any additional context?

The next vega data attribute demonstrates how this can be achievable using the following data parts:

  • rawdata-old: this is the older time based data aggregation query used to compare with and uses direct query to facilitate the 1 hour shift overlay.
  • rawdata: this is the current time based data aggregation query
  • flatdata: this is the combination of the two datasources (current + overlay)

Each data query is decorated with the time relevancy (Old,Current) transformation to indicate its origin

  "data": [
    {
      "name": "rawdata-old",
      "url": {
        "index": "otel-v1-apm-span-*",
        "body": {
          "query": {
            "bool": {
              "must": [
                "%dashboard_context-must_clause%"
                {
                  "range": {
                    "startTime": {
                      "%timefilter%": true,
                      "shift": 1,
                      "unit": "hour"
                    }
                  }
                }
              ]
              "must_not": [
                "%dashboard_context-must_not_clause%"
              ],
              "filter": [
                "%dashboard_context-filter_clause%"
              ]
            }
          }
          "aggs": {
            "services": {
              "terms": {
                "field": "serviceName",
                "size": 15
              },
              "aggs": {
                "time_buckets": {
                  "date_histogram": {
                    "field": "startTime",
                    "interval": {"%autointerval%": true},
                    "extended_bounds": {
                      "min": {"%timefilter%": "min"},
                      "max": {"%timefilter%": "max"}
                    },
                    "min_doc_count":0
                  },
                  "aggs": {
                    "duration": {
                      "avg": {
                        "missing": 0,
                        "script": {
                          "source": "!doc.containsKey('durationInNanos') || doc['durationInNanos'].empty ? 0 : doc['durationInNanos'].value / 1000000.0",
                          "lang": "painless"
                        }
                      }
                    }
                  }
                }
              }
            }
          },
          "size": 0
        }
      },
      "format": {"property": "aggregations.services.buckets"},
      "transform": [
        {"type": "formula", "expr": "'Old'", "as": "source"}
      ]
    },
    {
      "name": "rawdata",
      "url": {
        "index": "otel-v1-apm-span-*",
        "%context%": true,
        "%timefield%": "startTime",
        "body": {
          "aggs": {
            "services": {
              "terms": {
                "field": "serviceName",
                "size": 15
              },
              "aggs": {
                "time_buckets": {
                  "date_histogram": {
                    "field": "startTime",
                    "interval": {"%autointerval%": true},
                    "extended_bounds": {
                      "min": {"%timefilter%": "min"},
                      "max": {"%timefilter%": "max"}
                    },
                    "min_doc_count":0
                  },
                  "aggs": {
                    "duration": {
                      "avg": {
                        "missing": 0,
                        "script": {
                          "source": "!doc.containsKey('durationInNanos') || doc['durationInNanos'].empty ? 0 : doc['durationInNanos'].value / 1000000.0",
                          "lang": "painless"
                        }
                      }
                    }
                  }
                }
              }
            }
          },
          "size": 0
        }
      },
      "format": {"property": "aggregations.services.buckets"},
      "transform": [
        {"type": "formula", "expr": "'Current'", "as": "source"}
      ]
    },
    {
      "name": "flatdata",
      "source": ["rawdata", "rawdata-old"],
      "transform": [
        {
          "type": "flatten",
          "fields": ["time_buckets.buckets"],
          "as": ["val"]
        },
        {
          "type": "formula",
          "as": "timeInMs",
          "expr":"datum.val.key"
        },
        {
          "type": "formula",
          "as": "count",
          "expr":"datum.val.doc_count"
        },
        {
          "type": "formula",
          "as": "duration",
          "expr": "datum.val.duration.value == null ? 0 : datum.val.duration.value"
        },
        {
          "type": "formula",
          "as": "time",
          "expr": "timeFormat(utcParse(datum.val.key_as_string,'%Y-%m-%dT%H:%M:%S.%LZ'), '%B %d, %Y %H:%M')"
        }
      ]
    },
   ....
  ]

For rendering both the current data and old data the marks should consider the decorated source field

"marks": [
  {
    "type": "symbol",
    "from": {"data": "flatdata"},
    "encode": {
      "update": {
        "x": {"scale": "xScale", "field": "time"},
        "y": {"scale": "yScale", "field": "count"},
        "fill": {
          "condition": {"test": "datum.dataType === 'Old'", "value": "red"},
          "value": "blue"
        },
        "tooltip": {
          "signal": "{'Date': datum.time, 'Count': datum.count, 'Duration': datum.duration, 'Type': datum.dataType}"
        }
      }
    }
  }
]

@YANG-DB YANG-DB added enhancement New feature or request visualization a visual widget for a specific purpose labels Mar 8, 2024
@YANG-DB YANG-DB self-assigned this Mar 8, 2024
@YANG-DB YANG-DB removed the untriaged label Mar 27, 2024
@YANG-DB YANG-DB added the content integration / getting-started content label Aug 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
content integration / getting-started content enhancement New feature or request visualization a visual widget for a specific purpose
Projects
Status: No status
Development

No branches or pull requests

1 participant