Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: New logs page #2143

Closed
wants to merge 44 commits into from
Closed
Show file tree
Hide file tree
Changes from 30 commits
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
7759006
feat: add basic components
ogzhanolguncu Sep 26, 2024
5993ae5
feat: add log details section
ogzhanolguncu Sep 27, 2024
4344d3a
feat: move log details to separate component
ogzhanolguncu Sep 28, 2024
42e481d
refactor: log details section
ogzhanolguncu Sep 28, 2024
78eec7b
fix: overflow issue
ogzhanolguncu Sep 28, 2024
20ea6ae
feat: add custom time and calendar
ogzhanolguncu Sep 28, 2024
3e18a34
fix: server-client render issues
ogzhanolguncu Sep 29, 2024
d2b8b1f
feat: add time to calendar comp
ogzhanolguncu Sep 29, 2024
1cc48a8
feat: add response status filter
ogzhanolguncu Sep 29, 2024
d2a517c
feat: finalize search combobox
ogzhanolguncu Oct 1, 2024
9519fe4
feat: refactor search combobox and add search param persistance
ogzhanolguncu Oct 1, 2024
88667d5
feat: add filter persistance to all components
ogzhanolguncu Oct 1, 2024
9258053
feat: add highlight effect when row selected
ogzhanolguncu Oct 9, 2024
0fe8ba3
fix: clickhouse and also main pull I guess
chronark Oct 9, 2024
873e64e
feat: add initial data fetch for logs page
ogzhanolguncu Oct 10, 2024
0ae2d82
re organize files and use real data for charts component
ogzhanolguncu Oct 11, 2024
0da98b5
fix: trpc query
ogzhanolguncu Oct 14, 2024
6d231b5
chore: formatter
ogzhanolguncu Oct 14, 2024
b9c6346
feat: add badges to request and status
ogzhanolguncu Oct 16, 2024
ee7f73a
feat: add new tooltip for time
ogzhanolguncu Oct 16, 2024
7a2f3fc
feat: change how we show json fields in details modal
ogzhanolguncu Oct 16, 2024
ad2d539
Merge branch 'main' of github.com:unkeyed/unkey into logs-page
ogzhanolguncu Dec 2, 2024
3c22672
chore: update lock
ogzhanolguncu Dec 2, 2024
029e4d4
feat: add virtual list to logs
ogzhanolguncu Dec 2, 2024
5b21935
chore: add new local
ogzhanolguncu Dec 2, 2024
1714490
refactor: improvement
ogzhanolguncu Dec 3, 2024
444eb2e
refactor: improve query params type safety
ogzhanolguncu Dec 3, 2024
215d7b8
refactor: move click house to internal package
ogzhanolguncu Dec 3, 2024
8fe44eb
chore: fix fmt issues
ogzhanolguncu Dec 3, 2024
610531b
fix: fmt issues and clickhouse imports
ogzhanolguncu Dec 3, 2024
1eee930
fix: pnmp conflict
ogzhanolguncu Dec 3, 2024
5213279
Merge branch 'main' of github.com:unkeyed/unkey into logs-page
ogzhanolguncu Dec 3, 2024
cc79423
[autofix.ci] apply automated fixes
autofix-ci[bot] Dec 3, 2024
02b3251
fix: lock quote issue
ogzhanolguncu Dec 3, 2024
230fb5a
chore: add new lock file
ogzhanolguncu Dec 3, 2024
d247461
Merge branch 'logs-page' of https://github.com/ogzhanolguncu/unkey in…
ogzhanolguncu Dec 3, 2024
cb2b70e
fix: code rabbit issues
ogzhanolguncu Dec 3, 2024
56a14bf
[autofix.ci] apply automated fixes
autofix-ci[bot] Dec 3, 2024
80391cc
fix: revert some rabbit fixes and add review fixes
ogzhanolguncu Dec 3, 2024
9b0f94d
refactor: improve chart
ogzhanolguncu Dec 4, 2024
3bf4794
chore: remove shiki
ogzhanolguncu Dec 4, 2024
7f94805
Merge branch 'logs-page' of https://github.com/ogzhanolguncu/unkey in…
ogzhanolguncu Dec 4, 2024
b457ad7
Merge branch 'main' of github.com:unkeyed/unkey into logs-page
ogzhanolguncu Dec 4, 2024
542e577
Merge branch 'main' of github.com:unkeyed/unkey into logs-page
ogzhanolguncu Dec 4, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion apps/dashboard/app/(app)/layout.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ export default async function Layout({ children, breadcrumb }: LayoutProps) {
/>

<div className="isolate bg-background lg:border-l border-t lg:rounded-tl-[0.625rem] border-border w-full overflow-x-auto flex flex-col items-center lg:mt-2">
<div className="w-full max-w-[1152px] p-4 lg:p-8">
<div className="w-full p-4 lg:p-8">
{workspace.enabled ? (
<>
{/* Hacky way to make the breadcrumbs line up with the Teamswitcher on the left, because that also has h12 */}
Expand Down
128 changes: 128 additions & 0 deletions apps/dashboard/app/(app)/logs/components/chart.tsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,128 @@
"use client";

import {
type ChartConfig,
ChartContainer,
ChartTooltip,
ChartTooltipContent,
} from "@/components/ui/chart";
import { format } from "date-fns";
import { Bar, BarChart, CartesianGrid, XAxis } from "recharts";

export type Log = {
request_id: string;
time: number;
workspace_id: string;
host: string;
method: string;
path: string;
request_headers: string[];
request_body: string;
response_status: number;
response_headers: string[];
response_body: string;
error: string;
service_latency: number;
};

const chartConfig = {
success: {
label: "Success",
color: "hsl(var(--chart-3))",
},
warning: {
label: "Warning",
color: "hsl(var(--chart-4))",
},
error: {
label: "Error",
color: "hsl(var(--chart-1))",
},
} satisfies ChartConfig;

function aggregateData(data: Log[]) {
const aggregatedData: {
date: string;
success: number;
warning: number;
error: number;
}[] = [];
const intervalMs = 60 * 1000 * 10;

if (data.length === 0) {
return aggregatedData;
}

const startOfDay = new Date(data[0].time).setHours(0, 0, 0, 0);
const endOfDay = startOfDay + 24 * 60 * 60 * 1000;

for (let timestamp = startOfDay; timestamp < endOfDay; timestamp += intervalMs) {
const filteredLogs = data.filter((d) => d.time >= timestamp && d.time < timestamp + intervalMs);

const success = filteredLogs.filter(
(log) => log.response_status >= 200 && log.response_status < 300,
).length;
const warning = filteredLogs.filter(
(log) => log.response_status >= 400 && log.response_status < 500,
).length;
const error = filteredLogs.filter((log) => log.response_status >= 500).length;

aggregatedData.push({
date: format(timestamp, "yyyy-MM-dd'T'HH:mm:ss"),
success,
warning,
error,
});
}

return aggregatedData;
}
Copy link
Contributor

@coderabbitai coderabbitai bot Dec 3, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Optimize aggregateData function for performance

Currently, the aggregateData function filters the entire data array for each time interval, which can lead to performance issues with large datasets. Consider refactoring the function to process the data in a single pass.

Here is a suggested refactor:

 function aggregateData(data: Log[]) {
   const aggregatedData: {
     date: string;
     success: number;
     warning: number;
     error: number;
   }[] = [];
   const intervalMs = 60 * 1000 * 10;

+  if (data.length === 0) {
+    return aggregatedData;
+  }
+
+  // Sort the data by timestamp
+  const sortedData = data.slice().sort((a, b) => a.time - b.time);
+
+  let intervalStart = new Date(sortedData[0].time).setMinutes(0, 0, 0);
+  let intervalEnd = intervalStart + intervalMs;
+  let index = 0;
+
+  while (intervalStart < sortedData[sortedData.length - 1].time) {
+    let success = 0;
+    let warning = 0;
+    let error = 0;
+
+    while (index < sortedData.length && sortedData[index].time < intervalEnd) {
+      const log = sortedData[index];
+      if (log.response_status >= 200 && log.response_status < 300) {
+        success += 1;
+      } else if (log.response_status >= 400 && log.response_status < 500) {
+        warning += 1;
+      } else if (log.response_status >= 500) {
+        error += 1;
+      }
+      index += 1;
+    }
+
+    aggregatedData.push({
+      date: format(intervalStart, "yyyy-MM-dd'T'HH:mm:ss"),
+      success,
+      warning,
+      error,
+    });
+
+    intervalStart = intervalEnd;
+    intervalEnd += intervalMs;
+  }
+
   return aggregatedData;
 }

This refactor processes the data in a single pass, improving performance for large datasets.

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
function aggregateData(data: Log[]) {
const aggregatedData: {
date: string;
success: number;
warning: number;
error: number;
}[] = [];
const intervalMs = 60 * 1000 * 10;
if (data.length === 0) {
return aggregatedData;
}
const startOfDay = new Date(data[0].time).setHours(0, 0, 0, 0);
const endOfDay = startOfDay + 24 * 60 * 60 * 1000;
for (let timestamp = startOfDay; timestamp < endOfDay; timestamp += intervalMs) {
const filteredLogs = data.filter((d) => d.time >= timestamp && d.time < timestamp + intervalMs);
const success = filteredLogs.filter(
(log) => log.response_status >= 200 && log.response_status < 300,
).length;
const warning = filteredLogs.filter(
(log) => log.response_status >= 400 && log.response_status < 500,
).length;
const error = filteredLogs.filter((log) => log.response_status >= 500).length;
aggregatedData.push({
date: format(timestamp, "yyyy-MM-dd'T'HH:mm:ss"),
success,
warning,
error,
});
}
return aggregatedData;
}
function aggregateData(data: Log[]) {
const aggregatedData: {
date: string;
success: number;
warning: number;
error: number;
}[] = [];
const intervalMs = 60 * 1000 * 10;
if (data.length === 0) {
return aggregatedData;
}
// Sort the data by timestamp
const sortedData = data.slice().sort((a, b) => a.time - b.time);
let intervalStart = new Date(sortedData[0].time).setMinutes(0, 0, 0);
let intervalEnd = intervalStart + intervalMs;
let index = 0;
while (intervalStart < sortedData[sortedData.length - 1].time) {
let success = 0;
let warning = 0;
let error = 0;
while (index < sortedData.length && sortedData[index].time < intervalEnd) {
const log = sortedData[index];
if (log.response_status >= 200 && log.response_status < 300) {
success += 1;
} else if (log.response_status >= 400 && log.response_status < 500) {
warning += 1;
} else if (log.response_status >= 500) {
error += 1;
}
index += 1;
}
aggregatedData.push({
date: format(intervalStart, "yyyy-MM-dd'T'HH:mm:ss"),
success,
warning,
error,
});
intervalStart = intervalEnd;
intervalEnd += intervalMs;
}
return aggregatedData;
}

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah that feels a bit better, our arrays aren't huge, but as we might add more status types, we could end up looping over them quite a few times.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@chronark, I'm glad the refactored version feels better. Since you might add more status types, we can modify the aggregateData function to make it more flexible and scalable. By defining the status categories in a configuration array, we can easily add new types without changing the core aggregation logic.

Here's how you could refactor the code:

  1. Define the status categories:
const statusCategories = [
  {
    key: "success",
    label: "Success",
    range: [200, 299],
    color: "hsl(var(--chart-3))",
  },
  {
    key: "warning",
    label: "Warning",
    range: [400, 499],
    color: "hsl(var(--chart-4))",
  },
  {
    key: "error",
    label: "Error",
    range: [500, 599],
    color: "hsl(var(--chart-1))",
  },
  // Add more categories as needed
];
  1. Update chartConfig dynamically:
const chartConfig = statusCategories.reduce((config, category) => {
  config[category.key] = {
    label: category.label,
    color: category.color,
  };
  return config;
}, {} as ChartConfig);
  1. Refactor the aggregateData function:
function aggregateData(data: Log[]) {
  const aggregatedData: {
    date: string;
    [key: string]: number | string;
  }[] = [];

  const intervalMs = 10 * 60 * 1000; // 10 minutes

  if (data.length === 0) {
    return aggregatedData;
  }

  // Sort data by timestamp
  const sortedData = data.slice().sort((a, b) => a.time - b.time);

  let intervalStart = new Date(sortedData[0].time).setMinutes(0, 0, 0);
  let intervalEnd = intervalStart + intervalMs;
  let index = 0;

  while (intervalStart < sortedData[sortedData.length - 1].time) {
    const intervalCounts = statusCategories.reduce((counts, category) => {
      counts[category.key] = 0;
      return counts;
    }, {} as { [key: string]: number });

    while (index < sortedData.length && sortedData[index].time < intervalEnd) {
      const log = sortedData[index];
      for (const category of statusCategories) {
        if (
          log.response_status >= category.range[0] &&
          log.response_status <= category.range[1]
        ) {
          intervalCounts[category.key] += 1;
          break;
        }
      }
      index += 1;
    }

    aggregatedData.push({
      date: format(intervalStart, "yyyy-MM-dd'T'HH:mm:ss"),
      ...intervalCounts,
    });

    intervalStart = intervalEnd;
    intervalEnd += intervalMs;
  }

  return aggregatedData;
}

With this approach:

  • Easy Extension: You can add new status types by simply updating the statusCategories array without modifying the aggregation logic.
  • Maintainability: The code becomes more maintainable and easier to understand.
  • Scalability: It efficiently handles larger datasets and more status categories.

Let me know if you have any questions or need further assistance!


export function LogsChart({ logs }: { logs: Log[] }) {
const data = aggregateData(logs);

return (
<ChartContainer config={chartConfig} className="h-[125px] w-full">
<BarChart accessibilityLayer data={data}>
<XAxis
dataKey="date"
tickLine={false}
axisLine={false}
dx={-40}
tickFormatter={(value) => {
const date = new Date(value);
return date.toLocaleDateString("en-US", {
month: "short",
day: "2-digit",
hour: "2-digit",
minute: "2-digit",
hour12: true,
});
}}
/>
<CartesianGrid strokeDasharray="2" stroke="#DFE2E6" vertical={false} />
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how will this work in dark/light mode?
can we reference a css variable instead so it works automatically?

<ChartTooltip
content={
<ChartTooltipContent
className="w-[200px]"
nameKey="views"
labelFormatter={(value) => {
return new Date(value).toLocaleDateString("en-US", {
month: "short",
day: "numeric",
year: "numeric",
hour: "2-digit",
minute: "2-digit",
hour12: true,
});
}}
/>
}
/>
<Bar dataKey="success" stackId="a" fill="var(--color-success)" radius={3} />
<Bar dataKey="warning" stackId="a" fill="var(--color-warning)" radius={3} />
<Bar dataKey="error" stackId="a" fill="var(--color-error)" radius={3} />
</BarChart>
</ChartContainer>
);
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,162 @@
"use client";

import { addHours, format, setHours, setMinutes, setSeconds } from "date-fns";
import type { DateRange } from "react-day-picker";

import { Button } from "@/components/ui/button";
import { Calendar } from "@/components/ui/calendar";
import { Popover, PopoverContent, PopoverTrigger } from "@/components/ui/popover";
import { cn } from "@/lib/utils";
import { ArrowRight, Calendar as CalendarIcon } from "lucide-react";
import { useEffect, useState } from "react";
import { useLogSearchParams } from "../../../query-state";
import TimeSplitInput from "./time-split";

export function DatePickerWithRange({ className }: React.HTMLAttributes<HTMLDivElement>) {
const [interimDate, setInterimDate] = useState<DateRange>({
from: new Date(),
to: new Date(),
});
const [finalDate, setFinalDate] = useState<DateRange>();
const [startTime, setStartTime] = useState({ HH: "09", mm: "00", ss: "00" });
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

where do these defaults come from?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's just a random starting hour. Do you have something else on your mind?

const [endTime, setEndTime] = useState({ HH: "17", mm: "00", ss: "00" });
const [open, setOpen] = useState(false);
const { searchParams, setSearchParams } = useLogSearchParams();

useEffect(() => {
if (searchParams.startTime && searchParams.endTime) {
const from = new Date(searchParams.startTime);
const to = new Date(searchParams.endTime);
setFinalDate({ from, to });
setInterimDate({ from, to });
setStartTime({
HH: from.getHours().toString().padStart(2, "0"),
mm: from.getMinutes().toString().padStart(2, "0"),
ss: from.getSeconds().toString().padStart(2, "0"),
});
setEndTime({
HH: to.getHours().toString().padStart(2, "0"),
mm: to.getMinutes().toString().padStart(2, "0"),
ss: to.getSeconds().toString().padStart(2, "0"),
});
}
}, [searchParams.startTime, searchParams.endTime]);
ogzhanolguncu marked this conversation as resolved.
Show resolved Hide resolved

const handleFinalDate = (interimDate: DateRange | undefined) => {
setOpen(false);

if (interimDate?.from) {
let mergedFrom = setHours(interimDate.from, Number(startTime.HH));
addHours(interimDate?.from, Number(startTime.HH));
ogzhanolguncu marked this conversation as resolved.
Show resolved Hide resolved
mergedFrom = setMinutes(mergedFrom, Number(startTime.mm));
mergedFrom = setSeconds(mergedFrom, Number(startTime.ss));

let mergedTo: Date;
if (interimDate.to) {
mergedTo = setHours(interimDate.to, Number(endTime.HH));
mergedTo = setMinutes(mergedTo, Number(endTime.mm));
mergedTo = setSeconds(mergedTo, Number(endTime.ss));
} else {
mergedTo = setHours(interimDate.from, Number(endTime.HH));
mergedTo = setMinutes(mergedTo, Number(endTime.mm));
mergedTo = setSeconds(mergedTo, Number(endTime.ss));
}

setFinalDate({ from: mergedFrom, to: mergedTo });
setSearchParams({
startTime: mergedFrom.getTime(),
endTime: mergedTo.getTime(),
});
} else {
setFinalDate(interimDate);
setSearchParams({
startTime: null,
endTime: null,
});
}
};

return (
<div className={cn("grid gap-2", className)}>
<Popover open={open} onOpenChange={setOpen}>
<PopoverTrigger asChild>
<div
id="date"
className={cn(
"justify-start text-left font-normal flex gap-2 items-center",
!finalDate && "text-muted-foreground",
)}
>
<div className="flex gap-2 items-center w-fit">
<div>
<CalendarIcon className="h-4 w-4" />
</div>
{finalDate?.from ? (
finalDate.to ? (
<div className="truncate">
{format(finalDate.from, "LLL dd, y")} - {format(finalDate.to, "LLL dd, y")}
</div>
) : (
format(finalDate.from, "LLL dd, y")
)
) : (
<span>Custom</span>
)}
</div>
</div>
</PopoverTrigger>
<PopoverContent className="w-auto p-0 bg-background">
<Calendar
initialFocus
mode="range"
defaultMonth={interimDate?.from}
selected={interimDate}
onSelect={(date) =>
setInterimDate({
from: date?.from,
to: date?.to,
})
}
/>
ogzhanolguncu marked this conversation as resolved.
Show resolved Hide resolved
<div className="flex flex-col gap-2">
<div className="border-t border-border" />
<div className="flex gap-2 items-center w-full justify-evenly">
<TimeSplitInput
type="start"
startTime={startTime}
endTime={endTime}
time={startTime}
setTime={setStartTime}
setStartTime={setStartTime}
setEndTime={setEndTime}
startDate={interimDate.from}
endDate={interimDate.to}
/>
<ArrowRight strokeWidth={1.5} size={14} />
<TimeSplitInput
type="end"
startTime={startTime}
endTime={endTime}
time={endTime}
setTime={setEndTime}
setStartTime={setStartTime}
setEndTime={setEndTime}
startDate={interimDate.from}
endDate={interimDate.to}
/>
</div>
<div className="border-t border-border" />
</div>
<div className="flex gap-2 p-2 w-full justify-end bg-background-subtle">
<Button size="sm" variant="outline" onClick={() => handleFinalDate(undefined)}>
Clear
</Button>
<Button size="sm" variant="primary" onClick={() => handleFinalDate(interimDate)}>
Apply
</Button>
</div>
</PopoverContent>
</Popover>
</div>
);
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
import { Checkbox } from "@/components/ui/checkbox";
import { Popover, PopoverContent, PopoverTrigger } from "@/components/ui/popover";
import React, { useEffect, useState } from "react";
import { type ResponseStatus as Status, useLogSearchParams } from "../../../query-state";

interface CheckboxItemProps {
id: string;
label: string;
description: string;
checked: boolean;
onCheckedChange: (checked: boolean) => void;
}

const checkboxItems = [
{ id: "500", label: "Error", description: "500 error codes" },
ogzhanolguncu marked this conversation as resolved.
Show resolved Hide resolved
{ id: "200", label: "Success", description: "200 success codes" },
{ id: "400", label: "Warning", description: "400 warning codes" },
];

export const ResponseStatus = () => {
const [open, setOpen] = useState(false);
const { searchParams, setSearchParams } = useLogSearchParams();
const [checkedItems, setCheckedItems] = useState<Status[]>([]);

useEffect(() => {
if (searchParams.responseStatus) {
setCheckedItems(searchParams.responseStatus);
}
}, [searchParams.responseStatus]);

const handleItemChange = (status: Status, checked: boolean) => {
const newCheckedItems = checked
? [...checkedItems, status]
: checkedItems.filter((item) => item !== status);

setCheckedItems(newCheckedItems);
setSearchParams((prevState) => ({
...prevState,
responseStatus: newCheckedItems.length > 0 ? newCheckedItems : null,
}));
};

const getStatusDisplay = () => {
if (checkedItems.length === 0) {
return "Response Status";
}

const statusLabels = checkedItems
.map((status) => checkboxItems.find((item) => Number(item.id) === status)?.label)
.filter(Boolean)
.join(", ");

return `Response Status (${statusLabels})`;
};

return (
<Popover open={open} onOpenChange={setOpen}>
<PopoverTrigger asChild>
<div className="cursor-pointer">{getStatusDisplay()}</div>
</PopoverTrigger>
<PopoverContent className="w-80 bg-background p-0">
{checkboxItems.map((item, index) => (
<React.Fragment key={item.id}>
<CheckboxItem
{...item}
checked={checkedItems.includes(Number(item.id) as Status)}
onCheckedChange={(checked) => {
handleItemChange(Number(item.id) as Status, checked);
}}
/>
{index < checkboxItems.length - 1 && <div className="border-b border-border" />}
</React.Fragment>
))}
</PopoverContent>
</Popover>
);
};

const CheckboxItem = ({ id, label, description, checked, onCheckedChange }: CheckboxItemProps) => (
<div className="items-top flex space-x-2 p-4">
<Checkbox id={id} checked={checked} onCheckedChange={onCheckedChange} />
<div className="grid gap-1.5 leading-none">
<label
htmlFor={id}
className="text-sm font-medium leading-none peer-disabled:cursor-not-allowed peer-disabled:opacity-70"
>
{label}
</label>
<p className="text-xs text-content-subtle">{description}</p>
</div>
</div>
);
Loading
Loading