Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feature: support set extension for dump file #130

Closed
wants to merge 3 commits into from
Closed

feature: support set extension for dump file #130

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
4e88570
feature: support set extension for dump file
almas1992 Nov 10, 2022
3aaa888
doc: add doc in README
almas1992 Nov 14, 2022
a472151
doc: remove unused value and upgrade gopsutil
almas1992 Nov 29, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 6 additions & 7 deletions consts.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,13 +48,12 @@ const (
defaultThreadCoolDown = time.Hour
defaultGoroutineCoolDown = time.Minute * 10

defaultInterval = 5 * time.Second
defaultDumpProfileType = binaryDump
defaultDumpPath = "/tmp"
defaultLoggerName = "holmes.log"
defaultLoggerFlags = os.O_RDWR | os.O_CREATE | os.O_APPEND
defaultLoggerPerm = 0644
defaultShardLoggerSize = 5242880 // 5m
defaultInterval = 5 * time.Second
defaultDumpProfileType = binaryDump
defaultDumpPath = "/tmp"
defaultDumpFileExtension = ".log"
defaultLoggerFlags = os.O_RDWR | os.O_CREATE | os.O_APPEND
defaultLoggerPerm = 0644
)

type dumpProfileType int
Expand Down
2 changes: 2 additions & 0 deletions doc/zh.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,7 @@ h, _ := holmes.New(
holmes.WithCollectInterval("5s"),
holmes.WithDumpPath("/tmp"),
holmes.WithTextDump(),
holmes.WithDumpFileExtension(".out"),
holmes.WithDumpToLogger(true),
holmes.WithGoroutineDump(10, 25, 2000, 10*1000, time.Minute),
)
Expand All @@ -97,6 +98,7 @@ h.Stop()
* WithCollectInterval("5s") 每5s采集一次当前应用的各项指标，该值建议设置为大于1s。
* WithDumpPath("/tmp") profile文件保存路径。
* WithTextDump() 以文本格式保存profile内容。
* WithDumpFileExtension() 设置保存的profile文件的扩展名，默认是`.log`。
* WithDumpToLogger() profile内容将会输出到日志。
* WithGoroutineDump(10, 25, 2000, 100*1000, time.Minute) 当goroutine指标满足以下条件时，将会触发dump操作。
current_goroutine_num > `10` && current_goroutine_num < `100*1000` &&
Expand Down
4 changes: 3 additions & 1 deletion go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,9 @@ go 1.14

require (
github.com/gin-gonic/gin v1.7.7
github.com/shirou/gopsutil v3.20.11+incompatible
github.com/shirou/gopsutil v3.21.11+incompatible
github.com/stretchr/testify v1.7.0
github.com/tklauser/go-sysconf v0.3.11 // indirect
github.com/yusufpapurcu/wmi v1.2.2 // indirect
mosn.io/pkg v0.0.0-20211217101631-d914102d1baf
)
17 changes: 13 additions & 4 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,6 @@ github.com/Knetic/govaluate v3.0.1-0.20171022003610-9aa49832a739+incompatible/go
github.com/OneOfOne/xxhash v1.2.2/go.mod h1:HSdplMjZKSmBqAxg5vPj2TmRDmfkzw+cTzAElWljhcU=
github.com/Shopify/sarama v1.19.0/go.mod h1:FVkBWblsNy7DGZRfXLU0O9RCGt5g3g3yEuWXgklEdEo=
github.com/Shopify/toxiproxy v2.1.4+incompatible/go.mod h1:OXgGpZ6Cli1/URJOF1DMxUHB2q5Ap20/P/eIdh4G0pI=
github.com/StackExchange/wmi v0.0.0-20190523213315-cbe66965904d h1:G0m3OIz70MZUWq3EgK3CesDbo8upS2Vm9/P3FtgI+Jk=
github.com/StackExchange/wmi v0.0.0-20190523213315-cbe66965904d/go.mod h1:3eOhrUMpNV+6aFIbp5/iudMxNCF27Vw2OZgy4xEx0Fg=
github.com/VividCortex/gohistogram v1.0.0/go.mod h1:Pf5mBqqDxYaXu3hDrrU+w6nw50o/4+TcAqDqk/vUH7g=
github.com/afex/hystrix-go v0.0.0-20180502004556-fa1af6a1f4f5/go.mod h1:SkGFH1ia65gfNATL8TAiHDNxPzPdmEL5uirI2Uyuz6c=
Expand Down Expand Up @@ -118,8 +117,9 @@ github.com/go-kit/kit v0.10.0/go.mod h1:xUsJbQ/Fp4kEt7AFgCuvyX4a71u8h9jB8tj/ORgO
github.com/go-logfmt/logfmt v0.3.0/go.mod h1:Qt1PoO58o5twSAckw1HlFXLmHsOX5/0LbT9GBnD5lWE=
github.com/go-logfmt/logfmt v0.4.0/go.mod h1:3RMwSq7FuexP4Kalkev3ejPJsZTpXXBr9+V4qmtdjCk=
github.com/go-logfmt/logfmt v0.5.0/go.mod h1:wCYkCAKZfumFQihp8CzCvQ3paCTfi41vtzG1KdI/P7A=
github.com/go-ole/go-ole v1.2.4 h1:nNBDSCOigTSiarFpYE9J/KtEA1IOW4CNeqT9TQDqCxI=
github.com/go-ole/go-ole v1.2.4/go.mod h1:XCwSNxSkXRo4vlyPy93sltvi/qJq0jqQhjqQNIwKuxM=
github.com/go-ole/go-ole v1.2.6 h1:/Fpf6oFPoeFik9ty7siob0G6Ke8QvQEuVcuChpwXzpY=
github.com/go-ole/go-ole v1.2.6/go.mod h1:pprOEPIfldk/42T2oK7lQ4v4JSDwmV0As9GaiUsvbm0=
github.com/go-playground/assert/v2 v2.0.1 h1:MsBgLAaY856+nPRTKrp3/OZK38U/wa0CcBYNjji3q3A=
github.com/go-playground/assert/v2 v2.0.1/go.mod h1:VDjEfimB/XKnb+ZQfWdccd7VUvScMdVu0Titje2rxJ4=
github.com/go-playground/locales v0.13.0 h1:HyWk6mgj5qFqCT5fjGBuRArbVDfE4hi8+e8ceBS/t7Q=
Expand Down Expand Up @@ -410,8 +410,9 @@ github.com/ryanuber/columnize v0.0.0-20160712163229-9b3edd62028f/go.mod h1:sm1tb
github.com/samuel/go-zookeeper v0.0.0-20190923202752-2cc03de413da/go.mod h1:gi+0XIa01GRL2eRQVjQkKGqKF3SF9vZR/HnPullcV2E=
github.com/satori/go.uuid v1.2.0/go.mod h1:dA0hQrYB0VpLJoorglMZABFdXlWrHn1NEOzdhQKdks0=
github.com/sean-/seed v0.0.0-20170313163322-e2103e2c3529/go.mod h1:DxrIzT+xaE7yg65j358z/aeFdxmN0P9QXhEzd20vsDc=
github.com/shirou/gopsutil v3.20.11+incompatible h1:LJr4ZQK4mPpIV5gOa4jCOKOGb4ty4DZO54I4FGqIpto=
github.com/shirou/gopsutil v3.20.11+incompatible/go.mod h1:5b4v6he4MtMOwMlS0TUMTu2PcXUg8+E1lC7eC3UO/RA=
github.com/shirou/gopsutil v3.21.11+incompatible h1:+1+c1VGhc88SSonWP6foOcLhvnKlUeu/erjjvaPEYiI=
github.com/shirou/gopsutil v3.21.11+incompatible/go.mod h1:5b4v6he4MtMOwMlS0TUMTu2PcXUg8+E1lC7eC3UO/RA=
github.com/shurcooL/sanitized_anchor_name v1.0.0/go.mod h1:1NzhyTcUVG4SuEtjjoZeVRXNmyL/1OwPU0+IJeTBvfc=
github.com/sirupsen/logrus v1.2.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo=
github.com/sirupsen/logrus v1.4.2/go.mod h1:tLMulIdttU9McNUspp0xgXVQah82FyeX6MwdIuYE2rE=
Expand Down Expand Up @@ -446,6 +447,10 @@ github.com/stretchr/testify v1.7.0 h1:nwc3DEeHmmLAfoZucVR881uASk0Mfjw8xYJ99tb5Cc
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/subosito/gotenv v1.2.0/go.mod h1:N0PQaV/YGNqwC0u51sEeR/aUtSLEXKX9iv69rRypqCw=
github.com/tebeka/strftime v0.1.3/go.mod h1:7wJm3dZlpr4l/oVK0t1HYIc4rMzQ2XJlOMIUJUJH6XQ=
github.com/tklauser/go-sysconf v0.3.11 h1:89WgdJhk5SNwJfu+GKyYveZ4IaJ7xAkecBo+KdJV0CM=
github.com/tklauser/go-sysconf v0.3.11/go.mod h1:GqXfhXY3kiPa0nAXPDIQIWzJbMCB7AmcWpGR8lSZfqI=
github.com/tklauser/numcpus v0.6.0 h1:kebhY2Qt+3U6RNK7UqpYNA+tJ23IBEGKkB7JQBfDYms=
github.com/tklauser/numcpus v0.6.0/go.mod h1:FEZLMke0lhOUG6w2JadTzp0a+Nl8PF/GFkQ5UVIcaL4=
github.com/tmc/grpc-websocket-proxy v0.0.0-20170815181823-89b8d40f7ca8/go.mod h1:ncp9v5uamzpCO7NfCPTXjqaC+bZgJeR0sMTm6dMHP7U=
github.com/tmc/grpc-websocket-proxy v0.0.0-20190109142713-0ad062ec5ee5/go.mod h1:ncp9v5uamzpCO7NfCPTXjqaC+bZgJeR0sMTm6dMHP7U=
github.com/tmc/grpc-websocket-proxy v0.0.0-20200427203606-3cfed13b9966/go.mod h1:ncp9v5uamzpCO7NfCPTXjqaC+bZgJeR0sMTm6dMHP7U=
Expand All @@ -463,6 +468,8 @@ github.com/valyala/tcplisten v0.0.0-20161114210144-ceec8f93295a/go.mod h1:v3UYOV
github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2/go.mod h1:UETIi67q53MR2AWcXfiuqkDkRtnGDLqkBTpCHuJHxtU=
github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
github.com/yusufpapurcu/wmi v1.2.2 h1:KBNDSne4vP5mbSWnJbO+51IMOXJB67QiYCSBrubbPRg=
github.com/yusufpapurcu/wmi v1.2.2/go.mod h1:SBZ9tNy3G9/m5Oi98Zks0QjeHVDvuK0qfxQmPyzfmi0=
go.etcd.io/bbolt v1.3.2/go.mod h1:IbVyRI1SCnLcuJnV2u8VeU0CEYM7e686BmAb1XKL+uU=
go.etcd.io/bbolt v1.3.3/go.mod h1:IbVyRI1SCnLcuJnV2u8VeU0CEYM7e686BmAb1XKL+uU=
go.etcd.io/bbolt v1.3.5/go.mod h1:G5EMThwa9y8QZGBClrRx5EY+Yw9kAhnjy3bSjsnlVTQ=
Expand Down Expand Up @@ -586,6 +593,7 @@ golang.org/x/sys v0.0.0-20190606165138-5da285871e9c/go.mod h1:h1NjWce9XRLGQEsW7w
golang.org/x/sys v0.0.0-20190624142023-c5567b49c5d0/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190726091711-fc99dfbffb4e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190826190057-c7b8b68b1456/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190916202348-b4ddaad3f8a3/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191001151750-bb3f8db39f24/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191026070338-33540a1f6037/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191220142924-d4481acd189f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
Expand All @@ -604,8 +612,9 @@ golang.org/x/sys v0.0.0-20201009025420-dfb3f7c4e634/go.mod h1:h1NjWce9XRLGQEsW7w
golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20201214210602-f9fddec55a1e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20201223074533-0d417f636930/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210809222454-d867a43fc93e h1:WUoyKPm6nCo1BnNUvPGnFG3T5DUVem42yDJZZ4CNxMA=
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seems this change leads to the CI failure, since the CI use go1.14.
https://github.com/mosn/holmes/actions/runs/3573887532/jobs/6020361017

# golang.org/x/sys/unix
Error: ../../../go/pkg/mod/golang.org/x/[email protected]/unix/syscall.go:83:16: undefined: unsafe.Slice
Error: ../../../go/pkg/mod/golang.org/x/[email protected]/unix/syscall_linux.go:2256:9: undefined: unsafe.Slice
Error: ../../../go/pkg/mod/golang.org/x/[email protected]/unix/syscall_unix.go:118:7: undefined: unsafe.Slice
Error: ../../../go/pkg/mod/golang.org/x/[email protected]/unix/sysvshm_unix.go:33:7: undefined: unsafe.Slice

golang.org/x/sys v0.0.0-20210809222454-d867a43fc93e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.2.0 h1:ljd4t30dBnAvMZaQCevtY0xLLD0A+bRZXbgLMLU1F/A=
golang.org/x/sys v0.2.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
Expand Down
2 changes: 1 addition & 1 deletion holmes.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -598,7 +598,7 @@ func (h *Holmes) cpuProfile(curCPUUsage int, c typeOption) bool {
c.TriggerMin, c.TriggerDiff, c.TriggerAbs, NotSupportTypeMaxConfig,
h.cpuStats.sequentialData(), curCPUUsage)

bf, binFileName, err := getBinaryFileNameAndCreate(h.opts.DumpPath, cpu, "")
bf, binFileName, err := getBinaryFileNameAndCreate(h.opts.DumpPath, h.opts.DumpFileExtension, cpu, "")
if err != nil {
h.Errorf("[Holmes] failed to create cpu profile file: %v", err.Error())
return false
Expand Down
31 changes: 22 additions & 9 deletions options.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,8 @@ type DumpOptions struct {
DumpFullStack bool
// dump profile to logger. It will make huge log output if enable DumpToLogger option. issues/90
DumpToLogger bool
// set the dump file extension, default is .log
DumpFileExtension string
}

// ShrinkThrOptions contains the configuration about shrink thread
Expand Down Expand Up @@ -174,9 +176,10 @@ func newOptions() *options {
CollectInterval: defaultInterval,
intervalResetting: make(chan struct{}, 1),
DumpOptions: &DumpOptions{
DumpPath: defaultDumpPath,
DumpProfileType: defaultDumpProfileType,
DumpFullStack: false,
DumpPath: defaultDumpPath,
DumpProfileType: defaultDumpProfileType,
DumpFullStack: false,
DumpFileExtension: defaultDumpFileExtension,
},
ShrinkThrOptions: &ShrinkThrOptions{
Enable: false,
Expand Down Expand Up @@ -204,6 +207,14 @@ func WithDumpPath(dumpPath string) Option {
})
}

// WithDumpFileExtension set the dump file extension for holmes.
func WithDumpFileExtension(extension string) Option {
return optionFunc(func(opts *options) (err error) {
opts.DumpFileExtension = extension
return
})
}

// WithCollectInterval : interval must be valid time duration string,
// eg. "ns", "us" (or "µs"), "ms", "s", "m", "h".
func WithCollectInterval(interval string) Option {
Expand Down Expand Up @@ -320,8 +331,8 @@ func (base *typeOption) Set(min, abs, diff int, coolDown time.Duration) {

// newMemOptions
// enable the heap dumper, should dump if one of the following requirements is matched
// 1. memory usage > TriggerMin && memory usage diff > TriggerDiff
// 2. memory usage > TriggerAbs.
// 1. memory usage > TriggerMin && memory usage diff > TriggerDiff
// 2. memory usage > TriggerAbs.
func newMemOptions() *typeOption {
return newTypeOpts(
defaultMemTriggerMin,
Expand All @@ -341,8 +352,9 @@ func WithMemDump(min int, diff int, abs int, coolDown time.Duration) Option {

// newGCHeapOptions
// enable the heap dumper, should dump if one of the following requirements is matched
// 1. GC heap usage > TriggerMin && GC heap usage diff > TriggerDiff
// 2. GC heap usage > TriggerAbs
// 1. GC heap usage > TriggerMin && GC heap usage diff > TriggerDiff
// 2. GC heap usage > TriggerAbs
//
// in percent.
func newGCHeapOptions() *typeOption {
return newTypeOpts(
Expand Down Expand Up @@ -398,8 +410,9 @@ func WithThreadDump(min, diff, abs int, coolDown time.Duration) Option {
// newCPUOptions
// enable the cpu dumper, should dump if one of the following requirements is matched
// in percent
// 1. cpu usage > CPUTriggerMin && cpu usage diff > CPUTriggerDiff
// 2. cpu usage > CPUTriggerAbs
// 1. cpu usage > CPUTriggerMin && cpu usage diff > CPUTriggerDiff
// 2. cpu usage > CPUTriggerAbs
//
// in percent.
func newCPUOptions() *typeOption {
return newTypeOpts(
Expand Down
2 changes: 2 additions & 0 deletions readme.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,7 @@ h, _ := holmes.New(
holmes.WithCollectInterval("5s"),
holmes.WithDumpPath("/tmp"),
holmes.WithTextDump(),
holmes.WithDumpFileExtension(".out"),
holmes.WithDumpToLogger(true),
holmes.WithGoroutineDump(10, 25, 2000, 10*1000,time.Minute),
)
Expand All @@ -68,6 +69,7 @@ h.Stop()
* WithCollectInterval("5s") means the system metrics are collected once 5 seconds
* WithDumpPath("/tmp") means the dump binary file(binary mode) will write content to `/tmp` dir.
* WithTextDump() means not in binary mode, so it's text mode profiles
* WithDumpFileExtension() set the extension of the dump profile,default is `.log`
* WithDumpToLogger() means profiles information will be outputted to logger.
* WithGoroutineDump(10, 25, 2000, 100*1000,time.Minute) means dump will happen when current_goroutine_num > 10 &&
current_goroutine_num < `100*1000` && current_goroutine_num > `125%` * previous_average_goroutine_num or current_goroutine_num > `2000`,
Expand Down
10 changes: 5 additions & 5 deletions util.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -194,8 +194,8 @@ func matchRule(history ring, curVal, ruleMin, ruleAbs, ruleDiff, ruleMax int) (b
return false, ReasonCurlGreaterMin
}

func getBinaryFileName(filePath string, dumpType configureType, eventID string) string {
suffix := time.Now().Format("20060102150405.000") + ".log"
func getBinaryFileName(filePath, fileExtension string, dumpType configureType, eventID string) string {
suffix := time.Now().Format("20060102150405.000") + fileExtension
if len(eventID) == 0 {
return path.Join(filePath, check2name[dumpType]+"."+suffix)
}
Expand All @@ -204,8 +204,8 @@ func getBinaryFileName(filePath string, dumpType configureType, eventID string)
}

// fix #89
func getBinaryFileNameAndCreate(dump string, dumpType configureType, eventID string) (*os.File, string, error) {
filepath := getBinaryFileName(dump, dumpType, eventID)
func getBinaryFileNameAndCreate(dump, extension string, dumpType configureType, eventID string) (*os.File, string, error) {
filepath := getBinaryFileName(dump, extension, dumpType, eventID)
f, err := os.OpenFile(filepath, defaultLoggerFlags, defaultLoggerPerm)
if err != nil && os.IsNotExist(err) {
if err = os.MkdirAll(dump, 0o755); err != nil {
Expand Down Expand Up @@ -234,7 +234,7 @@ func writeFile(data bytes.Buffer, dumpType configureType, dumpOpts *DumpOptions,
buf = data.Bytes()
}

file, fileName, err := getBinaryFileNameAndCreate(dumpOpts.DumpPath, dumpType, eventID)
file, fileName, err := getBinaryFileNameAndCreate(dumpOpts.DumpPath, dumpOpts.DumpFileExtension, dumpType, eventID)
if err != nil {
return fileName, fmt.Errorf("pprof %v open file failed : %w", type2name[dumpType], err)
}
Expand Down