Add Huawei Cloud OBS Object Storage Support (#50)

* add obs support Signed-off-by: setoru <[email protected]> * fix obs code style Signed-off-by: setoru <[email protected]> * add factory config for obs Signed-off-by: setoru <[email protected]> * add obs config to bucketcfggen Signed-off-by: setoru <[email protected]> * add changelog entry Signed-off-by: setoru <[email protected]> * fix imports Signed-off-by: setoru <[email protected]> * fix return readable error Signed-off-by: setoru <[email protected]> * fix makefile and readme Signed-off-by: setoru <[email protected]> * fix readability problem and comment Signed-off-by: setoru <[email protected]> * fix comment Signed-off-by: setoru <[email protected]> --------- Signed-off-by: setoru <[email protected]>
thanos-io · May 22, 2023 · 23ebe2e · 23ebe2e
1 parent d1711f1
commit 23ebe2e
Show file tree

Hide file tree

Showing 10 changed files with 493 additions and 18 deletions.
diff --git a/.github/workflows/test.yaml b/.github/workflows/test.yaml
@@ -69,7 +69,7 @@ jobs:
       - name: Run unit tests
         env:
           # THANOS_TEST_OBJSTORE_SKIP: AZURE,COS,ALIYUNOSS,BOS
-          THANOS_TEST_OBJSTORE_SKIP: GCS,S3,SWIFT,AZURE,COS,ALIYUNOSS,BOS,OCI
+          THANOS_TEST_OBJSTORE_SKIP: GCS,S3,SWIFT,AZURE,COS,ALIYUNOSS,BOS,OCI,OBS
           # Variables for Swift testing.
           OS_AUTH_URL: http://127.0.0.1:5000/v2.0
           OS_PASSWORD: s3cr3t

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -21,6 +21,7 @@ We use *breaking :warning:* to mark changes that are not backward compatible (re
 - [#41](https://github.com/thanos-io/objstore/pull/41) S3: Support S3 session token.
 - [#43](https://github.com/thanos-io/objstore/pull/43) filesystem: abort filesystem bucket operations if the context has been cancelled
 - [#44](https://github.com/thanos-io/objstore/pull/44) Add new metric to count total number of fetched bytes from bucket
+- [#50](https://github.com/thanos-io/objstore/pull/50) Add Huawei Cloud OBS Object Storage Support
 
 ### Changed
 - [#38](https://github.com/thanos-io/objstore/pull/38) *: Upgrade minio-go version to `v7.0.45`.

diff --git a/Makefile b/Makefile
@@ -5,7 +5,7 @@ MDOX_VALIDATE_CONFIG ?= .mdox.validate.yaml
 
 .PHONY: test-local
 test-local:
-	THANOS_TEST_OBJSTORE_SKIP=GCS,S3,AZURE,SWIFT,COS,ALIYUNOSS,BOS,OCI $(MAKE) test
+	THANOS_TEST_OBJSTORE_SKIP=GCS,S3,AZURE,SWIFT,COS,ALIYUNOSS,BOS,OCI,OBS $(MAKE) test
 
 .PHONY: test
 test:

diff --git a/README.md b/README.md
@@ -128,6 +128,7 @@ Current object storage client implementations:
 | [Baidu BOS](#baidu-bos)                                                                   | Beta               | Production Usage      | no                | @yahaa                           |
 | [Local Filesystem](#filesystem)                                                           | Stable             | Testing and Demo only | yes               | @bwplotka                        |
 | [Oracle Cloud Infrastructure Object Storage](#oracle-cloud-infrastructure-object-storage) | Beta               | Production Usage      | yes               | @aarontams,@gaurav-05,@ericrrath |
+| [HuaweiCloud OBS](#huaweicloud-obs)                                                       | Beta               | Production Usage      | no               | @setoru                          |
 
 **Missing support to some object storage?** Check out [how to add your client section](#how-to-add-a-new-client-to-thanos)
 
@@ -289,7 +290,7 @@ Example working AWS IAM policy for user:
 To test the policy, set env vars for S3 access for *empty, not used* bucket as well as:
 
 ```
-THANOS_TEST_OBJSTORE_SKIP=GCS,AZURE,SWIFT,COS,ALIYUNOSS,OCI
+THANOS_TEST_OBJSTORE_SKIP=GCS,AZURE,SWIFT,COS,ALIYUNOSS,OCI,OBS
 THANOS_ALLOW_EXISTING_BUCKET_USE=true
 ```
 
@@ -323,7 +324,7 @@ We need access to CreateBucket and DeleteBucket and access to all buckets:
 }
 ```
 
-With this policy you should be able to run set `THANOS_TEST_OBJSTORE_SKIP=GCS,AZURE,SWIFT,COS,ALIYUNOSS,OCI` and unset `S3_BUCKET` and run all tests using `make test`.
+With this policy you should be able to run set `THANOS_TEST_OBJSTORE_SKIP=GCS,AZURE,SWIFT,COS,ALIYUNOSS,OCI,OBS` and unset `S3_BUCKET` and run all tests using `make test`.
 
 Details about AWS policies: https://docs.aws.amazon.com/AmazonS3/latest/dev/using-with-s3-actions.html
 
@@ -640,6 +641,41 @@ config:
 
 You can also include any of the optional configuration just like the example in `Default Provider`.
 
+##### HuaweiCloud OBS
+
+To use HuaweiCloud OBS as an object store, you should apply for a HuaweiCloud Account to create an object storage bucket at first.
+More details: [HuaweiCloud OBS](https://support.huaweicloud.com/obs/index.html)
+
+To configure HuaweiCloud Account to use OBS as storage store you need to set these parameters in YAML format stored in a file:
+
+```yaml mdox-exec="go run scripts/cfggen/main.go --name=cos.Config"
+type: OBS                    
+config:                      
+  bucket: ""                 
+  endpoint: ""               
+  access_key: ""             
+  secret_key: ""
+  http_config:
+    idle_conn_timeout: 1m30s
+    response_header_timeout: 2m
+    insecure_skip_verify: false
+    tls_handshake_timeout: 10s
+    expect_continue_timeout: 1s
+    max_idle_conns: 100
+    max_idle_conns_per_host: 100
+    max_conns_per_host: 0
+    tls_config:
+      ca_file: ""
+      cert_file: ""
+      key_file: ""
+      server_name: ""
+      insecure_skip_verify: false
+    disable_compression: false
+prefix: ""
+```
+
+The `access_key` and `secret_key` field is required. The `http_config` field is optional for optimize HTTP transport settings. 
+
 #### How to add a new client to Thanos?
 
 Following checklist allows adding new Go code client to supported providers:

diff --git a/client/factory.go b/client/factory.go
@@ -8,22 +8,23 @@ import (
 	"fmt"
 	"strings"
 
-	"github.com/go-kit/log"
-	"github.com/go-kit/log/level"
-	"github.com/pkg/errors"
-	"github.com/prometheus/client_golang/prometheus"
-	"gopkg.in/yaml.v2"
-
 	"github.com/thanos-io/objstore"
 	"github.com/thanos-io/objstore/providers/azure"
 	"github.com/thanos-io/objstore/providers/bos"
 	"github.com/thanos-io/objstore/providers/cos"
 	"github.com/thanos-io/objstore/providers/filesystem"
 	"github.com/thanos-io/objstore/providers/gcs"
+	"github.com/thanos-io/objstore/providers/obs"
 	"github.com/thanos-io/objstore/providers/oci"
 	"github.com/thanos-io/objstore/providers/oss"
 	"github.com/thanos-io/objstore/providers/s3"
 	"github.com/thanos-io/objstore/providers/swift"
+
+	"github.com/go-kit/log"
+	"github.com/go-kit/log/level"
+	"github.com/pkg/errors"
+	"github.com/prometheus/client_golang/prometheus"
+	"gopkg.in/yaml.v2"
 )
 
 type ObjProvider string
@@ -38,6 +39,7 @@ const (
 	ALIYUNOSS  ObjProvider = "ALIYUNOSS"
 	BOS        ObjProvider = "BOS"
 	OCI        ObjProvider = "OCI"
+	OBS        ObjProvider = "OBS"
 )
 
 type BucketConfig struct {
@@ -80,6 +82,8 @@ func NewBucket(logger log.Logger, confContentYaml []byte, reg prometheus.Registe
 		bucket, err = bos.NewBucket(logger, config, component)
 	case string(OCI):
 		bucket, err = oci.NewBucket(logger, config)
+	case string(OBS):
+		bucket, err = obs.NewBucket(logger, config)
 	default:
 		return nil, errors.Errorf("bucket with type %s is not supported", bucketConf.Type)
 	}

diff --git a/go.mod b/go.mod
@@ -12,6 +12,7 @@ require (
 	github.com/efficientgo/e2e v0.13.1-0.20220922081603-45de9fc588a8
 	github.com/fatih/structtag v1.2.0
 	github.com/go-kit/log v0.2.1
+	github.com/huaweicloud/huaweicloud-sdk-go-obs v3.23.3+incompatible
 	github.com/minio/minio-go/v7 v7.0.45
 	github.com/ncw/swift v1.0.53
 	github.com/opentracing/opentracing-go v1.2.0

diff --git a/go.sum b/go.sum
@@ -264,6 +264,8 @@ github.com/googleapis/gax-go/v2 v2.3.0/go.mod h1:b8LNqSzNabLiUpXKkY7HAR5jr6bIT99
 github.com/grpc-ecosystem/grpc-gateway v1.16.0/go.mod h1:BDjrQk3hbvj6Nolgz8mAMFbcEtjT1g+wF4CSlocrBnw=
 github.com/hashicorp/golang-lru v0.5.0/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
 github.com/hashicorp/golang-lru v0.5.1/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
+github.com/huaweicloud/huaweicloud-sdk-go-obs v3.23.3+incompatible h1:tKTaPHNVwikS3I1rdyf1INNvgJXWSf/+TzqsiGbrgnQ=
+github.com/huaweicloud/huaweicloud-sdk-go-obs v3.23.3+incompatible/go.mod h1:l7VUhRbTKCzdOacdT4oWCwATKyvZqUOlOqr0Ous3k4s=
 github.com/ianlancetaylor/demangle v0.0.0-20181102032728-5e5cf60278f6/go.mod h1:aSSvb/t6k1mPoxDqO4vJh6VOCGPwU4O0C2/Eqndh1Sc=
 github.com/ianlancetaylor/demangle v0.0.0-20200824232613-28f6c0f3b639/go.mod h1:aSSvb/t6k1mPoxDqO4vJh6VOCGPwU4O0C2/Eqndh1Sc=
 github.com/jmespath/go-jmespath v0.4.0/go.mod h1:T8mJZnbsbmF+m6zOOFylbeCJqk5+pHWvzYPziyZiYoo=

diff --git a/objtesting/foreach.go b/objtesting/foreach.go
@@ -8,19 +8,20 @@ import (
 	"strings"
 	"testing"
 
-	"github.com/efficientgo/core/testutil"
-
 	"github.com/thanos-io/objstore"
 	"github.com/thanos-io/objstore/client"
 	"github.com/thanos-io/objstore/providers/azure"
 	"github.com/thanos-io/objstore/providers/bos"
 	"github.com/thanos-io/objstore/providers/cos"
 	"github.com/thanos-io/objstore/providers/filesystem"
 	"github.com/thanos-io/objstore/providers/gcs"
+	"github.com/thanos-io/objstore/providers/obs"
 	"github.com/thanos-io/objstore/providers/oci"
 	"github.com/thanos-io/objstore/providers/oss"
 	"github.com/thanos-io/objstore/providers/s3"
 	"github.com/thanos-io/objstore/providers/swift"
+
+	"github.com/efficientgo/core/testutil"
 )
 
 // IsObjStoreSkipped returns true if given provider ID is found in THANOS_TEST_OBJSTORE_SKIP array delimited by comma e.g:
@@ -183,4 +184,18 @@ func ForeachStore(t *testing.T, testFn func(t *testing.T, bkt objstore.Bucket))
 			testFn(t, bkt)
 		})
 	}
+
+	// Optional OBS.
+	if !IsObjStoreSkipped(t, client.OBS) {
+		t.Run("obs", func(t *testing.T) {
+			bkt, closeFn, err := obs.NewTestBucket(t, "cn-south-1")
+			testutil.Ok(t, err)
+
+			t.Parallel()
+			defer closeFn()
+
+			testFn(t, bkt)
+			testFn(t, objstore.NewPrefixedBucket(bkt, "some_prefix"))
+		})
+	}
 }