Who is using Apache Celeborn? #2140

SteNicholas · 2023-12-11T03:41:12Z

Search before asking

I searched in the issues and found nothing similar.

Motivation

Who's using Apache Celeborn?

The Apache Celeborn community grows with the combined efforts of every user and developer and appreciates every user and developer who uses and cares about Celeborn. We are committed to listening to each one of you to make the community more prosperous.

Therefore, the purpose of this topic is to encourage you, as a user or developer, to leave your real-world scenarios so that we can subsequently explore new and better solutions, schedule the releases, and solve real-world problems based on these scenarios.

We look forward to your comments in the comments space, including but not limited to the following areas.

Organization: e.g. Alibaba Corp.
Region: e.g. Hangzhou, China
Contact: e.g. [email protected]
Usage Scenario: e.g., we use Apache Celeborn to improve the efficiency and elasticity of different map-reduce engines and provides an elastic, high-efficient management service for intermediate data including shuffle data, spilled data, result data, etc.

Regards,

Apache Celeborn Community

Solution

No response

Anything else?

No response

Are you willing to submit a PR?

I'm willing to submit a PR!

gaochao0509 · 2023-12-11T03:57:44Z

Organization: New Baatur
Region: Shen zhen, China
Contact information: [email protected]
Usage scenario: Use Apache Celeborn to improve the efficiency and resilience of the Spark/MapReduce engine, providing elastic and efficient management services for intermediate data such as Shuffle data, overflow data, and result data.
Location: Currently, 10+locations have been deployed, with significant stability improvement and performance improvement of over 30%.

cxzl25 · 2023-12-11T07:06:56Z

Organization: Trip.com Group
Region: Shanghai, China
Contact: [email protected]
Usage Scenario: We use Apache Celeborn to improve the stability and efficiency of Spark shuffle.

big-loser · 2023-12-11T08:40:09Z

Organization: inmobi Group
Region: Beijing, China
Contact: [email protected]
Usage Scenario: We use Apache Celeborn to improve the stability and efficiency of Spark shuffle.

RuiQin7 · 2023-12-11T09:13:54Z

Organization: zto.com Group
Region: Shanghai, China
Contact: [email protected]
Usage Scenario: We use Apache Celeborn to improve the stability and efficiency of Spark shuffle

BsoBird · 2023-12-11T09:37:46Z

Organization: shuyun Inc
Region: hangzhou, China
Contact: [email protected]
Usage Scenario: By using apache celeborn, We realised the separation of storage and compute in HIVE-MR3/SPARK-KYUUBI, which effectively reduces the cost of the server room.

SteNicholas · 2023-12-11T12:29:46Z

Organization: Bilibili Inc
Region: Shanghai, China
Contact: [email protected]
Usage Scenario: We use Apache Celeborn to improve the stability and efficiency of Spark and Flink(Batch) engine shuffle.

jiaoqingbo · 2023-12-12T02:21:59Z

Organization: AsiaInfo
Region: Beijing, China
Contact: [email protected]
Usage Scenario: We use Apache Celeborn to improve the stability and efficiency of Spark shuffle.

lifulong · 2023-12-12T05:36:15Z

Organization: Zhihu Inc
Region: Beijing, China
Contact: [email protected]
Usage Scenario: We use Apache Celeborn to improve the stability and efficiency of Spark engine shuffle.

TongWei1105 · 2023-12-12T07:21:19Z

Organization: Xiaohongshu Inc
Region: Shanghai, China
Contact: [email protected]
Usage Scenario: We use Apache Celeborn to improve the stability and efficiency of Spark and Flink(Batch) engine shuffle.

onebox-li · 2023-12-12T08:03:18Z

Organization: BIGO
Region: Guangzhou, China
Contact: [email protected]
Usage Scenario: We use Apache Celeborn to improve the stability and efficiency of Spark shuffle.

AngersZhuuuu · 2023-12-12T09:12:13Z

Organization: SHOPEE
Region: Shenzhen, China
Contact: [email protected]
Usage Scenario: We use Apache Celeborn to improve the stability and efficiency of Spark shuffle.

lianneli · 2023-12-21T02:07:15Z

Organization: DMALL
Region: Beijing, China
Contact: [email protected], [email protected], [email protected]
Usage Scenario: We use Apache Celeborn to improve the stability and efficiency of Spark shuffle.

camper42 · 2024-01-16T10:31:30Z

Organization: Douban
Region: Beijing, China
Contact: [email protected]
Usage Scenario: we use Apache Celeborn to support better Spark dynamic resource allocation in Kubernetes.

wfxxh · 2024-02-27T02:48:05Z

Organization: WanFang Data
Region: Beijing, China
Contact: [email protected]
Usage Scenario: We use Apache Celeborn to improve the stability and efficiency of Spark shuffle on k8s

wxplovecc · 2024-03-01T03:39:34Z

Organization: Tongcheng Travel（同程旅行）.
Region: Jiangsu, China
Contact: www.ly.com
Usage Scenario: we use Apache Celeborn to improve the stability and efficiency of spark shuffle.

CodingCat · 2024-03-08T15:21:28Z

Organization: e.g. Pinterest
Region: US
Contact: @CodingCat
Usage Scenario: we use Apache Celeborn to improve the efficiency and elasticity of Spark@K8S

akpatnam25 · 2024-03-12T06:15:39Z

Organization: LinkedIn
Region: US
Contact: @akpatnam25
Usage Scenario: we use Apache Celeborn to improve the stability and efficiency of spark shuffle.

liugs0213 · 2024-03-20T09:40:15Z

Organization: BOSS Zhipin
Region: Beijing, China
Contact: @liugs0213
Usage Scenario: we use Apache Celeborn to improve the efficiency and elasticity of Spark@K8S

lohitvijayarenu · 2024-08-06T03:52:33Z

Organization: Stripe
Region: US
Contact: @lohitvijayarenu
Usage Scenario: Apache Celeborn helps Spark workloads with efficient and elastic shuffle. Celeborn will also help in Storage and compute disaggregation for efficient use of resources.

wangshengjie123 · 2024-08-09T10:44:02Z

Organization: Xiaomi
Region: Beijing, China
Contact: [email protected]
Usage Scenario: We use Celeborn to improve the stability and efficiency of shuffle.

ASiegeLion · 2024-08-12T07:57:48Z

Organization: YY
Region: GuangZhou, China
Contact: https://www.yy.com/web/jinhong/#/connect
Usage Scenario: We use Celeborn to improve the stability and efficiency of shuffle.

jinyuchen724 · 2024-08-15T09:57:58Z

Organization: WuXianShengHuo
Region: HangZhou, China
Contact: https://www.weidian.com
Usage Scenario: We use Celeborn to improve the stability and efficiency of shuffle.

HYBG-1126 · 2024-10-09T03:35:58Z

Organization: 竞技世界(jjworld)
Region: Beijing, China
Contact: https://www.jj.cn/
Usage Scenario: we use Apache Celeborn to improve the efficiency and elasticity of Spark@K8S

LennonChin · 2024-10-10T11:59:17Z

Organization: 蔚来（www.nio.com）
Region: Shanghai, China
Contact: @LennonChin
Usage Scenario: we use Apache Celeborn to improve the efficiency and elasticity of Spark on K8S.

conorzhao · 2024-11-19T07:25:35Z

Organization: 金城银行
Region: ShenZhen, China
Contact: https://www.kcbebank.com/
Usage Scenario: we use Apache Celeborn to improve the efficiency and elasticity of Spark Shuffle

lzw7916002 · 2024-11-26T03:07:59Z

Organization: TUYA
Region: Hang zhou, China
Contact information: https://www.tuya.com/cn/
Usage scenario: Use Apache Celeborn to improve the efficiency and resilience of the Spark engine, providing elastic and efficient management services for intermediate data such as Shuffle data, overflow data, and result data.
Location: Currently, 3+locations have been deployed, with significant stability improvement and performance improvement of over 20%.

lukaizj · 2024-11-27T03:48:57Z

Organization: 企查查
Region: Suzhou, China
Contact: https://www.qcc.com
Usage Scenario: we use Apache Celeborn to improve the efficiency and elasticity of Kyuubi+Spark on K8S.

zhisheng17 · 2024-12-18T03:29:24Z

Organization: 哈啰
Region: Shanghai, China
Contact: https://www.hello-inc.com/
Usage Scenario: We use Apache Celeborn in the Flink/Spark On K8s hybrid deployment scenario in the production environment to improve the stability of tasks and clusters, improve efficiency and ensure stability.

This comment was marked as resolved.

Sign in to view

cxzl25 pinned this issue Dec 11, 2023

apache deleted a comment from AngersZhuuuu Aug 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Who is using Apache Celeborn? #2140

Who is using Apache Celeborn? #2140

SteNicholas commented Dec 11, 2023

gaochao0509 commented Dec 11, 2023 •

edited

Loading

This comment was marked as resolved.

cxzl25 commented Dec 11, 2023

big-loser commented Dec 11, 2023 •

edited

Loading

RuiQin7 commented Dec 11, 2023 •

edited

Loading

BsoBird commented Dec 11, 2023 •

edited

Loading

SteNicholas commented Dec 11, 2023 •

edited

Loading

jiaoqingbo commented Dec 12, 2023

lifulong commented Dec 12, 2023 •

edited

Loading

TongWei1105 commented Dec 12, 2023 •

edited

Loading

onebox-li commented Dec 12, 2023

AngersZhuuuu commented Dec 12, 2023

lianneli commented Dec 21, 2023

camper42 commented Jan 16, 2024

wfxxh commented Feb 27, 2024

wxplovecc commented Mar 1, 2024

CodingCat commented Mar 8, 2024 •

edited

Loading

akpatnam25 commented Mar 12, 2024 •

edited by AngersZhuuuu

Loading

liugs0213 commented Mar 20, 2024 •

edited

Loading

lohitvijayarenu commented Aug 6, 2024

wangshengjie123 commented Aug 9, 2024

ASiegeLion commented Aug 12, 2024

jinyuchen724 commented Aug 15, 2024

HYBG-1126 commented Oct 9, 2024 •

edited

Loading

LennonChin commented Oct 10, 2024 •

edited

Loading

conorzhao commented Nov 19, 2024 •

edited

Loading

lzw7916002 commented Nov 26, 2024

lukaizj commented Nov 27, 2024 •

edited

Loading

zhisheng17 commented Dec 18, 2024

Who is using Apache Celeborn? #2140

Who is using Apache Celeborn? #2140

Comments

SteNicholas commented Dec 11, 2023

Search before asking

Motivation

Who's using Apache Celeborn?

Solution

Anything else?

Are you willing to submit a PR?

gaochao0509 commented Dec 11, 2023 • edited Loading

This comment was marked as resolved.

cxzl25 commented Dec 11, 2023

big-loser commented Dec 11, 2023 • edited Loading

RuiQin7 commented Dec 11, 2023 • edited Loading

BsoBird commented Dec 11, 2023 • edited Loading

SteNicholas commented Dec 11, 2023 • edited Loading

jiaoqingbo commented Dec 12, 2023

lifulong commented Dec 12, 2023 • edited Loading

TongWei1105 commented Dec 12, 2023 • edited Loading

onebox-li commented Dec 12, 2023

AngersZhuuuu commented Dec 12, 2023

lianneli commented Dec 21, 2023

camper42 commented Jan 16, 2024

wfxxh commented Feb 27, 2024

wxplovecc commented Mar 1, 2024

CodingCat commented Mar 8, 2024 • edited Loading

akpatnam25 commented Mar 12, 2024 • edited by AngersZhuuuu Loading

liugs0213 commented Mar 20, 2024 • edited Loading

lohitvijayarenu commented Aug 6, 2024

wangshengjie123 commented Aug 9, 2024

ASiegeLion commented Aug 12, 2024

jinyuchen724 commented Aug 15, 2024

HYBG-1126 commented Oct 9, 2024 • edited Loading

LennonChin commented Oct 10, 2024 • edited Loading

conorzhao commented Nov 19, 2024 • edited Loading

lzw7916002 commented Nov 26, 2024

lukaizj commented Nov 27, 2024 • edited Loading

zhisheng17 commented Dec 18, 2024

gaochao0509 commented Dec 11, 2023 •

edited

Loading

big-loser commented Dec 11, 2023 •

edited

Loading

RuiQin7 commented Dec 11, 2023 •

edited

Loading

BsoBird commented Dec 11, 2023 •

edited

Loading

SteNicholas commented Dec 11, 2023 •

edited

Loading

lifulong commented Dec 12, 2023 •

edited

Loading

TongWei1105 commented Dec 12, 2023 •

edited

Loading

CodingCat commented Mar 8, 2024 •

edited

Loading

akpatnam25 commented Mar 12, 2024 •

edited by AngersZhuuuu

Loading

liugs0213 commented Mar 20, 2024 •

edited

Loading

HYBG-1126 commented Oct 9, 2024 •

edited

Loading

LennonChin commented Oct 10, 2024 •

edited

Loading

conorzhao commented Nov 19, 2024 •

edited

Loading

lukaizj commented Nov 27, 2024 •

edited

Loading