Skip to content

Commit

Permalink
[Chore](build) upgrade clang-format version to 16 && move thrift to f…
Browse files Browse the repository at this point in the history 
…e-common (#19155)

upgrade clang-format version to 16
move thrift to fe-common
fix core dump on pipeline engine when operator canceled and not prepared
  • Loading branch information
@BiteTheDDDDt
BiteTheDDDDt authored Apr 28, 2023
1 parent ffe27ba commit ec517a5
Show file tree
Hide file tree
Showing 54 changed files with 115 additions and 123 deletions.
2 changes: 1 addition & 1 deletion .github/actions/action-sh-checker
Submodule action-sh-checker updated 5 files
+2 −1 .github/workflows/ci.yml
+7 −2 Dockerfile
+3 −1 README.md
+3 −0 action.yml
+23 −2 entrypoint.sh
2 changes: 1 addition & 1 deletion .github/actions/ccache-action
Submodule ccache-action updated 8 files
+13 −0 .github/workflows/tests.yml
+4 −1 action.yml
+4,835 −4,629 dist/restore/index.js
+4,834 −4,626 dist/save/index.js
+177 −196 package-lock.json
+4 −4 package.json
+10 −6 src/restore.ts
+6 −1 src/save.ts
2 changes: 1 addition & 1 deletion .github/actions/clang-format-lint-action
Submodule clang-format-lint-action updated 2 files
+4 −4 README.md
+ − clang-format/clang-format16
2 changes: 1 addition & 1 deletion .github/actions/clang-tidy-review
Submodule clang-tidy-review updated 22 files
+0 −1 .dockerignore
+1 −1 .github/workflows/black.yml
+26 −0 .github/workflows/test.yml
+159 −1 .gitignore
+8 −13 Dockerfile
+87 −44 README.md
+8 −2 action.yml
+0 −3 entrypoint.sh
+3 −8 post/Dockerfile
+0 −0 post/__init__.py
+9 −0 post/action.yml
+258 −28 post/clang_tidy_review/clang_tidy_review/__init__.py
+47 −27 post/clang_tidy_review/clang_tidy_review/post.py
+48 −95 post/clang_tidy_review/clang_tidy_review/review.py
+42 −0 post/clang_tidy_review/pyproject.toml
+0 −4 post/requirements.txt
+0 −4 requirements.txt
+24 −0 tests/src/hello.cxx
+12 −0 tests/src/hello_original.cxx
+108 −0 tests/src/test_clang_tidy_review.yaml
+424 −0 tests/test_review.py
+16 −0 upload/action.yml
2 changes: 1 addition & 1 deletion .github/actions/get-workflow-origin
Submodule get-workflow-origin updated 4 files
+1 −1 .github/workflows/test.yml
+2,197 −65 dist/index.js
+13,700 −65 package-lock.json
+1 −1 package.json
2 changes: 1 addition & 1 deletion .github/actions/paths-filter
Submodule paths-filter updated 1 files
+1 −0 README.md
2 changes: 1 addition & 1 deletion .github/workflows/be-ut-clang.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ jobs:
export DEFAULT_DIR='/opt/doris'
mkdir "${DEFAULT_DIR}"
wget https://github.com/amosbird/ldb_toolchain_gen/releases/download/v0.14.2/ldb_toolchain_gen.sh \
wget https://github.com/amosbird/ldb_toolchain_gen/releases/download/v0.17/ldb_toolchain_gen.sh \
-q -O /tmp/ldb_toolchain_gen.sh
bash /tmp/ldb_toolchain_gen.sh "${DEFAULT_DIR}/ldb-toolchain"
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/build-thirdparty.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,7 @@ jobs:
sudo DEBIAN_FRONTEND=noninteractive apt install --yes "${packages[@]}"
mkdir -p "${DEFAULT_DIR}"
wget https://github.com/amosbird/ldb_toolchain_gen/releases/download/v0.14.2/ldb_toolchain_gen.sh \
wget https://github.com/amosbird/ldb_toolchain_gen/releases/download/v0.17/ldb_toolchain_gen.sh \
-q -O /tmp/ldb_toolchain_gen.sh
bash /tmp/ldb_toolchain_gen.sh "${DEFAULT_DIR}/ldb-toolchain"
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/clang-format.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ jobs:
id: be_clang_format
with:
source: "be/src be/test"
clangFormatVersion: 15
clangFormatVersion: 16
inplace: False

- name: Ignore it!
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/code-checks.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ jobs:
export DEFAULT_DIR='/opt/doris'
mkdir "${DEFAULT_DIR}"
wget https://github.com/amosbird/ldb_toolchain_gen/releases/download/v0.14.2/ldb_toolchain_gen.sh \
wget https://github.com/amosbird/ldb_toolchain_gen/releases/download/v0.17/ldb_toolchain_gen.sh \
-q -O /tmp/ldb_toolchain_gen.sh
bash /tmp/ldb_toolchain_gen.sh "${DEFAULT_DIR}/ldb-toolchain"
Expand Down
4 changes: 2 additions & 2 deletions .gitignore
View file
Original file line number Diff line number Diff line change
Expand Up @@ -74,8 +74,8 @@ fe/fe-core/gen/
fe/fe-core/src/main/resources/static/
fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisLexer.tokens
fe/fe-core/src/main/antlr4/org/apache/doris/nereids/gen/
fe/fe-core/src/main/java/org/apache/doris/thrift
fe/fe-core/src/main/java/org/apache/parquet
fe/fe-common/src/main/java/org/apache/doris/thrift
fe/fe-common/src/main/java/org/apache/parquet

# BE
be/build*/
Expand Down
46 changes: 29 additions & 17 deletions be/src/common/factory_creator.h
View file
Original file line number Diff line number Diff line change
Expand Up @@ -35,21 +35,33 @@
// during inherits
// TODO try to allow make_unique
//
#define ENABLE_FACTORY_CREATOR(TypeName) \
private: \
void* operator new(std::size_t size) { return ::operator new(size); } \
void* operator new[](std::size_t size) { return ::operator new[](size); } \
\
public: \
void* operator new(std::size_t count, void* ptr) { return ::operator new(count, ptr); } \
void operator delete(void* ptr) noexcept { ::operator delete(ptr); } \
void operator delete[](void* ptr) noexcept { ::operator delete[](ptr); } \
void operator delete(void* ptr, void* place) noexcept { ::operator delete(ptr, place); } \
template <typename... Args> \
static std::shared_ptr<TypeName> create_shared(Args&&... args) { \
return std::make_shared<TypeName>(std::forward<Args>(args)...); \
} \
template <typename... Args> \
static std::unique_ptr<TypeName> create_unique(Args&&... args) { \
return std::unique_ptr<TypeName>(new TypeName(std::forward<Args>(args)...)); \
#define ENABLE_FACTORY_CREATOR(TypeName) \
private: \
void* operator new(std::size_t size) { \
return ::operator new(size); \
} \
void* operator new[](std::size_t size) { \
return ::operator new[](size); \
} \
\
public: \
void* operator new(std::size_t count, void* ptr) { \
return ::operator new(count, ptr); \
} \
void operator delete(void* ptr) noexcept { \
::operator delete(ptr); \
} \
void operator delete[](void* ptr) noexcept { \
::operator delete[](ptr); \
} \
void operator delete(void* ptr, void* place) noexcept { \
::operator delete(ptr, place); \
} \
template <typename... Args> \
static std::shared_ptr<TypeName> create_shared(Args&&... args) { \
return std::make_shared<TypeName>(std::forward<Args>(args)...); \
} \
template <typename... Args> \
static std::unique_ptr<TypeName> create_unique(Args&&... args) { \
return std::unique_ptr<TypeName>(new TypeName(std::forward<Args>(args)...)); \
}
2 changes: 1 addition & 1 deletion be/src/olap/rowset/segment_v2/segment_writer.cpp
View file
Original file line number Diff line number Diff line change
Expand Up @@ -134,7 +134,7 @@ Status SegmentWriter::init(const std::vector<uint32_t>& col_ids, bool has_key,
_column_writers.reserve(_tablet_schema->columns().size());
_column_ids.insert(_column_ids.end(), col_ids.begin(), col_ids.end());
_olap_data_convertor = std::make_unique<vectorized::OlapBlockDataConvertor>();
auto create_column_writer = [&](uint32_t cid, const auto& column) -> auto{
auto create_column_writer = [&](uint32_t cid, const auto& column) -> auto {
ColumnWriterOptions opts;
opts.meta = _footer.add_columns();

Expand Down
22 changes: 15 additions & 7 deletions be/src/pipeline/exec/operator.h
View file
Original file line number Diff line number Diff line change
Expand Up @@ -42,13 +42,15 @@ class RuntimeState;
class TDataSink;
} // namespace doris

#define OPERATOR_CODE_GENERATOR(NAME, SUBCLASS) \
NAME##Builder::NAME##Builder(int32_t id, ExecNode* exec_node) \
: OperatorBuilder(id, #NAME, exec_node) {} \
\
OperatorPtr NAME##Builder::build_operator() { return std::make_shared<NAME>(this, _node); } \
\
NAME::NAME(OperatorBuilderBase* operator_builder, ExecNode* node) \
#define OPERATOR_CODE_GENERATOR(NAME, SUBCLASS) \
NAME##Builder::NAME##Builder(int32_t id, ExecNode* exec_node) \
: OperatorBuilder(id, #NAME, exec_node) {} \
\
OperatorPtr NAME##Builder::build_operator() { \
return std::make_shared<NAME>(this, _node); \
} \
\
NAME::NAME(OperatorBuilderBase* operator_builder, ExecNode* node) \
: SUBCLASS(operator_builder, node) {};

namespace doris::pipeline {
Expand Down Expand Up @@ -309,6 +311,9 @@ class DataSinkOperator : public OperatorBase {

protected:
void _fresh_exec_timer(NodeType* node) {
if (_runtime_profile == nullptr) {
return;
}
node->profile()->total_time_counter()->update(
_runtime_profile->total_time_counter()->value());
}
Expand Down Expand Up @@ -387,6 +392,9 @@ class StreamingOperator : public OperatorBase {

protected:
void _fresh_exec_timer(NodeType* node) {
if (_runtime_profile == nullptr) {
return;
}
node->runtime_profile()->total_time_counter()->update(
_runtime_profile->total_time_counter()->value());
}
Expand Down
10 changes: 6 additions & 4 deletions be/src/vec/common/hash_table/hash.h
View file
Original file line number Diff line number Diff line change
Expand Up @@ -122,10 +122,12 @@ inline size_t hash_crc32(doris::vectorized::Int128 u) {
doris::vectorized::UInt128((u >> 64) & int64_t(-1), u & int64_t(-1)));
}

#define DEFINE_HASH(T) \
template <> \
struct HashCRC32<T> { \
size_t operator()(T key) const { return hash_crc32<T>(key); } \
#define DEFINE_HASH(T) \
template <> \
struct HashCRC32<T> { \
size_t operator()(T key) const { \
return hash_crc32<T>(key); \
} \
};

DEFINE_HASH(doris::vectorized::UInt8)
Expand Down
2 changes: 1 addition & 1 deletion be/src/vec/functions/like.cpp
View file
Original file line number Diff line number Diff line change
Expand Up @@ -451,7 +451,7 @@ Status FunctionLikeBase::execute_impl(FunctionContext* context, Block& block,
context->get_function_state(FunctionContext::THREAD_LOCAL));
// for constant_substring_fn, use long run length search for performance
if (constant_substring_fn ==
*(state->function.target<doris::Status (*)(LikeSearchState * state, const ColumnString&,
*(state->function.target<doris::Status (*)(LikeSearchState* state, const ColumnString&,
const StringRef&, ColumnUInt8::Container&)>())) {
RETURN_IF_ERROR(execute_substring(values->get_chars(), values->get_offsets(), vec_res,
&state->search_state));
Expand Down
2 changes: 1 addition & 1 deletion bin/start_fe.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -108,7 +108,7 @@ if [[ -e "${DORIS_HOME}/bin/palo_env.sh" ]]; then
fi

if [[ -z "${JAVA_HOME}" ]]; then
JAVA="$(which java)"
JAVA="$(command -v java)"
else
JAVA="${JAVA_HOME}/bin/java"
fi
Expand Down
3 changes: 1 addition & 2 deletions build-for-release.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,7 @@ eval set -- "${OPTS}"

_USE_AVX2=1
TAR=0
HELP=0
VERSION=
if [[ "$#" == 1 ]]; then
_USE_AVX2=1
Expand Down Expand Up @@ -94,13 +95,11 @@ fi

if [[ "${HELP}" -eq 1 ]]; then
usage
exit
fi

if [[ -z ${VERSION} ]]; then
echo "Must specify version"
usage
exit 1
fi

echo "Get params:
Expand Down
2 changes: 1 addition & 1 deletion build-support/check-format.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,6 @@ DORIS_HOME=$(
)
export DORIS_HOME

CLANG_FORMAT="${CLANG_FORMAT_BINARY:=$(which clang-format)}"
CLANG_FORMAT="${CLANG_FORMAT_BINARY:=$(command -v clang-format)}"

python "${DORIS_HOME}/build-support/run_clang_format.py" "--clang-format-executable" "${CLANG_FORMAT}" "-r" "--style" "file" "--inplace" "false" "--extensions" "c,h,C,H,cpp,hpp,cc,hh,c++,h++,cxx,hxx" "--exclude" "none" "be/src be/test"
2 changes: 1 addition & 1 deletion build-support/clang-format.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,6 @@ DORIS_HOME=$(
)
export DORIS_HOME

CLANG_FORMAT="${CLANG_FORMAT_BINARY:=$(which clang-format)}"
CLANG_FORMAT="${CLANG_FORMAT_BINARY:=$(command -v clang-format)}"

python "${DORIS_HOME}/build-support/run_clang_format.py" "--clang-format-executable" "${CLANG_FORMAT}" "-r" "--style" "file" "--inplace" "true" "--extensions" "c,h,C,H,cpp,hpp,cc,hh,c++,h++,cxx,hxx" "--exclude" "none" "be/src be/test"
9 changes: 4 additions & 5 deletions build.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -232,7 +232,6 @@ fi

if [[ "${HELP}" -eq 1 ]]; then
usage
exit
fi
# build thirdparty libraries if necessary
if [[ ! -f "${DORIS_THIRDPARTY}/installed/lib/libbacktrace.a" ]]; then
Expand Down Expand Up @@ -391,9 +390,9 @@ echo "Build generated code"
cd "${DORIS_HOME}/gensrc"
# DO NOT using parallel make(-j) for gensrc
make
rm -rf "${DORIS_HOME}/fe/fe-core/src/main/java/org/apache/doris/thrift ${DORIS_HOME}/fe/fe-core/src/main/java/org/apache/parquet"
cp -r "build/gen_java/org/apache/doris/thrift" "${DORIS_HOME}/fe/fe-core/src/main/java/org/apache/doris"
cp -r "build/gen_java/org/apache/parquet" "${DORIS_HOME}/fe/fe-core/src/main/java/org/apache/"
rm -rf "${DORIS_HOME}/fe/fe-common/src/main/java/org/apache/doris/thrift ${DORIS_HOME}/fe/fe-common/src/main/java/org/apache/parquet"
cp -r "build/gen_java/org/apache/doris/thrift" "${DORIS_HOME}/fe/fe-common/src/main/java/org/apache/doris"
cp -r "build/gen_java/org/apache/parquet" "${DORIS_HOME}/fe/fe-common/src/main/java/org/apache/"

# Assesmble FE modules
FE_MODULES=''
Expand Down Expand Up @@ -432,7 +431,7 @@ if [[ "${BUILD_BE}" -eq 1 ]]; then
if [[ "${CLEAN}" -eq 1 ]]; then
clean_be
fi
MAKE_PROGRAM="$(which "${BUILD_SYSTEM}")"
MAKE_PROGRAM="$(command -v "${BUILD_SYSTEM}")"
echo "-- Make program: ${MAKE_PROGRAM}"
echo "-- Use ccache: ${CMAKE_USE_CCACHE}"
echo "-- Extra cxx flags: ${EXTRA_CXX_FLAGS:-}"
Expand Down
2 changes: 1 addition & 1 deletion build_plugin.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@ eval set -- "${OPTS}"

ALL_PLUGIN=1
CLEAN=0
HELP=0
if [[ "$#" == 1 ]]; then
# defuat
ALL_PLUGIN=1
Expand Down Expand Up @@ -89,7 +90,6 @@ fi

if [[ "${HELP}" -eq 1 ]]; then
usage
exit
fi

echo "Get params:
Expand Down
1 change: 0 additions & 1 deletion contrib/udf/build_udf.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,6 @@ fi

if [[ "${HELP}" -eq 1 ]]; then
usage
exit
fi

echo "Get params:
Expand Down
2 changes: 1 addition & 1 deletion dist/download_scripts/download_base.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ DOWNLOAD_DIR="$5"
FILE_SUFFIX=".tar.xz"

# Check if curl cmd exists
if [[ -n $(which curl >/dev/null 2>&1) ]]; then
if [[ -n $(command -v curl >/dev/null 2>&1) ]]; then
echo "curl command not found on the system"
exit 1
fi
Expand Down
3 changes: 0 additions & 3 deletions docker/thirdparties/run-thirdparties-docker.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -99,13 +99,11 @@ fi

if [[ "${HELP}" -eq 1 ]]; then
usage
exit 0
fi

if [[ "${COMPONENTS}"x == ""x ]]; then
echo "Invalid arguments"
usage
exit 1
fi

if [[ "${CONTAINER_UID}"x == "doris--"x ]]; then
Expand Down Expand Up @@ -150,7 +148,6 @@ for element in "${COMPONENTS_ARR[@]}"; do
else
echo "Invalid component: ${element}"
usage
exit 1
fi
done

Expand Down
2 changes: 1 addition & 1 deletion docs/en/docs/install/source-install/compilation-win.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,7 @@ This topic is about how to compile Doris from source with Windows.
- [Java8](https://doris-thirdparty-repo.bj.bcebos.com/thirdparty/jdk-8u131-linux-x64.tar.gz)
- [Apache Maven 3.6.3](https://doris-thirdparty-repo.bj.bcebos.com/thirdparty/apache-maven-3.6.3-bin.tar.gz)
- [Node v12.13.0](https://doris-thirdparty-repo.bj.bcebos.com/thirdparty/node-v12.13.0-linux-x64.tar.gz)
- [LDB_TOOLCHAIN](https://github.com/amosbird/ldb_toolchain_gen/releases/download/v0.14.2/ldb_toolchain_gen.sh)
- [LDB_TOOLCHAIN](https://github.com/amosbird/ldb_toolchain_gen/releases/download/v0.17/ldb_toolchain_gen.sh)

5. Configure environment variables

Expand Down
2 changes: 1 addition & 1 deletion docs/zh-CN/docs/install/source-install/compilation-win.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,7 @@ under the License.
- [Java8](https://doris-thirdparty-repo.bj.bcebos.com/thirdparty/jdk-8u131-linux-x64.tar.gz)
- [Apache Maven 3.6.3](https://doris-thirdparty-repo.bj.bcebos.com/thirdparty/apache-maven-3.6.3-bin.tar.gz)
- [Node v12.13.0](https://doris-thirdparty-repo.bj.bcebos.com/thirdparty/node-v12.13.0-linux-x64.tar.gz)
- [LDB_TOOLCHAIN](https://github.com/amosbird/ldb_toolchain_gen/releases/download/v0.14.2/ldb_toolchain_gen.sh)
- [LDB_TOOLCHAIN](https://github.com/amosbird/ldb_toolchain_gen/releases/download/v0.17/ldb_toolchain_gen.sh)

5. 配置环境变量

Expand Down
10 changes: 5 additions & 5 deletions env.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -117,7 +117,7 @@ fi
if [[ "${DORIS_TOOLCHAIN}" == "gcc" ]]; then
# set GCC HOME
if [[ -z "${DORIS_GCC_HOME}" ]]; then
DORIS_GCC_HOME="$(dirname "$(which gcc)")"/..
DORIS_GCC_HOME="$(dirname "$(command -v gcc)")"/..
export DORIS_GCC_HOME
fi

Expand All @@ -135,7 +135,7 @@ if [[ "${DORIS_TOOLCHAIN}" == "gcc" ]]; then
elif [[ "${DORIS_TOOLCHAIN}" == "clang" ]]; then
# set CLANG HOME
if [[ -z "${DORIS_CLANG_HOME}" ]]; then
DORIS_CLANG_HOME="$(dirname "$(which clang)")"/..
DORIS_CLANG_HOME="$(dirname "$(command -v clang)")"/..
export DORIS_CLANG_HOME
fi

Expand Down Expand Up @@ -163,7 +163,7 @@ if [[ -z "${DORIS_BIN_UTILS}" ]]; then
fi

if [[ -z "${DORIS_GCC_HOME}" ]]; then
DORIS_GCC_HOME="$(dirname "$(which gcc)")/.."
DORIS_GCC_HOME="$(dirname "$(command -v gcc)")/.."
export DORIS_GCC_HOME
fi

Expand Down Expand Up @@ -191,8 +191,8 @@ if test -z "${BUILD_THIRDPARTY_WIP:-}"; then

# check java home
if [[ -z "${JAVA_HOME}" ]]; then
JAVA="$(which java)"
JAVAP="$(which javap)"
JAVA="$(command -v java)"
JAVAP="$(command -v javap)"
else
JAVA="${JAVA_HOME}/bin/java"
JAVAP="${JAVA_HOME}/bin/javap"
Expand Down
Loading

0 comments on commit ec517a5

Please sign in to comment.