From 1d8e982b202cfefb2fbc543c01ec32ebd050d524 Mon Sep 17 00:00:00 2001 From: chenliang613 Date: Thu, 19 Oct 2017 18:10:29 +0530 Subject: [PATCH] [CARBONDATA-1598] Remove all spark 1.x info(CI, readme, documents) [CARBONDATA-1598] Remove all spark 1.x info(CI, readme, documents) --- README.md | 2 - assembly/pom.xml | 21 --------- build/README.md | 15 ++++--- docs/quick-start-guide.md | 62 --------------------------- integration/spark-common-test/pom.xml | 22 ---------- 5 files changed, 8 insertions(+), 114 deletions(-) diff --git a/README.md b/README.md index 06f1ce5a74c..297cece401e 100644 --- a/README.md +++ b/README.md @@ -29,8 +29,6 @@ You can find the latest CarbonData document and learn more at: ## Status Spark2.1: [![Build Status](https://builds.apache.org/buildStatus/icon?job=carbondata-master-spark-2.1)](https://builds.apache.org/view/CarbonData/job/carbondata-master-spark-2.1/) -Spark1.6: -[![Build Status](https://builds.apache.org/buildStatus/icon?job=carbondata-master-spark-1.6)](https://builds.apache.org/view/CarbonData/job/carbondata-master-spark-1.6/) ## Features CarbonData file format is a columnar store in HDFS, it has many features that a modern columnar format has, such as splittable, compression schema ,complex data type etc, and CarbonData has following unique features: diff --git a/assembly/pom.xml b/assembly/pom.xml index d705b66e1f4..b5652a5b188 100644 --- a/assembly/pom.xml +++ b/assembly/pom.xml @@ -124,28 +124,7 @@ - - - spark-1.5 - - - org.apache.carbondata - carbondata-spark - ${project.version} - - - - - spark-1.6 - - - org.apache.carbondata - carbondata-spark - ${project.version} - - - spark-2.1 diff --git a/build/README.md b/build/README.md index 5fa6814e889..50f6ce296fa 100644 --- a/build/README.md +++ b/build/README.md @@ -27,12 +27,17 @@ * [Apache Thrift 0.9.3](http://archive.apache.org/dist/thrift/0.9.3/) ## Build command -Build without test,by default carbondata takes Spark 1.6.2 to build the project +From 1.3.0 onwards, CarbonData supports spark 2.x, build without test,by default carbondata takes Spark 2.1.0 to build the project ``` mvn -DskipTests clean package ``` -Build with different supported versions of Spark. +Build with test +``` +mvn clean package +``` + +Before 1.3.0, build with different supported versions of Spark ``` mvn -DskipTests -Pspark-1.5 -Dspark.version=1.5.1 clean package mvn -DskipTests -Pspark-1.5 -Dspark.version=1.5.2 clean package @@ -44,13 +49,9 @@ mvn -DskipTests -Pspark-1.6 -Dspark.version=1.6.3 clean package mvn -DskipTests -Pspark-2.1 -Dspark.version=2.1.0 clean package ``` -Build with test -``` -mvn clean package -``` ## For contributors : To build the format code after any changes, please follow the below command. Note:Need install Apache Thrift 0.9.3 ``` -mvn clean -DskipTests -Pbuild-with-format -Pspark-1.6 package +mvn clean -DskipTests -Pbuild-with-format -Pspark-2.1 package ``` \ No newline at end of file diff --git a/docs/quick-start-guide.md b/docs/quick-start-guide.md index 5b39b9e1539..d8336798e06 100644 --- a/docs/quick-start-guide.md +++ b/docs/quick-start-guide.md @@ -98,65 +98,3 @@ scala>carbon.sql("SELECT city, avg(age), sum(age) FROM test_table GROUP BY city").show() ``` - -## Interactive Analysis with Spark Shell Version 1.6 - -#### Basics - -Start Spark shell by running the following command in the Spark directory: - -``` -./bin/spark-shell --jars -``` -**NOTE**: Assembly jar will be available after [building CarbonData](https://github.com/apache/carbondata/blob/master/build/README.md) and can be copied from `./assembly/target/scala-2.1x/carbondata_xxx.jar` - -**NOTE**: In this shell, SparkContext is readily available as `sc`. - -* In order to execute the Queries we need to import CarbonContext: - -``` -import org.apache.spark.sql.CarbonContext -``` - -* Create an instance of CarbonContext in the following manner : - -``` -val cc = new CarbonContext(sc, "") -``` -**NOTE**: If running on local machine without hdfs, configure the local machine's store path instead of hdfs store path - -#### Executing Queries - -###### Creating a Table - -``` -scala>cc.sql("CREATE TABLE - IF NOT EXISTS test_table ( - id string, - name string, - city string, - age Int) - STORED BY 'carbondata'") -``` -To see the table created : - -``` -scala>cc.sql("SHOW TABLES").show() -``` - -###### Loading Data to a Table - -``` -scala>cc.sql("LOAD DATA INPATH 'sample.csv file path' - INTO TABLE test_table") -``` -**NOTE**: Please provide the real file path of `sample.csv` for the above script. - -###### Querying Data from a Table - -``` -scala>cc.sql("SELECT * FROM test_table").show() -scala>cc.sql("SELECT city, avg(age), sum(age) - FROM test_table - GROUP BY city").show() -``` diff --git a/integration/spark-common-test/pom.xml b/integration/spark-common-test/pom.xml index b2ee316919d..8806c0a9419 100644 --- a/integration/spark-common-test/pom.xml +++ b/integration/spark-common-test/pom.xml @@ -327,28 +327,6 @@ - - spark-1.5 - - - org.apache.carbondata - carbondata-spark - ${project.version} - test - - - - - spark-1.6 - - - org.apache.carbondata - carbondata-spark - ${project.version} - test - - - spark-2.1