Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CARBONDATA-4304] Initialize CarbonEnv will try to create folder on local filesystem if storelocation is set without scheme #4234

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

jack86596
Copy link
Contributor

Why is this PR needed?

If storelocation doesn't have scheme, file system is treated as local by default, not the file system defined by fs.defaultFS.

What changes were proposed in this PR?

  1. call fs.makeQualified and get the final storepath with correct scheme.
  2. use storepath with scheme to create the folder, keep storepath without scheme in CarbonProperties, because storepath in CarbonProperties will be used to compare with spark.sql.warehouse.dir which doesn't have scheme also.

Does this PR introduce any user interface change?

  • No

Is any new testcase added?

  • Yes

…ocal filesystem if storepath is set without scheme
@CarbonDataQA2
Copy link

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6049/

@CarbonDataQA2
Copy link

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4305/

@CarbonDataQA2
Copy link

Build Success with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/439/

@jack86596
Copy link
Contributor Author

retest this please

@CarbonDataQA2
Copy link

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6051/

@CarbonDataQA2
Copy link

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4307/

@CarbonDataQA2
Copy link

Build Success with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/441/

@kunal642
Copy link
Contributor

@jack86596 does this problem exists only with "carbon.storelocation" property?

@jack86596
Copy link
Contributor Author

@jack86596 does this problem exists only with "carbon.storelocation" property?

yes

@kunal642
Copy link
Contributor

As carbon.storelocation is deprecated and would be removed soon, I don't think we should use it or fix issues related to it

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants