Move parquet cascading schemes to subprojects

[rubanm]

This moves all the parquet schemes to separate sub-projects than the parquet sources. This is mainly for easier upgrade to cascading3 and future versions perhaps. (As things stand now, we need to upgrade scalding-core in our cascading3 branch, before scalding-parquet for example.)

[johnynek]

can we move this raw version string with the rest at the top of this file?

[rubanm]

Updated. The global thriftVersion is already present and set to 0.5.0, which I think should be okay.

[johnynek]

👍

[ianoc]

This seems useful for the cascading 3 upgrade, and makes sense there. Do we want it in the 0.16 release cycle ? Or keep it closer to the big painful cascading one?

@rubanm would this be hard to take as an update at twitter? (just as an exemplar of issues elsewhere possibly).

[ianoc]

With the aliases and deps in place it should be handled mostly by ivy resolves I think?

Can we have the new types/classes be package private to scalding? They seem like an internal implementation detail as far as scalding is concerned, would be nice to not have them in our ABI if we could? thoughts @johnynek ?

[rubanm]

Re cascading3, I could cherry pick the sbt update commit (moves from Build.scala to build.sbt) to cascading3 branch, and apply this on top of it. If we don't merge this to develop, it will be one more thing to look at in the diff in the eventual cascading3 -> develop merge.

Yes given the pre-existing sub-projects will still pull in classes in the corresponding new *-cascading ones, the upgrade should be fine I think. I'm in favor of merging to develop (and cherry picking to cascading3 branch) if this seems reasonable as a general thing to do.

[rubanm]

Some of Schemes are consumed at Twitter in multiformat sources, for example. The rest could be made package private I think.

[johnynek]

@rubanm we will need help from you and @isnotinvain to make sure we keep twitter up to date and not let any forks happen!

So, if you think you can get the merge in at Twitter, I'm fine with a merge for 0.16.

[ianoc]

Yep, good way to put it. Merge when green from me if the hassle level is ok
for you guys to handle to merge.

On Friday, February 12, 2016, P. Oscar Boykin [email protected]
wrote:

@rubanm https://github.com/rubanm we will need help from you and
@isnotinvain https://github.com/isnotinvain to make sure we keep
twitter up to date and not let any forks happen!

So, if you think you can get the merge in at Twitter, I'm fine with a
merge for 0.16.

—
Reply to this email directly or view it on GitHub
#1514 (comment).

Sent from Gmail Mobile

[rubanm]

@ianoc @johnynek Makes sense. I'll wait for a green sandbox internally before merging. (So we block this on the new algebird version to release internally at Twitter first, with the related breaking tests fixed.) I'll just cherry pick this to the cascading3 branch meanwhile to keep that going.

[rubanm]

Sorry, closed by mistake.

[johnynek]

should we merge this now? I kind of lost track of it.

[CLAassistant]

All committers have signed the CLA.