Skip to content

A script that outputs a StructyType source code based on a DataFrame string schema

Notifications You must be signed in to change notification settings

RodrigoCSoares/struct-type-builder

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

3 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

This project aims to transform a Spark's simple string schema (printed when called printSchema method in a DataFrame) into a source code of a StructType based on the schema. Furthermore, this is a learning project of some Scala's fundamentals and reading the DataFrame directly from the data source could be achieved the goal, however, that would not be so useful for my learning purposes.

How to use:

  • Run the project passing the input text file that contains the string schema as a parameter, example: src/main/resources/schema.txt

Input example:
root
|-- topic: string (nullable = true)
|-- key: string (nullable = true)
|-- received_iso_at: timestamp (nullable = true)

Output example:
StructType(
Array(
StructField("topic", StringType, nullable = true),
StructField("key", StringType, nullable = true),
StructField("received_iso_at", TimestampType, nullable = true),
)
)

About

A script that outputs a StructyType source code based on a DataFrame string schema

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages