Remove teeplate dependency for builtin solution

[mdwagner]

Premise

While on my quest to add Windows support to Lucky framework, I've run into a bit of a snag with one our dependencies: teeplate. Teeplate is actually a fork of this shard, but it hasn't been updated in some time now. It also does not currently support Windows.

When diving into our fork's implementation to potentially add Windows support, I quickly realized this would be a harder task than I initially thought. I asked on Discord whether teeplate was worth the effort of refactoring/rewriting, or if we could instead leverage a simpler, builtin solution (maybe something that only relies on the stdlib?). The response was positive to look into a builtin solution to remove this dependency. The following suggestion is my idea of a builtin solution.

Builtin Solution

My idea for making a simpler version of teeplate can be best thought of in two parts, but only one of them requires an implementation on our part.

Template files

The first part is: how do we craft templates? We use ECR, which will define a #to_s method on a class/struct mapped to a template file. You could also use any other template engine (heredocs, crinja, kilt, slang, etc.), because we don't need to decide how you make a template, just that you can pass an object that respond's with #to_s. This means we don't need to provide any implementation.

Directory structure

The second part is: how do we define a directory structure the same way teeplate did? My idea for this is to make it in code instead of in the repo. This has a few benefits:

• We don't need to have weird file/folder names in the directory structure
• We can make multiple different directory structures without cluttering the repo
• We can still pass custom runtime names for files/folders
• We don't need to write any logic around existing files/permissions/etc.
• This would make Windows compatibility out-of-the-box

Pseudocode of a possible implementation

Example folder structure from teeplate:

templates/
  src/
    {{file}}/
      version.cr.ecr
    {{file}}.cr.ecr
  LICENSE.ecr
  README.md.ecr
  shard.yml.ecr

Possible implementation:

require "ecr"

class ShardYml
  ECR.def_to_s "shard.yml.ecr"
end
class Readme
  ECR.def_to_s "README.md.ecr"
end
class License
  ECR.def_to_s "LICENSE.ecr"
end
class MyCustomFile
  def initialize(@name : String)
  end
  ECR.def_to_s "my_custom_file.cr.ecr"
end

custom_file_name = "lucky_template"
tree = LuckyTemplate.create_tree(Path["./templates"]) do |folder|
  folder.add_file("shard.yml", ShardYml.new)
  folder.add_file("README.md", Readme.new)
  folder.add_file("LICENSE", License.new)
  folder.add_folder("src") do |src|
    src.add_file("#{custom_file_name}.cr", MyCustomFile.new(custom_file_name))
    src.add_folder(custom_file_name) do |cfn|
      cfn.add_file("version.cr", <<-VERSION)
      module #{custom_file_name.camelcase}
        VERSION = "0.1.0"
      end
      VERSION
    end
  end
end
tree.create! # will actually create the files/folders in ./templates directory

At the moment, it's a little verbose, but that's just to show off the intent of it. It's meant to be a simpler solution, almost purpose built, using Crystal's stdlib. It won't have a ton of bells and whistles up front, but I think this can provide most of what we need to get going.

[jwoertink]

I love this idea. One thing I'd like to try and keep in mind for this is ensuring the resulting code always compiles and can pass both a format and ameba checks. Though, generated apps don't come with ameba by default, they still get tested against currently being separate files.

So for example, we go this route, then down the road someone makes a change to a file, but with an extra space somewhere, or some syntax error, we need to make sure that the resulting apps fully catch all of that during the CI builds.

Related: luckyframework/lucky_cli#530

[mdwagner]

My understanding is that would be up to whatever that generator expects? Meaning the generator could create a test suite that runs ameba and/or format itself to ensure it satifies their requirements. I'm not sure how that applies to this discussion, since that sounds more specific to a type of template validation, and this is not that. This is more about defining a directory structure that you could use with templates.

However, if you are asking for ways to validate that this works, I do have an idea in mind: Allow this to statically create a representation that would allow you to apply your own validations. Let me know if that's something you're interesting in, but for now it's not my priority.

[robacarp]

Teeplate has been in desperate need of being replaced for a long time, and this seems like a great opportunity to improve the lucky generators overall. Here are some things I learned in working on both Amber and Lucky:

• It is very convenient to have the generator templating language not the same as the templating language you're generating templates for. Lucky has less of a risk to this than Amber does because the templates are written in Crystal, whereas Amber templates were written in ECR or slang. You end up with weird code like this <%= "<%=" %><%= object.name %><% "%>" %> or whatever.
• Slang is a template language which does not have good control over it's whitespace, and that may be an issue for you. It's also one of a few languages which is essentially optimized for generating HTML. I'd encourage focusing on a more general templating language. ECR is fine, but it's verbose and people seem to have negative feelings about it in general. Liquid could be useful here, I think? It can be run at both compile time and runtime, which is probably useful here.
• Even better yet, a way to have per-file template engine might be handy... generating a yaml with a liquid template might get confusing, for example.
• Lastly, and I think you're already well onto this, ditch the filesystem mapping stuff. Carrying around files with names which are themselves templates is such a mess.

[mdwagner]

• I hope to avoid this problem by allowing you to use any template language you want, but also allowing you to not use one at all if you'd like.
• Liquid, ECR, etc. The goal is they would all work, but if it's something outside of the Crystal ecosystem, it might need some small attention, but shouldn't be impossible.
• This seems entirely possible with the proposed idea. Based on my above impl: ShardYml could be ECR, but Readme could be Liquid, etc.
• Yep, it's one of the most challenging aspects of teeplate IMO.

[robacarp]

One of the current messes in the Lucky Generators is the compositional design. If memory serves right, you can generate these permutations:

• Bare lucky app
• lucky app with browser templates
• lucky app with api templates
• lucky app with browser templates with an auth engine
• lucky app with api templates with an auth engine

These are helpful constructs but the pattern itself doesn't scale and maintaining the generators is already an art -- "wait where does this line in particular come from?" is a question I ask frequently.

Building them in code as you've suggested could help this a lot. At least, I'm thinking you can use class inheritance to your aid. But another pattern exists in full template repositories like lucky jumpstart. I'm not yet suggesting to move everything into template repos, but maybe there are lessons to be learned there.

(Posting this as a different response because it's in a totally different vein.)

[jwoertink]

This is a good point. And I already need to add another layer with DB or no DB?... We had considered just making the generator generic where it pulls in some git repo, and then making separate repos for each template type, but that in itself comes with a slew of maintenance issues.

[robacarp]

At one point the Amber community was really excited about a thing called Recipes -- some of the ideas and discussion in that thread may be enlightening to this effort.

[mdwagner]

So, I'm building a POC of this idea, that seems like it could help with these different permutations really well. Hopefully will have more update on this soon.

I think your idea of using template repositories could also be useful, maybe in combination with git subtrees (think of a single repo that holds multiple other repos' state). Then you could have a root generator repo that would use this proposed idea to make other template repos, but would be distributed as multiple repos, to use as templates (that probably sounds ridiculous, but hopefully you get my point lol).

[mdwagner]

UPDATE

I have been working on a POC (proof-of-concept) that implements some of the ideas in my initial suggestion in the discussion, and I'm ready to share what I've created.

Behold...

LuckyTemplate

What is LuckyTemplate?

LuckyTemplate is very similar to Teeplate in its' goals, however, it's much simpler and a lot more capable.
Like Teeplate, you can create a file/folder structure templatized, but instead of using the filesystem, you do it in code.

Here are some of the major points:

• Allows you to create a virtual file/folder structure, that you can write to disk anywhere you want
• Supports using folders like building-blocks, enabling you to reuse folders within other folders
• File content can be both static (compiletime) or dynamic (runtime), using the following types:
  • String (static)
  • Inheritance (dynamic) by implementing an interface
  • Block (dynamic) with yielding an IO
  • Empty (static) when not caring about content
• After writing to disk, it comes with it's own Spec helper to validate that the files/folders within that location exist
• Provides a snapshot of a folder structure, to allow for custom validation or anything you want (e.g. using file paths for generating log output)

What does LuckyTemplate look like?

Simply put, this is the smallest example of what LuckyTemplate looks like:

require "lucky_template"

LuckyTemplate.write!(Path["."]) { }

However, let's see something more interesting.
Here is an example of how you might use LuckyTemplate:

README.md.ecr

# <%= @name %>!

lucky_template_example.cr

require "lucky_template"

class Readme
  include LuckyTemplate::Fileable

  def initialize(@name : String)
  end

  def to_file(io : IO) : Nil
    to_s(io)
  end

  ECR.def_to_s "#{__DIR__}/README.md.ecr"
end

name = "John"
folder = LuckyTemplate.write!(Path["."]) do |dir|
  dir.add_file("README.md", Readme.new("john"))

  dir.add_file("Welcome.md") do |io|
    io << "# Welcome " << name << "!\n"
  end

  dir.add_file("LICENSE", <<-LICENSE)
  The MIT License (MIT)
  ...
  LICENSE

  dir.add_folder("src") do |src|
    src.add_file(".keep")

    src.add_folder(name.downcase) do |name_dir|
      name_dir.add_file("#{name.downcase}.cr", <<-CR)
      class #{name}
      end

      pp! #{name}.new
      CR
    end
  end
end

LuckyTemplate.validate!(Path["."], folder)

The above code does the following:

• At path ., which is the current directory, it creates these files:
  • README.md using ECR template
  • Welcome.md using block yielding IO
  • LICENSE using static String
  • src/.keep (empty file)
  • src/john/john.cr using static String interpolation
• After success, it returns the folder it just used to write to disk
• It runs LuckyTemplate.validate! with path . and the returned folder, to determine if all the files/folders within are actually created

Another way to validate a folder is during testing, using the included Spec helper:

john_spec.cr

# Let's assume `folder` was still in scope

include LuckyTemplate::Spec

it "template folder for john worked" do
  folder.should be_valid_at(Path["."])
end

You can even create multiple folders at once, or embed them into other folders, without writing to disk yet:

folder1 = LuckyTemplate.create_folder do |dir|
  dir.add_file("folder_one.txt")
end
folder2 = LuckyTemplate.create_folder do |dir|
  dir.add_file("folder_two.txt")
end
folder3 = LuckyTemplate.create_folder do |dir|
  dir.add_file("folder_three.txt")
  dir.insert_folder("folder1", folder1)
  dir.insert_folder("folder2", folder2)
end
LuckyTemplate.write!(Path["."], folder3)

And perhaps the most flexible feature, you can even snapshot folders, for your own use:

# Assume above example is still in scope

snapshot = LuckyTemplate.snapshot(folder3)

snapshot.each do |path, type|
  puts "All files/folders in Snapshot:"
  puts "#{path} => #{type}"
end

snapshot.each do |path, type|
  puts "Only files in Snapshot:"
  puts "#{path} => #{type}" if type.file?
end

snapshot.each do |path, type|
  puts "Only folders in Snapshot:"
  puts "#{path} => #{type}" if type.folder?
end

Windows support?

Yep! LuckyTemplate works on Windows!
Since Path in Crystal handles most of the heavy lifting, LuckyTemplate is really just providing a nicer interface to creating files/folders.
Here is the Github Action workflow that runs both Linux & Windows

Examples are cool in theory, but what about actual real-world use?

I have a PR up in Lucky's Carbon, that replaces Teeplate with LuckyTemplate, while also improving some stuff that originally needed to be by hand, but can now be done with LuckyTemplate's snapshot feature. It also includes using the Spec helper for validation. Feel free to take a look!

Docs?

Yes, I've been deploying my docs here.

Final word

I know this was long, and tbh, I probably forgot some things I wanted to address. But hopefully, this is a good demonstration of LuckyTemplate and what I've been working on.
I've gone through a few different iterations at this point. This is the one I landed on that satisfied my goals and was simple to follow.
I think my next step will be to trying using it for other Lucky templates besides Carbon, and putting a lot of this in the actual Readme.

Let me know what you think!

Edit

One thing I was thinking I could add here that might be useful to some, is showing what would happen if you didn't use a Crystal implementation for making templates:

external_template

Welcome $NAME!

external_file.cr

class ExternalTemplate
  include LuckyTemplate::Fileable

  def initialize(@name : String)
  end

  def to_file(io : IO) : Nil
    File.open("#{__DIR__}/external_template") do |input|
      Process.run(
        command: "envsubst",
        input: input,
        output: io,
        env: {
          "NAME" => @name,
        }
      )
    end
  end
end

LuckyTemplate.write!(Path["."]) do |dir|
  dir.add_file("external_file", ExternalTemplate.new("John"))
end

As you can see above, I'm creating a running process using the gettext tool envsubst, which templates files using environment variables, and then passes that output to LuckyTemplate's file. This should be suitable in the event you aren't using Crystal for templating, but would still prefer to template out file/folders using LuckyTemplate.

[jwoertink]

Ok, so next big question is, where do you see this thing going? Do you want to transfer ownership to the Lucky org? Or just keep it in your name and we just link to that as a dependency? If we transfer to the Lucky org, I'm all for just giving you carte blanche so you can just commit and merge as needed to keep things moving. But if you'd rather just keep under your name, that's cool too!

[mdwagner]

This is the strategy I'm thinking of:

• Clean up/finalize POC implementation for normal use
• Transfer ownership to Lucky org, where any iteration will continue

I don't see why I need to tie this to myself for anything, but yeah it would be ideal to have some ownership when it's in the Lucky org, so it can continue to iterate and address any issues.

[mdwagner]

@jwoertink Ok, I think it's ready for the transfer to LuckyFramework.

The following are completed:

• Finalized docs
• Robust test suite
• Readme created

I'll make the first release as soon as we transfer it. Then we can start using it and continue to make improvements.

Let me know what needs to be done on my end. I'll try to make myself available.

[jwoertink]

I'm not sure exactly how the transfer works, but you can see if it just lets you send it over... if not, you may need to add me on as an admin? Or, I guess I could fork it... Yeah, fork it is probably fine... I'll do that.

[mdwagner]

Oh, I dont think you needed to fork it, but if you already did, just break away from the original repo. I'd rather not keep my original repo around, as I plan to do all my work on the luckyframework org.

I guess I initiate the transfer, but it says I dont have permissiont to create public repos for luckyframework. I would suggest we delete the fork and transfer it instead.

[mdwagner]

lucky_template has been transferred into LuckyFramework and all future work will take place there.