Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Automatically cluster with AWS execution environments #2797

Draft
wants to merge 6 commits into
base: main
Choose a base branch
from

Conversation

krainboltgreene
Copy link

@krainboltgreene krainboltgreene commented Sep 22, 2024

This will pick out the parts needed to cluster for any livebook application deployed on the AWS execution platforms.

  • Write documentation describing cloud map setup
  • Write documentation describing port setup
  • Write documentation describing security groups
  • Write documentation for partial automation

This will pick out the parts needed to cluster for any livebook application deployed on the AWS execution platforms.
@CLAassistant
Copy link

CLAassistant commented Sep 22, 2024

CLA assistant check
All committers have signed the CLA.

@krainboltgreene
Copy link
Author

Good news: There's like 5-6 different jq alternatives.
Bad news: They're all focused on speed/language availability rather than size.

One option we have is to utilize a call to the Elixir shipped binary to parse/query?

@josevalim
Copy link
Contributor

Oh, calling the Elixir binary is a good call. Let's do that. You can also call erl and use the new built-in json parser.

@jonatanklosko
Copy link
Member

@josevalim calling elixir and :json sounds best (otherwise we need to mix.install), but currently we only require OTP 25. That said, it should be fine to bump the requirement, since for Desktop and Docker we always use latest anyway.

@krainboltgreene
Copy link
Author

@jonatanklosko One of Livebook's deps is jason, so I used that instead to avoid bumping requirements.

@krainboltgreene
Copy link
Author

172.31.49.31root@ip-172-31-metadata=`curl --silent ${ECS_CONTAINER_METADATA_URI}`TADATA_URI}`
root@ip-172-31-49-31:/app# machine_ip=`echo $metadata | bin/core eval 'IO.read(:stdio, :eof) |> Jason.decode!() |> Map.get("Networks") |> Enum.at(0) |> Map.get("IPv4Addresses") |> Enum.at(0) |> IO.write()'`
root@ip-172-31-49-31:/app# image_id=`echo $metadata | bin/core eval 'IO.read(:stdio, :eof) |> Jason.decode!() |> Map.get("ImageID") |> IO.write()'`
root@ip-172-31-49-31:/app# deployment="$(echo $image_id | shasum | cut -c 1-10)"
root@ip-172-31-49-31:/app# echo "livebook-${deployment}@${machine_ip}"
[email protected]

Here it is working.

@krainboltgreene
Copy link
Author

Okay so the remaining work here is the documentation of Cloud Map and Security Groups. That said this is actually rather generic to Phoenix itself, or even Elixir, so should I suggest it as documentation to those projects?

@josevalim
Copy link
Contributor

I'd document it on Livebook first, for two reasons:

  1. Elixir itself does not have a specific page for documenting per provider option. There are just too many possible scenarios for the language docs to cover.

  2. Phoenix has a section on deployment, which you could add this, but you would need to cover all of the aspects of deployment, which will be more work.

WDYT?

@krainboltgreene
Copy link
Author

Roger, will start today.

For the future I think maybe there's a signal here telling us that an elixir cookbook, community run like Bevy's but outside the scope any framework or the language, might be worth organizing.

@jonatanklosko
Copy link
Member

@krainboltgreene good call parsing with release eval!

rel/server/env.sh.eex Outdated Show resolved Hide resolved
Copy link

github-actions bot commented Oct 1, 2024

Uffizzi Ephemeral Environment Deploying

☁️ https://app.uffizzi.com/github.com/livebook-dev/livebook/pull/2797

⚙️ Updating now by workflow run 11123556986.

What is Uffizzi? Learn more!

@jonatanklosko
Copy link
Member

Looks good to me, we just need the docs now. Ideally with some IaC snippets, I think either is fine, terraform, cloud formation or CLI, just so it's more standardised.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants