Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Signed-off-by: Pavel Abramov <[email protected]>
- Loading branch information
1 parent
1be9d4e
commit 6becead
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,199 @@ | ||
| # NeoEden: framework that allows you to write EVE tests in Golang | ||
|
|
||
| ## Motivation | ||
|
|
||
| Personally, I don't believe in the docs, which are against "all bad" and for all "the good", especially in engineering practicies, because definition of good and bad depends on the context of the problem: things evolve, requirements change and what used to be perfect fit back then seems like really bad idea now. So how one should navigate uncertainty, choose methods and evolve products? | ||
|
|
||
| *(Pro Tip: As a software engineer, remember that the correct answer to 90% of questions is, of course, 'it depends'—because who doesn't love a good mix of nuance, edge cases, and never-ending trade-offs? And never trust random estimates you see on the Internet)* | ||
|
|
||
| I think there's no one right answer for that, and if you ask different people you would get different opinions. If you ask me, after some unecessary references and jokes it would come down to following | ||
|
|
||
| > “Programs must be written for people to read, and only incidentally for machines to execute.” ― Harold Abelson, Structure and Interpretation of Computer Programs | ||
| So best practices would be | ||
|
|
||
| 1) Define interface first, implement later (most probably first version would be not as good as you would like it to be) | ||
| 2) Write documentation for people to understand contexts and ask less why? question when reading code or using it | ||
| 3) Learn by doing, there's a lot of unknown unknowns. So ship code, get feedback and iterate on it. | ||
|
|
||
| With that in mind let's talk about tests in Eden till version 0.9.12 | ||
|
|
||
| ## What is wrong with old way of doing things? Why do we need new one? | ||
|
|
||
| Why do we have a section about how things use to be? Well, history have a tendency to repeat itself, people involved in the project change and there could be a newcomer, who would think that Domain Specific Language (DSL) could be a good way to go. It would be nice to have some kind of summary, which basically says, yep, we tried that, that's how we did it and those are the problems we faced, that is why we switched. It doesn't mean to discourage, but rather learn from past experiences, who knows, maybe in some time there will be addition saying that DSL actually is the best way to go for that time requirements. Change is the only constant :) | ||
|
|
||
| When talking about Eden tests till version 0.9.12, we are talking about escript, a Domain Specific Language (DSL) which describes test case and uses Eden to setup, environment. Escript looks like this | ||
|
|
||
| ```bash | ||
| # 1 Setup environment variables | ||
| {{$port := "2223"}} | ||
| {{$network_name := "n1"}} | ||
| {{$app_name := "eclient"}} | ||
|
|
||
| # ... | ||
|
|
||
| # 2 run eden commands | ||
| eden -t 1m network create 10.11.12.0/24 -n {{$network_name}} | ||
|
|
||
| # 3 run escript commands | ||
| test eden.network.test -test.v -timewait 10m ACTIVATED {{$network_name}} | ||
|
|
||
| eden pod deploy -n {{$app_name}} --memory=512MB {{template "eclient_image"}} -p {{$port}}:22 --networks={{$network_name}} | ||
|
|
||
| # 4 execute shell script which are defined inside escript file | ||
| exec -t 5m bash ssh.sh | ||
| stdout 'Ubuntu' | ||
|
|
||
| # 5 overwrite configuration | ||
| -- eden-config.yml -- | ||
| {{/* Test's config. file */}} | ||
| test: | ||
| controller: adam://{{EdenConfig "adam.ip"}}:{{EdenConfig "adam.port"}} | ||
| eve: | ||
| {{EdenConfig "eve.name"}}: | ||
| onboard-cert: {{EdenConfigPath "eve.cert"}} | ||
| serial: "{{EdenConfig "eve.serial"}}" | ||
| model: {{EdenConfig "eve.devmodel"}} | ||
| -- ssh.sh -- | ||
| EDEN={{EdenConfig "eden.root"}}/{{EdenConfig "eden.bin-dist"}}/{{EdenConfig "eden.eden-bin"}} | ||
| for i in `seq 20` | ||
| do | ||
| sleep 20 | ||
| # Test SSH-access to container | ||
| echo $i\) $EDEN sdn fwd eth0 {{$port}} -- {{template "ssh"}} grep Ubuntu /etc/issue | ||
| $EDEN sdn fwd eth0 {{$port}} -- {{template "ssh"}} grep Ubuntu /etc/issue && break | ||
| done | ||
| ``` | ||
| So you can | ||
| 1) setup some environment variables | ||
| 2) run eden commands | ||
| 3) run escript commands with test | ||
| 4) execute user-defined shell scripts | ||
| 5) overwrite eden configuration | ||
| Escript file is fed as input to eden test command, which parses the variables using golang templates to substitute some variables using templates in golang, then it's going to read line by line and execute it via os.exec call in golang. test is actually a compiled golang binary, we compile it inside eden repo and then execute it, under the hood it is a manually forked version of standard golang test package with some added commands. | ||
|
|
||
| Sounds a tad complicated, doesn't it? | ||
| So there are two problems with that approach: | ||
| **You have to be an escript expert**: when new person comes to a project, they will have some industry skills, like C++ expertise, Rust, Golang, Computer Networking, etc. but if you never worked on this project, you never heard about escript, its' sole purpose was to be part of Eden and be useful language to describe tests for Eden. One might argue, but that's just bash on steroids. Well, true, but do you know what `!` means? It's not equal parameter in escript. So it is like bash, but not exactly it, and it might take time to figure out other hidden features, that could be solved by proper documentation. But before spiraling down that conversation lets think about tools? Imagine, that you wrote new fancy eden test and it doesn't work. Usual thing, you would say. Test Driven Development is all about writing failing tests first and then making them work. | ||
| **But how do you debug that test?** Because you're running an interpreter, which runs bash commands via os.exec plus you execute golang program, which is written and compiled inside the repository. Debug printing would work, but I find debuggers much more useful and efficient most of the times. It is a matter of taste, but better to debugger at your disposal, than not to have it in this case. Even worse, how do you debug escript problem? You need to know it's internals, there's no Stackoverflow or Google by your side, nor there are books about it. | ||
|
|
||
| **Bumping golang is actually manual action**: last but not least, we have to maintain custom test files which a basically copy-paste with one added function, bumping golang turns into some weird dances in that case. | ||
|
|
||
| I believe that those reasons are good enough to switch for golang-only approach to write tests, which is called NeoEden (kudos to @shjala for the name) | ||
|
|
||
| Also worst case what could happen with DSL is something like the Inner JSON effect [link](https://thedailywtf.com/articles/the-inner-json-effect) | ||
|
|
||
| ## Enter NeoEden. How tests look now? | ||
|
|
||
| They look like [this](../tests/sec/sec_test.go) code listing is in the end of this section. Conceptually, Eden contains of two entities: an entity which allows you setup some kind of environment and deploy EVE there. Think Infrastructure as a Code, like Teraform, which allows me using golang code to deploy, this is openevec package; second entity is evetestkit, which serves as an abstraction over missing features of adam as controller such as keeping track of state and provide you with toolkit to test things, namely send commands and access EVE or its AppInstances to see if the certain file exists or certain request gives you certain response. Those two things let you create golang-only test like one below. You use openevec to setup environment and evetestkit to help you describe what is passing criterion for particular test. The common denominator between openevec and evetestkit is EdenSetupArgs, openevec is just a wrapper over that configuration, which defines your state. The goal is to be able to define your state, write custom functions to state, like this: | ||
|
|
||
|
|
||
| ```go | ||
| pe := openevec.GetDefaultPatchEnvelope() | ||
| appInstance := openevec.GetDefaultAppInstance() | ||
| cfg := openevec.GetDefaultConfig(path) | ||
| .WithHyperVisor(openevec::HyperVisorQemu) | ||
| .WithAppInstances(appInstance) | ||
| .WithPatchEnvelopes(pe) | ||
| evec := openvec.CreateOpnEVEC(cfg) | ||
| evec.SetupEden(/* ... */) | ||
| evec.StartEden(/* ... */) | ||
| evec.OnboardEve(/* ... */) | ||
| node, err := tk.InitializeTestFromConfig(cfg) | ||
| got := node.GetPatchEnvelopes(appInstance.UUID) | ||
| t.assert_eq(pe, got) | ||
| got = node.AppInstanceSSH(fmt.Sprintf("curl %s", openevec.DEFAULT_PATH_ENVELOPE_URL)) | ||
| t.assert_eq(pe, got) | ||
| ``` | ||
|
|
||
| This is an example of how test for patch envelopes should look like. You might not know what PatchEnvelopes are, but from the definition you can see what does it mean to get patch envelopes and that if you ssh to AppInstance and do curl to some URL you will get that patch envelope. Below is an working example of a test which checks if app armor is enabled (if there is a file on a file system on EVE) | ||
|
|
||
| ```go | ||
| package sec_test | ||
| import ( | ||
| "os" | ||
| "path/filepath" | ||
| "strings" | ||
| "testing" | ||
| "github.com/lf-edge/eden/pkg/defaults" | ||
| tk "github.com/lf-edge/eden/pkg/evetestkit" | ||
| "github.com/lf-edge/eden/pkg/openevec" | ||
| log "github.com/sirupsen/logrus" | ||
| ) | ||
| const projectName = "security-test" | ||
| const appArmorStatus = "/sys/module/apparmor/parameters/enabled" | ||
| var eveNode *tk.EveNode | ||
| func TestMain(m *testing.M) { | ||
| log.Println("Security Test Suite started") | ||
| defer log.Println("Security Test Suite finished") | ||
| currentPath, err := os.Getwd() | ||
| if err != nil { | ||
| log.Fatal(err) | ||
| } | ||
| twoLevelsUp := filepath.Dir(filepath.Dir(currentPath)) | ||
| cfg := openevec.GetDefaultConfig(twoLevelsUp) | ||
| if err = openevec.ConfigAdd(cfg, cfg.ConfigName, "", false); err != nil { | ||
| log.Fatal(err) | ||
| } | ||
| evec := openevec.CreateOpenEVEC(cfg) | ||
| configDir := filepath.Join(twoLevelsUp, "eve-config-dir") | ||
| if err := evec.SetupEden("config", configDir, "", "", "", []string{}, false, false); err != nil { | ||
| log.Fatalf("Failed to setup Eden: %v", err) | ||
| } | ||
| if err := evec.StartEden(defaults.DefaultVBoxVMName, "", ""); err != nil { | ||
| log.Fatalf("Start eden failed: %s", err) | ||
| } | ||
| if err := evec.OnboardEve(cfg.Eve.CertsUUID); err != nil { | ||
| log.Fatalf("Eve onboard failed: %s", err) | ||
| } | ||
| node, err := tk.InitilizeTestFromConfig(projectName, cfg, tk.WithControllerVerbosity("debug")) | ||
| if err != nil { | ||
| log.Fatalf("Failed to initialize test: %v", err) | ||
| } | ||
| eveNode = node | ||
| res := m.Run() | ||
| os.Exit(res) | ||
| } | ||
| func TestAppArmorEnabled(t *testing.T) { | ||
| log.Println("TestAppArmorEnabled started") | ||
| defer log.Println("TestAppArmorEnabled finished") | ||
| t.Parallel() | ||
| out, err := eveNode.EveReadFile(appArmorStatus) | ||
| if err != nil { | ||
| t.Fatal(err) | ||
| } | ||
| exits := strings.TrimSpace(string(out)) | ||
| if exits != "Y" { | ||
| t.Fatal("AppArmor is not enabled") | ||
| } | ||
| } | ||
| ``` | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.