Issue/jperf 273 transplant ssh jira nodes #7

dagguh · 2019-04-01T15:38:44Z

Desired effects:

both aws-infrastructure main and virtual-users tests can reuse the same SSH Jira API
aws-infrastructure being able to optimize provisioning, e.g. :
- preemptively install all DC nodes and then start them in sequence without additional delays
- parallelize DB, shared home and DC node setup (only blocking before the Jira start)
flexibility to inject arbitrary logic into specific points during SSH Jira lifetime (profilers, upgrade task triggering and polling, JVM config, DB config, system metrics)
composition of a custom set of arbitrary hooks (e.g. do not run poll upgrade tasks if I don't want to)
smart report production and consumption, e.g. don't transfer analytics logs if I'm not consuming them, also don't install vmstat if I'm not going to run it)
thread-safety - let hooks be immutable
rich library of battle-tested hooks
ability to add PostgreSQL support within infrastructure (without aws-infrastructure, we can reuse the same port), or even without any changes to infrastructure at all (custom hook)
aws-infrastructure can add its own hook for the Jira plugins transported via S3

JPERF-273 would be resolved with the first bullet point, but if we simply move the aws-infrastructure StandaloneStoppedNode, etc., then we'll have to live with the bad API for a long time. So I'm trying to make new, designed, more flexible API, which should stay useful for a longer time.

src/main/kotlin/com/atlassian/performance/tools/infrastructure/api/profiler/AsyncProfiler.kt

sebapawlak · 2019-04-02T06:46:44Z

Could you explain what is the general purpose/value proposition of this PR?

dagguh · 2019-04-02T09:13:02Z

It's a draft, I'll add more context to the commit messages when it's ready (the scope is in flux as I discover new obstacles or opportunities). But the overall goal is to ship https://ecosystem.atlassian.net/browse/JPERF-273

src/main/kotlin/com/atlassian/performance/tools/infrastructure/api/jira/flow/install/Install.kt

src/test/kotlin/com/atlassian/performance/tools/infrastructure/api/jira/InstallableJiraIT.kt

src/test/kotlin/com/atlassian/performance/tools/infrastructure/api/jvm/AdoptOpenJdk11IT.kt

dagguh · 2019-04-02T17:29:12Z

src/main/kotlin/com/atlassian/performance/tools/infrastructure/api/database/MySqlDatabase.kt

+        Sed().replace(
+            connection = ssh,
+            expression = "(<url>.*(@(//)?|//))" + "([^:/]+)" + "(.*</url>)",


It still preserves the old assumption that the dbconfig.xml from Jira home is set up for MySQL already and all we need to do is to switch the IP. We can make it more robust in another PR.
Here's how the assumed dbconfig.xml looks like:

<?xml version="1.0" encoding="UTF-8"?> <jira-database-config> <name>defaultDS</name> <delegator-name>default</delegator-name> <database-type>mysql</database-type> <jdbc-datasource> <url>jdbc:mysql://18.184.122.128:3306/jiradb?useUnicode=true&characterEncoding=UTF8&sessionVariables=storage_engine=InnoDB</url> <driver-class>com.mysql.jdbc.Driver</driver-class> <username>jirauser</username> <password>jirauser</password> <pool-min-size>20</pool-min-size> <pool-max-size>20</pool-max-size> <pool-max-wait>30000</pool-max-wait> <validation-query>select 1</validation-query> <min-evictable-idle-time-millis>60000</min-evictable-idle-time-millis> <time-between-eviction-runs-millis>300000</time-between-eviction-runs-millis> <pool-max-idle>20</pool-max-idle> <pool-remove-abandoned>true</pool-remove-abandoned> <pool-remove-abandoned-timeout>300</pool-remove-abandoned-timeout> <pool-test-while-idle>true</pool-test-while-idle> <validation-query-timeout>3</validation-query-timeout> </jdbc-datasource> </jira-database-config>

IMO it's a good moment to stop relying on the assumption.

...main/kotlin/com/atlassian/performance/tools/infrastructure/api/jira/DataCenterNodeFormula.kt

.../atlassian/performance/tools/infrastructure/api/jira/flow/install/DefaultJiraInstallation.kt

...n/com/atlassian/performance/tools/infrastructure/api/jira/flow/start/DefaultPostStartHook.kt

dagguh · 2019-04-05T12:25:34Z

...n/com/atlassian/performance/tools/infrastructure/api/jira/flow/start/DefaultPostStartHook.kt

+        listOf(
+            JiraLogs(),
+            JstatHook()
+            //,RestUpgrade(config.launchTimeouts, "admin", "admin") TODO requires database to work


Can we control the order of things happening before or after the upgrade?

...kotlin/com/atlassian/performance/tools/infrastructure/api/jira/flow/install/UbuntuSysstat.kt

...main/kotlin/com/atlassian/performance/tools/infrastructure/api/jira/flow/install/JiraLogs.kt

pczuj · 2019-04-15T17:10:23Z

...otlin/com/atlassian/performance/tools/infrastructure/api/jira/flow/install/DataCenterHook.kt

+    ) {
+        val localSharedHome = sharedHome.localSharedHome
+        sharedHome.mount(ssh)
+        val jiraHome = jira.home  // TODO what's the difference between localSharedHome and jiraHome? should both be hookable?


Can't see a difference between "jirahome" and "jira-local-home": https://confluence.atlassian.com/adminjiraserver073/installing-jira-data-center-861253059.html#InstallingJIRADataCenter-InstallingJIRADataCenter

pczuj · 2019-04-15T17:20:00Z

.../atlassian/performance/tools/infrastructure/api/jira/flow/install/DefaultJiraInstallation.kt

+        flow: JiraNodeFlow
+    ): InstalledJira {
+        val installation = productDistribution.install(ssh, ".")
+        val home = jiraHomeSource.download(ssh)


Just a note: I'd remove the knowledge of jirahome being downloaded from JiraHomeSource and make it have #install(ssh) as well. The JiraHomeSource could in the future be implemented as offline jirahome generator.

pczuj · 2019-04-15T17:40:02Z

...lin/com/atlassian/performance/tools/infrastructure/api/jira/flow/install/JiraHomeProperty.kt

+import com.atlassian.performance.tools.infrastructure.api.jira.flow.JiraNodeFlow
+import com.atlassian.performance.tools.ssh.api.SshConnection
+
+class JiraHomeProperty : InstalledJiraHook {


This could be part of jirahome installation

pczuj · 2019-04-15T17:50:08Z

...lin/com/atlassian/performance/tools/infrastructure/api/jira/flow/install/JiraInstallation.kt

+import com.atlassian.performance.tools.infrastructure.api.jira.flow.JiraNodeFlow
+import com.atlassian.performance.tools.ssh.api.SshConnection
+
+interface JiraInstallation {


Funny that this is also indirectly TcpServerHook (logicly), but the TcpServerHook is also InstalledJiraHook, so basically you could install Jira on top of tcp server with Jira, which makes sense.

pczuj · 2019-04-15T18:03:01Z

...otlin/com/atlassian/performance/tools/infrastructure/api/jira/flow/install/DataCenterHook.kt

+import com.atlassian.performance.tools.infrastructure.api.jira.flow.JiraNodeFlow
+import com.atlassian.performance.tools.ssh.api.SshConnection
+
+class DataCenterHook(


I'd name it differently, like e.g. UpgradeToDataCenter or DataCenterUpgrade or DataCenterInstallation

pczuj · 2019-04-15T18:16:42Z

...main/kotlin/com/atlassian/performance/tools/infrastructure/api/jira/flow/server/SystemLog.kt

+import com.atlassian.performance.tools.infrastructure.api.jira.flow.report.StaticReport
+import com.atlassian.performance.tools.ssh.api.SshConnection
+
+class SystemLog : Report {


class SystemLog : Report by StaticReport("/var/log/syslog")

pczuj · 2019-04-15T18:34:17Z

...com/atlassian/performance/tools/infrastructure/api/jira/flow/start/DefaultStartedJiraHook.kt

+        listOf(
+            JiraLogs(),
+            JstatHook()
+            //,RestUpgrade(config.launchTimeouts, "admin", "admin") TODO requires database to work


Do you mean credentials? This knowledge from database could be bypassed if the user would always be admin/admin (or any defined in the test config).
Following KB could be the material for another "Hook" that would ensure that: https://confluence.atlassian.com/jira/retrieving-the-jira-administrator-192836.html
We would need to have an sql client that could work with every database Jira supports - possibly some kind of application to reset jira password.

I meant that RestUpgrade has a hidden dependency on DB. All other hooks only depended on what's injected in their constructors or method params. I guess the REST upgrade hook could become an injectable part of the DB hook.
If that happens then the credentials are an internal contract of the hook, so that's all good.

try injecting REST upgrades into DB hook

src/main/kotlin/com/atlassian/performance/tools/infrastructure/api/jira/flow/TcpServer.kt

dagguh · 2019-04-17T08:42:48Z

FYI I'm doing this on my 20% time, so expect slow progress

...in/kotlin/com/atlassian/performance/tools/infrastructure/api/database/InstallableDatabase.kt

wyrzyk · 2019-04-05T07:59:25Z

...in/kotlin/com/atlassian/performance/tools/infrastructure/api/database/InstallableDatabase.kt

+
+interface InstallableDatabase : Database {
+
+    fun installInJira(databaseIp: String): Install


Why not JDBC URL? Is it enough to set up a database by knowing only the IP address?

Because the callers don't know the JDBC URL. aws-infrastructure only generates the IP.
Perhaps this is the same smell as in https://github.com/atlassian/aws-infrastructure/pull/15/files#r273025547 : we're designing low-level API to match the existing caller patterns.
Perhaps it's the aws-infrastructure, who should ask for the fun install(databaseIp: String): Install

wyrzyk · 2019-04-05T08:01:00Z

src/main/kotlin/com/atlassian/performance/tools/infrastructure/api/database/MySqlDatabase.kt

+        Sed().replace(
+            connection = ssh,
+            expression = "(<url>.*(@(//)?|//))" + "([^:/]+)" + "(.*</url>)",


IMO it's a good moment to stop relying on the assumption.

wyrzyk · 2019-04-05T08:08:36Z

src/main/kotlin/com/atlassian/performance/tools/infrastructure/api/jvm/ThreadDump.kt

@@ -26,7 +26,7 @@ class ThreadDump(
     */
    fun gather(connection: SshConnection, destination: String) {
        val threadDumpName = Instant.now().toEpochMilli()
-        val command = "${jdk.use()}; jcmd $pid Thread.print > $destination/${threadDumpName}"
+        val command = "${jdk.use()}; jcmd $pid Thread.print > $destination/$threadDumpName"


You could limit the cosmetic changes in the large PR like this one.

Ok, I'll try to make this PR smaller. It's too big to comprehend.

wyrzyk · 2019-09-12T15:25:14Z

src/main/kotlin/com/atlassian/performance/tools/infrastructure/api/database/DatabaseIpConfig.kt

+import com.atlassian.performance.tools.infrastructure.api.jira.flow.install.InstalledJiraHook
+import com.atlassian.performance.tools.ssh.api.SshConnection
+
+class DatabaseIpConfig(


Is there a test to cover this?

Yes, but right now it's externalized.

We can include this coverage in here via SshUbuntu

wyrzyk · 2019-09-12T15:31:13Z

src/main/kotlin/com/atlassian/performance/tools/infrastructure/api/database/DatabaseIpConfig.kt

+
+class DatabaseIpConfig(
+    private val databaseIp: String
+) : InstalledJiraHook {


I remember we were discussing the meaning of this hooks F2F, but now I'm getting back to this pull-request after a while and I feel I don't have a clear picture on the hook system. Do you plan to add Kdocs?

Sure, all API should have KDocs.
Until then, we can IDE and "find usages".

KDoc for interfaces

KDoc for classes

wyrzyk · 2019-09-12T15:39:54Z

src/main/kotlin/com/atlassian/performance/tools/infrastructure/api/database/MysqlConnector.kt

+import com.atlassian.performance.tools.ssh.api.SshConnection
+import java.time.Duration
+
+class MysqlConnector : InstalledJiraHook {


MysqlConnector#run will not run MysqlConnector. Can we consider renaming to MysqlConnectorInstaller (It's ugly but more accurate)? WDYT?

Fair enough

src/main/kotlin/com/atlassian/performance/tools/infrastructure/api/jira/flow/JiraNodeFlow.kt

Allow reports to be contributed from arbitrary phase of the lifecycle. It means we can download partial results if the lifecycle fails mid-way.

It's already possible to hook in DB installation from `InstallableDatabase`.

Show the problem with hooking onto the tail of the hook queue during hook iteration.

This also means that if you run a hook, it gets unregistered

Use destructive polling for all JiraNodeFlow hook types.

Add JavaDoc. Use terminology of "insert" and "call" as per [common terminology]. Repackage hooks. [common terminology]: https://en.wikipedia.org/wiki/Hooking

src/main/kotlin/com/atlassian/performance/tools/infrastructure/api/jira/hook/JiraNodeHooks.kt

dagguh · 2021-01-22T16:02:53Z

src/main/kotlin/com/atlassian/performance/tools/infrastructure/api/jira/hook/JiraNodeHooks.kt

+    val postInstall = preInstall.postInstall
+    val preStart = postInstall.preStart
+    val postStart = preStart.postStart
+    val reports = postStart.reports


These are all already available via preInstall. They serve as syntax sugar to avoid long chains.
Before:

hooks.preInstall.postInstall.preStart.postStart.reports.list()

After:

hooks.reports.list()

dagguh · 2021-01-22T16:04:44Z

...in/com/atlassian/performance/tools/infrastructure/api/jira/hook/install/AsyncProfilerHook.kt

+import com.atlassian.performance.tools.ssh.api.SshConnection
+import java.net.URI
+
+class AsyncProfilerHook : PreInstallHook {


@mzyromski-atlassian This is one of the more advanced usage of hooks.
It installs a profiler before Jira is installed. Then starts the profiler after Jira starts (using PID provided by hooks). And it registers a flamegraph file for download.

dagguh · 2021-01-29T13:36:15Z

@mzyromski-atlassian this is too big, so I started splitting it.
Here are the chunks:

JPERF-273: Generalize Jira installation API #100
Generalize Jira start API #101
Hook into Jira provisoning #102
more coming...

sebapawlak reviewed Apr 2, 2019

View reviewed changes

src/main/kotlin/com/atlassian/performance/tools/infrastructure/api/profiler/AsyncProfiler.kt Outdated Show resolved Hide resolved

pczuj reviewed Apr 2, 2019

View reviewed changes

src/main/kotlin/com/atlassian/performance/tools/infrastructure/api/jira/flow/install/Install.kt Outdated Show resolved Hide resolved

dagguh commented Apr 2, 2019

View reviewed changes

src/test/kotlin/com/atlassian/performance/tools/infrastructure/api/jira/InstallableJiraIT.kt Outdated Show resolved Hide resolved

dagguh commented Apr 2, 2019

View reviewed changes

src/test/kotlin/com/atlassian/performance/tools/infrastructure/api/jvm/AdoptOpenJdk11IT.kt Outdated Show resolved Hide resolved

dagguh commented Apr 2, 2019

View reviewed changes

...main/kotlin/com/atlassian/performance/tools/infrastructure/api/jira/DataCenterNodeFormula.kt Outdated Show resolved Hide resolved

dagguh commented Apr 5, 2019

View reviewed changes

.../atlassian/performance/tools/infrastructure/api/jira/flow/install/DefaultJiraInstallation.kt Outdated Show resolved Hide resolved

dagguh commented Apr 5, 2019

View reviewed changes

...n/com/atlassian/performance/tools/infrastructure/api/jira/flow/start/DefaultPostStartHook.kt Outdated Show resolved Hide resolved

dagguh commented Apr 5, 2019

View reviewed changes

dagguh commented Apr 15, 2019

View reviewed changes

...kotlin/com/atlassian/performance/tools/infrastructure/api/jira/flow/install/UbuntuSysstat.kt Outdated Show resolved Hide resolved

...kotlin/com/atlassian/performance/tools/infrastructure/api/jira/flow/install/UbuntuSysstat.kt Outdated Show resolved Hide resolved

dagguh commented Apr 15, 2019

View reviewed changes

...main/kotlin/com/atlassian/performance/tools/infrastructure/api/jira/flow/install/JiraLogs.kt Outdated Show resolved Hide resolved

pczuj reviewed Apr 15, 2019

View reviewed changes

src/main/kotlin/com/atlassian/performance/tools/infrastructure/api/jira/flow/TcpServer.kt Outdated Show resolved Hide resolved

wyrzyk reviewed May 2, 2019

View reviewed changes

dagguh force-pushed the issue/JPERF-273-transplant-ssh-jira-nodes branch 2 times, most recently from e665f02 to d3c7a96 Compare July 5, 2019 18:19

dagguh force-pushed the issue/JPERF-273-transplant-ssh-jira-nodes branch from d3c7a96 to e18d03e Compare September 6, 2019 15:09

wyrzyk reviewed Sep 12, 2019

View reviewed changes

src/main/kotlin/com/atlassian/performance/tools/infrastructure/api/jira/flow/JiraNodeFlow.kt Outdated Show resolved Hide resolved

dagguh and others added 22 commits January 22, 2021 15:46

JPERF-273: Refactor the Jira lifecycle

6d1ffef

Allow reports to be contributed from arbitrary phase of the lifecycle. It means we can download partial results if the lifecycle fails mid-way.

JPERF-273: Stop jira manually, because we can

ce3b38a

Rename "track" to "flow"

f032696

Resolve DB TODO

06bee9a

It's already possible to hook in DB installation from `InstallableDatabase`.

Add DC hook

336c278

Let hooks be hooked in at any later stage

eec3e10

JPERF-273: Offer DB-related Jira hooks

2e3a2a9

JPERF-273: Test JiraNodeFlow concurrency

d129d08

JPERF-273: Fix JiraNodeFlowTest.shouldHookDuringListing

58ba182

JPERF-273: Add ParallelInstallation

1984d8e

JPERF-273: Distinguish running hooks from hooking them in

7192b70

Bump a bunch of timeouts, because my Internet is awful

fac32ba

Show the problem with hooking onto the tail of the hook queue during hook iteration.

JPERF-273: Poll hooks manually to avoid tail iteration

ebf7325

This also means that if you run a hook, it gets unregistered

JPERF-219: Block hooking to the past

d311c65

Fixup: 6ad30c0

c87d198

Use destructive polling for all JiraNodeFlow hook types.

Extract S3HostedJdk

f036201

JPERF-273: Define default JiraNodeFlow

13a0c5c

JPERF-273: Remove fluent hook API

1156aae

JPERF-273: Rename "flows" to "hooks"

23a7138

Expose Reports.listReports

c18a5b0

JPERF-273 Postgres 9.6.15

26bdabb

Refactor hooks: composition over inheritance

2742b75

Add JavaDoc. Use terminology of "insert" and "call" as per [common terminology]. Repackage hooks. [common terminology]: https://en.wikipedia.org/wiki/Hooking

dagguh force-pushed the issue/JPERF-273-transplant-ssh-jira-nodes branch from caa979a to 2742b75 Compare January 22, 2021 15:57

dagguh commented Jan 22, 2021

View reviewed changes

src/main/kotlin/com/atlassian/performance/tools/infrastructure/api/jira/hook/JiraNodeHooks.kt Outdated Show resolved Hide resolved

dagguh commented Jan 22, 2021

View reviewed changes

dagguh requested a review from a team January 22, 2021 16:05

dagguh added 2 commits January 29, 2021 10:50

Repackage hooks

a336862

Reuse StaticBackoff

8dc645b

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Issue/jperf 273 transplant ssh jira nodes #7

Issue/jperf 273 transplant ssh jira nodes #7

dagguh commented Apr 1, 2019 •

edited

Loading

sebapawlak commented Apr 2, 2019

dagguh commented Apr 2, 2019

dagguh Apr 2, 2019

wyrzyk Apr 5, 2019

dagguh Apr 5, 2019

pczuj Apr 15, 2019 •

edited

Loading

pczuj Apr 15, 2019 •

edited

Loading

pczuj Apr 15, 2019

pczuj Apr 15, 2019

pczuj Apr 15, 2019 •

edited

Loading

pczuj Apr 15, 2019

pczuj Apr 15, 2019

dagguh Dec 10, 2020 •

edited

Loading

dagguh commented Apr 17, 2019

wyrzyk Apr 5, 2019

dagguh May 2, 2019

wyrzyk Apr 5, 2019

wyrzyk Apr 5, 2019

dagguh Dec 10, 2020

wyrzyk Sep 12, 2019

dagguh Sep 12, 2019

wyrzyk Sep 12, 2019

dagguh Sep 12, 2019

dagguh Jan 22, 2021

wyrzyk Sep 12, 2019

dagguh Sep 12, 2019

dagguh Jan 22, 2021

dagguh Jan 22, 2021

dagguh commented Jan 29, 2021 •

edited

Loading


		interface InstallableDatabase : Database {

		fun installInJira(databaseIp: String): Install

Issue/jperf 273 transplant ssh jira nodes #7

Are you sure you want to change the base?

Issue/jperf 273 transplant ssh jira nodes #7

Conversation

dagguh commented Apr 1, 2019 • edited Loading

sebapawlak commented Apr 2, 2019

dagguh commented Apr 2, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pczuj Apr 15, 2019 • edited Loading

Choose a reason for hiding this comment

pczuj Apr 15, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pczuj Apr 15, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dagguh Dec 10, 2020 • edited Loading

Choose a reason for hiding this comment

dagguh commented Apr 17, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dagguh commented Jan 29, 2021 • edited Loading

dagguh commented Apr 1, 2019 •

edited

Loading

pczuj Apr 15, 2019 •

edited

Loading

pczuj Apr 15, 2019 •

edited

Loading

pczuj Apr 15, 2019 •

edited

Loading

dagguh Dec 10, 2020 •

edited

Loading

dagguh commented Jan 29, 2021 •

edited

Loading