Unable to upgrade/install Spinnaker Service on k8s 1.22 with Operator 1.3.0

[GenPage]

Spinnaker Service version: 1.29.2
Spinnaker Operator version: 1.3.0
Kubernetes version: 1.22

I was unable to upgrade our existing instance to 1.29.2 on the new 1.3.0 operator. I ended up having to nuke our dev instance and re-apply SpinService 1.28.1 with the operator 1.2.5 to complete the upgrade to 1.29.2. It seems the operator 1.3.0 doesn't allow to apply our configuration

{"level":"info","ts":1678227807.7857943,"logger":"spinnakerservice","msg":"reconciling SpinnakerService","Request.Namespace":"spinnaker","Request.Name":"spinnaker"}
{"level":"info","ts":1678227807.7859452,"logger":"spinnakerservice","msg":"checking spindeploy deployment","Request.Namespace":"spinnaker","Request.Name":"spinnaker"}
{"level":"info","ts":1678227807.786293,"logger":"spinnakerservice","msg":"*config.changeDetector detected a change that needs to be reconciled","Service":"spinnaker"}
{"level":"info","ts":1678227807.786315,"logger":"spinnakerservice","msg":"retrieving complete Spinnaker configuration","Service":"spinnaker"}
{"level":"info","ts":1678227807.786325,"logger":"spinnakerservice","msg":"applying options to Spinnaker config with 13 generators","Service":"spinnaker"}
{"level":"error","ts":1678227807.7864661,"logger":"controller.spinnakerservice-controller","msg":"Reconciler error","name":"spinnaker","namespace":"spinnaker","error":"Error creating decrypter for value 'encrypted:k8s!n:spinnaker-datadog!k:datadog-api-key':\n  secret context not initialized","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/opt/spinnaker-operator/build/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/opt/spinnaker-operator/build/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227"}

SpinService yaml:

apiVersion: spinnaker.io/v1alpha2
kind: SpinnakerService
metadata:
  creationTimestamp: "2022-01-11T17:47:30Z"
  generation: 102
  name: spinnaker
  namespace: spinnaker
  resourceVersion: "806761787"
  uid: 50bcf170-80e3-415c-b1e7-3b0d63de0c10
spec:
  accounts:
    dynamic: false
    enabled: false
  expose:
    service:
      overrides: {}
      type: NodePort
    type: service
  spinnakerConfig:
    config:
      artifacts:
        s3:
          accounts:
          - name: <redacted>
            region: <redacted>
          enabled: "true"
      canary:
        defaultJudge: NetflixACAJudge-v1.0
        defaultMetricsAccount: plangrid
        defaultMetricsStore: datadog
        enabled: "true"
        reduxLoggerEnabled: "true"
        serviceIntegrations:
        - accounts: []
          enabled: "false"
          gcsEnabled: "false"
          name: google
          stackdriverEnabled: "false"
        - accounts: []
          enabled: "false"
          name: prometheus
        - accounts:
          - apiKey: encrypted:k8s!n:spinnaker-datadog!k:datadog-api-key
            applicationKey: encrypted:k8s!n:spinnaker-datadog!k:datadog-app-key
            endpoint:
              baseUrl: https://api.datadoghq.com
            name: plangrid
            supportedTypes:
            - METRICS_STORE
          enabled: true
          name: datadog
        - accounts:
          - apiKey: encrypted:k8s!n:spinnaker-newrelic!k:newrelic-api-key
            applicationKey: encrypted:k8s!n:spinnaker-newrelic!k:newrelic-application-key
            name: <redacted>
            supportedTypes:
            - METRICS_STORE
          enabled: true
          name: newrelic
        - accounts:
          - bucket: <redacted>
            name: canaryaws
            region: <redacted>
            rootFolder: kayenta
            supportedTypes:
            - OBJECT_STORE
            - CONFIGURATION_STORE
          enabled: "true"
          name: aws
          s3Enabled: "true"
        showAllConfigsEnabled: "true"
        stagesEnabled: "true"
        templatesEnabled: "true"
      ci:
        jenkins:
          enabled: false
          masters:
          - address: <redacted>
            name: <redacted>
            password: encrypted:k8s!n:spinnaker-jenkins-user!k:password
            username: encrypted:k8s!n:spinnaker-jenkins-user!k:username
      deploymentEnvironment:
        accountName: <redacted>
        customSizing:
          clouddriver:
            limits:
              memory: 8000Mi
            replicas: 2
            requests:
              cpu: 3
              memory: 8000Mi
          spin-deck:
            limits:
              memory: 3000Mi
            replicas: 1
            requests:
              memory: 3000Mi
          spin-echo:
            limits:
              memory: 6000Mi
            replicas: 1
            requests:
              cpu: 1
              memory: 6000Mi
          spin-fiat:
            limits:
              memory: 3000Mi
            replicas: 1
            requests:
              cpu: 1
              memory: 3000Mi
          spin-front50:
            limits:
              memory: 3000Mi
            replicas: 1
            requests:
              cpu: 1
              memory: 3000Mi
          spin-gate:
            limits:
              memory: 3000Mi
            replicas: 1
            requests:
              cpu: 1
              memory: 3000Mi
          spin-igor:
            limits:
              memory: 3000Mi
            replicas: 1
            requests:
              cpu: 2
              memory: 3000Mi
          spin-kayenta:
            limits:
              memory: 2000Mi
            replicas: 1
            requests:
              cpu: 1
              memory: 2000Mi
          spin-orca:
            limits:
              memory: 6000Mi
            replicas: 1
            requests:
              cpu: 2
              memory: 6000Mi
        size: SMALL
        type: Distributed
      metricStores:
        datadog:
          api_key: <redacted>
          enabled: true
          tags:
          - app_name:spinnaker
          - region:us-west-2
        enabled: "true"
        period: "30"
        prometheus:
          add_source_metalabels: "true"
          enabled: "false"
        stackdriver:
          enabled: "false"
      persistentStorage:
        persistentStoreType: s3
        s3:
          bucket: <redacted>
          rootFolder: front50
      providers:
        dockerRegistry:
          accounts:
          - address: <redacted>
            cacheIntervalSeconds: "90"
            cacheThreads: "2"
            email: [email protected]
            name: ecr
            paginateSize: "200"
            passwordCommand: |
              read -r AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN <<< \
                    $(aws sts assume-role --role-arn "<redacted>" \
                    --role-session-name spinnaker-ecr-assume-dev \
                    --query "[Credentials.AccessKeyId, Credentials.SecretAccessKey, Credentials.SessionToken]" \
                    --output text ); \
              export AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN; \
              aws ecr get-authorization-token --region us-east-1 --output text \
                    --query authorizationData[].authorizationToken \
                    --registry-ids <redacted> | base64 -d | sed s/^AWS://
            providerVersion: V2
            requiredGroupMembership: []
            username: AWS
          enabled: true
          primaryAccount: ecr
        kubernetes:
          accounts:
          - configureImagePullSecrets: "true"
            context: kubelet
            customResources:
            - kubernetesKind: VirtualService
            - kubernetesKind: DestinationRule
            - kubernetesKind: Policy
            - kubernetesKind: PeerAuthentication
            dockerRegistries:
            - accountName: ecr
              namespaces: []
            kubeconfigFile: encryptedFile:k8s!n:local-cluster-kubeconfig!k:local-cluster-kubeconfig
            liveManifestCalls: "true"
            name: <redacted>
            namespaces:
            - default
            - spinnaker
            - istio-system
            omitNamespaces: []
            permissions:
              EXECUTE: []
              READ: []
              WRITE: []
            providerVersion: V2
          - customResources:
            - kubernetesKind: VirtualService
            - kubernetesKind: DestinationRule
            - kubernetesKind: Policy
            - kubernetesKind: PeerAuthentication
            dockerRegistries:
            - accountName: ecr
              namespaces: []
            kubeconfigFile: encryptedFile:k8s!n:<redacted>!k:kubeconfig
            liveManifestCalls: "true"
            name: <redacted>
            namespaces:
            - default
            - spinnaker
            - istio-system
            omitNamespaces: []
            permissions:
              <redacted>
            providerVersion: V2
          enabled: "true"
          primaryAccount: <redacted>
      pubsub:
        google:
          enabled: false
          subscriptions:
          - ackDeadlineSeconds: 10
            jsonPath: encryptedFile:k8s!n:pubsub-creds-json!k:pubsub-creds-json
            messageFormat: GCR
            name: pubsub-spinnaker
            project: pg-docker
            subscriptionName: <redacted>
      security:
        apiSecurity:
          overrideBaseUrl: <redacted>
          ssl:
            enabled: "false"
        authn:
          enabled: "true"
          saml:
            enabled: true
            issuerId: <redacted>
            keyStore: encryptedFile:k8s!n:saml-keystore!k:saml-keystore
            keyStoreAliasName: saml
            keyStorePassword: encrypted:k8s!n:saml-keystore-password!k:saml-keystore-password
            metadataLocal: encryptedFile:k8s!n:spinnaker-saml-metadata-xml!k:spinnaker-saml-metadata-xml
            serviceAddress: <redacted>
            userAttributeMapping:
              roles: memberOf
              rolesDelimiter: ','
        authz:
          enabled: true
          groupMembership:
            service: EXTERNAL
        uiSecurity:
          overrideBaseUrl: <redacted>
          ssl:
            enabled: "false"
      timezone: America/Los_Angeles
      version: 1.28.1
    files: {}
    profiles:
      clouddriver:
        kubernetes:
          jobs:
            append-suffix: false
        redis:
          cache:
            enabled: false
          enabled: false
          scheduler:
            enabled: false
          taskRepository:
            enabled: false
        server:
          max-http-header-size: 32KB
          tomcat:
            max-http-header-size: 32KB
        sql:
          cache:
            enabled: true
            readBatchSize: 500
            writeBatchSize: 300
          connectionPools:
            default:
              default: true
              jdbcUrl: encrypted:k8s!n:clouddriver-mysql-creds!k:jdbc-url
              password: encrypted:k8s!n:clouddriver-mysql-creds!k:service-password
              user: encrypted:k8s!n:clouddriver-mysql-creds!k:service-username
            tasks:
              jdbcUrl: encrypted:k8s!n:clouddriver-mysql-creds!k:jdbc-url
              password: encrypted:k8s!n:clouddriver-mysql-creds!k:service-password
              user: encrypted:k8s!n:clouddriver-mysql-creds!k:service-username
          enabled: true
          migration:
            jdbcUrl: encrypted:k8s!n:clouddriver-mysql-creds!k:jdbc-url
            password: encrypted:k8s!n:clouddriver-mysql-creds!k:migrate-password
            user: encrypted:k8s!n:clouddriver-mysql-creds!k:migrate-username
          read-only: false
          scheduler:
            enabled: true
          taskRepository:
            enabled: true
          unknown-agent-cleanup-agent:
            enabled: false
      deck:
        settings-local.js: |2
                        window.spinnakerSettings.kubernetesAdHocInfraWritesEnabled = true;
      echo:
        rest:
          enabled: false
          endpoints:
          - url: <redacted>
            wrap: "false"
        slack:
          enabled: true
          token: encrypted:k8s!n:spinnaker-slack-key!k:spinnaker-slack-key
        sql:
          connectionPools:
            default:
              default: true
              jdbcUrl: encrypted:k8s!n:echo-mysql-creds!k:jdbc-url
              password: encrypted:k8s!n:echo-mysql-creds!k:service-password
              user: encrypted:k8s!n:echo-mysql-creds!k:service-username
          enabled: "true"
          migration:
            jdbcUrl: encrypted:k8s!n:echo-mysql-creds!k:jdbc-url
            password: encrypted:k8s!n:echo-mysql-creds!k:migrate-password
            user: encrypted:k8s!n:echo-mysql-creds!k:migrate-username
      fiat:
        fiat:
          executeFallback: WRITE
      front50: {}
      gate:
        redis:
          configuration:
            secure: "true"
        saml:
          maxAuthenticationAge: "86400"
      igor:
        redis:
          connection: encrypted:k8s!n:external-redis-creds!k:redis-url
          enabled: "true"
        spinnaker:
          pollingSafeguard:
            itemUpperThreshold: "25000"
      kayenta:
        datadog:
          metadataCachingIntervalMS: "2700000"
        redis:
          connection: encrypted:k8s!n:external-redis-creds!k:redis-url
      orca:
        executionRepository:
          redis:
            enabled: "false"
          sql:
            enabled: "true"
        keiko:
          queue:
            pendingExecutionService:
              redis:
                enabled: "false"
              sql:
                enabled: "true"
            redis:
              enabled: "false"
            sql:
              enabled: "true"
            zombieCheck:
              enabled: "true"
        monitor:
          activeExecutions:
            redis: "false"
        pollers:
          oldPipelineCleanup:
            enabled: "true"
            intervalMs: "3600000"
            minimumPipelineExecutions: "5"
            thresholdDays: "14"
        queue:
          pending-execution-service:
            sql:
              enabled: "true"
        sql:
          connectionPool:
            connectionTimeout: 5000
            jdbcUrl: encrypted:k8s!n:orca-mysql-creds!k:jdbc-url
            maxLifetime: 30000
            maxPoolSize: 50
            password: encrypted:k8s!n:orca-mysql-creds!k:service-password
            user: encrypted:k8s!n:orca-mysql-creds!k:service-username
          enabled: "true"
          migration:
            jdbcUrl: encrypted:k8s!n:orca-mysql-creds!k:jdbc-url
            password: encrypted:k8s!n:orca-mysql-creds!k:migrate-password
            user: encrypted:k8s!n:orca-mysql-creds!k:migrate-username
        tasks:
          daysOfExecutionHistory: "25"
    service-settings:
      clouddriver:
        env:
          JAVA_OPTS: -Xms3400M -Xmx6550M
        kubernetes:
          podAnnotations:
            iam.amazonaws.com/role: <redacted>
            pg_app: spin-clouddriver
            sidecar.istio.io/inject: "true"
      deck:
        kubernetes:
          podAnnotations:
            iam.amazonaws.com/role: <redacted>
            pg_app: spin-deck
      echo:
        env:
          JAVA_OPTS: -Xms4800M -Xmx5100M
        kubernetes:
          podAnnotations:
            iam.amazonaws.com/role: <redacted>
            pg_app: spin-echo
      fiat:
        env:
          JAVA_OPTS: -Xms2400M -Xmx2550M
        kubernetes:
          podAnnotations:
            iam.amazonaws.com/role: <redacted>
            pg_app: spin-fiat
      front50:
        env:
          JAVA_OPTS: -Xms2400M -Xmx2550M
        kubernetes:
          podAnnotations:
            iam.amazonaws.com/role: <redacted>
            pg_app: spin-front50
            sidecar.istio.io/inject: "true"
      gate:
        env:
          JAVA_OPTS: -Xms2400M -Xmx2550M
        kubernetes:
          podAnnotations:
            iam.amazonaws.com/role: <redacted>
            pg_app: spin-gate
      igor:
        env:
          JAVA_OPTS: -Xms2400M -Xmx2550M
        kubernetes:
          podAnnotations:
            iam.amazonaws.com/role: <redacted>
            pg_app: spin-igor
      kayenta:
        env:
          JAVA_OPTS: -Xms960M -Xmx1020M
        kubernetes:
          podAnnotations:
            iam.amazonaws.com/role: <redacted>
            pg_app: spin-kayenta
      orca:
        env:
          JAVA_OPTS: -Xms4800M -Xmx5100M
        kubernetes:
          podAnnotations:
            iam.amazonaws.com/role: <redacted>
            pg_app: spin-orca
      redis:
        enabled: "true"
        kubernetes:
          podAnnotations:
            pg_app: spin-redis
            sidecar.istio.io/inject: "false"
        overrideBaseUrl: encrypted:k8s!n:external-redis-creds!k:redis-url
        skipLifeCycleManagement: "true"
      rosco:
        enabled: "false"
status:
  apiUrl: <redacted>
  lastDeployed:
    config:
      hash: d1de869f2a33b703a604c946e50b2d63
      lastUpdatedAt: "2023-03-07T16:41:18Z"
    kustomize:
      hash: 37a6259cc0c1dae299a7866489dff0bd
      lastUpdatedAt: "2023-03-07T16:41:18Z"
  serviceCount: 9
  services:
  - image: us-docker.pkg.dev/spinnaker-community/docker/clouddriver:5.76.2
    name: spin-clouddriver
    readyReplicas: 2
    replicas: 2
  - image: us-docker.pkg.dev/spinnaker-community/docker/deck:3.9.1
    name: spin-deck
    readyReplicas: 1
    replicas: 1
  - image: us-docker.pkg.dev/spinnaker-community/docker/echo:2.34.2
    name: spin-echo
    readyReplicas: 1
    replicas: 1
  - image: us-docker.pkg.dev/spinnaker-community/docker/fiat:1.31.2
    name: spin-fiat
    readyReplicas: 1
    replicas: 1
  - image: us-docker.pkg.dev/spinnaker-community/docker/front50:2.25.2
    name: spin-front50
    readyReplicas: 1
    replicas: 1
  - image: us-docker.pkg.dev/spinnaker-community/docker/gate:6.55.2
    name: spin-gate
    readyReplicas: 1
    replicas: 1
  - image: us-docker.pkg.dev/spinnaker-community/docker/igor:4.7.2
    name: spin-igor
    readyReplicas: 1
    replicas: 1
  - image: us-docker.pkg.dev/spinnaker-community/docker/kayenta:2.32.2
    name: spin-kayenta
    readyReplicas: 1
    replicas: 1
  - image: us-docker.pkg.dev/spinnaker-community/docker/orca:8.24.2
    name: spin-orca
    readyReplicas: 1
    replicas: 1
  status: OK
  uiUrl: <redacted>
  version: 1.28.1

[drewripa]

Update on 08/28/2023

here is the solution that might help those who don't want to downgrade the Operator.
Spinnaker Service version: 1.29.5
Spinnaker Operator version: 1.3.1
Kubernetes version: 1.27.4-eks-2d98532

The solution was suggested by @jasonmcintosh so I give credits to him for it. Tested and works just fine with Canary config.
I had an issue with the AWS part of it, but it could be translated to anything that contains an encrypted secret string.

canary-config.yml Patch. The key point here is to have only canary enabled flag.

apiVersion: spinnaker.io/v1alpha2
kind: SpinnakerService
metadata:
  name: spinnaker
spec:
  spinnakerConfig:
    config:
      canary:
        enabled: true
        reduxLoggerEnabled: true
        stagesEnabled: true
        templatesEnabled: true
        showAllConfigsEnabled: true

profiles-kayenta.yml Patch. This is where the actual AWS (and other) config is stored. kayenta.kayenta is not a typo, that's the proper Kayenta profile > kayenta-local config format used in the Halyard version of it.

apiVersion: spinnaker.io/v1alpha2
kind: SpinnakerService
metadata:
  name: spinnaker
spec:
  spinnakerConfig:
    profiles:
      kayenta:
        kayenta:
          aws:
            enabled: true
            accounts:
            - name: <redacted>
              bucket: <redacted>
              region: <redacted>
              rootFolder: kayenta
              explicitCredentials:
                accessKey: encrypted:k8s!n:config-canary-aws!k:accessKeyId
                secretKey: encrypted:k8s!n:config-canary-aws!k:secretAccessKey
              supportedTypes:
              - CONFIGURATION_STORE
              - OBJECT_STORE
          s3:
            enabled: true
          prometheus:
            enabled: true
            accounts:
            - name: <redacted>
              endpoint:
                baseUrl: <redacted>
              supportedTypes:
              - METRICS_STORE

Moving all the configs to profiles allows you to use encrypted functionality. The AWS credentials need to be injected in a slightly different way from the original Halyard config, ee above example.