Skip to content
This repository has been archived by the owner on Jan 8, 2024. It is now read-only.

Latest commit

 

History

History
764 lines (542 loc) · 24.2 KB

CHANGELOG.md

File metadata and controls

764 lines (542 loc) · 24.2 KB
Raw

Change Log

The Change Log summarizes the changes in Cloudbreak.

[v1.16.1]

Added

  • Added support for SmartSense and Flex subscriptions

Changed

  • update HDP version (2.6.0.3 -> 2.6.1.0)
  • update Ambari version (2.5.0.3 -> 2.5.1.0)
  • disable Knox from CB if blueprint contains Knox

[v1.15.0]

Added

  • Azure custom role support for credential creation

Changed

  • ability to query more information about the stack by queryparameters

[v1.14.4]

Fixed

  • fix settings generation
  • fix unable to create Azure cluster with WASB
  • preserve Knox settings when switching credentials
  • automatically create /var/run/knox after restart

Added

  • Azure Private IP support
  • add missing settings block for etl-edw blueprint
  • show all instance metadata information on shell "stack metadata"
  • add proper error handling for repair flow

Changed

  • update HDP version (2.2.0.1 -> 2.5.5.0 and 2.6.0.0 -> 2.6.0.3)
  • update Ambari version (2.5.0.0 -> 2.5.0.3)
  • disable Hive credential provider
  • Azure static IP allocation method instead of dynamic
  • Azure deallocate VM-s instead of just stopping them on cluster stop
  • conform to more strict Salt-api calls (since 2016.11.4)
  • use redhat6 repo on amazonlinux

[v1.14.0]

Added

  • new Azure connector based on Azure Java SDK
  • Azure interactive credential creation
  • new alerting system implemented based on prometheus

Changed

  • ability to specify multiple rdsconfig-s per cluster

[v1.6.3]

Changed

  • Upgraded a dependent component in order to address a network configuration issue.
  • Updated traefik to v1.1.2
  • Updated Google Cloud SDK to 1.22.0

[v1.6.2]

Fixed

  • return login error if user has multiple accounts in LDAP/AD
  • extra logging when creating the Ambari client
  • avoid deploy failures, due to concurrency in Salt highstate
  • incorrect error message collection from nodes during Salt highstate
  • reduce host name length on Azure in order to be sure that the fully qualified hostname is under 64 chars
  • do not overwrite manually changed kerberos config
  • 'securitygroup create' shell command caused a validation error due to missing provider value
  • autocreate default security groups for every provider
  • fix bower retry in web dockerfile to avoid UI build fails
  • Cloudbreak does not work with CentOS on AWS
  • Shiro config of Zepplein is always overwritten by Cloudbreak
  • fix cluster sync when a new service is added to Ambari through the wizard
  • rename all open port security group to denote unsecure
  • remove all port open security group as default

Added

  • Azure VMs supporting private deployments (when VMs have private ip only)
  • do not use the same host names for every cluster on Azure
  • configurable stack name prefix length in Azure host names
  • set up default HDP version
  • implement existing MIT KDC
  • existing AD support
  • support exsisting security group on AWS
  • separate security groups per provider
  • ability to specify image id on API to launch a cluster from that base AMI

Changed

  • OS parameter is optional when providing the HDP repo
  • modularize integration test Makefile to sh scripts

[v1.6.1]

Fixed

  • cannot send invite e-mails from Auth (aka Sultans) when smtp authentictaion is not enabled
  • recipe shell commands was not aligned with UI and with the implementation
  • avoid duplicated privateids for instances (they must be unique)
  • reserve ports on OS for well known Hadoop services
  • do not fail when there is no default VPC on AWS
  • reserve TCP/UDP ports on OS for well-known Hadoop services
  • invalid hostmetadata count after sync when a node from master hostgroup died
  • shell fix to avoid error when creating cluster using templates with encrypted EBS
  • fix concurrent removal of alerts in Autoscale (aka Periscope)
  • ensure that the gcs-connector.sh recipe runs only one time
  • fix multiple salt-minion start
  • short names for OpenStack VMs to avoid MySQL / Hive failures
  • fix uluwatu missing template volume type in some cases
  • avoid timeouts when IPV6 is disabled saltstack/salt#32719
  • do not allow cluster reset when external DB is used for Ambari
  • fix smtp related indentations in application.yml
  • introducing resource definiton cache, to avoid read the same definition from disk multiple times
  • invalid hdp repo config (update to 2.5 and support cnetos6 on AWS)
  • display AWS autoscaling group errors
  • sporadic ssh connectivity issue from on gcp
  • no transaction for cluster update
  • renamed s3role to instanceprofile on UI
  • avoid alphabetical order of recipes by filename
  • stop cluster is not possible due to invalid spot price check
  • sporadic classpath issue, since StackServiceComponentDescriptorMapFactory returns wrong object type

Added

  • OpenStack Liberty and Mitaka support
  • implement Availibility Zones for OpenStack
  • dynamic load of custom provider definitions
  • log every recipe by default (/var/log/recipses)
  • eventlog about recipe execution
  • ability to restart cluster installation in case of failed recipe

Changed

  • ambari client with sources and extra logging
  • remove unnecessary admin user
  • remove cloudbreak log highlighting
  • remove unused timeout value from recipes
  • improved logging for saltbootstrap
  • every recipe runs only once by default

[v1.6.0]

Fixed

  • proper handling of upstart config to avoid concurrency between sysv and upstart during ambari agent / server start
  • use m3 for dry-run check because it can be started in ec2-classic
  • consider auto scaling activity history in auto scaling group status checker
  • eliminate criticals reported by Sonar
  • ability to create bigger than 999 GB size volumes on Azure
  • fixed javascript test for nodecount
  • region is not a required field when posting a new stack

Added

  • ARM template JSON validation
  • added property to set static shared volume(s) in BYOS clusters
  • generate salt password per stack
  • added Hibernate field encryptor to encryt sensitive data in DB
  • ability to start GCP clusterts in a NATed environment without public IP address
  • Ambari and HDP versions are added to the /stack API call's response
  • when deploying a new subnet in an existing VPC on AWS, the CIDR range is automatically calculated when not specified explicitly on the API

Changed

  • Cloudbreak dev version uses latest image from image catalog
  • cleanup unnecessary pillar matchers
  • update default blueprints to add hive clients to worker nodes
  • API cleanup, remove invalid required fields from swagger
  • updated Ambari version to 2.4.1.0

[v1.5.0]

Fixed

  • ClusterBootstrapperTest randomly fails
  • slider does not work with ssl on python 2.7.9+
  • fix Ambari address on UI for clusters that provisioned by an older version of Cloudbreak
  • credential visibility on cluster details view
  • "Keystone Version" dropdown layouts

Added

  • added HDP version field to RDS config
  • ability to configure remote database for Ambari
  • stacks/clusters are associated with Cloudbreak version
  • new resource to configure LDAP service for Knox
  • support of Google's spot instances
  • support for c4 instance types in AWS
  • enable auto recovery for HDP services on failure via blueprint
  • ability to specify multiple subnets for the same VPC on AWS
  • ability to configure default regions

Changed

  • reinstate recipes on Uluwatu UI
  • increase Ambari password length and fix error message
  • automatic blueprint update during startup
  • on OpenStack stack name contains stack ID
  • disable upgrade of saltstack
  • remove unecessary require flags from API documentation
  • improve security of salt communication

[v1.4.0] - 2016-08-12

Fixed

  • remove invalid auto scaling group history check
  • fix sync to avoid node count corruption of instance groups
  • HDP on AWS can return 502 Bad Gateway while accessing Ambari
  • fix Ambari server ip selection on AWS
  • fix enable auto scaling with different user

Added

  • new RDS config resource
  • shell commands for RDS config operations
  • configurable image catalog
  • allow dynamically change Ambari version
  • added canada region to Azure
  • add basic auth to Zeppelin, Zeppelin is secured with the same password like Ambari
  • ability to configure a custom domain for the clusters with the CB_HOST_DISCOVERY_CUSTOM_DOMAIN env variable

Changed

  • update images for on-the-fly ambari update
  • increase poll timeout for ARM
  • disable automated map/reduce smoke test
  • parallel command execution in Ambari
  • update to Smartsense 1.3.0
  • add admin user to hadoop and hdfs groups after Ambari install
  • use only Cloudbreak address to connect Cloudbreak

[v1.3.2]

Fixed

  • Ambari restart
  • avoid duplicated alarmas
  • before upscale the sync won't change the stack state
  • upscale / downscale fix with Ambari 2.4

Added

  • reverse proxy for UI components
  • new supported volume type on aws: ST1
  • Spark reverse proxy settings
  • Spark jobhistory server reverse proxy settings
  • create admin user for Ambari views
  • added group name to AWS instance Name tags
  • add etl-edw default blueprint
  • configure Hive RDS through Cloudbreak
  • configurable default SmartSense configuration and installation
  • add SmartSense server to a single noded hostgroup if possible
  • generate SmartSense id as credential attribute
  • set SmartSense capture schedule on hosts by recipe

Changed

  • update to Ambari 2.4.0.0
  • use OpenJDK instead of Oracle JDK
  • use Ambari password as default service password
  • make Sultans base path configurable with '/sl' as default value
  • create individual root path for Cloudbreak and Periscope
  • set DNS TTL for AWS clients to <= 60s
  • rewrite url if ambari sessionid appears in cookies
  • remove MYSQL_SERVER component from blueprint if there is rds config present
  • handle AWS account id as String during SmartSense id generation
  • update proxy users hosts to avoid unauthorized connection for super-user in Oozie
  • set ONLY_STACK_DEFAULTS_APPLY as default since the new recommendation strategy does not work for Falcon and Ozzie
  • show blueprint name on review panel and hide hostroup panel by default
  • show only the previously selected network on the review panel

Removed

  • Spark and Zeppelin nginx proxy settings

[v1.3.0] - 2016-06-06

Fixed

  • delete cluster containers only if orchestrator type is container
  • show reason of failed commands when using the shell
  • login when password contains special characters
  • textarea placeholder on Internet Explorer 11
  • HDP repo verification on Uluwatu
  • wrong instance metadata status during cluster sync
  • manage platforms close button on UI
  • null pointer during router creation on non existing Openstack network

Added

  • use salt bootstrap instead of cloudbreak bootstrap
  • missing gateway port to stack response
  • enable spot price instances
  • AWS cluster creation with existing key pair
  • AWS existing SSH key pair could be configured by credential command
  • new description and output for CFN stack
  • add ssh port into instance metadata
  • seamless s3 connection
  • custom CIDR validator for subnet
  • custom tag to CloudFormation stack created by Cloudbreak
  • start termination flow stops other running flows on the same stack
  • Openstack API facing option

Changed

  • openstack network shell command improvement for the new network types
  • ability to select where to put ambari server
  • reorganize kerberos setup in Salt
  • using private address if only private address is available
  • always use ports on Openstack even there is no floating ip assigned
  • floating IP shall not be mandatory, and do not create separate ports
  • follow HBase port changes
  • move variant from advanced option to the basic options page
  • Spring update
  • elastic ips are managed by CloudFormation
  • sync starts automatically at startup
  • read nginx SSL port from parameter instead of static 443
  • sync to handle instances that are stopped on the provider side
  • Openstack metadata collection handles manually terminated instances
  • cleaned up Openstack resources in case of existing subnet

Removed

  • recipe and ssd config were removed
  • docker properties removed from UI

[v1.2.6] - 2016-05-19

Changed

  • use lates cloud images with ambari-agent:2.2.1-v20
  • map public ips to vms in case of existing vpc

[v1.2.5] - 2016-04-07

Fixed

  • slow lazy format on Azure
  • mounted disks are not visible in containers

Added

  • assume role without adding the keys into cbd Profile
  • ability to create cluster without public ip
  • high availability blueprint validator
  • support for older gcp projectids

[v1.2.4] - 2016-03-22

Fixed

  • azure storage location
  • openstack create network form
  • azure create network form
  • gcp disk type

Added

  • GCP subnetsupport to shell

Changed

  • use the provided public network id for allocation floating ips
  • allow 24 attached volumes for aws
  • cleaned up openstack resources in case of existing subnet

[v1.2.3] - 2016-03-22

Fixed

  • sudo right of CB user

Added

  • Azure default network

[v1.2.2] - 2016-03-21

Fixed

  • credential validation in case of Mesos/Marathon

[v1.2.1] - 2016-03-21

Fixed

  • live migration operations

[v1.2.0] - 2016-03-18

Fixed

  • consul recursor now exculdes both docker ip and bridge ip to avoid recursive dns recursor chain
  • docs fixed about getting default credentials (cbd login)
  • updates cb-shell to 0.5.37 to fix ssl issues

Added

  • Command cbd azure configure-arm will create your arm application which can used by cloudbreak
  • Command cbd azure deploy-dash will deploy a dash application in your Azure account
  • Command cbd start will execute the migration by default. If SKIP_DB_MIGRATION_ON_START envvar set to true in Profile, the migration will be skipped
  • Using Dns SRV record in our services instead of ambassador
  • Using docker linking system in third party services instead of ambassador
  • Integration tests are added, where cbd binary is called, not only sourced functions
  • Docker based CentOS integration test make target added
  • Uaa db migration
  • SMTP default parameters added: CLOUDBREAK_SMTP_AUTH and CLOUDBREAK_SMTP_STARTTLS_ENABLE and CLOUDBREAK_SMTP_TYPE
  • Local development Uluwatu configuration by ULUWATU_VOLUME_HOST environment variable
  • Local development Sultans configuration by SULTANS_VOLUME_HOST environment variable
  • install script for fixed version and install-latest for latest release added
  • Each snapshot artifact is uploaded as http://public-repo-1.hortonworks.com/HDP/cloudbreak/cbd-snapshot-$(uname).tgz
  • Configuration ability to enable or disable ssh fingerprint verification of virtual machines on GCP and AWS

Removed

  • Full removal of ambassador

Changed

  • cbd start doesn’t start if compose yaml regeneration is needed
  • cbd generate is less verbose, diff doesnt shown
  • cbd doctor shows diff if generate would change
  • cbd regenerate creates backup files if changes detected
  • sequenceiq/uaadb:1.0.1 is used instead of postgres:9.4.1

[v1.0.3] - 2015-09-03

Fixed

  • Authentication error with cloudbreak-shell and cloudbreak-shell-quiet is fixed
  • Command cbd update <branch> checks for artifact

Added

  • binary version of gnu-sed 4.2.2 is now included, to solve lot of osx/busybox issues
  • consul recursor test are added

Changed

  • sequenceiq/cloudbreak image updated to 1.0.3

  • debug() function made multiline capable. Use \n in messages

  • refactor bridge ip discovery to run helper docker command only once

  • consul recursor handling refactored to be more robust

[v1.0.2] - 2015-08-25

Added

  • DOCKER_CONSUL_OPTIONS config option to provide arbitrary consul option

Changed

  • Fixed docker version checker to be 1.8.1 compatible. (docker added --format option)
  • sequenceiq/cloudbreak image updated to 1.0.2
  • consul image changed from sequenceiq/consul to gliderlabs/consul
  • consul image updated to 0.5.2 (from 0.5.0)
  • consul discovers host dns settings, and uses the configured nameserver as recursor

[v1.0.0] - 2015-07-23

Fixed

  • GA Release

[v0.5.8] - 2015-07-23

Fixed

  • Fix CircleCI release. CircleCI doesn’t allow --rm on docker run

[v0.5.7] - 2015-07-23

Fixed

  • Fix make release dependency
  • Fix CHANGELOG generation at make release-next-ver avoid inserting extra -e

[v0.5.6] - 2015-07-23

Added

  • Release artifacts are published at public-repo-1.hortonworks.com/HDP/cloudbreak/

[v0.5.5] - 2015-07-10

Added

  • Command pull-parallel added for quicker/simultaneous image pull
  • Release process includes upload to public-repo s3 bucket

Changed

  • License changed from MIT to Apache v2
  • release artifact includes additional files: license/readme/notes

[v0.5.4] - 2015-07-03

Added

  • New cbd-cleanup command for removing old images or exited containers
  • Baywatch default parameters added: CB_BAYWATCH_ENABLED andCB_BAYWATCH_EXTERN_LOCATION
  • Logs are saved via lospout
  • TLS client certificate needed by Cloudbreak is generated with cbd generate
  • Command aws delete-role added
  • Command aws generate-role added
  • Command aws show-role added
  • Command cloudbreak-shell added
  • Command cloudbreak-shell-quiet added
  • Command local-dev added
  • Command token added

Changed

  • AWS authentication env varibale is fixed to use the correct AWS_SECRET_ACCESS_KEY (instead the old AWS_SECRET_KEY)
  • Using sequenceiq/ambassadord:0.5.0 docker image instead of progrium/ambassadord:latest

[v0.5.3] - 2015-06-03

Fixed

  • One-liner installer fixed, to work if previous cbd exists on path.
  • cbd update upstream changes on go-bahser broke the selfupdate functionality
  • In some environment cloudbreak starts really slow. See: details, see: commit

Added

  • New release proposal can be done by make release-next-ver

[v0.5.2] - 2015-05-21

Changed

  • Command doctor hints to run boot2docker shellinit if env is unset
  • Command init in case of OSX, DOCKER_XXX envs are initialized in local profile (Profile)
  • Default docker images are updated to:
    • sequenceiq/cloudbreak:0.5.93
    • sequenceiq/cbdb:0.5.92
    • sequenceiq/uluwatu:0.5.28
    • sequenceiq/sultans:0.5.3
    • sequenceiq/periscope:0.5.5

[v0.5.1] - 2015-05-18

Fixed

  • Issue #55: Sed handles more robust the issue with: curl includes an extra CR char in header fields.

Removed

  • deployer doesn’t specify cloud specific image defaults. If left empty, they fall back to defaults specified in java code
    • CB_AZURE_IMAGE_URI
    • CB_AWS_AMI_MAP
    • CB_OPENSTACK_IMAGE
    • CB_GCP_SOURCE_IMAGE_PATH

Changed

  • Command logs got usage example for specifying services as filter
  • Default docker images are updated to:
    • sequenceiq/cloudbreak:0.5.49
    • sequenceiq/uluwatu:0.5.16
    • sequenceiq/sutans:0.5.2

[v0.5.0] - 2015-05-08

Fixed

  • Command pull generates yaml files in case they are missing #31

Added

  • Command login Shows Uluwatu login url and credentials
  • Command regenerate deletes and generates docker-compose.yml and uaa.yml
  • Command delete added: deletes yamls and dbs
  • Command cloudbreak-shell added, right now it internale use DEBUG=1 fn fn-call
  • Command version does correct Semantic Versioning check to advise an upgrade
  • Command generate checks and shows if Profile change would result in yaml change.
  • Command start: prints uluwatu url and credential hint
  • Command doctor: fixes boot2docker date/time if not the same as on the host
  • Internal command: browse added to be able to automatically open a browser to a specified url.
  • Mini Getting Started guide added into README
  • make dev-debug installs a special cbd on local OSX, which doesn’t includes *.bash scrips, only refers them by path. No need to make dev to test small changes in bash scripts.
    • Load AWS key and AWS id from Profile
    • Command init helps to guess the PUBLIC_IP in public clouds: google, amazon

Changed

  • Command cbd env export adds export to the begining of each line
  • cbd logs accepts optional [services] parameter
  • docker-compose uses cbreak_ prefix for container naming instead of the directory name
  • Command generate prints out some more usefull info
  • uaa.yml generation wont overwrite, just instruct to move existing file (like docker-compose.yml generation)
  • Command init hint fixed on linux.
  • Command init advise to run generate if it finds a Profile
  • Command init set PRIVATE_IP the same as PUBLIC_IP for boot2docker
  • Command migrate is introduced for db migration see Migrate the databases section of README
  • Command startdb starts the cbdb and pcdb database containers only
  • Databases are not deleted after boot2docker restart
  • Import ULU_HOST_ADDRESS and ULU_SULTANS_ADDRESS from Profile

[v0.1.0] - 2015-04-16

Fixed

  • Selfupdate updates the actual running binary intead of the fixed /us/local/bin/cbd
  • SMTP default port is 25, to fix number conversion exception

Added

  • Command init creates Profile
  • Install cbd to a directory which is available on $PATH
  • Docker based test for the one-liner install from README.md: make install-test

Removed

  • update-snap command removed, replaced by parametrized update

Changed

  • Cloudbreak/Persicope/Uluwatu/Sultans Dcoker images upgraded to 0.4.x
  • Use the built in 'checksum' function instead of the external 'shasum' to generate secrets
  • Command update by default updates from latest Github release, parameter can point to branch on CircleCI
  • DOCKER_XXX env varibles are inherited, so they not needed in Profile
  • generate and compose specific commands are only available when Profile exists
  • generate command genertes docker-compose.yml and uaa.yml
  • PRIVATE_IP env var defaults to bridge IP (only PUBLC_IP is required in Profile)
  • use sulans-bin docker image istead of sultans

[v0.0.9] - 2015-04-14

Fixed

  • Bash 4.3 is included in the binary, extracted into .deps/bin upon start

[v0.0.8] - 2015-04-13

Fixed

  • Fixing deps module, golang fn: checksum added
  • CircleCI mdule defines required jq
  • Fixing PATH issue for binary deps

Added

  • uaadb start added
  • identity server start added
  • make dev added to mac based development
  • pull command added
  • logs command added

Changed

  • Docker containers are managed by docker-compose

[v0.0.7] - 2015-03-26

Added

  • make tests runs unit tests
  • docker unit tests are added
  • start command added: WIP consul, registrator starts
  • kill command addd: stops and removes cloudbreak specific containers
  • SKIP_XXX skips the container start

Changed

  • env command namespace is always exported, not only in DEBUG mode
  • env export: machine friendly config list
  • env show: human readable config list
  • circle runs unit tests
  • snapshot binaries include branch name in version string

[v0.0.6] - 2015-03-25

Fixed

  • removed dos2unix dependency for the update command

Added

  • doctor command added
  • docker-check-version command added
  • cci-latest accepts branch as parameter, needed for PR testing
  • export fn command in DEBUG mode
  • export env command in DEBUG mode
  • doctor: add instruction about setting DOCKER_XXX env vars in Profile
  • info() function added to print green text to STDOUT

Changed

  • HOME env var is also inherited (boot2docker version failed)
  • release process fully automatized

[v0.0.5] - 2015-03-23

  • update command works without dos2unix

[v0.0.4] - 2015-03-23

Fixed

  • debug function fixed
  • DEBUG, TRACE and CBD_DEFAULT_PROFILE env vars are inherited

Added

  • Profile handling added with docs
  • One-liner install added
  • Docs: install and update process described
  • Docs: release process described with sample git commands
  • Print version number in debug mode
  • update-snap downloads binary from latest os specific CircleCI binary artifact.

Changed

  • Tool specific library renamed from cloudbreak.bash to deployer.bash

[v0.0.3] - 2015-03-19

Fixed

  • make release creates binary with X.X.X version when on release branch otherwise X.X.X-gitrev

Added

  • Docs: release process described

[v0.0.2] - 2015-03-19

Added

Added

  • selfupdate command
  • gray debug to stderr

[v0.0.1] - 2015-03-18

Added

  • help command added
  • version command added
  • Added --version
  • CircleCI build
  • Linux/Darwin binary releases on github