Notes2.txt

D:\PraiseTheLord\HSBGInfotech\DevOps\sre\Notes.txt

################################################################################################
Cloud Engineering / Infrastructure as Code
################################################################################################
Infrastructure Coding - Terraform


################################################################################################
Services mesh Data planes & Control Planes - Envoy & Istio

Envoy
-----
https://www.xenonstack.com/insights/what-is-envoyproxy
https://www.envoyproxy.io/
	Envoy Proxy 
		modern, 
		high performant, 
		edge proxy, 
		
		works at both 
			L4 and 
			L7 proxies 
		most suitable for modern Cloud-Native applications 
			need proxy layer at L7. 
			
		Envoy is most comparable to software load balancers 
			like NGINX and HAProxy, 
		Has many advantages than typical proxies. 
		Allows SSL by default
		Really flexible around discovery and load balancing the workload.



	Why Envoy Proxy Matters?
	------------------------
		NGINX, HAProxy, and Envoy are all battle-tested L4 and L7 proxies
			
	Envoy has the following additional benefits -
		It’s developed by keeping modern Microservices in mind.
		It translates between HTTP-2 and HTTP-1.1.
		It proxies any TCP protocol.
		It proxies any raw data, web sockets, databases, etc.
		It enables SSL by default.
		It has in-built Service Discovery as well as Load Balancing.
		It does all the configuration dynamically
			hosts added dynamically, 
			not like writing the list to a static config file & re-reading it.
		Envoy stores the mapping of requests from clients (i.e., URLs) to services 
			in-built load balancer reads dynamically.	


		How Envoy works?
		----------------
		Configuring Envoy is a little complex 
		Envoy keeps 
			simple things very simple 
			allowing complex things to be possible to implement.

		Envoy Architecture
		-----------------
		Downstream/Upstream
		Clusters
			Backend-Downstream
		Listeners
			Frontend-Upstream
		Network Filters
			connect Listeners to Clusters
			options like
				tcp proxy
				network proxy
				http proxy etc.
		Thread Model
			Doesn't use process model
			Light weight 
			Much better performance
		Connection Pools


Envoy 
	L7 (application layer) proxy 
	communication bus designed for large modern service oriented architectures. 
	The project was born out of the belief that:
		Network should be transparent to applications. 
		When network and application problems do occur 
			quick and easy determine source.
		In practice, this goal is incredibly difficult. 
	Envoy attempts to solve this using:

		Out of process architecture: 
			Envoy is a self contained process 
			designed to run alongside every application server. 
			All of the Envoys 
				form a transparent communication mesh 
				each application sends and receives messages to and from localhost 
				unaware of the network topology. 
			out of process architecture 
				two substantial benefits 
					over the traditional library approach 
					to service to service communication:

			Envoy works with any application language. 
			A single Envoy deployment can form a mesh between 
				Java, 
				C++, 
				Go, 
				PHP, 
				Python, 
				etc. 
			Microservices can be polyglot.
			Envoy transparently bridges the gap.

			deploying library upgrades can be incredibly painful. 
			Envoy can be 
				deployed and 
				upgraded 
				quickly across an entire infrastructure transparently.

		L3/L4 filter architecture: 
			At its core, Envoy is an L3/L4 network proxy. 
			A pluggable filter chain mechanism 
				allows filters to be 
					written to perform different TCP/UDP proxy tasks and 
				inserted into the main server. 
			Filters have already been written 
				to support various tasks such as 
					raw TCP proxy, 
					UDP proxy, 
					HTTP proxy, 
					TLS client certificate authentication, 
					Redis, 
					MongoDB, 
					Postgres, 
					etc.

		HTTP L7 filter architecture: 
			HTTP 
				critical component of modern application 
			Envoy supports an additional HTTP L7 filter layer. 
			HTTP filters can be plugged into the HTTP connection management subsystem 
				that perform different tasks such as 
					buffering, 
					rate limiting, 
					routing/forwarding, 
					sniffing Amazon’s DynamoDB, etc.

			First class HTTP/2 support: 
				Envoy supports both 
					HTTP/1.1 and 
					HTTP/2. 
				Envoy can operate as a transparent HTTP/1.1 to HTTP/2 proxy in both directions. 
				Any combination of HTTP/1.1 and HTTP/2 clients and 
					target servers can be bridged. 
				The recommended service to service configuration 
					uses HTTP/2 between all Envoys 
					to create a mesh of persistent connections that 
					requests and responses can be multiplexed over.

			HTTP/3 support (currently in alpha): 
				From 1.19.0, 
					Envoy supports HTTP/3 upstream and downstream, 
					translating between any combination of HTTP/1.1, HTTP/2 and HTTP/3 in either direction.

			HTTP L7 routing: 
				Envoy supports a routing subsystem 
				capable of routing and redirecting requests based on 
					path, 
					authority, 
					content type, 
					runtime values, etc. 
				This functionality is most useful 
					when using Envoy as a front/edge proxy 
					but is also leveraged when building a service to service mesh.

		gRPC support: 
			gRPC 
				RPC framework from Google 
				uses HTTP/2 or above as the underlying multiplexed transport. 
			Envoy supports all of the HTTP/2 features 
				required to be used as the 
					routing and 
					load balancing substrate for 
						gRPC requests and responses. 
				The two systems are very complementary.

		Service discovery and dynamic configuration: 
			Envoy optionally consumes 
				layered set of dynamic configuration APIs for centralized management. 
			The layers provide an Envoy with dynamic updates about: 
				hosts within a backend cluster, 
				backend clusters themselves, 
				HTTP routing, 
				listening sockets, and 
				cryptographic material. 
			For a simpler deployment, 
				backend host discovery can be done through DNS resolution 
					(or even skipped entirely), 
					with the further layers replaced by static config files.

		Health checking: 
			The recommended way of building an Envoy mesh is to 
				treat service discovery as an eventually consistent process. 
			Envoy includes a health checking subsystem 
				which can optionally perform active health checking 
					of upstream service clusters. 
			Envoy uses 
				union of 
					service discovery and health checking 
						determine healthy load balancing targets. 
			
		Advanced load balancing: 
			Load balancing among different components in a 
				distributed system is a complex problem. 
			Envoy is also a self contained proxy 
				not a library, 
			implement advanced load balancing techniques 
				in a single place and 
				accessible to any application. 
			Currently Envoy includes support for 
				automatic retries, 
				circuit breaking, 
				global rate limiting 
					via an external rate limiting service, 
				request shadowing, and 
				outlier detection. 
			Work on request racing is WIP.

		Front/edge proxy support: 
			There is substantial benefit in using the same software at the edge 
				(observability, 
				management, 
				identical service discovery and 
				load balancing algorithms, etc.). 
		Envoy well suited as 
			edge proxy 
			for most modern web application use cases. 
		This includes 
			TLS termination, 
			HTTP/1.1 HTTP/2 and HTTP/3 support
			HTTP L7 routing.

		Best in class observability: 
			Primary goal of Envoy 
				make the network transparent. 
			But problems occur both at 
				network level and 
				application level. 
		Envoy includes robust statistics support for all subsystems. 
			statsd (and compatible providers) 
				currently supported statistics sink
				though plugging in a different one would not be difficult. 
		Statistics are also viewable via the administration port. 
		Envoy also supports distributed tracing via thirdparty providers.

			
		Downstream vs Upstream
		-----------------------
		Downstream - anything in front of envoy
		Upstream - anything in back of envoy
		
		Downstream <--> envoy <--> upstream
		image
		
		Service Mesh/Side car
		Downstream <--> envoy (http1) <--> envoy (http2)<--> upstream
		
		Clusters
		--------
		image
		Group of hosts/endpoints called cluster
		Cluster has a loadbalancing policy
		
		
		Listeners
		---------
		Listen on a port for downstream clients
		Network filters are applied to Listeners
			- TCP/HTTP filters
			- Transport sockets TLS
			
		Network Filters
		---------------
			Maps Listeners and Clusters
			Build to be extensible
				we can add our own proxy tomorrow
			TCP Proxy network filters
				envoy.filters.network.tcp_proxy
			HTTP Proxy network filters
				envoy.filters.network.http_connection_manager
			MongoDB/MySQL/GRPC Network filter
			

		Network Stack
		-------------
		Envoy works at the TCP level: 
			Layer 3/4 Network/Transport proxy. 
			It just read/write bits
				uses IP addresses and port numbers to make a decision 
					about where to route the request. 
		Working at the TCP level is drastically fast and simple. 
		Envoy also works at L7 as well simultaneously 
			to proxy different URLs (path based routing?) to different backends 
			since it needs application information available only at L7. 
		Working at 3, 4, and 7 layers allow it to 
			cover up all limitations and 
			have good performance.
		2 layers of Envoy
		Edge Envoy: 
			The standalone instance is a single point of ingress to the cluster. 
			All requests from outside first come here & it sends them to internal Envoys. 

		Sidecar Envoy: 
			This instance (replica) of an app has a sidecar Envoy, 
				runs in parallel with the app. 
				These Envoys monitors everything about the application.

		All these proxies are inside a mesh
			has internal routing 
			Each, side Envoys does health monitoring 
				identify services which go down. 
		All Envoys also gather stats from the application
			sends it to a telemetry component (Mixer in Istio). 
		All of the Envoys in Mesh configured differently by modifying Envoy configuration file, 
		according to a particular use case.


		Advantages of Envoy Proxy
		-------------------------
		Performance (much better than nginx).
		Scales horizontally.
		Proxies added/removed dynamically.
		Filter the request based on many parameters.
		For 
			Edge Envoys, 
				any number of servers
					(each of which points to its own array of hosts) 
				any number of routes added for different proxy URLs
					gives flexibility in Infra management.
			For sidecars
				Envoy have only one route
				it will proxy to the app running on localhost.
		Configuring a sidecar proxy is pretty straight-forward
			configuration updated dynamically.
		Envoy allows DNS
			easier to remember 
			new service instances added to the DNS dynamically.
		Envoy manages, observes and works best at L7.
		Envoy aligns well with Microservices world.
		Provides features such as 
			resilience, 
			observability, and 
			load balancing.
		Envoy embraces distributed architectures 
		Used in production at 
			Lyft, 
			Apple, 
			Google.

			Dropbox moved from nginx to Envoy
			
		
		
Steps
1. Install envoy 
		https://www.envoyproxy.io/docs/envoy/latest/start/install

		To run envoy as a docker container
			https://www.envoyproxy.io/docs/envoy/latest/start/docker
			
2. 
	http.yaml



docker run -d -p 1111:80 --name aspnetcore1111 mcr.microsoft.com/dotnet/samples:aspnetapp
docker run -d -p 2222:80 --name aspnetcore2222 mcr.microsoft.com/dotnet/samples:aspnetapp
docker run -d -p 3333:80 --name aspnetcore3333 mcr.microsoft.com/dotnet/samples:aspnetapp
docker run -d -p 4444:80 --name aspnetcore4444 mcr.microsoft.com/dotnet/samples:aspnetapp



Install 
https://computingforgeeks.com/how-to-install-envoy-proxy-on-centos/
  sudo yum install -y yum-utils
  yum update -y
  sudo rpm --import 'https://rpm.dl.getenvoy.io/public/gpg.CF716AF503183491.key'
  curl -sL 'https://rpm.dl.getenvoy.io/public/config.rpm.txt?distro=el&codename=7' > /tmp/tetrate-getenvoy-rpm-stable.repo
  sudo yum-config-manager --add-repo '/tmp/tetrate-getenvoy-rpm-stable.repo'
  sudo yum install -y getenvoy-envoy
  envoy --version

https://github.com/hnasr/javascript_playground/blob/master/envoy/allbackend.yaml
vi http.yaml - copy the content from above file.
envoy --config-path http.yaml
N.B: It doesn't accept .yml extension. It should be .yaml only.
http://<public ip address>:8080/


Service Mesh
------------
Handle the below infrastructure challenges
	Service Discovery
	Load Balancing
	Fault Tolerance
	Distributed Tracing 
	Telemetrics
	Security
	Mutual TLS between Services
	Network policies/Whitelisting
	Certificate expiry
	Granularity
	Bounded Contexts
	Data Modelling
	Independently releasable
	Service contracts
	Smart services, dumb-pipes


Istio can support
-----------------
Service Discovery
Load Balancing 
	(extensive set of algorithems)
Multi protocol support 
	(HTTP2/gRPC)
Fault Tolerance 
	(Circuirt breaking, Rate limiting, Auto-Retries)
Scaling
Telemetrics 
	(including wire-level like MongoDB, DynamoDB)
Distributed Tracing
Security (mTLS, policies)


Istio was initially designed with multiple services
	install each separately.
From version 1.5 they are all grouped toghether

	Control Plane: Istiod
	Worker nodes: Envoy proxy
	Istiod
		Service discovery
		Certficate 
			Has a CA 
		Configuration management



Istio handson
-------------
minikube start --cpu 6 --memory 8192
mkdir istio
cd istio
curl installable (refer documentation)
	https://istio.io/latest/docs/setup/install/istioctl/
	https://istio.io/latest/docs/setup/getting-started/#download

	curl -fsSL https://github.com/istio/istio/releases/download/1.14.3/istio-1.14.3-linux-amd64.tar.gz -o istio.tar.gz



tar xvfz <file>
set path to istio/<istio-version>/bin/
export PATH=$PATH:
istioctl


kubectl get ns
kubectl get pod
istioctl install

	core installed
	istiod installed
	ingress gateways installed
	installation complete
	
kubectl get ns
	Pick the sample app from https://istio.io/latest/docs/setup/getting-started/

kubectl apply -f <file.yaml>
cd /home/centos/istio/istio-1.14.3
kubectl apply -f samples/bookinfo/platform/kube/bookinfo.yaml
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.14/samples/bookinfo/platform/kube/bookinfo.yaml

	launch an application 
	
	istio core and ms are running
		in k8s cluster
	
	envoy proxies not injected.
	
	
kubectl get ns default --show-labels
kubectl label namespace default istio-injection=enabled

kubectl get ns default --show-labels
recreate application
kubectl delete -f manifest 	
kubectl apply -f manifest

kubectl get pod
	2/2 running
	
kubectl describe pod <any pod>

	init containers
	container id
	
	
	
Istio
	configuration
	service discovery
	certificate management
	gather telemetry data
	

Update all ClusterIP to NodePort
	inside sample/addons/ folder
	
Open the port on aws also


	kubectl apply -f samples/addons/
	
	kubectl get pod -n istio-system
	kubectl get svc -n istio-system
	
	#kubectl port-forward svc/kiali -n istio-system 20001
	
ip:20001/kiali/console/namespaces/default/services/frontend?


mandatory to have app: <name> label in deployment for kiali to work.

gitHub: https://github.com/istio/istio/tree/master/samples/addons
References: 
https://github.com/marcel-dempers/docker-development-youtube-series/tree/master/kubernetes/servicemesh/istio
################################################################################################

https://www.tutorialspoint.com/consul/consul_introduction.htm

Consul 
	Hashicorp tool 
	discovering and configuring a variety of different services 
	Built on Golang. 
	Some of the significant features that Consul provides are as follows.
		Service Discovery − 
			Using either DNS or HTTP
			applications can easily find the services they depend upon.
		Health Check Status − 
			Health checks. 
			Used by the service discovery components 
				to route traffic away from unhealthy hosts.
		Key/Value Store − 
			Consul's hierarchical key/value store - use for 
				dynamic configuration, 
				feature flagging, 
				coordination, 
				leader election, etc.
		Multi Datacenter Deployment − 
			Consul supports multiple datacenters. 
			Used for building additional layers of abstraction to grow to multiple regions.
		Web UI − 
			easy to use 
			manage all of the features in consul.

	Service Discovery
		Most important feature of Consul. 
		Defined as the detection of 
			different services and 
			network protocols 
				using which a service is found. 
		
		
	Comparison with ETCD and Zookeeper
https://www.tutorialspoint.com/consul/consul_introduction.htm
	
--------------------------------------------------------------------------------
Reference: https://www.youtube.com/watch?v=YqDZdPg3_tU
https://learn.hashicorp.com/tutorials/consul/get-started-install
Install consul based on documentation
Install consul agent 

start server
consul agent -dev -client 0.0.0.0 -bind <private ip>
yum install net-tools
netstat -ntlp

mkdir consul
cd consul

vi default.hcl
client_addr = "0.0.0.0"
bind_addr = "<private ip>"

consul agent -server -config-dir consul -data-dir /tmp/consultserver -bootstrap-expect=1 -ui=1


from browser
public-ip:8500


join client
consul agent -join <private ip of server> -bind <private ip of client> -data-dir /tmp/consuldata

stop client

mkdir consul_config_dir
cd consul_config_dir
vi consul-service.json
{
	"service":{
		"name":"myservice",
		"port":80
	}
}

consul agent -join <private ip of server> -bind <private ip of client> -data-dir /tmp/consuldata -config-dir ./consul_config_dir/


on server
dig @localhost -p 8600 myservice.service.consul

consul services 




















----------------------------------------------------------------------------------------------
Network configurations and Service Discovery - Consul

	kubectl get nodes
	kubectl get ns
	kubectl create ns consul
	
	
	install consul
	https://github.com/b1tsized/vault-tutorial/tree/main/01-getting-started
	
	consul agent -bootstrap-expect=1 -data-dir=consul-data
		-ui -bind=<ip address of where you are running it>
		
		
	should run on 
		localhost:8500/ui
		
	create two web ms with 
		web and consul discovery dependency
	https://github.com/b1tsized/vault-tutorial/tree/main/01-getting-started
		Modify node while copying 
		https://github.com/b1tsized/vault-tutorial/blob/main/01-getting-started/sys_file_templates/consul.service
	
Content of 
	vault-tutorial/01-getting-started/sys_file_templates/ui.json should be 
	vi /etc/consul.d/ui.json
	
{
  "bind_addr": "0.0.0.0",
  "advertise_addr": "10.0.2.15",
  "addresses": {
    "http": "0.0.0.0"
  }
}
################################################################################################
Continuous Integration - Jenkins
################################################################################################
Securing credentials - HashiCorp Vault & SSL & Certificates


Hashicorp Vault 
	secures, stores, and tightly controls access to confidential information like 
		tokens, 
		username/passwords, 
		certificates, 
		keys
			API keys, 
			ssh keys
		other secrets 
			in modern computing. 
	
	
	manage secrets in cloud agnostic way
	API-driven
	allows to safely store and manage 
		sensitive data in hybrid cloud env.
	Generate dynamic short lived credentials
		or encrypt application data on the fly.
		
		
	where secrets used to be?
		in file on dev. box.
		in db
		in repo like github
		etc.
		
		smart people used to encrypt it
		but where do you store the keys for encryption.
		
	Why Hashicorp vault
		Centrally 
			store
			access
			distribute 
				secrets
	
		Secure critical application data
		Encrypt Application data
			with ease
	
		Identity based access
			Authenticate to and 
			access different 
				cloud
				systems
				endpoints 
					using trusted identities

	Usecase 1
	---------
	Secrets management
		kv secrets engine
		
		client
			calls api (https://kv/secrets/foo)
				provide token
			to vault
				vault has policy
				which confirms if user has access to this key
				if user has access
					from keyvalue (kv)
					decrypted password/key is returned.
					
	
		
	Usecase 2
	---------
	Encryption as a service
	transit secret engine
	Webserver talks to Vault
		gives the token
		Vault checks the policy to confirm the access
		vault returns the encrypted data back to the server
		This can then be stored in db or s3 bucket
		when an application is trying to use it, 
			it would contact the vault to confirm if 
				the data is correct.	
				
	Usecase 3
	---------
	Vault with a cloud like aws
	client talks to vault passes the token
		vault checks the policy
		if the user is allowed, 
		vault creates credentials in aws
			aws returns keys/ userid/pwd to vault
			vault returns it to the 
				user/application.
				
	

	https://github.com/b1tsized/vault-tutorial/tree/main/01-getting-started
	https://servian.dev/get-started-with-hashicorp-vault-cc132dce627d
https://www.digitalocean.com/community/tutorials/how-to-securely-manage-secrets-with-hashicorp-vault-on-ubuntu-20-04
https://www.baeldung.com/vault	
	 
	 
-----------------------------------------------------------------------------------------------------------	 
https://www.youtube.com/watch?v=Oyvnicmxmbo
https://hub.docker.com/_/vault

COMMANDS
Get docker image: docker pull vault

Running image: 
	docker run -d --rm --name vault-server 	--cap-add=IPC_LOCK -e 'VAULT_DEV_ROOT_TOKEN_ID=myroot' -e 'VAULT_DEV_LISTEN_ADDRESS=0.0.0.0:8200' -p 8200:8200 vault
	
Get IP Address: docker inspect vault-server | grep IPAddress
Set environment variable: export VAULT_ADDR='http://172.17.0.2:8200'

Install the client by following https://www.vaultproject.io/downloads

	Basic vault cli commands
	vault
	
Authenticate to server from CLI: vault login

Write secret (CLI): vault kv put secret/mypwd mypassword=vilas1234

Read secret (CLI): vault kv get secret/mypwd 

INFO
vault info: https://vaultproject.io
vault CLI download: https://vaultproject.io/downloads
docker image at: https://hub.docker.com/_/vault
hvac info at: https://hvac.readthedocs.io/en/stable...

==========================

Python script (run pip install hvac first!):

import hvac

client = hvac.Client(url='http://172.17.0.2:8200',token="tdc-token")
print(client.is_authenticated())
read_response = client.secrets.kv.read_secret_version(path='tdc')

print(read_response['data']['data']['tdcpassword'])
-----------------------------------------------------------------------------------------------------------

	 
################################################################################################
Observability and telemetry

Observability, telemetry, and monitoring
https://cloud.ibm.com/docs/java?topic=cloud-native-observability-cn
Culture change around monitoring 
	in cloud native. 
	Applications should be 
		highly available and 
		resilient to failure
			the methods that are used to achieve those goals are different. 
	monitoring 
		not to avoid failure
		but to manage failure.

In on-premises environments
	infrastructure and middleware are provisioned 
		based on planned capacity and 
		high availability patterns, 
	for example, 
		active-active or active-passive. 
Unexpected failures can be complex in this environment
	requiring significant effort for problem determination and recovery. 
	As an example, consider the 
		tuning of 
			heap size, 
			timeouts, and 
			garbage collection policies for Java applications.
A cloud-native application is composed of 
	independent microservices and 
	required backing services. 
Even though a cloud-native application as a whole must 
	remain available and 
	continue to function, 
	individual service instances will 
		start or stop as to adjust for capacity requirements or to recover from failure.


Observability
-------------

Observability 
	data exposure and easy access 
		to information 
		required to find issues when the communications fail, 
		internal events do not occur as expected or 
		events occur when they shouldn’t. 
	
Monitoring this fluid system requires each participant to be observable. 
Each entity must produce appropriate data to support automated problem detection and alerting, manual debugging when necessary, and analysis of system health (historical trends and analytics).

What kinds of data should a service produce to be observable?

Health checks 
	(often custom HTTP endpoints) 
	help orchestrators, like Kubernetes or Cloud Foundry
		perform automated actions to maintain overall system health.
Metrics 
	are a numeric representation of data 
		collected at intervals into a time series. 
	Numerical time series data  
		is easy to store and query
		helps when looking for historical trends. 
	Over a longer period
		numerical data can be compressed into less granular aggregates, 
			daily or 
			weekly.
Log entries 
	represent discrete events. 
	Log entries are essential for debugging
	often include stack traces and other contextual information 
		can help identify the root cause of observed failures.
Distributed, request, or end-to-end tracing 
	captures the end-to-end flow of a request through the system. 
	Tracing essentially captures both relationships between services 
		(the services the request touched), and the 
		structure of work through the system 
		(synchronous or asynchronous processing, child-of or follows-from relationships).



Telemetry
---------
Cloud-native applications 
	rely on the environment for telemetry
		automatic collection and transmission of data 
			to centralized locations 
			for subsequent analysis. 
	one of the twelve factors 
		"treat logs as event streams"
		extends to all data a microservice produces to ensure it can be observed.
Kubernetes has some built-in telemetry capabilities	
	like Heapster
	but it's more likely telemetry is provided by other systems that integrate with the Kubernetes control plane. 
	As an example, 
		two of Istio's components, 
			Mixer and Envoy, act together to transparently collect telemetry from deployed applications.
Failures are no longer 
	rare, 
	disruptive occurrences. 


################################################################################################
Infrastructure Monitoring Tool 1 - Datadog
	Infrastructure Monitoring
	Introduced for infra monitoring
		Can now
			APM: performance monitor
			LOG: log monitoring
			
	SAAS
		Software as  a service
		No need to worry about
			Out of box Datadog is
				server service 
				config
				backup
				restore
				maintanance
				scalability
				availability
				security
				
	How to get started?
		https://www.datadoghq.com/
	Pricing
		https://www.datadoghq.com/pricing
	Docs
		https://docs.datadoghq.com/

		
	Setup free tier Datadog
		Integration - Agents
		Agent
			windows
			linux
			
		get centos instance
			according to the direction in datadog free account
				install datadog
				
		ps -eaf | grep data
		executables /opt/datadog-agent/bin/
		cat /etc/datadog-agent/datadog.yaml
			wait for sometime.
				check You have "1" 
		
		systemctl stop datadog-agent
		systemctl start datadog-agent
		


		
		we should be able to see the ui updated
		
		Flow
			create an account
			install agent
			Crate a dashboard
			Create a monitoring
			Install a Integration
		
		
		Architecture of Agent
		---------------------
		Agent has three main parts
			collector
			DogStatsD
			Forwarder
			
		Collector:
			Runs check on current machine
				for configured integrations
			Captures system metrics 
				memory
				CPU
		DogStatsD
			StatsD-compatible backend server
			send custom metrics to from application
		Forwarder
			Retrieves data from both DogStatsD and Collector
			Queues it up
			Send to Datadog
			
		
		New dashboard
			58 min
				add pie chart
				and timeseries	
				
		Monitoring
		----------
		Create a host related metric
			Create it 
			Test it
			Check if you got mail.
			
		
				
################################################################################################
Log Monitoring Tool 2 - ELK stake

	Why analyse log?
		Issue Debugging
		Predictive analysis
		Security analysis
		Performance analysis
		
	Problems with Log analysis
		Hetrogenous env. of multiple applications
			Inconsistent log format
			Inconsistent time format
			Decentralized logs
			Expert knowledge requirement

	Log Monitoring
	
	"ELK" 
		acronym for three open source projects: 
			Elasticsearch, 
			Logstash, 
			Kibana. 
	
	Elasticsearch 
		search and analytics engine. 
		open source
		full-text search and 
		analysis engine
		based on the Apache Lucene search engine.
		indexes and stores the data
		
	Logstash 
		server side data processing pipeline 
		log aggregator 
			collects data from various input sources
			parses and analyses very large data
				structured/unstructured data analysis.
			transforms it
				executes different 
					transformations and 
					enhancements and 
		sends it to a "stash" like Elasticsearch. 
		Plugin support 
			connect to various sources and platforms
	
	Kibana 
		works on top of Elasticsearch
		users visualize data with 
			charts and 
			graphs in 
				Elasticsearch.
		Real-time analysis.
		Customization 
			Summarization 
			Charting
			Help in Debugging
		
	Elastic Stack is the next evolution of the ELK Stack.		
	Beats 
		lightweight agents 
			installed on edge hosts to 
				collect different types of data for forwarding into the stack.
	
	
	Features of ELK
	---------------
		Search engine/search server
		NoSQL database i.e. 
			can't use SQL for queries
		Based on Apache lucene
			provides RESTful API
		Horizontally scale 
		Reliability
		Multi-tenancy
		Use indexes
			faster
		

	https://kaggle.com/
		collect data from there
		
	Installing Elasticsearch
		Dependency: Java 8 (+)
		Install methods
			Tar ball (.zip /.tar)
			Manual rpm
			yum
		Ports:
			9200: REST
			9300: (Node communication)
		Config file: /etc/elasticsearch/elasticsearch.yml
		Log files: /var/lob/elasticsearch
		Tweak to enable journal logging
		
		
	Installing Kibana
		Install methods
			Tar ball (.zip /.tar)
			Manual rpm
			yum
		Ports:
			5601: localhost
			Use nginx as reverse proxy
		Config file: /etc/kibana/kibana.yml
		Log files: Journal enabled (journalctl -f --unit=kibana)

		nginx reverse proxy
		/etc/nginx/nginx.conf
			- Delete the default server block
		/etc/nginx/conf.d/kibana.conf
			server{
				listen 80;
				server_name server.example.com;
				location/{
					proxy_pass http://localhost:5601
				}
			}


	Installing Logstash
		Install methods
			Tar ball (.zip /.tar)
			Manual rpm
			yum
		Ports:
			5044: localhost

	Filebeat
		Install methods
			Tar ball (.zip /.tar)
			Manual rpm
			yum
		Config file: /etc/filebeat/filebeat.yml
		
Follow this for hands on.
justmeandopensource
	https://github.com/justmeandopensource/elk/blob/master/INSTALL-ELK-6-CentOS7.md


	after installing elasticsearch
		systemctl status elasticsearch
		systemctl start elasticsearch
		sudo systemctl enable --now elasticsearch.service 
		systemctl status elasticsearch

		ls /var/log/elasticsearch
		jouralctl --unit elasticsearch
		yum install net-tools
		netstat -nltp
		cat /var/log/elasticsearch
		jouralctl --unit elasticsearch
		
		
	after installing kibana
	skip the nginx installation and do the below
		vi /etc/kibana/kibana.yml
			server.host: "0.0.0.0"
			server.port: 5601
		systemctl restart kibana	
		
		
		rpm -qc filebeat
		vi filebeat.yaml
		
		enabled: true
		path: 
			- /var/log/messages
			- /var/log/secure
			
			
		comment output:elasticsearch
			enable logstash output
			hosts: ["<hostname:5044>"]
			uncomment ssl.certificate_authorities: ["/etc/pki/tls/certs/logstash.crt"]
		
	systemctl restart filebeat	
---------------------------------------------------------------		
		vi /etc/
			(a)	enabled: true
			(b)	
paths:
    - /var/log/messages
    - /var/log/secure

			(c)
			comment elasticsearch output
#output.elasticsearch:
  # Array of hosts to connect to.
#  hosts: ["localhost:9200"]

			(d)
			enable logstash output
output.logstash:
  # The Logstash hosts
  hosts: ["localhost:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  ssl.certificate_authorities: ["/etc/pki/tls/certs/logstash.crt"]		

	systemctl restart filebeat
	journalctl --unit filebeat
	
	
	in kibana
		management
		Index Patterns
		Create Index Pattern
		copy paste filebeat-2022.08.15 into text field
			click next
			select timestamp
			create indexes
		Click discover on top left
			select filebeat on top left
				check client name
	-----------------------

################################################################################################
Ignore below



	yum update -y
	yum install -y java
	java -version
	hostname
	ip a
	rmp --import https://
	cat << EOF > /etc/yum.repos.
	
	yum install -y elasticsearch-oss
	vi /etc/elasticsearch/elasticsearch.yml
		/cluster
			change name
				elk
		node.name: <hostname>
		/network.host: <ip>
	add	
		ingest.geoip.downloader.enabled: false
	systemctl daemon-reload
	systemctl restart elasticsearch
	
	to debug 
		cat /etc/log/elasticsearch/elasticsearch.log
		cd /usr/share/elasticsearch/bin
./elasticsearch-create-enrollment-token -s node

			copy paste the token to kibana
			
			configure kibana host to ip address
			and restart the kibana
		
		cd /usr/share/kibana/bin
		./kibana-verification-code
		
		
	
################################################################################################
Performance & RUM (Real User Monitoring) Monitoring - NewRelic
################################################################################################

https://newrelic.com/
https://newrelic.com/signup

New Relic One
	Lets you 
		find
		visualize 
		understand 
			everything happening across 
				s/w env.
	Cut through complexity
		provide context
		see across artificial organizational bourndaries
			quickly finding and fixing problems.
			
	
New Relic supports
	New Relic APM
		Flagship product
		Montior performance and availability of web applications
	New Relic Browser
		Browser specific data 
		Understand software performance from end user perspective
	New Relic Mobile
		Quickly pinpoint performance issues in mobile apps
			operating in complext production env.
	New Relic Infrastructure
		Gives a precise picture of dynamically changing systems
		We can scale rapidly and deploy intelligently
		
	New Relic Insights
		Query application 
			business metrics
			performance data 
			customer behaviors
				at lightning speed
	New Relic Synthetics
		Test and find issues
			with software business critical functionlity
				before real users do.
				
				
	
New Relic infra
	Register at NewRelic - 30 days
	Create one VMs or use existing vm
	Install NewRelic infra agent
	Visualize a vm metrices on NewRelic one.
	
	
	
---------------------------------------------------------------------------------------------
Site Reliability Engineering

	Made of s/w engg.
	Build and implement s/w
	improve reliability of system/services
	
	Most important: 
		Reliability
			changes
			
	DevOps	
		1. Reduce organization silos
			improve throughput by	
				breaking org. barriers
				improve collaboration
		2. Accept failure as normal
			computers and machines are not 100% reliable
			human's are lot more unreliable than computers
		3. Implement gradual change	
			small changes are 
				easy to 
					review
					manage
					maintain
					ensure qos
					rollback/fix
						reduce MTTR
		4. Leverage tooling and automation
		5. Measure everything
		

	SRE class implements devops
	why SRE?
	
		
	Different views on SRE
	Google view
		
		SRE
		Defn. of Devops
		1. Reduce organization silos
			SRE takes ownership of production
			Define common 
				tooling
				goal
				path
		2. Accept failure as normal
			Accept accidents and failures
			blameless postmortem
			Don't repeat failure same way more than once
			Accept failures as normal by 
				encode error budget
					how much system is allowed to 
						go out of spec.
					Identify ways to mask it for customers
		3. Implement gradual change	
			Canary changes 
			test it for small users 
				before release big
		4. Leverage tooling and automation
			Eliminate manual work 
		5. Measure everything
			health check
			specific alerts
				pod x failed in y ns with 500 error at 	
	
	
	SRE
		Define availability
			ability of a 
				system or service 
				to perform its intended function 
				at a specified point in time. 
			Expressed as a percentage 
			(%)amount of time that the 
				system is operational and responsive to user requests, 
				compared to 
					total time that it could potentially be operational. 
			availability	= 
				total time a service is up and running / total time that has elapsed. 

			High availability systems 
				designed to minimize downtime 
				provide consistent and reliable service to users. 
		Level (Measure) of availability
			% of time service is operational and responsive to user requests
			expressed as a percentage. 
			Theoretically "The level of availability" 
				0% - 100% 
			Common levels of availability for systems include:
				99.999% (five 9s): 
					average of 5.26 minutes of downtime per year.
				99.99% (four 9s): 
					52.56 minutes of downtime per year.
				99.9% (three 9s): 
					8 hours and 45 minutes of downtime per year.
			The desired level of availability 
				depends on the specific 
					requirements and 
					priorities of the system or service, 
				specified as a Service Level Objective (SLO) in SRE.
		Plan in case of failure
			SRE teams 
				prioritize and plan for failures 
					to manage risk.

			This typically involves, including:

			Failure mode analysis: 
				Identify how system can fail 
					impact of those failures on users.

			Mitigation strategies: 
				Develop strategies to prevent failures 
					from 
						occurring or 
						reducing the impact of failures. 
					May use 
						redundancy, 
						auto-scaling, and 
						real-time monitoring.

			Incident response planning: 
				Creating and practicing an incident response plan 
					outlines the steps to be taken in the event of a failure. 
				
				assigning roles and responsibilities
				defining communication channels
				establishing a clear process for restoring service.

			Chaos engineering: 
				Proactively testing the system to identify and resolve potential failures before they occur in production.

			By planning for failures, SRE teams can minimize the impact of failures on users and restore service as quickly as possible. This helps to ensure the reliability and availability of the system over the long term.
		
	SLI (Service Level indicators)
		for a particular time.
		e.g.
			Request Latency
				Request latency 
					time it takes for 
						request to be processed and a response to be received. 
					measure of the delay experienced by a user 
						when making a request to a system or network. 
					can be affected by various factors like 
						network speed, 
						server load, and 
						distance between the client and server. 
					latency should be min.
			Batch throughput
				number of processing units 
					(e.g. records, transactions, items) 
					that a system can handle in a batch 
					or set of batches over a specific period of time. 
				measure of efficiency of the system to process
					batch jobs
				important metric to 
					monitor and optimize in many industries, like 
						finance, 
						retail, and 
						logistics, 
				High batch throughput 
					can reduce processing time and 
					increase system efficiency.
			Failures per request
				number of unsuccessful requests made to a system 
				usually percentage 
				failure/total number of requests. 
				measure of the reliability 
				can indicate the presence of issues like 
					system downtime, 
					errors, or 
					limitations 
						in capacity. 
				High failure rates may indicate a need for 
					additional resources, 
					improvements in software or hardware, or 
					changes in system configuration.
		SLI: watched closely by SRE
			on daily basis
	e.g. 
		a. 99th percentile latency of requests 
			received in the last 5 min.
				less than 300 ms
				
The 95th percentile and 99th percentile values 
	point at which 95% and 99% of your traffic is experiencing latency 
	i.e. 5% and 1% of your traffic is experiencing latency values that are out-of-range.				
				
				
		b. ratio of errors to total requests 
			last 5 min. 
				less than 1%
		
		
	
	SLO (Service Level Objectives)
		Binding target for a collection of SLIs
		e.g. Total amont of downtime in a year was less than 9 hours
		
		has upper and lower bounds
		higher SLO
			teams slow down
		lower SLO
			quality can be compromised
		watched between SRE and product
			on monthly/quaterly/annual
		
	SLA (Servie Level Agreement)
		Business agreement between 
			a customer and service provider
			based on SLOs
		watched between sales and customers
			violations result in payments
			
	
	SLA should be more lenient than SLO
		SLO failure should be a warning
			to act immediately.
		
	
	
	The silos continue between Dev and SRE? How to solve?
	-----------------------------------------------------
	Use Error budgets
	Risk and availability
	Don't attempt 100% availability
		expensive
		tech. complex
		user's don't realize benefit
			e.g. server 100% up but mobile is 99.9999% up. 
			so server uptime doesn't help.
			
	Work with PM to define
		availability target of the service
		error budet

	Accepts risks of system dictates 
		SLO		
		SLO mathematically defines Error budget
	

	SLO : 99.9%
	Error budget: 43.2 min/month failures
	
	99.9% uptime = .1% downtime
	= .001 * 30 days / monthly * 24hour/day *60 min/ hour 
	= 43.2 min/month

	Error budgets
		balance innovation and stability	
	PM's to accelerate delivery	
		should agree to lower SLO
		
	Track the SLO and SLO breaches using 
		monitoring system
		if it is about to breache
		reduce the delivery frequency 
		focus on stability
		less on innovation or new features
		
		
	But can dev. release more frequently
		result in SRE to work overtime
		Solution: Every one should buy Error budget
		For accelerating 
			ask exception 
			give golden bullets to VP for this.
			
	How about outside issues
		e.g. under cable wire cut
		will it affect my error budget
		
		Solution: error budget should be bought in by all
		
		How much error budget to 
			give to dependency?
			give to my dev?
			
			Another reason for not trying for 100% availability
			because dependencies can't be 100% up.
			
	
	Summary:
		Consider accidents as normal
		Quantify accidents
		Measure accidents
		Define acceptable accidents
		Define expectation/ratio between innovation and accidents
		
	
	Toil and Toil Budget?
	---------------------
	How about works like 
		email, 
		expense report
		meeting
		traveling
			etc.
		These are Overhead - not toil
		
	What is Toil?
	Characteristics of Toil
		Manual
		Repeatable
		Automatable 
		Tactical 
		Devoid of long term value
		Scales linearly as service grows
		
	Measuring Toil
		Don't measure toil and project work
		Account on call time as toil
		Survey, sample and log toil
	 
			
	But how to meet these?
	----------------------
	Risk Analysis
		List of items that may cause SLO violations
		Categorize as
		
					^
		Frequent	|
		Common		|
		Rare		|
					--------------------------------------->
					Minimal		Damaging	Catastrphic
					
		
Example use case
		e.g. needs db back up every 1 month.
			takes 2 hours to backup
			
			time between failures = 30 days
				30 * 24* 60 
			impact duration = 120 min.
			affects 100% users
			
			120 min * 12 months * 100% = 1440 bad min/year
			
		If error budget calculation(99.5% reliability)
		(100 - 99.5)/100 * 365 days/year * 24 hours/day * 60 min./hour = 2628 min./year 
		Monthly db backup is taking half of the error budget.
		
		
	are there more problems?
	every two weeks db becomes slow.
	takes 30 min. for people to realize and support ticket to come.
		(consider this time also)
	30 min. to resolve
	affects 50 % of users
	
	(30 min. to detect + 30 min. to resolve) * 26 weeks/year (happens every 2 weeks) * 50 % users
	= 780 bad min./year
	
	likewise
	
	
	TTD: Time to detect
	TTR: Time to resolve
	Calculated expected cost
	Risk				TTD	(min)	TTR	(min)	Freq/yr		Users		Bad/Year
	Prod. db backup		0			120			12			100%		1440 min
	Degrade QoS			30 			30			26			50			780
	Datacenter failure	15			480			1			100			495
	DDoS by IoT botnet	120			300			6			80			2016
	Bad code deploy		45			15			150			25			2250
	Upstream provider	5			30			6			75			157
		failure
																		------------
																		7138								
	all together go beyond error budget
	
	
	options
	1. Identify action items which can bring below error budget
		a. release less
		b. less DDoS attack testing
		etc.
		
		if none of them possible
			discuss with stakeholders
			modify SLO and have larger error budget
			
		
		
	How to reduce noise and manage alerts better?
	---------------------------------------------
	alert setup to know when something goes wrong
	But by the time i get up, LB has already taken care
	
	So alerts either not enough or 
	unnecessary as the issue is already taken care.
	
	Automated system keeps checking
	SLI
		is the operation successful?
		has it happened with in the defined timeframe.
	SLO
		integrated with SLI for last 30 days 
		measure it for 99.9% or 99.999%
		
	how about alerts?
		we should reduce 
			redundant alerts
				so we focus on real problems
	Solution:
		Define 
			slow burn alerts
				if your quarterly budget is under threat 
					after 2.5 months
				Raise a ticket for someone to fix during business hours
			fast burn alerts
				If your quarterly budget is under risk in few hours
					immediate alert.
	
		For doing this better you need better observability
	
	Observability
		Structured logs
		Metrics
		Traces
		
	
	Structured logs
	----------------------
		Design your logs very cleverly
		
		for e.g.
		for each request - it may include
			IP address
			timestamp
			process id
			req. id
			protocol
			type of request: GET/POST
			Request URL
			Response type
			Response code
			Response size
			user agent
			
		debug logs may include
			log level 
			date
			timestamp
			process id
			call site
			req. id
			detailed log statement
		
	Metrics
	-----------
	generally painted on a graph
		agreement type 
		e.g. number of queries
			latency
				distribution over a period of time
		cpu load
			over a period of time
			
	
		
		
	Traces
	-----------
	traces of each request
	like
	msA1
		msB5
			msC15
				
		includes 	
			timing 
			dependency
			sequence of events
			
	
	How?
	----
	Refer to metrics
		when SLI is broken
			identify where is it broken
			refer pattern 
				which 
					region
					service
					feature is broken
		Look at the traces to id
			identify the dependencies
				define context where you need to debug/attack
				identify which service is taking more/less time
				identify patterns
		Look at logs to 
			identify the real issue
			fix it
		
		Monitor to confirm the fix has brought your SLI within limit
		
		
	
	Different types of chaos
		1. too much data
		2. slow internet
		3. firewalls
		4. vpn
		5. too much communication or 
			too little communication
		
		
		Humans
		------
		Define very clear roles for each of the humans
			do exactly what they are expected to 
				without getting into way of others
			
	Need
		1. A process to declare incidents
		2. A dashboard for viewing current incidents
		3. A database for who to contact for each incidents
		
		Define threashhold
			when to declare and incident
			who can declare incidents
		
	Roles
	
		1. Incident commander (IC)
			make strategic decisions
			delegate additional roles
		
			should define
				2. operations lead
					detailed understanding
					change state of the system
					e.g.
						run commands
						grab log files
							
			3. stakeholders
			4. support team
			5. Communication lead
				communicating with external sources
			6. Planning lead
				documentation of plans
			7. Logistics load
				ensure people have all supplies as required.
				
		
		Blameless postmortem or Retrospective
		-------------------------------------
		Ensure it is blameless
		Collect metadata
		e.g.
		1. What systems were affected?
		2. Who was involved in responding?
		3. How do we find out about the events
		4. When did we start responding?
		5. What mitigations did we deploy?
		6. When the incident conclude?
		
		This should be collaborative effort
		Draft prepared by IC
			updated by everybody in the team
		
		
		Define what fix was temporary and needs rollback/fix
		e.g. 
			increase cpu on vm temporarily
			directing traffic out of a zone.
			
		Remember 
			1. failures may have multiple 
				reasons
				learning opportunities
				fixes 
					etc.
				so eliminate focusing on root cause
				
		File issue in system for each issues
			Tag it with a postmortem tag
				should help to identify 
					how much risk you are taking
			focus not on 
				how many are remaining
				how many fixed
			but focus on 
				how much risk you are taking
				what is the probability of that on 
					error budget violations
			
			
		Also try to identify positive motivators
			like
				people who went beyond
				machines that performed beyond
			appreciate that
			
	Define machine readable metadata
		track progress to overall incidence response managemnt process
			e.g. 
				decrease in total time to resolve incidence
				know whether a release was involved in failure
				
################################################################################################
Emergency Response & Alerting & Chat & Notification SMTP, SES, SNS,Pagerduty & Slack - Pagerduty & Slack
################################################################################################
Designing for fault tolerance \ HA
----------------------------------

	What is fault tolerance
	What is HA
	
	
https://www.devopsinstitute.com/sre-building-a-culture-of-reliability-resiliency-and-risk-management/




Site Reliability Engineering (SRE) 
	culture of building and operating 
		reliable, 
		resilient, 
		risk managed software systems. 

	E: for engineering.
	
Traditional software development 
	reliability 
		in design phase. 
	Changes to the functionality 
		reliability requirements. 
	
	Non-functional requirements 
		not reviewed as often. 
	Quality of Service (QoS) parameters 
		overlooked at development 
		
		Result: operational issues subsequently.

The goal 
	not push the software to the production 
	but run and manage 
		efficiently and 
		effectively 
			once it is live.  
	SRE bridges this gap 
		well-defined set of 
			practices, 
			principles and 
			culture 
				built on DevOps foundations with strong emphasis on engineering capabilities.

SRE 
	sets measurable engineering objectives 
		mapping to Service Level Objective (SLO) and 
		enables monitoring and tracking of QoS parameters such as:

	Reliability – 
		Ability of the system to function correctly
		failure free software operation
	Availability – 
		System response to disruption and fault tolerance
		avoid down-time
		Stateless application design
		fail forward database design
		HA data management
	Recoverability – 
		System ability to recover from incidences 
			through actionable alerts and 
			next-gen automation
	Serviceability – 
		Speed with which system can be repaired
		System health assessment
		monitoring and logging mechanism
		end user experience
	Elasticity – 
		System scalability and performance with reference to 
			data, 
			traffic, 
			peak load and 
			response time
	Resiliency – 
		System ability to withstand potential failure
		focus on 
			Mean Time to Repair (MTTR) over 
				average time it takes to recover from a product or system failure
			Mean Time Between Failures (MTBF)
				average time elapsed between a 
					failure and 
					next time it occurs.
	Risk Budgeting – 
		Ongoing process of risk measurement, 
		attribution and 
		allocation. 
		Optimal risk allocation to maximize expected return
	It’s of paramount importance to 
		standardize SLO, 
		identify KPIs (Key performance indicators), 
		create balanced scorecards and 
		continuously drive measurement, 
		monitoring and tracking. 
		
How you balance 
	change velocity 
	vs. 
	availability, 
	reliability, 
	security 
		and other operational attributes 
	
	Implementation of 
		continuous delivery, 
		continuous integration, 
		continuous testing, 
		continuous release and 
		deployment 
			coupled with collaboration will drive the required cultural change. 
	
	The system must recover from failure by automation.
"Anything that can go wrong will go wrong” -- Murphy’s Law


SRE team should be responsible for 
	system design and 
	development, 
	release management, 
	capacity management, 
	change management, 
	incidence management, 
	automation, 
	availability, 
	latency, 
	performance, 
	security and 
	monitoring of their services.

SRE will deliver differentiated value proposition 
	in digital reinvention journey 
	provide fast and uninterrupted services 
		through resilient systems, 
		drive operational excellence and 
		cost optimization 
			by adopting automation and 
			best practices, 
			risk management frameworks 
				to address risk tolerance of services and 
				bridge the relationship gap 
					between development and 
					operations teams and 
					enable them to communicate with 
					cost of reliability.



SRE should 
	design, 
	build, 
	operate and 
	enhance 
		software systems 

Culture of 
	risk managed, 
	reliable and 
	resilient 
		digital footprint and 
SRE is at the heart of all these happenings.	
	
	
https://linkedin.github.io/school-of-sre/level101/systems_design/fault-tolerance	
Failures 
	not avoidable 
	will happen all the time
build systems 
	can tolerate failures or 
	recover from them.

“Complex systems contain changing mixtures of failures latent within them” 
	-- How Complex Systems Fail.


Fault Tolerance - Failure Metrics
Common failure metrics that get measured and tracked for any system.

	Mean time to repair (MTTR): 
		Average time to repair and restore a failed system.
	Mean time between failures (MTBF): 
		Average operational time between one device failure or system breakdown and the next.
	Mean time to failure (MTTF): 
		Average time a device or system is expected to function before it fails.
	Mean time to detect (MTTD): 
		Average time between the onset of a problem and when the organization detects it.
	Mean time to investigate (MTTI): 
		Average time between 
			detection of an incident and 
			when the organization begins to investigate its cause and solution.
	Mean time to restore service (MTRS): 
		Average elapsed time from the detection of an incident 
			until the affected system or component is again available to users.
	Mean time between system incidents (MTBSI): 
		Average elapsed time between 
			detection of two consecutive incidents. 
		(MTBSI = MTBF + MTRS).
	Failure rate: 
		Another reliability metric, 
		frequency of system fails. 
		Number of failures over a unit of time.

Refer
https://www.splunk.com/en_us/data-insider/what-is-mean-time-to-repair.html
Fault Tolerance - Fault Isolation Terms
---------------------------------------
	Systems should have a short circuit. 
	-----------------------------------
	Say in our content sharing system, 
		if “Notifications” is not working, 
		the site should gracefully handle that failure 
			by removing/replacing the functionality 
			instead of taking the whole site down.
	Swimlane 
		a commonly used fault isolation methodologies. 
		Swimlane 
			adds a barrier to the service 
			from other services 
			so that failure on either of them won’t affect the other. 
		Say we roll out a new feature ‘Advertisement’ in our content sharing app. 
		We can have two architectures Swimlane
	
	Summary of belwo: Make calls async. 
		Let the called app populate into a storage or db.
		Calling app picks it from there.
		Better error handling
	If Ads are generated on the fly synchronously 
		during each Newsfeed request, 
			the faults in the Ads feature 
				get propagated to the Newsfeed feature. 
		Instead if we swimlane 
			the “Generation of Ads” service 
				use a shared storage 
					to populate Newsfeed App, 
				Ads failures won’t cascade to Newsfeed
			worst case if Ads don’t meet SLA 
				we can have Newsfeed without Ads.
	Let's take another example, we have come up with a new model for our Content sharing App. Here we roll out an enterprise content sharing App where enterprises pay for the service and the content should never be shared outside the enterprise.

Swimlane Principles
	Principle 1: 
		Nothing is shared 
			(also known as “share as little as possible”). 
		The less that is shared within a swim lane
			more fault isolative the swim lane becomes. 
				(as shown in Enterprise use-case)
	Principle 2: Nothing crosses a swim lane boundary. Synchronous (defined by expecting a request—not the transfer protocol) communication never crosses a swim lane boundary; if it does, the boundary is drawn incorrectly. (as shown in Ads feature)

Swimlane Approaches
	Approach 1: Swim lane the money-maker. Never allow your cash register to be compromised by other systems. (Tier 1 vs Tier 2 in enterprise use case)
	Approach 2: Swim lane the biggest sources of incidents. Identify the recurring causes of pain and isolate them. (if Ads feature is in code yellow, swim laning it is the best option)
	Approach 3: Swim lane natural barriers. Customer boundaries make good swim lanes. (Public vs Enterprise customers)

Refer
https://learning.oreilly.com/library/view/the-art-of/9780134031408/ch21.html#ch21
Applications in SRE role
Work with the DC tech or cloud team to distribute infrastructure such that its immune to switch or power failures by creating fault zones within a Data Center https://docs.microsoft.com/en-us/azure/virtual-machines/manage-availability#use-availability-zones-to-protect-from-datacenter-level-failures
Work with the partners and design interaction between services such that one service breakdown is not amplified in a cascading fashion to all upstreams	
	

################################################################################################
Training in relevant topics from areas like https://aws.amazon.com/builders-library/
################################################################################################
Minimizing correlated failures in distributed systems
	https://aws.amazon.com/builders-library/minimizing-correlated-failures-in-distributed-systems/?did=ba_card&trk=ba_card
Monolith typical architecture	
	
SOA architecture

Achieving fault tolerance
	Early days
		ran on multiple physical servers 
			behind a load balancer 
			distributed incoming requests among those servers. 
		To deal with server failures, 
			the load balancer sent a periodic health check 
				(typically a small HTTP request to a well-known URL) to each server. 
		Servers failed to respond 
			several times in a row
			load balancer take server out 
		More details: https://aws.amazon.com/builders-library/implementing-health-checks/
		
		To avoid the load balancer itself becoming a single point of failure
			two load balancers were arranged 
				primary and 
				secondary pair
		domain name system (DNS) 
			used to fail over to the secondary 
			if the primary failed. 
			multiple IP's (IP pool) attached to single dns. 
			If one fails, automatically attempt other
		If the system needed persistent storage
			similar technique was used 
			pair of replicated relational databases
			with an active primary and a standby secondary. 
		


The problem with correlated failures
	Above HA is fine with until correlated failures
		if the failures are uncorrelated
		if individual failures are always independent. 
	Unfortunately
		in the real world 
		A single underlying cause can trigger multiple failures. 
		for e.g. a bulb getting fused every day at your home.

	Generally correlated failure occur 
		when multiple independent components fail due to the same underlying cause. 
	causes can vary in complexity
		can be hard to predict. 
		(there is also cascading failure
			relatively easy to diagnose and fix.
	

	Risks of correlated failures 
		can exist in distributed systems as well
	destroy availability gains 
		from redundancy. 
	e.g. issues with 
		data center power, 
		network, and 
		cooling. 
	With very little
		dedicated engineering work and 
		careful attention, 
			common infrastructure dependencies 
				like DNS can cause correlated failures 
	Popular joke among infrastructure engineers: 
		“It wasn’t DNS. 
		It definitely wasn’t DNS! . . . It was DNS.”

	A batch of hardware components could have the same latent manufacturing defect
		or a single operator action could have an impact on multiple servers. 
		As hardware components age, 
			failure probabilities increase 
			making it more likely that multiple servers will fail at the same time. 
	These factors work against the distributed system, 
		increasing the probability of multiple simultaneous failures, 
		and thus reducing the effectiveness of redundancy in our architectures.

Using Regions and Availability Zones
	At Amazon
		reducing such risks 
			organize infrastructure into 
				multiple Regions and 
				Availability Zones. 
	Each Region is designed to be isolated 
		from all the other Regions
		it consists of multiple Availability Zones. 
	Each Availability Zone 
		1 or more discrete data centers 
		have redundant 
			power, 
			network, and 
			connectivity, and are 
			physically separated 
				by a meaningful distance from other Availability Zones. 
	Infrastructure services like 
		Amazon Elastic Compute Cloud (EC2), 
		Amazon Elastic Block Store (EBS), 
		Elastic Load Balancing (ELB), and 
		Amazon Virtual Private Cloud (VPC) 
			run independent stacks in each Availability Zone 
				share little to nothing with other Availability Zones. 
	Any change brings with it a risk of failure

	Separation of infrastructure into 
		independent Availability Zones 
		allows service teams to architect and deploy 
			their services in a way that 
				reduces the risk that a single infrastructure failure 
					will have an impact on the majority of their fleet. 

	Availability Zones 
		very powerful at reducing the risk of correlated infrastructure failures
			
	These days, AWS customers and internal service teams alike use Availability Zones to build highly available applications 
		on top of EC2. 
	
Noninfrastructure causes of correlated failures
	Physical infrastructure not the only factor causing correlated failures. 
	Any action performed by an 
		operator or 
		any tool 
			can act on multiple servers 
		at the same time 
			has the potential to introduce correlated failure modes into the system. 
	Early 
		it was common to execute a command 
		on all the servers in parallel—
			for example to grep for a specific log line, 
			or to adjust some operating system parameter. 
	These days we learned 
		such actions and 
		using such tools 
			can introduce correlated failure modes 
			into the system. 
	Anything goes wrong, 
		it will impact every server at the same time.

	While using such operational tools and practices, 
		we always make sure to introduce 
			velocity controls and 
			feedback loops. 
	In unlikely event 
		it will only impact a small percentage of the servers. 
	Don’t stop there. 
	Operational tools and practices 
		take Availability Zones into account. 
	Such operational tools 
		not directly related to the underlying infrastructure boundaries, 
		Our services are architected to 
			withstand an Availability Zone failure, 
			thus giving our teams a well understood and 
			practiced fault boundary. 
	For example, 
		a system responsible for automatically patching our servers 
			has both built-in velocity controls and 
			fail-safe mechanisms, 
		And never operates on servers from multiple Availability Zones at the same time.

	Don’t stop at Availability Zones. 
	At AWS, 
		a common practice 
			services adopt cellular architectures. 
	Instead of operating a single stack per Availability Zone or Region 
		we subdivide a service 
			into a set of identical and independent partitions 
				called cells. 
		Among other benefits, 
			cells allow us to limit the impact of unforeseen problems or failures. 
	For services that adopt cellular architectures, 
		operational tooling and practices 
			take cells into account and 
			avoid touching multiple cells at the same time.

Reducing correlated behaviors
	Inspite of all these
	Most distributed systems 
		still have one important correlating factor—
			each server in the system runs the same exact software. 
	
	The premise of the game was that you controlled a group of adorable little creatures called lemmings, 
		who were traversing a landscape filled with obstacles and deadly traps. 
	The catch was that lemmings lacked free will, so each lemming moved in a predictable algorithmic path. 
	When faced with an obstacle, a lemming would turn around. 
	When faced with a cliff or a trap, a lemming would mindlessly walk straight ahead. 
	And because every lemming followed the same algorithm, if one lemming perished, so did the rest of them. 
	As a player, you had to use the features of the terrain, and a small number of tools, to safely guide the group to the exit.

	Although most distributed systems run software that is significantly more sophisticated than the AI in control of a lemming, they can still suffer a similar fate. For most distributed systems, every server in the fleet runs the same exact software, with the same limits and failure modes. Load balancers and other work distribution mechanisms tend to do a really good job of evenly distributing the load across all the servers, which is exactly what we want them to do. However, when a software bug or a load-related failure causes one of the servers in such distributed system to fail, the same cause can also impact the rest of the fleet at the same time.

	Early in my career, one common example of such a failure was running out of open file descriptors. In many operating systems, a file descriptor is a data structure that represents an open file or a network socket. There is usually a limit on how many such file descriptors any single process can open at a time. If the limit is exceeded it can cause the application to fail in unpredictable ways. If incoming load caused the software running on one of the servers to run out of open file descriptors and fail, an evenly distributed load would frequently cause the other servers to experience the same fate.

	These days, applications are not likely to run out of open file descriptors. Thanks to many hard-won lessons, and the ample memory present in modern servers, most systems default to a high enough file descriptor limit to make running out impractical. Still, other limits, behaviors, and even bugs can exist, and they can introduce correlated failure modes into the system. Where possible, one powerful approach to minimizing risk of correlated failures that we like to use is shuffle sharding, a workload isolation technique that makes sure no two servers in the fleet see the same exact workload. That way, if anything about its workload causes a server to fail, it is unlikely to impact all the servers in the fleet. To learn more, go to the Amazon Builders’ Library article Colm MàcCarthaigh wrote about shuffle sharding: Workload isolation using shuffle-sharding.

	Correlated behaviors can be another challenge for redundant systems. This reminds me of another experience I had at home: Several months ago, I was working from home and my home internet service provider (ISP) suffered an outage in my neighborhood. I had several important meetings to attend. Luckily, I had a cellular service from a different provider and was able to quickly set up a Wi-Fi hotspot on my cellphone and tether it to my laptop. To my surprise, the cellular service began experiencing a slowdown as well. I later found out that many of my neighbors, who were also working from home (if you are reading this article at a later date, this was during the time of COVID-19, where many people in North America worked from home), had the same idea, and the cellular network in our neighborhood simply wasn’t ready to handle all that additional load. In this situation, the correlated behavior of the larger system (ISP customers failing over at the same time) caused a failure in the second, completely unrelated, system (the cellular network).

	Similar risks exist in distributed systems as well, where it’s possible to end up in a situation where a large number of clients change their behavior at the same time, creating a highly-correlated spike in load. Changing human behavior is really hard. However, changing the behavior of computers is easier. Where possible, one technique we like to use at Amazon is introducing jitter, which is adding a small amount of randomness to common behaviors and limits. Our teams are on the lookout for parts of their system that can result in a highly correlated behavior, and they use jitter to reduce such risks. Here are just a few examples of using jitter to reduce risks of correlated behaviors:
	Varying the frequency at which each server runs housekeeping jobs.
	Adding a small amount of randomness to the delay before a failed API is retried. To learn more about this, go to Marc Brooker’s Amazon Builders’ Library article on adding jitter to retries: Timeouts, retries, and backoff with jitter.
	If the system contains mode shifts (e.g., refreshing its state from a remote data store), varying the thresholds at which such mode shifts occur.
	Varying expiration times of short-term credentials and secrets.
	Varying cache time to live (TTL) across servers.
	Where applicable, adding a small amount of variability to the various resource limits, such as Java virtual machine (JVM) heap and connection pool sizes.
	I want to note that one challenge with using jitter is that it can make debugging the overall system more challenging. If the behavior of the application varies across servers and time, bugs and other failures can become harder to reproduce. We found that one approach to manage that downside is to seed the random function that is used to add jitter with a well-known parameter, like the name or IP address of the server. That way, each server will always use consistent and predictable values for its jitter, yet be different from any other server in the fleet.

Conclusion
Redundancy has been instrumental in helping us build highly available systems, even while using relatively inexpensive hardware components. By architecting our systems to run on multiple servers, we can make sure the system remains available even if some of those servers fail. However, both known and hidden factors can introduce risks of correlated failures, which eat away at the benefits of redundancy in the distributed systems. Some examples of factors that could cause correlated failures are:



Infrastructure components like power, cooling, and network.
Common dependencies like DNS.
Operator actions that touch every server.
Every server in the fleet having identical behavior and resource limits.
At Amazon, we look at ways to reduce such risks. Availability Zones are one such powerful mechanism. By organizing infrastructure into independent locations that share as little as possible with each other we give our own teams and customers a powerful tool to improve availability of their services. Beyond that, our service teams look for techniques like shuffle sharding and jitter to reduce the risk of a single software problem impacting all the servers in the fleet.



Computers: They do exactly as you tell them
Now, unlike making coffee by hand, one of the great things about computers is that everything is very repeatable, and you don’t have to trade away quality for scale. Teach a computer how to perform something once, and it can do it again and again. Each time is exactly the same. There’s still craft and a human touch, but the quality goes into how you teach computers to do things. If you skillfully teach it all of the parameters it needs to make a great cup of coffee, a computer will do it millions of times over.

Still, doing something millions of times takes more time than doing something thousands or hundreds of times. Ask a computer to add two plus two a million times. It’ll get four every time, but it will take longer than if you only asked it to do it once. When we’re operating highly reliable systems, variability is our biggest challenge. This is never truer than when we handle increases in load, state changes like reconfigurations, or when we respond to failures, like a power or network outage. Times of high stress on a system, with a lot of changes, are the worst times for things to get slower. Getting slower means queues get longer, just like they do in a barista-powered café. However, unlike a queue in a café, these system queues can set off a spiral of doom. As the system gets slower, clients retry, which makes the system slower still. This feeds itself.

Marc Brooker and David Yanacek have written in the Amazon Builders’ Library about how to get timeouts and retries right to avoid this kind of storm. However, even when you get all of that right, slowdowns are still bad. Delay when responding to failures and faults means downtime.

This is why many of our most reliable systems use very simple, very dumb, very reliable constant work patterns. Just like coffee urns. These patterns have three key features. One, they don’t scale up or slow down with load or stress. Two, they don’t have modes, which means they do the same operations in all conditions. Three, if they have any variation, it’s to do less work in times of stress so they can perform better when you need them most. There’s that anti-fragility again.

Whenever I mention anti-fragility, someone reminds me that another example of an anti-fragile pattern is a cache. Caches improve response times, and they tend to improve those response times even better under load. But most caches have modes. So, when a cache is empty, response times get much worse, and that can make the system unstable. Worse still, when a cache is rendered ineffective by too much load, it can cause a cascading failure where the source it was caching for now falls over from too much direct load. Caches appear to be anti-fragile at first, but most amplify fragility when over-stressed. Because this article isn’t focused on caches, I won’t say more here. However, if you want to learn more using caches, Matt Brinkley and Jas Chhabra have written in detail about what it takes to build a truly anti-fragile cache.

This article also isn’t just about how to serve coffee at scale, it’s about how we’ve applied constant work patterns at Amazon. I’m going to discuss two examples. Each example is simplified and abstracted a little from the real-world implementation, mainly to avoid getting into some mechanisms and proprietary technology that powers other features. Think of these examples as a distillation of the important aspects of the constant work approach.
Amazon Route 53 health checks and healthiness
It’s hard to think of a more critical function than health checks. If an instance, server, or Availability Zone loses power or networking, health checks notice and ensure that requests and traffic are directed elsewhere. Health checks are integrated into the Amazon Route 53 DNS service, into Elastic Load Balancing load balancers, and other services. Here we cover how the Route 53 health checks work. They’re the most critical of all. If DNS isn’t sending traffic to healthy endpoints, there’s no other opportunity to recover.

From a customer’s perspective, Route 53 health checks work by associating a DNS name with two or more answers (like the IP addresses for a service’s endpoints). The answers might be weighted, or they might be in a primary and secondary configuration, where one answer takes precedence as long as it’s healthy. The health of an endpoint is determined by associating each potential answer with a health check. Health checks are created by configuring a target, usually the same IP address that’s in the answer, such as a port, a protocol, timeouts, and so on. If you use Elastic Load Balancing, Amazon Relational Database Service, or any number of other AWS services that use Route 53 for high availability and failover, those services configure all of this in Route 53 on your behalf.

Route 53 has a fleet of health checkers, broadly distributed across many AWS Regions. There’s a lot of redundancy. Every few seconds, tens of health checkers send requests to their targets and check the results. These health-check results are then sent to a smaller fleet of aggregators. It’s at this point that some smart logic about health-check sensitivity is applied. Just because one of the ten in the latest round of health checks failed doesn’t mean the target is unhealthy. Health checks can be subject to noise. The aggregators apply some conditioning. For example, we might only consider a target unhealthy if at least three individual health checks have failed. Customers can configure these options too, so the aggregators apply whatever logic a customer has configured for each of their targets.

So far, everything we’ve described lends itself to constant work. It doesn’t matter if the targets are healthy or unhealthy, the health checkers and aggregators do the same work every time. Of course, customers might configure new health checks, against new targets, and each one adds slightly to the work that the health checkers and aggregators are doing. But we don’t need to worry about that as much.

One reason why we don’t worry about these new customer configurations is that our health checkers and aggregators use a cellular design. We’ve tested how many health checks each cell can sustain, and we always know where each health checking cell is relative to that limit. If the system starts approaching those limits, we add another health checking cell or aggregator cell, whichever is needed.

The next reason not to worry might be the best trick in this whole article. Even when there are only a few health checks active, the health checkers send a set of results to the aggregators that is sized to the maximum. For example, if only 10 health checks are configured on a particular health checker, it’s still constantly sending out a set of (for example) 10,000 results, if that’s how many health checks it could ultimately support. The other 9,990 entries are dummies. However, this ensures that the network load, as well as the work the aggregators are doing, won’t increase as customers configure more health checks. That’s a significant source of variance … gone.

What’s most important is that even if a very large number of targets start failing their health checks all at once—say, for example, as the result of an Availability Zone losing power—it won’t make any difference to the health checkers or aggregators. They do what they were already doing. In fact, the overall system might do a little less work. That’s because some of the redundant health checkers might themselves be in the impacted Availability Zone.

So far so good. Route 53 can check the health of targets and aggregate those health check results using a constant work pattern. But that’s not very useful on its own. We need to do something with those health check results. This is where things get interesting. It would be very natural to take our health check results and to turn them into DNS changes. We could compare the latest health check status to the previous one. If a status turns unhealthy, we’d create an API request to remove any associated answers from DNS. If a status turns healthy, we’d add it back. Or to avoid adding and removing records, we could support some kind of “is active” flag that could be set or unset on demand.

If you think of Route 53 as a sort of database, this appears to make sense, but that would be a mistake. First, a single health check might be associated with many DNS answers. The same IP address might appear many times for different DNS names. When a health check fails, making a change might mean updating one record, or hundreds. Next, in the unlikely event that an Availability Zone loses power, tens of thousands of health checks might start failing, all at the same time. There could be millions of DNS changes to make. That would take a while, and it’s not a good way to respond to an event like a loss of power.

The Route 53 design is different. Every few seconds, the health check aggregators send a fixed-size table of health check statuses to the Route 53 DNS servers. When the DNS servers receive it, they store the table in memory, pretty much as-is. That’s a constant work pattern. Every few seconds, receive a table, store it in memory. Why does Route 53 push the data to the DNS servers, rather than pull from them? That’s because there are more DNS severs than there are health check aggregators. If you want to learn more about these design choices, check out Joe Magerramov’s article on putting the smaller service in control.

Next, when a Route 53 DNS server gets a DNS query, it looks up all of the potential answers for a name. Then, at query time, it cross-references these answers with the relevant health check statuses from the in-memory table. If a potential answer’s status is healthy, that answer is eligible for selection. What’s more, even if the first answer it tried is healthy and eligible, the server checks the other potential answers anyway. This approach ensures that even if a status changes, the DNS server is still performing the same work that it was before. There’s no increase in scan or retrieval time.

I like to think that the DNS servers simply don’t care how many health checks are healthy or unhealthy, or how many suddenly change status, the code performs the very same actions. There’s no new mode of operation here. We didn’t make a large set of changes, nor did we pull a lever that activated some kind of “Availability Zone unreachable” mode. The only difference is the answers that Route 53 chooses as results. The same memory is accessed and the same amount of computer time is spent. That makes the process extremely reliable.
Amazon S3 as a configuration loop
Another application that demands extreme reliability is the configuration of foundational components from AWS, such as Network Load Balancers. When a customer makes a change to their Network Load Balancer, such as adding a new instance or container as a target, it is often critical and urgent. The customer might be experiencing a flash crowd and needs to add capacity quickly. Under the hood, Network Load Balancers run on AWS Hyperplane, an internal service that is embedded in the Amazon Elastic Compute Cloud (EC2) network. AWS Hyperplane could handle configuration changes by using a workflow. So, whenever a customer makes a change, the change is turned into an event and inserted into a workflow that pushes that change out to all of the AWS Hyperplane nodes that need it. They can then ingest the change.

The problem with this approach is that when there are a large number of changes all at once, the system will very likely slow down. More changes mean more work. When systems slow down, customers naturally resort to trying again, which slows the system down even further. That isn’t what we want.

The solution is surprisingly simple. Rather than generate events, AWS Hyperplane integrates customer changes into a configuration file that’s stored in Amazon S3. This happens right when the customer makes the change. Then, rather than respond to a workflow, AWS Hyperplane nodes fetch this configuration from Amazon S3 every few seconds. The AWS Hyperplane nodes then process and load this configuration file. This happens even if nothing has changed. Even if the configuration is completely identical to what it was the last time, the nodes process and load the latest copy anyway. Effectively, the system is always processing and loading the maximum number of configuration changes. Whether one load balancer changed or hundreds, it behaves the same.

You can probably see this coming now, but the configuration is also sized to its maximum size right from the beginning. Even when we activate a new Region and there are only a handful of Network Load Balancers active, the configuration file is still as big as it will ever be. There are dummy configuration “slots” waiting to be filled with customer configuration. However, as far the workings of AWS Hyperplane are concerned, the configuration slots there nonetheless.

Because AWS Hyperplane is a highly redundant system, there is anti-fragility in this design. If AWS Hyperplane nodes are lost, the amount of work in the system goes down, not up. There are fewer requests to Amazon S3, instead of more attempts in a workflow.

Besides being simple and robust, this approach is very cost effective. Storing a file in Amazon S3 and fetching it over and over again in a loop, even from hundreds of machines, costs far less than the engineering time and opportunity cost spent building something more complex.
Constant work and self-healing
There’s another interesting property of these constant-work designs that I haven’t mentioned yet. The designs tend to be naturally self-healing and will automatically correct for a variety of problems without intervention. For example, let’s say a configuration file was somehow corrupted while being applied. Perhaps it was mistakenly truncated by a network problem. This problem will be corrected by the next pass. Or say a DNS server missed an update entirely. It will get the next update, without building up any kind of backlog. Since a constant work system is constantly starting from a clean slate, it’s always operating in “repair everything” mode.

In contrast, a workflow type system is usually edge-triggered, which means that changes in configuration or state are what kick off the occurrence of workflow actions. These changes first have to be detected, and then actions often have to occur in a perfect sequence to work. The system needs complex logic to handle cases where some actions don’t succeed or need to be repaired because of transient corruption. The system is also prone to the build-up of backlogs. In other words, workflows aren’t naturally self-healing, you have to make them self-healing.
Design and manageability
I wrote about big-O notation earlier, and how constant work systems are usually notated as O(1). Something important to remember is that O(1) doesn’t mean that a process or algorithm only uses one operation. It means that it uses a constant number of operations regardless of the size of the input. The notation should really be O(C). Both our Network Load Balancer configuration system, and our Route 53 health check system are actually doing many thousands of operations for every “tick” or “cycle” that they iterate. But those operations don’t change because the health check statuses did, or because of customer configurations. That’s the point. They’re like coffee urns, which hold hundreds of cups of coffee at a time no matter how many customers are looking for a cup.

In the physical world, constant work patterns usually come at the cost of waste. If you brew a whole coffee urn but only get a handful of coffee drinkers, you’re going to be pouring coffee down the drain. You lose the energy it took to heat the coffee urn, the energy it took to sanitize and transport the water, and the coffee grounds. Now for coffee, those costs turn out to be small and very acceptable for a café or a caterer. There may even be more waste brewing one cup at a time because some economies of scale are lost.

For most configuration systems, or a propagation system like our health checks, this issue doesn’t arise. The difference in energy cost between propagating one health check result and propagating 10,000 health check results is negligible. Because a constant work pattern doesn’t need separate retries and state machines, it can even save energy in comparison to a design that uses a workflow.

At the same time, there are cases where the constant work pattern doesn’t fit quite as well. If you’re running a large website that requires 100 web servers at peak, you could choose to always run 100 web servers. This certainly reduces a source of variance in the system, and is in the spirit of the constant work design pattern, but it’s also wasteful. For web servers, scaling elastically can be a better fit because the savings are large. It’s not unusual to require half as many web servers off peak time as during the peak. Because that scaling happens day in and day out, the overall system can still experience the dynamism regularly enough to shake out problems. The savings can be enjoyed by the customer and the planet.
The value of a simple design
I’ve used the word “simple” several times in this article. The designs I’ve covered, including coffee urns, don’t have a lot of moving parts. That’s a kind of simplicity, but it’s not what I mean. Counting moving parts can be deceptive. A unicycle has fewer moving parts than a bicycle, but it’s much harder to ride. That’s not simpler. A good design has to handle many stresses and faults, and over enough time “survival of the fittest” tends to eliminate designs that have too many or too few moving parts or are not practical.

When I say a simple design, I mean a design that is easy to understand, use, and operate. If a design makes sense to a team that had nothing to do with its inception, that’s a good sign. At AWS, we’ve re-used the constant work design pattern many times. You might be surprised how many configuration systems can be as simple as “apply a full configuration each time in a loop.”







The ops win
The retrospective
ops meeting 
investment in agility
architectural choice