Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Title: Implement Distributed Circuit Breaker #70

Merged

Conversation

call-stack
Copy link

@call-stack call-stack commented Oct 16, 2024

Distributed Circuit Breaker Implementation

Overview

This implementation introduces a distributed version of our circuit breaker pattern. The new DistributedCircuitBreaker extends the existing CircuitBreaker functionality by supporting any storage backend through a generic CacheClient interface, enabling distributed state management across multiple instances of an application.

Key Changes

  • Added DistributedCircuitBreaker[T] struct that embeds the original CircuitBreaker[T]
  • Implemented generic CacheClient interface that can be implemented by any storage solution (Redis, Memcached, etcd, etc.)
  • Created StorageSettings to configure the distributed circuit breaker
  • Implemented StoredState struct to represent the circuit breaker state in the storage backend
  • Added helper methods for state retrieval and updates that work with any storage implementation

Features

Storage Flexibility

  • Any storage system can be used by implementing the CacheClient interface
  • Distributed state management across multiple application instances
  • Fallback to in-memory state if the storage backend is unavailable
  • Consistent circuit breaker behavior in distributed environments

Design Considerations

  • Storage operations are abstracted through a simple interface requiring only GetState and SetState methods
  • Generic type parameter T maintains type safety for the protected operation's result
  • State synchronization handles race conditions through generation numbers
  • Graceful degradation to local state when storage is unavailable

Next Steps

  • Review the implementation for any potential race conditions or edge cases.
  • Evaluate the need for additional configuration options
  • Add more examples demonstrating integration with different storage solutions

This enhancement makes our circuit breaker implementation more flexible and adaptable to different distributed storage requirements while maintaining its core functionality and reliability.

@call-stack call-stack marked this pull request as draft October 16, 2024 07:51
@call-stack call-stack marked this pull request as ready for review October 22, 2024 04:27
@call-stack call-stack changed the title Title: Implement Redis-backed Circuit BreakerFeature/redis circuit breaker Title: Implement Redis-backed Circuit Breaker Oct 22, 2024
@YoshiyukiMineo
Copy link
Member

Thank you for your PR.

The latest implementation is under v2 directory. Could you move this change to v2?

@call-stack
Copy link
Author

@YoshiyukiMineo I have move the implementation under v2. Could you please review it?

go.mod Outdated Show resolved Hide resolved
go.sum Outdated Show resolved Hide resolved
v2/redis_circuit_breaker.go Outdated Show resolved Hide resolved
@call-stack call-stack changed the title Title: Implement Redis-backed Circuit Breaker Title: Implement Distributed Circuit Breaker Nov 6, 2024
@call-stack
Copy link
Author

Hi reviewers, I've addressed the previous feedback. Would appreciate another look when you have a chance. Thanks!

@YoshiyukiMineo YoshiyukiMineo changed the base branch from master to distributed November 14, 2024 23:55
@YoshiyukiMineo YoshiyukiMineo merged commit fc36f2f into sony:distributed Nov 14, 2024
2 checks passed
@YoshiyukiMineo
Copy link
Member

Thanks. I have merged your PR into distributed branch temporarily. I will merge it into master after some slight change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants