Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support custom SASL mechanisms #2256

Open
wants to merge 22 commits into
base: master
Choose a base branch
from

Commits on Aug 20, 2021

  1. Support custom SASL mechanisms

    There is some interest in supporting various SASL mechanisms not
    currently included in the library:
    
    * dpkp#2110 (DMS)
    * dpkp#2204 (SSPI)
    * dpkp#2232 (AWS_MSK_IAM)
    
    Adding these mechanisms in the core library may be undesirable due to:
    
    * Increased maintenance burden.
    * Unavailable testing environments.
    * Vendor specificity.
    
    This commit provides a quick prototype for a pluggable SASL system.
    
    ---
    
    **Example**
    
    To define a custom SASL mechanism a module must implement two methods:
    
    ```py
    
    def validate_config(conn):
        # Check configuration values, available libraries, etc.
        assert conn.config['vendor_specific_setting'] is not None, (
            'vendor_specific_setting required when sasl_mechanism=MY_SASL'
        )
    
    def try_authenticate(conn, future):
        # Do authentication routine and return resolved Future with failed
        # or succeeded state.
    ```
    
    And then the custom mechanism should be registered before initializing
    a KafkaAdminClient, KafkaConsumer, or KafkaProducer:
    
    ```py
    
    import kafka.sasl
    from kafka import KafkaProducer
    
    import my_sasl
    
    kafka.sasl.register_mechanism('MY_SASL', my_sasl)
    
    producer = KafkaProducer(sasl_mechanism='MY_SASL')
    ```
    
    ---
    
    **Notes**
    
    **ABCs**
    
    This prototype does not implement an ABC for custom SASL mechanisms.
    Using an ABC would reduce a few of the explicit assertions involved with
    registering a mechanism and is a viable option. Due to differing feature
    sets between py2/py3 this option was not explored, but shouldn't be
    difficult.
    
    **Private Methods**
    
    This prototype relies on some methods that are currently marked as
    **private** in `BrokerConnection`.
    
    * `._can_send_recv`
    * `._lock`
    * `._recv_bytes_blocking`
    * `._send_bytes_blocking`
    
    A pluggable system would require stable interfaces for these actions.
    
    **Alternative Approach**
    
    If the module-scoped dict modification in `register_mechanism` feels too
    clunky maybe the addtional mechanisms can be specified via an argument
    when initializing one of the `Kafka*` classes?
    mattoberle committed Aug 20, 2021
    Configuration menu
    Copy the full SHA
    03c357f View commit details
    Browse the repository at this point in the history

Commits on Mar 7, 2024

  1. Configuration menu
    Copy the full SHA
    b95e46d View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    78c74c0 View commit details
    Browse the repository at this point in the history

Commits on Mar 8, 2024

  1. Configuration menu
    Copy the full SHA
    e796019 View commit details
    Browse the repository at this point in the history
  2. Reconfigure tests to complete in a more timely manner and skip some i…

    …terations for Kafka 0.8.2 and Python 3.12 (dpkp#159)
    
    * skip failing tests for PyPy since they work locally
    
    * Reconfigure tests for PyPy and 3.12
    
    * Skip partitioner tests in test_partitioner.py if 3.12 and 0.8.2
    
    * Update test_partitioner.py
    
    * Update test_producer.py
    
    * Timeout tests after ten minutes
    
    * Set 0.8.2.2 to be experimental from hereon
    
    * Formally support PyPy 3.9
    wbarnha authored Mar 8, 2024
    Configuration menu
    Copy the full SHA
    38e159a View commit details
    Browse the repository at this point in the history

Commits on Mar 9, 2024

  1. Test Kafka 0.8.2.2 using Python 3.10 in the meantime (dpkp#161)

    * Test Kafka 0.8.2.2 using Python 3.11 in the meantime
    
    * Override PYTHON_LATEST conditionally in python-package.yml
    
    * Update python-package.yml
    
    * add python annotation to kafka version test matrix
    
    * Update python-package.yml
    
    * try python 3.10
    wbarnha authored Mar 9, 2024
    Configuration menu
    Copy the full SHA
    e762321 View commit details
    Browse the repository at this point in the history
  2. Remove support for EOL'ed versions of Python (dpkp#160)

    * Remove support for EOL'ed versions of Python
    
    * Update setup.py
    wbarnha authored Mar 9, 2024
    Configuration menu
    Copy the full SHA
    00750aa View commit details
    Browse the repository at this point in the history
  3. Stop testing Python 3.13 in python-package.yml (dpkp#162)

    Too many MRs to review... so little time.
    wbarnha authored Mar 9, 2024
    Configuration menu
    Copy the full SHA
    5bd1323 View commit details
    Browse the repository at this point in the history
  4. Avoid 100% CPU usage while socket is closed (dpkp#156)

    After stop/start kafka service, kafka-python may use 100% CPU caused by
    busy-retry while the socket was closed. This fix the issue by unregister
    the socket if the fd is negative.
    
    Co-authored-by: Orange Kao <[email protected]>
    wbarnha and orange-kao authored Mar 9, 2024
    Configuration menu
    Copy the full SHA
    cda8f81 View commit details
    Browse the repository at this point in the history
  5. Fix DescribeConfigsResponse_v1 config_source (dpkp#150)

    Co-authored-by: Ryar Nyah <[email protected]>
    wbarnha and ryarnyah authored Mar 9, 2024
    Configuration menu
    Copy the full SHA
    c02df08 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    7f262e0 View commit details
    Browse the repository at this point in the history

Commits on Mar 10, 2024

  1. Fix base class of DescribeClientQuotasResponse_v0 (dpkp#144)

    Co-authored-by: Denis Otkidach <[email protected]>
    wbarnha and ods authored Mar 10, 2024
    Configuration menu
    Copy the full SHA
    65eacfb View commit details
    Browse the repository at this point in the history
  2. Update license_file to license_files (dpkp#131)

    The former has been deprecated since setuptools 56
    
    Co-authored-by: micwoj92 <[email protected]>
    wbarnha and micwoj92 authored Mar 10, 2024
    Configuration menu
    Copy the full SHA
    e0ebe5d View commit details
    Browse the repository at this point in the history
  3. Update some RST documentation syntax (dpkp#130)

    * docs: Update syntax in README.rst
    
    * docs: Update code block syntax in docs/index.rst
    
    ---------
    
    Co-authored-by: HalfSweet <[email protected]>
    wbarnha and HalfSweet authored Mar 10, 2024
    Configuration menu
    Copy the full SHA
    26bb3eb View commit details
    Browse the repository at this point in the history
  4. Fix crc32c's __main__ for Python 3 (dpkp#142)

    * Fix crc32c's __main__ for Python 3
    
    * Remove TODO from _crc32c.py
    
    ---------
    
    Co-authored-by: Yonatan Goldschmidt <[email protected]>
    wbarnha and Jongy authored Mar 10, 2024
    Configuration menu
    Copy the full SHA
    88763da View commit details
    Browse the repository at this point in the history
  5. Strip trailing dot off hostname. (dpkp#133)

    Co-authored-by: Dave Voutila <[email protected]>
    wbarnha and voutilad authored Mar 10, 2024
    Configuration menu
    Copy the full SHA
    b1a4c53 View commit details
    Browse the repository at this point in the history
  6. Handle OSError to properly recycle SSL connection, fix infinite loop (d…

    …pkp#155)
    
    * handling OSError
    
    * better error output
    
    * removed traceback logging
    
    ---------
    
    Co-authored-by: Alexander Sibiryakov <[email protected]>
    wbarnha and sibiryakov authored Mar 10, 2024
    Configuration menu
    Copy the full SHA
    18eaa2d View commit details
    Browse the repository at this point in the history
  7. client_async: Allow throwing an exception upon socket error during (d…

    …pkp#134)
    
    wakeup
    
    When wakeup() is called, we sometime notice that we get
    an endless prints:
    "Unable to send to wakeup socket!".
    
    Those prints are spamming the logs.
    This commit aims to address it by allowing restating the
    application via an intentional exception raise.
    This behavior is configurable and its default is backward compatible.
    
    Signed-off-by: shimon-armis <[email protected]>
    Co-authored-by: shimon-armis <[email protected]>
    wbarnha and shimonturjeman authored Mar 10, 2024
    Configuration menu
    Copy the full SHA
    54cbd63 View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    82fae7a View commit details
    Browse the repository at this point in the history

Commits on Mar 12, 2024

  1. Log connection errors at ERROR level (dpkp#139)

    Co-authored-by: drewdogg <[email protected]>
    wbarnha and drewdogg authored Mar 12, 2024
    Configuration menu
    Copy the full SHA
    eb6fd9b View commit details
    Browse the repository at this point in the history

Commits on Mar 17, 2024

  1. Configuration menu
    Copy the full SHA
    035af5b View commit details
    Browse the repository at this point in the history
  2. Add test_msk.py by @mattoberle

    wbarnha authored Mar 17, 2024
    Configuration menu
    Copy the full SHA
    ebcfcb1 View commit details
    Browse the repository at this point in the history