Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updates for consent signal processing #5200

Merged
merged 37 commits into from
Aug 27, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
91e635d
Updates for consent signal processing
galvana Aug 15, 2024
224b5dc
Updating RequestOverrideFunction type
galvana Aug 15, 2024
7c8cda0
Minor cleanup of consentable item access
galvana Aug 15, 2024
9521550
Adding source and submitted_by fields to privacy request
galvana Aug 15, 2024
4eca107
Updating type for Admin UI
galvana Aug 15, 2024
1b0f595
Updating source values in Privacy Center to use enums
galvana Aug 15, 2024
5753e94
Storing submitted_by for privacy requests
galvana Aug 16, 2024
aa1ad40
Refactoring logic to filter out consent webhook requests
galvana Aug 16, 2024
146df41
Adding tests
galvana Aug 16, 2024
9066caf
Hiding privacy request source data behind plus flag
galvana Aug 16, 2024
ddd554f
Merge branch 'main' into PROD-1788-track-request-origin
galvana Aug 16, 2024
296fd54
Fixing tests
galvana Aug 17, 2024
b6dec0a
Adding connections to client
galvana Aug 20, 2024
d0adc40
Merge branch 'main' into PROD-1788-track-request-origin
galvana Aug 20, 2024
a967723
Merge branch 'main' into PROD-2601-updates-for-consent-signal-processing
galvana Aug 20, 2024
e22f113
Adding connections to client model
galvana Aug 20, 2024
a20d2e4
Adding token_duration_override to is_token_expired
galvana Aug 21, 2024
fd674ed
Merge branch 'main' into PROD-1788-track-request-origin
galvana Aug 21, 2024
0b7bb0f
Merge branch 'PROD-1788-track-request-origin' into PROD-2601-updates-…
galvana Aug 21, 2024
27e8774
Fixing downrev
galvana Aug 21, 2024
1d69b06
Adding support for Fides.js as a source
galvana Aug 22, 2024
22176f7
Merge branch 'PROD-1788-track-request-origin' into PROD-2601-updates-…
galvana Aug 22, 2024
7d95255
Updating client model tests
galvana Aug 22, 2024
03a389a
Merge branch 'main' into PROD-1788-track-request-origin
galvana Aug 23, 2024
227ddbd
Changes based on PR feedback
galvana Aug 23, 2024
1301015
Refactoring PrivacyRequestSource to avoid cyclic dependency
galvana Aug 23, 2024
837e347
Prettier fix
galvana Aug 23, 2024
f0b86b3
Merge branch 'main' into PROD-1788-track-request-origin
galvana Aug 24, 2024
13a0893
Fixing tests
galvana Aug 24, 2024
2593fe3
Merge branch 'PROD-1788-track-request-origin' into PROD-2601-updates-…
galvana Aug 24, 2024
1aef805
Updating Fides dataset
galvana Aug 24, 2024
384d8d5
Enabling custom privacy request field collection for tests
galvana Aug 24, 2024
2273cc4
Merge branch 'PROD-1788-track-request-origin' into PROD-2601-updates-…
galvana Aug 24, 2024
f58ed0c
Constraining the keys for the ConsentWebhookResult identity map
galvana Aug 25, 2024
2c9c8cb
Fixing tests
galvana Aug 25, 2024
30a18d5
Merge branch 'main' into PROD-2601-updates-for-consent-signal-processing
galvana Aug 26, 2024
6de8e72
Updating change log
galvana Aug 27, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .fides/db_dataset.yml
Original file line number Diff line number Diff line change
Expand Up @@ -126,6 +126,9 @@ dataset:
- name: systems
data_categories:
- system.operations
- name: connections
data_categories:
- system.operations
- name: updated_at
data_categories:
- system.operations
Expand Down
2 changes: 1 addition & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,14 +17,14 @@ The types of changes are:

## [Unreleased](https://github.com/ethyca/fides/compare/2.43.1...main)


### Added
- Added Gzip Middleware for responses [#5225](https://github.com/ethyca/fides/pull/5225)
- Adding source and submitted_by fields to privacy requests (Fidesplus) [#5206](https://github.com/ethyca/fides/pull/5206)

### Changed
- Removed unused `username` parameter from the Delighted integration configuration [#5220](https://github.com/ethyca/fides/pull/5220)
- Removed unused `ad_account_id` parameter from the Snap integration configuration [#5229](https://github.com/ethyca/fides/pull/5220)
- Updates to support consent signal processing (Fidesplus) [#5200](https://github.com/ethyca/fides/pull/5200)

### Developer Experience
- Sourcemaps are now working for fides-js in debug mode [#5222](https://github.com/ethyca/fides/pull/5222)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -250,7 +250,6 @@ export const ConsentAutomationForm = ({
color="white"
isDisabled={isSubmitting}
isLoading={isSubmitting}
loadingText="Submitting"
galvana marked this conversation as resolved.
Show resolved Hide resolved
size="sm"
variant="solid"
type="submit"
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
"""add connections to client

Revision ID: d9064e71f69d
Revises: 896ea3803770
Create Date: 2024-08-20 22:11:34.351186

"""

import sqlalchemy as sa
from alembic import op

# revision identifiers, used by Alembic.
revision = "d9064e71f69d"
down_revision = "896ea3803770"
galvana marked this conversation as resolved.
Show resolved Hide resolved
branch_labels = None
depends_on = None


def upgrade():
op.add_column(
"client",
sa.Column(
"connections", sa.ARRAY(sa.String()), server_default="{}", nullable=False
),
)


def downgrade():
op.drop_column("client", "connections")
1 change: 1 addition & 0 deletions src/fides/api/cryptography/schemas/jwt.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,3 +3,4 @@
JWE_ISSUED_AT = "iat"
JWE_PAYLOAD_ROLES = "roles"
JWE_PAYLOAD_SYSTEMS = "systems"
JWE_PAYLOAD_CONNECTIONS = "connections"
13 changes: 13 additions & 0 deletions src/fides/api/models/client.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@
from fides.api.cryptography.schemas.jwt import (
JWE_ISSUED_AT,
JWE_PAYLOAD_CLIENT_ID,
JWE_PAYLOAD_CONNECTIONS,
JWE_PAYLOAD_ROLES,
JWE_PAYLOAD_SCOPES,
JWE_PAYLOAD_SYSTEMS,
Expand All @@ -28,6 +29,7 @@
DEFAULT_SCOPES: list[str] = []
DEFAULT_ROLES: list[str] = []
DEFAULT_SYSTEMS: list[str] = []
DEFAULT_CONNECTIONS: list[str] = []


class ClientDetail(Base):
Expand All @@ -42,6 +44,9 @@ def __tablename__(self) -> str:
scopes = Column(ARRAY(String), nullable=False, server_default="{}", default=dict)
roles = Column(ARRAY(String), nullable=False, server_default="{}", default=dict)
systems = Column(ARRAY(String), nullable=False, server_default="{}", default=dict)
connections = Column(
ARRAY(String), nullable=False, server_default="{}", default=dict
)
fides_key = Column(String, index=True, unique=True, nullable=True)
user_id = Column(
String, ForeignKey(FidesUser.id_field_path), nullable=True, unique=True
Expand All @@ -60,6 +65,7 @@ def create_client_and_secret(
encoding: str = "UTF-8",
roles: list[str] | None = None,
systems: list[str] | None = None,
connections: list[str] | None = None,
) -> tuple["ClientDetail", str]:
"""Creates a ClientDetail and returns that along with the unhashed secret
so it can be returned to the user on create
Expand All @@ -77,6 +83,9 @@ def create_client_and_secret(
if not systems:
systems = DEFAULT_SYSTEMS

if not connections:
connections = DEFAULT_CONNECTIONS

salt = generate_salt()
hashed_secret = hash_with_salt(
secret.encode(encoding),
Expand All @@ -94,6 +103,7 @@ def create_client_and_secret(
"user_id": user_id,
"roles": roles,
"systems": systems,
"connections": connections,
},
)
return client, secret # type: ignore
Expand Down Expand Up @@ -122,6 +132,7 @@ def create_access_code_jwe(self, encryption_key: str) -> str:
JWE_ISSUED_AT: datetime.now().isoformat(),
JWE_PAYLOAD_ROLES: self.roles,
JWE_PAYLOAD_SYSTEMS: self.systems,
JWE_PAYLOAD_CONNECTIONS: self.connections,
}
return generate_jwe(json.dumps(payload), encryption_key)

Expand Down Expand Up @@ -155,6 +166,7 @@ def _get_root_client_detail(
scopes=scopes,
roles=roles,
systems=[],
connections=[],
)

return ClientDetail(
Expand All @@ -164,4 +176,5 @@ def _get_root_client_detail(
scopes=DEFAULT_SCOPES,
roles=DEFAULT_ROLES,
systems=DEFAULT_SYSTEMS,
connections=DEFAULT_CONNECTIONS,
)
9 changes: 6 additions & 3 deletions src/fides/api/oauth/utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
from datetime import datetime
from functools import update_wrapper
from types import FunctionType
from typing import Any, Callable, Dict, List, Tuple
from typing import Any, Callable, Dict, List, Optional, Tuple

from fastapi import Depends, HTTPException, Security
from fastapi.security import SecurityScopes
Expand Down Expand Up @@ -276,7 +276,10 @@ async def verify_oauth_client(


def extract_token_and_load_client(
authorization: str = Security(oauth2_scheme), db: Session = Depends(get_db)
authorization: str = Security(oauth2_scheme),
db: Session = Depends(get_db),
*,
token_duration_override: Optional[int] = None,
) -> Tuple[Dict, ClientDetail]:
"""Extract the token, verify it's valid, and likewise load the client as part of authorization"""
if authorization is None:
Expand All @@ -298,7 +301,7 @@ def extract_token_and_load_client(

if is_token_expired(
datetime.fromisoformat(issued_at),
CONFIG.security.oauth_access_token_expire_minutes,
token_duration_override or CONFIG.security.oauth_access_token_expire_minutes,
):
raise AuthorizationError(detail="Not Authorized for this action")

Expand Down
6 changes: 4 additions & 2 deletions src/fides/api/schemas/consentable_item.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from typing import Any, Dict, List, Optional
from typing import Dict, List, Literal, Optional

from pydantic import BaseModel, Field

Expand Down Expand Up @@ -83,7 +83,9 @@ class ConsentWebhookResult(BaseModel):
A wrapper class for the identity map and notice map values returned from a `PROCESS_CONSENT_WEBHOOK` function.
"""

identity_map: Dict[str, Any] = {}
identity_map: Dict[
Literal["email", "phone_number", "fides_user_device", "external_id"], str
] = {}
notice_map: Dict[str, UserConsentPreference] = {}
galvana marked this conversation as resolved.
Show resolved Hide resolved

@property
Expand Down
10 changes: 10 additions & 0 deletions src/fides/api/schemas/saas/shared_schemas.py
Original file line number Diff line number Diff line change
Expand Up @@ -39,3 +39,13 @@ class IdentityParamRef(BaseModel):
"""A reference to the identity type in the filter Post Processor Config"""

identity: str


class ConsentPropagationStatus(Enum):
"""
An enum for the different statuses that can be returned from a consent propagation request.
"""

executed = "executed"
no_update_needed = "no_update_needed"
missing_data = "missing_data"
57 changes: 34 additions & 23 deletions src/fides/api/service/connectors/saas_connector.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,6 @@
)
from fides.api.graph.execution import ExecutionNode
from fides.api.models.connectionconfig import ConnectionConfig, ConnectionTestStatus
from fides.api.models.consent_automation import ConsentAutomation
from fides.api.models.policy import Policy
from fides.api.models.privacy_notice import UserConsentPreference
from fides.api.models.privacy_request import PrivacyRequest, RequestTask
Expand All @@ -35,7 +34,10 @@
ReadSaaSRequest,
SaaSRequest,
)
from fides.api.schemas.saas.shared_schemas import SaaSRequestParams
from fides.api.schemas.saas.shared_schemas import (
ConsentPropagationStatus,
SaaSRequestParams,
)
from fides.api.service.connectors.base_connector import BaseConnector
from fides.api.service.connectors.saas.authenticated_client import AuthenticatedClient
from fides.api.service.connectors.saas_query_config import SaaSQueryConfig
Expand Down Expand Up @@ -609,15 +611,15 @@ def relevant_consent_identities(

@staticmethod
def build_notice_based_consentable_item_hierarchy(
session: Session, connection_config_id: str
) -> Optional[List[ConsentableItem]]:
"""Helper function to construct list of consentable items to later pass into update consent function"""
consent_automation: Optional[ConsentAutomation] = ConsentAutomation.get_by(
session, field="connection_config_id", value=connection_config_id
galvana marked this conversation as resolved.
Show resolved Hide resolved
)
if consent_automation:
connection_config: ConnectionConfig,
) -> List[ConsentableItem]:
"""
Helper function to construct list of consentable items to later pass into update consent function.
"""

if consent_automation := connection_config.consent_automation:
return build_consent_item_hierarchy(consent_automation.consentable_items)
return None
return []

@staticmethod
def obtain_notice_based_update_consent_function_or_none(
Expand Down Expand Up @@ -654,21 +656,23 @@ def run_consent_request(
identity_data: Dict[str, Any],
session: Session,
) -> bool:
"""Execute a consent request. Return whether the consent request to the third party succeeded.
# pylint: disable=too-many-branches, too-many-statements
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's a lot happening in run_consent_request now, a followup ticket could try to break some of this logic into separate functions -

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed! I thought about doing it during this pass but a follow up ticket sound better https://ethyca.atlassian.net/browse/PROD-2659

"""
Execute a consent request. Return whether the consent request to the third party succeeded.
Should only propagate either the entire set of opt in or opt out requests.
Return True if 200 OK. Raises a SkippingConsentPropagation exception if no action is taken
against the service.
"""

logger.info(
"Starting consent request for node: '{}'",
node.address.value,
)
self.set_privacy_request_state(privacy_request, node, request_task)
query_config = self.query_config(node)
saas_config = self.saas_config
fired: bool = (
False # True if the SaaS connector was successfully called / completed
)

consent_propagation_status: Optional[ConsentPropagationStatus] = None

notice_based_override_function: Optional[RequestOverrideFunction] = (
self.obtain_notice_based_update_consent_function_or_none(saas_config.type)
Expand Down Expand Up @@ -699,10 +703,8 @@ def run_consent_request(
relevant_preferences=filtered_preferences,
relevant_user_identities=identity_data,
)
notice_based_consentable_item_hierarchy: Optional[List[ConsentableItem]] = (
self.build_notice_based_consentable_item_hierarchy(
session, self.configuration.id
)
notice_based_consentable_item_hierarchy: List[ConsentableItem] = (
self.build_notice_based_consentable_item_hierarchy(self.configuration)
)
if not notice_based_consentable_item_hierarchy:
logger.info(
Expand All @@ -712,7 +714,7 @@ def run_consent_request(
raise SkippingConsentPropagation(
f"Skipping consent propagation for node {node.address.value} - no actionable consent preferences to propagate"
)
fired = self._invoke_consent_request_override(
consent_propagation_status = self._invoke_consent_request_override(
notice_based_override_function,
self.create_client(),
policy,
Expand All @@ -722,6 +724,10 @@ def run_consent_request(
notice_id_to_preference_map, # type: ignore[arg-type]
notice_based_consentable_item_hierarchy,
)
if consent_propagation_status == ConsentPropagationStatus.no_update_needed:
raise SkippingConsentPropagation(
"Consent preferences are already up-to-date"
)

else:
# follow the basic (global opt-in/out) SaaS consent flow
Expand Down Expand Up @@ -785,7 +791,7 @@ def run_consent_request(
SaaSRequestType(query_config.action),
)
)
fired = self._invoke_consent_request_override(
consent_propagation_status = self._invoke_consent_request_override(
override_function,
self.create_client(),
policy,
Expand All @@ -806,17 +812,22 @@ def run_consent_request(
node.address.value,
exc,
)
consent_propagation_status = (
ConsentPropagationStatus.missing_data
)
Comment on lines +815 to +817
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems net new - if I'm following this will now cause an exception to be raised on L826 when before it was just ignored? Is this intentional?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We previously set fired = False at the start, and if fired remained False by the end, we assumed data was missing and raised a SkippingConsentPropagation exception with the message "Missing needed values to propagate request". Now, we're setting this explicitly if we get a ValueError when building a consent request and skip_missing_param_values is set to True

except ValueError as exc:
    if consent_request.skip_missing_param_values:

Same behavior as before but now it's more explicit

continue
raise exc
client: AuthenticatedClient = self.create_client()
client.send(prepared_request)
fired = True
consent_propagation_status = ConsentPropagationStatus.executed

self.unset_connector_state()
if not fired:

if consent_propagation_status == ConsentPropagationStatus.missing_data:
raise SkippingConsentPropagation(
"Missing needed values to propagate request."
)

add_complete_system_status_for_consent_reporting(
session, privacy_request, self.configuration
)
Expand Down Expand Up @@ -986,7 +997,7 @@ def _invoke_consent_request_override(
identity_data: Optional[Dict[str, Any]] = None,
notice_id_to_preference_map: Optional[Dict[str, UserConsentPreference]] = None,
consentable_items_hierarchy: Optional[List[ConsentableItem]] = None,
) -> bool:
) -> ConsentPropagationStatus:
"""
Invokes the appropriate user-defined SaaS request override for consent requests
and performs error handling for uncaught exceptions coming out of the override.
Expand Down
Loading
Loading