[FSTORE-1090] Concepts & Guides for helper columns and on-demand features #333

davitbzh · 2023-11-24T13:25:52Z

No description provided.

SirOibaf

The PR also changed the behaviour around primary keys and event time. We should update the Training Data/Batch Data and Feature Vectors page to document this (e.g. how to include the pk or event time int he data returned when getting training data)

SirOibaf · 2023-12-17T18:47:55Z

docs/concepts/fs/feature_group/on_demand_feature.md

@@ -0,0 +1,6 @@
+On-demand is, a feature that is computed at request-time using application-supplied inputs for an online model.


Can you add a title and the description tag to this page?

SirOibaf · 2023-12-17T18:48:36Z

docs/user_guides/fs/feature_view/helper-columns.md

@@ -0,0 +1,127 @@
+# Helper columns


Can you add the description field here?

docs/concepts/fs/feature_group/on_demand_feature.md

SirOibaf · 2023-12-17T21:31:27Z

docs/concepts/fs/feature_group/on_demand_feature.md

@@ -0,0 +1,6 @@
+On-demand is, a feature that is computed at request-time using application-supplied inputs for an online model.
+
+In the image below shows an example of a housing price model that demonstrates how to implement an on-demand feature, a zip code (or post code) that is computed using longitude/latitude parameters. In your online application, longitude and latitude are provided as parameters to the application, and the same python function used to calculate the zip code in the feature pipeline is used to compute the zip code in the Online Inference pipeline. This is achieved by implementing the on-demand features as a Python function in a Python module. Also ensure that the same version of the Python module is installed in both the feature and inference pipelines.


Switch the order here - put the "this is achived by implementing the on-demand feature as python function" first and then present the example.

SirOibaf · 2023-12-17T21:50:55Z

docs/user_guides/fs/feature_view/helper-columns.md

@@ -0,0 +1,127 @@
+# Helper columns
+
+HSFS provides functionality to define two types of helper columns `inference_helper_columns` and `training_helper_columns` to [feature views](./overview.md).


Don't mention HSFS - people might be using the feature store using the Hopsworks library and they get confused on what hsfs means.

Say something like - when defining a feature view users can mark certain features as helper columns or training columns....

SirOibaf · 2023-12-17T22:00:47Z

docs/user_guides/fs/feature_view/helper-columns.md

+product to assign different weights during the training time.
+
+## Definition
+Both inference and training helper column name(s) must be part of the `Query` object. If helper column name(s) belong to feature group that is part of a `Join` with `prefix` defined, then this prefix needs to prepended


Use a !!! note section to call out this prefix thing.

SirOibaf · 2023-12-17T22:01:55Z

docs/user_guides/fs/feature_view/helper-columns.md

+        query = label_fg.select("fraud_label")\
+                        .join(cc_profile.select("expiry_date"))\
+                        .join(trans_fg.select(["category", "amount",  "days_until_card_expires", "date_of_transaction"
+                                               "locaton_delta", "longitude", "latitude", "category"])) \
+                        .join(window_aggs_fg.select_except(["trans_volume_mstd", "trans_volume_mavg", "trans_freq", 


Do we need to have some many feature groups here? It's not htat people can copy paste it directly, so I think we should limit the setup code to the bare minimum and focus on the important part, the feature view definition.

SirOibaf · 2023-12-17T22:02:38Z

docs/user_guides/fs/feature_view/helper-columns.md

+        ```
+
+## Retrieval
+When replaying a `Query` during model inference, helper columns will be omitted. However, they can be optionally fetched with inference or training data.


You don't "replay a query" - You are retrieving data for model inference.

SirOibaf · 2023-12-17T22:03:54Z

docs/user_guides/fs/feature_view/helper-columns.md

+        # compute location delta
+        df['loc_delta_t_minus_1'] = df.apply(lambda row: location_delta(row['longitude'], 
+                                                                        row['latitute'],
+                                                                        row['longitude_prev'], 
+                                                                        row['latitute_prev']), axis=1)
+
+        # compute time delta
+        df['days_until_card_expires'] = df.apply(lambda row: time_delta(row['date_of_transaction'],  


Same comment as above, have a single example so that what we care about doesn't get lost in all this code.

SirOibaf · 2023-12-17T22:04:26Z

docs/user_guides/fs/feature_view/helper-columns.md

+        # here cc_num, longitute, lattitude and date_of_transaction are provided as parameters to the application
+        cc_num = ...
+        longitude = ...
+        latitute = ...
+        date_of_transaction = ...
+
+        # get previous transaction location of this credit card
+        inference_helper = feature_view.get_inference_helper({"cc_num": cc_num}, return_type="dict")
+
+        # compute location delta 
+        loc_delta_t_minus_1 = location_delta(longitude, 
+                                             latitute, 
+                                             inference_helper['longitude'], 
+                                             inference_helper['latitute'])
+
+        # compute time delta 
+        days_until_card_expires = time_delta(date_of_transaction,
+                                             inference_helper['expiry_date'])
+
+        # Now get assembled feature vector for prediction
+        feature_vector = feature_view.get_feature_vector({"cc_num": cc_num}, 
+                                                          passed_features={"loc_delta_t_minus_1": loc_delta_t_minus_1, 


Same number of example comment.

Co-authored-by: Fabio Buso <[email protected]>

docs/user_guides/fs/feature_view/batch-data.md

docs/user_guides/fs/feature_view/training-data.md

docs/user_guides/fs/feature_view/helper-columns.md

SirOibaf · 2023-12-18T22:16:59Z

docs/user_guides/fs/feature_view/helper-columns.md

+    Both inference and training helper column name(s) must be part of the `Query` object. If helper column name(s) belong to feature group that is part of a `Join` with `prefix` defined, then this prefix needs to prepended
+    to the original column name when defining helper column list.
+
+# Inference Helper columns


This needs to be a h2 title not h1, i.e., put another # at the beginning

SirOibaf · 2023-12-18T22:18:31Z

docs/user_guides/fs/feature_view/helper-columns.md

+### Retrieval
+When retrieving data for model inference, helper columns will be omitted. However, they can be optionally fetched with inference or training data.
+
+### Batch inference


I think you want #### here.

SirOibaf · 2023-12-18T22:18:39Z

docs/user_guides/fs/feature_view/helper-columns.md

+        df = df[[f.name for f in feature_view.features if not (f.label or f.inference_helper_column or f.training_helper_column)]]
+        ```
+
+### Online inference


#### here.

Co-authored-by: Fabio Buso <[email protected]>

…davitbzh/logicalclocks.github.io into ondemad_features_helper_columns

…ures (#333) Co-authored-by: Fabio Buso <[email protected]> Co-authored-by: davitbzh <[email protected]>

helper-columns

16e8922

davitbzh requested review from jimdowling and SirOibaf November 24, 2023 13:25

type

671effe

SirOibaf requested changes Dec 17, 2023

View reviewed changes

davitbzh and others added 2 commits December 17, 2023 23:57

Update docs/concepts/fs/feature_group/on_demand_feature.md

eacb635

Co-authored-by: Fabio Buso <[email protected]>

address comments

6ea595f

SirOibaf requested changes Dec 18, 2023

View reviewed changes

davitbzh and others added 7 commits December 19, 2023 00:12

Update docs/user_guides/fs/feature_view/batch-data.md

b2a3aaa

Co-authored-by: Fabio Buso <[email protected]>

Update docs/user_guides/fs/feature_view/batch-data.md

d0e288e

Co-authored-by: Fabio Buso <[email protected]>

Update docs/user_guides/fs/feature_view/training-data.md

051d0ae

Co-authored-by: Fabio Buso <[email protected]>

Update docs/user_guides/fs/feature_view/training-data.md

ab3841d

Co-authored-by: Fabio Buso <[email protected]>

Update docs/user_guides/fs/feature_view/helper-columns.md

d6149b2

Co-authored-by: Fabio Buso <[email protected]>

address comments

67e6bbb

Merge branch 'ondemad_features_helper_columns' of https://github.com/…

aa96bd4

…davitbzh/logicalclocks.github.io into ondemad_features_helper_columns

davitbzh requested a review from SirOibaf December 18, 2023 23:33

SirOibaf approved these changes Dec 19, 2023

View reviewed changes

SirOibaf merged commit e88d96f into logicalclocks:main Dec 19, 2023
1 check passed

SirOibaf added a commit that referenced this pull request Dec 19, 2023

[FSTORE-1090] Concepts & Guides for helper columns and on-demand feat…

d9bf08e

…ures (#333) Co-authored-by: Fabio Buso <[email protected]> Co-authored-by: davitbzh <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FSTORE-1090] Concepts & Guides for helper columns and on-demand features #333

[FSTORE-1090] Concepts & Guides for helper columns and on-demand features #333

davitbzh commented Nov 24, 2023

SirOibaf left a comment

SirOibaf Dec 17, 2023

SirOibaf Dec 17, 2023

SirOibaf Dec 17, 2023

SirOibaf Dec 17, 2023

SirOibaf Dec 17, 2023

SirOibaf Dec 17, 2023

SirOibaf Dec 17, 2023

SirOibaf Dec 17, 2023

SirOibaf Dec 17, 2023

SirOibaf Dec 18, 2023

SirOibaf Dec 18, 2023

SirOibaf Dec 18, 2023 •

edited

Loading

		@@ -0,0 +1,6 @@
		On-demand is, a feature that is computed at request-time using application-supplied inputs for an online model.

		@@ -0,0 +1,6 @@
		On-demand is, a feature that is computed at request-time using application-supplied inputs for an online model.

		In the image below shows an example of a housing price model that demonstrates how to implement an on-demand feature, a zip code (or post code) that is computed using longitude/latitude parameters. In your online application, longitude and latitude are provided as parameters to the application, and the same python function used to calculate the zip code in the feature pipeline is used to compute the zip code in the Online Inference pipeline. This is achieved by implementing the on-demand features as a Python function in a Python module. Also ensure that the same version of the Python module is installed in both the feature and inference pipelines.

		@@ -0,0 +1,127 @@
		# Helper columns

		HSFS provides functionality to define two types of helper columns `inference_helper_columns` and `training_helper_columns` to [feature views](./overview.md).

[FSTORE-1090] Concepts & Guides for helper columns and on-demand features #333

[FSTORE-1090] Concepts & Guides for helper columns and on-demand features #333

Conversation

davitbzh commented Nov 24, 2023

SirOibaf left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SirOibaf Dec 18, 2023 • edited Loading

Choose a reason for hiding this comment

SirOibaf Dec 18, 2023 •

edited

Loading