Process positional arguments #134

tatianacv · 2023-01-09T14:35:03Z

Fixes #108.

Commits

I slowed down and took a deep breadth before committing. Otherwise, I will start a pristine branch and reissue this pull request.
Before committing my changes, I checked my working directory carefully. I only staged the changes I wanted and ensured it was of the utmost quality.
I used git diff and git status to check my work carefully before committing.
I completely understand what I am committing and why.
I completely understand why things are the way they are.
The change I am proposing makes complete sense to me.
For the most part, the changes in this pull request correspond to the problem or task I am solving or performing. In other words, only the changes that are supposed to be there are the ones included in this pull request.

Expectations

We have added code to be able to parse through the positional arguments. In this PR, there is restructuring of the method getDeclaringModuleName() in Util.java from the analysis plugin. There is a new method getDeclaringDefinitionName() using some extracted parts of the getDeclaringModuleName() to be able to access the definition of the decorator so we can get the correct ordering of the decorator’s arguments for the positionals. Then, in the Function->HybridizationParameters class, we added code to get the tf.function definition for the user's TF version. Then, depending on the position we were parsing, we would link the definition to the argument getting to get the resulting tf.function parameter.

Testing

If this is a bug fix, I was able to reproduce the problem locally before I made any changes (can use git stash to temporarily reset the working directory).
When I made (or put back) my changes, the problem I reproduced above was fixed.

Contain tests for each tf.function parameter, test for a mix of positional and keyword, test for default values (to know we are counting them as existing), test for no parameters.

Final Checks

The changes I am proposing meet the expectations I have described above.
I have performed a self-review of my code.
I have commented my code, particularly in hard-to-understand areas.
I have made corresponding changes to the documentation if applicable.
I have added tests that prove my fix is effective or that my feature works if applicable.
New and existing unit tests pass locally with my changes.

edu.cuny.hunter.hybridize.tests/resources/HybridizeFunction/testPositionalParameters/in/A.py

khatchad · 2023-01-09T14:59:01Z

...idize.tests/test cases/edu/cuny/hunter/hybridize/tests/HybridizeFunctionRefactoringTest.java

+	 * Test for #106. This tests whether we can parse one tf.function positional argument.
+	 */
+	@Test
+	public void testPositionalParameters() throws Exception {


We need to know what we are doing with the default values. Why are those special?

Added a comment explaining this in the test code.

What test code?

In testPositionalParameters12

Why in testPositionalParameters12?

Because that test handles the default values.

...idize.tests/test cases/edu/cuny/hunter/hybridize/tests/HybridizeFunctionRefactoringTest.java

edu.cuny.hunter.hybridize.tests/resources/HybridizeFunction/testPositionalParameters11/in/A.py

...idize.tests/test cases/edu/cuny/hunter/hybridize/tests/HybridizeFunctionRefactoringTest.java

…nctions-Refactoring into issue_108

edu.cuny.hunter.hybridize.core/src/edu/cuny/hunter/hybridize/core/analysis/Function.java

khatchad · 2023-03-08T20:42:42Z

edu.cuny.hunter.hybridize.core/src/edu/cuny/hunter/hybridize/core/analysis/Function.java

+						Set<Definition> potentialDeclaringDefitinion = Util.getDeclaringDefinition(selection,
+								Function.this.containingModuleName, Function.this.containingFile, Function.this.nature, monitor);
+						if (potentialDeclaringDefitinion.size() == 1)
+							declaringDefinition = potentialDeclaringDefitinion.iterator().next();


Why is there not an else case? I'm not seeing the logic here. Can be greatly helped with some comments in the code of the case analysis.

I added comments for this conditional.

I don't know what variable would be null and why. Rewrite the logic to handle the else case here.

I have rewritten so we handle the else case here

khatchad · 2023-03-09T15:03:24Z

edu.cuny.hunter.hybridize.core/src/edu/cuny/hunter/hybridize/core/analysis/Util.java

 	/**
-	 * Get the name of the module defining the entity described in the given {@link PySelection}.
+	 * Get the set of potential declaring definitions of the entity described in the given {@link PySelection}.


I don't understand. How can there be multiple definitions of the same thing?

khatchad

Please do not resolve the comments I make. I will resolve them when I review them. Thanks.

khatchad · 2023-03-16T18:13:01Z

edu.cuny.hunter.hybridize.core/src/edu/cuny/hunter/hybridize/core/analysis/Function.java

+						Set<Definition> potentialDeclaringDefitinion = Util.getDeclaringDefinition(selection,
+								Function.this.containingModuleName, Function.this.containingFile, Function.this.nature, monitor);
+						if (potentialDeclaringDefitinion.size() == 1)
+							declaringDefinition = potentialDeclaringDefitinion.iterator().next();


I don't know what variable would be null and why. Rewrite the logic to handle the else case here.

khatchad · 2023-03-16T18:25:52Z

edu.cuny.hunter.hybridize.core/src/edu/cuny/hunter/hybridize/core/analysis/Function.java

+					// We iterate over the tf.function's parameters positions.
+					for (int i = 0; i < callFunction.args.length; i++) {
+						// From the position i, we use the tf.function's definition to verify which parameter we are analyzing.
+						String evaluatedArgument = argumentIdDeclaringDefintion.get(i);


Not sure what happened to my comment here, but my question about why a for each loop is not being used here still pertains. Note that this is a question.

Looks like it was on outdated code. But, either way, it popped up here again.

We don't want to for-each over the values of the parsed tf.function arguments (callFunction.args: exprType[]) because those expressions are the values of the arguments since we are evaluating positional arguments. Additionally, we also don't want to for-each over the values of the tf.function definition (argumentIdDeclaringDefintion :ArrayList<String>) because there we have all keywords of tf.function arguments.

We want to see which arguments are being used for the parsed tf.function and match them with the position of the tf.function decorator argument, so we would need the index value. In this code, we get how many arguments the decorator we are parsing has and then tie those positions to the positions in the tf.function definition.

I see the problem now. You are indexing argumentIdDeclaringDefintion using callFunction.args.length as a length instead of argumentIdDeclaringDefintion.size(). But, nowhere do I see a comparison of callFunction.args.length with argumentIdDeclaringDefintion.size(). That's very dangerous if callFunction.args.length > argumentIdDeclaringDefintion.size().

For it to be valid Python code, you cannot have this case callFunction.args.length > argumentIdDeclaringDefintion.size(). You would receive TypeError: function() takes from 0 to 10 positional arguments but 11 were given (If 11 were given)

I added a check for this.

edu.cuny.hunter.hybridize.core/src/edu/cuny/hunter/hybridize/core/analysis/Function.java

khatchad · 2023-03-16T18:28:16Z

edu.cuny.hunter.hybridize.core/src/edu/cuny/hunter/hybridize/core/analysis/Util.java

 	/**
-	 * Get the name of the module defining the entity described in the given {@link PySelection}.
+	 * Get the set of potential declaring definitions of the entity described in the given {@link PySelection}.


Sorry, I am still not convinced. To put it another way, why was it working before but now it is not working specifically for this feature?

khatchad · 2023-03-16T18:32:52Z

...idize.tests/test cases/edu/cuny/hunter/hybridize/tests/HybridizeFunctionRefactoringTest.java

+	 * Test for #106. This tests whether we can parse one tf.function positional argument.
+	 */
+	@Test
+	public void testPositionalParameters() throws Exception {


What test code?

khatchad · 2023-03-16T18:33:45Z

...idize.tests/test cases/edu/cuny/hunter/hybridize/tests/HybridizeFunctionRefactoringTest.java

+		Function function = functions.iterator().next();
+		assertNotNull(function);
+
+		Function.HybridizationParameters args = function.getHybridizationParameters();


Not sure how this is resolved if it's blocked.

…nctions-Refactoring into issue_108

khatchad · 2023-03-21T00:10:29Z

edu.cuny.hunter.hybridize.core/src/edu/cuny/hunter/hybridize/core/analysis/Function.java

@@ -108,6 +142,10 @@ public HybridizationParameters(IProgressMonitor monitor) throws BadLocationExcep
 			// Will contain the last tf.function decorator
 			decoratorsType tfFunctionDecorator = null;

+			// Declaring definitions of the decorator, if it contains multiple definitions there might be more than one in this set. Since
+			// we are dealing with tf.function, we expect only one.


We can't "expect." We must check.

I modified the comment.

khatchad · 2023-03-21T00:23:15Z

edu.cuny.hunter.hybridize.core/src/edu/cuny/hunter/hybridize/core/analysis/Function.java

+						if (potentialDeclaringDefitinion.iterator().hasNext())
+							declaringDefinition = potentialDeclaringDefitinion.iterator().next();
+						else
+							throw new IllegalStateException("Can't determine tf.function decorator defintion.");


What is the point of changing existing code from returning a non scalar value to a scalar value and then only using one of them anyway?

Also, the error message here isn't very informative.

I added more information about which param couldn't be determined.

khatchad · 2023-03-21T00:24:37Z

edu.cuny.hunter.hybridize.core/src/edu/cuny/hunter/hybridize/core/analysis/Function.java

+
+						// Matching the arguments from the definition and the arguments from the code being analyzed.
+						if (evaluatedArgument.equals(FUNC))
+							// Found parameter func


The comment here is not necessary. It's obvious what is being "found."

I removed it.

khatchad · 2023-03-21T00:24:52Z

edu.cuny.hunter.hybridize.core/src/edu/cuny/hunter/hybridize/core/analysis/Function.java

+							// Found parameter experimental_follow_type_hints
+							this.experimentaFollowTypeHintsParamExists = true;
+						else
+							throw new IllegalArgumentException("Unable to process tf.function argument.");


Added more information.

khatchad · 2023-03-21T00:25:40Z

edu.cuny.hunter.hybridize.core/src/edu/cuny/hunter/hybridize/core/analysis/Function.java

@@ -164,11 +256,48 @@ else if (name.id.equals(EXPERIMENTAL_AUTOGRAPH_OPTIONS))
 							else if (name.id.equals(EXPERIMENTAL_FOLLOW_TYPE_HINTS))
 								// Found parameter experimental_follow_type_hints
 								this.experimentaFollowTypeHintsParamExists = true;
+							else
+								throw new IllegalArgumentException("Unable to process tf.function argument.");


In a PR for positional arguments, why are there changes to keyword arguments?

khatchad · 2023-03-21T00:27:00Z

edu.cuny.hunter.hybridize.core/src/edu/cuny/hunter/hybridize/core/analysis/Function.java

+		 * @param declaringDefinition The Definition to use.
+		 * @return An array with the names of the arguments given by {@link Definition}.
+		 */
+		private ArrayList<String> getTfFunctionPythonDefinitionArguments(Definition declaringDefinition) {


This method doesn't seem very specific to tf.function at all. Isn't it more general?

I modified the name of the method and the comments so that it is more general. Thanks.

khatchad

Many things are unclear and puzzling.

khatchad

Potentially dangerous scalar indexing.

khatchad · 2023-03-21T18:51:01Z

edu.cuny.hunter.hybridize.core/src/edu/cuny/hunter/hybridize/core/analysis/Function.java

+					// We iterate over the tf.function's parameters positions.
+					for (int i = 0; i < callFunction.args.length; i++) {
+						// From the position i, we use the tf.function's definition to verify which parameter we are analyzing.
+						String evaluatedArgument = argumentIdDeclaringDefintion.get(i);


I see the problem now. You are indexing argumentIdDeclaringDefintion using callFunction.args.length as a length instead of argumentIdDeclaringDefintion.size(). But, nowhere do I see a comparison of callFunction.args.length with argumentIdDeclaringDefintion.size(). That's very dangerous if callFunction.args.length > argumentIdDeclaringDefintion.size().

…nctions-Refactoring into issue_108

…nition, more general method name, more information on exception messages

khatchad · 2023-03-28T18:45:02Z

edu.cuny.hunter.hybridize.core/src/edu/cuny/hunter/hybridize/core/analysis/Function.java

@@ -143,7 +143,7 @@ public HybridizationParameters(IProgressMonitor monitor) throws BadLocationExcep
 			decoratorsType tfFunctionDecorator = null;

 			// Declaring definitions of the decorator, if it contains multiple definitions there might be more than one in this set. Since
-			// we are dealing with tf.function, we expect only one.
+			// we are dealing with tf.function, we check we it has one definition.


It's not only the comment. The checking must be in the code.

There is a check in the code.

khatchad · 2023-03-28T18:46:12Z

edu.cuny.hunter.hybridize.core/src/edu/cuny/hunter/hybridize/core/analysis/Function.java

@@ -164,61 +164,62 @@ public HybridizationParameters(IProgressMonitor monitor) throws BadLocationExcep
 						if (potentialDeclaringDefitinion.iterator().hasNext())
 							declaringDefinition = potentialDeclaringDefitinion.iterator().next();
 						else
-							throw new IllegalStateException("Can't determine tf.function decorator defintion.");
+							throw new IllegalStateException(String.format(
+									"Can't find declaring definition for selection: %s in line: %s, file: %s, and project: %s.",


This error message is vague and doesn't resemble the one that was modified.

khatchad · 2023-03-28T18:46:37Z

edu.cuny.hunter.hybridize.core/src/edu/cuny/hunter/hybridize/core/analysis/Function.java

 					}
 				} catch (AmbiguousDeclaringModuleException e) {
 					throw new IllegalStateException("Can't determine whether decorator: " + decorator + " is hybrid.", e);
 				}
 			} // We expect to have the last tf.function decorator in tfFunctionDecorator

 			// Getting tf.functions Python definition arguments.
-			ArrayList<String> argumentIdDeclaringDefintion = getTfFunctionPythonDefinitionArguments(declaringDefinition);
+			ArrayList<String> argumentIdDeclaringDefintion = getPythonDefinitionArguments(declaringDefinition);


What is a "Python definition argument?"

Getting the argument from the Python definition

edu.cuny.hunter.hybridize.core/src/edu/cuny/hunter/hybridize/core/analysis/Function.java

khatchad · 2023-03-28T18:47:48Z

edu.cuny.hunter.hybridize.core/src/edu/cuny/hunter/hybridize/core/analysis/Function.java

@@ -256,20 +257,18 @@ else if (name.id.equals(EXPERIMENTAL_AUTOGRAPH_OPTIONS))
 							else if (name.id.equals(EXPERIMENTAL_FOLLOW_TYPE_HINTS))
 								// Found parameter experimental_follow_type_hints
 								this.experimentaFollowTypeHintsParamExists = true;
-							else
-								throw new IllegalArgumentException("Unable to process tf.function argument.");


Why was this removed? Now, we are missing the else case here.

Because this is for keyword arguments

tatianacv added 23 commits December 11, 2022 22:58

parsing positional arguments for tf.function decorator

df69fd5

fixing comments

3b881a6

adding tests

2899de3

fix trailing whitespace

731b664

fix

a4cf2cd

not counting the default values as existing

d6b16ed

fix, might need to tidy up

aeca489

fix

61f1b1b

fetching from upstream

e8b508c

Progress

109e0e4

fix trailing whitespace

b7d9d5c

Throwing exceptions

6dbbadd

adding tests for another tf version

e5c5b80

changing test's tf.function version and some formatting

1046cff

fixing python code

8c489aa

fixing to a non-default value

130f807

update

55cadea

adding requirement

c83eccf

update

4361e2e

fix checkstyle

53c44f1

update

23ae949

fixing comments

94a005f

Merge branch 'ponder-lab:main' into issue_108

528a3a7

khatchad reviewed Jan 9, 2023

View reviewed changes

edu.cuny.hunter.hybridize.tests/resources/HybridizeFunction/testPositionalParameters/in/A.py Outdated Show resolved Hide resolved

khatchad reviewed Jan 9, 2023

View reviewed changes

...idize.tests/test cases/edu/cuny/hunter/hybridize/tests/HybridizeFunctionRefactoringTest.java Show resolved Hide resolved

khatchad reviewed Jan 9, 2023

View reviewed changes

edu.cuny.hunter.hybridize.tests/resources/HybridizeFunction/testPositionalParameters11/in/A.py Outdated Show resolved Hide resolved

khatchad reviewed Jan 9, 2023

View reviewed changes

...idize.tests/test cases/edu/cuny/hunter/hybridize/tests/HybridizeFunctionRefactoringTest.java Show resolved Hide resolved

tatianacv added 2 commits January 9, 2023 17:13

progress on comments

a3cfae6

Merge branch 'issue_108' of https://github.com/tatianacv/Hybridize-Fu…

92cf918

…nctions-Refactoring into issue_108

Updates on requested changes

c182a37

khatchad disabled auto-merge March 8, 2023 20:40

khatchad requested changes Mar 9, 2023

View reviewed changes

tatianacv and others added 5 commits March 10, 2023 15:10

update

d3239b4

update

9a60cbf

Renaming for clarity

1cb7502

update

f710611

Merge branch 'main' into issue_108

dacd17a

khatchad requested changes Mar 16, 2023

View reviewed changes

tatianacv and others added 3 commits March 17, 2023 13:00

update

5673690

Merge branch 'issue_108' of https://github.com/tatianacv/Hybridize-Fu…

474acdf

…nctions-Refactoring into issue_108

Merge branch 'main' into issue_108

ccce776

khatchad reviewed Mar 21, 2023

View reviewed changes

khatchad requested changes Mar 21, 2023

View reviewed changes

tatianacv and others added 7 commits March 21, 2023 15:10

Merge branch 'main' into issue_108

4e16587

Changing comments, removing keyword args else

5d72476

Merge branch 'issue_108' of https://github.com/tatianacv/Hybridize-Fu…

b4358fd

…nctions-Refactoring into issue_108

Checking that the size of the parse tffunction args is less than defi…

54ba275

…nition, more general method name, more information on exception messages

Adding more info

afd01ff

Merge branch 'main' into issue_108

1ca2997

Merge branch 'main' into issue_108

afdbbb0

khatchad reviewed Mar 28, 2023

View reviewed changes

khatchad requested changes Mar 28, 2023

View reviewed changes

Merge branch 'main' into issue_108

4161ad7

Process positional arguments #134

Are you sure you want to change the base?

Process positional arguments #134

Conversation

tatianacv commented Jan 9, 2023 • edited Loading

Commits

Expectations

Testing

Final Checks

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

khatchad left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

khatchad Mar 21, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

khatchad left a comment • edited Loading

Choose a reason for hiding this comment

khatchad left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tatianacv commented Jan 9, 2023 •

edited

Loading

khatchad Mar 21, 2023 •

edited

Loading

khatchad left a comment •

edited

Loading