The Go driver becomes "undefined: parser", and the application cannot be built.

[maehara-n]

Describe the bug
When trying to build an application that uses the Go driver, an error occurs. "undefined: parser" is output by the Go driver.

What data setup do we need to do?
What is the necessary configuration info needed?
Specify the following version of the Go driver in go.mod.

github.com/apache/age/drivers/golang v0.0.0-20231212232227-0ea118689d78

What is the command that caused the error?
Build the application that is using the Go driver.

#0 5.670 # github.com/apache/age/drivers/golang/age
#0 5.670 /go/pkg/mod/github.com/apache/age/drivers/[email protected]/age/builder.go:42:22: undefined: parser.AgeParser
#0 5.670 /go/pkg/mod/github.com/apache/age/drivers/[email protected]/age/builder.go:43:21: undefined: parser.AgeVisitor
#0 5.670 /go/pkg/mod/github.com/apache/age/drivers/[email protected]/age/builder.go:114:9: undefined: parser.AgeVisitor
#0 5.670 /go/pkg/mod/github.com/apache/age/drivers/[email protected]/age/builder.go:129:52: undefined: parser.AgeoutContext
#0 5.670 /go/pkg/mod/github.com/apache/age/drivers/[email protected]/age/builder.go:135:52: undefined: parser.VertexContext
#0 5.670 /go/pkg/mod/github.com/apache/age/drivers/[email protected]/age/builder.go:151:50: undefined: parser.EdgeContext
#0 5.670 /go/pkg/mod/github.com/apache/age/drivers/[email protected]/age/builder.go:164:50: undefined: parser.PathContext
#0 5.670 /go/pkg/mod/github.com/apache/age/drivers/[email protected]/age/builder.go:186:51: undefined: parser.ValueContext
#0 5.670 /go/pkg/mod/github.com/apache/age/drivers/[email protected]/age/builder.go:201:56: undefined: parser.PropertiesContext
#0 5.670 /go/pkg/mod/github.com/apache/age/drivers/[email protected]/age/builder.go:216:50: undefined: parser.PairContext
#0 5.670 /go/pkg/mod/github.com/apache/age/drivers/[email protected]/age/builder.go:216:50: too many errors

Expected behavior
The application can be built.

Environment (please complete the following information):

• golang 1.21.5

Additional context

[mjholub]

I think the reason behind this might be broken antlr dependencies, which are ridden with issues such as invalid function signatures and seem pretty low maintenance, so I'll try discussing ditching these with the project maintainers. I've tried using pgx as an alternative but it causes a memory leak while scanning 😞. @maehara-n, did you figure out any temporary solution?

[jrgemignani]

@maehara-n You need to build those files by executing generate.sh -

[jgemignani@cent7nuc7i5bnk age]$ ll ../parser/
total 100
-rw-rw-r--. 1 jgemignani jgemignani  2732 Jan 17 16:34 age_base_listener.go
-rw-rw-r--. 1 jgemignani jgemignani  1064 Jan 17 16:34 age_base_visitor.go
-rw-rw-r--. 1 jgemignani jgemignani  2193 Sep 20 17:44 Age.g4
-rw-rw-r--. 1 jgemignani jgemignani  2887 Jan 17 16:34 Age.interp
-rw-rw-r--. 1 jgemignani jgemignani 10646 Jan 17 16:34 age_lexer.go
-rw-rw-r--. 1 jgemignani jgemignani  7366 Jan 17 16:34 AgeLexer.interp
-rw-rw-r--. 1 jgemignani jgemignani   251 Jan 17 16:34 AgeLexer.tokens
-rw-rw-r--. 1 jgemignani jgemignani  1774 Jan 17 16:34 age_listener.go
-rw-rw-r--. 1 jgemignani jgemignani 37680 Jan 17 16:34 age_parser.go
-rw-rw-r--. 1 jgemignani jgemignani   251 Jan 17 16:34 Age.tokens
-rw-rw-r--. 1 jgemignani jgemignani  1082 Jan 17 16:34 age_visitor.go
-rw-rw-r--. 1 jgemignani jgemignani    44 Nov 28 16:50 generate.go
-rwxrwxr-x. 1 jgemignani jgemignani   400 Jan 17 16:02 generate.sh
[jgemignani@cent7nuc7i5bnk age]$

[jrgemignani]

Both the README and install.sh files aren't very good. If you look at the install file, you will basically need to -

1. Make sure that you have installed and selected the latest version of JAVA.
2. Make sure that you have installed the latest version of ANTLR and it is in the CLASSPATH.
3. You need to build the files in the parser directory using generate.sh. This is not in the install.sh file.
4. You will need to run that last command in install.sh for Age.g4 and give it the correct location parser/Age.g4
5. You will then need to modify the DSN information to reflect your database connect information.

After that you should be good to go.

[maehara-n]

@jrgemignani Prepare the JAVA environment, follow the steps to create the files, and place the files to eliminate the errors.
However, this cannot be called automated generation.
What troubles me is that I have to manually generate them.

[jrgemignani]

@maehara-n The JAVA environment is up to the user to provide as is the database connection information. The other stuff, yeah, if it is stated to be automatic or automated, it isn't. The individual who wrote the scripts didn't write them very well. We will need to fix them. But, this is an open source project and it can be hard to manage all of the community provided code.

As for having to manually generate files, that shouldn't be an issue for you, provided the documentation either states how to do it or there are scripts to do it automatically.

[maehara-n]

@jrgemignani The explanation you provided seems to be for setting up the development environment of Apache AGE.
The issue I am facing is the build failure in the system that uses Apache AGE.
There is no JAVA environment.

[maehara-n]

@mjholub We specify the following older version in go.mod.
github.com/apache/age/drivers/golang v0.0.0-20230125234102-2a233e3e2da9

[jrgemignani]

@maehara-n What I gave are directions for building the Go driver for AGE. Building it requires JAVA and ANTLR to be installed.

[maehara-n]

@jrgemignani You suggested that building is possible if the generated file is placed, but where exactly should it be placed? Can you please provide more specific information?

The following steps can reproduce the error when creating the application:

Create a directory named "myapp"
Execute the following command:
go mod init myapp
Place the files under the directory "https://github.com/apache/age/tree/master/drivers/golang/samples" into the "myapp" directory.
Run the following command:
go mod tidy
-> The following go.mod file is created:

module myapp

go 1.21.5

require (
	github.com/apache/age/drivers/golang v0.0.0-20240131070327-1e11acb19d14
	github.com/lib/pq v1.10.9
)

require (
	github.com/antlr/antlr4/runtime/Go/antlr/v4 v4.0.0-20230321174746-8dcc6526cfb1 // indirect
	golang.org/x/exp v0.0.0-20230321023759-10a507213a29 // indirect

)

Run the following command:
go build -o /out
-> An error will be outputted.

[jrgemignani]

@maehara-n Building requires -

• Java 11 or higher to be installed.
• ANTLR 4.11.1 or higher to be installed and included in the CLASSPATH
• Golang 1.18 or higher to be installed.

As I mentioned, the README.md and associated installer scripts aren't very good. I have updated the README.md and the install.sh script in PR #1527 to verify that the above conditions are met. However, I have removed the installation of Java, ANTLR, and Golang from install.sh script as this should be done by the system administrator i.e. it is not our responsibility. Currently, I am waiting for that PR to be merged.

Once the above criteria are met, the parser and lexer can be built in the driver/golang/age directory via the install.sh script. The current generate.sh script in that directory generally works, but also needs to be updated or removed.

[jrgemignani]

Update -

I need to clarify that the install.sh script will verify the minimum prerequisites listed above and then it will build and install the Go driver. The generate.sh script in the parser directory is for the Github CI scripts, so just ignore it.

All of the changes are now in the master branch.

[maehara-n]

@jrgemignani I have reviewed the update, but it will not solve the problem.
The solution you provided assumes working within the age project, which includes:

age
  drivers
    golang
      age
      parser
      samples  
      myapp
        age_wrapper_sample.go
        main.go
        sql_api_sample.go

However, my "myapp" project is structured as follows:

myapp
  age_wrapper_sample.go
  main.go
  sql_api_sample.go
  go.mod
  go.sum

The content of go.mod is:

module myapp

go 1.21.5

require (
	github.com/apache/age/drivers/golang v0.0.0-20240131070327-1e11acb19d14
	github.com/lib/pq v1.10.9
)

require (
	github.com/antlr/antlr4/runtime/Go/antlr/v4 v4.0.0-20230321174746-8dcc6526cfb1 // indirect
	golang.org/x/exp v0.0.0-20230321023759-10a507213a29 // indirect

)

There is no "install.sh" in myapp project and Java is not installed either.
The reason for the error with the go driver is that there is no file in the age repository.

[jrgemignani]

The reason for the error with the go driver is that there is no file in the age repository.

Which file(s) are you referring to? The ones in parser?

[maehara-n]

@jrgemignani The code for "myapp" is just importing and using "github.com/apache/age/drivers/golang/age".
The driver is referencing the parser.

"myapp" project is structured as follows:

myapp
  age_wrapper_sample.go
  main.go
  sql_api_sample.go
  go.mod
  go.sum

main.go

package main

import (
	"fmt"

	_ "github.com/lib/pq"
)

// var dsn string = "host={host} port={port} dbname={dbname} user={username} password={password} sslmode=disable"
var dsn string = "host=127.0.0.1 port=5432 dbname=postgres user=postgres password=agens sslmode=disable"

// var graphName string = "{graph_path}"
var graphName string = "testGraph"

func main() {

	// Do cypher query to AGE with database/sql Tx API transaction control
	fmt.Println("# Do cypher query with SQL API")
	doWithSqlAPI(dsn, graphName)

	// Do cypher query to AGE with Age API
	fmt.Println("# Do cypher query with Age API")
	doWithAgeWrapper(dsn, graphName)
}

sql_api_sample.go

package main

import (
	"database/sql"
	"fmt"

	"github.com/apache/age/drivers/golang/age"
)

// Do cypher query to AGE with database/sql Tx API transaction control
func doWithSqlAPI(dsn string, graphName string) {

	// Connect to PostgreSQL
	db, err := sql.Open("postgres", dsn)
	if err != nil {
		panic(err)
	}

	// Confirm graph_path created
	_, err = age.GetReady(db, graphName)
	if err != nil {
		panic(err)
	}

	// Tx begin for execute create vertex
	tx, err := db.Begin()
	if err != nil {
		panic(err)
	}

	// Create vertices with Cypher
	_, err = age.ExecCypher(tx, graphName, 0, "CREATE (n:Person {name: '%s', weight:%f})", "Joe", 67.3)
	if err != nil {
		panic(err)
	}

	_, err = age.ExecCypher(tx, graphName, 0, "CREATE (n:Person {name: '%s', weight:77.3, roles:['Dev','marketing']})", "Jack")
	if err != nil {
		panic(err)
	}

	_, err = age.ExecCypher(tx, graphName, 0, "CREATE (n:Person {name: '%s', weight:%d})", "Andy", 59)
	if err != nil {
		panic(err)
	}

	// Commit Tx
	tx.Commit()

	// Tx begin for queries
	tx, err = db.Begin()
	if err != nil {
		panic(err)
	}
	// Query cypher
	cypherCursor, err := age.ExecCypher(tx, graphName, 1, "MATCH (n:Person) RETURN n")
	if err != nil {
		panic(err)
	}
	// Unmarshal result data to Vertex row by row
	for cypherCursor.Next() {
		row, err := cypherCursor.GetRow()
		if err != nil {
			panic(err)
		}
		vertex := row[0].(*age.Vertex)
		fmt.Println(vertex.Id(), vertex.Label(), vertex.Props())
	}

	// Create Paths (Edges)
	_, err = age.ExecCypher(tx, graphName, 0, "MATCH (a:Person), (b:Person) WHERE a.name='%s' AND b.name='%s' CREATE (a)-[r:workWith {weight: %d}]->(b)", "Jack", "Joe", 3)
	if err != nil {
		panic(err)
	}

	_, err = age.ExecCypher(tx, graphName, 0, "MATCH (a:Person {name: '%s'}), (b:Person {name: '%s'}) CREATE (a)-[r:workWith {weight: %d}]->(b)", "Joe", "Andy", 7)
	if err != nil {
		panic(err)
	}

	tx.Commit()

	tx, err = db.Begin()
	if err != nil {
		panic(err)
	}
	// Query Paths with Cypher
	cypherCursor, err = age.ExecCypher(tx, graphName, 1, "MATCH p=()-[:workWith]-() RETURN p")
	if err != nil {
		panic(err)
	}

	for cypherCursor.Next() {
		row, err := cypherCursor.GetRow()
		if err != nil {
			panic(err)
		}

		path := row[0].(*age.Path)
		vertexStart := path.GetAsVertex(0)
		edge := path.GetAsEdge(1)
		vertexEnd := path.GetAsVertex(2)

		fmt.Println(vertexStart, edge, vertexEnd)
	}

	// Query with return many columns
	cursor, err := age.ExecCypher(tx, graphName, 3, "MATCH (a:Person)-[l:workWith]-(b:Person) RETURN a, l, b")
	if err != nil {
		panic(err)
	}

	count := 0
	for cursor.Next() {
		row, err := cursor.GetRow()
		if err != nil {
			panic(err)
		}
		count++
		v1 := row[0].(*age.Vertex)
		edge := row[1].(*age.Edge)
		v2 := row[2].(*age.Vertex)
		fmt.Println("ROW ", count, ">>", "\n\t", v1, "\n\t", edge, "\n\t", v2)
	}

	// Delete Vertices
	_, err = age.ExecCypher(tx, graphName, 0, "MATCH (n:Person) DETACH DELETE n RETURN *")
	if err != nil {
		panic(err)
	}
	tx.Commit()
}

age_wrapper_sample.go

package main

import (
	"fmt"

	"github.com/apache/age/drivers/golang/age"
)

// Do cypher query to AGE with Age API
func doWithAgeWrapper(dsn string, graphName string) {

	ag, err := age.ConnectAge(graphName, dsn)

	if err != nil {
		panic(err)
	}

	tx, err := ag.Begin()
	if err != nil {
		panic(err)
	}

	_, err = tx.ExecCypher(0, "CREATE (n:Person {name: '%s'})", "Joe")
	if err != nil {
		panic(err)
	}

	_, err = tx.ExecCypher(0, "CREATE (n:Person {name: '%s', age: %d})", "Smith", 10)
	if err != nil {
		panic(err)
	}

	_, err = tx.ExecCypher(0, "CREATE (n:Person {name: '%s', weight:%f})", "Jack", 70.3)
	if err != nil {
		panic(err)
	}

	tx.Commit()

	tx, err = ag.Begin()
	if err != nil {
		panic(err)
	}

	cursor, err := tx.ExecCypher(1, "MATCH (n:Person) RETURN n")
	if err != nil {
		panic(err)
	}

	count := 0
	for cursor.Next() {
		entities, err := cursor.GetRow()
		if err != nil {
			panic(err)
		}
		count++
		vertex := entities[0].(*age.Vertex)
		fmt.Println(count, "]", vertex.Id(), vertex.Label(), vertex.Props())
	}

	fmt.Println("Vertex Count:", count)

	_, err = tx.ExecCypher(0, "MATCH (a:Person), (b:Person) WHERE a.name='%s' AND b.name='%s' CREATE (a)-[r:workWith {weight: %d}]->(b)",
		"Jack", "Joe", 3)
	if err != nil {
		panic(err)
	}

	_, err = tx.ExecCypher(0, "MATCH (a:Person {name: '%s'}), (b:Person {name: '%s'}) CREATE (a)-[r:workWith {weight: %d}]->(b)",
		"Joe", "Smith", 7)
	if err != nil {
		panic(err)
	}

	tx.Commit()

	tx, err = ag.Begin()
	if err != nil {
		panic(err)
	}

	cursor, err = tx.ExecCypher(1, "MATCH p=()-[:workWith]-() RETURN p")
	if err != nil {
		panic(err)
	}

	count = 0
	for cursor.Next() {
		entities, err := cursor.GetRow()
		if err != nil {
			panic(err)
		}
		count++

		path := entities[0].(*age.Path)

		vertexStart := path.GetAsVertex(0)
		edge := path.GetAsEdge(1)
		vertexEnd := path.GetAsVertex(2)

		fmt.Println(count, "]", vertexStart, edge.Props(), vertexEnd)
	}

	// Query with return many columns
	cursor, err = tx.ExecCypher(3, "MATCH (a:Person)-[l:workWith]-(b:Person) RETURN a, l, b")
	if err != nil {
		panic(err)
	}

	count = 0
	for cursor.Next() {
		row, err := cursor.GetRow()
		if err != nil {
			panic(err)
		}

		count++

		v1 := row[0].(*age.Vertex)
		edge := row[1].(*age.Edge)
		v2 := row[2].(*age.Vertex)

		fmt.Println("ROW ", count, ">>", "\n\t", v1, "\n\t", edge, "\n\t", v2)
	}

	_, err = tx.ExecCypher(0, "MATCH (n:Person) DETACH DELETE n RETURN *")
	if err != nil {
		panic(err)
	}
	tx.Commit()
}

If I describe go.mod as follows, the build will fail.

module myapp

go 1.21.5

require (
	github.com/apache/age/drivers/golang v0.0.0-20240131202942-03a22b5269ec
	github.com/lib/pq v1.10.9
)

require (
	github.com/antlr/antlr4/runtime/Go/antlr/v4 v4.0.0-20230321174746-8dcc6526cfb1 // indirect
	golang.org/x/exp v0.0.0-20240119083558-1b970713d09a // indirect
)

If I describe go.mod as follows, the build will succeed.

module myapp

go 1.21.5

require (
	github.com/apache/age/drivers/golang v0.0.0-20230125234102-2a233e3e2da9
	github.com/lib/pq v1.10.9
)

require github.com/antlr/antlr4/runtime/Go/antlr v0.0.0-20210521184019-c5ad59b459ec // indirect

[jrgemignani]

@maehara-n If you can get it to work with your app, then I'm not sure what the issue is. The most that I can do is to add, remove, or update files, if necessary. I am only able to test the installation with the included test app which works on my end.

[maehara-n]

@jrgemignani The problem is not resolved. Have you checked the version of the go driver I wrote?

github.com/apache/age/drivers/golang v0.0.0-20230125234102-2a233e3e2da9
This is a version of the go driver that builds successfully.
It's a version that was released before the following fix was made:
#714

After the fix mentioned in pull request 714 was made, the go driver no longer builds successfully.
This is because the files required for building were removed from the repository.

A pull request was created to address this issue, but it has been left unattended:
#1205

To make the build successful, one of the following actions needs to be taken:

• Undo the fix in pull request 714, which is incorrect for go lang.

• Approve pull request 1205.

[jrgemignani]

@maehara-n I feel like we're using the driver two different ways and that is causing some confusion -

You are using it as, what sounds like, a prebuilt package. Whereas, I'm using it and building it (as are the Github CI scripts), as necessary. Those components are automatically built for me, but apparently not for you.

I can't revert #714 as it has security improvements. I can't merge #1205 as there have been updates to the driver since that PR. Additionally, @mjholub submitted a PR #1504 to make changes.

Unfortunately, I'm not an expert, or even a novice, at Golang so bare with me. If necessary, I will just update the whole driver and add those files back in.

[jrgemignani]

I will also add PR #1203 to the mix as needing review.

[jrgemignani]

@maehara-n I have created PR #1582 to add in those files. Please check that PR out locally to see if it corrects your issue.

Please understand that it still needs to be reviewed and approved by others on the team. There is no guarantee that they will necessarily agree with providing files that can be built by the user. By verifying that this PR corrects the issue for you, it may help with consideration.

[maehara-n]

@jrgemignani I have looked at PR #1582 . It seems like it might solve this issue.

[jrgemignani]

@maehara-n Could you verify this by pulling down the PR and testing it? I would rather not merge a PR that doesn't resolve an issue. Might, isn't compelling enough especially since everything on my side works without this PR.

[sanket-uptycs]

After fixing dir permissions for go pkg
install.sh / generate.sh scripts are working fine.
I am using go get to install driver. but facing this issue any idea how to fix this?

[jrgemignani]

@sanket-uptycs This looks to be the same error from above. Could you try either building those parser files or testing out the PR linked above, which may resolve it?

[sanket-uptycs]

@jrgemignani I tried building files using generate.sh and was able to generate files successfully.
But even after generating the files, I can see the errors in the builder.go and mapper.go
not declared by package parser. (I am using go get)
I tried replacing age/drivers/golang => jrgemignani/age/drivers/golang v0.0.0-20240206212258-1ac037aca6c8

on which #1582 is targeted.
I was able to generate files successfully in the parser in
go/pkg/mod/github.com/jrgemignani/age/drivers/[email protected]
but somehow new files in the package parser are not getting detected by builder.go and mapper.go

Also if I build drivers from the source no errors at all on the master branch.

But if we build module from the source and import it then it works fine.

Could be a configuration issue?

[maehara-n]

@sanket-uptycs I think there may be a mismatch in the specified version. You need to specify the following version.
replace github.com/apache/age/drivers/golang => github.com/jrgemignani/age/drivers/golang v0.0.0-20240209012620-7c153e1a9fb2

@jrgemignani I was able to successfully build the application by specifying the go.mod as follows.

module myapp

go 1.21.5

replace github.com/apache/age/drivers/golang => github.com/jrgemignani/age/drivers/golang v0.0.0-20240209012620-7c153e1a9fb2

require (
	github.com/apache/age/drivers/golang v0.0.0-00010101000000-000000000000
	github.com/lib/pq v1.10.9
)

require (
	github.com/antlr/antlr4/runtime/Go/antlr/v4 v4.0.0-20230321174746-8dcc6526cfb1 // indirect
	golang.org/x/exp v0.0.0-20240205201215-2c58cdc269a3 // indirect
)

[maehara-n]

@jrgemignani The issue will be resolved by #1582.

[sanket-uptycs]

@jrgemignani @maehara-n My bad, missed to specify the version.
#1582 looks good.
Thanks.

[jrgemignani]

I will have a team member look at merging #1582

[jrgemignani]

@sanket-uptycs @maehara-n The master branch has been updated. Please verify that this resolves the issue, and if so, close the ticket :)

[sanket-uptycs]

Verified on the latest master branch. we can close the issue.

[jrgemignani]

@sanket-uptycs @maehara-n Once this is closed, I will push create PRs for the other branches.

[maehara-n]

@jrgemignani Thank you for your support.

[sanket-uptycs]

@jrgemignani Thanks man !