Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

auto detect latest ARDC versions #170

Merged
merged 8 commits into from
Dec 10, 2024
Merged

auto detect latest ARDC versions #170

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
73db9ff
auto detect latest ARDC versions
vietnguyengit Dec 2, 2024
0cee6c6
Update tests
vietnguyengit Dec 2, 2024
cbcf9c1
Add tests
vietnguyengit Dec 2, 2024
ea2f214
refactor
vietnguyengit Dec 2, 2024
eabab6d
Refactor and update tests
vietnguyengit Dec 3, 2024
d3a135c
Only update vocabs if there are version changes
vietnguyengit Dec 3, 2024
137366b
Extract specific version from current
vietnguyengit Dec 9, 2024
b54a8a7
minor update
vietnguyengit Dec 9, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 0 additions & 37 deletions ardcvocabs/src/main/java/au/org/aodn/ardcvocabs/configuration/VocabApiPaths.java
View file
Open in desktop

This file was deleted.

29 changes: 29 additions & 0 deletions ardcvocabs/src/main/java/au/org/aodn/ardcvocabs/model/ArdcCurrentPaths.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
package au.org.aodn.ardcvocabs.model;

import lombok.Getter;

@Getter
public enum ArdcCurrentPaths {
PARAMETER_VOCAB(
"/aodn-parameter-category-vocabulary/current/concept.json",
"/aodn-discovery-parameter-vocabulary/current/concept.json"
),
PLATFORM_VOCAB(
"/aodn-platform-category-vocabulary/current/concept.json",
"/aodn-platform-vocabulary/current/concept.json"
),
ORGANISATION_VOCAB(
"/aodn-organisation-category-vocabulary/current/concept.json",
"/aodn-organisation-vocabulary/current/concept.json"
);


private final String categoryCurrent;
private final String vocabCurrent;

ArdcCurrentPaths(String categoryCurrent, String vocabCurrent) {
String baseUrl = "https://vocabs.ardc.edu.au/repository/api/lda/aodn";
this.categoryCurrent = baseUrl + categoryCurrent;
this.vocabCurrent = baseUrl + vocabCurrent;
}
}
8 changes: 8 additions & 0 deletions ardcvocabs/src/main/java/au/org/aodn/ardcvocabs/model/PathName.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
package au.org.aodn.ardcvocabs.model;

public enum PathName {
categoryApi,
categoryDetailsApi,
vocabApi,
vocabDetailsApi
}
43 changes: 21 additions & 22 deletions ardcvocabs/src/main/java/au/org/aodn/ardcvocabs/model/VocabApiPaths.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,37 +1,36 @@
package au.org.aodn.ardcvocabs.model;

import lombok.Getter;

@Getter
public enum VocabApiPaths {
PARAMETER_VOCAB(
"/aodn-parameter-category-vocabulary/version-2-1/concept.json",
"/aodn-parameter-category-vocabulary/version-2-1/resource.json?uri=%s",
"/aodn-discovery-parameter-vocabulary/version-1-6/concept.json",
"/aodn-discovery-parameter-vocabulary/version-1-6/resource.json?uri=%s"
"/aodn-parameter-category-vocabulary/%s/concept.json",
"/aodn-parameter-category-vocabulary/%s/resource.json?uri=%s",
"/aodn-discovery-parameter-vocabulary/%s/concept.json",
"/aodn-discovery-parameter-vocabulary/%s/resource.json?uri=%s"
Comment on lines +7 to +10
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

although I could replace version directly with current, writing tests is a bit headache, after debugging I found that links like this for example:

https://vocabs.ardc.edu.au/repository/api/lda/aodn/aodn-platform-vocabulary/current/concept.json
or
https://vocabs.ardc.edu.au/repository/api/lda/aodn/aodn-platform-vocabulary/current/concept.json?_page=1

the next and prev fields refer to paths with specific version instead of current.

So less complex tests and later on ... check if there are versions change to update caches, I keep this template enum, and have separate method for extracting latest specific versions from current, current is just a starting point wrapper.

),
PLATFORM_VOCAB(
"/aodn-platform-category-vocabulary/version-1-2/concept.json",
"/aodn-platform-category-vocabulary/version-1-2/resource.json?uri=%s",
"/aodn-platform-vocabulary/version-6-1/concept.json",
"/aodn-platform-vocabulary/version-6-1/resource.json?uri=%s"
"/aodn-platform-category-vocabulary/%s/concept.json",
"/aodn-platform-category-vocabulary/%s/resource.json?uri=%s",
"/aodn-platform-vocabulary/%s/concept.json",
"/aodn-platform-vocabulary/%s/resource.json?uri=%s"
),
ORGANISATION_VOCAB(
"/aodn-organisation-category-vocabulary/version-2-5/concept.json",
"/aodn-organisation-category-vocabulary/version-2-5/resource.json?uri=%s",
"/aodn-organisation-vocabulary/version-2-5/concept.json",
"/aodn-organisation-vocabulary/version-2-5/resource.json?uri=%s"
"/aodn-organisation-category-vocabulary/%s/concept.json",
"/aodn-organisation-category-vocabulary/%s/resource.json?uri=%s",
"/aodn-organisation-vocabulary/%s/concept.json",
"/aodn-organisation-vocabulary/%s/resource.json?uri=%s"
);

private final String vocabCategoryApiPath;
private final String vocabCategoryDetailsApiPath;
private final String vocabApiPath;
private final String vocabDetailsApiPath;
private final String categoryApiTemplate;
private final String categoryDetailsTemplate;
private final String vocabApiTemplate;
private final String vocabDetailsTemplate;

VocabApiPaths(String vocabCategoryApiPath, String vocabCategoryDetailsApiPath, String vocabApiPath, String vocabDetailsApiPath) {
this.vocabCategoryApiPath = vocabCategoryApiPath;
this.vocabCategoryDetailsApiPath = vocabCategoryDetailsApiPath;
this.vocabApiPath = vocabApiPath;
this.vocabDetailsApiPath = vocabDetailsApiPath;
VocabApiPaths(String categoryApiTemplate, String categoryDetailsTemplate, String vocabApiTemplate, String vocabDetailsTemplate) {
this.categoryApiTemplate = categoryApiTemplate;
this.categoryDetailsTemplate = categoryDetailsTemplate;
this.vocabApiTemplate = vocabApiTemplate;
this.vocabDetailsTemplate = vocabDetailsTemplate;
}
}
5 changes: 4 additions & 1 deletion ardcvocabs/src/main/java/au/org/aodn/ardcvocabs/service/ArdcVocabService.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,13 @@
package au.org.aodn.ardcvocabs.service;

import au.org.aodn.ardcvocabs.model.PathName;
import au.org.aodn.ardcvocabs.model.VocabApiPaths;
import au.org.aodn.ardcvocabs.model.VocabModel;

import java.util.List;
import java.util.Map;

public interface ArdcVocabService {
List<VocabModel> getVocabTreeFromArdcByType(VocabApiPaths vocabApiPaths);
Map<String, Map<PathName, String>> getResolvedPathCollection();
List<VocabModel> getVocabTreeFromArdcByType(Map<PathName, String> resolvedPaths);
}
114 changes: 98 additions & 16 deletions ardcvocabs/src/main/java/au/org/aodn/ardcvocabs/service/ArdcVocabServiceImpl.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
package au.org.aodn.ardcvocabs.service;

import au.org.aodn.ardcvocabs.model.ArdcCurrentPaths;
import au.org.aodn.ardcvocabs.model.PathName;
import au.org.aodn.ardcvocabs.model.VocabApiPaths;
import au.org.aodn.ardcvocabs.model.VocabModel;
import com.fasterxml.jackson.databind.JsonNode;
Expand All @@ -22,14 +24,94 @@

public class ArdcVocabServiceImpl implements ArdcVocabService {

protected Logger log = LoggerFactory.getLogger(ArdcVocabServiceImpl.class);
protected static Logger log = LoggerFactory.getLogger(ArdcVocabServiceImpl.class);

@Value("${ardcvocabs.baseUrl:https://vocabs.ardc.edu.au/repository/api/lda/aodn}")
protected String vocabApiBase;

protected RestTemplate restTemplate;
protected RetryTemplate retryTemplate;

protected static final String VERSION_REGEX = "/(version-\\d+-\\d+)(?:/|$)";

public Map<String, Map<PathName, String>> getResolvedPathCollection() {
Map<String, Map<PathName, String>> resolvedPathCollection = new HashMap<>();
for (ArdcCurrentPaths currentPath : ArdcCurrentPaths.values()) {
try {
ObjectNode categoryCurrentContent = fetchCurrentContents(currentPath.getCategoryCurrent());
ObjectNode vocabCurrentContent = fetchCurrentContents(currentPath.getVocabCurrent());

if (categoryCurrentContent != null && vocabCurrentContent != null) {
// Extract versions
String categoryVersion = extractVersionFromCurrentContent(categoryCurrentContent);
String vocabVersion = extractVersionFromCurrentContent(vocabCurrentContent);

if (categoryVersion != null && vocabVersion != null) {
log.info("Fetched ARDC category version for {}: {}", currentPath.name(), categoryVersion);
log.info("Fetched ARDC vocab version for {}: {}", currentPath.name(), vocabVersion);

// Build and store resolved paths
Map<PathName, String> resolvedPaths = buildResolvedPaths(currentPath, categoryVersion, vocabVersion);
resolvedPathCollection.put(currentPath.name(), resolvedPaths);
} else {
log.error("Failed to extract versions for {}", currentPath.name());
}
} else {
log.error("Failed to fetch HTML content for {}", currentPath.name());
}
} catch (Exception e) {
log.error("Error initialising versions for {}: {}", currentPath.name(), e.getMessage(), e);
}
}
return resolvedPathCollection;
}

private ObjectNode fetchCurrentContents(String url) {
try {
return retryTemplate.execute(context -> restTemplate.getForObject(url, ObjectNode.class));
} catch (RestClientException e) {
log.error("Failed to fetch HTML content from URL {}: {}", url, e.getMessage());
} catch (Exception e) {
log.error("Unexpected error while fetching HTML content from URL {}: {}", url, e.getMessage(), e);
}
return null;
}

protected Map<PathName, String> buildResolvedPaths(ArdcCurrentPaths currentPaths, String categoryVersion, String vocabVersion) {
Map<PathName, String> resolvedPaths = new HashMap<>();
for (VocabApiPaths vocabApiPath : VocabApiPaths.values()) {
if (currentPaths.name().equals(vocabApiPath.name())) {
resolvedPaths.put(PathName.categoryApi, String.format(vocabApiPath.getCategoryApiTemplate(), categoryVersion));
resolvedPaths.put(PathName.categoryDetailsApi, String.format(vocabApiPath.getCategoryDetailsTemplate(), categoryVersion, "%s"));
resolvedPaths.put(PathName.vocabApi, String.format(vocabApiPath.getVocabApiTemplate(), vocabVersion));
resolvedPaths.put(PathName.vocabDetailsApi, String.format(vocabApiPath.getVocabDetailsTemplate(), vocabVersion, "%s"));
}
}
return resolvedPaths;
}

protected String extractVersionFromCurrentContent(ObjectNode currentContent) {
if (currentContent != null && !currentContent.isEmpty()) {
JsonNode node = currentContent.get("result");
if (!about.apply(node).isEmpty()) {
Pattern pattern = Pattern.compile(VERSION_REGEX);
Matcher matcher = pattern.matcher(about.apply(node));

if (matcher.find()) {
String version = matcher.group(1);
log.info("Valid Version Found: {}", version);
return version;
} else {
log.warn("Version does not match the required format: {}", about.apply(node));
}
}

} else {
log.warn("Current content is empty or null.");
}
return null;
}

protected Function<JsonNode, String> extractSingleText(String key) {
return (node) -> {
JsonNode labelNode = node.get(key);
Expand Down Expand Up @@ -90,10 +172,10 @@ public ArdcVocabServiceImpl(RestTemplate restTemplate, RetryTemplate retryTempla
this.retryTemplate = retryTemplate;
}

protected VocabModel buildVocabByResourceUri(String vocabUri, String vocabApiBase, VocabApiPaths vocabApiPaths) {
protected VocabModel buildVocabByResourceUri(String vocabUri, String vocabApiBase, Map<PathName, String> resolvedPaths) {
String resourceDetailsApi = vocabUri.contains("_classes")
? vocabApiPaths.getVocabCategoryDetailsApiPath()
: vocabApiPaths.getVocabDetailsApiPath();
? resolvedPaths.get(PathName.categoryDetailsApi)
: resolvedPaths.get(PathName.vocabDetailsApi);

String detailsUrl = String.format(vocabApiBase + resourceDetailsApi, vocabUri);

Expand Down Expand Up @@ -123,7 +205,7 @@ protected VocabModel buildVocabByResourceUri(String vocabUri, String vocabApiBas
for (JsonNode j : target.get("narrower")) {
if (!about.apply(j).isEmpty()) {
// recursive call
VocabModel narrowerNode = buildVocabByResourceUri(about.apply(j), vocabApiBase, vocabApiPaths);
VocabModel narrowerNode = buildVocabByResourceUri(about.apply(j), vocabApiBase, resolvedPaths);
if (narrowerNode != null) {
narrowerNodes.add(narrowerNode);
}
Expand All @@ -143,7 +225,7 @@ protected VocabModel buildVocabByResourceUri(String vocabUri, String vocabApiBas
return null;
}

protected <T> VocabModel buildVocabModel(T currentNode, String vocabApiBase, VocabApiPaths vocabApiPaths) {
protected <T> VocabModel buildVocabModel(T currentNode, String vocabApiBase, Map<PathName, String> resolvedPaths) {
String resourceUri = null;

if (currentNode instanceof ObjectNode objectNode) {
Expand All @@ -161,12 +243,12 @@ protected <T> VocabModel buildVocabModel(T currentNode, String vocabApiBase, Voc
throw new IllegalArgumentException("Unsupported node type: " + currentNode.getClass().getName());
}

return buildVocabByResourceUri(resourceUri, vocabApiBase, vocabApiPaths);
return buildVocabByResourceUri(resourceUri, vocabApiBase, resolvedPaths);
}

protected Map<String, List<VocabModel>> getVocabLeafNodes(String vocabApiBase, VocabApiPaths vocabApiPaths) {
protected Map<String, List<VocabModel>> getVocabLeafNodes(String vocabApiBase, Map<PathName, String> resolvedPaths) {
Map<String, List<VocabModel>> results = new HashMap<>();
String url = String.format(vocabApiBase + vocabApiPaths.getVocabApiPath());
String url = String.format(vocabApiBase + resolvedPaths.get(PathName.vocabApi));

while (url != null && !url.isEmpty()) {
try {
Expand All @@ -180,7 +262,7 @@ protected Map<String, List<VocabModel>> getVocabLeafNodes(String vocabApiBase, V
if (isNodeValid.apply(node, "items")) {
for (JsonNode j : node.get("items")) {
// Now we need to construct link to detail resources
String dl = String.format(vocabApiBase + vocabApiPaths.getVocabDetailsApiPath(), about.apply(j));
String dl = String.format(vocabApiBase + resolvedPaths.get(PathName.vocabDetailsApi), about.apply(j));
try {
log.debug("getVocabLeafNodes -> {}", dl);
ObjectNode d = retryTemplate.execute(context -> restTemplate.getForObject(dl, ObjectNode.class));
Expand All @@ -205,7 +287,7 @@ protected Map<String, List<VocabModel>> getVocabLeafNodes(String vocabApiBase, V
List<VocabModel> vocabNarrower = new ArrayList<>();
if(target.has("narrower") && !target.get("narrower").isEmpty()) {
for(JsonNode currentNode : target.get("narrower")) {
VocabModel narrowerNode = buildVocabModel(currentNode, vocabApiBase, vocabApiPaths);
VocabModel narrowerNode = buildVocabModel(currentNode, vocabApiBase, resolvedPaths);
if (narrowerNode != null) {
vocabNarrower.add(narrowerNode);
}
Expand All @@ -230,7 +312,7 @@ protected Map<String, List<VocabModel>> getVocabLeafNodes(String vocabApiBase, V
List<VocabModel> completedInternalNodes = new ArrayList<>();
vocab.getNarrower().forEach(currentInternalNode -> {
// rebuild currentInternalNode (no linked leaf nodes) to completedInternalNode (with linked leaf nodes)
VocabModel completedInternalNode = buildVocabModel(currentInternalNode, vocabApiBase, vocabApiPaths);
VocabModel completedInternalNode = buildVocabModel(currentInternalNode, vocabApiBase, resolvedPaths);
if (completedInternalNode != null) {
// each internal node now will have linked narrower nodes (if available)
completedInternalNodes.add(completedInternalNode);
Expand Down Expand Up @@ -269,9 +351,9 @@ protected Map<String, List<VocabModel>> getVocabLeafNodes(String vocabApiBase, V
}

@Override
public List<VocabModel> getVocabTreeFromArdcByType(VocabApiPaths vocabApiPaths) {
Map<String, List<VocabModel>> vocabLeafNodes = getVocabLeafNodes(vocabApiBase, vocabApiPaths);
String url = String.format(vocabApiBase + vocabApiPaths.getVocabCategoryApiPath());
public List<VocabModel> getVocabTreeFromArdcByType(Map<PathName, String> resolvedPaths) {
Map<String, List<VocabModel>> vocabLeafNodes = getVocabLeafNodes(vocabApiBase, resolvedPaths);
String url = String.format(vocabApiBase + resolvedPaths.get(PathName.categoryApi));
List<VocabModel> vocabCategoryNodes = new ArrayList<>();
while (url != null && !url.isEmpty()) {
try {
Expand Down Expand Up @@ -299,7 +381,7 @@ public List<VocabModel> getVocabTreeFromArdcByType(VocabApiPaths vocabApiPaths)
Map<String, List<VocabModel>> internalVocabCategoryNodes = new HashMap<>();
if (j.has("narrower") && !j.get("narrower").isEmpty()) {
j.get("narrower").forEach(currentNode -> {
VocabModel internalNode = buildVocabModel(currentNode, vocabApiBase, vocabApiPaths);
VocabModel internalNode = buildVocabModel(currentNode, vocabApiBase, resolvedPaths);
if (internalNode != null) {
List<VocabModel> leafNodes = vocabLeafNodes.getOrDefault(internalNode.getAbout(), Collections.emptyList());
if (!leafNodes.isEmpty()) {
Expand Down
Loading
Loading