Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FEATURE] Performance improvement #79

Open
afillatre opened this issue Apr 11, 2021 · 3 comments
Open

[FEATURE] Performance improvement #79

afillatre opened this issue Apr 11, 2021 · 3 comments
Assignees
@cesarsotovalero
Labels
feature New feature request

Comments

@afillatre
Copy link
Contributor

Is your feature request related to a problem? Please describe.
On large project, depclean takes a very long time to execute, most of the time is spent on excluding dependencies.

Here are some details. Given this analysis result:

USED DIRECT DEPENDENCIES [37]
USED INHERITED DEPENDENCIES [0]: 
USED TRANSITIVE DEPENDENCIES [218]:
POTENTIALLY UNUSED DIRECT DEPENDENCIES [18]:
POTENTIALLY UNUSED INHERITED DEPENDENCIES [0]: 
POTENTIALLY UNUSED TRANSITIVE DEPENDENCIES [225]:

It took 5h29min to complete depclean debloated-pom generation.

I have no timestamps, but I'd say it was between 20/30 min to analyse the dependencies, then the rest to exclude them and generate the debloated pom

Describe the solution you'd like
Better perfs, so this can be included in some regular CI jobs

Describe alternatives you've considered
None ATM
@afillatre afillatre added the feature New feature request label Apr 11, 2021
@ABHAY0O7 ABHAY0O7 mentioned this issue Jul 4, 2021
Closed
@cesarsotovalero cesarsotovalero self-assigned this Apr 7, 2022
@cesarsotovalero
Copy link
Collaborator

cesarsotovalero commented Apr 7, 2022 •
edited
Loading

Hi @afillatre,
Thanks for reporting this issue (and YES!... performance is a feature 😄).
At the moment, DepClean generates the debloated-pom.xml using the following method writePom https://github.com/castor-software/depclean/blob/e1f510d347d72d4fe45ba5e2cf74dbe16ec8ac81/depclean-maven-plugin/src/main/java/se/kth/depclean/util/MavenDebloater.java#L103
It relies on the Maven dependency plugins for the actual task of writing the file, so I suspect that the performance issues are located there and not in DepClean itself.
WDYT?

@afillatre
Copy link
Contributor Author

It would be nice to run a performance analysis tool during the run to understand where the issue is. I cant' say for sure where the problem is ATM

@algomaster99
Copy link
Member

algomaster99 commented Apr 27, 2023 •
edited
Loading

I faced the same issue with org.apache.accumulo:accumulo-test on commit apache/accumulo@85796e5. It took 525 minutes and 3 seconds to complete the analysis, but I at least have the report 😄

[INFO] -----------------< org.apache.accumulo:accumulo-test >------------------
[INFO] Building Apache Accumulo Testing 3.0.0-SNAPSHOT                  [15/17]
[INFO]   from test/pom.xml
[INFO] --------------------------------[ jar ]---------------------------------
Downloading from apache.snapshots: https://repository.apache.org/snapshots/org/apache/accumulo/accumulo-minicluster/3.0.0-SNAPSHOT/maven-metadata.xml
Downloaded from apache.snapshots: https://repository.apache.org/snapshots/org/apache/accumulo/accumulo-minicluster/3.0.0-SNAPSHOT/maven-metadata.xml (792 B at 3.7 kB/s)
Downloading from apache.snapshots: https://repository.apache.org/snapshots/org/apache/accumulo/accumulo-native/3.0.0-SNAPSHOT/maven-metadata.xml
Downloaded from apache.snapshots: https://repository.apache.org/snapshots/org/apache/accumulo/accumulo-native/3.0.0-SNAPSHOT/maven-metadata.xml (790 B at 2.6 kB/s)
Downloading from apache.snapshots: https://repository.apache.org/snapshots/org/apache/accumulo/accumulo-shell/3.0.0-SNAPSHOT/maven-metadata.xml
Downloaded from apache.snapshots: https://repository.apache.org/snapshots/org/apache/accumulo/accumulo-shell/3.0.0-SNAPSHOT/maven-metadata.xml (786 B at 6.8 kB/s)
Downloading from apache.snapshots: https://repository.apache.org/snapshots/org/apache/accumulo/accumulo-iterator-test-harness/3.0.0-SNAPSHOT/maven-metadata.xml
Downloaded from apache.snapshots: https://repository.apache.org/snapshots/org/apache/accumulo/accumulo-iterator-test-harness/3.0.0-SNAPSHOT/maven-metadata.xml (802 B at 6.2 kB/s)
Downloading from apache.snapshots: https://repository.apache.org/snapshots/org/apache/accumulo/accumulo-compaction-coordinator/3.0.0-SNAPSHOT/maven-metadata.xml
Downloaded from apache.snapshots: https://repository.apache.org/snapshots/org/apache/accumulo/accumulo-compaction-coordinator/3.0.0-SNAPSHOT/maven-metadata.xml (803 B at 5.9 kB/s)
Downloading from apache.snapshots: https://repository.apache.org/snapshots/org/apache/accumulo/accumulo-compactor/3.0.0-SNAPSHOT/maven-metadata.xml
Downloaded from apache.snapshots: https://repository.apache.org/snapshots/org/apache/accumulo/accumulo-compactor/3.0.0-SNAPSHOT/maven-metadata.xml (790 B at 8.8 kB/s)
Downloading from apache.snapshots: https://repository.apache.org/snapshots/org/apache/accumulo/accumulo-gc/3.0.0-SNAPSHOT/maven-metadata.xml
Downloaded from apache.snapshots: https://repository.apache.org/snapshots/org/apache/accumulo/accumulo-gc/3.0.0-SNAPSHOT/maven-metadata.xml (783 B at 3.0 kB/s)
Downloading from apache.snapshots: https://repository.apache.org/snapshots/org/apache/accumulo/accumulo-manager/3.0.0-SNAPSHOT/maven-metadata.xml
Downloaded from apache.snapshots: https://repository.apache.org/snapshots/org/apache/accumulo/accumulo-manager/3.0.0-SNAPSHOT/maven-metadata.xml (788 B at 7.2 kB/s)
Downloading from apache.snapshots: https://repository.apache.org/snapshots/org/apache/accumulo/accumulo-monitor/3.0.0-SNAPSHOT/maven-metadata.xml
Downloaded from apache.snapshots: https://repository.apache.org/snapshots/org/apache/accumulo/accumulo-monitor/3.0.0-SNAPSHOT/maven-metadata.xml (788 B at 6.5 kB/s)
Downloading from apache.snapshots: https://repository.apache.org/snapshots/org/apache/accumulo/accumulo-server-base/3.0.0-SNAPSHOT/maven-metadata.xml
Downloaded from apache.snapshots: https://repository.apache.org/snapshots/org/apache/accumulo/accumulo-server-base/3.0.0-SNAPSHOT/maven-metadata.xml (792 B at 2.1 kB/s)
Downloading from apache.snapshots: https://repository.apache.org/snapshots/org/apache/accumulo/accumulo-tserver/3.0.0-SNAPSHOT/maven-metadata.xml
Downloaded from apache.snapshots: https://repository.apache.org/snapshots/org/apache/accumulo/accumulo-tserver/3.0.0-SNAPSHOT/maven-metadata.xml (788 B at 10 kB/s)
[INFO] 
[INFO] --- depclean:2.0.6:depclean (default-cli) @ accumulo-test ---
-------------------------------------------------------
[INFO] Starting DepClean dependency analysis
[INFO] Ignoring scope Scope(value=import)
[INFO] Ignoring scope Scope(value=runtime)
[INFO] Ignoring scope Scope(value=system)
[INFO] Ignoring scope Scope(value=provided)
[INFO] Ignoring scope Scope(value=test)
-------------------------------------------------------
 D E P C L E A N   A N A L Y S I S   R E S U L T S
-------------------------------------------------------
USED DIRECT DEPENDENCIES [43]: 
	org.apache.hadoop:hadoop-client-minicluster:3.3.5:compile (35 MB)
	org.apache.hadoop:hadoop-client-runtime:3.3.5:runtime (28 MB)
	org.apache.hadoop:hadoop-client-api:3.3.5:compile (18 MB)
	org.apache.accumulo:accumulo-core:3.0.0-20230426.091822-78:compile (6 MB)
	org.bouncycastle:bcprov-jdk15on:1.70:compile (5 MB)
	io.micrometer:micrometer-registry-statsd:1.10.6:compile (5 MB)
	org.apache.curator:curator-test:5.3.0:compile (2 MB)
	com.google.guava:guava:31.1-jre:compile (2 MB)
	org.apache.zookeeper:zookeeper:3.8.1:compile (1 MB)
	org.jline:jline:3.22.0:compile (1 MB)
	org.bouncycastle:bcpkix-jdk15on:1.70:compile (941 KB)
	io.micrometer:micrometer-core:1.10.6:compile (760 KB)
	com.github.ben-manes.caffeine:caffeine:3.1.4:compile (734 KB)
	org.apache.accumulo:accumulo-monitor:3.0.0-20230426.091823-78:compile (716 KB)
	org.apache.accumulo:accumulo-server-base:3.0.0-20230426.091823-78:compile (644 KB)
	org.apache.commons:commons-configuration2:2.8.0:compile (617 KB)
	org.apache.commons:commons-lang3:3.12.0:compile (573 KB)
	org.easymock:easymock:4.3:compile (563 KB)
	org.apache.accumulo:accumulo-tserver:3.0.0-20230426.091824-78:compile (537 KB)
	org.apache.thrift:libthrift:0.17.0:compile (345 KB)
	org.apache.logging.log4j:log4j-1.2-api:2.20.0:compile (336 KB)
	commons-io:commons-io:2.11.0:compile (319 KB)
	org.apache.accumulo:accumulo-manager:3.0.0-20230426.091823-78:compile (287 KB)
	org.apache.accumulo:accumulo-shell:3.0.0-20230426.091823-78:compile (272 KB)
	com.google.code.gson:gson:2.9.1:compile (258 KB)
	org.apache.zookeeper:zookeeper-jute:3.8.1:compile (250 KB)
	org.junit.jupiter:junit-jupiter-engine:5.9.2:compile (240 KB)
	org.junit.jupiter:junit-jupiter-api:5.9.2:compile (202 KB)
	org.apache.hadoop:hadoop-distcp:3.3.5:compile (154 KB)
	io.opentelemetry:opentelemetry-api:1.25.0:compile (126 KB)
	com.beust:jcommander:1.82:compile (86 KB)
	org.apache.accumulo:accumulo-minicluster:3.0.0-20230426.091823-78:compile (82 KB)
	org.slf4j:slf4j-api:2.0.6:compile (61 KB)
	commons-cli:commons-cli:1.5.0:compile (56 KB)
	org.apache.accumulo:accumulo-gc:3.0.0-20230426.091822-78:compile (48 KB)
	io.opentelemetry:opentelemetry-context:1.25.0:compile (46 KB)
	org.apache.accumulo:accumulo-compaction-coordinator:3.0.0-20230426.091822-78:compile (43 KB)
	org.apache.accumulo:accumulo-iterator-test-harness:3.0.0-20230426.091823-78:test (33 KB)
	org.apache.accumulo:accumulo-compactor:3.0.0-20230426.091822-78:compile (32 KB)
	org.apache.logging.log4j:log4j-slf4j2-impl:2.20.0:test (25 KB)
	org.apache.hadoop:hadoop-minikdc:3.3.5:compile (16 KB)
	org.apache.accumulo:accumulo-start:3.0.0-20230426.091824-78:compile (16 KB)
	com.google.auto.service:auto-service:1.0.1:compile (12 KB)
USED TRANSITIVE DEPENDENCIES [98]: 
	org.apache.commons:commons-math3:3.6.1:compile (2 MB)
	org.xerial.snappy:snappy-java:1.1.8.4:compile (1 MB)
	org.apache.logging.log4j:log4j-core:2.20.0:compile (1 MB)
	org.freemarker:freemarker:2.3.32:compile (1 MB)
	com.fasterxml.jackson.core:jackson-databind:2.14.2:compile (1 MB)
	org.hibernate.validator:hibernate-validator:7.0.5.Final:runtime (1 MB)
	org.glassfish.jersey.core:jersey-common:3.0.9:compile (1 MB)
	org.glassfish.jersey.core:jersey-server:3.0.9:compile (934 KB)
	org.apache.datasketches:datasketches-java:3.3.0:compile (812 KB)
	org.eclipse.jetty:jetty-server:11.0.13:compile (777 KB)
	org.javassist:javassist:3.29.2-GA:runtime (775 KB)
	org.apache.commons:commons-collections4:4.4:compile (734 KB)
	io.netty:netty-common:4.1.89.Final:compile (642 KB)
	org.eclipse.jetty.toolchain:jetty-jakarta-servlet-api:5.0.2:compile (590 KB)
	org.junit.jupiter:junit-jupiter-params:5.9.2:test (566 KB)
	org.eclipse.jetty:jetty-util:11.0.13:compile (554 KB)
	io.netty:netty-handler:4.1.89.Final:compile (532 KB)
	io.netty:netty-transport:4.1.89.Final:compile (476 KB)
	org.bouncycastle:bcutil-jdk15on:1.70:compile (471 KB)
	com.fasterxml.jackson.core:jackson-core:2.14.2:compile (448 KB)
	junit:junit:4.13.2:compile (375 KB)
	commons-codec:commons-codec:1.15:compile (345 KB)
	io.netty:netty-codec:4.1.89.Final:compile (337 KB)
	org.yaml:snakeyaml:1.33:compile (323 KB)
	org.apache.logging.log4j:log4j-api:2.20.0:compile (305 KB)
	jakarta.servlet:jakarta.servlet-api:5.0.0:compile (300 KB)
	io.netty:netty-buffer:4.1.89.Final:compile (297 KB)
	org.glassfish.jersey.core:jersey-client:3.0.9:compile (261 KB)
	org.eclipse.jetty:jetty-http:11.0.13:compile (243 KB)
	org.apache.commons:commons-text:1.10.0:compile (232 KB)
	org.apache.kerby:kerb-core:1.0.1:compile (221 KB)
	org.checkerframework:checker-qual:3.26.0:compile (217 KB)
	org.eclipse.jetty:jetty-io:11.0.13:compile (211 KB)
	org.apache.kerby:kerby-pkix:1.0.1:compile (199 KB)
	org.glassfish.hk2:hk2-locator:3.0.3:runtime (199 KB)
	org.glassfish.hk2:hk2-api:3.0.3:compile (196 KB)
	org.junit.platform:junit-platform-engine:1.9.2:compile (184 KB)
	org.hdrhistogram:HdrHistogram:2.1.12:runtime (169 KB)
	org.glassfish:jakarta.el:4.0.2:runtime (169 KB)
	org.eclipse.jetty:jetty-servlet:11.0.13:compile (162 KB)
	io.netty:netty-transport-classes-epoll:4.1.89.Final:compile (141 KB)
	jakarta.ws.rs:jakarta.ws.rs-api:3.0.0:compile (135 KB)
	io.dropwizard.metrics:metrics-core:3.2.5:compile (133 KB)
	org.glassfish.hk2:hk2-utils:3.0.3:compile (128 KB)
	org.apache.datasketches:datasketches-memory:2.1.0:compile (126 KB)
	jakarta.xml.bind:jakarta.xml.bind-api:3.0.1:compile (125 KB)
	org.eclipse.jetty:jetty-security:11.0.13:compile (118 KB)
	org.apache.kerby:kerb-crypto:1.0.1:compile (113 KB)
	org.apache.kerby:kerb-client:1.0.1:compile (110 KB)
	com.google.auto:auto-common:1.2:compile (108 KB)
	org.junit.platform:junit-platform-commons:1.9.2:compile (100 KB)
	org.apache.kerby:kerby-asn1:1.0.1:compile (99 KB)
	jakarta.validation:jakarta.validation-api:3.0.0:compile (90 KB)
	org.glassfish.jersey.ext:jersey-entity-filtering:3.0.9:compile (81 KB)
	org.apache.kerby:kerb-server:1.0.1:compile (80 KB)
	org.apache.kerby:kerb-admin:1.0.1:compile (79 KB)
	jakarta.el:jakarta.el-api:4.0.0:runtime (78 KB)
	org.glassfish.jersey.media:jersey-media-json-jackson:3.0.9:compile (78 KB)
	org.glassfish.jersey.inject:jersey-hk2:3.0.9:runtime (77 KB)
	com.fasterxml.jackson.core:jackson-annotations:2.14.2:compile (74 KB)
	org.glassfish.jersey.containers:jersey-container-servlet-core:3.0.9:compile (71 KB)
	org.glassfish.jaxb:txw2:4.0.2:runtime (71 KB)
	com.fasterxml:classmate:1.5.1:runtime (66 KB)
	org.apache.kerby:kerb-common:1.0.1:compile (63 KB)
	io.micrometer:micrometer-observation:1.10.6:compile (63 KB)
	org.jboss.logging:jboss-logging:3.5.0.Final:runtime (61 KB)
	com.sun.activation:jakarta.activation:2.0.1:compile (60 KB)
	commons-logging:commons-logging:1.2:compile (60 KB)
	org.glassfish.jersey.ext:jersey-bean-validation:3.0.9:runtime (58 KB)
	org.glassfish.jersey.ext:jersey-mvc:3.0.9:compile (52 KB)
	org.objenesis:objenesis:3.3:compile (48 KB)
	org.hamcrest:hamcrest-core:1.3:compile (43 KB)
	io.netty:netty-transport-native-unix-common:4.1.89.Final:compile (42 KB)
	org.apache.kerby:kerby-util:1.0.1:compile (39 KB)
	io.micrometer:micrometer-commons:1.10.6:compile (38 KB)
	io.netty:netty-transport-native-epoll:4.1.89.Final:runtime (38 KB)
	io.netty:netty-resolver:4.1.89.Final:compile (36 KB)
	io.opentelemetry:opentelemetry-semconv:1.25.0-alpha:compile (36 KB)
	org.apache.kerby:kerb-util:1.0.1:compile (35 KB)
	org.apache.logging.log4j:log4j-web:2.20.0:compile (34 KB)
	com.fasterxml.jackson.module:jackson-module-jakarta-xmlbind-annotations:2.14.2:compile (31 KB)
	org.apache.kerby:kerby-config:1.0.1:compile (29 KB)
	org.latencyutils:LatencyUtils:2.0.3:runtime (29 KB)
	org.glassfish.hk2.external:aopalliance-repackaged:3.0.3:compile (26 KB)
	jakarta.annotation:jakarta.annotation-api:2.0.0:compile (24 KB)
	org.glassfish.jersey.ext:jersey-mvc-freemarker:3.0.9:compile (23 KB)
	org.apache.yetus:audience-annotations:0.12.0:compile (20 KB)
	org.apache.kerby:kerb-simplekdc:1.0.1:compile (19 KB)
	org.apache.kerby:kerb-identity:1.0.1:compile (19 KB)
	com.google.code.findbugs:jsr305:3.0.2:compile (19 KB)
	org.glassfish.hk2:osgi-resource-locator:1.0.3:compile (19 KB)
	com.google.errorprone:error_prone_annotations:2.15.0:compile (15 KB)
	jakarta.inject:jakarta.inject-api:2.0.0:compile (9 KB)
	com.google.j2objc:j2objc-annotations:1.3:compile (8 KB)
	org.opentest4j:opentest4j:1.2.0:compile (7 KB)
	org.apiguardian:apiguardian-api:1.1.2:compile (6 KB)
	com.google.guava:failureaccess:1.0.1:compile (4 KB)
	com.google.auto.service:auto-service-annotations:1.0.1:compile (3 KB)
USED INHERITED DIRECT DEPENDENCIES [1]: 
	com.github.spotbugs:spotbugs-annotations:4.7.3:compile (14 KB)
USED INHERITED TRANSITIVE DEPENDENCIES [0]: 
POTENTIALLY UNUSED DIRECT DEPENDENCIES [1]: 
	org.apache.accumulo:accumulo-native:3.0.0-20230426.091823-78:compile (13 KB)
POTENTIALLY UNUSED TRANSITIVE DEPENDENCIES [4]: 
	org.apache.kerby:kerby-xdr:1.0.1:compile (28 KB)
	org.apache.kerby:token-provider:1.0.1:compile (18 KB)
	io.netty:netty-transport-native-epoll:4.1.89.Final:compile (5 KB)
	com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava:compile (2 KB)
POTENTIALLY UNUSED INHERITED DIRECT DEPENDENCIES [0]: 
POTENTIALLY UNUSED INHERITED TRANSITIVE DEPENDENCIES [0]: 
[INFO] Analysis done in 525min 3s

Summary for complete accumulo project

Command to reproduce:

mvn se.kth.castor:depclean-maven-plugin:2.0.6:depclean -DfailIfUnusedDirect=false -DfailIfUnusedTransitive=false -DignoreScopes=provided,test,runtime,system,import
[INFO] --- depclean:2.0.6:depclean (default-cli) @ accumulo ---
-------------------------------------------------------
[INFO] Starting DepClean dependency analysis
[INFO] Skipping because packaging type is pom
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Apache Accumulo Project 3.0.0-SNAPSHOT:
[INFO] 
[INFO] Apache Accumulo Project ............................ SUCCESS [  0.684 s]
[INFO] Apache Accumulo Start .............................. SUCCESS [ 13.324 s]
[INFO] Apache Accumulo Core ............................... SUCCESS [09:42 min]
[INFO] Apache Accumulo Server Base ........................ SUCCESS [20:34 min]
[INFO] Apache Accumulo Compaction Coordinator ............. SUCCESS [21:55 min]
[INFO] Apache Accumulo Compactor .......................... SUCCESS [21:32 min]
[INFO] Apache Accumulo GC Server .......................... SUCCESS [21:53 min]
[INFO] Apache Accumulo Manager Server ..................... SUCCESS [21:48 min]
[INFO] Apache Accumulo Monitor Server ..................... SUCCESS [22:17 min]
[INFO] Apache Accumulo Tablet Server ...................... SUCCESS [22:46 min]
[INFO] Apache Accumulo MiniCluster ........................ SUCCESS [23:00 min]
[INFO] Apache Accumulo Native Libraries ................... SUCCESS [  0.023 s]
[INFO] Apache Accumulo Shell .............................. SUCCESS [  02:58 h]
[INFO] Apache Accumulo Iterator Test Harness .............. SUCCESS [27:17 min]
[INFO] Apache Accumulo Testing ............................ SUCCESS [  08:45 h]
[INFO] Apache Accumulo Hadoop MapReduce ................... SUCCESS [31:15 min]
[INFO] Apache Accumulo .................................... SUCCESS [  0.489 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  15:47 h
[INFO] Finished at: 2023-04-27T09:56:22+02:00
[INFO] ------------------------------------------------------------------------

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@cesarsotovalero cesarsotovalero

Labels
feature New feature request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@afillatre @cesarsotovalero @algomaster99