diff --git a/README.md b/README.md index 4130bf02b..24dd015d0 100644 --- a/README.md +++ b/README.md @@ -2,410 +2,17 @@ An addon that provide tools to display platform usage statistics using charts. -# Prerequisites - -The addon requires Meeds server with Elasticsearch. - ## Configuration options -You can can change some options of the addon using properties that you can add into [exo.properties](https://docs.exoplatform.org/en/latest/Configuration.html#configuration-configurationoverview) file - | VARIABLE | MANDATORY | DEFAULT VALUE | DESCRIPTION | |------------------------|-----------|---------------|-------------------------------------------------------------------------------------------| -| exo.analytics.admin.permission | NO | *:/platform/analytics | Group of users that can modify Charts application settings. All other users, *even members of /platform/administrators* will not be able to modify charts settings | -| exo.analytics.viewall.permission | NO | *:/platform/administrators | Group of users that can consult all data in Graphs. | -| exo.analytics.view.permission | NO | *:/platform/users | Group of users that can consult their personal and their spaces Graphs. | -| exo.analytics.aggregation.terms.doc_size | NO | 200 | Limit of number of resturned documents in aggregations result of type 'terms' (not used for aggregations of type : sum, avg, date_histogram, histogram and cardinality) | -| exo.addon.analytics.es.mapping.path | NO | jar:/analytics-es-mapping.json | Initial [ES mapping](https://github.com/Meeds-io/analytics/blob/master/analytics-services/src/main/resources/analytics-es-mapping.json) that will be used to generate daily [ES index](https://www.elastic.co/guide/en/elasticsearch/reference/5.6/mapping.html) | -| exo.cache.analytics.queue.MaxNodes | NO | 2000 | Number of maximum entries in in-memory cached Analytics Queue that is processed each 10 seconds | -| exo.cache.analytics.queue.TimeToLive | NO | -1 | lifetime of entries in Analytics Queue. Default: infinite. | -| exo.es.analytics.index.server.url | NO | Same as used [ES for eXo](https://docs.exoplatform.org/en/latest/Configuration.html#properties-of-the-elasticsearch-client) | Elasticsearch server URL used for indexing and searching analytics content | -| exo.es.analytics.index.server.username | NO | Same as used [ES for eXo](https://docs.exoplatform.org/en/latest/Configuration.html#properties-of-the-elasticsearch-client) | Elasticsearch server username used for indexing and searching analytics content | -| exo.es.analytics.index.server.password | NO | Same as used [ES for eXo](https://docs.exoplatform.org/en/latest/Configuration.html#properties-of-the-elasticsearch-client) | Elasticsearch server password used for indexing and searching analytics content | -| exo.es.analytics.index.per.day | YES | true | Whether create one index per day or not. You can switch this flag to true or false whener you want, it will not affect retrieved data result. In fact, the Analytics engine searches in all present indexes in server by including systematically a documents field filter that is added automatically to all statistics data : `isAnalytics = true` | -| exo.es.analytics.index.prefix | YES | analytics | Elasticsearch index names prefix. IT will be exactly equals to this value when `exo.es.analytics.index.per.day=false`, else the index names will be of pattern `${exo.es.analytics.index.prefix}_NUMBER_OF_DAYS` | - -## How to build addon - -1. install Apache maven 3.5+ -1. Go to Home path of addon -2. Execute the following command : - -```bash -mvn clean verify -Prun-its -``` - -## How to use it - -Once the addon installed and eXo Platform started, -and after [logged-in on eXo](https://docs.exoplatform.org/en/latest/GettingStarted.html#welcome-to-exo-platform), -you can follow those steps: - -1. Add a user into analytics administrators (Under Platform => Analytics) group [using group managment UI](https://docs.exoplatform.org/en/latest/Administration.html#managing-groups) -2. Go to application registry and [add Analytics portlet](https://docs.exoplatform.org/en/latest/Administration.html#managing-applications) in an application category -3. Go to home page and [add a new page](https://docs.exoplatform.org/en/latest/Administration.html#adding-a-new-page) in current site -4. You can add multiple instances of analytics portlet inside the page to show multiple statistics data -5. Save the page - -You will see the analytics application added in your page and displaying by default a default chart including all statistics data. -You can change settings of each application to show you different charts type, title and statistics data. - -By default, the user root is a member of `/platform/*analytics*` group, so he can see the charts, -while for other users (if you added right access permissions on page and added portlets in page), -he will see only his data without being able to change charts settings. -In fact, the user and space filtering on statistics data is made automatically switch loggedin user and page context (inside a space or in public page) - -The default collected statistics data are: -* [Activities statistics data](https://github.com/Meeds-io/analytics/blob/master/analytics-listeners/src/main/java/org/exoplatform/analytics/listener/AnalyticsActivityListener.java) : - * create activity - * create comment - * update activity - * update comment - * like activity - * like comment. -* [Space statistics data](https://github.com/Meeds-io/analytics/blob/master/analytics-listeners/src/main/java/org/exoplatform/analytics/listener/SpaceAnalyticsListener.java): - * create a space - * remove a space - * rename a space - * update space description - * update space registration - * update space access - * update space banner - * update space avatar - * add space application - * disable space application - * delete space application - * grant space member as manager - * revoke space member as manager - * joining a space - * leaving a space - * inviting a user to a space - * requesting join a space by a user -* [User statistics data](https://github.com/Meeds-io/analytics/blob/master/analytics-listeners/src/main/java/org/exoplatform/analytics/listener/UserAnalyticsEventListener.java): - * Create a user - * Delete a user - * Enable/Disable a user -* [User Profile statistics data](https://github.com/Meeds-io/analytics/blob/master/analytics-listeners/src/main/java/org/exoplatform/analytics/listener/AnalyticsProfileListener.java): - * create new user social profile (generally triggered at same time than create a user event) - * update avatar - * update banner - * update contact section - * update experience section -* [User login statistics data](https://github.com/Meeds-io/analytics/blob/master/analytics-listeners/src/main/java/org/exoplatform/analytics/listener/LoginAnalyticsListener.java): - * login / logout of a user -* [User relationship statistics data](https://github.com/Meeds-io/analytics/blob/master/analytics-listeners/src/main/java/org/exoplatform/analytics/listener/AnalyticsRelationshipListener.java): - * request connection with other user - * accept connection with other user - * deny connection with other user - * ignore connection request of other user - * remove connection with other user -* [Displayed page statistics data](https://github.com/Meeds-io/analytics/blob/1.0.0-M04/analytics-listeners/src/main/java/org/exoplatform/analytics/listener/PageAccessListener.java) -* [Extensible UI and DOM Events statistics data](https://github.com/Meeds-io/analytics/blob/master/analytics-listeners/src/main/java/org/exoplatform/analytics/listener/WebSocketUIStatisticListener.java): - * [Home link click](https://github.com/Meeds-io/analytics/blob/1.0.0-M04/analytics-webapps/src/main/webapp/WEB-INF/conf/analytics/analytics-ui-watchers-configuration.xml#L8-L84) - * [Hamburger menu topbar click](https://github.com/Meeds-io/analytics/blob/1.0.0-M04/analytics-webapps/src/main/webapp/WEB-INF/conf/analytics/analytics-ui-watchers-configuration.xml#L436-L498) - * [Hamburger menu page link click](https://github.com/Meeds-io/analytics/blob/1.0.0-M04/analytics-webapps/src/main/webapp/WEB-INF/conf/analytics/analytics-ui-watchers-configuration.xml#L85-L158) - * [Hamburger menu space link click](https://github.com/Meeds-io/analytics/blob/1.0.0-M04/analytics-webapps/src/main/webapp/WEB-INF/conf/analytics/analytics-ui-watchers-configuration.xml#L159-L232) - * [Post activity using drawer composer](https://github.com/Meeds-io/analytics/blob/1.0.0-M04/analytics-webapps/src/main/webapp/WEB-INF/conf/analytics/analytics-ui-watchers-configuration.xml#L291-L362) - * [Post activity using inpage composer](https://github.com/Meeds-io/analytics/blob/1.0.0-M04/analytics-webapps/src/main/webapp/WEB-INF/conf/analytics/analytics-ui-watchers-configuration.xml#L363-L435) - -Technical note: By default, the produced and persisted `statistic samples/data` by those `listeners` -has basic properties stored into it and that could be extended any time because a dynamic mapping is used on each daily -created Elasticsearch index. - -The default produced data for all events in general: -* `id` : Technical id of statistics sample -* `spaceId` : Technical id of space (space name is displayed in UI in list of samples) - (Perspective: could be retrieved using suggester for data filtering and charts setttings) -* `userId` : Technical id of user making the event (user display name is displayed in UI in list of samples) - (Perspective: could be retrieved using suggester for data filtering and charts setttings) -* `module` : eXo project name: `social`, `portal`... -* `subModule` : `activity`, `space`, `login`, `relationship`, `profile`, `relationship`... -* `operation` : type of made operation, like `createActivity`, `login`, `logout`, `login`, `confirmed`... -* `status` : status of operation `OK` if operation succeeded else `KO` -* `timestamp` : time of statistic event in ms - -Specifically for each listener, it has default produced data are: (data can be extend, like adding number of likes, comments...) -* Activity statistic data: - * `module` = `social` - * `subModule` = `activity` - * `operation` = - * `createActivity` - * `updateActivity` - * `createComment` - * `updateComment` - * `likeActivity` - * `likeComment` - * `streamIdentityId` : Technical identity ID of stream owner (Space ID or User ID) - (Perspective: could be retrieved using suggester for data filtering and charts setttings) - * `activityId` : Technical activity ID - * `commentId` : Technical comment ID - * `subCommentId` : Technical sub comment ID - * `activityType` : activity type, like : `exosocial:people`, `USER_PROFILE_ACTIVITY`... (See complete list of [actitivy types](https://docs.exoplatform.org/en/latest/Configuration.html#enabling-disabling-any-activity-type)) -* Space statistic data: - * `module` = `social` - * `subModule` = `space` - * `operation` = - * `spaceAccessEdited` - * `spaceBannerEdited` - * `spaceCreated` - * `spaceRemoved` - * `spaceRenamed` - * `spaceDescriptionEdited` - * `spaceRegistrationEdited` - * `spaceAvatarEdited` - * `applicationActivated` - * `applicationDeactivated` - * `applicationRemoved` - * `grantedLead` - * `joined` - * `revokedLead` - * `addInvitedUser` - * `addPendingUser` - * `modifierUserId`: Technical identity ID of user having modified portlet - (Perspective: could be retrieved using suggester for data filtering and charts setttings) - * `spaceTemplate`: space template name -* Login statistic data: - * `module` = `portal` - * `subModule` = `login` - * `operation` = - * `login` - * `logout` -* User statistic data: - * `module` = `organization` - * `subModule` = `user` - * `operation` = - * `createUser` - * `saveUser` - * `enableUser` - * `deleteUser` - * `isEnabled` : if user is enabled -* Profile statistic data: - * `module` = `social` - * `subModule` = `profile` - * `operation` = - * `avatar` - * `banner` - * `contactSection` - * `experienceSection` - * `createProfile` -* Relationship statistic data: - * `module` = `social` - * `subModule` = `relationship` - * `operation` = - * `avatar` - * `banner` - * `contactSection` - * `experienceSection` - * `createProfile` - * `from`: Technical identity ID of user (Perspective: could be retrieved using suggester for data filtering and charts setttings) - * `to` : Technical identity ID of user (Perspective: could be retrieved using suggester for data filtering and charts setttings) -* Displayed page statistics data - * `module` = `portal` - * `subModule` = - * `webui` (in case of WebUI ajax query) - * `page` (in case of page display) - * `operation` = - * `ajaxRequest` (in case of WebUI ajax query) - * `pageDisplay` (in case of page display) - * `spaceTemplate` = Current space template - * `userLocale` = Current user locale - * `portalOwner` = Current PORTAL site (dw, intranet, ...) - * `portalUri` = Current URI - * `pageUri` = Current page URI (without portal URI, like 'documents', 'activities', 'wiki/user/root' ) - * `pageName` = current page name/uri last part - * `httpRequestMethod` - * `httpRequestUri` - * `httpRequestProtocol` - * `httpRequestMethod` - * `httpRequestContentType` - * `httpRequestContentLength` - * `httpResponseContentType` - * `httpResponseContentLength` - * `httpResponseStatus` -* Extensible **UI and DOM Events** statistics data: - * `module` = `portal` - * `subModule` = `ui` - * `portalUri` = current portal URI when user triggered the event - * For **DOM event** *Home page link* : - * `watcher` = `Home page link` (defined by [configuration](https://github.com/Meeds-io/analytics/blob/1.0.0-M04/analytics-webapps/src/main/webapp/WEB-INF/conf/analytics/analytics-ui-watchers-configuration.xml#L18)) - * `ui` = `toolbar` (defined by [configuration](https://github.com/Meeds-io/analytics/blob/1.0.0-M04/analytics-webapps/src/main/webapp/WEB-INF/conf/analytics/analytics-ui-watchers-configuration.xml#L32)) - * `application` = `NavigationToolbarPortlet` (defined by [configuration](https://github.com/Meeds-io/analytics/blob/1.0.0-M04/analytics-webapps/src/main/webapp/WEB-INF/conf/analytics/analytics-ui-watchers-configuration.xml#L40)) - * `operation` = `click` (defined by [configuration](https://github.com/Meeds-io/analytics/blob/1.0.0-M04/analytics-webapps/src/main/webapp/WEB-INF/conf/analytics/analytics-ui-watchers-configuration.xml#L22)) - * `dom-href` = link of target page (defined by [configuration](https://github.com/Meeds-io/analytics/blob/1.0.0-M04/analytics-webapps/src/main/webapp/WEB-INF/conf/analytics/analytics-ui-watchers-configuration.xml#L71)) - * `dom-class` = DOM class value of element having triggered the event(defined by [configuration](https://github.com/Meeds-io/analytics/blob/1.0.0-M04/analytics-webapps/src/main/webapp/WEB-INF/conf/analytics/analytics-ui-watchers-configuration.xml#L74)) - * `dom-id` = DOM class value of element having triggered the event(defined by [configuration](https://github.com/Meeds-io/analytics/blob/1.0.0-M04/analytics-webapps/src/main/webapp/WEB-INF/conf/analytics/analytics-ui-watchers-configuration.xml#L77)) - * `event-type` = event type having triggered the event (defined by [configuration](https://github.com/Meeds-io/analytics/blob/1.0.0-M04/analytics-webapps/src/main/webapp/WEB-INF/conf/analytics/analytics-ui-watchers-configuration.xml#L57)) - * `event-pageX`: horizontal coordinate of Element having triggered the event (defined by [configuration](https://github.com/Meeds-io/analytics/blob/1.0.0-M04/analytics-webapps/src/main/webapp/WEB-INF/conf/analytics/analytics-ui-watchers-configuration.xml#L69)) - * `event-pageY`: vertical coordinate of Element having triggered the event (defined by [configuration](https://github.com/Meeds-io/analytics/blob/1.0.0-M04/analytics-webapps/src/main/webapp/WEB-INF/conf/analytics/analytics-ui-watchers-configuration.xml#L63)) - * For **DOM event** *Hamburger menu icon* : - * `watcher` = `Hamburger menu icon` (defined by [configuration](https://github.com/Meeds-io/analytics/blob/1.0.0-M04/analytics-webapps/src/main/webapp/WEB-INF/conf/analytics/analytics-ui-watchers-configuration.xml#L437)) - * `ui` = `toolbar` (defined by [configuration](https://github.com/Meeds-io/analytics/blob/1.0.0-M04/analytics-webapps/src/main/webapp/WEB-INF/conf/analytics/analytics-ui-watchers-configuration.xml#L460)) - * `application` = `HamburgerNavigationMenuLink` (defined by [configuration](https://github.com/Meeds-io/analytics/blob/1.0.0-M04/analytics-webapps/src/main/webapp/WEB-INF/conf/analytics/analytics-ui-watchers-configuration.xml#L468)) - * `operation` = `click` (defined by [configuration](https://github.com/Meeds-io/analytics/blob/1.0.0-M04/analytics-webapps/src/main/webapp/WEB-INF/conf/analytics/analytics-ui-watchers-configuration.xml#L450)) - * `event-type` = event type having triggered the event (defined by [configuration](https://github.com/Meeds-io/analytics/blob/1.0.0-M04/analytics-webapps/src/main/webapp/WEB-INF/conf/analytics/analytics-ui-watchers-configuration.xml#L485)) - * `event-pageX`: horizontal coordinate of Element having triggered the event (defined by [configuration](https://github.com/Meeds-io/analytics/blob/1.0.0-M04/analytics-webapps/src/main/webapp/WEB-INF/conf/analytics/analytics-ui-watchers-configuration.xml#L488)) - * `event-pageY`: vertical coordinate of Element having triggered the event (defined by [configuration](https://github.com/Meeds-io/analytics/blob/1.0.0-M04/analytics-webapps/src/main/webapp/WEB-INF/conf/analytics/analytics-ui-watchers-configuration.xml#L491)) - * For **DOM event** *Hamburger menu site navigation link* : - * `watcher` = `Hamburger menu site navigation link` (defined by [configuration](https://github.com/Meeds-io/analytics/blob/1.0.0-M04/analytics-webapps/src/main/webapp/WEB-INF/conf/analytics/analytics-ui-watchers-configuration.xml#L95)) - * `ui` = `HamburgerMenu` (defined by [configuration](https://github.com/Meeds-io/analytics/blob/1.0.0-M04/analytics-webapps/src/main/webapp/WEB-INF/conf/analytics/analytics-ui-watchers-configuration.xml#L109)) - * `application` = `SiteHamburgerNavigation` (defined by [configuration](https://github.com/Meeds-io/analytics/blob/1.0.0-M04/analytics-webapps/src/main/webapp/WEB-INF/conf/analytics/analytics-ui-watchers-configuration.xml#L117)) - * `operation` = `click` (defined by [configuration](https://github.com/Meeds-io/analytics/blob/1.0.0-M04/analytics-webapps/src/main/webapp/WEB-INF/conf/analytics/analytics-ui-watchers-configuration.xml#L99)) - * `dom-href` = link of target page (defined by [configuration](https://github.com/Meeds-io/analytics/blob/1.0.0-M04/analytics-webapps/src/main/webapp/WEB-INF/conf/analytics/analytics-ui-watchers-configuration.xml#L148)) - * `dom-data-aria-selected` = DOM aria-selected data value, used to determine whether the element is current page or not. (defined by [configuration](https://github.com/Meeds-io/analytics/blob/1.0.0-M04/analytics-webapps/src/main/webapp/WEB-INF/conf/analytics/analytics-ui-watchers-configuration.xml#L151)) - * `event-type` = event type having triggered the event (defined by [configuration](https://github.com/Meeds-io/analytics/blob/1.0.0-M04/analytics-webapps/src/main/webapp/WEB-INF/conf/analytics/analytics-ui-watchers-configuration.xml#L134)) - * `event-pageX`: horizontal coordinate of Element having triggered the event (defined by [configuration](https://github.com/Meeds-io/analytics/blob/1.0.0-M04/analytics-webapps/src/main/webapp/WEB-INF/conf/analytics/analytics-ui-watchers-configuration.xml#L137)) - * `event-pageY`: vertical coordinate of Element having triggered the event (defined by [configuration](https://github.com/Meeds-io/analytics/blob/1.0.0-M04/analytics-webapps/src/main/webapp/WEB-INF/conf/analytics/analytics-ui-watchers-configuration.xml#L140)) - * For **DOM event** *Hamburger menu spaces navigation link* : - * `watcher` = `Hamburger menu spaces navigation link` (defined by [configuration](https://github.com/Meeds-io/analytics/blob/1.0.0-M04/analytics-webapps/src/main/webapp/WEB-INF/conf/analytics/analytics-ui-watchers-configuration.xml#L169)) - * `ui` = `HamburgerMenu` (defined by [configuration](https://github.com/Meeds-io/analytics/blob/1.0.0-M04/analytics-webapps/src/main/webapp/WEB-INF/conf/analytics/analytics-ui-watchers-configuration.xml#L183)) - * `application` = `SpacesHamburgerNavigation` (defined by [configuration](https://github.com/Meeds-io/analytics/blob/1.0.0-M04/analytics-webapps/src/main/webapp/WEB-INF/conf/analytics/analytics-ui-watchers-configuration.xml#L191)) - * `operation` = `click` (defined by [configuration](https://github.com/Meeds-io/analytics/blob/1.0.0-M04/analytics-webapps/src/main/webapp/WEB-INF/conf/analytics/analytics-ui-watchers-configuration.xml#L173)) - * `dom-href` = link of target page (defined by [configuration](https://github.com/Meeds-io/analytics/blob/1.0.0-M04/analytics-webapps/src/main/webapp/WEB-INF/conf/analytics/analytics-ui-watchers-configuration.xml#L222)) - * `dom-data-aria-selected` = DOM aria-selected data value, used to determine whether the element is current page or not. (defined by [configuration](https://github.com/Meeds-io/analytics/blob/1.0.0-M04/analytics-webapps/src/main/webapp/WEB-INF/conf/analytics/analytics-ui-watchers-configuration.xml#L225)) - * `event-type` = event type having triggered the event (defined by [configuration](https://github.com/Meeds-io/analytics/blob/1.0.0-M04/analytics-webapps/src/main/webapp/WEB-INF/conf/analytics/analytics-ui-watchers-configuration.xml#L208)) - * `event-pageX`: horizontal coordinate of Element having triggered the event (defined by [configuration](https://github.com/Meeds-io/analytics/blob/1.0.0-M04/analytics-webapps/src/main/webapp/WEB-INF/conf/analytics/analytics-ui-watchers-configuration.xml#L211)) - * `event-pageY`: vertical coordinate of Element having triggered the event (defined by [configuration](https://github.com/Meeds-io/analytics/blob/1.0.0-M04/analytics-webapps/src/main/webapp/WEB-INF/conf/analytics/analytics-ui-watchers-configuration.xml#L214)) - * For **DOM event** *Activities refresh button* : - * `watcher` = `Activities refresh button` (defined by [configuration](https://github.com/Meeds-io/analytics/blob/1.0.0-M04/analytics-webapps/src/main/webapp/WEB-INF/conf/analytics/analytics-ui-watchers-configuration.xml#L243)) - * `ui` = `ActivityStream` (defined by [configuration](https://github.com/Meeds-io/analytics/blob/1.0.0-M04/analytics-webapps/src/main/webapp/WEB-INF/conf/analytics/analytics-ui-watchers-configuration.xml#L257)) - * `application` = `uiActivitiesDisplay` (defined by [configuration](https://github.com/Meeds-io/analytics/blob/1.0.0-M04/analytics-webapps/src/main/webapp/WEB-INF/conf/analytics/analytics-ui-watchers-configuration.xml#L265)) - * `operation` = `click` (defined by [configuration](https://github.com/Meeds-io/analytics/blob/1.0.0-M04/analytics-webapps/src/main/webapp/WEB-INF/conf/analytics/analytics-ui-watchers-configuration.xml#L247)) - * `event-type` = event type having triggered the event (defined by [configuration](https://github.com/Meeds-io/analytics/blob/1.0.0-M04/analytics-webapps/src/main/webapp/WEB-INF/conf/analytics/analytics-ui-watchers-configuration.xml#L282)) - * For **DOM event** *Post activity using Composer Drawer* : - * `watcher` = `Post activity using Composer Drawer` (defined by [configuration](https://github.com/Meeds-io/analytics/blob/1.0.0-M04/analytics-webapps/src/main/webapp/WEB-INF/conf/analytics/analytics-ui-watchers-configuration.xml#L300)) - * `ui` = `ActivityStream` (defined by [configuration](https://github.com/Meeds-io/analytics/blob/1.0.0-M04/analytics-webapps/src/main/webapp/WEB-INF/conf/analytics/analytics-ui-watchers-configuration.xml#L314)) - * `component` = `composer` (defined by [configuration](https://github.com/Meeds-io/analytics/blob/1.0.0-M04/analytics-webapps/src/main/webapp/WEB-INF/conf/analytics/analytics-ui-watchers-configuration.xml#L330)) - * `application` = `ActivityComposerDrawer` (defined by [configuration](https://github.com/Meeds-io/analytics/blob/1.0.0-M04/analytics-webapps/src/main/webapp/WEB-INF/conf/analytics/analytics-ui-watchers-configuration.xml#L322)) - * `operation` = `click` (defined by [configuration](https://github.com/Meeds-io/analytics/blob/1.0.0-M04/analytics-webapps/src/main/webapp/WEB-INF/conf/analytics/analytics-ui-watchers-configuration.xml#L304)) - * `event-type` = event type having triggered the event (defined by [configuration](https://github.com/Meeds-io/analytics/blob/1.0.0-M04/analytics-webapps/src/main/webapp/WEB-INF/conf/analytics/analytics-ui-watchers-configuration.xml#L347)) - * `dom-disabled`: DOM value of attribute `disabled` (defined by [configuration](https://github.com/Meeds-io/analytics/blob/1.0.0-M04/analytics-webapps/src/main/webapp/WEB-INF/conf/analytics/analytics-ui-watchers-configuration.xml#L355)) - * For **DOM event** *Post activity using legacy Composer* : - * `watcher` = `Post activity using legacy Composer` (defined by [configuration](https://github.com/Meeds-io/analytics/blob/1.0.0-M04/analytics-webapps/src/main/webapp/WEB-INF/conf/analytics/analytics-ui-watchers-configuration.xml#L373)) - * `ui` = `ActivityStream` (defined by [configuration](https://github.com/Meeds-io/analytics/blob/1.0.0-M04/analytics-webapps/src/main/webapp/WEB-INF/conf/analytics/analytics-ui-watchers-configuration.xml#L384)) - * `component` = `composer` (defined by [configuration](https://github.com/Meeds-io/analytics/blob/1.0.0-M04/analytics-webapps/src/main/webapp/WEB-INF/conf/analytics/analytics-ui-watchers-configuration.xml#L330)) - * `application` = `EmbeddedActivityComposer` (defined by [configuration](https://github.com/Meeds-io/analytics/blob/1.0.0-M04/analytics-webapps/src/main/webapp/WEB-INF/conf/analytics/analytics-ui-watchers-configuration.xml#L395)) - * `operation` = `click` (defined by [configuration](https://github.com/Meeds-io/analytics/blob/1.0.0-M04/analytics-webapps/src/main/webapp/WEB-INF/conf/analytics/analytics-ui-watchers-configuration.xml#L377)) - * `event-type` = event type having triggered the event (defined by [configuration](https://github.com/Meeds-io/analytics/blob/1.0.0-M04/analytics-webapps/src/main/webapp/WEB-INF/conf/analytics/analytics-ui-watchers-configuration.xml#L420)) - * `dom-disabled`: DOM value of attribute `disabled` (defined by [configuration](https://github.com/Meeds-io/analytics/blob/1.0.0-M04/analytics-webapps/src/main/webapp/WEB-INF/conf/analytics/analytics-ui-watchers-configuration.xml#L428)) - - -You can consult the data used to display analytics chart using `View samples` button on each portlet. -By viewing samples, you will be able to know what `fields` to use in settings to : -* choose X Axis field (date by default) - * Per day - * Per week - * Per month - * Per quarter - * Per year - * Per custom field : you can choose a custom field from the selectbox: - for now, the fields are retrieved without name, just its technical name that you can find in displayed `statistic samples/data` in drawer. - For example, if you choose `spaceId`, then in the X Axis, you will get the space name. -* choose Y Axis: - * *count* of `statistic samples/data` per choosen X Axis field (date period or custom field) - * *count samples with distinct fields*. - For example, to get the number logins per 'unique users' per day, you will choose `Per day` as X Axis and this option as Y Axis. - * *sum of* `statistic samples/data numeric field` : this will display in Y Axis the sum of a specific `Numeric` field. - * *average of* `statistic samples/data numeric field` : this will display in Y Axis the average rate of a specific `Numeric` field. -* Data filters : a list of conditions on to filter on used data in computing results -* Multiple charts : - * When enabled, this will display multiple charts in one single. - -# Examples of settings - -For example, if you want to display the number of created activities per day and per space (pay attention to the number of spaces). - -ou can choose the following parameters: - * X Axis = `Per day` - * Y Axis = `count samples` - * Multiple charts = true, in addition choose field `spaceId` (a chart will be retrieved per spaceId) - * Data filters : - ** `operation` (`equals`) `createActivity` - ** `spaceId` (`greater`) `1` (or if you have a lot of spaces, you will have to choose a set of `spaceId`, like : - `spaceId` (`in set`) `1,2,3,895`) - -# DOM Events configuration plugin - -In order to add new DOM Event watcher, you can add a [component-plugin](https://docs.exoplatform.org/en/5.3/eXo_Kernel.html#external-plugin) to inject into Service `org.exoplatform.analytics.api.service.AnalyticsService`. (You can follow some example from [here](https://github.com/Meeds-io/analytics/blob/1.0.0-M04/analytics-webapps/src/main/webapp/WEB-INF/conf/analytics/analytics-ui-watchers-configuration.xml)) - -The DOM statistic data are collected in a sync way, but sent to server in async way in a dedicated Process using [Cometd Worker](https://docs.cometd.org/current3/reference/#_javascript_configure) (used parameter: **`useWorkerScheduler`**) - -To understand more about properties of plugin, please read the commented description in the following example: -```xml - - Home page link - addUIWatcherPlugin - org.exoplatform.analytics.api.service.StatisticUIWatcherPlugin - - - watcher - - - - #brandingTopBar a - - - - mousedown - - - - Home page link - - - - click - - - - - - - ui - - - toolbar - - - - - application - - - NavigationToolbarPortlet - - - - - - - - - type - - - pageX - - - pageY - - - - - - - - href - - - class - - - id - - - - - - - -``` \ No newline at end of file +| analytics.admin.permission | NO | *:/platform/analytics | Group of users that can modify Charts application settings. All other users, *even members of /platform/administrators* will not be able to modify charts settings | +| analytics.viewall.permission | NO | *:/platform/administrators | Group of users that can consult all data in Graphs. | +| analytics.view.permission | NO | *:/platform/users | Group of users that can consult their personal and their spaces Graphs. | +| analytics.aggregation.terms.doc_size | NO | 200 | Limit of number of resturned documents in aggregations result of type 'terms' (not used for aggregations of type : sum, avg, date_histogram, histogram and cardinality) | +| analytics.es.index.server.url | NO | Same as used for Meeds | Elasticsearch server URL used for indexing and searching analytics content | +| analytics.es.index.server.username | NO | Same as used for Meeds | Elasticsearch server username used for indexing and searching analytics content | +| analytics.es.index.server.password | NO | Same as used for Meeds | Elasticsearch server password used for indexing and searching analytics content | +| analytics.es.index.writePeriod | YES | 7 | Number of days to keep an Analytics index available for Read/Write mode. Once the period is passed, another index is created with a different suffix. | +| analytics.queue.MaxNodes | NO | 2000 | Number of maximum entries in in-memory cached Analytics Queue that is processed each 10 seconds | +| analytics.queue.TimeToLive | NO | -1 | lifetime of entries in Analytics Queue. Default: infinite. | diff --git a/analytics-api/pom.xml b/analytics-api/pom.xml index 03ee4677b..cefd57f8c 100644 --- a/analytics-api/pom.xml +++ b/analytics-api/pom.xml @@ -1,16 +1,19 @@ 4.0.0 - org.exoplatform.addons.analytics + io.meeds.analytics analytics-parent 7.0.x-exo-SNAPSHOT analytics-listeners - eXo Analytics - Listeners + Meeds:: Analytics - Listeners 0 diff --git a/analytics-listeners/src/main/java/io/meeds/analytics/job/SpacesStatisticsCountJob.java b/analytics-listeners/src/main/java/io/meeds/analytics/job/SpacesStatisticsCountJob.java new file mode 100644 index 000000000..749c5adc4 --- /dev/null +++ b/analytics-listeners/src/main/java/io/meeds/analytics/job/SpacesStatisticsCountJob.java @@ -0,0 +1,63 @@ +/** + * This file is part of the Meeds project (https://meeds.io/). + * + * Copyright (C) 2020 - 2024 Meeds Association contact@meeds.io + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 3 of the License, or (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public License + * along with this program; if not, write to the Free Software Foundation, + * Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + */ +package io.meeds.analytics.job; + +import java.util.concurrent.TimeUnit; + +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.context.annotation.Configuration; +import org.springframework.scheduling.annotation.EnableScheduling; +import org.springframework.scheduling.annotation.Scheduled; + +import org.exoplatform.social.core.space.spi.SpaceService; + +import io.meeds.analytics.model.StatisticData; +import io.meeds.analytics.utils.AnalyticsUtils; +import io.meeds.common.ContainerTransactional; + +import lombok.SneakyThrows; + +/** + * A job to collect statistics of users count + */ +@Configuration +@EnableScheduling +public class SpacesStatisticsCountJob { + + @Autowired + private SpaceService spaceService; + + @SneakyThrows + @ContainerTransactional + @Scheduled(initialDelay = 2, fixedDelay = 180, timeUnit = TimeUnit.MINUTES) + public void run() { + long startTime = System.currentTimeMillis(); + int allSpacesCount = spaceService.getAllSpacesWithListAccess().getSize(); + StatisticData statisticData = new StatisticData(); + statisticData.setModule("social"); + statisticData.setSubModule("space"); + statisticData.setOperation("spacesCount"); + statisticData.setDuration(System.currentTimeMillis() - startTime); + statisticData.addParameter("countType", "allSpaces"); + statisticData.addParameter("count", allSpacesCount); + AnalyticsUtils.addStatisticData(statisticData); + } + +} diff --git a/analytics-listeners/src/main/java/io/meeds/analytics/job/UsersStatisticsCountJob.java b/analytics-listeners/src/main/java/io/meeds/analytics/job/UsersStatisticsCountJob.java new file mode 100644 index 000000000..3ed634e0c --- /dev/null +++ b/analytics-listeners/src/main/java/io/meeds/analytics/job/UsersStatisticsCountJob.java @@ -0,0 +1,100 @@ +/** + * This file is part of the Meeds project (https://meeds.io/). + * + * Copyright (C) 2020 - 2024 Meeds Association contact@meeds.io + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 3 of the License, or (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public License + * along with this program; if not, write to the Free Software Foundation, + * Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + */ +package io.meeds.analytics.job; + +import java.util.concurrent.TimeUnit; + +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.context.annotation.Configuration; +import org.springframework.scheduling.annotation.EnableScheduling; +import org.springframework.scheduling.annotation.Scheduled; + +import org.exoplatform.commons.utils.ListAccess; +import org.exoplatform.services.organization.Group; +import org.exoplatform.services.organization.Membership; +import org.exoplatform.services.organization.OrganizationService; +import org.exoplatform.services.organization.User; +import org.exoplatform.services.organization.UserStatus; +import org.exoplatform.social.core.identity.model.Identity; +import org.exoplatform.social.core.identity.provider.OrganizationIdentityProvider; +import org.exoplatform.social.core.manager.IdentityManager; + +import io.meeds.analytics.model.StatisticData; +import io.meeds.analytics.utils.AnalyticsUtils; +import io.meeds.common.ContainerTransactional; + +import lombok.SneakyThrows; + +/** + * A job to collect statistics of users count + */ +@Configuration +@EnableScheduling +public class UsersStatisticsCountJob { + + @Autowired + private OrganizationService organizationService; + + @Autowired + private IdentityManager identityManager; + + @SneakyThrows + @ContainerTransactional + @Scheduled(initialDelay = 2, fixedDelay = 180, timeUnit = TimeUnit.MINUTES) + public void run() { + long startTime = System.currentTimeMillis(); + + ListAccess allUsers = organizationService.getUserHandler().findAllUsers(UserStatus.ANY); + int allUsersCount = allUsers.getSize(); + + ListAccess enabledIdentities = identityManager.getIdentitiesByProfileFilter(OrganizationIdentityProvider.NAME, + null, + false); + int enabledUsersCount = enabledIdentities.getSize(); + int disabledUsersCount = allUsersCount - enabledUsersCount; + + addUsersCountStatistic("allUsers", allUsersCount, startTime); + addUsersCountStatistic("enabledUsers", enabledUsersCount, startTime); + addUsersCountStatistic("disabledUsers", disabledUsersCount, startTime); + + startTime = System.currentTimeMillis(); + Group externalsGroup = organizationService.getGroupHandler().findGroupById("/platform/externals"); + int enabledExternalUsersCount = 0; + if (externalsGroup != null) { + ListAccess externalMemberships = organizationService.getMembershipHandler() + .findAllMembershipsByGroup(externalsGroup); + enabledExternalUsersCount = externalMemberships.getSize(); + } + addUsersCountStatistic("enabledExternalUsers", enabledExternalUsersCount, startTime); + addUsersCountStatistic("enabledInternalUsers", (enabledUsersCount - enabledExternalUsersCount), startTime); + } + + private void addUsersCountStatistic(String countType, int count, long startTime) { + StatisticData statisticData = new StatisticData(); + statisticData.setModule("portal"); + statisticData.setSubModule("account"); + statisticData.setOperation("usersCount"); + statisticData.setDuration(System.currentTimeMillis() - startTime); + statisticData.addParameter("countType", countType); + statisticData.addParameter("count", count); + AnalyticsUtils.addStatisticData(statisticData); + } + +} diff --git a/analytics-listeners/src/main/java/org/exoplatform/analytics/listener/portal/LoginAnalyticsListener.java b/analytics-listeners/src/main/java/io/meeds/analytics/listener/portal/LoginAnalyticsListener.java similarity index 66% rename from analytics-listeners/src/main/java/org/exoplatform/analytics/listener/portal/LoginAnalyticsListener.java rename to analytics-listeners/src/main/java/io/meeds/analytics/listener/portal/LoginAnalyticsListener.java index 654bfe9d8..53aa21ed6 100644 --- a/analytics-listeners/src/main/java/org/exoplatform/analytics/listener/portal/LoginAnalyticsListener.java +++ b/analytics-listeners/src/main/java/io/meeds/analytics/listener/portal/LoginAnalyticsListener.java @@ -1,36 +1,60 @@ /** * This file is part of the Meeds project (https://meeds.io/). - * Copyright (C) 2022 Meeds Association - * contact@meeds.io + * + * Copyright (C) 2020 - 2024 Meeds Association contact@meeds.io + * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU Lesser General Public * License as published by the Free Software Foundation; either * version 3 of the License, or (at your option) any later version. + * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU * Lesser General Public License for more details. + * * You should have received a copy of the GNU Lesser General Public License * along with this program; if not, write to the Free Software Foundation, * Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. */ -package org.exoplatform.analytics.listener.portal; +package io.meeds.analytics.listener.portal; -import static org.exoplatform.analytics.utils.AnalyticsUtils.addStatisticData; -import static org.exoplatform.analytics.utils.AnalyticsUtils.getUserIdentityId; +import static io.meeds.analytics.utils.AnalyticsUtils.addStatisticData; +import static io.meeds.analytics.utils.AnalyticsUtils.getUserIdentityId; + +import java.util.Arrays; +import java.util.List; import org.apache.commons.lang3.StringUtils; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.stereotype.Component; -import org.exoplatform.analytics.model.StatisticData; import org.exoplatform.services.listener.*; import org.exoplatform.services.log.ExoLogger; import org.exoplatform.services.log.Log; import org.exoplatform.services.security.ConversationRegistry; import org.exoplatform.services.security.ConversationState; +import io.meeds.analytics.model.StatisticData; + +import jakarta.annotation.PostConstruct; + @Asynchronous +@Component public class LoginAnalyticsListener extends Listener { - private static final Log LOG = ExoLogger.getLogger(LoginAnalyticsListener.class); + + private static final Log LOG = ExoLogger.getLogger(LoginAnalyticsListener.class); + + private static final List EVENT_NAMES = Arrays.asList("exo.core.security.ConversationRegistry.register", + "exo.core.security.ConversationRegistry.unregister"); + + @Autowired + private ListenerService listenerService; + + @PostConstruct + public void init() { + EVENT_NAMES.forEach(name -> listenerService.addListener(name, this)); + } @Override public void onEvent(Event event) throws Exception { diff --git a/analytics-listeners/src/main/java/org/exoplatform/analytics/listener/portal/LoginFailedAnalyticsListener.java b/analytics-listeners/src/main/java/io/meeds/analytics/listener/portal/LoginFailedAnalyticsListener.java similarity index 64% rename from analytics-listeners/src/main/java/org/exoplatform/analytics/listener/portal/LoginFailedAnalyticsListener.java rename to analytics-listeners/src/main/java/io/meeds/analytics/listener/portal/LoginFailedAnalyticsListener.java index 4ed2ad14f..1a54327d0 100644 --- a/analytics-listeners/src/main/java/org/exoplatform/analytics/listener/portal/LoginFailedAnalyticsListener.java +++ b/analytics-listeners/src/main/java/io/meeds/analytics/listener/portal/LoginFailedAnalyticsListener.java @@ -1,35 +1,58 @@ /** * This file is part of the Meeds project (https://meeds.io/). - * Copyright (C) 2022 Meeds Association - * contact@meeds.io + * + * Copyright (C) 2020 - 2024 Meeds Association contact@meeds.io + * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU Lesser General Public * License as published by the Free Software Foundation; either * version 3 of the License, or (at your option) any later version. + * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU * Lesser General Public License for more details. + * * You should have received a copy of the GNU Lesser General Public License * along with this program; if not, write to the Free Software Foundation, * Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. */ -package org.exoplatform.analytics.listener.portal; +package io.meeds.analytics.listener.portal; -import static org.exoplatform.analytics.utils.AnalyticsUtils.addStatisticData; +import static io.meeds.analytics.utils.AnalyticsUtils.addStatisticData; +import java.util.Arrays; +import java.util.List; import java.util.Map; -import org.exoplatform.analytics.model.StatisticData; -import org.exoplatform.analytics.utils.AnalyticsUtils; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.stereotype.Component; + import org.exoplatform.commons.api.persistence.ExoTransactional; import org.exoplatform.services.listener.Asynchronous; import org.exoplatform.services.listener.Event; import org.exoplatform.services.listener.Listener; +import org.exoplatform.services.listener.ListenerService; + +import io.meeds.analytics.model.StatisticData; +import io.meeds.analytics.utils.AnalyticsUtils; + +import jakarta.annotation.PostConstruct; @Asynchronous +@Component public class LoginFailedAnalyticsListener extends Listener> { + private static final List EVENT_NAMES = Arrays.asList("login.failed"); + + @Autowired + private ListenerService listenerService; + + @PostConstruct + public void init() { + EVENT_NAMES.forEach(name -> listenerService.addListener(name, this)); + } + @Override @ExoTransactional public void onEvent(Event> event) throws Exception { @@ -40,8 +63,8 @@ public void onEvent(Event> event) throws Exception { statisticData.setSubModule("login"); statisticData.setOperation("login"); statisticData.setUserId(AnalyticsUtils.getUserIdentityId(data.get("user_id"))); - statisticData.setStatus(data.get("status").equals("ko") ? StatisticData.StatisticStatus.KO - : StatisticData.StatisticStatus.OK); + statisticData.setStatus(data.get("status").equals("ko") ? StatisticData.StatisticStatus.KO : + StatisticData.StatisticStatus.OK); statisticData.addParameter("reason", data.get("reason")); addStatisticData(statisticData); } diff --git a/analytics-listeners/src/main/java/org/exoplatform/analytics/listener/portal/PageAccessListener.java b/analytics-listeners/src/main/java/io/meeds/analytics/listener/portal/PageAccessListener.java similarity index 84% rename from analytics-listeners/src/main/java/org/exoplatform/analytics/listener/portal/PageAccessListener.java rename to analytics-listeners/src/main/java/io/meeds/analytics/listener/portal/PageAccessListener.java index c891cedaa..2f3afe94f 100644 --- a/analytics-listeners/src/main/java/org/exoplatform/analytics/listener/portal/PageAccessListener.java +++ b/analytics-listeners/src/main/java/io/meeds/analytics/listener/portal/PageAccessListener.java @@ -1,30 +1,35 @@ /** * This file is part of the Meeds project (https://meeds.io/). - * Copyright (C) 2022 Meeds Association - * contact@meeds.io + * + * Copyright (C) 2020 - 2024 Meeds Association contact@meeds.io + * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU Lesser General Public * License as published by the Free Software Foundation; either * version 3 of the License, or (at your option) any later version. + * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU * Lesser General Public License for more details. + * * You should have received a copy of the GNU Lesser General Public License * along with this program; if not, write to the Free Software Foundation, * Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. */ -package org.exoplatform.analytics.listener.portal; - -import static org.exoplatform.analytics.utils.AnalyticsUtils.*; +package io.meeds.analytics.listener.portal; +import jakarta.annotation.PostConstruct; import jakarta.servlet.http.HttpServletRequest; import jakarta.servlet.http.HttpServletResponse; -import org.exoplatform.analytics.model.StatisticData; -import org.exoplatform.analytics.model.StatisticData.StatisticStatus; +import static io.meeds.analytics.utils.AnalyticsUtils.*; + +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.beans.factory.annotation.Value; +import org.springframework.stereotype.Component; + import org.exoplatform.container.component.BaseComponentPlugin; -import org.exoplatform.container.xml.InitParams; import org.exoplatform.portal.application.PortalRequestContext; import org.exoplatform.portal.mop.user.UserNode; import org.exoplatform.portal.webui.portal.UIPortal; @@ -36,18 +41,25 @@ import org.exoplatform.web.application.*; import org.exoplatform.webui.application.WebuiRequestContext; +import io.meeds.analytics.model.StatisticData; +import io.meeds.analytics.model.StatisticData.StatisticStatus; + +@Component public class PageAccessListener extends BaseComponentPlugin implements ApplicationLifecycle { - private static final Log LOG = ExoLogger.getLogger(PageAccessListener.class); + private static final Log LOG = ExoLogger.getLogger(PageAccessListener.class); - private ThreadLocal operationStartTime = new ThreadLocal<>(); + @Autowired + private ApplicationLifecycleExtension applicationLifecycleExtension; - private boolean collectAjaxQueries = false; + @Value("${analytics.collectAjaxQueries:false}") + private boolean collectAjaxQueries = false; - public PageAccessListener(InitParams params) { - if (params != null && params.containsKey("collectAjaxQueries")) { - this.collectAjaxQueries = Boolean.parseBoolean(params.getValueParam("collectAjaxQueries").getValue()); - } + private ThreadLocal operationStartTime = new ThreadLocal<>(); + + @PostConstruct + public void init() { + applicationLifecycleExtension.addPortalApplicationLifecycle(this); } @Override diff --git a/analytics-listeners/src/main/java/org/exoplatform/analytics/listener/portal/UserAnalyticsEventListener.java b/analytics-listeners/src/main/java/io/meeds/analytics/listener/portal/UserAnalyticsEventListener.java similarity index 69% rename from analytics-listeners/src/main/java/org/exoplatform/analytics/listener/portal/UserAnalyticsEventListener.java rename to analytics-listeners/src/main/java/io/meeds/analytics/listener/portal/UserAnalyticsEventListener.java index 62c303b2e..bd1770ae9 100644 --- a/analytics-listeners/src/main/java/org/exoplatform/analytics/listener/portal/UserAnalyticsEventListener.java +++ b/analytics-listeners/src/main/java/io/meeds/analytics/listener/portal/UserAnalyticsEventListener.java @@ -1,70 +1,94 @@ /** * This file is part of the Meeds project (https://meeds.io/). - * Copyright (C) 2022 Meeds Association - * contact@meeds.io + * + * Copyright (C) 2020 - 2024 Meeds Association contact@meeds.io + * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU Lesser General Public * License as published by the Free Software Foundation; either * version 3 of the License, or (at your option) any later version. + * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU * Lesser General Public License for more details. + * * You should have received a copy of the GNU Lesser General Public License * along with this program; if not, write to the Free Software Foundation, * Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. */ -package org.exoplatform.analytics.listener.portal; +package io.meeds.analytics.listener.portal; -import static org.exoplatform.analytics.utils.AnalyticsUtils.*; +import static io.meeds.analytics.utils.AnalyticsUtils.FIELD_SOCIAL_IDENTITY_ID; +import static io.meeds.analytics.utils.AnalyticsUtils.addStatisticData; +import static io.meeds.analytics.utils.AnalyticsUtils.getCurrentUserIdentityId; +import static io.meeds.analytics.utils.AnalyticsUtils.getUserIdentityId; + +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.stereotype.Component; -import org.exoplatform.analytics.model.StatisticData; -import org.exoplatform.commons.api.persistence.ExoTransactional; import org.exoplatform.services.log.ExoLogger; import org.exoplatform.services.log.Log; +import org.exoplatform.services.organization.OrganizationService; import org.exoplatform.services.organization.User; import org.exoplatform.services.organization.UserEventListener; +import io.meeds.analytics.model.StatisticData; +import io.meeds.common.ContainerTransactional; + +import jakarta.annotation.PostConstruct; +import lombok.SneakyThrows; + +@Component public class UserAnalyticsEventListener extends UserEventListener { - private static final Log LOG = ExoLogger.getLogger(UserAnalyticsEventListener.class); + private static final Log LOG = ExoLogger.getLogger(UserAnalyticsEventListener.class); - private ThreadLocal operationStartTime = new ThreadLocal<>(); + private ThreadLocal operationStartTime = new ThreadLocal<>(); + + @Autowired + private OrganizationService organizationService; + + @SneakyThrows + @PostConstruct + public void init() { + organizationService.addListenerPlugin(this); + } @Override - @ExoTransactional + @ContainerTransactional public void preSave(User user, boolean isNew) throws Exception { operationStartTime.set(System.currentTimeMillis()); } @Override - @ExoTransactional + @ContainerTransactional public void preSetEnabled(User user) throws Exception { operationStartTime.set(System.currentTimeMillis()); } @Override - @ExoTransactional + @ContainerTransactional public void preDelete(User user) throws Exception { operationStartTime.set(System.currentTimeMillis()); } @Override - @ExoTransactional + @ContainerTransactional public void postSave(User user, boolean isNew) throws Exception { StatisticData statisticData = buildStatisticData(isNew ? "createUser" : "saveUser", user); addStatisticData(statisticData); } @Override - @ExoTransactional + @ContainerTransactional public void postSetEnabled(User user) throws Exception { StatisticData statisticData = buildStatisticData("enableUser", user); addStatisticData(statisticData); } @Override - @ExoTransactional + @ContainerTransactional public void postDelete(User user) throws Exception { StatisticData statisticData = buildStatisticData("deleteUser", user); addStatisticData(statisticData); diff --git a/analytics-listeners/src/main/java/org/exoplatform/analytics/listener/social/ActivityAttachmentAnalyticsListener.java b/analytics-listeners/src/main/java/io/meeds/analytics/listener/social/ActivityAttachmentAnalyticsListener.java similarity index 58% rename from analytics-listeners/src/main/java/org/exoplatform/analytics/listener/social/ActivityAttachmentAnalyticsListener.java rename to analytics-listeners/src/main/java/io/meeds/analytics/listener/social/ActivityAttachmentAnalyticsListener.java index efdaea73e..22a9104ba 100644 --- a/analytics-listeners/src/main/java/org/exoplatform/analytics/listener/social/ActivityAttachmentAnalyticsListener.java +++ b/analytics-listeners/src/main/java/io/meeds/analytics/listener/social/ActivityAttachmentAnalyticsListener.java @@ -1,45 +1,71 @@ -/* +/** * This file is part of the Meeds project (https://meeds.io/). * - * Copyright (C) 2023 Meeds Association contact@meeds.io + * Copyright (C) 2020 - 2024 Meeds Association contact@meeds.io * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU Lesser General Public * License as published by the Free Software Foundation; either * version 3 of the License, or (at your option) any later version. + * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU * Lesser General Public License for more details. - * + * * You should have received a copy of the GNU Lesser General Public License * along with this program; if not, write to the Free Software Foundation, * Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. */ -package org.exoplatform.analytics.listener.social; +package io.meeds.analytics.listener.social; + +import static io.meeds.analytics.utils.AnalyticsUtils.addActivityStatisticsData; -import static org.exoplatform.analytics.utils.AnalyticsUtils.addActivityStatisticsData; +import java.util.Arrays; +import java.util.Collections; +import java.util.List; + +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.stereotype.Component; -import org.exoplatform.analytics.model.StatisticData; -import org.exoplatform.container.xml.InitParams; import org.exoplatform.services.listener.Asynchronous; +import org.exoplatform.services.listener.ListenerService; import org.exoplatform.social.attachment.AttachmentService; import org.exoplatform.social.attachment.model.ObjectAttachmentId; import org.exoplatform.social.core.activity.model.ExoSocialActivity; import org.exoplatform.social.core.manager.ActivityManager; import org.exoplatform.social.core.space.spi.SpaceService; +import io.meeds.analytics.model.StatisticData; + +import jakarta.annotation.PostConstruct; +import lombok.Getter; + @Asynchronous +@Component public class ActivityAttachmentAnalyticsListener extends BaseAttachmentAnalyticsListener { - private ActivityManager activityManager; + private static final List SUPPORTED_OBJECT_TYPES = Collections.singletonList("activity"); + + private static final List EVENT_NAMES = Arrays.asList("attachment.created", "attachment.deleted"); + + @Getter + @Autowired + private AttachmentService attachmentService; - public ActivityAttachmentAnalyticsListener(AttachmentService attachmentService, - SpaceService spaceService, - ActivityManager activityManager, - InitParams initParam) { - super(attachmentService, spaceService, initParam); - this.activityManager = activityManager; + @Getter + @Autowired + private SpaceService spaceService; + + @Autowired + private ActivityManager activityManager; + + @Autowired + private ListenerService listenerService; + + @PostConstruct + public void init() { + EVENT_NAMES.forEach(name -> listenerService.addListener(name, this)); } @Override @@ -50,6 +76,11 @@ protected void extendStatisticData(StatisticData statisticData, ObjectAttachment } } + @Override + protected List getSupportedObjectType() { + return SUPPORTED_OBJECT_TYPES; + } + @Override protected String getModule(ObjectAttachmentId objectAttachment) { return "social"; diff --git a/analytics-listeners/src/main/java/org/exoplatform/analytics/listener/social/AnalyticsActivityListener.java b/analytics-listeners/src/main/java/io/meeds/analytics/listener/social/AnalyticsActivityListener.java similarity index 87% rename from analytics-listeners/src/main/java/org/exoplatform/analytics/listener/social/AnalyticsActivityListener.java rename to analytics-listeners/src/main/java/io/meeds/analytics/listener/social/AnalyticsActivityListener.java index 8bacd200b..264f76187 100644 --- a/analytics-listeners/src/main/java/org/exoplatform/analytics/listener/social/AnalyticsActivityListener.java +++ b/analytics-listeners/src/main/java/io/meeds/analytics/listener/social/AnalyticsActivityListener.java @@ -1,27 +1,35 @@ /** * This file is part of the Meeds project (https://meeds.io/). - * Copyright (C) 2022 Meeds Association - * contact@meeds.io + * + * Copyright (C) 2020 - 2024 Meeds Association contact@meeds.io + * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU Lesser General Public * License as published by the Free Software Foundation; either * version 3 of the License, or (at your option) any later version. + * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU * Lesser General Public License for more details. + * * You should have received a copy of the GNU Lesser General Public License * along with this program; if not, write to the Free Software Foundation, * Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. */ -package org.exoplatform.analytics.listener.social; +package io.meeds.analytics.listener.social; -import static org.exoplatform.analytics.utils.AnalyticsUtils.*; +import static io.meeds.analytics.utils.AnalyticsUtils.addActivityStatisticsData; +import static io.meeds.analytics.utils.AnalyticsUtils.addSpaceStatistics; +import static io.meeds.analytics.utils.AnalyticsUtils.addStatisticData; +import static io.meeds.analytics.utils.AnalyticsUtils.getCurrentUserIdentityId; +import static io.meeds.analytics.utils.AnalyticsUtils.getIdentity; +import static io.meeds.analytics.utils.AnalyticsUtils.getUserIdentityId; import org.apache.commons.lang3.StringUtils; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.stereotype.Component; -import org.exoplatform.analytics.model.StatisticData; -import org.exoplatform.commons.utils.CommonsUtils; import org.exoplatform.services.log.ExoLogger; import org.exoplatform.services.log.Log; import org.exoplatform.social.core.activity.ActivityLifeCycleEvent; @@ -36,9 +44,25 @@ import org.exoplatform.social.core.space.model.Space; import org.exoplatform.social.core.space.spi.SpaceService; +import io.meeds.analytics.model.StatisticData; + +import jakarta.annotation.PostConstruct; + +@Component public class AnalyticsActivityListener extends ActivityListenerPlugin { - private static final Log LOG = ExoLogger.getLogger(AnalyticsActivityListener.class); + private static final Log LOG = ExoLogger.getLogger(AnalyticsActivityListener.class); + + @Autowired + private ActivityManager activityManager; + + @Autowired + private SpaceService spaceService; + + @PostConstruct + public void init() { + activityManager.addActivityEventListener(this); + } @Override public void saveActivity(ActivityLifeCycleEvent event) { @@ -59,15 +83,17 @@ public void updateActivity(ActivityLifeCycleEvent event) { handleErrorProcessingOperation(event, e); } } + @Override public void deleteActivity(ActivityLifeCycleEvent event) { try { - StatisticData statisticData = addActivityStatisticEvent(event,"deleteActivity"); + StatisticData statisticData = addActivityStatisticEvent(event, "deleteActivity"); addStatisticData(statisticData); } catch (Exception e) { handleErrorProcessingOperation(event, e); } } + @Override public void shareActivity(ActivityLifeCycleEvent event) { try { @@ -97,15 +123,17 @@ public void updateComment(ActivityLifeCycleEvent event) { handleErrorProcessingOperation(event, e); } } + @Override public void deleteComment(ActivityLifeCycleEvent event) { try { - StatisticData statisticData = addActivityStatisticEvent(event,"deleteComment"); + StatisticData statisticData = addActivityStatisticEvent(event, "deleteComment"); addStatisticData(statisticData); } catch (Exception e) { handleErrorProcessingOperation(event, e); } } + @Override public void likeActivity(ActivityLifeCycleEvent event) { try { @@ -163,7 +191,6 @@ private StatisticData addActivityStatisticEvent(ActivityLifeCycleEvent event, St ActivityStream activityStream = activity.getActivityStream(); if ((activityStream == null || activityStream.getType() == null || activityStream.getPrettyId() == null) && StringUtils.isNotBlank(activity.getParentId())) { - ActivityManager activityManager = CommonsUtils.getService(ActivityManager.class); ExoSocialActivity parentActivity = activityManager.getActivity(activity.getParentId()); activityStream = parentActivity.getActivityStream(); } @@ -188,7 +215,6 @@ private StatisticData addActivityStatisticEvent(ActivityLifeCycleEvent event, St StatisticData statisticData = new StatisticData(); if (streamIdentity != null) { if (StringUtils.equals(streamIdentity.getProviderId(), SpaceIdentityProvider.NAME)) { - SpaceService spaceService = CommonsUtils.getService(SpaceService.class); Space space = spaceService.getSpaceByPrettyName(streamIdentity.getRemoteId()); addSpaceStatistics(statisticData, space); } diff --git a/analytics-listeners/src/main/java/org/exoplatform/analytics/listener/social/AnalyticsActivityTagsListener.java b/analytics-listeners/src/main/java/io/meeds/analytics/listener/social/AnalyticsActivityTagsListener.java similarity index 81% rename from analytics-listeners/src/main/java/org/exoplatform/analytics/listener/social/AnalyticsActivityTagsListener.java rename to analytics-listeners/src/main/java/io/meeds/analytics/listener/social/AnalyticsActivityTagsListener.java index 597b9c85e..5f10f9981 100644 --- a/analytics-listeners/src/main/java/org/exoplatform/analytics/listener/social/AnalyticsActivityTagsListener.java +++ b/analytics-listeners/src/main/java/io/meeds/analytics/listener/social/AnalyticsActivityTagsListener.java @@ -1,32 +1,37 @@ /** * This file is part of the Meeds project (https://meeds.io/). - * Copyright (C) 2022 Meeds Association - * contact@meeds.io + * + * Copyright (C) 2020 - 2024 Meeds Association contact@meeds.io + * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU Lesser General Public * License as published by the Free Software Foundation; either * version 3 of the License, or (at your option) any later version. + * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU * Lesser General Public License for more details. + * * You should have received a copy of the GNU Lesser General Public License * along with this program; if not, write to the Free Software Foundation, * Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. */ -package org.exoplatform.analytics.listener.social; +package io.meeds.analytics.listener.social; import java.util.Date; import java.util.Set; import org.apache.commons.lang3.Validate; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.stereotype.Component; import org.apache.commons.lang3.StringUtils; -import org.exoplatform.analytics.model.StatisticData; -import org.exoplatform.analytics.utils.AnalyticsUtils; + import org.exoplatform.commons.api.persistence.ExoTransactional; import org.exoplatform.services.listener.Asynchronous; import org.exoplatform.services.listener.Event; import org.exoplatform.services.listener.Listener; +import org.exoplatform.services.listener.ListenerService; import org.exoplatform.social.core.activity.model.ExoSocialActivity; import org.exoplatform.social.core.activity.model.ExoSocialActivityImpl; import org.exoplatform.social.core.identity.model.Identity; @@ -35,16 +40,29 @@ import org.exoplatform.social.metadata.tag.model.TagName; import org.exoplatform.social.metadata.tag.model.TagObject; +import io.meeds.analytics.model.StatisticData; +import io.meeds.analytics.utils.AnalyticsUtils; + +import jakarta.annotation.PostConstruct; + @Asynchronous +@Component public class AnalyticsActivityTagsListener extends Listener> { - private ActivityManager activityManager; + private static final String EVENT_NAME = "metadata.tag.added"; + + @Autowired + private ActivityManager activityManager; + + @Autowired + private IdentityManager identityManager; - private IdentityManager identityManager; + @Autowired + private ListenerService listenerService; - public AnalyticsActivityTagsListener(ActivityManager activityManager, IdentityManager identityManager) { - this.activityManager = activityManager; - this.identityManager = identityManager; + @PostConstruct + public void init() { + listenerService.addListener(EVENT_NAME, this); } @Override diff --git a/analytics-listeners/src/main/java/org/exoplatform/analytics/listener/social/AnalyticsProfileListener.java b/analytics-listeners/src/main/java/io/meeds/analytics/listener/social/AnalyticsProfileListener.java similarity index 78% rename from analytics-listeners/src/main/java/org/exoplatform/analytics/listener/social/AnalyticsProfileListener.java rename to analytics-listeners/src/main/java/io/meeds/analytics/listener/social/AnalyticsProfileListener.java index a24e249c9..c8e07f55b 100644 --- a/analytics-listeners/src/main/java/org/exoplatform/analytics/listener/social/AnalyticsProfileListener.java +++ b/analytics-listeners/src/main/java/io/meeds/analytics/listener/social/AnalyticsProfileListener.java @@ -1,47 +1,69 @@ /** * This file is part of the Meeds project (https://meeds.io/). - * Copyright (C) 2022 Meeds Association - * contact@meeds.io + * + * Copyright (C) 2020 - 2024 Meeds Association contact@meeds.io + * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU Lesser General Public * License as published by the Free Software Foundation; either * version 3 of the License, or (at your option) any later version. + * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU * Lesser General Public License for more details. + * * You should have received a copy of the GNU Lesser General Public License * along with this program; if not, write to the Free Software Foundation, * Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. */ -package org.exoplatform.analytics.listener.social; +package io.meeds.analytics.listener.social; -import static org.exoplatform.analytics.utils.AnalyticsUtils.AVATAR; -import static org.exoplatform.analytics.utils.AnalyticsUtils.FIELD_SOCIAL_IDENTITY_ID; -import static org.exoplatform.analytics.utils.AnalyticsUtils.IMAGE_SIZE; -import static org.exoplatform.analytics.utils.AnalyticsUtils.IMAGE_TYPE; -import static org.exoplatform.analytics.utils.AnalyticsUtils.addStatisticData; -import static org.exoplatform.analytics.utils.AnalyticsUtils.getIdentity; +import static io.meeds.analytics.utils.AnalyticsUtils.FIELD_SOCIAL_IDENTITY_ID; +import static io.meeds.analytics.utils.AnalyticsUtils.addStatisticData; +import static io.meeds.analytics.utils.AnalyticsUtils.getIdentity; + +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.stereotype.Component; -import org.exoplatform.analytics.model.StatisticData; import org.exoplatform.services.log.ExoLogger; import org.exoplatform.services.log.Log; import org.exoplatform.social.core.identity.model.Identity; import org.exoplatform.social.core.identity.provider.OrganizationIdentityProvider; +import org.exoplatform.social.core.manager.IdentityManager; import org.exoplatform.social.core.model.AvatarAttachment; import org.exoplatform.social.core.profile.ProfileLifeCycleEvent; import org.exoplatform.social.core.profile.ProfileListenerPlugin; +import io.meeds.analytics.model.StatisticData; + +import jakarta.annotation.PostConstruct; + +@Component public class AnalyticsProfileListener extends ProfileListenerPlugin { - private static final Log LOG = ExoLogger.getLogger(AnalyticsProfileListener.class); + public static final String IMAGE_SIZE = "imageSize"; + + public static final String IMAGE_TYPE = "imageType"; + + public static final String AVATAR = "avatar"; + + private static final Log LOG = ExoLogger.getLogger(AnalyticsProfileListener.class); + + @Autowired + private IdentityManager identityManager; + + @PostConstruct + public void init() { + identityManager.registerProfileListener(this); + } @Override public void avatarUpdated(ProfileLifeCycleEvent event) { try { AvatarAttachment avatarAttachment = ((AvatarAttachment) event.getPayload().getProperty(AVATAR)); if (avatarAttachment != null) { - StatisticData statisticData = buildStatisticData("avatar", event.getSource()); + StatisticData statisticData = buildStatisticData(AVATAR, event.getSource()); if (statisticData != null) { statisticData.addParameter(IMAGE_SIZE, avatarAttachment.getSize()); statisticData.addParameter(IMAGE_TYPE, avatarAttachment.getMimeType()); diff --git a/analytics-listeners/src/main/java/org/exoplatform/analytics/listener/social/AnalyticsRelationshipListener.java b/analytics-listeners/src/main/java/io/meeds/analytics/listener/social/AnalyticsRelationshipListener.java similarity index 83% rename from analytics-listeners/src/main/java/org/exoplatform/analytics/listener/social/AnalyticsRelationshipListener.java rename to analytics-listeners/src/main/java/io/meeds/analytics/listener/social/AnalyticsRelationshipListener.java index e8cc5490c..0303b5630 100644 --- a/analytics-listeners/src/main/java/org/exoplatform/analytics/listener/social/AnalyticsRelationshipListener.java +++ b/analytics-listeners/src/main/java/io/meeds/analytics/listener/social/AnalyticsRelationshipListener.java @@ -1,34 +1,54 @@ /** * This file is part of the Meeds project (https://meeds.io/). - * Copyright (C) 2022 Meeds Association - * contact@meeds.io + * + * Copyright (C) 2020 - 2024 Meeds Association contact@meeds.io + * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU Lesser General Public * License as published by the Free Software Foundation; either * version 3 of the License, or (at your option) any later version. + * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU * Lesser General Public License for more details. + * * You should have received a copy of the GNU Lesser General Public License * along with this program; if not, write to the Free Software Foundation, * Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. */ -package org.exoplatform.analytics.listener.social; +package io.meeds.analytics.listener.social; -import static org.exoplatform.analytics.utils.AnalyticsUtils.addStatisticData; -import static org.exoplatform.analytics.utils.AnalyticsUtils.getCurrentUserIdentityId; +import static io.meeds.analytics.utils.AnalyticsUtils.addStatisticData; +import static io.meeds.analytics.utils.AnalyticsUtils.getCurrentUserIdentityId; + +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.stereotype.Component; -import org.exoplatform.analytics.model.StatisticData; import org.exoplatform.services.log.ExoLogger; import org.exoplatform.services.log.Log; +import org.exoplatform.social.core.manager.RelationshipManager; +import org.exoplatform.social.core.manager.RelationshipManagerImpl; import org.exoplatform.social.core.relationship.RelationshipEvent; import org.exoplatform.social.core.relationship.RelationshipListenerPlugin; import org.exoplatform.social.core.relationship.model.Relationship; +import io.meeds.analytics.model.StatisticData; + +import jakarta.annotation.PostConstruct; + +@Component public class AnalyticsRelationshipListener extends RelationshipListenerPlugin { - private static final Log LOG = ExoLogger.getLogger(AnalyticsRelationshipListener.class); + private static final Log LOG = ExoLogger.getLogger(AnalyticsRelationshipListener.class); + + @Autowired + private RelationshipManager relationshipManager; + + @PostConstruct + public void init() { + ((RelationshipManagerImpl) relationshipManager).addListenerPlugin(this); + } @Override public void requested(RelationshipEvent event) { diff --git a/analytics-listeners/src/main/java/org/exoplatform/analytics/listener/social/AnalyticsSpaceListener.java b/analytics-listeners/src/main/java/io/meeds/analytics/listener/social/AnalyticsSpaceListener.java similarity index 91% rename from analytics-listeners/src/main/java/org/exoplatform/analytics/listener/social/AnalyticsSpaceListener.java rename to analytics-listeners/src/main/java/io/meeds/analytics/listener/social/AnalyticsSpaceListener.java index d12dc077f..681becc9f 100644 --- a/analytics-listeners/src/main/java/org/exoplatform/analytics/listener/social/AnalyticsSpaceListener.java +++ b/analytics-listeners/src/main/java/io/meeds/analytics/listener/social/AnalyticsSpaceListener.java @@ -1,36 +1,55 @@ /** * This file is part of the Meeds project (https://meeds.io/). - * Copyright (C) 2022 Meeds Association - * contact@meeds.io + * + * Copyright (C) 2020 - 2024 Meeds Association contact@meeds.io + * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU Lesser General Public * License as published by the Free Software Foundation; either * version 3 of the License, or (at your option) any later version. + * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU * Lesser General Public License for more details. + * * You should have received a copy of the GNU Lesser General Public License * along with this program; if not, write to the Free Software Foundation, * Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. */ -package org.exoplatform.analytics.listener.social; +package io.meeds.analytics.listener.social; -import static org.exoplatform.analytics.utils.AnalyticsUtils.addSpaceStatistics; -import static org.exoplatform.analytics.utils.AnalyticsUtils.addStatisticData; -import static org.exoplatform.analytics.utils.AnalyticsUtils.getCurrentUserIdentityId; +import static io.meeds.analytics.utils.AnalyticsUtils.addSpaceStatistics; +import static io.meeds.analytics.utils.AnalyticsUtils.addStatisticData; +import static io.meeds.analytics.utils.AnalyticsUtils.getCurrentUserIdentityId; + +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.stereotype.Component; -import org.exoplatform.analytics.model.StatisticData; import org.exoplatform.services.log.ExoLogger; import org.exoplatform.services.log.Log; import org.exoplatform.social.core.space.SpaceListenerPlugin; import org.exoplatform.social.core.space.model.Space; import org.exoplatform.social.core.space.spi.SpaceLifeCycleEvent; +import org.exoplatform.social.core.space.spi.SpaceService; + +import io.meeds.analytics.model.StatisticData; + +import jakarta.annotation.PostConstruct; +@Component public class AnalyticsSpaceListener extends SpaceListenerPlugin { private static final Log LOG = ExoLogger.getLogger(AnalyticsSpaceListener.class); + @Autowired + private SpaceService spaceService; + + @PostConstruct + public void init() { + spaceService.registerSpaceListenerPlugin(this); + } + @Override public void spaceAccessEdited(SpaceLifeCycleEvent event) { try { diff --git a/analytics-listeners/src/main/java/org/exoplatform/analytics/listener/social/AnalyticsSpaceWebNotificationListener.java b/analytics-listeners/src/main/java/io/meeds/analytics/listener/social/AnalyticsSpaceWebNotificationListener.java similarity index 74% rename from analytics-listeners/src/main/java/org/exoplatform/analytics/listener/social/AnalyticsSpaceWebNotificationListener.java rename to analytics-listeners/src/main/java/io/meeds/analytics/listener/social/AnalyticsSpaceWebNotificationListener.java index 896440abf..d7d5f141e 100644 --- a/analytics-listeners/src/main/java/org/exoplatform/analytics/listener/social/AnalyticsSpaceWebNotificationListener.java +++ b/analytics-listeners/src/main/java/io/meeds/analytics/listener/social/AnalyticsSpaceWebNotificationListener.java @@ -1,41 +1,66 @@ -/* +/** * This file is part of the Meeds project (https://meeds.io/). - * Copyright (C) 2020 - 2022 Meeds Association contact@meeds.io + * + * Copyright (C) 2020 - 2024 Meeds Association contact@meeds.io + * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU Lesser General Public * License as published by the Free Software Foundation; either * version 3 of the License, or (at your option) any later version. + * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU * Lesser General Public License for more details. + * * You should have received a copy of the GNU Lesser General Public License * along with this program; if not, write to the Free Software Foundation, - * Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + * Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. */ -package org.exoplatform.analytics.listener.social; +package io.meeds.analytics.listener.social; -import static org.exoplatform.analytics.utils.AnalyticsUtils.addSpaceStatistics; -import static org.exoplatform.analytics.utils.AnalyticsUtils.addStatisticData; +import static io.meeds.analytics.utils.AnalyticsUtils.addSpaceStatistics; +import static io.meeds.analytics.utils.AnalyticsUtils.addStatisticData; + +import java.util.Arrays; +import java.util.List; import org.apache.commons.lang3.StringUtils; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.stereotype.Component; -import org.exoplatform.analytics.model.StatisticData; import org.exoplatform.commons.api.persistence.ExoTransactional; +import org.exoplatform.services.listener.Asynchronous; import org.exoplatform.services.listener.Event; import org.exoplatform.services.listener.Listener; +import org.exoplatform.services.listener.ListenerService; import org.exoplatform.social.core.space.model.Space; import org.exoplatform.social.core.space.spi.SpaceService; import org.exoplatform.social.notification.model.SpaceWebNotificationItem; import org.exoplatform.social.notification.model.SpaceWebNotificationItemUpdate; import org.exoplatform.social.notification.service.SpaceWebNotificationService; +import io.meeds.analytics.model.StatisticData; + +import jakarta.annotation.PostConstruct; + +@Asynchronous +@Component public class AnalyticsSpaceWebNotificationListener extends Listener { - private SpaceService spaceService; + private static final List EVENT_NAMES = Arrays.asList("notification.read.item", + "notification.unread.item", + "notification.read.allItems"); + + @Autowired + private SpaceService spaceService; + + @Autowired + private ListenerService listenerService; - public AnalyticsSpaceWebNotificationListener(SpaceService spaceService) { - this.spaceService = spaceService; + @PostConstruct + public void init() { + EVENT_NAMES.forEach(name -> listenerService.addListener(name, this)); } @Override diff --git a/analytics-listeners/src/main/java/org/exoplatform/analytics/listener/social/BaseAttachmentAnalyticsListener.java b/analytics-listeners/src/main/java/io/meeds/analytics/listener/social/BaseAttachmentAnalyticsListener.java similarity index 67% rename from analytics-listeners/src/main/java/org/exoplatform/analytics/listener/social/BaseAttachmentAnalyticsListener.java rename to analytics-listeners/src/main/java/io/meeds/analytics/listener/social/BaseAttachmentAnalyticsListener.java index 6d1cab43a..e4e5dcee3 100644 --- a/analytics-listeners/src/main/java/org/exoplatform/analytics/listener/social/BaseAttachmentAnalyticsListener.java +++ b/analytics-listeners/src/main/java/io/meeds/analytics/listener/social/BaseAttachmentAnalyticsListener.java @@ -1,27 +1,31 @@ -/* +/** * This file is part of the Meeds project (https://meeds.io/). * - * Copyright (C) 2023 Meeds Association contact@meeds.io + * Copyright (C) 2020 - 2024 Meeds Association contact@meeds.io * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU Lesser General Public * License as published by the Free Software Foundation; either * version 3 of the License, or (at your option) any later version. + * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU * Lesser General Public License for more details. - * + * * You should have received a copy of the GNU Lesser General Public License * along with this program; if not, write to the Free Software Foundation, * Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. */ -package org.exoplatform.analytics.listener.social; +package io.meeds.analytics.listener.social; + +import static io.meeds.analytics.utils.AnalyticsUtils.addSpaceStatistics; +import static io.meeds.analytics.utils.AnalyticsUtils.addStatisticData; + +import java.util.Date; +import java.util.List; +import java.util.Map; -import org.exoplatform.analytics.model.StatisticData; -import org.exoplatform.analytics.utils.AnalyticsUtils; -import org.exoplatform.commons.api.persistence.ExoTransactional; -import org.exoplatform.container.xml.InitParams; import org.exoplatform.services.listener.Event; import org.exoplatform.services.listener.Listener; import org.exoplatform.social.attachment.AttachmentPlugin; @@ -30,46 +34,32 @@ import org.exoplatform.social.core.space.model.Space; import org.exoplatform.social.core.space.spi.SpaceService; -import java.util.Date; -import java.util.List; -import java.util.Map; - -import static org.exoplatform.analytics.utils.AnalyticsUtils.addSpaceStatistics; -import static org.exoplatform.analytics.utils.AnalyticsUtils.addStatisticData; +import io.meeds.analytics.model.StatisticData; +import io.meeds.analytics.utils.AnalyticsUtils; +import io.meeds.common.ContainerTransactional; public abstract class BaseAttachmentAnalyticsListener extends Listener { - public static final String STATISTICS_ATTACH_OPERATION = "attachImages"; - - public static final String STATISTICS_DETACH_OPERATION = "removeImageAttachments"; - - public static final String ATTACHMENT_CREATED_EVENT = "attachment.created"; - - public static final String ATTACHMENT_DELETED_EVENT = "attachment.deleted"; - private final AttachmentService attachmentService; + public static final String STATISTICS_ATTACH_OPERATION = "attachImages"; - private SpaceService spaceService; + public static final String STATISTICS_DETACH_OPERATION = "removeImageAttachments"; - private List supportedObjectType; + public static final String ATTACHMENT_CREATED_EVENT = "attachment.created"; - protected BaseAttachmentAnalyticsListener(AttachmentService attachmentService, SpaceService spaceService, InitParams initParams) { - this.attachmentService = attachmentService; - this.spaceService = spaceService; - this.supportedObjectType = initParams.getValuesParam("supported-type").getValues(); - } + public static final String ATTACHMENT_DELETED_EVENT = "attachment.deleted"; @Override - @ExoTransactional + @ContainerTransactional public void onEvent(Event event) throws Exception { String username = event.getSource(); ObjectAttachmentId objectAttachment = event.getData(); - if (objectAttachment == null || !supportedObjectType.contains(objectAttachment.getObjectType())) { + if (objectAttachment == null || !getSupportedObjectType().contains(objectAttachment.getObjectType())) { return; } - Map attachmentPlugins = attachmentService.getAttachmentPlugins(); + Map attachmentPlugins = getAttachmentService().getAttachmentPlugins(); AttachmentPlugin attachmentPlugin = attachmentPlugins.get(objectAttachment.getObjectType()); long spaceId = attachmentPlugin.getSpaceId(objectAttachment.getObjectId()); @@ -93,7 +83,7 @@ public void onEvent(Event event) throws Exception { } private StatisticData buildStatisticData(String operation, ObjectAttachmentId objectAttachment, long spaceId, long userId) { - Space space = spaceService.getSpaceById(String.valueOf(spaceId)); + Space space = getSpaceService().getSpaceById(String.valueOf(spaceId)); StatisticData statisticData = new StatisticData(); statisticData.setModule(getModule(objectAttachment)); statisticData.setSubModule(getSubModule(objectAttachment)); @@ -107,6 +97,12 @@ private StatisticData buildStatisticData(String operation, ObjectAttachmentId ob protected void extendStatisticData(StatisticData statisticData, ObjectAttachmentId objectAttachment) { } + protected abstract AttachmentService getAttachmentService(); + + protected abstract SpaceService getSpaceService(); + + protected abstract List getSupportedObjectType(); + protected abstract String getModule(ObjectAttachmentId objectAttachment); protected abstract String getSubModule(ObjectAttachmentId objectAttachment); diff --git a/analytics-listeners/src/main/java/org/exoplatform/analytics/listener/websocket/WebSocketUIStatisticListener.java b/analytics-listeners/src/main/java/io/meeds/analytics/listener/websocket/WebSocketUIStatisticListener.java similarity index 76% rename from analytics-listeners/src/main/java/org/exoplatform/analytics/listener/websocket/WebSocketUIStatisticListener.java rename to analytics-listeners/src/main/java/io/meeds/analytics/listener/websocket/WebSocketUIStatisticListener.java index bb2b6bd14..a678c9f17 100644 --- a/analytics-listeners/src/main/java/org/exoplatform/analytics/listener/websocket/WebSocketUIStatisticListener.java +++ b/analytics-listeners/src/main/java/io/meeds/analytics/listener/websocket/WebSocketUIStatisticListener.java @@ -1,50 +1,67 @@ /** * This file is part of the Meeds project (https://meeds.io/). - * Copyright (C) 2022 Meeds Association - * contact@meeds.io + * + * Copyright (C) 2020 - 2024 Meeds Association contact@meeds.io + * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU Lesser General Public * License as published by the Free Software Foundation; either * version 3 of the License, or (at your option) any later version. + * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU * Lesser General Public License for more details. + * * You should have received a copy of the GNU Lesser General Public License * along with this program; if not, write to the Free Software Foundation, * Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. */ -package org.exoplatform.analytics.listener.websocket; +package io.meeds.analytics.listener.websocket; -import static org.exoplatform.analytics.utils.AnalyticsUtils.*; +import static io.meeds.analytics.utils.AnalyticsUtils.*; import java.util.*; import java.util.Map.Entry; import org.apache.commons.lang3.StringUtils; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.stereotype.Component; -import org.exoplatform.analytics.api.service.AnalyticsService; -import org.exoplatform.analytics.api.service.StatisticWatcher; -import org.exoplatform.analytics.api.websocket.AnalyticsWebSocketMessage; -import org.exoplatform.analytics.api.websocket.AnalyticsWebSocketService; -import org.exoplatform.analytics.model.StatisticData; import org.exoplatform.services.listener.*; import org.exoplatform.services.log.ExoLogger; import org.exoplatform.services.log.Log; import org.exoplatform.social.core.space.model.Space; +import io.meeds.analytics.api.service.AnalyticsService; +import io.meeds.analytics.api.websocket.AnalyticsWebSocketService; +import io.meeds.analytics.api.websocket.listener.AnalyticsWebSocketListener; +import io.meeds.analytics.model.AnalyticsWebSocketMessage; +import io.meeds.analytics.model.StatisticData; +import io.meeds.analytics.model.StatisticWatcher; + +import jakarta.annotation.PostConstruct; + @Asynchronous +@Component public class WebSocketUIStatisticListener extends Listener { - private static final Log LOG = ExoLogger.getLogger(WebSocketUIStatisticListener.class); + private static final Log LOG = ExoLogger.getLogger(WebSocketUIStatisticListener.class); + + private static final List EVENT_NAMES = Arrays.asList(AnalyticsWebSocketListener.EXO_ANALYTICS_MESSAGE_EVENT); + + @Autowired + private AnalyticsService analyticsService; - private AnalyticsService analyticsService; + @Autowired + private ListenerService listenerService; - public WebSocketUIStatisticListener(AnalyticsService analyticsService) { - this.analyticsService = analyticsService; + @PostConstruct + public void init() { + EVENT_NAMES.forEach(name -> listenerService.addListener(name, this)); } @Override - public void onEvent(Event event) throws Exception { + public void onEvent(Event event) throws Exception { // NOSONAR AnalyticsWebSocketMessage message = event.getData(); long userId = getUserIdentityId(message.getUserName()); if (userId <= 0) { diff --git a/analytics-listeners/src/main/java/org/exoplatform/analytics/job/SpacesStatisticsCountJob.java b/analytics-listeners/src/main/java/org/exoplatform/analytics/job/SpacesStatisticsCountJob.java deleted file mode 100644 index aca87136a..000000000 --- a/analytics-listeners/src/main/java/org/exoplatform/analytics/job/SpacesStatisticsCountJob.java +++ /dev/null @@ -1,81 +0,0 @@ -/** - * This file is part of the Meeds project (https://meeds.io/). - * Copyright (C) 2022 Meeds Association - * contact@meeds.io - * This program is free software; you can redistribute it and/or - * modify it under the terms of the GNU Lesser General Public - * License as published by the Free Software Foundation; either - * version 3 of the License, or (at your option) any later version. - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * Lesser General Public License for more details. - * You should have received a copy of the GNU Lesser General Public License - * along with this program; if not, write to the Free Software Foundation, - * Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. - */ -package org.exoplatform.analytics.job; - -import org.quartz.*; - -import org.exoplatform.analytics.model.StatisticData; -import org.exoplatform.analytics.utils.AnalyticsUtils; -import org.exoplatform.commons.utils.ListAccess; -import org.exoplatform.container.*; -import org.exoplatform.container.component.RequestLifeCycle; -import org.exoplatform.services.log.ExoLogger; -import org.exoplatform.services.log.Log; -import org.exoplatform.social.core.space.model.Space; -import org.exoplatform.social.core.space.spi.SpaceService; - -/** - * A job to collect statistics of users count - */ -@DisallowConcurrentExecution -public class SpacesStatisticsCountJob implements Job { - - private static final Log LOG = ExoLogger.getLogger(SpacesStatisticsCountJob.class); - - private ExoContainer container; - - private SpaceService spaceService; - - public SpacesStatisticsCountJob() { - this.container = PortalContainer.getInstance(); - } - - @Override - public void execute(JobExecutionContext context) throws JobExecutionException { - long startTime = System.currentTimeMillis(); - - ExoContainer currentContainer = ExoContainerContext.getCurrentContainer(); - ExoContainerContext.setCurrentContainer(container); - RequestLifeCycle.begin(this.container); - try { - ListAccess allSpaces = getSpaceService().getAllSpacesWithListAccess(); - int allSpacesCount = allSpaces.getSize(); - - StatisticData statisticData = new StatisticData(); - statisticData.setModule("social"); - statisticData.setSubModule("space"); - statisticData.setOperation("spacesCount"); - statisticData.setDuration(System.currentTimeMillis() - startTime); - statisticData.addParameter("countType", "allSpaces"); - statisticData.addParameter("count", allSpacesCount); - AnalyticsUtils.addStatisticData(statisticData); - } catch (Exception e) { - LOG.error("Error while computing spaces statistics", e); - } finally { - RequestLifeCycle.end(); - ExoContainerContext.setCurrentContainer(currentContainer); - } - } - - private SpaceService getSpaceService() { - if (spaceService == null) { - spaceService = this.container.getComponentInstanceOfType(SpaceService.class); - } - return spaceService; - } - -} diff --git a/analytics-listeners/src/main/java/org/exoplatform/analytics/job/UsersStatisticsCountJob.java b/analytics-listeners/src/main/java/org/exoplatform/analytics/job/UsersStatisticsCountJob.java deleted file mode 100644 index 8b8f457d3..000000000 --- a/analytics-listeners/src/main/java/org/exoplatform/analytics/job/UsersStatisticsCountJob.java +++ /dev/null @@ -1,116 +0,0 @@ -/** - * This file is part of the Meeds project (https://meeds.io/). - * Copyright (C) 2022 Meeds Association - * contact@meeds.io - * This program is free software; you can redistribute it and/or - * modify it under the terms of the GNU Lesser General Public - * License as published by the Free Software Foundation; either - * version 3 of the License, or (at your option) any later version. - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * Lesser General Public License for more details. - * You should have received a copy of the GNU Lesser General Public License - * along with this program; if not, write to the Free Software Foundation, - * Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. - */ -package org.exoplatform.analytics.job; - -import org.quartz.*; - -import org.exoplatform.analytics.model.StatisticData; -import org.exoplatform.analytics.utils.AnalyticsUtils; -import org.exoplatform.commons.utils.ListAccess; -import org.exoplatform.container.*; -import org.exoplatform.container.component.RequestLifeCycle; -import org.exoplatform.services.log.ExoLogger; -import org.exoplatform.services.log.Log; -import org.exoplatform.services.organization.*; -import org.exoplatform.social.core.identity.model.Identity; -import org.exoplatform.social.core.identity.provider.OrganizationIdentityProvider; -import org.exoplatform.social.core.manager.IdentityManager; - -/** - * A job to collect statistics of users count - */ -@DisallowConcurrentExecution -public class UsersStatisticsCountJob implements Job { - - private static final Log LOG = ExoLogger.getLogger(UsersStatisticsCountJob.class); - - private ExoContainer container; - - private OrganizationService organizationService; - - private IdentityManager identityManager; - - public UsersStatisticsCountJob() { - this.container = PortalContainer.getInstance(); - } - - @Override - public void execute(JobExecutionContext context) throws JobExecutionException { - long startTime = System.currentTimeMillis(); - - ExoContainer currentContainer = ExoContainerContext.getCurrentContainer(); - ExoContainerContext.setCurrentContainer(container); - RequestLifeCycle.begin(this.container); - try { - ListAccess allUsers = getOrganizationService().getUserHandler().findAllUsers(UserStatus.ANY); - int allUsersCount = allUsers.getSize(); - - ListAccess enabledIdentities = - getIdentityManager().getIdentitiesByProfileFilter(OrganizationIdentityProvider.NAME, - null, - false); - int enabledUsersCount = enabledIdentities.getSize(); - int disabledUsersCount = allUsersCount - enabledUsersCount; - - addUsersCountStatistic("allUsers", allUsersCount, startTime); - addUsersCountStatistic("enabledUsers", enabledUsersCount, startTime); - addUsersCountStatistic("disabledUsers", disabledUsersCount, startTime); - - startTime = System.currentTimeMillis(); - Group externalsGroup = getOrganizationService().getGroupHandler().findGroupById("/platform/externals"); - int enabledExternalUsersCount = 0; - if (externalsGroup != null) { - ListAccess externalMemberships = getOrganizationService().getMembershipHandler() - .findAllMembershipsByGroup(externalsGroup); - enabledExternalUsersCount = externalMemberships.getSize(); - } - addUsersCountStatistic("enabledExternalUsers", enabledExternalUsersCount, startTime); - addUsersCountStatistic("enabledInternalUsers", (enabledUsersCount - enabledExternalUsersCount), startTime); - } catch (Exception e) { - LOG.error("Error while computing users statistics", e); - } finally { - RequestLifeCycle.end(); - ExoContainerContext.setCurrentContainer(currentContainer); - } - } - - private void addUsersCountStatistic(String countType, int count, long startTime) { - StatisticData statisticData = new StatisticData(); - statisticData.setModule("portal"); - statisticData.setSubModule("account"); - statisticData.setOperation("usersCount"); - statisticData.setDuration(System.currentTimeMillis() - startTime); - statisticData.addParameter("countType", countType); - statisticData.addParameter("count", count); - AnalyticsUtils.addStatisticData(statisticData); - } - - private OrganizationService getOrganizationService() { - if (organizationService == null) { - organizationService = this.container.getComponentInstanceOfType(OrganizationService.class); - } - return organizationService; - } - - private IdentityManager getIdentityManager() { - if (identityManager == null) { - identityManager = this.container.getComponentInstanceOfType(IdentityManager.class); - } - return identityManager; - } - -} diff --git a/analytics-packaging/pom.xml b/analytics-packaging/pom.xml index 4eddbb67f..e4ff96127 100644 --- a/analytics-packaging/pom.xml +++ b/analytics-packaging/pom.xml @@ -2,14 +2,13 @@ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd"> 4.0.0 - org.exoplatform.addons.analytics + io.meeds.analytics analytics-parent 7.0.x-exo-SNAPSHOT analytics-packaging pom - eXo Add-on:: eXo Analytics - Packaging - eXo Add-on:: eXo Analytics - Packaging + Meeds:: Analytics - Packaging ${project.groupId} diff --git a/analytics-services/pom.xml b/analytics-services/pom.xml index 6f98156a1..5ea770ddb 100644 --- a/analytics-services/pom.xml +++ b/analytics-services/pom.xml @@ -1,16 +1,19 @@ diff --git a/analytics-services/src/main/java/io/meeds/analytics/elasticsearch/model/ElasticsearchResponse.java b/analytics-services/src/main/java/io/meeds/analytics/elasticsearch/model/ElasticsearchResponse.java new file mode 100644 index 000000000..8d5be4b6a --- /dev/null +++ b/analytics-services/src/main/java/io/meeds/analytics/elasticsearch/model/ElasticsearchResponse.java @@ -0,0 +1,34 @@ +/** + * This file is part of the Meeds project (https://meeds.io/). + * + * Copyright (C) 2020 - 2024 Meeds Association contact@meeds.io + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 3 of the License, or (at your option) any later version. + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public License + * along with this program; if not, write to the Free Software Foundation, + * Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + */ +package io.meeds.analytics.elasticsearch.model; + +import lombok.AllArgsConstructor; +import lombok.Data; +import lombok.NoArgsConstructor; + +@Data +@AllArgsConstructor +@NoArgsConstructor +public class ElasticsearchResponse { + + private String message; + + private int statusCode; + +} diff --git a/analytics-services/src/main/java/org/exoplatform/analytics/es/service/ESAnalyticsService.java b/analytics-services/src/main/java/io/meeds/analytics/elasticsearch/service/ElasticsearchAnalyticsService.java similarity index 65% rename from analytics-services/src/main/java/org/exoplatform/analytics/es/service/ESAnalyticsService.java rename to analytics-services/src/main/java/io/meeds/analytics/elasticsearch/service/ElasticsearchAnalyticsService.java index 8455f65c4..4cd1880c2 100644 --- a/analytics-services/src/main/java/org/exoplatform/analytics/es/service/ESAnalyticsService.java +++ b/analytics-services/src/main/java/io/meeds/analytics/elasticsearch/service/ElasticsearchAnalyticsService.java @@ -1,152 +1,204 @@ /** * This file is part of the Meeds project (https://meeds.io/). - * Copyright (C) 2022 Meeds Association - * contact@meeds.io + * + * Copyright (C) 2020 - 2024 Meeds Association contact@meeds.io + * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU Lesser General Public * License as published by the Free Software Foundation; either * version 3 of the License, or (at your option) any later version. + * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU * Lesser General Public License for more details. + * * You should have received a copy of the GNU Lesser General Public License * along with this program; if not, write to the Free Software Foundation, - * Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + * Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. */ -package org.exoplatform.analytics.es.service; +package io.meeds.analytics.elasticsearch.service; -import static org.exoplatform.analytics.utils.AnalyticsUtils.*; +import static io.meeds.analytics.utils.AnalyticsUtils.DEFAULT_FIELDS; +import static io.meeds.analytics.utils.AnalyticsUtils.FIELD_TIMESTAMP; +import static io.meeds.analytics.utils.AnalyticsUtils.collectionToJSONString; +import static io.meeds.analytics.utils.AnalyticsUtils.fixJSONStringFormat; +import static io.meeds.analytics.utils.AnalyticsUtils.fromJsonString; +import static io.meeds.analytics.utils.AnalyticsUtils.getJSONObject; +import static io.meeds.analytics.utils.AnalyticsUtils.toJsonString; import java.time.ZoneId; import java.time.ZoneOffset; -import java.util.*; -import java.util.concurrent.*; -import java.util.stream.Collectors; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashMap; +import java.util.HashSet; +import java.util.Iterator; +import java.util.LinkedHashMap; +import java.util.LinkedHashSet; +import java.util.List; +import java.util.Map; +import java.util.Objects; +import java.util.Set; +import java.util.concurrent.Executors; +import java.util.concurrent.ScheduledExecutorService; +import java.util.concurrent.TimeUnit; import org.apache.commons.lang3.StringUtils; -import org.json.*; -import org.picocontainer.Startable; - -import org.exoplatform.analytics.api.service.*; -import org.exoplatform.analytics.es.AnalyticsESClient; -import org.exoplatform.analytics.model.*; -import org.exoplatform.analytics.model.StatisticData.StatisticStatus; -import org.exoplatform.analytics.model.chart.*; -import org.exoplatform.analytics.model.filter.*; -import org.exoplatform.analytics.model.filter.AnalyticsFilter.Range; -import org.exoplatform.analytics.model.filter.aggregation.*; -import org.exoplatform.analytics.model.filter.search.AnalyticsFieldFilter; +import org.json.JSONArray; +import org.json.JSONException; +import org.json.JSONObject; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.beans.factory.annotation.Value; +import org.springframework.context.annotation.Primary; +import org.springframework.stereotype.Service; + import org.exoplatform.commons.api.settings.SettingService; import org.exoplatform.commons.api.settings.SettingValue; import org.exoplatform.commons.api.settings.data.Context; import org.exoplatform.commons.api.settings.data.Scope; -import org.exoplatform.container.ExoContainerContext; -import org.exoplatform.container.PortalContainer; -import org.exoplatform.container.component.RequestLifeCycle; -import org.exoplatform.container.xml.InitParams; import org.exoplatform.services.log.ExoLogger; import org.exoplatform.services.log.Log; -public class ESAnalyticsService implements AnalyticsService, Startable { +import io.meeds.analytics.api.service.AnalyticsService; +import io.meeds.analytics.elasticsearch.storage.ElasticsearchAnalyticsStorage; +import io.meeds.analytics.model.StatisticData; +import io.meeds.analytics.model.StatisticData.StatisticStatus; +import io.meeds.analytics.model.StatisticFieldMapping; +import io.meeds.analytics.model.StatisticFieldValue; +import io.meeds.analytics.model.StatisticWatcher; +import io.meeds.analytics.model.chart.ChartAggregationLabel; +import io.meeds.analytics.model.chart.ChartAggregationResult; +import io.meeds.analytics.model.chart.ChartAggregationValue; +import io.meeds.analytics.model.chart.ChartData; +import io.meeds.analytics.model.chart.ChartDataList; +import io.meeds.analytics.model.chart.PercentageChartResult; +import io.meeds.analytics.model.chart.PercentageChartValue; +import io.meeds.analytics.model.chart.TableColumnItemValue; +import io.meeds.analytics.model.chart.TableColumnResult; +import io.meeds.analytics.model.filter.AnalyticsFilter; +import io.meeds.analytics.model.filter.AnalyticsFilter.Range; +import io.meeds.analytics.model.filter.AnalyticsPercentageFilter; +import io.meeds.analytics.model.filter.AnalyticsPeriod; +import io.meeds.analytics.model.filter.AnalyticsPeriodType; +import io.meeds.analytics.model.filter.AnalyticsTableColumnFilter; +import io.meeds.analytics.model.filter.AnalyticsTableFilter; +import io.meeds.analytics.model.filter.aggregation.AnalyticsAggregation; +import io.meeds.analytics.model.filter.aggregation.AnalyticsAggregationType; +import io.meeds.analytics.model.filter.aggregation.AnalyticsPercentageLimit; +import io.meeds.analytics.model.filter.search.AnalyticsFieldFilter; +import io.meeds.common.ContainerTransactional; - private static final Log LOG = - ExoLogger.getLogger(ESAnalyticsService.class); +import jakarta.annotation.PostConstruct; +import jakarta.annotation.PreDestroy; +import lombok.Getter; - private static final String ANALYTICS_ADMIN_PERMISSION_PARAM_NAME = "exo.analytics.admin.permissions"; +@Primary +@Service +public class ElasticsearchAnalyticsService implements AnalyticsService { - private static final String ANALYTICS_VIEW_ALL_PERMISSION_PARAM_NAME = "exo.analytics.viewall.permissions"; + private static final String VALUE_PARAM = "value"; - private static final String ANALYTICS_VIEW_PERMISSION_PARAM_NAME = "exo.analytics.view.permissions"; + private static final String SORT_DIRECTION_REQUEST_BODY_PARAM = "$sortDirection"; - private static final String RETURNED_AGGREGATION_DOCS_COUNT_PARAM_NAME = - "exo.analytics.aggregation.terms.doc_size"; + private static final String SORT_FIELD_REQUEST_BODY_PARAM = "$sortField"; - private static final String AGGREGATION_KEYS_SEPARATOR = "-"; + private static final String SIZE_REQUEST_BODY_PARAM = "$size"; - private static final String AGGREGATION_RESULT_PARAM = "aggregation_result"; + private static final String MAX_BOUND_REQUEST_BODY_PARAM = "$maxBound"; - private static final String AGGREGATION_RESULT_VALUE_PARAM = "aggregation_result_value"; + private static final String MIN_BOUND_REQUEST_BODY_PARAM = "$minBound"; - private static final String AGGREGATION_BUCKETS_VALUE_PARAM = "aggregation_buckets_value"; + private static final String MAX_REQUEST_BODY_PARAM = "$max"; - private static final Context CONTEXT = Context.GLOBAL.id("ANALYTICS"); + private static final String MIN_REQUEST_BODY_PARAM = "$min"; - private static final Scope ES_SCOPE = Scope.GLOBAL.id("elasticsearch"); + private static final String TIME_ZONE_ID_REQUEST_BODY_PARAM = "$timeZoneId"; - private static final String ES_AGGREGATED_MAPPING = "ES_AGGREGATED_MAPPING"; + private static final String INTERVAL_OFFSET_REQUEST_BODY_PARAM = "$intervalOffset"; - private List uiWatcherPlugins = new ArrayList<>(); + private static final String INTERVAL_REQUEST_BODY_PARAM = "$interval"; - private List uiWatchers = new ArrayList<>(); + private static final String AGG_FIELD_NAME_REQUEST_BODY_PARAM = "$aggFieldName"; - private AnalyticsESClient esClient; + private static final String AGG_NAME_REQUEST_BODY_PARAM = "$aggName"; - private SettingService settingService; + private static final String VALUE_REQUEST_BODY_PARAM = "$value"; - private Map esMappings = new HashMap<>(); + private static final String NAME_REQUEST_BODY_PARAM = "$name"; - private ScheduledExecutorService esMappingUpdater = Executors.newScheduledThreadPool(1); + private static final String BUCKETS_RESPONSE_BODY = "buckets"; - private List administratorsPermissions; + private static final String AGGREGATIONS_RESPONSE_BODY = "aggregations"; - private List viewAllPermissions; + private static final String ERROR_PARSING_RESULTS_MESSAGE = + "Error parsing results with - filter: %s - query: %s - response: %s"; - private List viewPermissions; + private static final String FILTER_AGGREGATIONS_IS_MANDATORY_MESSAGE = "Filter aggregations is mandatory"; - private int aggregationReturnedDocumentsSize = 1000; + private static final String FILTER_IS_MANDATORY_MESSAGE = "Filter is mandatory"; - public ESAnalyticsService(AnalyticsESClient esClient, - SettingService settingService, - InitParams params) { - this.esClient = esClient; - this.settingService = settingService; + private static final Log LOG = + ExoLogger.getLogger(ElasticsearchAnalyticsService.class); - if (params != null && params.containsKey(ANALYTICS_ADMIN_PERMISSION_PARAM_NAME)) { - this.administratorsPermissions = params.getValuesParam(ANALYTICS_ADMIN_PERMISSION_PARAM_NAME).getValues(); - } else { - this.administratorsPermissions = Collections.emptyList(); - } - if (params != null && params.containsKey(ANALYTICS_VIEW_ALL_PERMISSION_PARAM_NAME)) { - this.viewAllPermissions = params.getValuesParam(ANALYTICS_VIEW_ALL_PERMISSION_PARAM_NAME).getValues(); - } else { - this.viewAllPermissions = Collections.emptyList(); - } - if (params != null && params.containsKey(ANALYTICS_VIEW_PERMISSION_PARAM_NAME)) { - this.viewPermissions = params.getValuesParam(ANALYTICS_VIEW_PERMISSION_PARAM_NAME).getValues(); - } else { - this.viewPermissions = Collections.emptyList(); - } - if (params != null && params.containsKey(RETURNED_AGGREGATION_DOCS_COUNT_PARAM_NAME)) { - this.aggregationReturnedDocumentsSize = Integer.parseInt(params.getValueParam(RETURNED_AGGREGATION_DOCS_COUNT_PARAM_NAME) - .getValue()); - } - } + private static final String AGGREGATION_KEYS_SEPARATOR = "-"; - @Override + private static final String AGGREGATION_RESULT_PARAM = "aggregation_result"; + + private static final String AGGREGATION_RESULT_VALUE_PARAM = "aggregation_result_value"; + + private static final String AGGREGATION_BUCKETS_VALUE_PARAM = "aggregation_buckets_value"; + + private static final Context CONTEXT = Context.GLOBAL.id("ANALYTICS"); + + private static final Scope ES_SCOPE = Scope.GLOBAL.id("elasticsearch"); + + private static final String ES_AGGREGATED_MAPPING = "ES_AGGREGATED_MAPPING"; + + @Autowired + private ElasticsearchAnalyticsStorage elasticsearchStorage; + + @Autowired + private SettingService settingService; + + private List uiWatchers = new ArrayList<>(); + + private Map esMappings = new HashMap<>(); + + private ScheduledExecutorService esMappingUpdater = Executors.newScheduledThreadPool(1); + + @Value("${analytics.aggregation.terms.doc_size:200}") + private int aggregationReturnedDocumentsSize; + + @Getter + @Value("${analytics.admin.permission:*:/platform/analytics}") + List administratorsPermissions; + + @Getter + @Value("${analytics.viewall.permissions:*:/platform/administrators}") + List viewAllPermissions; + + @Getter + @Value("${analytics.view.permission:*:/platform/users}") + List viewPermissions; + + @PostConstruct public void start() { // Can't be job, because the mapping retrival must be executed on each // cluster node - esMappingUpdater.scheduleAtFixedRate(() -> { - PortalContainer container = PortalContainer.getInstance(); - ExoContainerContext.setCurrentContainer(container); - RequestLifeCycle.begin(container); - try { - retrieveMapping(true); - } catch (Exception e) { - LOG.warn("Error while getting mapping from elasticsearch", e); - } finally { - RequestLifeCycle.end(); - } - }, 1, 2, TimeUnit.MINUTES); + esMappingUpdater.scheduleAtFixedRate(() -> retrieveMapping(true), + 1, + 2, + TimeUnit.MINUTES); } - @Override + @PreDestroy public void stop() { esMappingUpdater.shutdown(); } @Override + @ContainerTransactional public Set retrieveMapping(boolean forceRefresh) { if (!forceRefresh) { if (esMappings.isEmpty()) { @@ -155,7 +207,7 @@ public Set retrieveMapping(boolean forceRefresh) { return new HashSet<>(esMappings.values()); } try { - String mappingJsonString = esClient.retrieveAllAnalyticsIndexesMapping(); + String mappingJsonString = elasticsearchStorage.retrieveAllAnalyticsIndexesMapping(); if (StringUtils.isBlank(mappingJsonString)) { return new HashSet<>(esMappings.values()); } @@ -209,7 +261,7 @@ public List retrieveFieldValues(String field, int limit) { null, 0, 0); - String jsonResponse = this.esClient.sendRequest(esQuery); + String jsonResponse = this.elasticsearchStorage.search(esQuery); try { return buildFieldValuesResponse(jsonResponse); } catch (JSONException e) { @@ -220,7 +272,7 @@ public List retrieveFieldValues(String field, int limit) { @Override public PercentageChartResult computePercentageChartData(AnalyticsPercentageFilter percentageFilter) { if (percentageFilter == null) { - throw new IllegalArgumentException("Filter is mandatory"); + throw new IllegalArgumentException(FILTER_IS_MANDATORY_MESSAGE); } AnalyticsPeriod currentPeriod = percentageFilter.getCurrentAnalyticsPeriod(); AnalyticsPeriod previousPeriod = percentageFilter.getPreviousAnalyticsPeriod(); @@ -279,10 +331,10 @@ public TableColumnResult computeTableColumnData(TableColumnResult columnResult, int columnIndex, boolean isValue) { if (filter == null) { - throw new IllegalArgumentException("Filter is mandatory"); + throw new IllegalArgumentException(FILTER_IS_MANDATORY_MESSAGE); } if (filter.getAggregations() == null || filter.getAggregations().isEmpty()) { - throw new IllegalArgumentException("Filter aggregations is mandatory"); + throw new IllegalArgumentException(FILTER_AGGREGATIONS_IS_MANDATORY_MESSAGE); } String esQueryString = buildAnalyticsQuery(filter.getAggregations(), @@ -291,7 +343,7 @@ public TableColumnResult computeTableColumnData(TableColumnResult columnResult, filter.getOffset(), filter.getLimit()); - String jsonResponse = this.esClient.sendRequest(esQueryString); + String jsonResponse = this.elasticsearchStorage.search(esQueryString); try { return buildTableColumnDataFromESResponse(columnResult, tableFilter, @@ -302,18 +354,21 @@ public TableColumnResult computeTableColumnData(TableColumnResult columnResult, jsonResponse, filter.getLimit()); } catch (JSONException e) { - throw new IllegalStateException("Error parsing results with - filter: " + filter + " - query: " + esQueryString - + " - response: " + jsonResponse, e); + throw new IllegalStateException(String.format(ERROR_PARSING_RESULTS_MESSAGE, + filter, + esQueryString, + jsonResponse), + e); } } @Override public ChartDataList computeChartData(AnalyticsFilter filter) { if (filter == null) { - throw new IllegalArgumentException("Filter is mandatory"); + throw new IllegalArgumentException(FILTER_IS_MANDATORY_MESSAGE); } if (filter.getAggregations() == null || filter.getAggregations().isEmpty()) { - throw new IllegalArgumentException("Filter aggregations is mandatory"); + throw new IllegalArgumentException(FILTER_AGGREGATIONS_IS_MANDATORY_MESSAGE); } String esQueryString = buildAnalyticsQuery(filter.getAggregations(), @@ -322,12 +377,15 @@ public ChartDataList computeChartData(AnalyticsFilter filter) { filter.getOffset(), filter.getLimit()); - String jsonResponse = this.esClient.sendRequest(esQueryString); + String jsonResponse = this.elasticsearchStorage.search(esQueryString); try { return buildChartDataFromESResponse(filter, jsonResponse); } catch (JSONException e) { - throw new IllegalStateException("Error parsing results with - filter: " + filter + " - query: " + esQueryString - + " - response: " + jsonResponse, e); + throw new IllegalStateException(String.format(ERROR_PARSING_RESULTS_MESSAGE, + filter, + esQueryString, + jsonResponse), + e); } } @@ -337,14 +395,14 @@ public PercentageChartValue computePercentageChartData(AnalyticsFilter filter, AnalyticsPeriod previousPeriod, boolean hasLimitAggregation) { if (filter == null) { - throw new IllegalArgumentException("Filter is mandatory"); + throw new IllegalArgumentException(FILTER_IS_MANDATORY_MESSAGE); } if (filter.getAggregations() == null || filter.getAggregations().isEmpty()) { - throw new IllegalArgumentException("Filter aggregations is mandatory"); + throw new IllegalArgumentException(FILTER_AGGREGATIONS_IS_MANDATORY_MESSAGE); } AnalyticsAggregation yAxisAggregation = filter.getYAxisAggregation(); - AnalyticsAggregationType aggregationType = yAxisAggregation == null ? AnalyticsAggregationType.COUNT - : yAxisAggregation.getType(); + AnalyticsAggregationType aggregationType = yAxisAggregation == null ? AnalyticsAggregationType.COUNT : + yAxisAggregation.getType(); String esQueryString = buildAnalyticsQuery(filter.getAggregations(), filter.getFilters(), aggregationType, @@ -353,12 +411,15 @@ public PercentageChartValue computePercentageChartData(AnalyticsFilter filter, filter.getOffset(), filter.getLimit()); - String jsonResponse = this.esClient.sendRequest(esQueryString); + String jsonResponse = this.elasticsearchStorage.search(esQueryString); try { return buildPercentageChartValuesFromESResponse(jsonResponse, currentPeriod, previousPeriod); } catch (JSONException e) { - throw new IllegalStateException("Error parsing results with - filter: " + filter + " - query: " + esQueryString - + " - response: " + jsonResponse, e); + throw new IllegalStateException(String.format(ERROR_PARSING_RESULTS_MESSAGE, + filter, + esQueryString, + jsonResponse), + e); } } @@ -369,7 +430,7 @@ public List retrieveData(AnalyticsFilter filter) { long limit = filter == null ? 10 : filter.getLimit(); ZoneId timeZone = filter == null ? null : filter.zoneId(); String esQueryString = buildAnalyticsQuery(null, filters, timeZone, offset, limit); - String jsonResponse = this.esClient.sendRequest(esQueryString); + String jsonResponse = this.elasticsearchStorage.search(esQueryString); try { return buildSearchResultFromESResponse(jsonResponse); } catch (JSONException e) { @@ -377,21 +438,6 @@ public List retrieveData(AnalyticsFilter filter) { } } - @Override - public List getAdministratorsPermissions() { - return administratorsPermissions; - } - - @Override - public List getViewAllPermissions() { - return viewAllPermissions; - } - - @Override - public List getViewPermissions() { - return viewPermissions; - } - @Override public List getUIWatchers() { return uiWatchers; @@ -399,18 +445,20 @@ public List getUIWatchers() { @Override public StatisticWatcher getUIWatcher(String name) { - return getUIWatchers().stream().filter(watcher -> StringUtils.equals(name, watcher.getName())).findFirst().orElse(null); + return uiWatchers.stream() + .filter(watcher -> StringUtils.equals(name, watcher.getName())) + .findFirst() + .orElse(null); } @Override - public void addUIWatcherPlugin(StatisticUIWatcherPlugin uiWatcherPlugin) { - uiWatcherPlugins.add(uiWatcherPlugin); - uiWatchers.add(uiWatcherPlugin.getStatisticWatcher()); + public void addUIWatcher(StatisticWatcher uiWatcher) { + uiWatchers.add(uiWatcher); } private List buildFieldValuesResponse(String jsonResponse) throws JSONException { JSONObject json = new JSONObject(jsonResponse); - JSONObject aggregations = json.has("aggregations") ? json.getJSONObject("aggregations") : null; + JSONObject aggregations = json.has(AGGREGATIONS_RESPONSE_BODY) ? json.getJSONObject(AGGREGATIONS_RESPONSE_BODY) : null; if (aggregations == null) { return Collections.emptyList(); } @@ -418,7 +466,7 @@ private List buildFieldValuesResponse(String jsonResponse) if (result == null) { return Collections.emptyList(); } - JSONArray buckets = result.has("buckets") ? result.getJSONArray("buckets") : null; + JSONArray buckets = result.has(BUCKETS_RESPONSE_BODY) ? result.getJSONArray(BUCKETS_RESPONSE_BODY) : null; if (buckets == null) { return Collections.emptyList(); } @@ -506,11 +554,12 @@ private void appendSearchFilterConditions(List filters, St } else { filters = new ArrayList<>(filters); } - - esQuery.append(","); - esQuery.append(" \"query\": {"); - esQuery.append(" \"bool\" : {"); - esQuery.append(" \"must\" : ["); + esQuery.append(""" + , + "query": { + "bool" : { + "must" : [ + """); for (AnalyticsFieldFilter fieldFilter : filters) { String esFieldName = fieldFilter.getField(); StatisticFieldMapping fieldMapping = this.esMappings.get(esFieldName); @@ -518,83 +567,77 @@ private void appendSearchFilterConditions(List filters, St esFieldName = fieldMapping.getAggregationFieldName(); } - String esQueryValue = fieldMapping == null ? StatisticFieldMapping.computeESQueryValue(fieldFilter.getValueString()) - : fieldMapping.getESQueryValue(fieldFilter.getValueString()); + String esQueryValue = fieldMapping == null ? StatisticFieldMapping.computeESQueryValue(fieldFilter.getValueString()) : + fieldMapping.getESQueryValue(fieldFilter.getValueString()); switch (fieldFilter.getType()) { - case NOT_NULL: - esQuery.append(" {\"exists\" : {\"") - .append("field") - .append("\" : \"") - .append(esFieldName) - .append("\" }},"); - break; - case IS_NULL: - esQuery.append(" {\"bool\": {\"must_not\": {\"exists\": {\"field\": \"") - .append(esFieldName) - .append("\" }}}},"); - break; - case EQUAL: - esQuery.append(" {\"match\" : {\"") - .append(esFieldName) - .append("\" : ") - .append(esQueryValue) - .append(" }},"); - break; - case NOT_EQUAL: - esQuery.append(" {\"bool\": {\"must_not\": {\"match\" : {\"") - .append(esFieldName) - .append("\" : ") - .append(esQueryValue) - .append(" }}}},"); - break; - case GREATER: - esQuery.append(" {\"range\" : {\"") - .append(esFieldName) - .append("\" : {") - .append("\"gte\" : ") - .append(esQueryValue) - .append(" }}},"); - break; - case LESS: - esQuery.append(" {\"range\" : {\"") - .append(esFieldName) - .append("\" : {") - .append("\"lte\" : ") - .append(esQueryValue) - .append(" }}},"); - break; - case RANGE: - Range range = fieldFilter.getRange(); - esQuery.append(" {\"range\" : {\"") - .append(esFieldName) - .append("\" : {") - .append("\"gte\" : ") - .append(range.getMin()) - .append(",\"lte\" : ") - .append(range.getMax()) - .append(" }}},"); - break; - case IN_SET: - esQuery.append(" {\"terms\" : {\"") - .append(esFieldName) - .append("\" : ") - .append(collectionToJSONString(fieldFilter.getValueString())) - .append(" }},"); - break; - case NOT_IN_SET: - esQuery.append(" {\"bool\": {\"must_not\": {\"terms\" : {\"") - .append(esFieldName) - .append("\" : ") - .append(collectionToJSONString(fieldFilter.getValueString())) - .append(" }}}},"); - break; - default: - break; + case NOT_NULL: + esQuery.append(""" + {"exists" : {"field": "$name"}}, + """.replace(NAME_REQUEST_BODY_PARAM, esFieldName)); + break; + case IS_NULL: + esQuery.append(""" + {"bool" : {"must_not": {"exists": {"field": "$name"}}}}, + """.replace(NAME_REQUEST_BODY_PARAM, esFieldName)); + break; + case EQUAL: + esQuery.append(""" + {"match" : {"$name": $value}}, + """.replace(NAME_REQUEST_BODY_PARAM, esFieldName) + .replace(VALUE_REQUEST_BODY_PARAM, esQueryValue)); + break; + case NOT_EQUAL: + esQuery.append(""" + {"bool": {"must_not": {"match" : {"$name": $value}}}}, + """.replace(NAME_REQUEST_BODY_PARAM, esFieldName) + .replace(VALUE_REQUEST_BODY_PARAM, esQueryValue)); + break; + case GREATER: + esQuery.append(""" + {"range": {"$name": {"gte": $value}}}, + """.replace(NAME_REQUEST_BODY_PARAM, esFieldName) + .replace(VALUE_REQUEST_BODY_PARAM, esQueryValue)); + break; + case LESS: + esQuery.append(""" + {"range": {"$name": {"lte": $value}}}, + """.replace(NAME_REQUEST_BODY_PARAM, esFieldName) + .replace(VALUE_REQUEST_BODY_PARAM, esQueryValue)); + break; + case RANGE: + Range range = fieldFilter.getRange(); + esQuery.append(""" + {"range": { + "$name": { + "gte": $min, + "lte": $max + } + }}, + """.replace(NAME_REQUEST_BODY_PARAM, esFieldName) + .replace(MIN_REQUEST_BODY_PARAM, range.getMin()) + .replace(MAX_REQUEST_BODY_PARAM, range.getMax())); + break; + case IN_SET: + esQuery.append(""" + {"terms": {"$name": $value}}, + """.replace(NAME_REQUEST_BODY_PARAM, esFieldName) + .replace(VALUE_REQUEST_BODY_PARAM, collectionToJSONString(fieldFilter.getValueString()))); + break; + case NOT_IN_SET: + esQuery.append(""" + {"bool": {"must_not": {"terms": {"$name": $value}}}}, + """.replace(NAME_REQUEST_BODY_PARAM, esFieldName) + .replace(VALUE_REQUEST_BODY_PARAM, collectionToJSONString(fieldFilter.getValueString()))); + break; + default: + break; } } - esQuery.append(" ],"); - esQuery.append(" },"); - esQuery.append(" },"); + esQuery.append(""" + ], + }, + }, + """); } private void buildAggregationQuery(StringBuilder esQuery, @@ -610,8 +653,8 @@ private void buildAggregationQuery(StringBuilder esQuery, AnalyticsAggregationType aggregationType = aggregation.getType(); if (aggregationType.isUseInterval() && StringUtils.isBlank(aggregation.getInterval())) { - throw new IllegalStateException("Analytics aggregation type '" + aggregationType - + "' is using intervals while it has empty interval"); + throw new IllegalStateException("Analytics aggregation type '" + aggregationType + + "' is using intervals while it has empty interval"); } String fieldName = getAggregationFieldName(aggregationType); @@ -620,130 +663,180 @@ private void buildAggregationQuery(StringBuilder esQuery, limit = aggregationReturnedDocumentsSize; } - esQuery.append(" ,\"aggs\": {"); - esQuery.append(" \"") - .append(fieldName) - .append("\": {"); - esQuery.append(" \"") - .append(aggregationType.getAggName()) - .append("\": {"); String aggregationFieldName = aggregation.getField(); StatisticFieldMapping aggregationField = getFieldMapping(aggregationFieldName); - if (aggregationField == null || !aggregationField.isScriptedField()) { - esQuery.append(" \"field\": \"") - .append(aggregationFieldName) - .append("\""); - } else { - esQuery.append(" \"script\": {") - .append(" \"lang\": \"painless\",") - .append(" \"source\": \"") - .append(aggregationFieldName) - .append("\"") - .append("}"); - } - - if (aggregationType.isUseInterval()) { - esQuery.append(","); - if (Arrays.asList(AnalyticsAggregation.YEAR_INTERVAL, - AnalyticsAggregation.QUARTER_INTERVAL, - AnalyticsAggregation.MONTH_INTERVAL, - AnalyticsAggregation.WEEK_INTERVAL, - AnalyticsAggregation.DAY_INTERVAL, - AnalyticsAggregation.HOUR_INTERVAL, - AnalyticsAggregation.MINUTE_INTERVAL) - .contains(aggregation.getInterval())) { - esQuery.append(" \"calendar_interval\": \""); - } else { - esQuery.append(" \"fixed_interval\": \""); - } - esQuery.append(aggregation.getInterval()) - .append("\""); - if (aggregation.getOffset() != null) { - esQuery.append(",") - .append(" \"offset\": \"") - .append(aggregation.getOffset()) - .append("\""); - } - if (timeZone != null - && !ZoneOffset.UTC.equals(timeZone) - && aggregationFieldName.equals(FIELD_TIMESTAMP)) { - esQuery.append(",") - .append(" \"time_zone\": \"") - .append(timeZone.getId()) - .append("\""); - } - if (aggregation.isUseBounds()) { - esQuery.append(",") - .append(" \"min_doc_count\": 0,") - .append("") - .append(" \"extended_bounds\": {") - .append(" \"min\": ") - .append(aggregation.getMinBound()) - .append(",") - .append(" \"max\": ") - .append(aggregation.getMaxBound()) - .append(" }"); - } - } - if (aggregationType.isUseLimit() && limit > 0) { - esQuery.append(" ,\"size\": ").append(limit); - } - if (aggregationType.isUseSort()) { - String sortField = null; - if ((i + 1) < aggregationsSize) { - AnalyticsAggregation nextAggregation = aggregations.get(i + 1); - sortField = getSortField(nextAggregation); - } else if (aggregationType == AnalyticsAggregationType.TERMS) { - sortField = "_count"; - } - if (sortField != null) { - esQuery.append(", \"order\": {\"") - .append(sortField) - .append("\": \"") - .append(aggregation.getSortDirection()) - .append("\"}"); - } + appendStartAggregationFieldQuery(esQuery, aggregationType, fieldName); + { // NOSONAR + appendAggregationNameQuery(esQuery, aggregationFieldName, aggregationField); + appendIntervalQuery(esQuery, timeZone, aggregation, aggregationType, aggregationFieldName); + appendLimitQuery(esQuery, aggregationType, limit); + appendSortQuery(esQuery, aggregations, aggregationsSize, i, aggregation, aggregationType); } - esQuery.append(" }"); + appendEndAggregationFieldQuery(esQuery); // Appended at the end - endOfQuery.append(" }"); - if (hasLimitAggregation && percentageAggregationType != null) { - if (i != (aggregationsSize - 1)) { - endOfQuery.append(" }"); - } - if (i == (aggregationsSize - 2)) { - String bucketAggregationType = null; - switch (percentageAggregationType) { - case MIN: - bucketAggregationType = "min_bucket"; - break; - case MAX: - bucketAggregationType = "max_bucket"; - break; - case AVG: - bucketAggregationType = "avg_bucket"; - break; - default: - bucketAggregationType = "sum_bucket"; - } - String aggregationResultBucketName = AGGREGATION_RESULT_PARAM + ">" + AGGREGATION_RESULT_VALUE_PARAM + ".value"; - endOfQuery.append(" },"); - endOfQuery.append(" \"").append(AGGREGATION_BUCKETS_VALUE_PARAM).append("\": {"); - endOfQuery.append(" \"").append(bucketAggregationType).append("\": {"); - endOfQuery.append(" \"buckets_path\": \"" + aggregationResultBucketName + "\""); - endOfQuery.append(" }"); - endOfQuery.append(" }"); - } - } else { - endOfQuery.append(" }"); - } + appendEndOfAggregations(endOfQuery, percentageAggregationType, hasLimitAggregation, aggregationsSize, i); } esQuery.append(endOfQuery); } } + private void appendStartAggregationFieldQuery(StringBuilder esQuery, + AnalyticsAggregationType aggregationType, + String fieldName) { + esQuery.append(""" + , + "aggs": { + "$name": { + "$aggName": { + + """.replace(NAME_REQUEST_BODY_PARAM, fieldName) + .replace(AGG_NAME_REQUEST_BODY_PARAM, aggregationType.getAggName())); + } + + private void appendAggregationNameQuery(StringBuilder esQuery, + String aggregationFieldName, + StatisticFieldMapping aggregationField) { + if (aggregationField == null || !aggregationField.isScriptedField()) { + esQuery.append(""" + "field": "$aggFieldName" + """.replace(AGG_FIELD_NAME_REQUEST_BODY_PARAM, aggregationFieldName)); + } else { + esQuery.append(""" + "script": { + "lang": "painless", + "source": "$aggFieldName" + } + """.replace(AGG_FIELD_NAME_REQUEST_BODY_PARAM, aggregationFieldName)); + } + } + + private void appendIntervalQuery(StringBuilder esQuery, + ZoneId timeZone, + AnalyticsAggregation aggregation, + AnalyticsAggregationType aggregationType, + String aggregationFieldName) { + if (aggregationType.isUseInterval()) { + appendIntervalTypeQuery(esQuery, aggregation); + appendIntervalOffsetQuery(esQuery, aggregation); + appendIntervalTimeZoneQuery(esQuery, timeZone, aggregationFieldName); + appendIntervalBoundQuery(esQuery, aggregation); + } + } + + private void appendIntervalTypeQuery(StringBuilder esQuery, AnalyticsAggregation aggregation) { + if (AnalyticsAggregation.ALL_INTERVALS.contains(aggregation.getInterval())) { + esQuery.append(""" + , + "calendar_interval": "$interval" + """.replace(INTERVAL_REQUEST_BODY_PARAM, aggregation.getInterval())); + } else { + esQuery.append(""" + , + "fixed_interval": "$interval" + """.replace(INTERVAL_REQUEST_BODY_PARAM, aggregation.getInterval())); + } + } + + private void appendIntervalOffsetQuery(StringBuilder esQuery, AnalyticsAggregation aggregation) { + if (aggregation.getOffset() != null) { + esQuery.append(""" + , + "offset": "$intervalOffset" + """.replace(INTERVAL_OFFSET_REQUEST_BODY_PARAM, aggregation.getOffset())); + } + } + + private void appendIntervalTimeZoneQuery(StringBuilder esQuery, ZoneId timeZone, String aggregationFieldName) { + if (timeZone != null + && !ZoneOffset.UTC.equals(timeZone) + && aggregationFieldName.equals(FIELD_TIMESTAMP)) { + esQuery.append(""" + , + "time_zone": "$timeZoneId" + """.replace(TIME_ZONE_ID_REQUEST_BODY_PARAM, timeZone.getId())); + } + } + + private void appendIntervalBoundQuery(StringBuilder esQuery, AnalyticsAggregation aggregation) { + if (aggregation.isUseBounds()) { + esQuery.append(""" + , + "min_doc_count": 0, + "extended_bounds": { + "min": $minBound, + "max": $maxBound + } + """.replace(MIN_BOUND_REQUEST_BODY_PARAM, String.valueOf(aggregation.getMinBound())) + .replace(MAX_BOUND_REQUEST_BODY_PARAM, String.valueOf(aggregation.getMaxBound()))); + } + } + + private void appendLimitQuery(StringBuilder esQuery, AnalyticsAggregationType aggregationType, long limit) { + if (aggregationType.isUseLimit() && limit > 0) { + esQuery.append(""" + , + "size": $size + """.replace(SIZE_REQUEST_BODY_PARAM, String.valueOf(limit))); + } + } + + private void appendSortQuery(StringBuilder esQuery, + List aggregations, + int aggregationsSize, + int i, + AnalyticsAggregation aggregation, + AnalyticsAggregationType aggregationType) { + if (aggregationType.isUseSort()) { + String sortField = null; + if ((i + 1) < aggregationsSize) { + AnalyticsAggregation nextAggregation = aggregations.get(i + 1); + sortField = getSortField(nextAggregation); + } else if (aggregationType == AnalyticsAggregationType.TERMS) { + sortField = "_count"; + } + if (sortField != null) { + esQuery.append(""" + , + "order": {"$sortField": "$sortDirection"} + """.replace(SORT_FIELD_REQUEST_BODY_PARAM, sortField) + .replace(SORT_DIRECTION_REQUEST_BODY_PARAM, aggregation.getSortDirection())); + } + } + } + + private void appendEndAggregationFieldQuery(StringBuilder esQuery) { + esQuery.append(""" + } + """); // End $aggName + } + + private void appendEndOfAggregations(StringBuffer endOfQuery, + AnalyticsAggregationType percentageAggregationType, + boolean hasLimitAggregation, + int aggregationsSize, + int i) { + endOfQuery.append(""" + } + """); // End $name + if (hasLimitAggregation && percentageAggregationType != null) { + if (i != (aggregationsSize - 1)) { + endOfQuery.append(" }"); + } + if (i == (aggregationsSize - 2)) { + endOfQuery.append(""" + }, + "aggregation_buckets_value": { + "$bucketAggregationType": {"buckets_path": "aggregation_result>aggregation_result_value.value"} + } + """.replace("$bucketAggregationType", getBucketAggregationType(percentageAggregationType))); + } + } else { + endOfQuery.append(" }"); + } + } + private String getAggregationFieldName(AnalyticsAggregationType aggregationType) { String fieldName = null; if (AnalyticsAggregationType.TERMS == aggregationType @@ -782,10 +875,8 @@ private void computePercentageLimits(AnalyticsPercentageFilter percentageFilter, percentageChartResult.setCurrentPeriodLimit(currentPeriodLimit); percentageChartResult.setPreviousPeriodLimit(previousPeriodLimit); - percentageFilter.setCurrentPeriodLimit(Math.round(currentPeriodLimit * percentageLimit.getPercentage() - / 100)); - percentageFilter.setPreviousPeriodLimit(Math.round(previousPeriodLimit * percentageLimit.getPercentage() - / 100)); + percentageFilter.setCurrentPeriodLimit(Math.round(currentPeriodLimit * percentageLimit.getPercentage() / 100)); + percentageFilter.setPreviousPeriodLimit(Math.round(previousPeriodLimit * percentageLimit.getPercentage() / 100)); } private void computePercentageValuesPerPeriod(PercentageChartResult percentageResult, @@ -825,7 +916,7 @@ private TableColumnResult buildTableColumnDataFromESResponse(TableColumnResult t tableColumnResult = new TableColumnResult(); } JSONObject json = new JSONObject(jsonResponse); - JSONObject aggregations = json.has("aggregations") ? json.getJSONObject("aggregations") : null; + JSONObject aggregations = json.has(AGGREGATIONS_RESPONSE_BODY) ? json.getJSONObject(AGGREGATIONS_RESPONSE_BODY) : null; if (aggregations == null) { return tableColumnResult; } @@ -833,7 +924,7 @@ private TableColumnResult buildTableColumnDataFromESResponse(TableColumnResult t if (result == null) { return tableColumnResult; } - JSONArray buckets = result.has("buckets") ? result.getJSONArray("buckets") : null; + JSONArray buckets = result.has(BUCKETS_RESPONSE_BODY) ? result.getJSONArray(BUCKETS_RESPONSE_BODY) : null; if (buckets == null) { return tableColumnResult; } @@ -850,8 +941,8 @@ private TableColumnResult buildTableColumnDataFromESResponse(TableColumnResult t if (!isCurrent && !previousPeriod.isInPeriod(timestamp)) { continue; } - if (bucket.has(AGGREGATION_RESULT_PARAM) && bucket.getJSONObject(AGGREGATION_RESULT_PARAM).has("buckets")) { - JSONArray subBuckets = bucket.getJSONObject(AGGREGATION_RESULT_PARAM).getJSONArray("buckets"); + if (bucket.has(AGGREGATION_RESULT_PARAM) && bucket.getJSONObject(AGGREGATION_RESULT_PARAM).has(BUCKETS_RESPONSE_BODY)) { + JSONArray subBuckets = bucket.getJSONObject(AGGREGATION_RESULT_PARAM).getJSONArray(BUCKETS_RESPONSE_BODY); for (int j = 0; j < subBuckets.length(); j++) { JSONObject subBucket = subBuckets.getJSONObject(j); String key = getResultKeyAsString(subBucket); @@ -881,7 +972,7 @@ private TableColumnResult buildTableColumnDataFromESResponse(TableColumnResult t } List itemsList = null; if (limit > 0) { - itemsList = itemValues.values().stream().limit(limit).collect(Collectors.toList()); + itemsList = itemValues.values().stream().limit(limit).toList(); } else { itemsList = new ArrayList<>(itemValues.values()); } @@ -895,20 +986,20 @@ private void computeColumnItemValue(TableColumnItemValue itemValue, boolean isValue) throws JSONException { Object value; if (bucket.has(AGGREGATION_RESULT_VALUE_PARAM)) { - value = bucket.getJSONObject(AGGREGATION_RESULT_VALUE_PARAM).get("value"); + value = bucket.getJSONObject(AGGREGATION_RESULT_VALUE_PARAM).get(VALUE_PARAM); } else if (bucket.has(AGGREGATION_RESULT_PARAM)) { JSONObject subAggregationResult = bucket.getJSONObject(AGGREGATION_RESULT_PARAM); List values = new ArrayList<>(); - if (subAggregationResult.has("buckets")) { - JSONArray subAggregationBuckets = subAggregationResult.getJSONArray("buckets"); + if (subAggregationResult.has(BUCKETS_RESPONSE_BODY)) { + JSONArray subAggregationBuckets = subAggregationResult.getJSONArray(BUCKETS_RESPONSE_BODY); for (int j = 0; j < subAggregationBuckets.length(); j++) { JSONObject subAggregationBucket = subAggregationBuckets.getJSONObject(j); values.add(getResultKeyAsString(subAggregationBucket)); } } value = values; - } else if (bucket.has("value")) { - value = bucket.get("value"); + } else if (bucket.has(VALUE_PARAM)) { + value = bucket.get(VALUE_PARAM); } else { value = null; } @@ -927,22 +1018,22 @@ private void computeColumnItemValue(TableColumnItemValue itemValue, } } - private PercentageChartValue buildPercentageChartValuesFromESResponse(String jsonResponse, + private PercentageChartValue buildPercentageChartValuesFromESResponse(String jsonResponse, // NOSONAR AnalyticsPeriod currentPeriod, AnalyticsPeriod previousPeriod) throws JSONException { PercentageChartValue percentageChartValue = new PercentageChartValue(); JSONObject json = new JSONObject(jsonResponse); - JSONObject aggregations = json.getJSONObject("aggregations"); + JSONObject aggregations = json.getJSONObject(AGGREGATIONS_RESPONSE_BODY); if (aggregations == null) { return percentageChartValue; } percentageChartValue.setComputingTime(json.getLong("took")); JSONObject hitsResult = json.getJSONObject("hits"); - percentageChartValue.setDataCount(hitsResult.getJSONObject("total").getLong("value")); + percentageChartValue.setDataCount(hitsResult.getJSONObject("total").getLong(VALUE_PARAM)); if (aggregations.has(AGGREGATION_BUCKETS_VALUE_PARAM)) { - String value = toString(aggregations.getJSONObject(AGGREGATION_BUCKETS_VALUE_PARAM).get("value")); + String value = toString(aggregations.getJSONObject(AGGREGATION_BUCKETS_VALUE_PARAM).get(VALUE_PARAM)); double valueDouble = StringUtils.isBlank(value) || StringUtils.equals("null", value) ? 0d : Double.parseDouble(value); if (currentPeriod != null) { percentageChartValue.setCurrentPeriodValue(valueDouble); @@ -950,18 +1041,18 @@ private PercentageChartValue buildPercentageChartValuesFromESResponse(String jso percentageChartValue.setPreviousPeriodValue(valueDouble); } } else if (aggregations.has(AGGREGATION_RESULT_PARAM)) { - JSONArray buckets = aggregations.getJSONObject(AGGREGATION_RESULT_PARAM).getJSONArray("buckets"); + JSONArray buckets = aggregations.getJSONObject(AGGREGATION_RESULT_PARAM).getJSONArray(BUCKETS_RESPONSE_BODY); Map values = new HashMap<>(); if (buckets.length() > 0) { for (int i = 0; i < buckets.length(); i++) { JSONObject bucket = buckets.getJSONObject(i); Long timestamp = bucket.getLong("key"); if (bucket.has(AGGREGATION_BUCKETS_VALUE_PARAM)) { - String value = toString(bucket.getJSONObject(AGGREGATION_BUCKETS_VALUE_PARAM).get("value")); + String value = toString(bucket.getJSONObject(AGGREGATION_BUCKETS_VALUE_PARAM).get(VALUE_PARAM)); values.put(timestamp, StringUtils.isBlank(value) || StringUtils.equals("null", value) ? 0d : Double.parseDouble(value)); } else if (bucket.has(AGGREGATION_RESULT_VALUE_PARAM)) { - String value = toString(bucket.getJSONObject(AGGREGATION_RESULT_VALUE_PARAM).get("value")); + String value = toString(bucket.getJSONObject(AGGREGATION_RESULT_VALUE_PARAM).get(VALUE_PARAM)); values.put(timestamp, StringUtils.isBlank(value) || StringUtils.equals("null", value) ? 0d : Double.parseDouble(value)); } else { @@ -975,7 +1066,7 @@ private PercentageChartValue buildPercentageChartValuesFromESResponse(String jso .filter(aggregationValue -> { long timestamp = aggregationValue.getKey(); return timestamp < currentPeriod.getToInMS() - && timestamp >= currentPeriod.getFromInMS(); + && timestamp >= currentPeriod.getFromInMS(); }) .map(Map.Entry::getValue) @@ -990,7 +1081,7 @@ private PercentageChartValue buildPercentageChartValuesFromESResponse(String jso .filter(aggregationValue -> { long timestamp = aggregationValue.getKey(); return timestamp < previousPeriod.getToInMS() - && timestamp >= previousPeriod.getFromInMS(); + && timestamp >= previousPeriod.getFromInMS(); }) .map(Map.Entry::getValue) @@ -1009,14 +1100,14 @@ private ChartDataList buildChartDataFromESResponse(AnalyticsFilter filter, Strin ChartDataList chartsData = new ChartDataList(lang); JSONObject json = new JSONObject(jsonResponse); - JSONObject aggregations = json.getJSONObject("aggregations"); + JSONObject aggregations = json.getJSONObject(AGGREGATIONS_RESPONSE_BODY); if (aggregations == null) { return chartsData; } JSONObject hitsResult = (JSONObject) json.get("hits"); chartsData.setComputingTime(json.getLong("took")); - chartsData.setDataCount(hitsResult.getJSONObject("total").getLong("value")); + chartsData.setDataCount(hitsResult.getJSONObject("total").getLong(VALUE_PARAM)); int level = multipleChartsAggregation == null ? 0 : -1; computeAggregatedResultEntry(filter, aggregations, chartsData, multipleChartsAggregation, null, null, level); @@ -1024,7 +1115,7 @@ private ChartDataList buildChartDataFromESResponse(AnalyticsFilter filter, Strin return chartsData; } - private void computeAggregatedResultEntry(AnalyticsFilter filter, + private void computeAggregatedResultEntry(AnalyticsFilter filter, // NOSONAR JSONObject aggregations, ChartDataList chartsData, AnalyticsAggregation multipleChartsAggregation, @@ -1034,7 +1125,7 @@ private void computeAggregatedResultEntry(AnalyticsFilter filter, String lang = filter.getLang(); JSONObject aggsResult = aggregations.getJSONObject(AGGREGATION_RESULT_PARAM); - JSONArray buckets = aggsResult.getJSONArray("buckets"); + JSONArray buckets = aggsResult.getJSONArray(BUCKETS_RESPONSE_BODY); if (buckets.length() > 0) { int nextLevel = level + 1; for (int i = 0; i < buckets.length(); i++) { @@ -1053,13 +1144,13 @@ private void computeAggregatedResultEntry(AnalyticsFilter filter, result = bucketResult.get("doc_count").toString(); } else { JSONObject valueResult = bucketResult.getJSONObject(AGGREGATION_RESULT_VALUE_PARAM); - result = valueResult.get("value").toString(); + result = valueResult.get(VALUE_PARAM).toString(); } addAggregationValue(key, filter, childAggregationValues, level); List labels = childAggregationValues.stream() .map(ChartAggregationValue::getFieldLabel) - .collect(Collectors.toList()); + .toList(); String label = StringUtils.join(labels, AGGREGATION_KEYS_SEPARATOR); ChartAggregationLabel chartLabel = new ChartAggregationLabel(childAggregationValues, label, lang); @@ -1154,6 +1245,19 @@ private StatisticFieldMapping getFieldMapping(String fieldName) { return esMappings.get(fieldName); } + private String getBucketAggregationType(AnalyticsAggregationType percentageAggregationType) { + switch (percentageAggregationType) { + case MIN: + return "min_bucket"; + case MAX: + return "max_bucket"; + case AVG: + return "avg_bucket"; + default: + return "sum_bucket"; + } + } + private void readFieldsMapping() { SettingValue existingMapping = settingService.get(CONTEXT, ES_SCOPE, ES_AGGREGATED_MAPPING); if (existingMapping == null) { diff --git a/analytics-services/src/main/java/org/exoplatform/analytics/es/processor/ElasticSearchStatisticDataProcessor.java b/analytics-services/src/main/java/io/meeds/analytics/elasticsearch/service/ElasticsearchStatisticDataProcessorService.java similarity index 51% rename from analytics-services/src/main/java/org/exoplatform/analytics/es/processor/ElasticSearchStatisticDataProcessor.java rename to analytics-services/src/main/java/io/meeds/analytics/elasticsearch/service/ElasticsearchStatisticDataProcessorService.java index cad76297a..021ab6c25 100644 --- a/analytics-services/src/main/java/org/exoplatform/analytics/es/processor/ElasticSearchStatisticDataProcessor.java +++ b/analytics-services/src/main/java/io/meeds/analytics/elasticsearch/service/ElasticsearchStatisticDataProcessorService.java @@ -1,51 +1,47 @@ /** * This file is part of the Meeds project (https://meeds.io/). - * Copyright (C) 2022 Meeds Association - * contact@meeds.io + * + * Copyright (C) 2020 - 2024 Meeds Association contact@meeds.io + * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU Lesser General Public * License as published by the Free Software Foundation; either * version 3 of the License, or (at your option) any later version. + * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU * Lesser General Public License for more details. + * * You should have received a copy of the GNU Lesser General Public License * along with this program; if not, write to the Free Software Foundation, * Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. */ -package org.exoplatform.analytics.es.processor; - -import static org.exoplatform.analytics.utils.AnalyticsUtils.ES_ANALYTICS_PROCESSOR_ID; +package io.meeds.analytics.elasticsearch.service; import java.util.List; -import org.exoplatform.analytics.api.processor.StatisticDataProcessorPlugin; -import org.exoplatform.analytics.es.AnalyticsESClient; -import org.exoplatform.analytics.model.StatisticDataQueueEntry; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.stereotype.Component; -public class ElasticSearchStatisticDataProcessor extends StatisticDataProcessorPlugin { +import io.meeds.analytics.elasticsearch.storage.ElasticsearchAnalyticsStorage; +import io.meeds.analytics.model.StatisticDataQueueEntry; +import io.meeds.analytics.plugin.StatisticDataProcessorPlugin; - private AnalyticsESClient analyticsIndexingClient; +@Component +public class ElasticsearchStatisticDataProcessorService implements StatisticDataProcessorPlugin { - public ElasticSearchStatisticDataProcessor(AnalyticsESClient analyticsIndexingClient) { - this.analyticsIndexingClient = analyticsIndexingClient; - } + @Autowired + private ElasticsearchAnalyticsStorage elasticsearchStorage; @Override public String getId() { - return ES_ANALYTICS_PROCESSOR_ID; + return "analytics.processor.elasticsearch"; } @Override public void process(List processorQueueEntries) { - analyticsIndexingClient.sendCreateBulkDocumentsRequest(processorQueueEntries); - } - - @Override - public void init() { - analyticsIndexingClient.init(); - setInitialized(true); + elasticsearchStorage.sendCreateBulkDocumentsRequest(processorQueueEntries); } } diff --git a/analytics-services/src/main/java/io/meeds/analytics/elasticsearch/storage/ElasticsearchAnalyticsStorage.java b/analytics-services/src/main/java/io/meeds/analytics/elasticsearch/storage/ElasticsearchAnalyticsStorage.java new file mode 100644 index 000000000..96000c7f2 --- /dev/null +++ b/analytics-services/src/main/java/io/meeds/analytics/elasticsearch/storage/ElasticsearchAnalyticsStorage.java @@ -0,0 +1,434 @@ +/** + * This file is part of the Meeds project (https://meeds.io/). + * + * Copyright (C) 2020 - 2024 Meeds Association contact@meeds.io + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 3 of the License, or (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public License + * along with this program; if not, write to the Free Software Foundation, + * Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + */ +package io.meeds.analytics.elasticsearch.storage; + +import static io.meeds.analytics.utils.AnalyticsUtils.FIELD_DURATION; +import static io.meeds.analytics.utils.AnalyticsUtils.FIELD_ERROR_CODE; +import static io.meeds.analytics.utils.AnalyticsUtils.FIELD_ERROR_MESSAGE; +import static io.meeds.analytics.utils.AnalyticsUtils.FIELD_IS_ANALYTICS; +import static io.meeds.analytics.utils.AnalyticsUtils.FIELD_MODULE; +import static io.meeds.analytics.utils.AnalyticsUtils.FIELD_OPERATION; +import static io.meeds.analytics.utils.AnalyticsUtils.FIELD_SPACE_ID; +import static io.meeds.analytics.utils.AnalyticsUtils.FIELD_STATUS; +import static io.meeds.analytics.utils.AnalyticsUtils.FIELD_SUB_MODULE; +import static io.meeds.analytics.utils.AnalyticsUtils.FIELD_TIMESTAMP; +import static io.meeds.analytics.utils.AnalyticsUtils.FIELD_USER_ID; + +import java.io.IOException; +import java.time.Instant; +import java.time.ZoneOffset; +import java.time.format.DateTimeFormatter; +import java.time.format.ResolverStyle; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Objects; +import java.util.Set; +import java.util.concurrent.CompletableFuture; + +import org.apache.commons.httpclient.HttpStatus; +import org.apache.commons.lang3.StringUtils; +import org.apache.hc.client5.http.classic.HttpClient; +import org.apache.hc.client5.http.classic.methods.HttpDelete; +import org.apache.hc.client5.http.classic.methods.HttpGet; +import org.apache.hc.client5.http.classic.methods.HttpHead; +import org.apache.hc.client5.http.classic.methods.HttpPost; +import org.apache.hc.client5.http.classic.methods.HttpPut; +import org.apache.hc.core5.http.ClassicHttpResponse; +import org.apache.hc.core5.http.ContentType; +import org.apache.hc.core5.http.HttpEntity; +import org.apache.hc.core5.http.io.entity.EntityUtils; +import org.apache.hc.core5.http.io.entity.StringEntity; +import org.json.JSONException; +import org.json.JSONObject; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.beans.factory.annotation.Qualifier; +import org.springframework.cache.annotation.CacheEvict; +import org.springframework.cache.annotation.Cacheable; +import org.springframework.stereotype.Component; + +import org.exoplatform.commons.search.domain.Document; +import org.exoplatform.services.log.ExoLogger; +import org.exoplatform.services.log.Log; + +import io.meeds.analytics.elasticsearch.model.ElasticsearchResponse; +import io.meeds.analytics.model.StatisticData; +import io.meeds.analytics.model.StatisticDataQueueEntry; + +import jakarta.annotation.PostConstruct; +import lombok.SneakyThrows; + +@Component +public class ElasticsearchAnalyticsStorage { + + private static final Log LOG = + ExoLogger.getExoLogger(ElasticsearchAnalyticsStorage.class); + + private static final long DAY_IN_MS = 86400000L; + + private static final String DAY_DATE_FORMAT = "yyyy-MM-dd"; + + public static final DateTimeFormatter DAY_DATE_FORMATTER = DateTimeFormatter.ofPattern(DAY_DATE_FORMAT) + .withResolverStyle(ResolverStyle.LENIENT); + + @Autowired + private ElasticsearchConfiguration elasticsearchConfiguration; + + @Autowired + @Qualifier("elasticsearchHttpClient") + private HttpClient httpClient; + + @PostConstruct + public void init() { + try { + checkIndexTemplateExistence(); + CompletableFuture.runAsync(this::sendRolloverRequest); + } catch (Exception e) { + LOG.warn("Error while initializing Elasticsearch connection", e); + } + } + + public void sendCreateBulkDocumentsRequest(List dataQueueEntries) { + if (dataQueueEntries == null || dataQueueEntries.isEmpty()) { + return; + } + + LOG.debug("Indexing in bulk {} documents", dataQueueEntries.size()); + sendCreateIndexRequest(); + + StringBuilder request = new StringBuilder(); + for (StatisticDataQueueEntry statisticDataQueueEntry : dataQueueEntries) { + String singleDocumentQuery = getCreateDocumentRequestContent(String.valueOf(statisticDataQueueEntry.getId()), + statisticDataQueueEntry.getStatisticData()); + request.append(singleDocumentQuery); + } + + LOG.debug("Create documents request to ES: {}", request); + sendPutRequest("_bulk", request.toString()); + + sendRefreshIndex(); + } + + public String search(String esQuery) { + ElasticsearchResponse elasticResponse = sendPostRequest(elasticsearchConfiguration.getIndexAlias() + "/_search", esQuery); + String response = elasticResponse.getMessage(); + int statusCode = elasticResponse.getStatusCode(); + if (StringUtils.isBlank(response)) { + response = "Empty response was sent by ES"; + } else if (!isError(elasticResponse)) { + org.json.JSONObject json = null; + try { + json = new JSONObject(response); + if (json.has("status") && isError(json.getInt("status"))) { + throw new IllegalStateException("Error occured while requesting ES HTTP error code: '" + statusCode + + "', HTTP response: '" + response + "'"); + } + } catch (JSONException e) { + throw new IllegalStateException("Error occured while requesting ES HTTP code: '" + statusCode + + "', Error parsing response to JSON format, content = '" + response + "'", e); + } + } + return response; + } + + public String retrieveAllAnalyticsIndexesMapping() { + ElasticsearchResponse response = sendGetRequest(elasticsearchConfiguration.getIndexAlias() + "/_mapping", false); + if (isError(response)) { + LOG.warn("Error getting mapping of analytics : - \t\tcode : {} - \t\tmessage: {}", + response.getStatusCode(), + response.getMessage()); + return null; + } else { + return response.getMessage(); + } + } + + public ElasticsearchResponse sendGetRequest(String uri) { + return sendGetRequest(uri, true); + } + + public ElasticsearchResponse sendGetRequest(String uri, boolean handleResponse) { + ElasticsearchResponse response = sendHttpGetRequest(elasticsearchConfiguration.getUrlClient() + "/" + uri); + if (handleResponse) { + return handleESResponse(response, uri, null); + } else { + return response; + } + } + + public ElasticsearchResponse sendHeadRequest(String uri) { + ElasticsearchResponse response = sendHttpHeadRequest(elasticsearchConfiguration.getUrlClient() + "/" + uri); + return handleESResponse(response, uri, null); + } + + public ElasticsearchResponse sendPutRequest(String uri, String content) { + ElasticsearchResponse response = sendHttpPutRequest(elasticsearchConfiguration.getUrlClient() + "/" + uri, content); + return handleESResponse(response, uri, content); + } + + public ElasticsearchResponse sendDeleteRequest(String uri) { + ElasticsearchResponse response = sendHttpDeleteRequest(elasticsearchConfiguration.getUrlClient() + "/" + uri); + return handleESResponse(response, uri, null); + } + + public ElasticsearchResponse sendPostRequest(String uri, String content) { + ElasticsearchResponse response = sendHttpPostRequest(elasticsearchConfiguration.getUrlClient() + "/" + uri, content); + return handleESResponse(response, uri, content); + } + + private boolean sendCreateIndexRequest() { + String index = getIndex(); + if (sendIsIndexExistsRequest(index)) { + LOG.debug("Index {} already exists. Index creation requests will not be sent.", index); + return false; + } else { + sendTurnOffWriteOnAllAnalyticsIndexes(); + sendCreateIndex(index); + if (sendIsIndexExistsRequest(index)) { + LOG.info("New analytics index {} created.", index); + return true; + } else { + throw new IllegalStateException("Error creating index " + index + " on elasticsearch"); + } + } + } + + private void sendTurnOffWriteOnAllAnalyticsIndexes() { + if (sendIsIndexExistsRequest(elasticsearchConfiguration.getIndexAlias())) { + String esQuery = getTurnOffWriteOnAllAnalyticsIndexes(); + try { + sendPostRequest("_aliases", esQuery); + LOG.info("All analytics indexes switched to RO mode to prepare creation of a new index"); + } catch (Exception e) { + LOG.warn("Analytics old indexes seems to not be turned off on write access"); + } + } + } + + @Cacheable("analytics.indexExists") + private boolean sendIsIndexExistsRequest(String esIndex) { + ElasticsearchResponse responseExists = sendGetRequest(esIndex, false); + return responseExists.getStatusCode() == HttpStatus.SC_OK; + } + + @CacheEvict("analytics.indexExists") + private void sendCreateIndex(String index) { + sendPutRequest(index, getCreateIndexRequestContent()); + CompletableFuture.runAsync(this::sendRolloverRequest); + } + + private boolean sendIsIndexTemplateExistsRequest() { + ElasticsearchResponse responseExists = sendGetRequest("_index_template/" + elasticsearchConfiguration.getIndexTemplateName(), + false); + return responseExists.getStatusCode() == HttpStatus.SC_OK; + } + + private void sendRefreshIndex() { + sendRefreshIndex(elasticsearchConfiguration.getIndexAlias()); + } + + private void sendRefreshIndex(String index) { + sendPostRequest(index + "/_refresh", null); + } + + @SneakyThrows + private ElasticsearchResponse sendHttpPostRequest(String url, String content) { + HttpPost httpTypeRequest = new HttpPost(url); + if (StringUtils.isNotBlank(content)) { + httpTypeRequest.setEntity(new StringEntity(content, ContentType.APPLICATION_JSON)); + } + return httpClient.execute(httpTypeRequest, this::handleHttpResponse); + } + + @SneakyThrows + private ElasticsearchResponse sendHttpPutRequest(String url, String content) { + HttpPut httpTypeRequest = new HttpPut(url); + if (StringUtils.isNotBlank(content)) { + httpTypeRequest.setEntity(new StringEntity(content, ContentType.APPLICATION_JSON)); + } + return httpClient.execute(httpTypeRequest, this::handleHttpResponse); + } + + @SneakyThrows + private ElasticsearchResponse sendHttpDeleteRequest(String url) { + HttpDelete httpDeleteRequest = new HttpDelete(url); + return httpClient.execute(httpDeleteRequest, this::handleHttpResponse); + } + + @SneakyThrows + private ElasticsearchResponse sendHttpGetRequest(String url) { + HttpGet httpGetRequest = new HttpGet(url); + return httpClient.execute(httpGetRequest, this::handleHttpResponse); + } + + @SneakyThrows + private ElasticsearchResponse sendHttpHeadRequest(String url) { + HttpHead httpHeadRequest = new HttpHead(url); + return httpClient.execute(httpHeadRequest, this::handleHttpResponse); + } + + private String getCreateIndexRequestContent() { + return " {" + + "\"aliases\": {" + + " \"" + elasticsearchConfiguration.getIndexAlias() + "\": {" + + " \"is_write_index\" : true" + + " }" + + "}" + + "}"; + } + + private String getTurnOffWriteOnAllAnalyticsIndexes() { + return "{" + + "\"actions\": [" + + " {" + + " \"add\": {" + + " \"index\": \"" + elasticsearchConfiguration.getIndexPrefix() + "*\"," + + " \"alias\": \"" + elasticsearchConfiguration.getIndexAlias() + "\"," + + " \"is_write_index\": false" + + " }" + + " }" + + "]" + + "}"; + } + + private String getCreateDocumentRequestContent(String id, StatisticData data) { + JSONObject jsonObject = createCUDHeaderRequestContent(id); + String timestampString = String.valueOf(data.getTimestamp()); + + Map fields = new HashMap<>(); + fields.put("id", id); + fields.put(FIELD_TIMESTAMP, timestampString); + fields.put(FIELD_USER_ID, String.valueOf(data.getUserId())); + fields.put(FIELD_SPACE_ID, String.valueOf(data.getSpaceId())); + fields.put(FIELD_MODULE, data.getModule()); + fields.put(FIELD_SUB_MODULE, data.getSubModule()); + fields.put(FIELD_OPERATION, data.getOperation()); + fields.put(FIELD_STATUS, String.valueOf(data.getStatus().ordinal())); + fields.put(FIELD_ERROR_CODE, String.valueOf(data.getErrorCode())); + fields.put(FIELD_ERROR_MESSAGE, data.getErrorMessage()); + fields.put(FIELD_DURATION, String.valueOf(data.getDuration())); + fields.put(FIELD_IS_ANALYTICS, "true"); + if (data.getParameters() != null && !data.getParameters().isEmpty()) { + fields.putAll(data.getParameters()); + } + Document document = new Document(String.valueOf(id), + null, + null, + (Set) null, + fields); + if (data.getListParameters() != null && !data.getListParameters().isEmpty()) { + document.setListFields(data.getListParameters()); + } + JSONObject createRequest = new JSONObject(); + createRequest.put("create", jsonObject); + return createRequest.toString() + "\n" + document.toJSON() + "\n"; + } + + private JSONObject createCUDHeaderRequestContent(String id) { + JSONObject cudHeader = new JSONObject(); + cudHeader.put("_index", elasticsearchConfiguration.getIndexAlias()); + cudHeader.put("_id", id); + return cudHeader; + } + + /** + * Handle Http response receive from ES Log an INFO if the return status code + * is 2xx Log an ERROR if the return code is different from 2xx + * + * @param httpResponse The Http Response to handle + */ + @SneakyThrows + private ElasticsearchResponse handleHttpResponse(ClassicHttpResponse httpResponse) throws IOException { + final HttpEntity entity = httpResponse.getEntity(); + int statusCode = httpResponse.getCode(); + return new ElasticsearchResponse(EntityUtils.toString(entity), statusCode); + } + + private boolean isError(ElasticsearchResponse response) { + return isError(response.getStatusCode()); + } + + private boolean isError(int status) { + return status / 100 != 2; + } + + private ElasticsearchResponse handleESResponse(ElasticsearchResponse response, String uri, String content) { + if (isError(response) || StringUtils.contains(response.getMessage(), "\"errors\":true")) { + throw new IllegalStateException(String.format("Error message returned from ES: %s. URI: %s. Content: %s", + response.getMessage(), + uri, + content)); + } else if (StringUtils.contains(response.getMessage(), "\"type\":\"version_conflict_engine_exception\"")) { + LOG.warn("ID conflict in some content: {}", response.getMessage()); + } + return response; + } + + private void checkIndexTemplateExistence() { + if (!sendIsIndexTemplateExistsRequest()) { + String indexTemplate = elasticsearchConfiguration.getIndexTemplateName(); + sendPostRequest("_index_template/" + indexTemplate, elasticsearchConfiguration.getIndexTemplateMapping()); + if (sendIsIndexTemplateExistsRequest()) { + LOG.info("Index Template {} created.", indexTemplate); + } else { + throw new IllegalStateException("Error while creating Index Template " + indexTemplate); + } + } + } + + private void sendRolloverRequest() { + LOG.info("Analytics Indices rollover process start"); + ElasticsearchResponse response = sendGetRequest(elasticsearchConfiguration.getIndexPrefix() + + "_*?allow_no_indices=true&ignore_unavailable=true"); + String indexListJsonString = response.getMessage(); + JSONObject jsonObject = new JSONObject(indexListJsonString); + List outdatedIndices = jsonObject.keySet() + .stream() + .sorted((s1, s2) -> StringUtils.compare(s2, s1)) + .skip(elasticsearchConfiguration.getMaxIndexCount()) + .filter(Objects::nonNull) + .toList(); + while (!outdatedIndices.isEmpty()) { + List outdatedIndicesSubList = outdatedIndices.stream().limit(10).toList(); + String outdatedIndiceNames = StringUtils.join(outdatedIndicesSubList, ","); + LOG.info("Deleting {} outdated analytics Indices: [{}]", outdatedIndicesSubList.size(), outdatedIndiceNames); + sendDeleteRequest(outdatedIndiceNames); + outdatedIndices = outdatedIndices.stream().skip(10).toList(); + } + LOG.info("Analytics Indices rollover process finished successfully."); + } + + private final String getIndex() { + return getIndex(System.currentTimeMillis() / getIndexPerDaysMs()); + } + + @Cacheable("analytics.indexName") + private final String getIndex(long indexPeriodIndex) { + long periodEpochMs = indexPeriodIndex * getIndexPerDaysMs(); + String indexSuffix = DAY_DATE_FORMATTER.format(Instant.ofEpochMilli(periodEpochMs) + .atZone(ZoneOffset.UTC)); + return elasticsearchConfiguration.getIndexPrefix() + "_" + indexSuffix; + } + + private long getIndexPerDaysMs() { + return DAY_IN_MS * Math.max(elasticsearchConfiguration.getIndexPerDays(), 1); + } + +} diff --git a/analytics-services/src/main/java/io/meeds/analytics/elasticsearch/storage/ElasticsearchConfiguration.java b/analytics-services/src/main/java/io/meeds/analytics/elasticsearch/storage/ElasticsearchConfiguration.java new file mode 100644 index 000000000..4945c37a3 --- /dev/null +++ b/analytics-services/src/main/java/io/meeds/analytics/elasticsearch/storage/ElasticsearchConfiguration.java @@ -0,0 +1,187 @@ +/** + * This file is part of the Meeds project (https://meeds.io/). + * + * Copyright (C) 2020 - 2024 Meeds Association contact@meeds.io + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 3 of the License, or (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public License + * along with this program; if not, write to the Free Software Foundation, + * Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + */ +package io.meeds.analytics.elasticsearch.storage; + +import java.io.InputStream; + +import org.apache.commons.lang3.StringUtils; +import org.apache.hc.client5.http.auth.AuthScope; +import org.apache.hc.client5.http.auth.UsernamePasswordCredentials; +import org.apache.hc.client5.http.classic.HttpClient; +import org.apache.hc.client5.http.impl.auth.BasicCredentialsProvider; +import org.apache.hc.client5.http.impl.classic.HttpClientBuilder; +import org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager; +import org.apache.hc.client5.http.io.HttpClientConnectionManager; +import org.apache.hc.core5.http.impl.DefaultConnectionReuseStrategy; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.beans.factory.annotation.Value; +import org.springframework.context.annotation.Bean; +import org.springframework.stereotype.Component; + +import org.exoplatform.commons.api.settings.SettingService; +import org.exoplatform.commons.api.settings.SettingValue; +import org.exoplatform.commons.api.settings.data.Context; +import org.exoplatform.commons.api.settings.data.Scope; +import org.exoplatform.commons.utils.IOUtil; +import org.exoplatform.services.log.ExoLogger; +import org.exoplatform.services.log.Log; + +import jakarta.annotation.PostConstruct; +import lombok.Getter; +import lombok.SneakyThrows; + +@Component +public class ElasticsearchConfiguration { + + private static final Log LOG = + ExoLogger.getLogger(ElasticsearchConfiguration.class); + + private static final String DEFAULT_ES_CLIENT_SERVER_URL = "http://127.0.0.1:9200"; + + private static final String ES_CLIENT_SERVER_URL = "exo.es.index.server.url"; + + private static final String ES_CLIENT_USERNAME = "exo.es.index.server.username"; + + private static final String ES_CLIENT_PWD = "exo.es.index.server.password"; // NOSONAR + + public static final String ES_ANALYTICS_INDEX_TEMPLATE = "exo.es.analytics.index.template"; + + public static final String DEFAULT_ES_INDEX_TEMPLATE = "analytics_template"; + + public static final Context ES_ANALYTICS_CONTEXT = Context.GLOBAL.id("analytics"); + + public static final Scope ES_ANALYTICS_SCOPE = Scope.APPLICATION.id("analytics"); + + @Autowired + private SettingService settingService; + + @Getter + @Value("${analytics.es.index.prefix:analytics}") + private String indexPrefix; + + @Getter + @Value("${analytics.es.index.template:analytics_template}") + private String indexTemplateName; + + @Getter + @Value("${analytics.es.index.alias:analytics_alias}") + private String indexAlias; + + @Getter + @Value("${analytics.es.index.writePeriod:7}") + private long indexPerDays; + + @Getter + @Value("${analytics.es.index.maxCount:500}") + private long maxIndexCount; + + @Value("${analytics.es.replicas:0}") + private int replicas; + + @Value("${analytics.es.shards:1}") + private int shards; + + @Value("${analytics.es.index.template.path:analytics-es-template.json}") + private String indexTemplateMappingFilePath; + + @Getter + private String indexTemplateMapping; + + @Value("${analytics.es.index.server.username:}") + private String username; + + @Value("${analytics.es.index.server.password:}") + private String password; + + @Getter + @Value("${analytics.es.index.server.url:}") + private String urlClient; + + @SneakyThrows + @PostConstruct + public void init() { + computeConnectionProperties(); + computeIndexTemplateName(); + computeIndexTemplateMapping(); + } + + @Bean("elasticsearchHttpClient") + private HttpClient elasticsearchHttpClient() { + // Check if Basic Authentication need to be used + HttpClientConnectionManager clientConnectionManager = new PoolingHttpClientConnectionManager(); + HttpClientBuilder httpClientBuilder = HttpClientBuilder + .create() + .disableAutomaticRetries() + .setConnectionManager(clientConnectionManager) + .setConnectionReuseStrategy(new DefaultConnectionReuseStrategy()); + if (StringUtils.isNotBlank(username)) { + BasicCredentialsProvider credsProvider = new BasicCredentialsProvider(); + credsProvider.setCredentials( + new AuthScope(null, -1), + new UsernamePasswordCredentials(username, + password.toCharArray())); + return httpClientBuilder.setDefaultCredentialsProvider(credsProvider) + .build(); + } else { + return httpClientBuilder.build(); + } + } + + @SneakyThrows + private void computeIndexTemplateMapping() { + InputStream mappingFileIS = getClass().getClassLoader().getResourceAsStream("analytics-es-template.json"); + indexTemplateMapping = IOUtil.getStreamContentAsString(mappingFileIS); + indexTemplateMapping = indexTemplateMapping.replace(ElasticsearchConfiguration.DEFAULT_ES_INDEX_TEMPLATE, + getIndexAlias()) + .replace("replica.number", + String.valueOf(replicas)) + .replace("shard.number", + String.valueOf(shards)); + } + + private void computeConnectionProperties() { + if (StringUtils.isBlank(this.urlClient)) { + this.urlClient = System.getProperty(ES_CLIENT_SERVER_URL); + this.username = System.getProperty(ES_CLIENT_USERNAME); + this.password = System.getProperty(ES_CLIENT_PWD); + } + if (StringUtils.isBlank(this.urlClient)) { + this.urlClient = DEFAULT_ES_CLIENT_SERVER_URL; + } + } + + private void computeIndexTemplateName() { + SettingValue indexTemplateValue = this.settingService.get(ES_ANALYTICS_CONTEXT, + ES_ANALYTICS_SCOPE, + ES_ANALYTICS_INDEX_TEMPLATE); + if (indexTemplateValue == null || indexTemplateValue.getValue() == null) { + this.settingService.set(ES_ANALYTICS_CONTEXT, + ES_ANALYTICS_SCOPE, + ES_ANALYTICS_INDEX_TEMPLATE, + SettingValue.create(indexTemplateName)); + } else { + String storedIndexTemplate = indexTemplateValue.getValue().toString(); + if (!StringUtils.equals(storedIndexTemplate, indexTemplateName)) { + LOG.warn("Can't change index template from {} to {}. New index will be ignored.", storedIndexTemplate, indexTemplateName); + indexTemplateName = storedIndexTemplate; + } + } + } +} diff --git a/analytics-services/src/main/java/org/exoplatform/analytics/queue/service/DummyStatisticDataQueueService.java b/analytics-services/src/main/java/io/meeds/analytics/queue/service/DummyStatisticDataQueueService.java similarity index 71% rename from analytics-services/src/main/java/org/exoplatform/analytics/queue/service/DummyStatisticDataQueueService.java rename to analytics-services/src/main/java/io/meeds/analytics/queue/service/DummyStatisticDataQueueService.java index 5913dd972..9e96888de 100644 --- a/analytics-services/src/main/java/org/exoplatform/analytics/queue/service/DummyStatisticDataQueueService.java +++ b/analytics-services/src/main/java/io/meeds/analytics/queue/service/DummyStatisticDataQueueService.java @@ -1,55 +1,70 @@ /** * This file is part of the Meeds project (https://meeds.io/). - * Copyright (C) 2022 Meeds Association - * contact@meeds.io + * + * Copyright (C) 2020 - 2024 Meeds Association contact@meeds.io + * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU Lesser General Public * License as published by the Free Software Foundation; either * version 3 of the License, or (at your option) any later version. + * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU * Lesser General Public License for more details. + * * You should have received a copy of the GNU Lesser General Public License * along with this program; if not, write to the Free Software Foundation, * Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. */ -package org.exoplatform.analytics.queue.service; +package io.meeds.analytics.queue.service; import java.math.BigInteger; import java.util.List; -import java.util.concurrent.*; +import java.util.concurrent.Executors; +import java.util.concurrent.ScheduledExecutorService; +import java.util.concurrent.TimeUnit; -import org.picocontainer.Startable; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.context.annotation.Primary; +import org.springframework.stereotype.Service; import com.google.common.util.concurrent.ThreadFactoryBuilder; -import org.exoplatform.analytics.api.service.StatisticDataProcessorService; -import org.exoplatform.analytics.api.service.StatisticDataQueueService; -import org.exoplatform.analytics.model.StatisticData; -import org.exoplatform.analytics.model.StatisticDataQueueEntry; -import org.exoplatform.container.ExoContainerContext; -import org.exoplatform.container.PortalContainer; -import org.exoplatform.container.component.RequestLifeCycle; import org.exoplatform.services.cache.CacheService; import org.exoplatform.services.cache.ExoCache; import org.exoplatform.services.log.ExoLogger; import org.exoplatform.services.log.Log; -public class DummyStatisticDataQueueService implements StatisticDataQueueService, Startable { +import io.meeds.analytics.api.service.StatisticDataProcessorService; +import io.meeds.analytics.api.service.StatisticDataQueueService; +import io.meeds.analytics.model.StatisticData; +import io.meeds.analytics.model.StatisticDataQueueEntry; +import io.meeds.common.ContainerTransactional; + +import jakarta.annotation.PostConstruct; +import jakarta.annotation.PreDestroy; + +@Primary +@Service +public class DummyStatisticDataQueueService implements StatisticDataQueueService { private static final Log LOG = ExoLogger.getLogger(DummyStatisticDataQueueService.class); private static final String ANALYTICS_QUEUE_CACHE_NAME = "analytics.queue"; - private ExoCache statisticQueueCache = null; - + @Autowired private StatisticDataProcessorService statisticDataProcessorService; - private ScheduledExecutorService queueProcessingExecutor = null; + @Autowired + private CacheService cacheService = null; - private PortalContainer container = null; + private ExoCache statisticQueueCache = null; + + private ScheduledExecutorService queueProcessingExecutor = + Executors.newSingleThreadScheduledExecutor(new ThreadFactoryBuilder().setNameFormat("Analytics-ingestor-%d") + .build()); private BigInteger totalExecutionTime = BigInteger.ZERO; @@ -57,29 +72,21 @@ public class DummyStatisticDataQueueService implements StatisticDataQueueService private long executionCount = 0; - public DummyStatisticDataQueueService(PortalContainer container, - StatisticDataProcessorService statisticDataProcessorService, - CacheService cacheService) { - this.statisticDataProcessorService = statisticDataProcessorService; - this.container = container; + @PostConstruct + public void init() { this.statisticQueueCache = cacheService.getCacheInstance(ANALYTICS_QUEUE_CACHE_NAME); + // Can't be job, because each cluster node must process its in-memory queue + this.queueProcessingExecutor.scheduleAtFixedRate(this::processQueueTransactional, 0, 10, TimeUnit.SECONDS); + } - ThreadFactory namedThreadFactory = new ThreadFactoryBuilder().setNameFormat("Analytics-ingestor-%d").build(); - this.queueProcessingExecutor = Executors.newSingleThreadScheduledExecutor(namedThreadFactory); + @PreDestroy + public void shutdown() { + queueProcessingExecutor.shutdown(); } - @Override - public void start() { - // Can't be job, because each cluster node must process its in-memory queue - queueProcessingExecutor.scheduleAtFixedRate(() -> { - ExoContainerContext.setCurrentContainer(this.container); - RequestLifeCycle.begin(this.container); - try { - processQueue(); - } finally { - RequestLifeCycle.end(); - } - }, 0, 10, TimeUnit.SECONDS); + @ContainerTransactional + public void processQueueTransactional() { + processQueue(); } @Override @@ -103,11 +110,6 @@ public void processQueue() { } } - @Override - public void stop() { - queueProcessingExecutor.shutdown(); - } - @Override public void put(StatisticData data) { StatisticDataQueueEntry statisticDataQueueEntry = new StatisticDataQueueEntry(data); diff --git a/analytics-services/src/main/java/org/exoplatform/analytics/es/AnalyticsESClient.java b/analytics-services/src/main/java/org/exoplatform/analytics/es/AnalyticsESClient.java deleted file mode 100644 index bd6bfb783..000000000 --- a/analytics-services/src/main/java/org/exoplatform/analytics/es/AnalyticsESClient.java +++ /dev/null @@ -1,429 +0,0 @@ -/** - * This file is part of the Meeds project (https://meeds.io/). - * Copyright (C) 2022 Meeds Association - * contact@meeds.io - * This program is free software; you can redistribute it and/or - * modify it under the terms of the GNU Lesser General Public - * License as published by the Free Software Foundation; either - * version 3 of the License, or (at your option) any later version. - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * Lesser General Public License for more details. - * You should have received a copy of the GNU Lesser General Public License - * along with this program; if not, write to the Free Software Foundation, - * Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. - */ -package org.exoplatform.analytics.es; - -import java.io.InputStream; -import java.time.Instant; -import java.time.ZoneOffset; -import java.time.format.DateTimeFormatter; -import java.time.format.ResolverStyle; -import java.util.HashMap; -import java.util.HashSet; -import java.util.List; -import java.util.Map; -import java.util.Set; - -import org.apache.commons.httpclient.HttpStatus; -import org.apache.commons.lang3.StringUtils; -import org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager; -import org.apache.hc.client5.http.io.HttpClientConnectionManager; -import org.json.JSONException; - -import org.exoplatform.analytics.model.StatisticDataQueueEntry; -import org.exoplatform.commons.search.es.client.ElasticClient; -import org.exoplatform.commons.search.es.client.ElasticClientException; -import org.exoplatform.commons.search.es.client.ElasticIndexingAuditTrail; -import org.exoplatform.commons.search.es.client.ElasticResponse; -import org.exoplatform.commons.utils.IOUtil; -import org.exoplatform.container.configuration.ConfigurationManager; -import org.exoplatform.container.xml.InitParams; -import org.exoplatform.services.log.ExoLogger; -import org.exoplatform.services.log.Log; - -public class AnalyticsESClient extends ElasticClient { - - private static final Log LOG = - ExoLogger.getExoLogger(AnalyticsESClient.class); - - private static final String DEFAULT_ES_CLIENT_SERVER_URL = "http://127.0.0.1:9200"; - - private static final String ES_CLIENT_SERVER_URL = "exo.es.index.server.url"; - - private static final String ES_CLIENT_USERNAME = "exo.es.index.server.username"; - - private static final String ES_CLIENT_PWD = "exo.es.index.server.password"; // NOSONAR - - private static final String ES_ANALYTICS_CLIENT_SERVER_URL = "exo.es.analytics.index.server.url"; - - private static final String ES_ANALYTICS_CLIENT_USERNAME = "exo.es.analytics.index.server.username"; - - private static final String ES_ANALYTICS_CLIENT_PWD = "exo.es.analytics.index.server.password"; // NOSONAR - - private static final String INDEX_TEMPLATE_FILE_PATH_PARAM = "index.template.file.path"; - - private static final String ES_ANALYTICS_INDEX_PER_DAYS = "exo.es.analytics.index.per.days"; - - private static final long DAY_IN_MS = 86400000L; - - private static final String DAY_DATE_FORMAT = "yyyy-MM-dd"; - - public static final DateTimeFormatter DAY_DATE_FORMATTER = DateTimeFormatter.ofPattern(DAY_DATE_FORMAT) - .withResolverStyle(ResolverStyle.LENIENT); - - private AnalyticsIndexingServiceConnector analyticsIndexingConnector; - - private AnalyticsElasticContentRequestBuilder elasticContentRequestBuilder; - - private Set existingIndexes = new HashSet<>(); - - private Map indexSuffixPerDayIndice = new HashMap<>(); - - private int indexPerDays; - - private String esIndexTemplateQuery; - - private String username; - - private String password; - - public AnalyticsESClient(ConfigurationManager configurationManager, // NOSONAR - AnalyticsElasticContentRequestBuilder elasticContentRequestBuilder, - AnalyticsIndexingServiceConnector analyticsIndexingConnector, - ElasticIndexingAuditTrail auditTrail, - InitParams initParams) { - super(auditTrail); - this.analyticsIndexingConnector = analyticsIndexingConnector; - this.elasticContentRequestBuilder = elasticContentRequestBuilder; - - if (initParams != null) { - if (initParams.containsKey(ES_ANALYTICS_CLIENT_SERVER_URL)) { - this.urlClient = initParams.getValueParam(ES_ANALYTICS_CLIENT_SERVER_URL).getValue(); - } - if (initParams.containsKey(ES_ANALYTICS_CLIENT_USERNAME)) { - this.username = initParams.getValueParam(ES_ANALYTICS_CLIENT_USERNAME).getValue(); - } - if (initParams.containsKey(ES_ANALYTICS_CLIENT_PWD)) { - this.password = initParams.getValueParam(ES_ANALYTICS_CLIENT_PWD).getValue(); - } - if (initParams.containsKey(INDEX_TEMPLATE_FILE_PATH_PARAM)) { - String mappingFilePath = initParams.getValueParam(INDEX_TEMPLATE_FILE_PATH_PARAM).getValue(); - try { - this.esIndexTemplateQuery = getFileContent(configurationManager, mappingFilePath); - } catch (Exception e) { - LOG.error("Can't read elasticsearch index mapping from path {}", mappingFilePath, e); - } - } - if (initParams.containsKey(ES_ANALYTICS_INDEX_PER_DAYS)) { - this.indexPerDays = Integer.parseInt(initParams.getValueParam(ES_ANALYTICS_INDEX_PER_DAYS).getValue()); - } - } - if (StringUtils.isBlank(this.urlClient)) { - this.urlClient = System.getProperty(ES_CLIENT_SERVER_URL); - this.username = System.getProperty(ES_CLIENT_USERNAME); - this.password = System.getProperty(ES_CLIENT_PWD); - } - if (StringUtils.isBlank(this.urlClient)) { - this.urlClient = DEFAULT_ES_CLIENT_SERVER_URL; - } - if (StringUtils.isBlank(this.esIndexTemplateQuery)) { - LOG.error("Empty elasticsearch index mapping file path parameter"); - } - initHttpClient(); - } - - public void init() { - checkIndexTemplateExistence(); - LOG.info("Analytics client initialized and is ready to proceed analytics data"); - } - - public boolean sendCreateIndexRequest(String index) { - if (sendIsIndexExistsRequest(index)) { - LOG.debug("Index {} already exists. Index creation requests will not be sent.", index); - return false; - } else { - sendTurnOffWriteOnAllAnalyticsIndexes(); - - String esIndexSettings = elasticContentRequestBuilder.getCreateIndexRequestContent(analyticsIndexingConnector); - ElasticResponse createIndexResponse = sendHttpPutRequest(index, esIndexSettings); - - if (sendIsIndexExistsRequest(index)) { - LOG.info("New analytics index {} created.", index); - return true; - } else { - throw new IllegalStateException("Error creating index " + index + " on elasticsearch, response code = " - + createIndexResponse.getStatusCode() + " , response content : " - + createIndexResponse.getMessage()); - } - } - } - - public void sendTurnOffWriteOnAllAnalyticsIndexes() { - if (sendIsIndexExistsRequest(analyticsIndexingConnector.getIndexAlias())) { - String esQuery = elasticContentRequestBuilder.getTurnOffWriteOnAllAnalyticsIndexes(analyticsIndexingConnector); - try { - sendHttpPostRequest("_aliases", esQuery); - LOG.info("All analytics indexes switched to RO mode to prepare creation of a new index"); - } catch (ElasticClientException e) { - LOG.warn("Analytics old indexes seems to not be turned off on write access"); - } - } - } - - public boolean sendIsIndexExistsRequest(String esIndex) { - if (existingIndexes.contains(esIndex)) { - return true; - } - String url = urlClient + "/" + esIndex; - ElasticResponse responseExists = super.sendHttpGetRequest(url); - boolean indexExists = responseExists.getStatusCode() == HttpStatus.SC_OK; - if (indexExists) { - existingIndexes.add(esIndex); - } - return indexExists; - } - - public boolean sendIsIndexTemplateExistsRequest() { - String url = urlClient + "/_index_template/" + analyticsIndexingConnector.getIndexTemplate(); - ElasticResponse responseExists = super.sendHttpGetRequest(url); - return responseExists.getStatusCode() == HttpStatus.SC_OK; - } - - public void sendCreateBulkDocumentsRequest(List dataQueueEntries) { - if (dataQueueEntries == null || dataQueueEntries.isEmpty()) { - return; - } - - LOG.info("Indexing in bulk {} documents", dataQueueEntries.size()); - - checkIndexExistence(dataQueueEntries); - - StringBuilder request = new StringBuilder(); - for (StatisticDataQueueEntry statisticDataQueueEntry : dataQueueEntries) { - String documentId = String.valueOf(statisticDataQueueEntry.getId()); - String singleDocumentQuery = elasticContentRequestBuilder.getCreateDocumentRequestContent(analyticsIndexingConnector, - documentId); - request.append(singleDocumentQuery); - } - - LOG.debug("Create documents request to ES: {}", request); - sendHttpPutRequest("_bulk", request.toString()); - - refreshIndex(); - } - - public String sendRequest(String esQuery) { - ElasticResponse elasticResponse = sendHttpPostRequest(analyticsIndexingConnector.getIndexAlias() + "/_search", esQuery); - String response = elasticResponse.getMessage(); - int statusCode = elasticResponse.getStatusCode(); - if (ElasticIndexingAuditTrail.isError(statusCode) || StringUtils.isBlank(response)) { - if (StringUtils.isBlank(response)) { - response = "Empty response was sent by ES"; - } - } else { - org.json.JSONObject json = null; - try { - json = new org.json.JSONObject(response); - if (json.has("status") && ElasticIndexingAuditTrail.isError(json.getInt("status"))) { - throw new IllegalStateException("Error occured while requesting ES HTTP error code: '" + statusCode - + "', HTTP response: '" - + response + "'"); - } - } catch (JSONException e) { - throw new IllegalStateException("Error occured while requesting ES HTTP code: '" + statusCode - + "', Error parsing response to JSON format, content = '" + response + "'", e); - } - } - return response; - } - - public String retrieveAllAnalyticsIndexesMapping() { - String url = urlClient + "/" + analyticsIndexingConnector.getIndexAlias() + "/_mapping"; - ElasticResponse response = super.sendHttpGetRequest(url); - if (ElasticIndexingAuditTrail.isError(response.getStatusCode())) { - LOG.warn("Error getting mapping of analytics : - \t\tcode : {} - \t\tmessage: {}", - response.getStatusCode(), - response.getMessage()); - return null; - } else { - return response.getMessage(); - } - } - - @Override - protected ElasticResponse sendHttpHeadRequest(String uri) { - String url = urlClient + "/" + uri; - ElasticResponse response = super.sendHttpHeadRequest(url); - try { - handleESResponse(response); - } catch (Exception e) { - throw new ElasticClientException("Error sending HEAD request '" + uri + "'", e); - } - return response; - } - - @Override - public ElasticResponse sendHttpGetRequest(String uri) { - return sendHttpGetRequest(null, uri); - } - - public ElasticResponse sendHttpGetRequest(String urlClient, String uri) { - if (StringUtils.isBlank(urlClient)) { - urlClient = this.urlClient; - } - String url = urlClient + "/" + uri; - ElasticResponse response = super.sendHttpGetRequest(url); - try { - handleESResponse(response); - } catch (Exception e) { - throw new ElasticClientException("Error sending GET request '" + url + "'", e); - } - return response; - } - - @Override - public ElasticResponse sendHttpPutRequest(String uri, String content) { - String url = urlClient + "/" + uri; - ElasticResponse response = super.sendHttpPutRequest(url, content); - try { - handleESResponse(response); - } catch (Exception e) { - throw new ElasticClientException("Error sending PUT request '" + url + "' with content = '" + content + "'", e); - } - return response; - } - - @Override - public ElasticResponse sendHttpDeleteRequest(String uri) { - String url = urlClient + "/" + uri; - ElasticResponse response = super.sendHttpDeleteRequest(url); - try { - handleESResponse(response); - } catch (Exception e) { - throw new ElasticClientException("Error sending 'DELETE' request '" + url + "'", e); - } - return response; - } - - @Override - public ElasticResponse sendHttpPostRequest(String uri, String content) { - String url = urlClient + "/" + uri; - ElasticResponse response = super.sendHttpPostRequest(url, content); - try { - handleESResponse(response); - } catch (Exception e) { - throw new ElasticClientException("Error sending POST request '" + url + "' with content = '" + content + "'", e); - } - return response; - } - - public String getIndexSuffix(long timestamp) { - long indexSuffixLong = timestamp / (DAY_IN_MS * indexPerDays); - String indexSuffix = indexSuffixPerDayIndice.get(indexSuffixLong); - if (indexSuffix != null) { - return indexSuffix; - } - indexSuffix = DAY_DATE_FORMATTER.format(Instant.ofEpochMilli(timestamp).atZone(ZoneOffset.UTC)); - indexSuffixPerDayIndice.put(indexSuffixLong, indexSuffix); - return indexSuffix; - } - - public void refreshIndex() { - refreshIndex(analyticsIndexingConnector.getIndexAlias()); - } - - public void refreshIndex(String index) { - sendHttpPostRequest(index + "/_refresh", null); - } - - public int getIndexPerDays() { - return indexPerDays; - } - - @Override - protected String getEsUsernameProperty() { - return username; - } - - @Override - protected String getEsPasswordProperty() { - return password; - } - - @Override - protected HttpClientConnectionManager getClientConnectionManager() { - return new PoolingHttpClientConnectionManager(); - } - - private void handleESResponse(ElasticResponse response) { - if (response.getStatusCode() != 200) { - throw new ElasticClientException(response.getMessage()); - } - if (StringUtils.contains(response.getMessage(), "\"type\":\"version_conflict_engine_exception\"")) { - LOG.warn("ID conflict in some content", response.getMessage()); - return; - } - if (response.getStatusCode() != 200 || StringUtils.contains(response.getMessage(), "\"errors\":true")) { - throw new ElasticClientException(response.getMessage()); - } - } - - private void checkIndexExistence(List dataQueueEntries) { - Set indexes = new HashSet<>(); - for (StatisticDataQueueEntry statisticDataQueueEntry : dataQueueEntries) { - long timestamp = statisticDataQueueEntry.getStatisticData().getTimestamp(); - indexes.add(getIndex(timestamp)); - } - for (String index : indexes) { - sendCreateIndexRequest(index); - } - } - - private void checkIndexTemplateExistence() { - String indexTemplate = analyticsIndexingConnector.getIndexTemplate(); - if (sendIsIndexTemplateExistsRequest()) { - LOG.debug("Index Template {} already exists. Index creation requests will not be sent.", - indexTemplate); - } else { - long startTime = System.currentTimeMillis(); - esIndexTemplateQuery = esIndexTemplateQuery.replace(AnalyticsIndexingServiceConnector.DEFAULT_ES_INDEX_TEMPLATE, - analyticsIndexingConnector.getIndexAlias()) - .replace("replica.number", - String.valueOf(analyticsIndexingConnector.getReplicas())) - .replace("shard.number", String.valueOf(analyticsIndexingConnector.getShards())); - ElasticResponse responseCreate = sendHttpPostRequest("_index_template/" + indexTemplate, esIndexTemplateQuery); - auditTrail.audit("create_index_template", - null, - indexTemplate, - responseCreate.getStatusCode(), - responseCreate.getMessage(), - (System.currentTimeMillis() - startTime)); - - if (sendIsIndexTemplateExistsRequest()) { - LOG.info("Index Template {} created.", indexTemplate); - analyticsIndexingConnector.storeCreatedIndexTemplate(); - } else { - throw new IllegalStateException("Index Template " + indexTemplate - + " isn't created successfully"); - } - } - } - - private final String getIndex(long timestamp) { - if (indexPerDays > 0) { - String indexSuffix = getIndexSuffix(timestamp); - return analyticsIndexingConnector.getIndexPrefix() + "_" + indexSuffix; - } else { - return null; - } - } - - private String getFileContent(ConfigurationManager configurationManager, String filePath) throws Exception { - InputStream mappingFileIS = configurationManager.getInputStream(filePath); - return IOUtil.getStreamContentAsString(mappingFileIS); - } - -} diff --git a/analytics-services/src/main/java/org/exoplatform/analytics/es/AnalyticsElasticContentRequestBuilder.java b/analytics-services/src/main/java/org/exoplatform/analytics/es/AnalyticsElasticContentRequestBuilder.java deleted file mode 100644 index 9a92ccf19..000000000 --- a/analytics-services/src/main/java/org/exoplatform/analytics/es/AnalyticsElasticContentRequestBuilder.java +++ /dev/null @@ -1,48 +0,0 @@ -/** - * This file is part of the Meeds project (https://meeds.io/). - * Copyright (C) 2022 Meeds Association - * contact@meeds.io - * This program is free software; you can redistribute it and/or - * modify it under the terms of the GNU Lesser General Public - * License as published by the Free Software Foundation; either - * version 3 of the License, or (at your option) any later version. - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * Lesser General Public License for more details. - * You should have received a copy of the GNU Lesser General Public License - * along with this program; if not, write to the Free Software Foundation, - * Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. - */ -package org.exoplatform.analytics.es; - -import org.exoplatform.commons.search.es.client.ElasticContentRequestBuilder; -import org.exoplatform.commons.search.index.impl.ElasticIndexingServiceConnector; - -public class AnalyticsElasticContentRequestBuilder extends ElasticContentRequestBuilder { - - @Override - public String getCreateIndexRequestContent(ElasticIndexingServiceConnector connector) { - return " {" + - "\"aliases\": {" + - " \"" + connector.getIndexAlias() + "\": {" + - " \"is_write_index\" : true" + - " }" + - "}" + - "}"; - } - - public String getTurnOffWriteOnAllAnalyticsIndexes(AnalyticsIndexingServiceConnector connector) { - return "{" + - "\"actions\": [" + - " {" + - " \"add\": {" + - " \"index\": \"" + connector.getIndexPrefix() + "*\"," + - " \"alias\": \"" + connector.getIndexAlias() + "\"," + - " \"is_write_index\": false" + - " }" + - " }" + - "]" + - "}"; - } -} diff --git a/analytics-services/src/main/java/org/exoplatform/analytics/es/AnalyticsIndexingServiceConnector.java b/analytics-services/src/main/java/org/exoplatform/analytics/es/AnalyticsIndexingServiceConnector.java deleted file mode 100644 index 632fea0ee..000000000 --- a/analytics-services/src/main/java/org/exoplatform/analytics/es/AnalyticsIndexingServiceConnector.java +++ /dev/null @@ -1,178 +0,0 @@ -/** - * This file is part of the Meeds project (https://meeds.io/). - * Copyright (C) 2022 Meeds Association - * contact@meeds.io - * This program is free software; you can redistribute it and/or - * modify it under the terms of the GNU Lesser General Public - * License as published by the Free Software Foundation; either - * version 3 of the License, or (at your option) any later version. - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * Lesser General Public License for more details. - * You should have received a copy of the GNU Lesser General Public License - * along with this program; if not, write to the Free Software Foundation, - * Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. - */ -package org.exoplatform.analytics.es; - -import static org.exoplatform.analytics.utils.AnalyticsUtils.*; - -import java.util.*; - -import org.apache.commons.lang3.StringUtils; -import org.picocontainer.Startable; - -import org.exoplatform.analytics.api.service.StatisticDataQueueService; -import org.exoplatform.analytics.model.StatisticData; -import org.exoplatform.commons.api.settings.SettingService; -import org.exoplatform.commons.api.settings.SettingValue; -import org.exoplatform.commons.api.settings.data.Context; -import org.exoplatform.commons.api.settings.data.Scope; -import org.exoplatform.commons.search.domain.Document; -import org.exoplatform.commons.search.index.impl.ElasticIndexingServiceConnector; -import org.exoplatform.container.xml.InitParams; -import org.exoplatform.services.log.ExoLogger; -import org.exoplatform.services.log.Log; - -public class AnalyticsIndexingServiceConnector extends ElasticIndexingServiceConnector implements Startable { - - private static final Log LOG = - ExoLogger.getLogger(AnalyticsIndexingServiceConnector.class); - - public static final String DEFAULT_ES_INDEX_TEMPLATE = "analytics_template"; - - public static final String DEFAULT_ES_ANALYTICS_INDEX_NAME = "analytics"; - - public static final String ES_ANALYTICS_INDEX_PREFIX = "exo.es.analytics.index.prefix"; - - public static final String ES_ANALYTICS_INDEX_TEMPLATE = "exo.es.analytics.index.template"; - - public static final Context ES_ANALYTICS_CONTEXT = Context.GLOBAL.id("analytics"); - - public static final Scope ES_ANALYTICS_SCOPE = Scope.APPLICATION.id("analytics"); - - private SettingService settingService; - - private StatisticDataQueueService analyticsQueueService; - - private String indexPrefix; - - private String indexTemplate; - - public AnalyticsIndexingServiceConnector(StatisticDataQueueService analyticsQueueService, - SettingService settingService, - InitParams initParams) { - super(initParams); - this.settingService = settingService; - this.analyticsQueueService = analyticsQueueService; - if (initParams != null) { - if (initParams.containsKey(ES_ANALYTICS_INDEX_PREFIX)) { - this.indexPrefix = initParams.getValueParam(ES_ANALYTICS_INDEX_PREFIX).getValue(); - } - if (initParams.containsKey(ES_ANALYTICS_INDEX_TEMPLATE)) { - this.indexTemplate = initParams.getValueParam(ES_ANALYTICS_INDEX_TEMPLATE).getValue(); - } - } - if (StringUtils.isBlank(this.indexPrefix)) { - this.indexPrefix = DEFAULT_ES_ANALYTICS_INDEX_NAME; - } - if (StringUtils.isBlank(this.indexTemplate)) { - this.indexTemplate = DEFAULT_ES_INDEX_TEMPLATE; - } - } - - @Override - public void start() { - SettingValue indexTemplateValue = this.settingService.get(ES_ANALYTICS_CONTEXT, - ES_ANALYTICS_SCOPE, - ES_ANALYTICS_INDEX_TEMPLATE); - if (indexTemplateValue != null && indexTemplateValue.getValue() != null) { - String storedIndexTemplate = indexTemplateValue.getValue().toString(); - if (!StringUtils.equals(storedIndexTemplate, indexTemplate)) { - LOG.warn("Can't change index template from {} to {}. New index will be ignored.", storedIndexTemplate, indexTemplate); - indexTemplate = storedIndexTemplate; - } - } - } - - @Override - public void stop() { - // Nothing to stop - } - - @Override - public String getConnectorName() { - throw new UnsupportedOperationException(); - } - - @Override - public String getMapping() { - throw new UnsupportedOperationException(); - } - - @Override - public Document create(String idString) { - if (StringUtils.isBlank(idString)) { - throw new IllegalArgumentException("id is mandatory"); - } - long id = Long.parseLong(idString); - StatisticData data = this.analyticsQueueService.get(id); - if (data == null) { - LOG.warn("Can't find document with id {}", id); - return null; - } - String timestampString = String.valueOf(data.getTimestamp()); - - Map fields = new HashMap<>(); - fields.put("id", idString); - fields.put(FIELD_TIMESTAMP, timestampString); - fields.put(FIELD_USER_ID, String.valueOf(data.getUserId())); - fields.put(FIELD_SPACE_ID, String.valueOf(data.getSpaceId())); - fields.put(FIELD_MODULE, data.getModule()); - fields.put(FIELD_SUB_MODULE, data.getSubModule()); - fields.put(FIELD_OPERATION, data.getOperation()); - fields.put(FIELD_STATUS, String.valueOf(data.getStatus().ordinal())); - fields.put(FIELD_ERROR_CODE, String.valueOf(data.getErrorCode())); - fields.put(FIELD_ERROR_MESSAGE, data.getErrorMessage()); - fields.put(FIELD_DURATION, String.valueOf(data.getDuration())); - fields.put(FIELD_IS_ANALYTICS, "true"); - if (data.getParameters() != null && !data.getParameters().isEmpty()) { - fields.putAll(data.getParameters()); - } - Document esDocument = new Document(String.valueOf(id), - null, - null, - (Set) null, - fields); - if (data.getListParameters() != null && !data.getListParameters().isEmpty()) { - esDocument.setListFields(data.getListParameters()); - } - return esDocument; - } - - @Override - public Document update(String id) { - throw new UnsupportedOperationException(); - } - - @Override - public List getAllIds(int offset, int limit) { - throw new UnsupportedOperationException(); - } - - public String getIndexPrefix() { - return indexPrefix; - } - - public String getIndexTemplate() { - return indexTemplate; - } - - public void storeCreatedIndexTemplate() { - this.settingService.set(ES_ANALYTICS_CONTEXT, - ES_ANALYTICS_SCOPE, - ES_ANALYTICS_INDEX_TEMPLATE, - SettingValue.create(indexTemplate)); - } -} diff --git a/analytics-services/src/main/resources/analytics-ui-watchers.json b/analytics-services/src/main/resources/analytics-ui-watchers.json new file mode 100644 index 000000000..b9ea3417d --- /dev/null +++ b/analytics-services/src/main/resources/analytics-ui-watchers.json @@ -0,0 +1,14 @@ +{ + "descriptors":[ + { + "name":"Unified search click", + "operation":"click", + "parameters":{ + "ui":"toolbar", + "application":"SearchPortlet" + }, + "domSelector":"#SearchApplication", + "domEvent":"mousedown" + } + ] +} \ No newline at end of file diff --git a/analytics-services/src/main/resources/conf/portal/configuration.xml b/analytics-services/src/main/resources/conf/portal/configuration.xml deleted file mode 100644 index c2efd1e57..000000000 --- a/analytics-services/src/main/resources/conf/portal/configuration.xml +++ /dev/null @@ -1,122 +0,0 @@ - - - - - - org.exoplatform.analytics.es.AnalyticsElasticContentRequestBuilder - org.exoplatform.analytics.es.AnalyticsElasticContentRequestBuilder - - - - org.exoplatform.analytics.es.AnalyticsESClient - - - exo.es.analytics.index.server.url - ${exo.es.analytics.index.server.url:} - - - exo.es.analytics.index.server.username - ${exo.es.analytics.index.server.username:} - - - exo.es.analytics.index.server.password - ${exo.es.analytics.index.server.password:} - - - exo.es.analytics.index.per.days - ${exo.es.analytics.index.per.days:7} - - - index.template.file.path - ${exo.es.analytics.index.template.path:jar:/analytics-es-template.json} - - - - - - org.exoplatform.analytics.api.service.AnalyticsService - org.exoplatform.analytics.es.service.ESAnalyticsService - - - exo.analytics.aggregation.terms.doc_size - ${exo.analytics.aggregation.terms.doc_size:200} - - - exo.analytics.admin.permissions - ${exo.analytics.admin.permission:*:/platform/analytics} - - - exo.analytics.viewall.permissions - ${exo.analytics.viewall.permission:*:/platform/administrators} - - - exo.analytics.view.permissions - ${exo.analytics.view.permission:*:/platform/users} - - - - - - org.exoplatform.analytics.api.websocket.AnalyticsWebSocketService - - - - org.exoplatform.analytics.es.AnalyticsIndexingServiceConnector - - - exo.es.analytics.index.prefix - ${exo.es.analytics.index.prefix:analytics} - - - exo.es.analytics.index.template - ${exo.es.analytics.index.template:analytics_template} - - - constructor.params - - - - - - - - - - org.exoplatform.analytics.api.service.StatisticDataQueueService - org.exoplatform.analytics.queue.service.DummyStatisticDataQueueService - - - - org.exoplatform.analytics.api.service.StatisticDataProcessorService - - - - org.exoplatform.analytics.api.service.ManagedStatisticDataQueueService - - - - org.exoplatform.analytics.api.service.StatisticDataProcessorService - - elasticsearch - addProcessor - org.exoplatform.analytics.es.processor.ElasticSearchStatisticDataProcessor - ElasticSearch analytics data persister - - - - diff --git a/analytics-webapps/pom.xml b/analytics-webapps/pom.xml index adce243d6..e030c8acd 100644 --- a/analytics-webapps/pom.xml +++ b/analytics-webapps/pom.xml @@ -1,16 +1,19 @@ - - - - org.exoplatform.analytics.api.service.AnalyticsService - - Home page link - addUIWatcherPlugin - org.exoplatform.analytics.api.service.StatisticUIWatcherPlugin - - - watcher - - - - Home page link - - - - click - - - - - - - ui - - - toolbar - - - - - application - - - NavigationToolbarPortlet - - - - - - - #brandingTopBar a - - - - mousedown - - - - - - type - - - pageX - - - pageY - - - - - - - - href - - - class - - - id - - - - - - - - - Hamburger menu site navigation link - addUIWatcherPlugin - org.exoplatform.analytics.api.service.StatisticUIWatcherPlugin - - - watcher - - - - Hamburger menu site navigation link - - - - click - - - - - - - ui - - - HamburgerMenu - - - - - application - - - SiteHamburgerNavigation - - - - - - - #HamburgerMenuNavigation #SiteHamburgerNavigation a[role="listitem"] - - - - mousedown - - - - - - type - - - pageX - - - pageY - - - - - - - - href - - - aria-selected - - - - - - - - - Hamburger menu spaces navigation link - addUIWatcherPlugin - org.exoplatform.analytics.api.service.StatisticUIWatcherPlugin - - - watcher - - - - Hamburger menu spaces navigation link - - - - click - - - - - - - ui - - - HamburgerMenu - - - - - application - - - SpacesHamburgerNavigation - - - - - - - #HamburgerMenuNavigation .spacesNavigationContent a[role="listitem"] - - - - mousedown - - - - - - type - - - pageX - - - pageY - - - - - - - - href - - - aria-selected - - - - - - - - - Hamburger menu icon - addUIWatcherPlugin - org.exoplatform.analytics.api.service.StatisticUIWatcherPlugin - - - watcher - - - - Hamburger menu icon - - - - click - - - - - - - ui - - - toolbar - - - - - application - - - HamburgerNavigationMenuLink - - - - - - - .HamburgerNavigationMenuLink - - - - mousedown - - - - - - type - - - pageX - - - pageY - - - - - - - - - Activity Link click - addUIWatcherPlugin - org.exoplatform.analytics.api.service.StatisticUIWatcherPlugin - - - watcher - - - - Activity Link click - - - - click - - - - - - - ui - - - activityStream - - - - - application - - - activityHeader - - - - - - - .activity-stream .activity-head-time a - - - - mousedown - - - - - - - Activity Poster click - addUIWatcherPlugin - org.exoplatform.analytics.api.service.StatisticUIWatcherPlugin - - - watcher - - - - Activity Poster click - - - - click - - - - - - - ui - - - activityStream - - - - - application - - - activityHeader - - - - - - - .activity-stream .activity-head-user-link - - - - mousedown - - - - - - - Activity Space click - addUIWatcherPlugin - org.exoplatform.analytics.api.service.StatisticUIWatcherPlugin - - - watcher - - - - Activity Space click - - - - click - - - - - - - ui - - - activityStream - - - - - application - - - activityHeader - - - - - - - .activity-stream .activity-head-space-link - - - - mousedown - - - - - - - - Unified search click - addUIWatcherPlugin - org.exoplatform.analytics.api.service.StatisticUIWatcherPlugin - - - watcher - - - - Unified search click - - - - click - - - - - - - ui - - - toolbar - - - - - application - - - SearchPortlet - - - - - - - #SearchApplication - - - - mousedown - - - - - - - - - - org.exoplatform.groovyscript.text.TemplateService - - UIPortalApplication-head - addTemplateExtension - org.exoplatform.groovyscript.text.TemplateExtensionPlugin - - - templates - war:/groovy/UIPageDisplayStatisticCollection.gtmpl - - - - - - diff --git a/analytics-webapps/src/main/webapp/WEB-INF/conf/analytics/cache-configuration.xml b/analytics-webapps/src/main/webapp/WEB-INF/conf/analytics/cache-configuration.xml index c581be626..c4c81559d 100644 --- a/analytics-webapps/src/main/webapp/WEB-INF/conf/analytics/cache-configuration.xml +++ b/analytics-webapps/src/main/webapp/WEB-INF/conf/analytics/cache-configuration.xml @@ -1,16 +1,19 @@ - + + + org.exoplatform.groovyscript.text.TemplateService + + UIPortalApplication-head + addTemplateExtension + org.exoplatform.groovyscript.text.TemplateExtensionPlugin + + + templates + war:/groovy/UIPageDisplayStatisticCollection.gtmpl + + + + + org.exoplatform.commons.addons.AddOnService diff --git a/analytics-webapps/src/main/webapp/WEB-INF/conf/analytics/listeners-configuration.xml b/analytics-webapps/src/main/webapp/WEB-INF/conf/analytics/listeners-configuration.xml deleted file mode 100644 index a3510a001..000000000 --- a/analytics-webapps/src/main/webapp/WEB-INF/conf/analytics/listeners-configuration.xml +++ /dev/null @@ -1,209 +0,0 @@ - - - - - - org.exoplatform.social.core.space.spi.SpaceService - - AnalyticsSpaceListener - addSpaceListener - org.exoplatform.analytics.listener.social.AnalyticsSpaceListener - - - - - org.exoplatform.services.listener.ListenerService - - metadata.tag.added - addListener - org.exoplatform.analytics.listener.social.AnalyticsActivityTagsListener - - - - - org.exoplatform.services.listener.ListenerService - - attachment.created - addListener - org.exoplatform.analytics.listener.social.ActivityAttachmentAnalyticsListener - - - supported-type - activity - - - - - attachment.deleted - addListener - org.exoplatform.analytics.listener.social.ActivityAttachmentAnalyticsListener - - - supported-type - activity - - - - - - - org.exoplatform.services.organization.OrganizationService - - new.user.event.listener - addListenerPlugin - org.exoplatform.analytics.listener.portal.UserAnalyticsEventListener - This listener will compute again users count each time a new user is added/deleted - - - - - org.exoplatform.services.listener.ListenerService - - exo.core.security.ConversationRegistry.register - addListener - org.exoplatform.analytics.listener.portal.LoginAnalyticsListener - Listener for user login event statistics - - - exo.core.security.ConversationRegistry.unregister - addListener - org.exoplatform.analytics.listener.portal.LoginAnalyticsListener - Listener for user logout event statistics - - - - - org.exoplatform.services.listener.ListenerService - - login.failed - addListener - org.exoplatform.analytics.listener.portal.LoginFailedAnalyticsListener - Listener for user login failed event statistics - - - - - - org.exoplatform.social.core.manager.ActivityManager - - AnalyticsActivityListener - addActivityEventListener - org.exoplatform.analytics.listener.social.AnalyticsActivityListener - - - - - org.exoplatform.social.core.manager.IdentityManager - - AnalyticsProfileListener - registerProfileListener - org.exoplatform.analytics.listener.social.AnalyticsProfileListener - - - - - org.exoplatform.social.core.manager.RelationshipManager - - AnalyticsRelationshipListener - addListenerPlugin - org.exoplatform.analytics.listener.social.AnalyticsRelationshipListener - - - - - org.exoplatform.web.application.ApplicationLifecycleExtension - - PageAccessListener - addPortalApplicationLifecycle - org.exoplatform.analytics.listener.portal.PageAccessListener - - - collectAjaxQueries - Whether collect ajax queries or not - ${exo.analytics.collectAjaxQueries:false} - - - - - - - org.exoplatform.services.listener.ListenerService - - exo.analytics.websocket.messageReceived - addListener - org.exoplatform.analytics.listener.websocket.WebSocketUIStatisticListener - - - - - org.exoplatform.services.listener.ListenerService - - notification.read.item - addListener - org.exoplatform.analytics.listener.social.AnalyticsSpaceWebNotificationListener - - - notification.unread.item - addListener - org.exoplatform.analytics.listener.social.AnalyticsSpaceWebNotificationListener - - - notification.read.allItems - addListener - org.exoplatform.analytics.listener.social.AnalyticsSpaceWebNotificationListener - - - - - org.exoplatform.services.scheduler.JobSchedulerService - - UsersStatisticsCountJob - addPeriodJob - org.exoplatform.services.scheduler.PeriodJob - A cron job to daily collect users count statistics - - - job.info - - - - - - - - - - - - - - - - SpacesStatisticsCountJob - addPeriodJob - org.exoplatform.services.scheduler.PeriodJob - A cron job to daily collect spaces count statistics - - - job.info - - - - - - - - - - - - - - - - - diff --git a/analytics-webapps/src/main/webapp/WEB-INF/conf/analytics/organization-configuration.xml b/analytics-webapps/src/main/webapp/WEB-INF/conf/analytics/organization-configuration.xml index 7a60e10d7..bef3513ca 100644 --- a/analytics-webapps/src/main/webapp/WEB-INF/conf/analytics/organization-configuration.xml +++ b/analytics-webapps/src/main/webapp/WEB-INF/conf/analytics/organization-configuration.xml @@ -1,16 +1,19 @@ - + diff --git a/analytics-webapps/src/main/webapp/WEB-INF/jsp/analytics-rate.jsp b/analytics-webapps/src/main/webapp/WEB-INF/jsp/analytics-rate.jsp index ffb0c9972..ef59193f0 100644 --- a/analytics-webapps/src/main/webapp/WEB-INF/jsp/analytics-rate.jsp +++ b/analytics-webapps/src/main/webapp/WEB-INF/jsp/analytics-rate.jsp @@ -1,16 +1,19 @@ <% /** * This file is part of the Meeds project (https://meeds.io/). - * Copyright (C) 2022 Meeds Association - * contact@meeds.io + * + * Copyright (C) 2020 - 2024 Meeds Association contact@meeds.io + * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU Lesser General Public * License as published by the Free Software Foundation; either * version 3 of the License, or (at your option) any later version. + * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU * Lesser General Public License for more details. + * * You should have received a copy of the GNU Lesser General Public License * along with this program; if not, write to the Free Software Foundation, * Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. diff --git a/analytics-webapps/src/main/webapp/WEB-INF/jsp/analytics-table.jsp b/analytics-webapps/src/main/webapp/WEB-INF/jsp/analytics-table.jsp index 73f7ca57b..70caababc 100644 --- a/analytics-webapps/src/main/webapp/WEB-INF/jsp/analytics-table.jsp +++ b/analytics-webapps/src/main/webapp/WEB-INF/jsp/analytics-table.jsp @@ -1,16 +1,19 @@ <% /** * This file is part of the Meeds project (https://meeds.io/). - * Copyright (C) 2022 Meeds Association - * contact@meeds.io + * + * Copyright (C) 2020 - 2024 Meeds Association contact@meeds.io + * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU Lesser General Public * License as published by the Free Software Foundation; either * version 3 of the License, or (at your option) any later version. + * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU * Lesser General Public License for more details. + * * You should have received a copy of the GNU Lesser General Public License * along with this program; if not, write to the Free Software Foundation, * Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. diff --git a/analytics-webapps/src/main/webapp/WEB-INF/jsp/analytics.jsp b/analytics-webapps/src/main/webapp/WEB-INF/jsp/analytics.jsp index d4880bc0d..e8da3faa1 100644 --- a/analytics-webapps/src/main/webapp/WEB-INF/jsp/analytics.jsp +++ b/analytics-webapps/src/main/webapp/WEB-INF/jsp/analytics.jsp @@ -1,16 +1,19 @@ <% /** * This file is part of the Meeds project (https://meeds.io/). - * Copyright (C) 2022 Meeds Association - * contact@meeds.io + * + * Copyright (C) 2020 - 2024 Meeds Association contact@meeds.io + * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU Lesser General Public * License as published by the Free Software Foundation; either * version 3 of the License, or (at your option) any later version. + * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU * Lesser General Public License for more details. + * * You should have received a copy of the GNU Lesser General Public License * along with this program; if not, write to the Free Software Foundation, * Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. diff --git a/analytics-webapps/src/main/webapp/WEB-INF/jsp/breadcrumb.jsp b/analytics-webapps/src/main/webapp/WEB-INF/jsp/breadcrumb.jsp index a3c54ddad..e44ffff3f 100644 --- a/analytics-webapps/src/main/webapp/WEB-INF/jsp/breadcrumb.jsp +++ b/analytics-webapps/src/main/webapp/WEB-INF/jsp/breadcrumb.jsp @@ -1,16 +1,19 @@ <% /** * This file is part of the Meeds project (https://meeds.io/). - * Copyright (C) 2022 Meeds Association - * contact@meeds.io + * + * Copyright (C) 2020 - 2024 Meeds Association contact@meeds.io + * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU Lesser General Public * License as published by the Free Software Foundation; either * version 3 of the License, or (at your option) any later version. + * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU * Lesser General Public License for more details. + * * You should have received a copy of the GNU Lesser General Public License * along with this program; if not, write to the Free Software Foundation, * Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. diff --git a/analytics-webapps/src/main/webapp/WEB-INF/jsp/statistics-collection.jsp b/analytics-webapps/src/main/webapp/WEB-INF/jsp/statistics-collection.jsp index 86d51889d..abcf32ccf 100644 --- a/analytics-webapps/src/main/webapp/WEB-INF/jsp/statistics-collection.jsp +++ b/analytics-webapps/src/main/webapp/WEB-INF/jsp/statistics-collection.jsp @@ -1,16 +1,19 @@ <% /** * This file is part of the Meeds project (https://meeds.io/). - * Copyright (C) 2022 Meeds Association - * contact@meeds.io + * + * Copyright (C) 2020 - 2024 Meeds Association contact@meeds.io + * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU Lesser General Public * License as published by the Free Software Foundation; either * version 3 of the License, or (at your option) any later version. + * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU * Lesser General Public License for more details. + * * You should have received a copy of the GNU Lesser General Public License * along with this program; if not, write to the Free Software Foundation, * Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. diff --git a/analytics-webapps/src/main/webapp/WEB-INF/portlet.xml b/analytics-webapps/src/main/webapp/WEB-INF/portlet.xml index df87452dc..a9378bd97 100644 --- a/analytics-webapps/src/main/webapp/WEB-INF/portlet.xml +++ b/analytics-webapps/src/main/webapp/WEB-INF/portlet.xml @@ -1,16 +1,19 @@