Skip to content

Commit

Permalink
Merge pull request #466 from idaholab/v24.04.0_merge_idaholab
Browse files Browse the repository at this point in the history
Malcolm v24.04.0

* Features and enhancements
    - Zeek-extracted files scanned and preserved on a [Hedgehog Linux](https://idaholab.github.io/Malcolm/docs/malcolm-hedgehog-e2e-iso-install.html#HedgehogZeekFileExtraction) sensor can now be accessed via [the extracted files download user interface](https://idaholab.github.io/Malcolm/docs/file-scanning.html#ZeekFileExtractionUI) (#331).
    - Improvements to creation of index templates, dashboards, and other saved objects on startup (#208) to ensure that saved objects get created correctly upon upgrade (see [this comment](#208 (comment)) for more details on this feature).
    - [Populating the NetBox inventory via passively-gathered network traffic metadata](https://idaholab.github.io/Malcolm/docs/asset-interaction-analysis.html#NetBoxPopPassive) now uses network traffic logs for DNS, NTLM, and DHCP to identify assets' host names when possible for use when populating device and VM names (#415). Autopopulated devices now have their *status* field set to `Active` rather than `Stage`, and uses *tags* instead to indicated that they were created through autopopulation.
    - Users can now specify pruning thresholds for [carved files](https://idaholab.github.io/Malcolm/docs/file-scanning.html#ZeekFileExtraction) so that old files are deleted in order to avoid filling available storage (#453). See a new section of documentation on [Managing disk usage](https://idaholab.github.io/Malcolm/docs/malcolm-config.html#DiskUsage) for more information about this and similar settings.
    - Users can now specify a prefix that will be prepended to dashboards as they are imported into OpenSearch Dashboards or Kibana, allowing users who have dashboards from other sources to differentiate between those and Malcolm's (#455).
    - The default anomaly detectors created for the OpenSearch Anomaly Detection plugin are now created with [category fields for high cardinality](https://opensearch.org/docs/latest/observing-your-data/ad/index/#optional-set-category-fields-for-high-cardinality) to allow for better breakdown of contributing values to anomalies discovered (#464).
    - Include [JA4+ plugin in Arkime](https://arkime.com/settings#ja4plus). See #419 for status on upcoming full JA4+ support in Malcolm.
    - Hedgehog Linux sensors can now [periodically refresh](https://github.com/idaholab/Malcolm/blob/bceee4616dd5676a010a3dd7b0410856257948e8/hedgehog-iso/interface/sensor_ctl/control_vars.conf#L75) their [Zeek inteligence files](https://idaholab.github.io/Malcolm/docs/hedgehog-config-zeek-intel.html#HedgehogZeekIntel).
        + **NOTE**: Due to an oversight, a value is missing from the default Hedgehog Linux configuration in this release, preventing the intel refresh cron job from executing. As a workaround, appending the line `export INTEL_DIR=/opt/sensor/sensor_ctl/zeek/intel` to `/opt/sensor/sensor_ctl/control_vars.conf` and restarting the sensor services will remedy the situation. This will be corrected in the next Malcolm release.
    - Assorted documentation improvements.
* Component version updates
    - Arkime to [v5.1.2](https://github.com/arkime/arkime/blob/bcd9d7e68be8e4a52a17c35211c5d5a7fdcc1a1c/CHANGELOG#L36-L41)
    - OpenSearch and OpenSearch Dashboards to [v2.13.0](https://github.com/opensearch-project/opensearch-build/blob/main/release-notes/opensearch-release-notes-2.13.0.md)
    - Beats to [v8.13.2](https://www.elastic.co/guide/en/beats/libbeat/current/release-notes-8.13.2.html)
    - Logstash to [v8.13.2](https://www.elastic.co/guide/en/logstash/current/logstash-8-13-2.html)
    - gunicorn to v22.0.0 to address [CVE-2024-1135](GHSA-w3h3-4rj7-4ph4).
    - elasticsearch-dsl to [v8.13.0](https://github.com/elastic/elasticsearch-dsl-py/releases/tag/v8.13.0)
    - elasticsearch-py to [v8.13.0](https://github.com/elastic/elasticsearch-py/releases/tag/v8.13.0)
    - idna to v3.7 to address [CVE-2024-3651](GHSA-jjg7-2v4v-x38h)
    - Fluent Bit to [v3.0.3](https://fluentbit.io/announcements/v3.0.3/)
* Bug fixes
    - The documentation for [Windows host system configuration](https://idaholab.github.io/Malcolm/docs/host-config-windows.html#HostSystemConfigWindows) was out of date and has been updated for the latest version of Microsoft Windows Subsystem for Linux (#421).
    - An issue was fixed in which Malcolm's list of users and their password hashes could become corrupted if the file did not initially end with a newline character (#426).
    - The manner in which Zeek intel files are generated has been changed to avoid problems found in Kubernetes deployments when scaling out the number of `zeek-live` containers (#456). See [this comment](#456 (comment)) for more details.
    - Removed the version top-level element from `docker-compose.yml` files as it is [now obsolete](https://docs.docker.com/compose/compose-file/04-version-and-name/) and caused a warning message that sometimes was not handled correctly.
    - Fix Malcolm ISO not correctly detecting if it's in a live boot ISO environment or installed mode.
    - Restart live Zeek instances with `zeekctl deploy` instead of `zeekctl restart`.
* Configuration changes (in [environment variables](https://idaholab.github.io/Malcolm/docs/malcolm-config.html#MalcolmConfigEnvVars) in [`./config/`](https://github.com/idaholab/Malcolm/blob/v24.04.0/config))
    - `ARKIME_QUERY_ALL_INDICES` in [`arkime.env`](https://github.com/idaholab/Malcolm/blob/bceee4616dd5676a010a3dd7b0410856257948e8/config/arkime.env.example#L9-L11) can be set to control the [`queryAllIndices` setting](https://arkime.com/settings#queryAllIndices) in Arkime's `config.ini`.
    - `DASHBOARDS_PREFIX` in [`dashboards-helper.env`](https://github.com/idaholab/Malcolm/blob/bceee4616dd5676a010a3dd7b0410856257948e8/config/dashboards-helper.env.example#L3C1-L4C19) has been added for #455 (see above in **Features and Enhancements**).
    - `LOGSTASH_NETBOX_ENRICHMENT_DATASETS` in [`logstash.env`](https://github.com/idaholab/Malcolm/blob/bceee4616dd5676a010a3dd7b0410856257948e8/config/logstash.env.example#L13) has been changed to include `zeek.dhcp`, `zeek.dns`, and `zeek.ntlm` to support #415 (see above in **Features and Enhancements**).
    - `LOGSTASH_ZEEK_IGNORED_LOGS` in [`logstash.env`](https://github.com/idaholab/Malcolm/blob/bceee4616dd5676a010a3dd7b0410856257948e8/config/logstash.env.example#L15) has been changed to remove `capture_loss` and `stats` so that those diagnostic Zeek logs can be parsed without the user having to manually change this variable.
    - `ZEEK_CRON` has been removed from [`zeek-live.env`](https://github.com/idaholab/Malcolm/blob/bceee4616dd5676a010a3dd7b0410856257948e8/config/zeek-live.env.example) and `ZEEK_INTEL_REFRESH_CRON_EXPRESSION` was removed from [`zeek.env`](https://github.com/idaholab/Malcolm/blob/bceee4616dd5676a010a3dd7b0410856257948e8/config/zeek.env.example) and moved to the "offline" version of the container in [`zeek-offline.env`](https://github.com/idaholab/Malcolm/blob/bceee4616dd5676a010a3dd7b0410856257948e8/config/zeek-offline.env.example#L17-L19) for #456.
    - `EXTRACTED_FILE_PRUNE_THRESHOLD_MAX_SIZE`, `EXTRACTED_FILE_PRUNE_THRESHOLD_TOTAL_DISK_USAGE_PERCENT`, and `EXTRACTED_FILE_PRUNE_INTERVAL_SECONDS` were added to [`zeek.env`](https://github.com/idaholab/Malcolm/blob/bceee4616dd5676a010a3dd7b0410856257948e8/config/zeek.env.example#L32-L37) for #453. See a new section of documentation on [Managing disk usage](https://idaholab.github.io/Malcolm/docs/malcolm-config.html#DiskUsage) for more information about these and similar settings.
  • Loading branch information
mmguero authored Apr 30, 2024
2 parents 99f68ec + bceee46 commit a6248b6
Show file tree
Hide file tree
Showing 230 changed files with 2,858 additions and 1,601 deletions.
2 changes: 2 additions & 0 deletions .github/workflows/api-build-and-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@ on:
- '!shared/bin/common-init.sh'
- '!shared/bin/sensor-init.sh'
- '!shared/bin/os-disk-config.py'
- '!shared/bin/extracted_files_http_server.py'
- '!shared/bin/web-ui-asset-download.sh'
- '!shared/bin/preseed_late_user_config.sh'
- '!shared/bin/configure-interfaces.py'
- '!shared/bin/configure-capture.py'
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/arkime-build-and-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@ on:
- '!shared/bin/agg-init.sh'
- '!shared/bin/common-init.sh'
- '!shared/bin/sensor-init.sh'
- '!shared/bin/extracted_files_http_server.py'
- '!shared/bin/web-ui-asset-download.sh'
- '!shared/bin/os-disk-config.py'
- '!shared/bin/preseed_late_user_config.sh'
- '!shared/bin/configure-interfaces.py'
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/dashboards-build-and-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@ on:
- '!shared/bin/agg-init.sh'
- '!shared/bin/common-init.sh'
- '!shared/bin/sensor-init.sh'
- '!shared/bin/extracted_files_http_server.py'
- '!shared/bin/web-ui-asset-download.sh'
- '!shared/bin/os-disk-config.py'
- '!shared/bin/preseed_late_user_config.sh'
- '!shared/bin/configure-interfaces.py'
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/dashboards-helper-build-and-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@ on:
- '!shared/bin/agg-init.sh'
- '!shared/bin/common-init.sh'
- '!shared/bin/sensor-init.sh'
- '!shared/bin/extracted_files_http_server.py'
- '!shared/bin/web-ui-asset-download.sh'
- '!shared/bin/os-disk-config.py'
- '!shared/bin/preseed_late_user_config.sh'
- '!shared/bin/configure-interfaces.py'
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/file-upload-build-and-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@ on:
- '!shared/bin/agg-init.sh'
- '!shared/bin/common-init.sh'
- '!shared/bin/sensor-init.sh'
- '!shared/bin/extracted_files_http_server.py'
- '!shared/bin/web-ui-asset-download.sh'
- '!shared/bin/os-disk-config.py'
- '!shared/bin/preseed_late_user_config.sh'
- '!shared/bin/configure-interfaces.py'
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/filebeat-build-and-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@ on:
- '!shared/bin/agg-init.sh'
- '!shared/bin/common-init.sh'
- '!shared/bin/sensor-init.sh'
- '!shared/bin/extracted_files_http_server.py'
- '!shared/bin/web-ui-asset-download.sh'
- '!shared/bin/os-disk-config.py'
- '!shared/bin/preseed_late_user_config.sh'
- '!shared/bin/configure-interfaces.py'
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/freq-build-and-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@ on:
- '!shared/bin/agg-init.sh'
- '!shared/bin/common-init.sh'
- '!shared/bin/sensor-init.sh'
- '!shared/bin/extracted_files_http_server.py'
- '!shared/bin/web-ui-asset-download.sh'
- '!shared/bin/os-disk-config.py'
- '!shared/bin/preseed_late_user_config.sh'
- '!shared/bin/configure-interfaces.py'
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,9 @@ jobs:
cp -r ./arkime/patch ./hedgehog-iso/shared/arkime_patch
mkdir -p ./hedgehog-iso/suricata
cp -r ./suricata/rules-default ./hedgehog-iso/suricata/
mkdir -p ./hedgehog-iso/nginx
cp -r ./nginx/landingpage/css ./hedgehog-iso/nginx/
cp -r ./nginx/landingpage/js ./hedgehog-iso/nginx/
pushd ./hedgehog-iso
echo "${{ steps.extract_malcolm_version.outputs.mversion }}" > ./shared/version.txt
echo "${{ secrets.MAXMIND_GEOIP_DB_LICENSE_KEY }}" > ./shared/maxmind_license.txt
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/htadmin-build-and-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@ on:
- '!shared/bin/agg-init.sh'
- '!shared/bin/common-init.sh'
- '!shared/bin/sensor-init.sh'
- '!shared/bin/extracted_files_http_server.py'
- '!shared/bin/web-ui-asset-download.sh'
- '!shared/bin/os-disk-config.py'
- '!shared/bin/preseed_late_user_config.sh'
- '!shared/bin/configure-interfaces.py'
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/logstash-build-and-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@ on:
- '!shared/bin/agg-init.sh'
- '!shared/bin/common-init.sh'
- '!shared/bin/sensor-init.sh'
- '!shared/bin/extracted_files_http_server.py'
- '!shared/bin/web-ui-asset-download.sh'
- '!shared/bin/os-disk-config.py'
- '!shared/bin/preseed_late_user_config.sh'
- '!shared/bin/configure-interfaces.py'
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/malcolm-iso-build-docker-wrap-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,8 @@ on:
- 'malcolm-iso/**'
- 'shared/bin/*'
- '!shared/bin/configure-capture.py'
- '!shared/bin/extracted_files_http_server.py'
- '!shared/bin/web-ui-asset-download.sh'
- '!shared/bin/zeek*'
- '!shared/bin/suricata*'
- '.trigger_iso_workflow_build'
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/netbox-build-and-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@ on:
- '!shared/bin/agg-init.sh'
- '!shared/bin/common-init.sh'
- '!shared/bin/sensor-init.sh'
- '!shared/bin/extracted_files_http_server.py'
- '!shared/bin/web-ui-asset-download.sh'
- '!shared/bin/os-disk-config.py'
- '!shared/bin/preseed_late_user_config.sh'
- '!shared/bin/configure-interfaces.py'
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/nginx-build-and-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@ on:
- '!shared/bin/agg-init.sh'
- '!shared/bin/common-init.sh'
- '!shared/bin/sensor-init.sh'
- '!shared/bin/extracted_files_http_server.py'
- '!shared/bin/web-ui-asset-download.sh'
- '!shared/bin/os-disk-config.py'
- '!shared/bin/preseed_late_user_config.sh'
- '!shared/bin/configure-interfaces.py'
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/opensearch-build-and-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@ on:
- '!shared/bin/common-init.sh'
- '!shared/bin/sensor-init.sh'
- '!shared/bin/os-disk-config.py'
- '!shared/bin/extracted_files_http_server.py'
- '!shared/bin/web-ui-asset-download.sh'
- '!shared/bin/preseed_late_user_config.sh'
- '!shared/bin/configure-interfaces.py'
- '!shared/bin/configure-capture.py'
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/pcap-capture-build-and-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@ on:
- '!shared/bin/common-init.sh'
- '!shared/bin/sensor-init.sh'
- '!shared/bin/os-disk-config.py'
- '!shared/bin/extracted_files_http_server.py'
- '!shared/bin/web-ui-asset-download.sh'
- '!shared/bin/preseed_late_user_config.sh'
- '!shared/bin/configure-interfaces.py'
- '!shared/bin/configure-capture.py'
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/pcap-monitor-build-and-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@ on:
- '!shared/bin/common-init.sh'
- '!shared/bin/sensor-init.sh'
- '!shared/bin/os-disk-config.py'
- '!shared/bin/extracted_files_http_server.py'
- '!shared/bin/web-ui-asset-download.sh'
- '!shared/bin/preseed_late_user_config.sh'
- '!shared/bin/configure-interfaces.py'
- '!shared/bin/configure-capture.py'
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/postgresql-build-and-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@ on:
- '!shared/bin/common-init.sh'
- '!shared/bin/sensor-init.sh'
- '!shared/bin/os-disk-config.py'
- '!shared/bin/extracted_files_http_server.py'
- '!shared/bin/web-ui-asset-download.sh'
- '!shared/bin/preseed_late_user_config.sh'
- '!shared/bin/configure-interfaces.py'
- '!shared/bin/configure-capture.py'
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/redis-build-and-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@ on:
- '!shared/bin/common-init.sh'
- '!shared/bin/sensor-init.sh'
- '!shared/bin/os-disk-config.py'
- '!shared/bin/extracted_files_http_server.py'
- '!shared/bin/web-ui-asset-download.sh'
- '!shared/bin/preseed_late_user_config.sh'
- '!shared/bin/configure-interfaces.py'
- '!shared/bin/configure-capture.py'
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/suricata-build-and-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@ on:
- '!shared/bin/common-init.sh'
- '!shared/bin/sensor-init.sh'
- '!shared/bin/os-disk-config.py'
- '!shared/bin/extracted_files_http_server.py'
- '!shared/bin/web-ui-asset-download.sh'
- '!shared/bin/preseed_late_user_config.sh'
- '!shared/bin/configure-interfaces.py'
- '!shared/bin/configure-capture.py'
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/zeek-build-and-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@ on:
- '!shared/bin/common-init.sh'
- '!shared/bin/sensor-init.sh'
- '!shared/bin/os-disk-config.py'
- '!shared/bin/extracted_files_http_server.py'
- '!shared/bin/web-ui-asset-download.sh'
- '!shared/bin/preseed_late_user_config.sh'
- '!shared/bin/configure-interfaces.py'
- '!shared/bin/configure-capture.py'
Expand Down
10 changes: 7 additions & 3 deletions Dockerfiles/arkime.Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ ENV TERM xterm
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1

ENV ARKIME_VERSION "v5.0.1"
ENV ARKIME_VERSION "v5.1.2"
ENV ARKIME_DIR "/opt/arkime"
ENV ARKIME_URL "https://github.com/arkime/arkime.git"
ENV ARKIME_LOCALELASTICSEARCH no
Expand All @@ -16,7 +16,8 @@ ENV ARKIME_INET yes
ADD arkime/scripts/bs4_remove_div.py /opt/
ADD arkime/patch/* /opt/patches/

RUN apt-get -q update && \
RUN export DEBARCH=$(dpkg --print-architecture) && \
apt-get -q update && \
apt-get -y -q --no-install-recommends upgrade && \
apt-get install -q -y --no-install-recommends \
binutils \
Expand Down Expand Up @@ -73,7 +74,10 @@ RUN apt-get -q update && \
make install && \
npm cache clean --force && \
rm -f ${ARKIME_DIR}/wiseService/source.* ${ARKIME_DIR}/etc/*.systemd.service && \
bash -c "file ${ARKIME_DIR}/bin/* ${ARKIME_DIR}/node-v*/bin/* | grep 'ELF 64-bit' | sed 's/:.*//' | xargs -l -r strip -v --strip-unneeded"
bash -c "file ${ARKIME_DIR}/bin/* ${ARKIME_DIR}/node-v*/bin/* | grep 'ELF 64-bit' | sed 's/:.*//' | xargs -l -r strip -v --strip-unneeded" && \
mkdir -p "${ARKIME_DIR}"/plugins && \
curl -fsSL -o "${ARKIME_DIR}/plugins/ja4plus.${DEBARCH}.so" "https://github.com/arkime/arkime/releases/download/${ARKIME_VERSION}/ja4plus.${DEBARCH}.so" && \
chmod 755 "${ARKIME_DIR}/plugins/ja4plus.${DEBARCH}.so"

FROM debian:12-slim

Expand Down
4 changes: 2 additions & 2 deletions Dockerfiles/dashboards-helper.Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ ADD scripts/malcolm_utils.py /data/

RUN apk update --no-cache && \
apk upgrade --no-cache && \
apk --no-cache add bash python3 py3-pip curl openssl procps psmisc npm rsync shadow jq tini && \
apk --no-cache add bash python3 py3-pip curl openssl procps psmisc moreutils npm rsync shadow jq tini && \
npm install -g http-server && \
pip3 install --break-system-packages supervisor humanfriendly requests && \
curl -fsSLO "$SUPERCRONIC_URL" && \
Expand Down Expand Up @@ -95,7 +95,7 @@ RUN apk update --no-cache && \
/opt/templates && \
chmod 755 /data/*.sh /data/*.py /data/init && \
chmod 400 /opt/maps/* && \
(echo -e "*/2 * * * * /data/create-arkime-sessions-index.sh\n0 10 * * * /data/index-refresh.py --index MALCOLM_NETWORK_INDEX_PATTERN --template malcolm_template --unassigned\n30 */2 * * * /data/index-refresh.py --index MALCOLM_OTHER_INDEX_PATTERN --template malcolm_beats_template --unassigned\n*/20 * * * * /data/opensearch_index_size_prune.py" > ${SUPERCRONIC_CRONTAB})
(echo -e "*/2 * * * * /data/shared-object-creation.sh\n0 10 * * * /data/index-refresh.py --index MALCOLM_NETWORK_INDEX_PATTERN --template malcolm_template --unassigned\n30 */2 * * * /data/index-refresh.py --index MALCOLM_OTHER_INDEX_PATTERN --template malcolm_beats_template --unassigned\n*/20 * * * * /data/opensearch_index_size_prune.py" > ${SUPERCRONIC_CRONTAB})
EXPOSE $OFFLINE_REGION_MAPS_PORT
Expand Down
8 changes: 4 additions & 4 deletions Dockerfiles/dashboards.Dockerfile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM opensearchproject/opensearch-dashboards:2.12.0
FROM opensearchproject/opensearch-dashboards:2.13.0

LABEL maintainer="[email protected]"
LABEL org.opencontainers.image.authors='[email protected]'
Expand All @@ -20,7 +20,7 @@ ENV PUSER_PRIV_DROP true
ENV TERM xterm

ENV TINI_VERSION v0.19.0
ENV OSD_TRANSFORM_VIS_VERSION 2.12.0
ENV OSD_TRANSFORM_VIS_VERSION 2.13.0

ARG NODE_OPTIONS="--max_old_space_size=4096"
ENV NODE_OPTIONS $NODE_OPTIONS
Expand All @@ -40,8 +40,8 @@ RUN yum upgrade -y && \
/usr/share/opensearch-dashboards/bin/opensearch-dashboards-plugin remove securityDashboards --allow-root && \
cd /tmp && \
# unzip transformVis.zip opensearch-dashboards/transformVis/opensearch_dashboards.json opensearch-dashboards/transformVis/package.json && \
# sed -i "s/2\.12\.0/2\.12\.0/g" opensearch-dashboards/transformVis/opensearch_dashboards.json && \
# sed -i "s/2\.12\.0/2\.12\.0/g" opensearch-dashboards/transformVis/package.json && \
# sed -i "s/2\.12\.0/2\.13\.0/g" opensearch-dashboards/transformVis/opensearch_dashboards.json && \
# sed -i "s/2\.12\.0/2\.13\.0/g" opensearch-dashboards/transformVis/package.json && \
# zip transformVis.zip opensearch-dashboards/transformVis/opensearch_dashboards.json opensearch-dashboards/transformVis/package.json && \
cd /usr/share/opensearch-dashboards/plugins && \
/usr/share/opensearch-dashboards/bin/opensearch-dashboards-plugin install file:///tmp/transformVis.zip --allow-root && \
Expand Down
40 changes: 21 additions & 19 deletions Dockerfiles/file-monitor.Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,9 @@ ARG EXTRACTED_FILE_SCANNER_START_SLEEP=10
ARG EXTRACTED_FILE_LOGGER_START_SLEEP=5
ARG EXTRACTED_FILE_MIN_BYTES=64
ARG EXTRACTED_FILE_MAX_BYTES=134217728
ARG EXTRACTED_FILE_PRUNE_THRESHOLD_MAX_SIZE=1TB
ARG EXTRACTED_FILE_PRUNE_THRESHOLD_TOTAL_DISK_USAGE_PERCENT=0
ARG EXTRACTED_FILE_PRUNE_INTERVAL_SECONDS=300
ARG VTOT_API2_KEY=0
ARG VTOT_REQUESTS_PER_MINUTE=4
ARG EXTRACTED_FILE_ENABLE_CLAMAV=false
Expand Down Expand Up @@ -65,6 +68,9 @@ ENV EXTRACTED_FILE_SCANNER_START_SLEEP $EXTRACTED_FILE_SCANNER_START_SLEEP
ENV EXTRACTED_FILE_LOGGER_START_SLEEP $EXTRACTED_FILE_LOGGER_START_SLEEP
ENV EXTRACTED_FILE_MIN_BYTES $EXTRACTED_FILE_MIN_BYTES
ENV EXTRACTED_FILE_MAX_BYTES $EXTRACTED_FILE_MAX_BYTES
ENV EXTRACTED_FILE_PRUNE_THRESHOLD_MAX_SIZE $EXTRACTED_FILE_PRUNE_THRESHOLD_MAX_SIZE
ENV EXTRACTED_FILE_PRUNE_THRESHOLD_TOTAL_DISK_USAGE_PERCENT $EXTRACTED_FILE_PRUNE_THRESHOLD_TOTAL_DISK_USAGE_PERCENT
ENV EXTRACTED_FILE_PRUNE_INTERVAL_SECONDS $EXTRACTED_FILE_PRUNE_INTERVAL_SECONDS
ENV VTOT_API2_KEY $VTOT_API2_KEY
ENV VTOT_REQUESTS_PER_MINUTE $VTOT_REQUESTS_PER_MINUTE
ENV EXTRACTED_FILE_ENABLE_CLAMAV $EXTRACTED_FILE_ENABLE_CLAMAV
Expand Down Expand Up @@ -103,6 +109,11 @@ ENV SUPERCRONIC_SHA1SUM "cd48d45c4b10f3f0bfdd3a57d054cd05ac96812b"
ENV SUPERCRONIC_CRONTAB "/etc/crontab"

COPY --chmod=755 shared/bin/yara_rules_setup.sh /usr/local/bin/
ADD nginx/landingpage/css "${EXTRACTED_FILE_HTTP_SERVER_ASSETS_DIR}/css"
ADD nginx/landingpage/js "${EXTRACTED_FILE_HTTP_SERVER_ASSETS_DIR}/js"
ADD --chmod=644 docs/images/logo/Malcolm_background.png "${EXTRACTED_FILE_HTTP_SERVER_ASSETS_DIR}/assets/img/bg-masthead.png"
COPY --chmod=644 docs/images/icon/favicon.ico "${EXTRACTED_FILE_HTTP_SERVER_ASSETS_DIR}/favicon.ico"
COPY --chmod=755 shared/bin/web-ui-asset-download.sh /usr/local/bin/

RUN sed -i "s/main$/main contrib non-free/g" /etc/apt/sources.list.d/debian.sources && \
apt-get -q update && \
Expand All @@ -129,7 +140,7 @@ RUN sed -i "s/main$/main contrib non-free/g" /etc/apt/sources.list.d/debian.sour
pkg-config \
tini \
unzip && \
apt-get -y -q install \
apt-get -y -q install \
inotify-tools \
libzmq5 \
psmisc \
Expand All @@ -143,6 +154,7 @@ RUN sed -i "s/main$/main contrib non-free/g" /etc/apt/sources.list.d/debian.sour
python3 -m pip install --break-system-packages --no-compile --no-cache-dir \
clamd \
dominate \
humanfriendly \
psutil \
pycryptodome \
python-magic \
Expand Down Expand Up @@ -170,6 +182,8 @@ RUN sed -i "s/main$/main contrib non-free/g" /etc/apt/sources.list.d/debian.sour
rm -rf "${SRC_BASE_DIR}"/yara* && \
cd "${YARA_RULES_SRC_DIR}" && \
/usr/local/bin/yara_rules_setup.sh -r "${YARA_RULES_SRC_DIR}" -y "${YARA_RULES_DIR}" && \
cd /tmp && \
/usr/local/bin/web-ui-asset-download.sh -o "${EXTRACTED_FILE_HTTP_SERVER_ASSETS_DIR}/css" && \
cd /tmp && \
curl -fsSL -o ./capa.zip "${CAPA_URL}" && \
unzip ./capa.zip && \
Expand All @@ -190,9 +204,6 @@ RUN sed -i "s/main$/main contrib non-free/g" /etc/apt/sources.list.d/debian.sour
libtool \
make \
python3-dev && \
apt-get -y -q --allow-downgrades --allow-remove-essential --allow-change-held-packages autoremove && \
apt-get clean && \
rm -rf /var/lib/apt/lists/* /tmp/* && \
mkdir -p /var/log/clamav "${CLAMAV_RULES_DIR}" && \
groupadd --gid ${DEFAULT_GID} ${PGROUP} && \
useradd -m --uid ${DEFAULT_UID} --gid ${DEFAULT_GID} ${PUSER} && \
Expand All @@ -214,31 +225,22 @@ RUN sed -i "s/main$/main contrib non-free/g" /etc/apt/sources.list.d/debian.sour
ln -r -s /usr/local/bin/zeek_carve_scanner.py /usr/local/bin/clam_scan.py && \
ln -r -s /usr/local/bin/zeek_carve_scanner.py /usr/local/bin/yara_scan.py && \
ln -r -s /usr/local/bin/zeek_carve_scanner.py /usr/local/bin/capa_scan.py && \
echo "0 */6 * * * /bin/bash /usr/local/bin/capa-update.sh\n0 */6 * * * /usr/local/bin/yara_rules_setup.sh -r \"${YARA_RULES_SRC_DIR}\" -y \"${YARA_RULES_DIR}\"" > ${SUPERCRONIC_CRONTAB}
echo "0 */6 * * * /bin/bash /usr/local/bin/capa-update.sh\n0 */6 * * * /usr/local/bin/yara_rules_setup.sh -r \"${YARA_RULES_SRC_DIR}\" -y \"${YARA_RULES_DIR}\"" > ${SUPERCRONIC_CRONTAB} && \
apt-get -y -q --allow-downgrades --allow-remove-essential --allow-change-held-packages autoremove && \
apt-get clean && \
rm -rf /var/lib/apt/lists/* /tmp/*

USER ${PUSER}

RUN /usr/bin/freshclam freshclam --config-file=/etc/clamav/freshclam.conf

USER root

ADD nginx/landingpage/css "${EXTRACTED_FILE_HTTP_SERVER_ASSETS_DIR}/css"
ADD nginx/landingpage/js "${EXTRACTED_FILE_HTTP_SERVER_ASSETS_DIR}/js"
ADD --chmod=644 docs/images/logo/Malcolm_background.png "${EXTRACTED_FILE_HTTP_SERVER_ASSETS_DIR}/assets/img/bg-masthead.png"
ADD --chmod=644 https://fonts.gstatic.com/s/lato/v24/S6u_w4BMUTPHjxsI9w2_Gwfo.ttf "${EXTRACTED_FILE_HTTP_SERVER_ASSETS_DIR}/css/"
ADD --chmod=644 https://fonts.gstatic.com/s/lato/v24/S6u8w4BMUTPHjxsAXC-v.ttf "${EXTRACTED_FILE_HTTP_SERVER_ASSETS_DIR}/css/"
ADD --chmod=644 https://fonts.gstatic.com/s/lato/v24/S6u_w4BMUTPHjxsI5wq_Gwfo.ttf "${EXTRACTED_FILE_HTTP_SERVER_ASSETS_DIR}/css/"
ADD --chmod=644 https://fonts.gstatic.com/s/lato/v24/S6u9w4BMUTPHh7USSwiPHA.ttf "${EXTRACTED_FILE_HTTP_SERVER_ASSETS_DIR}/css/"
ADD --chmod=644 https://fonts.gstatic.com/s/lato/v24/S6uyw4BMUTPHjx4wWw.ttf "${EXTRACTED_FILE_HTTP_SERVER_ASSETS_DIR}/css/"
ADD --chmod=644 https://fonts.gstatic.com/s/lato/v24/S6u9w4BMUTPHh6UVSwiPHA.ttf "${EXTRACTED_FILE_HTTP_SERVER_ASSETS_DIR}/css/"
ADD --chmod=644 'https://cdn.jsdelivr.net/npm/[email protected]/font/fonts/bootstrap-icons.woff2?856008caa5eb66df68595e734e59580d' "${EXTRACTED_FILE_HTTP_SERVER_ASSETS_DIR}/css/bootstrap-icons.woff2"
ADD --chmod=644 'https://cdn.jsdelivr.net/npm/[email protected]/font/fonts/bootstrap-icons.woff?856008caa5eb66df68595e734e59580d' "${EXTRACTED_FILE_HTTP_SERVER_ASSETS_DIR}/css/bootstrap-icons.woff"

COPY --chmod=644 docs/images/icon/favicon.ico "${EXTRACTED_FILE_HTTP_SERVER_ASSETS_DIR}/favicon.ico"
COPY --chmod=755 shared/bin/docker-uid-gid-setup.sh /usr/local/bin/
COPY --chmod=755 shared/bin/prune_files.sh /usr/local/bin/
COPY --chmod=755 shared/bin/service_check_passthrough.sh /usr/local/bin/
COPY --chmod=755 shared/bin/zeek_carve*.py /usr/local/bin/
COPY --chmod=755 file-monitor/scripts/*.py /usr/local/bin/
COPY --chmod=755 shared/bin/extracted_files_http_server.py /usr/local/bin/
COPY --chmod=644 shared/bin/watch_common.py /usr/local/bin/
COPY --chmod=644 scripts/malcolm_utils.py /usr/local/bin/
COPY --chmod=644 file-monitor/supervisord.conf /etc/supervisord.conf
Expand Down
2 changes: 1 addition & 1 deletion Dockerfiles/filebeat.Dockerfile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM docker.elastic.co/beats/filebeat-oss:8.12.1
FROM docker.elastic.co/beats/filebeat-oss:8.13.2

# Copyright (c) 2024 Battelle Energy Alliance, LLC. All rights reserved.
LABEL maintainer="[email protected]"
Expand Down
2 changes: 1 addition & 1 deletion Dockerfiles/logstash.Dockerfile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM docker.elastic.co/logstash/logstash-oss:8.12.1
FROM docker.elastic.co/logstash/logstash-oss:8.13.2

LABEL maintainer="[email protected]"
LABEL org.opencontainers.image.authors='[email protected]'
Expand Down
2 changes: 1 addition & 1 deletion Dockerfiles/opensearch.Dockerfile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM opensearchproject/opensearch:2.12.0
FROM opensearchproject/opensearch:2.13.0

# Copyright (c) 2024 Battelle Energy Alliance, LLC. All rights reserved.
LABEL maintainer="[email protected]"
Expand Down
Loading

0 comments on commit a6248b6

Please sign in to comment.