Skip to content

Parseable is a disk less, cloud native database for logs, observability, security, and compliance. Parseable is built with focus on simplicity & resource efficiency.

License

Notifications You must be signed in to change notification settings

parseablehq/parseable

Folders and files

NameName
Last commit message
Last commit date
May 14, 2023
Mar 13, 2023
May 18, 2023
May 18, 2023
May 18, 2023
Jan 10, 2023
Sep 19, 2022
Feb 10, 2023
Mar 30, 2023
Sep 8, 2022
Aug 4, 2022
May 18, 2023
Jan 22, 2022
Mar 7, 2023
Mar 7, 2023
Mar 11, 2023
Aug 10, 2022
May 29, 2022
May 1, 2023
May 14, 2023
May 14, 2023
Apr 16, 2023
May 18, 2023

Repository files navigation

Parseable Logo
Cloud native log analytics

Docker Pulls Slack Docs Build

Parseable is a lightweight, cloud native log observability and analytics engine. It is written in Rust and uses Apache Arrow and Parquet.

Parseable uses a simple, index-free mechanism to organize and query data allowing low latency, and high throughput ingestion and query. It can use either a local mount point or object storage (S3/compatible stores) for data storage.

For comparison, Parseable consumes up to ~80% lower memory and ~50% lower CPU than Elastic for similar ingestion throughput. Read more in the benchmarks directory.

πŸš€ Features

  • Choose your own storage backend - local drive or S3 (or compatible) object store.
  • Ingestion API compatible with HTTP + JSON output of log agents.
  • Query log data with PostgreSQL compatible SQL.
  • Grafana β†—οΈŽ for visualization.
  • Send alerts β†—οΈŽ to webhook targets including Slack.
  • Stats API β†—οΈŽ to track ingestion and compressed data.
  • Single binary includes all components - ingestion, store and query. Built-in UI.

βœ… Getting Started

Run the below command to deploy Parseable in local storage mode with Docker.

mkdir -p $HOME/parseable/data
mkdir -p $HOME/parseable/staging

docker run -p 8000:8000 \
  -v $HOME/parseable/data:/parseable/data \
  -v $HOME/parseable/staging:/parseable/staging \
  -e P_FS_DIR=/parseable/data \
  -e P_STAGING_DIR=/parseable/staging \
  parseable/parseable:latest \
  parseable local-store

Once this runs successfully, you'll see dashboard at http://localhost:8000. You can login to the dashboard default credentials admin, admin.

Send log events

curl --location --request POST 'http://localhost:8000/api/v1/ingest' \
--header 'X-P-META-meta1: value1' \
--header 'X-P-TAG-tag1: value1' \
--header 'X-P-Stream: demo' \
--header 'Authorization: Basic YWRtaW46YWRtaW4=' \
--header 'Content-Type: application/json' \
--data-raw '[
    {
        "id": "434a5f5e-2f5f-11ed-a261-0242ac120002",
        "datetime": "24/Jun/2022:14:12:15 +0000",
        "host": "153.10.110.81",
        "user-identifier": "Mozilla/5.0 Gecko/20100101 Firefox/64.0",
        "method": "PUT",
        "status": 500,
        "referrer": "http://www.google.com/"
    }
]'

Note: The X-P-Stream header is used to specify the log stream where data is sent. Parseable will create the stream automatically if it doesn't exist already.

Query the stream

You can see the events in Parseable UI, or use the below curl command to see the query response on CLI.

NOTE: Please change the startTime and endTime to the time range corresponding to the event you sent in the previous step.

curl --location --request POST 'http://localhost:8000/api/v1/query' \
--header 'Authorization: Basic YWRtaW46YWRtaW4=' \
--header 'Content-Type: application/json' \
--data-raw '{
    "query":"select * from demo",
    "startTime":"2023-06-09T00:00:00+00:00",
    "endTime":"2023-06-09T23:59:00+00:00"
}'

πŸ“ˆ Benchmarking

Parseable is benchmarked with K6. Please find the results and details on how to run the benchmark in your environment in the benchmarks directory.

πŸ“š Documentation

🎯 Motivation

Traditionally, logging has been seen as a text search problem. Log volumes were not high, and data ingestion or storage were not really issues. This led us to today, where all the logging platforms are primarily text search engines.

But with log data growing exponentially, today's log data challenges involve whole lot more – Data ingestion, storage, and observation, all at scale. We are building Parseable to address these challenges.

🩺 Support

πŸ† Contributing

Refer to the contributing guide here β†—οΈŽ.

Contributors

Supported by