This is a personal hobby project; based on the observation, sometimes we need a simple enough CLI tool with auditable capability for Data Masking/Encryption/Decryption for CSV/JSON files.
- Rust powered performance.
- Provide Masking and Encyption/Decryption capabilities.
- Auditable with a built-in SQLite-powered Audit table.
There are two central crates in this package.
- MED_CLI - the CLI interface for the med binary.
- MED_CORE - the core engineer to execution the CSV/JSON files Masking, Encryption, and Decryption, which you can use in your use-case/project/context implementation.
- Introduction M.E.D.
- Build a CLI Tool for Data Masking, Encryption, and Decryption With Rust
- Reduce memory footprint by about 600% for M.E.D. — Performance Matters
The binary name for M.E.D. is med; it depends on the med_core.
Archives of precompiled binaries for med are available for Windows, macOS, and Linux. Users of platforms not explicitly mentioned below are advised to download one of these archives.
dnf install medbrew tap jayhuang75/med
brew install medModel Name: MacBook Pro
Processor Name: 6-Core Intel Core i7
Processor Speed: 2.6 GHz
Total Number of Cores: 6
Memory: 16 GB| File Type | Records | File Size | File Counts | Mode | Field num for mask/encrypt | Elapsed Time | Memory Consumption |
|---|---|---|---|---|---|---|---|
| CSV | 120,000,000 | 2.8G | 3 | mask | 1 | ~78 seconds (1.3 mins) | ~2 MB |
| CSV | 120,000,000 | 2.8G | 3 | encrypt (DES64) | 1 | ~182 seconds (3 mins) | ~1.9 MB |
| CSV | 120,000,000 | 2.8G | 3 | encrypt (AES128) | 1 | ~221 seconds (3.6 mins) | ~1.9 MB |
| CSV | 20,000,000 | 471M | 2 | mask | 1 | ~7 seconds | ~1.8 MB |
| CSV | 10,000,000 | 236M | 1 | mask | 1 | ~5 seconds | ~1.8 MB |
| JSON | 129,220 | 10G | 129,220 | mask | 1 | ~2200 seconds(36 mins) | ~62 MB |
| JSON | 64,641 | 5.1G | 64,641 | mask | 1 | ~792 seconds(13 mins) | ~30 MB |
| JSON | 23,519 | 1.9G | 23,519 | mask | 1 | ~284 seconds(4.7 mins) | ~18 MB |
| JSON | 23,519 | 1.9G | 23,519 | encrypt | 1 | ~282 seconds(4.4 mins) | ~18 MB |
This project follows the all-contributors specification.
Please complete the template before the PR.
Contributions of any kind are welcome!
Give a ⭐️ if this project helped you!
Copyright © 2023 Wei Huang
This project is Licensed under Apache.