xixu-me/tzst
1
0 Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity
Files
698eb9cf09543cc8d0187ea36011df481ba0dad2
tzst/README.md
T
google-labs-jules[bot] e2fca92d4b docs: remove year from copyright info in READMEs and docs 
Removes the year "2025" from the copyright section in all README files (all languages) and docs/index.md, as requested. The copyright line now reads "Copyright (c) [Author]..." without a specific year.
2025-12-31 17:24:55 +00:00

526 lines
15 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
<h1 align="center">
<img src="https://raw.githubusercontent.com/xixu-me/tzst/refs/heads/main/docs/_static/tzst-logo.png" width="300">
</h1><br>
[![codecov](https://codecov.io/gh/xixu-me/tzst/graph/badge.svg?token=2AIN1559WU)](https://codecov.io/gh/xixu-me/tzst)
[![CodeQL](https://github.com/xixu-me/tzst/actions/workflows/github-code-scanning/codeql/badge.svg)](https://github.com/xixu-me/tzst/actions/workflows/github-code-scanning/codeql)
[![CI/CD](https://github.com/xixu-me/tzst/actions/workflows/ci.yml/badge.svg)](https://github.com/xixu-me/tzst/actions/workflows/ci.yml)
[![PyPI - Version](https://img.shields.io/pypi/v/tzst)](https://pypi.org/project/tzst/)
[![PyPI - Downloads](https://img.shields.io/pypi/dm/tzst)](https://pypistats.org/packages/tzst)
[![GitHub License](https://img.shields.io/github/license/xixu-me/tzst)](LICENSE)
[![Sponsor](https://img.shields.io/badge/Sponsor-violet)](https://xi-xu.me/#sponsorships)
[![Documentation](https://img.shields.io/badge/Documentation-blue)](https://tzst.xi-xu.me)
**🇺🇸 English** | [🇨🇳 汉语](./README.zh.md) | [🇪🇸 español](./README.es.md) | [🇯🇵 日本語](./README.ja.md) | [🇦🇪 العربية](./README.ar.md) | [🇷🇺 русский](./README.ru.md) | [🇩🇪 Deutsch](./README.de.md) | [🇫🇷 français](./README.fr.md) | [🇰🇷 한국어](./README.ko.md) | [🇧🇷 português](./README.pt.md)
**tzst** is a next-generation Python library engineered for modern archive management, leveraging cutting-edge Zstandard compression to deliver superior performance, security, and reliability. Built exclusively for Python 3.12+, this enterprise-grade solution combines atomic operations, streaming efficiency, and a meticulously crafted API to redefine how developers handle `.tzst`/`.tar.zst` archives in production environments. 🚀
In-depth technical analysis article published: **[Deep Dive into tzst: A Modern Python Archiving Library Based on Zstandard](https://blog.xi-xu.me/2025/11/01/deep-dive-into-tzst-en.html)**.
## ✨ Features
- **🗜️ High Compression**: Zstandard compression for excellent compression ratios and speed
- **📁 Tar Compatibility**: Creates standard tar archives compressed with Zstandard
- **💻 Command Line Interface**: Intuitive CLI with streaming support and comprehensive options
- **🐍 Python API**: Clean, Pythonic API for programmatic use
- **🌍 Cross-Platform**: Works on Windows, macOS, and Linux
- **📂 Multiple Extensions**: Supports both `.tzst` and `.tar.zst` extensions
- **💾 Memory Efficient**: Streaming mode for handling large archives with minimal memory usage
- **⚡ Atomic Operations**: Safe file operations with automatic cleanup on interruption
- **🔒 Secure by Default**: Uses the 'data' filter for maximum security during extraction
- **🚨 Enhanced Error Handling**: Clear error messages with helpful alternatives
## 📥 Installation
### From GitHub Releases
Download standalone executables that don't require Python installation:
#### Supported Platforms
| Platform | Architecture | File |
|----------|-------------|------|
| **🐧 Linux** | x86_64 | `tzst-{version}-linux-amd64.zip` |
| **🐧 Linux** | ARM64 | `tzst-{version}-linux-arm64.zip` |
| **🪟 Windows** | x64 | `tzst-{version}-windows-amd64.zip` |
| **🪟 Windows** | ARM64 | `tzst-{version}-windows-arm64.zip` |
| **🍎 macOS** | Intel | `tzst-{version}-darwin-amd64.zip` |
| **🍎 macOS** | Apple Silicon | `tzst-{version}-darwin-arm64.zip` |
#### 🛠️ Installation Steps
1. **📥 Download** the appropriate archive for your platform from the [latest releases page](https://github.com/xixu-me/tzst/releases/latest)
2. **📦 Extract** the archive to get the `tzst` executable (or `tzst.exe` on Windows)
3. **📂 Move** the executable to a directory in your PATH:
- **🐧 Linux/macOS**: `sudo mv tzst /usr/local/bin/`
- **🪟 Windows**: Add the directory containing `tzst.exe` to your PATH environment variable
4. **✅ Verify** installation: `tzst --help`
#### 🎯 Benefits of Binary Installation
- ✅ **No Python required** - Standalone executable
- ✅ **Faster startup** - No Python interpreter overhead
- ✅ **Easy deployment** - Single file distribution
- ✅ **Consistent behavior** - Bundled dependencies
### 📦 From PyPI
Using pip:
```bash
pip install tzst
```
Or using uv (recommended):
```bash
uv tool install tzst
```
### 🔧 From Source
```bash
git clone https://github.com/xixu-me/tzst.git
cd tzst
pip install .
```
### 🚀 Development Installation
This project uses modern Python packaging standards:
```bash
git clone https://github.com/xixu-me/tzst.git
cd tzst
pip install -e .[dev]
```
## 🚀 Quick Start
### 💻 Command Line Usage
```bash
# 📁 Create an archive
tzst a archive.tzst file1.txt file2.txt directory/
# 📤 Extract an archive
tzst x archive.tzst
# 📋 List archive contents
tzst l archive.tzst
# 🧪 Test archive integrity
tzst t archive.tzst
```
### 🐍 Python API Usage
```python
from tzst import create_archive, extract_archive, list_archive
# Create an archive
create_archive("archive.tzst", ["file1.txt", "file2.txt", "directory/"])
# Extract an archive
extract_archive("archive.tzst", "output_directory/")
# List archive contents
contents = list_archive("archive.tzst", verbose=True)
for item in contents:
print(f"{item['name']}: {item['size']} bytes")
```
## 💻 Command Line Interface
### 📁 Archive Operations
#### ➕ Create Archive
```bash
# Basic usage
tzst a archive.tzst file1.txt file2.txt
# With compression level (1-22, default: 3)
tzst a archive.tzst files/ -l 15
# Alternative commands
tzst add archive.tzst files/
tzst create archive.tzst files/
```
#### 📤 Extract Archive
```bash
# Extract with full directory structure
tzst x archive.tzst
# Extract to specific directory
tzst x archive.tzst -o output/
# Extract specific files
tzst x archive.tzst file1.txt dir/file2.txt
# Extract without directory structure (flat)
tzst e archive.tzst -o output/
# Use streaming mode for large archives
tzst x archive.tzst --streaming -o output/
```
#### 📋 List Contents
```bash
# Simple listing
tzst l archive.tzst
# Verbose listing with details
tzst l archive.tzst -v
# Use streaming mode for large archives
tzst l archive.tzst --streaming -v
```
#### 🧪 Test Integrity
```bash
# Test archive integrity
tzst t archive.tzst
# Test with streaming mode
tzst t archive.tzst --streaming
```
### 📊 Command Reference
| Command | Aliases | Description | Streaming Support |
|---------|---------|-------------|-------------------|
| `a` | `add`, `create` | Create or add to archive | N/A |
| `x` | `extract` | Extract with full paths | ✓ `--streaming` |
| `e` | `extract-flat` | Extract without directory structure | ✓ `--streaming` |
| `l` | `list` | List archive contents | ✓ `--streaming` |
| `t` | `test` | Test archive integrity | ✓ `--streaming` |
### ⚙️ CLI Options
- `-v, --verbose`: Enable verbose output
- `-o, --output DIR`: Specify output directory (extract commands)
- `-l, --level LEVEL`: Set compression level 1-22 (create command)
- `--streaming`: Enable streaming mode for memory-efficient processing
- `--filter FILTER`: Security filter for extraction (data/tar/fully_trusted)
- `--no-atomic`: Disable atomic file operations (not recommended)
### 🔒 Security Filters
```bash
# Extract with maximum security (default)
tzst x archive.tzst --filter data
# Extract with standard tar compatibility
tzst x archive.tzst --filter tar
# Extract with full trust (dangerous - only for trusted archives)
tzst x archive.tzst --filter fully_trusted
```
**🔐 Security Filter Options:**
- `data` (default): Most secure. Blocks dangerous files, absolute paths, and paths outside extraction directory
- `tar`: Standard tar compatibility. Blocks absolute paths and directory traversal
- `fully_trusted`: No security restrictions. Only use with completely trusted archives
## 🐍 Python API
### 📦 TzstArchive Class
```python
from tzst import TzstArchive
# Create a new archive
with TzstArchive("archive.tzst", "w", compression_level=5) as archive:
archive.add("file.txt")
archive.add("directory/", recursive=True)
# Read an existing archive
with TzstArchive("archive.tzst", "r") as archive:
# List contents
contents = archive.list(verbose=True)
# Extract with security filter
archive.extract("file.txt", "output/", filter="data")
# Test integrity
is_valid = archive.test()
# For large archives, use streaming mode
with TzstArchive("large_archive.tzst", "r", streaming=True) as archive:
archive.extract(path="output/")
```
**⚠️ Important Limitations:**
- **❌ Append Mode Not Supported**: Create multiple archives or recreate the entire archive instead
### 🎯 Convenience Functions
#### 📁 create_archive()
```python
from tzst import create_archive
# Create with atomic operations (default)
create_archive(
archive_path="backup.tzst",
files=["documents/", "photos/", "config.txt"],
compression_level=10
)
```
#### 📤 extract_archive()
```python
from tzst import extract_archive
# Extract with security (default: 'data' filter)
extract_archive("backup.tzst", "restore/")
# Extract specific files
extract_archive("backup.tzst", "restore/", members=["config.txt"])
# Flatten directory structure
extract_archive("backup.tzst", "restore/", flatten=True)
# Use streaming for large archives
extract_archive("large_backup.tzst", "restore/", streaming=True)
```
#### 📋 list_archive()
```python
from tzst import list_archive
# Simple listing
files = list_archive("backup.tzst")
# Detailed listing
files = list_archive("backup.tzst", verbose=True)
# Streaming for large archives
files = list_archive("large_backup.tzst", streaming=True)
```
#### 🧪 test_archive()
```python
from tzst import test_archive
# Basic integrity test
if test_archive("backup.tzst"):
print("Archive is valid")
# Test with streaming
if test_archive("large_backup.tzst", streaming=True):
print("Large archive is valid")
```
## 🔧 Advanced Features
### 📂 File Extensions
The library automatically handles file extensions with intelligent normalization:
- `.tzst` - Primary extension for tar+zstandard archives
- `.tar.zst` - Alternative standard extension
- Auto-detection when opening existing archives
- Automatic extension addition when creating archives
```python
# These all create valid archives
create_archive("backup.tzst", files) # Creates backup.tzst
create_archive("backup.tar.zst", files) # Creates backup.tar.zst
create_archive("backup", files) # Creates backup.tzst
create_archive("backup.txt", files) # Creates backup.tzst (normalized)
```
### 🗜️ Compression Levels
Zstandard compression levels range from 1 (fastest) to 22 (best compression):
- **Level 1-3**: Fast compression, larger files
- **Level 3** (default): Good balance of speed and compression
- **Level 10-15**: Better compression, slower
- **Level 20-22**: Maximum compression, much slower
### 🌊 Streaming Mode
Use streaming mode for memory-efficient processing of large archives:
**✅ Benefits:**
- Significantly reduced memory usage
- Better performance for archives that don't fit in memory
- Automatic cleanup of resources
**🎯 When to Use:**
- Archives larger than 100MB
- Limited memory environments
- Processing archives with many large files
```python
# Example: Processing a large backup archive
from tzst import extract_archive, list_archive, test_archive
large_archive = "backup_500gb.tzst"
# Memory-efficient operations
is_valid = test_archive(large_archive, streaming=True)
contents = list_archive(large_archive, streaming=True, verbose=True)
extract_archive(large_archive, "restore/", streaming=True)
```
### ⚡ Atomic Operations
All file creation operations use atomic file operations by default:
- Archives created in temporary files first, then atomically moved
- Automatic cleanup if process is interrupted
- No risk of corrupted or incomplete archives
- Cross-platform compatibility
```python
# Atomic operations enabled by default
create_archive("important.tzst", files) # Safe from interruption
# Can be disabled if needed (not recommended)
create_archive("test.tzst", files, use_temp_file=False)
```
### 🚨 Error Handling
```python
from tzst import TzstArchive
from tzst.exceptions import (
TzstError,
TzstArchiveError,
TzstCompressionError,
TzstDecompressionError,
TzstFileNotFoundError
)
try:
with TzstArchive("archive.tzst", "r") as archive:
archive.extract()
except TzstDecompressionError:
print("Failed to decompress archive")
except TzstFileNotFoundError:
print("Archive file not found")
except KeyboardInterrupt:
print("Operation interrupted by user")
# Cleanup handled automatically
```
## 🚀 Performance and Comparison
### 💡 Performance Tips
1. **🗜️ Compression levels**: Level 3 is optimal for most use cases
2. **🌊 Streaming**: Use for archives larger than 100MB
3. **📦 Batch operations**: Add multiple files in single session
4. **📄 File types**: Already compressed files won't compress much further
### 🆚 vs Other Tools
**vs tar + gzip:**
- ✅ Better compression ratios
- ⚡ Faster decompression
- 🔄 Modern algorithm
**vs tar + xz:**
- 🚀 Significantly faster compression
- 📊 Similar compression ratios
- ⚖️ Better speed/compression trade-off
**vs zip:**
- 🗜️ Better compression
- 🔐 Preserves Unix permissions and metadata
- 🌊 Better streaming support
## 📋 Requirements
- 🐍 Python 3.12 or higher
- 📦 zstandard >= 0.19.0
## 🛠️ Development
### 🚀 Setting up Development Environment
This project uses modern Python packaging standards:
```bash
git clone https://github.com/xixu-me/tzst.git
cd tzst
pip install -e .[dev]
```
### 🧪 Running Tests
```bash
# Run tests with coverage
pytest --cov=tzst --cov-report=html
# Or use the simpler command (coverage settings are in pyproject.toml)
pytest
```
### ✨ Code Quality
```bash
# Check code quality
ruff check src tests
# Format code
ruff format src tests
```
## 🤝 Contributing
We welcome contributions! Please read our [Contributing Guide](CONTRIBUTING.md) for:
- Development setup and project structure
- Code style guidelines and best practices
- Testing requirements and writing tests
- Pull request process and review workflow
### 🚀 Quick Start for Contributors
```bash
git clone https://github.com/xixu-me/tzst.git
cd tzst
pip install -e .[dev]
python -m pytest tests/
```
### 🎯 Types of Contributions Welcome
- 🐛 **Bug fixes** - Fix issues in existing functionality
- ✨ **Features** - Add new capabilities to the library
- 📚 **Documentation** - Improve or add documentation
- 🧪 **Tests** - Add or improve test coverage
- ⚡ **Performance** - Optimize existing code
- 🔒 **Security** - Address security vulnerabilities
## 🙏 Acknowledgments
- [Meta Zstandard](https://github.com/facebook/zstd) for the excellent compression algorithm
- [python-zstandard](https://github.com/indygreg/python-zstandard) for Python bindings
- The Python community for inspiration and feedback
## 📄 License
Copyright &copy; [Xi Xu](https://xi-xu.me). All rights reserved.
Licensed under the [BSD 3-Clause](LICENSE) license.
Reference in new issue View Git Blame Copy Permalink