tzst/docs/performance.md

myst

myst

html_meta description keywords og:title og:description twitter:title twitter:description og:type og:image og:url twitter:card twitter:image tzst Performance Guide - Compression level optimization, performance tips, and comparison with other archive tools tzst performance, compression benchmarks, tar gzip comparison, archive performance optimization tzst Performance Guide Performance optimization tips and comparison with other archive tools for tzst tzst Performance Guide Performance optimization tips and comparison with other archive tools for tzst website https://tzst.xi-xu.me/_static/tzst-square-logo.png https://tzst.xi-xu.me/ summary_large_image https://tzst.xi-xu.me/_static/tzst-square-logo.png

Performance Guide

This guide covers performance optimization techniques and provides detailed comparisons with other archive tools.

Performance Tips

1. Compression Levels

Choose the right compression level for your use case:

• Level 1-3: Fast compression, larger files (good for temporary archives or real-time processing)
• Level 3 (default): Optimal balance for most use cases
• Level 6-9: Higher compression, moderate speed (good for regular backups)
• Level 15-22: Maximum compression, slower (for long-term storage or bandwidth-limited scenarios)

from tzst import create_archive

# For temporary files or frequent operations
create_archive("temp.tzst", files, compression_level=1)

# Balanced default (recommended)
create_archive("backup.tzst", files, compression_level=3)

# Long-term storage
create_archive("archive.tzst", files, compression_level=9)

# Maximum compression for critical space savings
create_archive("minimal.tzst", files, compression_level=22)

2. Streaming

Use streaming mode for archives larger than 100MB:

from tzst import extract_archive, list_archive, test_archive

# Memory-efficient operations for large archives
extract_archive("large-backup.tzst", "restore/", streaming=True)
contents = list_archive("large-backup.tzst", streaming=True)
is_valid = test_archive("large-backup.tzst", streaming=True)

Streaming Benefits:

• Significantly reduced memory usage
• Better performance for large archives
• Handles archives that don't fit in memory

3. Batch Operations

Add multiple files in a single session when possible:

from tzst import TzstArchive

# Efficient: Single archive session
with TzstArchive("backup.tzst", "w") as archive:
    archive.add("file1.txt")
    archive.add("file2.txt")
    archive.add("directory/", recursive=True)

# Less efficient: Multiple separate operations
create_archive("backup1.tzst", ["file1.txt"])
create_archive("backup2.tzst", ["file2.txt"])

4. File Type Considerations

• Already compressed files (.jpg, .png, .mp4, .pdf) won't compress much further
• Text files, source code, and logs compress very well
• Consider compression level based on your data types

Comparison with Other Tools

vs tar + gzip

tzst Advantages:

• Better compression ratios: 10-40% smaller archives
• Faster decompression: 2-3x faster extraction
• Modern algorithm: Better handling of various file types
• Streaming support: Better memory efficiency

When to use tar + gzip:

• Legacy system compatibility requirements
• Very old systems without zstd support

vs tar + xz

tzst Advantages:

• Significantly faster compression: 3-10x faster creation
• Faster decompression: 2-4x faster extraction
• Better speed/compression trade-off: Similar compression with much better speed
• More compression levels: Fine-grained control (22 levels vs 9)

When to use tar + xz:

• Maximum compression is critical and time is not a factor
• Systems that don't support zstd

vs zip

tzst Advantages:

• Better compression: 15-30% smaller archives
• Preserves Unix permissions and metadata: Full POSIX compatibility
• Better streaming support: Memory-efficient for large archives
• Better directory handling: Preserves directory structure and timestamps

When to use zip:

• Cross-platform compatibility with very old systems
• Individual file access without full extraction is required
• Windows-centric environments with no command-line tools

Benchmarking Examples

Compression Level Benchmark

import time
from pathlib import Path
from tzst import create_archive

def benchmark_compression_levels(files, output_prefix="benchmark"):
    """Compare different compression levels."""
    levels_to_test = [1, 3, 6, 9, 15, 22]
    
    results = []
    for level in levels_to_test:
        output_file = f"{output_prefix}_level_{level}.tzst"
        
        # Measure compression time
        start_time = time.time()
        create_archive(output_file, files, compression_level=level)
        compress_time = time.time() - start_time
        
        # Get file size
        file_size = Path(output_file).stat().st_size
        
        results.append({
            'level': level,
            'time': compress_time,
            'size': file_size,
            'size_mb': file_size / (1024 * 1024)
        })
        
        print(f"Level {level}: {compress_time:.2f}s, {file_size/1024/1024:.1f} MB")
    
    return results

# Example usage
files = ["documents/", "projects/"]
results = benchmark_compression_levels(files)

Memory Usage Comparison

import psutil
import os
from tzst import extract_archive

def monitor_memory_usage(func, *args, **kwargs):
    """Monitor memory usage during function execution."""
    process = psutil.Process(os.getpid())
    initial_memory = process.memory_info().rss / 1024 / 1024  # MB
    
    func(*args, **kwargs)
    
    peak_memory = process.memory_info().rss / 1024 / 1024  # MB
    return peak_memory - initial_memory

# Compare streaming vs non-streaming extraction
large_archive = "large-dataset.tzst"

memory_normal = monitor_memory_usage(extract_archive, large_archive, "output1/")
memory_streaming = monitor_memory_usage(extract_archive, large_archive, "output2/", streaming=True)

print(f"Normal extraction: {memory_normal:.1f} MB")
print(f"Streaming extraction: {memory_streaming:.1f} MB")
print(f"Memory savings: {memory_normal - memory_streaming:.1f} MB")

Best Practices

For Development

# Fast compression for frequent builds
create_archive("build-artifacts.tzst", ["build/"], compression_level=1)

For Backups

# Balanced compression for regular backups
create_archive("daily-backup.tzst", ["data/"], compression_level=6)

For Distribution

# Higher compression for software distribution
create_archive("software-package.tzst", ["app/"], compression_level=9)

For Archival Storage

# Maximum compression for long-term storage
create_archive("archive-2024.tzst", ["historical-data/"], compression_level=22)

Hardware Considerations

CPU Usage

• Higher compression levels use more CPU but for shorter time periods
• Modern multi-core systems handle zstd compression very efficiently
• Consider system load when choosing compression levels

Memory Usage

• Streaming mode: ~16-32 MB memory usage regardless of archive size
• Normal mode: Memory usage proportional to archive size
• Use streaming for archives >100 MB or on memory-constrained systems

Storage

• SSDs benefit from higher compression (less I/O)
• HDDs may prefer lower compression levels (CPU vs I/O trade-off)
• Network storage benefits from higher compression (bandwidth savings)

Integration with Build Systems

Makefile Example

# Fast compression for development
build-dev: 
 tzst a build-dev.tzst build/ -l 1

# Production compression
build-prod:
 tzst a build-prod.tzst build/ -l 9

# CI/CD artifacts
artifacts:
 tzst a artifacts.tzst dist/ logs/ -l 6

GitHub Actions Example

- name: Create release archive
  run: |
    tzst a release-${{ github.ref_name }}.tzst \
      build/ docs/ \
      --compression-level 9

This performance guide helps you choose the right settings for your specific use case and understand how tzst compares to alternative archive tools.