xixu-me/xget
1
0 Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

Revise and expand project documentation in CLAUDE.md

Browse Source 
Updated the CLAUDE.md file to provide a more comprehensive and structured overview of the Xget project. The changes include expanded feature lists, clearer architecture and testing sections, detailed command references, platform support instructions, security considerations, and deployment guidance. The documentation is now more accessible for new contributors and better reflects the current state and best practices of the project.
This commit is contained in:
xixu-me committed 2025-11-16 21:51:09 +08:00
1 parent 973daacf2c
commit f74850e5a2
1 file changed
+201 -180
Download Patch File Download Diff File Expand all files Collapse all files
CLAUDE.md
+201 -180
View File
@@ -4,226 +4,247 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
## Project Overview ## Project Overview
**Xget** is a high-performance acceleration engine for developer resources deployed on Cloudflare Workers. It acts as a unified proxy/acceleration layer for 40+ platforms including code repositories (GitHub, GitLab, Gitea), package managers (npm, PyPI, conda, Maven, etc.), AI inference providers (OpenAI, Anthropic, Gemini), container registries (Docker Hub, GHCR), and Linux distributions. Xget is a high-performance acceleration engine for developer resources built on Cloudflare Workers. It provides unified acceleration for code repositories, package managers, container registries, AI inference providers, and more. The application proxies requests to various platforms, applies intelligent caching, and implements security measures while maintaining protocol compliance (Git, Docker/OCI, AI APIs).
**Primary purpose**: Accelerate access to developer resources for users in regions with connectivity challenges, particularly mainland China. **Key Features:**
## Technology Stack - Multi-platform support (40+ platforms: GitHub, GitLab, npm, PyPI, Docker Hub, OpenAI, etc.)
- Smart caching with protocol-aware strategies (no cache for Git/Docker/AI, 30-min cache for downloads)
- HTTP Range request support for partial downloads
- Container registry authentication proxy
- Retry logic with exponential backoff
- Comprehensive security headers (HSTS, CSP, X-Frame-Options)
- **Runtime**: Cloudflare Workers (serverless edge computing) ## Commands
- **Language**: JavaScript (ES2022) with JSDoc type annotations
- **Testing**: Vitest with `@cloudflare/vitest-pool-workers`
- **Deployment**: Wrangler CLI + GitHub Actions
- **Code Quality**: ESLint + Prettier
- **Type Checking**: TypeScript via JSDoc (no build step)
## Development Commands ### Development
```bash ```bash
# Development # Start local development server with Wrangler
npm run dev # Start Wrangler dev server (same as npm start) npm run dev
# Testing # Type check without emitting files
npm test # Run Vitest in watch mode npm run type-check
npm run test:run # Run tests once (CI mode) ```
npm run test:coverage # Run tests with coverage report
npm run test:ui # Open Vitest UI
# Code Quality ### Testing
npm run lint # Run ESLint
npm run lint:fix # Fix ESLint errors automatically
npm run format # Format code with Prettier
npm run format:check # Check formatting without changes
npm run type-check # TypeScript/JSDoc type checking (no emit)
# Deployment ```bash
npm run deploy # Deploy to Cloudflare Workers # Run all tests in watch mode
npm test
# Run tests once (CI mode)
npm run test:run
# Run tests with UI
npm run test:ui
# Generate coverage report (requires 80% coverage)
npm run test:coverage
# Watch mode
npm run test:watch
```
### Code Quality
```bash
# Run ESLint
npm run lint
# Auto-fix ESLint issues
npm run lint:fix
# Format code with Prettier
npm run format
# Check formatting without modifying files
npm run format:check
```
### Deployment
```bash
# Deploy to Cloudflare Workers
npm run deploy
# or
wrangler deploy
# Start production preview
npm start
```
### Docker Deployment
```bash
# Build and run with Docker
docker build -t xget .
docker run -p 8080:8080 xget
# Or with Podman
podman build -t xget .
podman run -p 8080:8080 xget
``` ```
## Architecture ## Architecture
### Core Components ### Core Components
1. **Main Request Handler** ([src/index.js](src/index.js)) **`src/index.js`** (1432 lines) - Main request handler
- Entry point with `fetch()` event handler
- Protocol detection (Git, Git LFS, Docker, AI inference)
- Request validation, retry logic, timeout handling
- Performance monitoring (`PerformanceMonitor` class)
- Security header management
2. **Configuration** ([src/config/index.js](src/config/index.js)) - `handleRequest()` - Core request processor with caching, retry logic, security validation
- `createConfig(env)` - runtime config with environment overrides - `PerformanceMonitor` - Tracks timing metrics throughout request lifecycle
- `CONFIG` object with defaults (cache TTL, timeouts, retries) - Protocol detection: Git (git-upload-pack), Git LFS, Docker/OCI (v2 API), AI inference providers
- URL transformation and platform routing
- Docker registry authentication proxy (token fetch, WWW-Authenticate handling)
- Response rewriting for protocol compliance
3. **Platform Definitions** ([src/config/platforms.js](src/config/platforms.js)) **`src/config/index.js`** (180 lines) - Configuration management
- `PLATFORMS` object mapping prefixes to base URLs (40+ entries)
- `transformPath()` function for URL transformations - `createConfig(env)` - Creates config with environment variable overrides
- Platform-specific special cases (crates.io API, Jenkins paths, etc.) - `CONFIG` - Default config object
- Settings: timeout (30s), retries (3), cache duration (1800s), security rules
**`src/config/platforms.js`** (395 lines) - Platform definitions
- `PLATFORMS` - Maps 40+ platform prefixes to base URLs
- Code repos: `gh`, `gl`, `gitea`, `codeberg`, `sf`, `aosp`, `hf`, `civitai`
- Package managers: `npm`, `pypi`, `conda`, `maven`, `gradle`, `rubygems`, `cran`, `cpan`, `golang`, `nuget`, `crates`, `packagist`
- Linux distros: `debian`, `ubuntu`, `fedora`, `rocky`, `opensuse`, `arch`
- AI providers: `ip-openai`, `ip-anthropic`, `ip-gemini`, etc. (prefix: `ip-`)
- Container registries: `cr-docker`, `cr-ghcr`, `cr-gcr`, etc. (prefix: `cr-`)
- `transformPath()` - Converts prefixed URLs to actual platform URLs
### Request Flow ### Request Flow
``` ```
Request → Validate → Detect Protocol → Transform URL → Check Cache 1. Receive request → Validate security (method, path length, origin)
→ Fetch Upstream (with retries) → Handle Docker Auth → Rewrite Response 2. Detect protocol → Git/Git LFS/Docker/AI/Standard
→ Add Security Headers → Cache → Return with Performance Metrics 3. Parse URL → Extract platform prefix and path
4. Transform → Convert to actual platform URL via PLATFORMS config
5. Cache check → Skip for Git/Docker/AI protocols
6. Fetch upstream → With timeout, retries, exponential backoff
7. Handle auth → Docker token proxy for container registries
8. Rewrite response → Docker Content-Digest, URL rewriting
9. Add headers → Security headers, performance metrics, CORS
10. Cache → Store successful responses (protocol-dependent)
11. Return → Final response with all headers
``` ```
### Protocol Detection
The system handles different protocols with specialized logic:
- **Git requests**: Via `/info/refs`, `/git-upload-pack` endpoints, User-Agent patterns
- **Git LFS**: LFS-specific headers and endpoints
- **Docker/OCI registries**: `/v2/` prefix, Docker headers, token auth flow
- **AI inference**: `/ip/` prefix for AI API proxying
- **Regular downloads**: Default behavior with intelligent caching
### Security Features
All responses include security headers:
- `Strict-Transport-Security` (HSTS)
- `X-Frame-Options: DENY`
- `X-XSS-Protection`
- `Content-Security-Policy`
- `Referrer-Policy`
- `Permissions-Policy`
Request validation:
- Method whitelist (GET/HEAD default, POST/PUT/PATCH for Git/Docker/AI)
- Path length limit (2048 characters)
- 30-second timeout protection
### Caching Strategy ### Caching Strategy
- **Cache bypass**: Git, Git LFS, Docker, AI inference requests skip cache - **No cache:** Git operations, Git LFS, Docker/OCI registry, AI inference APIs
- **Cache TTL**: 1800 seconds (30 minutes) default - **30-min cache:** Standard file downloads, package registry files
- **Range requests**: Full support for HTTP 206 partial content - **Range requests:** Cache full content when partial range requested
- **Smart cache keys**: Different keys for Range vs full-content requests - Cache key: Full request URL including query parameters
### Content Transformation ### Testing Structure
- **PyPI URLs**: Rewrites `files.pythonhosted.org` URLs in HTML responses - `test/index.test.js` - Core handler tests
- **npm URLs**: Rewrites tarball URLs in JSON registry responses - `test/platforms.test.js` - Platform URL transformation tests
- **Streaming**: Uses `ReadableStream` for efficient large file handling - `test/integration.test.js` - End-to-end integration tests
- `test/security.test.js` - Security validation tests
- `test/performance.test.js` - Performance benchmarking
- `test/container-registry.test.js` - Docker/OCI registry tests
- `test/range-cache.test.js` - HTTP Range request tests
- `test/setup.js` - Global test configuration
- `test/helpers/` - Test utilities
- `test/fixtures/` - Test data
## Testing **Test configuration:**
### Test Structure - Framework: Vitest with Cloudflare Workers pool (`@cloudflare/vitest-pool-workers`)
- Coverage: 80% minimum (branches, functions, lines, statements)
Tests are organized by functionality in `test/`: - Timeout: 30 seconds per test
- Retries: 2 attempts
- `index.test.js` - Core request handling
- `integration.test.js` - End-to-end platform tests
- `security.test.js` - Security headers and validation
- `performance.test.js` - Performance monitoring
- `platforms.test.js` - URL transformation logic
- `container-registry.test.js` - Docker/OCI registry flows
- `range-cache.test.js` - HTTP Range requests
- Platform-specific: `npm-fix.test.js`, `crates.test.js`, `homebrew.test.js`, `opensuse.test.js`, `cran.test.js`, `jenkins.test.js`
### Coverage Requirements
Minimum 80% coverage for:
- Branches
- Functions
- Lines
- Statements
### Running Specific Tests
```bash
# Run specific test file
npm test -- index.test.js
# Run tests matching pattern
npm test -- --grep "Docker"
# Run with coverage
npm run test:coverage
```
## Code Style ## Code Style
- **JSDoc annotations**: All functions must have JSDoc type annotations **ESLint configuration (eslint.config.js):**
- **ES modules**: Use `import/export`, not `require()`
- **Async/await**: Prefer over Promise chains
- **Error handling**: All fetch operations must have retry logic and timeout protection
## Important Patterns - 2-space indentation
- Single quotes for strings
- Semicolons required
- Camelcase naming (except properties)
- No trailing spaces
- Unix line endings
- No `var`, use `const` or `let`
- Prefer arrow functions and template literals
### Adding a New Platform **Naming conventions:**
1. Add entry to `PLATFORMS` object in [src/config/platforms.js](src/config/platforms.js): - Variables/functions: `camelCase`
- Constants: `UPPER_SNAKE_CASE`
- Classes: `PascalCase`
- Files: `kebab-case` (though this repo uses `.js` files)
```js **JSDoc comments required:**
'prefix': 'https://example.com'
```
2. If special URL transformation needed, add case in `transformPath()`: - All functions must have JSDoc with `@param`, `@returns`, `@throws`
- Include `@example` blocks for non-trivial functions
```js - Extensive inline comments for complex logic
if (platform === 'prefix' && /* condition */) {
// Custom transformation
}
```
3. Add integration test in [test/integration.test.js](test/integration.test.js)
4. Update README.md with usage examples
### Retry Logic Pattern
All upstream fetches use:
```js
for (let attempt = 1; attempt <= maxRetries; attempt++) {
try {
// Fetch with AbortController for timeout
const response = await fetch(url, { signal: controller.signal });
// Handle response
break;
} catch (error) {
// Only retry on 5xx or network errors, not 4xx
if (attempt === maxRetries || isClientError(response)) throw error;
}
}
```
### Performance Monitoring
Use `PerformanceMonitor` class to track timing:
```js
const perfMon = new PerformanceMonitor();
perfMon.mark('operation_start');
// ... do work ...
perfMon.mark('operation_complete');
const metrics = perfMon.getMetrics(); // Returns timing data
```
## Deployment
- **Target**: Cloudflare Workers
- **Config**: [wrangler.toml](wrangler.toml)
- **CI/CD**: GitHub Actions ([.github/workflows/depoly.yml](.github/workflows/depoly.yml))
- **Secrets required**: `CLOUDFLARE_API_TOKEN`, `CLOUDFLARE_ACCOUNT_ID`
## Environment Variables ## Environment Variables
Configuration can be overridden via Cloudflare Workers environment variables: Configure via Cloudflare Workers environment or `.dev.vars`:
- `CACHE_TTL` - Cache time-to-live in seconds (default: 1800) - `TIMEOUT_SECONDS` - Request timeout (default: 30)
- `MAX_RETRIES` - Maximum retry attempts (default: 3) - `MAX_RETRIES` - Max retry attempts (default: 3)
- `TIMEOUT` - Request timeout in milliseconds (default: 30000) - `RETRY_DELAY_MS` - Retry delay (default: 1000)
- Platform-specific base URLs can be overridden by setting env var matching platform key - `CACHE_DURATION` - Cache TTL in seconds (default: 1800)
- `ALLOWED_METHODS` - Comma-separated HTTP methods (default: 'GET,HEAD')
- `ALLOWED_ORIGINS` - Comma-separated CORS origins (default: '*')
- `MAX_PATH_LENGTH` - Max URL path length (default: 2048)
## Key Files to Reference ## Platform Support
- [src/index.js](src/index.js) - Main application logic (~900 lines) ### Adding New Platforms
- [src/config/platforms.js](src/config/platforms.js) - All platform definitions and transformations
- [src/config/index.js](src/config/index.js) - Configuration management 1. Edit `src/config/platforms.js`:
- [wrangler.toml](wrangler.toml) - Cloudflare Workers deployment config
- [vitest.config.js](vitest.config.js) - Test configuration ```javascript
export const PLATFORMS = {
// ... existing platforms
'new-prefix': 'https://platform-url.com'
};
```
2. Add URL transformation logic in `transformPath()` if needed (for special path handling)
3. Add tests in `test/platforms.test.js`:
```javascript
it('should transform new-prefix URLs', () => {
const result = transformPath('new-prefix/path/to/resource');
expect(result).toBe('https://platform-url.com/path/to/resource');
});
```
4. Update README.md with platform badge and usage instructions
### Platform Categories
- **Code repositories:** Direct path passthrough
- **Package managers:** May require special path handling (e.g., npm scoped packages)
- **Container registries (cr- prefix):** Docker/OCI protocol, authentication proxy
- **AI providers (ip- prefix):** No caching, streaming support
## Security Considerations
- All GPLv3 licensed - include license header in new files
- Validate HTTP methods (allow POST for Git operations)
- Path length limits (2048 chars default)
- Security headers applied to all responses
- No XSS or injection vulnerabilities - all URLs validated
- Docker auth tokens proxied securely (never stored)
- CORS headers configurable per deployment
## Deployment Targets
1. **Cloudflare Workers** (primary): Zero-config deployment with global edge network
2. **Docker/Podman**: Self-hosted with workerd runtime (see Dockerfile)
3. **Local development**: Wrangler dev server on localhost
## License
GPLv3 - All new source files must include the GPL header (see existing files for template).