xixu-me/xget
1
0 Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

Add Vercel Functions workflow and sync automation

Browse Source 
Introduces a new GitHub Actions workflow for deploying to Vercel Functions and extends the sync workflow to auto-convert and push the codebase to a 'functions' branch for Vercel deployment. Updates workflow YAML files for consistent quoting and naming, refines documentation in CLAUDE.md for platform support and development, and updates package-lock.json dependencies.
This commit is contained in:
xixu-me committed 2025-11-25 14:49:42 +08:00
1 parent 988de5b6e2
commit 06561b5b2b
7 files changed
+546 -216
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

CLAUDE.md
+240 -169
View File
@@ -4,247 +4,318 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
## Project Overview
Xget is a high-performance acceleration engine for developer resources built on Cloudflare Workers. It provides unified acceleration for code repositories, package managers, container registries, AI inference providers, and more. The application proxies requests to various platforms, applies intelligent caching, and implements security measures while maintaining protocol compliance (Git, Docker/OCI, AI APIs).
Xget is a high-performance, security-focused acceleration engine for developer resources. It's a Cloudflare Workers application written in JavaScript that proxies and accelerates access to 40+ platforms including code repositories (GitHub, GitLab), package managers (npm, PyPI, Maven), container registries (Docker Hub, ghcr.io), AI inference providers (OpenAI, Anthropic), and Linux distributions.
**Key Features:**
**Key characteristics:**
- Multi-platform support (40+ platforms: GitHub, GitLab, npm, PyPI, Docker Hub, OpenAI, etc.)
- Smart caching with protocol-aware strategies (no cache for Git/Docker/AI, 30-min cache for downloads)
- HTTP Range request support for partial downloads
- Container registry authentication proxy
- Retry logic with exponential backoff
- Comprehensive security headers (HSTS, CSP, X-Frame-Options)
- Runs on Cloudflare Workers (edge computing platform)
- Single-file architecture: all core logic in `src/index.js` (~1500 lines)
- Configuration-driven platform support via `src/config/platforms.js`
- No external dependencies in production (uses only Web APIs and Cloudflare Workers APIs)
- Supports HTTP/3, intelligent caching, retry logic, and performance monitoring
## Commands
## Development Commands
### Development
### Local Development
```bash
# Start local development server with Wrangler
npm run dev
# Type check without emitting files
npm run type-check
npm run dev # Start local development server with Wrangler
npm start # Alias for npm run dev
```
### Testing
```bash
# Run all tests in watch mode
npm test
npm test # Run all tests with Vitest (watch mode)
npm run test:run # Run tests once without watch mode
npm run test:coverage # Generate coverage report (requires 80% threshold)
npm run test:ui # Launch Vitest UI
npm run test:watch # Explicitly run in watch mode
```
# Run tests once (CI mode)
npm run test:run
**Run a single test file:**
# Run tests with UI
npm run test:ui
```bash
npx vitest run test/platforms.test.js
```
# Generate coverage report (requires 80% coverage)
npm run test:coverage
**Run tests matching a pattern:**
# Watch mode
npm run test:watch
```bash
npm test -- --grep "GitHub"
```
### Code Quality
```bash
# Run ESLint
npm run lint
# Auto-fix ESLint issues
npm run lint:fix
# Format code with Prettier
npm run format
# Check formatting without modifying files
npm run format:check
npm run lint # Run ESLint on src/ and test/
npm run lint:fix # Auto-fix ESLint issues
npm run format # Format code with Prettier
npm run format:check # Check formatting without modifying
npm run type-check # Run TypeScript type checking (no emit)
```
### Deployment
```bash
# Deploy to Cloudflare Workers
npm run deploy
# or
wrangler deploy
# Start production preview
npm start
npm run deploy # Deploy to Cloudflare Workers
```
### Docker Deployment
```bash
# Build and run with Docker
docker build -t xget .
docker run -p 8080:8080 xget
# Or with Podman
podman build -t xget .
podman run -p 8080:8080 xget
```
**Note:** Deployment requires Cloudflare account and proper wrangler authentication (`wrangler login`).
## Architecture
### Core Components
### Core Request Flow
**`src/index.js`** (1432 lines) - Main request handler
1. **Request Reception** (`src/index.js` fetch handler)
- Validates HTTP method (GET/HEAD by default, POST allowed for Git operations)
- Enforces path length limits (max 2048 characters)
- Creates `PerformanceMonitor` instance for timing metrics
- `handleRequest()` - Core request processor with caching, retry logic, security validation
- `PerformanceMonitor` - Tracks timing metrics throughout request lifecycle
- Protocol detection: Git (git-upload-pack), Git LFS, Docker/OCI (v2 API), AI inference providers
- URL transformation and platform routing
- Docker registry authentication proxy (token fetch, WWW-Authenticate handling)
- Response rewriting for protocol compliance
2. **Platform Detection & URL Transformation**
- Extracts platform prefix from URL path (e.g., `/gh/user/repo` → platform: `gh`)
- Uses `transformPath()` from `src/config/platforms.js` to convert Xget URL to upstream URL
- Platform config in `PLATFORMS` object maps prefixes to base URLs
**`src/config/index.js`** (180 lines) - Configuration management
3. **Protocol Detection**
- **Git detection** (`isGitRequest`): checks for `/info/refs`, `/git-upload-pack`, `/git-receive-pack`, Git User-Agent
- **Docker detection** (`isDockerRequest`): checks for `/v2/` paths, Docker User-Agent, OCI manifest headers
- **Git LFS detection** (`isGitLFSRequest`): checks for `.git/info/lfs`, `application/vnd.git-lfs+json`
- **AI inference detection** (`isAIInferenceRequest`): checks for `ip-` platform prefix
- `createConfig(env)` - Creates config with environment variable overrides
- `CONFIG` - Default config object
- Settings: timeout (30s), retries (3), cache duration (1800s), security rules
4. **Caching Layer**
- Cache is **skipped** for Git, Git LFS, Docker, and AI inference requests
- Uses Cloudflare Cache API for HTTP GET/HEAD requests
- Default TTL: 1800 seconds (30 minutes)
- Supports HTTP Range requests with intelligent cache lookup
**`src/config/platforms.js`** (395 lines) - Platform definitions
5. **Upstream Fetch with Retry**
- Retry logic: max 3 attempts with linear backoff (1000ms × retry count)
- Timeout: 30 seconds per request
- Request headers are proxied (User-Agent, Authorization, Range, etc.)
- Special handling for Git (`service=git-upload-pack` query params) and Docker (authentication flow)
- `PLATFORMS` - Maps 40+ platform prefixes to base URLs
- Code repos: `gh`, `gl`, `gitea`, `codeberg`, `sf`, `aosp`, `hf`, `civitai`
- Package managers: `npm`, `pypi`, `conda`, `maven`, `gradle`, `rubygems`, `cran`, `cpan`, `golang`, `nuget`, `crates`, `packagist`
- Linux distros: `debian`, `ubuntu`, `fedora`, `rocky`, `opensuse`, `arch`
- AI providers: `ip-openai`, `ip-anthropic`, `ip-gemini`, etc. (prefix: `ip-`)
- Container registries: `cr-docker`, `cr-ghcr`, `cr-gcr`, etc. (prefix: `cr-`)
- `transformPath()` - Converts prefixed URLs to actual platform URLs
6. **Response Processing**
- Adds security headers (HSTS, CSP, X-Frame-Options, etc.)
- Adds performance metrics via `X-Performance-Metrics` header
- Caches successful responses (200-299 status codes)
- Handles CORS for allowed origins
### Request Flow
### Platform Configuration System
```
1. Receive request → Validate security (method, path length, origin)
2. Detect protocol → Git/Git LFS/Docker/AI/Standard
3. Parse URL → Extract platform prefix and path
4. Transform → Convert to actual platform URL via PLATFORMS config
5. Cache check → Skip for Git/Docker/AI protocols
6. Fetch upstream → With timeout, retries, exponential backoff
7. Handle auth → Docker token proxy for container registries
8. Rewrite response → Docker Content-Digest, URL rewriting
9. Add headers → Security headers, performance metrics, CORS
10. Cache → Store successful responses (protocol-dependent)
11. Return → Final response with all headers
```
**File:** `src/config/platforms.js`
### Caching Strategy
This file contains:
- **No cache:** Git operations, Git LFS, Docker/OCI registry, AI inference APIs
- **30-min cache:** Standard file downloads, package registry files
- **Range requests:** Cache full content when partial range requested
- Cache key: Full request URL including query parameters
- `PLATFORMS` object: maps platform prefixes to base URLs (e.g., `gh: 'https://github.com'`)
- `transformPath(effectivePath, platform)` function: converts Xget paths to upstream paths
- Handles special cases like container registries (`cr-*` platforms)
- Strips platform prefix and reconstructs correct upstream URL structure
### Testing Structure
**Adding a new platform:**
- `test/index.test.js` - Core handler tests
- `test/platforms.test.js` - Platform URL transformation tests
1. Add entry to `PLATFORMS` object with prefix and base URL
2. If URL transformation needs special logic, add case in `transformPath()`
3. Add tests in `test/platforms.test.js`
4. Update documentation (README.md)
### Configuration & Environment
**File:** `src/config/index.js`
The `createConfig(env)` function creates runtime configuration from environment variables:
- `TIMEOUT_SECONDS` (default: 30)
- `MAX_RETRIES` (default: 3)
- `RETRY_DELAY_MS` (default: 1000)
- `CACHE_DURATION` (default: 1800)
- `ALLOWED_METHODS` (default: 'GET,HEAD')
- `ALLOWED_ORIGINS` (default: '*')
- `MAX_PATH_LENGTH` (default: 2048)
Environment variables are set via Cloudflare Workers dashboard or `wrangler.toml`.
## Testing Strategy
**Framework:** Vitest with `@cloudflare/vitest-pool-workers` (simulates Cloudflare Workers environment)
**Test Organization:**
- `test/index.test.js` - Core request handling logic
- `test/platforms.test.js` - Platform URL transformation logic
- `test/integration.test.js` - End-to-end integration tests
- `test/security.test.js` - Security validation tests
- `test/performance.test.js` - Performance benchmarking
- `test/container-registry.test.js` - Docker/OCI registry tests
- `test/range-cache.test.js` - HTTP Range request tests
- `test/setup.js` - Global test configuration
- `test/helpers/` - Test utilities
- `test/fixtures/` - Test data
- `test/security.test.js` - Security validation (headers, input validation)
- `test/performance.test.js` - Performance monitoring tests
- `test/range-cache.test.js` - HTTP Range request caching
- `test/container-registry.test.js` - Docker/OCI registry support
- `test/[platform].test.js` - Platform-specific tests (npm, homebrew, jenkins, etc.)
- `test/helpers/test-utils.js` - Shared test utilities and mock helpers
**Test configuration:**
**Coverage requirements:** 80% minimum (branches, functions, lines, statements)
- Framework: Vitest with Cloudflare Workers pool (`@cloudflare/vitest-pool-workers`)
- Coverage: 80% minimum (branches, functions, lines, statements)
- Timeout: 30 seconds per test
- Retries: 2 attempts
**Important testing notes:**
## Code Style
- Tests run in a simulated Cloudflare Workers environment (not Node.js)
- Use `SELF.fetch()` to test the worker (provided by vitest-pool-workers)
- Mock external fetch calls using `vi.spyOn(globalThis, 'fetch')`
- Test fixtures in `test/fixtures/` for sample responses
**ESLint configuration (eslint.config.js):**
## Code Style & Conventions
**Enforced by ESLint + Prettier:**
- 2-space indentation
- Single quotes for strings
- Semicolons required
- Camelcase naming (except properties)
- No trailing spaces
- Unix line endings
- No `var`, use `const` or `let`
- Prefer arrow functions and template literals
- camelCase for variables/functions
- UPPER_SNAKE_CASE for constants
- PascalCase for classes
- Use JSDoc comments for all exported functions and classes
**Naming conventions:**
**Example JSDoc:**
- Variables/functions: `camelCase`
- Constants: `UPPER_SNAKE_CASE`
- Classes: `PascalCase`
- Files: `kebab-case` (though this repo uses `.js` files)
```javascript
/**
* Transforms incoming request path to upstream platform URL.
*
* @param {string} effectivePath - The path from the Xget URL
* @param {string} platform - Platform prefix (e.g., 'gh', 'npm')
* @returns {string} Transformed path for upstream request
*
* @example
* transformPath('/gh/user/repo/file.txt', 'gh')
* // Returns: '/user/repo/file.txt'
*/
```
**JSDoc comments required:**
## Common Development Tasks
- All functions must have JSDoc with `@param`, `@returns`, `@throws`
- Include `@example` blocks for non-trivial functions
- Extensive inline comments for complex logic
### Adding Support for a New Platform
## Environment Variables
Configure via Cloudflare Workers environment or `.dev.vars`:
- `TIMEOUT_SECONDS` - Request timeout (default: 30)
- `MAX_RETRIES` - Max retry attempts (default: 3)
- `RETRY_DELAY_MS` - Retry delay (default: 1000)
- `CACHE_DURATION` - Cache TTL in seconds (default: 1800)
- `ALLOWED_METHODS` - Comma-separated HTTP methods (default: 'GET,HEAD')
- `ALLOWED_ORIGINS` - Comma-separated CORS origins (default: '*')
- `MAX_PATH_LENGTH` - Max URL path length (default: 2048)
## Platform Support
### Adding New Platforms
1. Edit `src/config/platforms.js`:
1. Add platform to `src/config/platforms.js`:
```javascript
export const PLATFORMS = {
// ... existing platforms
'new-prefix': 'https://platform-url.com'
'newplatform': 'https://newplatform.com'
};
```
2. Add URL transformation logic in `transformPath()` if needed (for special path handling)
2. If special URL transformation needed, update `transformPath()` in `src/config/platforms.js`
3. Add tests in `test/platforms.test.js`:
3. Add tests in `test/platforms.test.js` or create `test/newplatform.test.js`
```javascript
it('should transform new-prefix URLs', () => {
const result = transformPath('new-prefix/path/to/resource');
expect(result).toBe('https://platform-url.com/path/to/resource');
});
```
4. Update README.md with platform prefix and examples
4. Update README.md with platform badge and usage instructions
### Debugging Cloudflare Workers Locally
### Platform Categories
- Wrangler provides local development server with hot reload
- Use `console.log()` - output appears in terminal
- Performance metrics available via `X-Performance-Metrics` response header
- Use `npm run dev` and test with `curl` or browser
- **Code repositories:** Direct path passthrough
- **Package managers:** May require special path handling (e.g., npm scoped packages)
- **Container registries (cr- prefix):** Docker/OCI protocol, authentication proxy
- **AI providers (ip- prefix):** No caching, streaming support
### Working with Docker/Container Registries
Docker registry protocol requires:
- Authentication flow handling (`/v2/auth` endpoint)
- WWW-Authenticate header parsing
- Token fetching and proxying
- Special MIME types (`application/vnd.docker.distribution.manifest.v2+json`)
See `isDockerRequest()` and Docker-specific logic in `src/index.js:800-850`.
### Handling Git Operations
Git protocol detection checks:
- URL paths: `/info/refs`, `/git-upload-pack`, `/git-receive-pack`
- Query params: `service=git-upload-pack`
- User-Agent: `git/` prefix
Git requests **bypass cache** to ensure real-time data.
## Security Considerations
- All GPLv3 licensed - include license header in new files
- Validate HTTP methods (allow POST for Git operations)
- Path length limits (2048 chars default)
- Security headers applied to all responses
- No XSS or injection vulnerabilities - all URLs validated
- Docker auth tokens proxied securely (never stored)
- CORS headers configurable per deployment
**Never disable security headers** - they protect against XSS, clickjacking, and MITM attacks.
## Deployment Targets
**Input validation:**
1. **Cloudflare Workers** (primary): Zero-config deployment with global edge network
2. **Docker/Podman**: Self-hosted with workerd runtime (see Dockerfile)
3. **Local development**: Wrangler dev server on localhost
- Always check path length before processing
- Validate platform prefix exists in `PLATFORMS` config
- Sanitize user input to prevent path traversal
## License
**Request method restrictions:**
GPLv3 - All new source files must include the GPL header (see existing files for template).
- Default: GET/HEAD only
- Git operations: also allow POST
- Enforce via `ALLOWED_METHODS` configuration
**CORS policy:**
- Default allows all origins (`*`)
- Can be restricted via `ALLOWED_ORIGINS` environment variable
- Always set appropriate CORS headers
## Deployment Options
**Cloudflare Workers (Primary):**
- Run `npm run deploy` after `wrangler login`
- Configure environment variables in Cloudflare dashboard
- Uses `wrangler.toml` for worker configuration
**Self-hosted (Docker):**
- Multi-stage Dockerfile builds worker with Wrangler
- Runs with `workerd` runtime (Cloudflare's open-source Workers runtime)
- Exposes port 8080
- Uses `config.capnp` for workerd configuration
**EdgeOne Pages (Alternative edge platform):**
- Similar to Cloudflare Workers
- See README.md for deployment instructions
## Performance Optimization
**Key performance features:**
- HTTP/3 support (via Cloudflare)
- Brotli/gzip compression
- Connection keep-alive and pre-warming
- Edge caching with 30-minute default TTL
- Parallel retry logic with linear backoff
- Performance monitoring built-in (`PerformanceMonitor` class)
**Monitoring metrics:**
- `cache_hit` - Successful cache retrieval
- `attempt_N` - Nth retry attempt timestamp
- `success` - Successful upstream fetch
- Response includes `X-Performance-Metrics` with timing data
## Troubleshooting
**Tests failing with "fetch is not defined":**
- Ensure using `@cloudflare/vitest-pool-workers` pool
- Check `vitest.config.js` has correct pool configuration
**Wrangler deployment fails:**
- Run `wrangler login` first
- Check `wrangler.toml` configuration
- Verify Cloudflare account has Workers enabled
**Cache not working:**
- Ensure not testing Git/Docker/AI requests (cache bypassed)
- Check Cloudflare Workers cache API is available (not available in `wrangler dev`)
- Verify HTTP status is 2xx
**Platform URL transformation incorrect:**
- Check `transformPath()` logic in `src/config/platforms.js`
- Add console.log to debug URL construction
- Verify platform prefix matches `PLATFORMS` key