fix(skills): harden URL conversion

This commit is contained in:
xixu-me committed 2026-03-15 12:16:31 +08:00
1 parent e8eba9645a
commit ff3e61ae22
4 files changed
+429 -48

No files matched your search

+10 -5
View File
@@ -29,8 +29,8 @@ self-hosted option exists.
- `XGET_BASE_URL` from the environment
- if neither exists and the task is not just writing docs, ask the user for
their self-hosted Xget domain and configure `XGET_BASE_URL`
- `https://xget.example.com` only for docs or templates when a real domain
is not available yet
- `https://xget.example.com` only for docs or templates when a real domain is
not available yet
- `https://xget.xi-xu.me` only as an explicitly labeled fallback after the
user declines or cannot provide a self-hosted domain
2. Keep platform data fresh. Do not hardcode the full prefix list from memory.
@@ -51,7 +51,8 @@ node scripts/xget.mjs convert --base-url https://xget.example.com --url https://
1. Identify the user's goal:
- convert one or more upstream URLs
- generate config snippets for npm, pip, Go, NuGet, Cargo, Docker, or AI SDKs
- generate config snippets for npm, pip, Go, NuGet, Docker, or AI SDKs
- explain the current Cargo limitation for registry source replacement
- explain which Xget prefix to use
- propose or document a self-hosted deployment
2. Refresh the live platform map with `scripts/xget.mjs` if the answer depends
@@ -62,6 +63,7 @@ node scripts/xget.mjs convert --base-url https://xget.example.com --url https://
5. Before finishing, sanity-check that every example uses the right Xget path
shape:
- repo/content: `/{prefix}/...`
- crates.io HTTP URLs: `/crates/...` rather than `/crates/api/v1/crates/...`
- inference APIs: `/ip/{provider}/...`
- OCI registries: `/cr/{registry}/...`
@@ -94,7 +96,10 @@ node scripts/xget.mjs platforms --format table
[references/REFERENCE.md](references/REFERENCE.md).
- If an upstream URL does not match any known platform, do not invent a prefix.
Report that no current Xget mapping was found.
- When writing pip config for HTTPS domains, keep `trusted-host` aligned with
the actual host only if the user really needs it.
- The default pip snippet should omit `trusted-host`; add it only when the
deployment really needs it and keep it aligned with the actual host.
- The `cargo` preset is informational for now. Xget can rewrite direct
`crates.io` HTTP URLs under `/crates/...`, but this skill should not emit
Cargo source replacement config until Xget exposes a registry index endpoint.
- When generating docs or templates without a real domain, prefer
`https://xget.example.com` over the public demo.
+20 -8
View File
@@ -9,7 +9,9 @@ Use these defaults in order:
3. `https://xget.example.com` only for templates and docs
4. `https://xget.xi-xu.me` only as a clearly labeled public-demo fallback
The Xget README explicitly labels `xget.xi-xu.me` as a pre-deployed instance with no reliability guarantee, while the self-hosting docs and DigitalOcean guide show recommended self-hosted domains such as `xget.example.com`.
The Xget README explicitly labels `xget.xi-xu.me` as a pre-deployed instance
with no reliability guarantee, while the self-hosting docs and DigitalOcean
guide show recommended self-hosted domains such as `xget.example.com`.
## Live platform source
@@ -44,7 +46,11 @@ The script derives these path shapes from platform keys:
- pip simple index: `https://{base}/pypi/simple/`
- Go proxy: `https://{base}/golang`
- NuGet v3 index: `https://{base}/nuget/v3/index.json`
- Cargo registry: `https://{base}/crates/`
- Direct crates.io HTTP URLs: `https://{base}/crates/...`
Cargo registry source replacement is not currently emitted by this skill. Xget's
`/crates/...` route rewrites direct crates.io HTTP API and download URLs, but it
does not expose a Cargo registry index endpoint.
### Container registries
@@ -76,7 +82,7 @@ Representative presets:
- `pip`
- `go`
- `nuget`
- `cargo`
- `cargo` (returns the current limitation instead of source replacement config)
- `docker-ghcr`
- `openai`
- `anthropic`
@@ -98,7 +104,7 @@ services:
image: ghcr.io/xixu-me/xget:latest
container_name: xget
ports:
- "127.0.0.1:8080:8080"
- '127.0.0.1:8080:8080'
restart: unless-stopped
```
@@ -109,8 +115,14 @@ Representative reverse-proxy outcome:
## Troubleshooting heuristics
- `404` on converted URLs often means the wrong prefix or an unmatched upstream platform.
- pip issues often come from mixing the right `index-url` with the wrong host in `trusted-host`.
- `404` on converted URLs often means the wrong prefix or an unmatched upstream
platform.
- crates.io conversions should strip the upstream `/api/v1/crates` prefix before
adding `/crates/...`.
- pip issues often come from adding `trusted-host` unnecessarily or pointing it
at the wrong host.
- Docker examples must use `/cr/{registry}` prefixes, not plain `/{prefix}`.
- AI SDK examples usually need the Xget base URL changed but keep the original API key behavior.
- If the user asks for the “latest” supported platform, refresh the live platform map before answering.
- AI SDK examples usually need the Xget base URL changed but keep the original
API key behavior.
- If the user asks for the “latest” supported platform, refresh the live
platform map before answering.
+312 -35
View File
@@ -2,12 +2,51 @@
import { get } from 'node:https';
import process from 'node:process';
import { pathToFileURL } from 'node:url';
import vm from 'node:vm';
const DEFAULT_SOURCE_URL = 'https://raw.gitcode.com/xixu-me/xget/raw/main/src/config/platforms.js';
const DEFAULT_BASE_PLACEHOLDER = 'https://xget.example.com';
const CRATES_API_PREFIX = '/api/v1/crates';
/**
* @typedef {'resource' | 'registry' | 'inference'} PlatformCategory
*/
/**
* @typedef {{ key: string, upstream: string, pathPrefix: string, category: PlatformCategory }} PlatformEntry
*/
/**
* @typedef {{
* help?: boolean,
* format?: string,
* url?: string,
* preset?: string,
* 'source-url'?: string,
* 'base-url'?: string,
* [key: string]: string | boolean | undefined
* }} CliOptions
*/
/**
* @typedef {{ command: string, options: CliOptions }} ParsedArgs
*/
/**
* @typedef {{
* preset: string,
* summary: string,
* commands?: string[],
* env?: Record<string, string>,
* files?: Record<string, string>,
* notes?: string[],
* supported?: boolean
* }} Snippet
*/
function printHelp() {
console.log(`Usage: node scripts/xget.mjs <command> [options]
@@ -33,7 +72,9 @@ convert options:
snippet options:
--base-url URL Xget base URL. Defaults to XGET_BASE_URL.
--preset NAME One of: npm, pip, go, nuget, cargo, docker-ghcr,
openai, anthropic, gemini.
openai, anthropic, gemini. The cargo preset
explains the current limitation instead of
emitting source replacement config.
--format json|text
Examples:
@@ -43,12 +84,17 @@ Examples:
`);
}
/**
* @param {string[]} argv
* @returns {ParsedArgs}
*/
function parseArgs(argv) {
const [command = 'help', ...rest] = argv;
if (command === '--help') {
return { command: 'help', options: { help: true } };
}
/** @type {CliOptions} */
const options = {};
for (let index = 0; index < rest.length; index += 1) {
@@ -75,11 +121,28 @@ function parseArgs(argv) {
return { command, options };
}
/**
* @param {unknown} error
* @returns {string}
*/
function getErrorMessage(error) {
return error instanceof Error ? error.message : String(error);
}
/**
* @param {string} message
* @param {number} [code]
* @returns {never}
*/
function fail(message, code = 1) {
console.error(`Error: ${message}`);
process.exit(code);
}
/**
* @param {string} url
* @returns {Promise<string>}
*/
function httpGet(url) {
return new Promise((resolve, reject) => {
get(url, response => {
@@ -109,7 +172,11 @@ function httpGet(url) {
});
}
function extractPlatformsModule(jsSource) {
/**
* @param {string} jsSource
* @returns {Record<string, string>}
*/
export function extractPlatformsModule(jsSource) {
const match = jsSource.match(/export const PLATFORMS = (\{[\s\S]*?\n\});/);
if (!match) {
@@ -119,13 +186,15 @@ function extractPlatformsModule(jsSource) {
try {
return vm.runInNewContext(`(${match[1]})`);
} catch (error) {
fail(`Could not parse remote PLATFORMS object: ${error.message}`);
fail(`Could not parse remote PLATFORMS object: ${getErrorMessage(error)}`);
}
}
async function loadPlatforms(sourceUrl) {
const jsSource = await httpGet(sourceUrl);
const platforms = extractPlatformsModule(jsSource);
/**
* @param {Record<string, string>} platforms
* @returns {PlatformEntry[]}
*/
export function createPlatformEntries(platforms) {
return Object.entries(platforms)
.sort(([left], [right]) => left.localeCompare(right))
.map(([key, upstream]) => ({
@@ -140,8 +209,30 @@ async function loadPlatforms(sourceUrl) {
}));
}
/**
* @param {string} jsSource
* @returns {PlatformEntry[]}
*/
export function loadPlatformsFromSource(jsSource) {
const platforms = extractPlatformsModule(jsSource);
return createPlatformEntries(platforms);
}
/**
* @param {string} sourceUrl
* @returns {Promise<PlatformEntry[]>}
*/
async function loadPlatforms(sourceUrl) {
const jsSource = await httpGet(sourceUrl);
return loadPlatformsFromSource(jsSource);
}
/**
* @param {string | undefined} value
* @returns {string | null}
*/
function normalizeBaseUrl(value) {
if (!value) {
if (typeof value !== 'string' || !value) {
return null;
}
@@ -156,6 +247,11 @@ function normalizeBaseUrl(value) {
}
}
/**
* @param {string} value
* @param {string} flagName
* @returns {URL}
*/
function normalizeAbsoluteUrl(value, flagName) {
try {
return new URL(value);
@@ -164,19 +260,168 @@ function normalizeAbsoluteUrl(value, flagName) {
}
}
function findPlatformForUrl(platforms, originUrl) {
const origin = originUrl.origin;
return platforms.find(({ upstream }) => upstream === origin) ?? null;
/**
* @param {string} pathname
* @returns {string}
*/
function normalizePathname(pathname) {
if (!pathname || pathname === '/') {
return '';
}
return pathname.replace(/\/+$/, '');
}
function buildConvertedUrl(baseUrl, platform, originUrl) {
const suffix = originUrl.pathname + originUrl.search + originUrl.hash;
/**
* @param {string} pathname
* @param {string} prefix
* @param {boolean} [caseInsensitive]
* @returns {boolean}
*/
function matchesPathPrefix(pathname, prefix, caseInsensitive = false) {
const normalizedPath = normalizePathname(pathname);
const normalizedPrefix = normalizePathname(prefix);
if (!normalizedPrefix) {
return true;
}
if (!normalizedPath) {
return false;
}
if (caseInsensitive) {
const lowerPath = normalizedPath.toLowerCase();
const lowerPrefix = normalizedPrefix.toLowerCase();
return lowerPath === lowerPrefix || lowerPath.startsWith(`${lowerPrefix}/`);
}
return normalizedPath === normalizedPrefix || normalizedPath.startsWith(`${normalizedPrefix}/`);
}
/**
* @param {string} pathname
* @param {string} prefix
* @param {boolean} [caseInsensitive]
* @returns {string}
*/
function stripPathPrefix(pathname, prefix, caseInsensitive = false) {
const normalizedPrefix = normalizePathname(prefix);
if (!normalizedPrefix) {
return pathname;
}
const flags = caseInsensitive ? 'i' : '';
const escapedPrefix = normalizedPrefix.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
return pathname.replace(new RegExp(`^${escapedPrefix}(?=/|$)`, flags), '');
}
/**
* @param {PlatformEntry[]} platforms
* @param {string} key
* @returns {PlatformEntry | null}
*/
function findPlatformByKey(platforms, key) {
return platforms.find(platform => platform.key === key) ?? null;
}
/**
* @param {PlatformEntry[]} platforms
* @param {URL} originUrl
* @returns {PlatformEntry | null}
*/
function findSpecialPlatformForUrl(platforms, originUrl) {
if (originUrl.hostname === 'ghcr.io') {
if (originUrl.pathname.startsWith('/v2/homebrew/')) {
return findPlatformByKey(platforms, 'homebrew-bottles');
}
return findPlatformByKey(platforms, 'cr-ghcr');
}
return null;
}
/**
* @param {PlatformEntry[]} platforms
* @param {URL} originUrl
* @returns {PlatformEntry | null}
*/
export function findPlatformForUrl(platforms, originUrl) {
const specialPlatform = findSpecialPlatformForUrl(platforms, originUrl);
if (specialPlatform) {
return specialPlatform;
}
const matchingPlatforms = platforms
.filter(platform => {
const upstreamUrl = new URL(platform.upstream);
if (upstreamUrl.origin !== originUrl.origin) {
return false;
}
const caseInsensitive = platform.key === 'homebrew' || platform.key === 'homebrew-api';
return matchesPathPrefix(originUrl.pathname, upstreamUrl.pathname, caseInsensitive);
})
.sort((left, right) => {
const leftPathLength = normalizePathname(new URL(left.upstream).pathname).length;
const rightPathLength = normalizePathname(new URL(right.upstream).pathname).length;
return rightPathLength - leftPathLength;
});
return matchingPlatforms[0] ?? null;
}
/**
* @param {PlatformEntry} platform
* @param {URL} originUrl
* @returns {string}
*/
export function getConvertedSuffix(platform, originUrl) {
let pathname = originUrl.pathname;
if (platform.key === 'homebrew') {
pathname = stripPathPrefix(pathname, '/Homebrew', true);
} else if (platform.key === 'homebrew-api') {
pathname = stripPathPrefix(pathname, '/api', true);
} else if (platform.key === 'crates') {
pathname = stripPathPrefix(pathname, CRATES_API_PREFIX, true);
} else {
const upstreamPath = new URL(platform.upstream).pathname;
pathname = stripPathPrefix(pathname, upstreamPath);
}
if (!pathname) {
pathname = '/';
}
if (!pathname.startsWith('/')) {
pathname = `/${pathname}`;
}
return `${pathname}${originUrl.search}${originUrl.hash}`;
}
/**
* @param {string} baseUrl
* @param {PlatformEntry} platform
* @param {URL} originUrl
* @returns {string}
*/
export function buildConvertedUrl(baseUrl, platform, originUrl) {
const suffix = getConvertedSuffix(platform, originUrl);
return `${baseUrl}${platform.pathPrefix}${suffix.replace(/^\/+/, '')}`;
}
function createSnippet(baseUrl, preset) {
/**
* @param {string} baseUrl
* @param {string} preset
* @returns {Snippet}
*/
export function createSnippet(baseUrl, preset) {
const host = new URL(baseUrl).host;
/** @type {Record<string, Snippet>} */
const snippets = {
npm: {
preset,
@@ -186,10 +431,9 @@ function createSnippet(baseUrl, preset) {
pip: {
preset,
summary: 'Configure pip to use the Xget PyPI simple index.',
commands: [
`pip config set global.index-url ${baseUrl}/pypi/simple/`,
`pip config set global.trusted-host ${host}`,
'pip config list'
commands: [`pip config set global.index-url ${baseUrl}/pypi/simple/`, 'pip config list'],
notes: [
`Only add "pip config set global.trusted-host ${host}" when the deployment really needs it.`
]
},
go: {
@@ -207,16 +451,12 @@ function createSnippet(baseUrl, preset) {
},
cargo: {
preset,
summary: 'Route crates.io traffic through Xget.',
files: {
'~/.cargo/config.toml': [
'[source.crates-io]',
'replace-with = "xget"',
'',
'[source.xget]',
`registry = "${baseUrl}/crates/"`
].join('\n')
}
supported: false,
summary: 'Cargo registry source replacement is not currently supported by Xget.',
notes: [
'Xget can convert direct crates.io HTTP URLs under /crates/..., but it does not expose a Cargo registry index.',
'Do not generate ~/.cargo/config.toml source replacement entries until Xget provides a registry index endpoint.'
]
},
'docker-ghcr': {
preset,
@@ -257,16 +497,29 @@ function createSnippet(baseUrl, preset) {
return result;
}
/**
* @param {unknown} value
* @returns {void}
*/
function renderJson(value) {
console.log(JSON.stringify(value, null, 2));
}
/**
* @param {PlatformEntry[]} rows
* @returns {void}
*/
function renderTable(rows) {
/** @type {Array<keyof PlatformEntry>} */
const headers = ['key', 'category', 'pathPrefix', 'upstream'];
const widths = headers.map(header =>
Math.max(header.length, ...rows.map(row => String(row[header]).length))
);
/**
* @param {Record<string, string>} row
* @returns {string}
*/
const formatRow = row =>
headers.map((header, index) => String(row[header]).padEnd(widths[index])).join(' ');
@@ -275,6 +528,10 @@ function renderTable(rows) {
rows.forEach(row => console.log(formatRow(row)));
}
/**
* @param {Snippet} snippet
* @returns {void}
*/
function renderTextSnippet(snippet) {
console.log(snippet.summary);
@@ -295,6 +552,21 @@ function renderTextSnippet(snippet) {
console.log(content);
});
}
if (snippet.notes) {
console.log('\nNotes:');
snippet.notes.forEach(note => console.log(note));
}
}
/**
* @param {CliOptions} options
* @param {string} key
* @returns {string | undefined}
*/
function getStringOption(options, key) {
const value = options[key];
return typeof value === 'string' ? value : undefined;
}
async function main() {
@@ -305,8 +577,8 @@ async function main() {
return;
}
const sourceUrl = options['source-url'] ?? DEFAULT_SOURCE_URL;
const format = options.format ?? 'json';
const sourceUrl = getStringOption(options, 'source-url') ?? DEFAULT_SOURCE_URL;
const format = getStringOption(options, 'format') ?? 'json';
if (command === 'platforms') {
const platforms = await loadPlatforms(sourceUrl);
@@ -329,10 +601,10 @@ async function main() {
if (command === 'convert') {
const baseUrl =
normalizeBaseUrl(options['base-url'] ?? process.env.XGET_BASE_URL) ??
normalizeBaseUrl(getStringOption(options, 'base-url') ?? process.env.XGET_BASE_URL) ??
fail(`Missing --base-url and XGET_BASE_URL. For docs, use ${DEFAULT_BASE_PLACEHOLDER}.`, 2);
const rawUrl = options.url;
const rawUrl = getStringOption(options, 'url');
if (!rawUrl) {
fail('Missing --url for convert.', 2);
}
@@ -345,12 +617,13 @@ async function main() {
fail(`No current Xget platform matched upstream origin ${originUrl.origin}.`, 3);
}
const convertedUrl = buildConvertedUrl(baseUrl, platform, originUrl);
const payload = {
sourceUrl,
baseUrl,
upstreamUrl: originUrl.toString(),
matchedPlatform: platform,
convertedUrl: buildConvertedUrl(baseUrl, platform, originUrl)
convertedUrl
};
if (format === 'json') {
@@ -368,10 +641,10 @@ async function main() {
if (command === 'snippet') {
const baseUrl =
normalizeBaseUrl(options['base-url'] ?? process.env.XGET_BASE_URL) ??
normalizeBaseUrl(getStringOption(options, 'base-url') ?? process.env.XGET_BASE_URL) ??
fail(`Missing --base-url and XGET_BASE_URL. For docs, use ${DEFAULT_BASE_PLACEHOLDER}.`, 2);
const preset = options.preset;
const preset = getStringOption(options, 'preset');
if (!preset) {
fail('Missing --preset for snippet.', 2);
}
@@ -393,4 +666,8 @@ async function main() {
fail(`Unknown command "${command}". Use --help for supported commands.`, 2);
}
main().catch(error => fail(error.message));
const entryHref = process.argv[1] ? pathToFileURL(process.argv[1]).href : null;
if (entryHref === import.meta.url) {
main().catch(error => fail(getErrorMessage(error)));
}
+87
View File
@@ -0,0 +1,87 @@
import { describe, expect, it } from 'vitest';
import { PLATFORMS } from '../../src/config/platforms.js';
import {
buildConvertedUrl,
createPlatformEntries,
createSnippet,
findPlatformForUrl
} from '../../skills/xget/scripts/xget.mjs';
const BASE_URL = 'https://xget.example.com';
const platforms = createPlatformEntries(PLATFORMS);
/**
* Convert an upstream URL with the xget skill helpers.
* @param {string} url
* @returns {{ platform: { key: string } | null, convertedUrl: string | null }} Converted result.
*/
function convert(url) {
const upstreamUrl = new URL(url);
const platform = findPlatformForUrl(platforms, upstreamUrl);
return {
platform,
convertedUrl: platform ? buildConvertedUrl(BASE_URL, platform, upstreamUrl) : null
};
}
describe('xget skill helpers', () => {
describe('URL conversion', () => {
it('converts Homebrew repository URLs to the homebrew prefix', () => {
const result = convert('https://github.com/Homebrew/homebrew-core/raw/HEAD/Formula/g/git.rb');
expect(result.platform?.key).toBe('homebrew');
expect(result.convertedUrl).toBe(
'https://xget.example.com/homebrew/homebrew-core/raw/HEAD/Formula/g/git.rb'
);
});
it('converts Homebrew API URLs to the homebrew/api prefix', () => {
const result = convert('https://formulae.brew.sh/api/formula/git.json');
expect(result.platform?.key).toBe('homebrew-api');
expect(result.convertedUrl).toBe('https://xget.example.com/homebrew/api/formula/git.json');
});
it('routes Homebrew bottle URLs to homebrew/bottles instead of cr/ghcr', () => {
const result = convert('https://ghcr.io/v2/homebrew/core/git/manifests/2.39.0');
expect(result.platform?.key).toBe('homebrew-bottles');
expect(result.convertedUrl).toBe(
'https://xget.example.com/homebrew/bottles/v2/homebrew/core/git/manifests/2.39.0'
);
});
it('normalizes crates.io API URLs to the documented /crates path shape', () => {
const result = convert('https://crates.io/api/v1/crates/serde/1.0.0/download');
expect(result.platform?.key).toBe('crates');
expect(result.convertedUrl).toBe('https://xget.example.com/crates/serde/1.0.0/download');
});
});
describe('snippets', () => {
it('omits trusted-host from the default pip preset', () => {
const snippet = createSnippet(BASE_URL, 'pip');
expect(snippet.commands).toEqual([
'pip config set global.index-url https://xget.example.com/pypi/simple/',
'pip config list'
]);
expect(snippet.notes).toContain(
'Only add "pip config set global.trusted-host xget.example.com" when the deployment really needs it.'
);
});
it('returns an explicit unsupported notice for cargo instead of a broken config file', () => {
const snippet = createSnippet(BASE_URL, 'cargo');
expect(snippet.supported).toBe(false);
expect(snippet.files).toBeUndefined();
expect(snippet.notes).toContain(
'Do not generate ~/.cargo/config.toml source replacement entries until Xget provides a registry index endpoint.'
);
});
});
});