162 lines
6.2 KiB
Markdown
162 lines
6.2 KiB
Markdown
# Xread
|
|
|
|
**_[汉语](./README.zh.md)_**
|
|
|
|
**Your LLMs deserve better input.**
|
|
|
|
Turn messy webpages into clean Markdown, run web search for LLMs, and self-host the whole stack.
|
|
|
|
Xread is an open-source, self-hosted web ingestion service for LLM workflows. It gives you three main surfaces:
|
|
|
|
- `crawl`: fetch a URL and turn it into readable Markdown
|
|
- `search`: run a search and return text that is easy to paste into a prompt
|
|
- `serp`: run a search and return structured JSON
|
|
|
|
It is a standalone fork of the public [Jina AI Reader](https://jina.ai/reader) repository in [`jina-ai/reader`](https://github.com/jina-ai/reader), adapted for self-hosted use without the original internal `thinapps-shared` dependency.
|
|
|
|
## What This Repository Adds
|
|
|
|
- Self-hosted packaging around the upstream Reader repository
|
|
- Local-first storage and cache persistence
|
|
- OCI image published to GHCR
|
|
- Ready-to-run reverse-proxy deployment assets with the same image, Compose stack, and environment model across Docker, Podman, and `nerdctl`
|
|
- CI, CodeQL, dependency governance, and container publishing already wired up
|
|
|
|
## Quick Look
|
|
|
|
These examples use the published image directly on its HTTP/1 port (`8081`). The deployment guide later covers the Compose backend loopback ports (`3001`, `3101`, `3201`) and the public reverse-proxy hostnames.
|
|
|
|
### Crawl a page
|
|
|
|
```bash
|
|
curl "http://127.0.0.1:8081/http://example.com"
|
|
```
|
|
|
|
### Search the web as text
|
|
|
|
```bash
|
|
curl "http://127.0.0.1:8081/search?q=example%20domain&num=5&provider=google"
|
|
```
|
|
|
|
### Search the web as JSON
|
|
|
|
```bash
|
|
curl "http://127.0.0.1:8081/?q=example%20domain&num=5&provider=google"
|
|
```
|
|
|
|
## Quick Start
|
|
|
|
### Run locally
|
|
|
|
```bash
|
|
npm ci
|
|
npm run build
|
|
npm start
|
|
```
|
|
|
|
That starts the `crawl` server. The standalone services are:
|
|
|
|
- `build/stand-alone/crawl.js`
|
|
- `build/stand-alone/search.js`
|
|
- `build/stand-alone/serp.js`
|
|
|
|
### Run with a container engine
|
|
|
|
```bash
|
|
docker run --rm -p 8081:8081 ghcr.io/xixu-me/xread:latest
|
|
```
|
|
|
|
The published image defaults to `crawl`. Any OCI-compatible engine can run it. The examples below use Docker syntax. To run the other entrypoints:
|
|
|
|
```bash
|
|
docker run --rm -p 8081:8081 --entrypoint node ghcr.io/xixu-me/xread:latest build/stand-alone/search.js
|
|
docker run --rm -p 8081:8081 --entrypoint node ghcr.io/xixu-me/xread:latest build/stand-alone/serp.js
|
|
```
|
|
|
|
## Built for Self-Hosting
|
|
|
|
- Chrome is bundled in the image for page rendering
|
|
- HTTP/2 cleartext (`h2c`) is supported internally
|
|
- HTTP/1 fallback is available on `PORT + 1`
|
|
- If no paid search provider is configured, `search` and `serp` still work through the standalone fallback path
|
|
|
|
Published image:
|
|
|
|
- `ghcr.io/xixu-me/xread:latest`
|
|
|
|
For production, pin a digest instead of following `latest`.
|
|
|
|
## Deploy It
|
|
|
|
The repository includes a complete reverse-proxy deployment example:
|
|
|
|
- [Deployment guide](./docs/deploy.md)
|
|
- [Container engine guide](./docs/container-engines.md)
|
|
- [Compose stack](./deploy/docker-compose.yml)
|
|
- [`Caddyfile`](./deploy/Caddyfile)
|
|
- [Environment template](./deploy/.env.example)
|
|
|
|
The deployment docs use the same image, Compose stack, and environment variables across Docker, Podman, and `nerdctl`/containerd.
|
|
|
|
The documented deployment pattern is:
|
|
|
|
- `r.your-domain.example` -> `crawl`
|
|
- `s.your-domain.example/search` -> `search`
|
|
- `s.your-domain.example/?q=...` -> `serp`
|
|
|
|
## Configuration
|
|
|
|
The standalone build reads configuration from environment variables.
|
|
|
|
The deployment template in [`deploy/.env.example`](./deploy/.env.example) includes the shared deployment variables. The full runtime surface is:
|
|
|
|
| Variable | Purpose |
|
|
| --------------------------------- | --------------------------------------------------------------------------- |
|
|
| `PORT` | Main service port. HTTP/1 fallback listens on `PORT + 1`. |
|
|
| `PUBLIC_HTTP_PORT` | Public HTTP port exposed by the reverse proxy in the Compose deployment. |
|
|
| `PUBLIC_HTTPS_PORT` | Public HTTPS port exposed by the reverse proxy in the Compose deployment. |
|
|
| `XREAD_LOOPBACK_HOST` | Host IP used for the backend loopback ports in the Compose deployment. |
|
|
| `XREAD_CRAWL_LOOPBACK_PORT` | Host loopback port for the `crawl` backend. |
|
|
| `XREAD_SEARCH_LOOPBACK_PORT` | Host loopback port for the `search` backend. |
|
|
| `XREAD_SERP_LOOPBACK_PORT` | Host loopback port for the `serp` backend. |
|
|
| `STORAGE_ROOT` | Root directory for stored artifacts such as snapshots and generated assets. |
|
|
| `LOCAL_DB_ROOT` | Root directory for local metadata, caches, and lightweight persistence. |
|
|
| `SERPER_SEARCH_API_KEY` | Enables Serper-backed Google/Bing search. |
|
|
| `BRAVE_SEARCH_API_KEY` | Enables Brave Search integration where supported. |
|
|
| `CLOUD_FLARE_API_KEY` | Enables Cloudflare-backed capabilities where configured. |
|
|
| `LOCAL_PROXY_URLS` | Optional proxy pool for outbound requests. |
|
|
| `OVERRIDE_CHROME_EXECUTABLE_PATH` | Optional override for the Chrome executable path. |
|
|
|
|
Useful request controls for crawling include:
|
|
|
|
- `x-no-cache`
|
|
- `x-target-selector`
|
|
- `x-wait-for-selector`
|
|
- `x-respond-with`
|
|
- `x-timeout`
|
|
|
|
## Build and Verify
|
|
|
|
```bash
|
|
npm run lint
|
|
npm run security:audit
|
|
npm run test:ci
|
|
npm run build
|
|
```
|
|
|
|
The image build also dry-runs all three standalone servers.
|
|
|
|
## Repository Layout
|
|
|
|
```text
|
|
src/api/ RPC hosts and public HTTP-facing methods
|
|
src/stand-alone/ standalone server entrypoints
|
|
src/services/ crawling, formatting, search, and runtime services
|
|
src/shared/ local shared primitives for storage, rate limits, and config
|
|
public/ static assets
|
|
deploy/ ready-to-run production deployment templates
|
|
docs/ operational documentation
|
|
scripts/ build, licensing, and security helper scripts
|
|
tests/ regression tests for build, runtime, and automation
|
|
```
|