Initial commit: project documentation
This commit is contained in:
@@ -1,3 +1,236 @@
|
||||
# afakzaid-rag
|
||||
# AfakZaid RAG System
|
||||
|
||||
AfakZaid RAG System
|
||||
Arabic AI library system with hybrid search, Celery task queue, Cloudflare R2 storage, and Typesense full-text search.
|
||||
|
||||
## Stack
|
||||
|
||||
| Service | Purpose | Port (internal) |
|
||||
|---|---|---|
|
||||
| FastAPI | Main API | 8100 |
|
||||
| PostgreSQL 15 | Metadata, users, sessions | 5432 |
|
||||
| Qdrant | Vector embeddings | 6333 |
|
||||
| Redis | Cache + Celery broker | 6379 |
|
||||
| Ollama | Local LLM (qwen2:7b + nomic-embed-text) | 11434 |
|
||||
| Typesense 0.25.2 | Arabic full-text search | 8108 |
|
||||
| Celery | Async task queue (PDF pipeline) | — |
|
||||
| Flower | Celery monitor dashboard | 5555 |
|
||||
| Cloudflare R2 | PDF + cover CDN storage | — |
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Upload → VPS disk → Compress (Ghostscript) → Ingest (Qdrant vectors + PG chunks)
|
||||
→ Celery task → Meta extract (Ollama) → R2 upload → Typesense index
|
||||
→ BM25 rebuild
|
||||
|
||||
Query → Dense retrieval (Qdrant top 15) → BM25 (rank_bm25 top 15)
|
||||
→ RRF fusion → CrossEncoder rerank → MMR diversity
|
||||
→ Ollama qwen2:7b response with Arabic citations
|
||||
```
|
||||
|
||||
## Requirements
|
||||
|
||||
- Ubuntu 22.04 or 24.04
|
||||
- Docker + Docker Compose
|
||||
- Minimum 8GB RAM (16GB recommended for Ollama)
|
||||
- 50GB disk space
|
||||
- Domain with DNS control (for SSL)
|
||||
- Cloudflare R2 bucket (optional but recommended)
|
||||
|
||||
## Quick Install
|
||||
|
||||
```bash
|
||||
# 1. Clone this repo
|
||||
git clone https://git.asktam.net/colorlink/afakzaid-rag
|
||||
cd afakzaid-rag
|
||||
|
||||
# 2. Run setup
|
||||
bash scripts/setup.sh your-domain.com
|
||||
|
||||
# 3. Follow post-install steps printed at the end
|
||||
```
|
||||
|
||||
## Manual Install
|
||||
|
||||
### Step 1 — Prepare directories
|
||||
|
||||
```bash
|
||||
mkdir -p /opt/rag /mnt/rag-media/pdfs /mnt/rag-media/covers
|
||||
mkdir -p /opt/typesense/data /opt/qdrant/storage
|
||||
cp -r . /opt/rag/
|
||||
```
|
||||
|
||||
### Step 2 — Install Ollama
|
||||
|
||||
```bash
|
||||
curl -fsSL https://ollama.ai/install.sh | sh
|
||||
systemctl enable --now ollama
|
||||
ollama pull nomic-embed-text
|
||||
ollama pull qwen2:7b
|
||||
```
|
||||
|
||||
### Step 3 — Create Docker network
|
||||
|
||||
```bash
|
||||
docker network create rag-net
|
||||
```
|
||||
|
||||
### Step 4 — Generate secrets
|
||||
|
||||
```bash
|
||||
bash scripts/gen-secrets.sh
|
||||
# Secrets written to /opt/rag/.secrets
|
||||
# Edit /opt/rag/.secrets to add R2 credentials
|
||||
```
|
||||
|
||||
### Step 5 — Start services
|
||||
|
||||
```bash
|
||||
bash scripts/start-all.sh
|
||||
```
|
||||
|
||||
### Step 6 — Setup reverse proxy
|
||||
|
||||
```bash
|
||||
bash scripts/setup-nginx.sh your-domain.com
|
||||
```
|
||||
|
||||
## Environment Variables
|
||||
|
||||
All secrets stored in `/opt/rag/.secrets` (chmod 600):
|
||||
|
||||
```env
|
||||
REDIS_PASSWORD= # Redis auth password
|
||||
QDRANT_API_KEY= # Qdrant service API key
|
||||
TYPESENSE_API_KEY= # Typesense collection key
|
||||
FLOWER_USER= # Flower dashboard user
|
||||
FLOWER_PASSWORD= # Flower dashboard password
|
||||
CELERY_BROKER_URL= # Auto-generated from REDIS_PASSWORD
|
||||
RAG_API_KEY= # Public API key for queries
|
||||
ADMIN_KEY= # Admin endpoint key
|
||||
JWT_SECRET= # JWT signing secret (64 chars)
|
||||
R2_ENDPOINT= # https://<account>.r2.cloudflarestorage.com
|
||||
R2_ACCESS_KEY= # Cloudflare R2 access key ID
|
||||
R2_SECRET_KEY= # Cloudflare R2 secret key
|
||||
R2_BUCKET= # R2 bucket name (e.g. library)
|
||||
R2_PUBLIC_URL= # CDN URL (e.g. https://cdn.yourdomain.com)
|
||||
```
|
||||
|
||||
## PDF Upload Pipeline
|
||||
|
||||
1. File saved to `/mnt/rag-media/pdfs/`
|
||||
2. Ghostscript compression (skipped if file < 200KB or compressed > original)
|
||||
3. Text extraction + chunking (~150 words/chunk, 200w max)
|
||||
4. Dense embeddings via `nomic-embed-text` → Qdrant
|
||||
5. Metadata extraction via `qwen2:7b` (title, author, category, description)
|
||||
6. Upload PDF + cover to Cloudflare R2
|
||||
7. Index in Typesense (Arabic full-text)
|
||||
8. BM25 index rebuild
|
||||
|
||||
## RAG Query Pipeline
|
||||
|
||||
1. Dense retrieval: top 15 chunks from Qdrant (threshold 0.20)
|
||||
2. BM25 retrieval: top 15 chunks (Arabic tokenization)
|
||||
3. RRF fusion (k=60)
|
||||
4. CrossEncoder rerank: `cross-encoder/mmarco-mMiniLMv2-L12-H384-v1`
|
||||
5. MMR diversity filter (λ=0.6, final 5 chunks)
|
||||
6. Answer generation: `qwen2:7b` with Arabic system prompt
|
||||
7. Numbered citations [١][٢] in response
|
||||
|
||||
## API Endpoints
|
||||
|
||||
```
|
||||
GET /stats System statistics
|
||||
POST /query RAG query (requires API key)
|
||||
GET /api/library List books
|
||||
POST /api/library/upload Upload single PDF
|
||||
POST /api/library/bulk-upload Bulk upload PDFs
|
||||
GET /api/library/{id}/pdf Serve/redirect PDF
|
||||
GET /api/library/{id}/download Download PDF
|
||||
GET /library/{id}/cover Book cover image
|
||||
GET /admin/overview Admin dashboard stats
|
||||
GET /admin/services Live service health
|
||||
GET /admin/books Book management
|
||||
GET /admin/r2-test Test R2 connection
|
||||
POST /admin/r2-config Save R2 credentials
|
||||
```
|
||||
|
||||
## Security
|
||||
|
||||
- All internal service ports blocked from external access via iptables
|
||||
- Redis password protected
|
||||
- Qdrant API key enforced
|
||||
- FastAPI: docs/openapi endpoints disabled in production
|
||||
- CORS restricted to configured domain
|
||||
- Rate limiting: login 10/min, query 30/min, ingest 20/min
|
||||
- JWT fails fast if secret not set
|
||||
- File uploads: 50MB limit, path traversal protection
|
||||
|
||||
## File Structure
|
||||
|
||||
```
|
||||
/opt/rag/
|
||||
├── api.py FastAPI app + admin endpoints
|
||||
├── routes.py Library API router
|
||||
├── brain.py Hybrid RAG pipeline
|
||||
├── ingest.py PDF ingestion + chunking
|
||||
├── tasks.py Celery task definitions
|
||||
├── celery_app.py Celery app configuration
|
||||
├── compress.py Ghostscript PDF compression
|
||||
├── storage.py Cloudflare R2 file operations
|
||||
├── typesense_client.py Typesense search integration
|
||||
├── meta_extract.py Ollama metadata extraction
|
||||
├── auth.py JWT authentication
|
||||
├── worker.py Legacy threading worker (fallback)
|
||||
├── .secrets All credentials (chmod 600)
|
||||
└── static/
|
||||
├── index.html Home page + RAG query interface
|
||||
├── library.html Book library with grid/list view
|
||||
├── reader.html PDF reader (redirects to R2)
|
||||
└── admin.html Admin dashboard
|
||||
|
||||
/mnt/rag-media/
|
||||
├── pdfs/ PDF files (permanent, used for re-ingestion)
|
||||
└── covers/ Cover images extracted from PDFs
|
||||
|
||||
scripts/
|
||||
├── setup.sh Full automated setup
|
||||
├── gen-secrets.sh Generate fresh credentials
|
||||
├── start-all.sh Start all Docker containers
|
||||
└── setup-nginx.sh Nginx + SSL configuration
|
||||
```
|
||||
|
||||
## Version History
|
||||
|
||||
- v1.7.4 — Celery for all uploads, R2 delivery, Typesense search
|
||||
- v1.7.3 — PDF compression via Ghostscript
|
||||
- v1.7.2 — Cloudflare R2 redirect for download/read
|
||||
- v1.7.0 — Typesense + Celery + MinIO (MinIO later disabled)
|
||||
- v1.5.x — Hybrid RAG (BM25 + dense + RRF + CrossEncoder + MMR)
|
||||
- v1.0.0 — Initial FastAPI + Qdrant + Ollama
|
||||
|
||||
## Database Schema
|
||||
|
||||
Key tables in PostgreSQL (`ragdb`):
|
||||
|
||||
- `users` — user accounts with roles and permissions
|
||||
- `books` — book metadata (title, author, category, pdf_url, cover_url)
|
||||
- `chunks` — text chunks with book reference (used for BM25)
|
||||
- `conversations` — query history
|
||||
- `memory` — Mem0 conversation memory
|
||||
- `bulk_jobs` / `bulk_job_files` — Celery upload job tracking
|
||||
- `api_settings` — runtime configuration (Claude API key, etc.)
|
||||
- `favorites` — user favorite books
|
||||
|
||||
## Notes for AI Assistants
|
||||
|
||||
When helping install this system on a new server:
|
||||
|
||||
1. Ollama runs on the HOST (not in Docker) — containers reach it via the Docker gateway IP (usually `172.17.0.1` or `172.16.x.1`)
|
||||
2. The Docker image `rag-api-img:secure` is pre-built with: Python 3.12, PyTorch, sentence-transformers, Ghostscript, Celery, Typesense client, boto3
|
||||
3. All inter-service communication uses container names (e.g., `rag-postgres`, `rag-redis`) within the Docker network
|
||||
4. The `R2_PUBLIC_URL` must be updated per deployment — point to your CDN domain for that server
|
||||
5. Qdrant vectors must be restored from snapshots — they cannot be regenerated without re-uploading all PDFs
|
||||
6. The BM25 index is rebuilt in-memory at startup from the `chunks` table — no migration needed
|
||||
7. Typesense collection `books` is auto-created on first startup — data restored via `index_all_books()`
|
||||
8. After import, run: `docker exec rag-api python3 -c "from typesense_client import index_all_books; index_all_books()"`
|
||||
|
||||
Reference in New Issue
Block a user