Initial commit: project documentation

README.md

@@ -1,3 +1,236 @@
-# afakzaid-rag
+# AfakZaid RAG System
 
-AfakZaid RAG System
+Arabic AI library system with hybrid search, Celery task queue, Cloudflare R2 storage, and Typesense full-text search.
+
+## Stack
+
+| Service | Purpose | Port (internal) |
+|---|---|---|
+| FastAPI | Main API | 8100 |
+| PostgreSQL 15 | Metadata, users, sessions | 5432 |
+| Qdrant | Vector embeddings | 6333 |
+| Redis | Cache + Celery broker | 6379 |
+| Ollama | Local LLM (qwen2:7b + nomic-embed-text) | 11434 |
+| Typesense 0.25.2 | Arabic full-text search | 8108 |
+| Celery | Async task queue (PDF pipeline) | — |
+| Flower | Celery monitor dashboard | 5555 |
+| Cloudflare R2 | PDF + cover CDN storage | — |
+
+## Architecture
+
+```
+Upload → VPS disk → Compress (Ghostscript) → Ingest (Qdrant vectors + PG chunks)
+       → Celery task → Meta extract (Ollama) → R2 upload → Typesense index
+       → BM25 rebuild
+
+Query → Dense retrieval (Qdrant top 15) → BM25 (rank_bm25 top 15)
+      → RRF fusion → CrossEncoder rerank → MMR diversity
+      → Ollama qwen2:7b response with Arabic citations
+```
+
+## Requirements
+
+- Ubuntu 22.04 or 24.04
+- Docker + Docker Compose
+- Minimum 8GB RAM (16GB recommended for Ollama)
+- 50GB disk space
+- Domain with DNS control (for SSL)
+- Cloudflare R2 bucket (optional but recommended)
+
+## Quick Install
+
+```bash
+# 1. Clone this repo
+git clone https://git.asktam.net/colorlink/afakzaid-rag
+cd afakzaid-rag
+
+# 2. Run setup
+bash scripts/setup.sh your-domain.com
+
+# 3. Follow post-install steps printed at the end
+```
+
+## Manual Install
+
+### Step 1 — Prepare directories
+
+```bash
+mkdir -p /opt/rag /mnt/rag-media/pdfs /mnt/rag-media/covers
+mkdir -p /opt/typesense/data /opt/qdrant/storage
+cp -r . /opt/rag/
+```
+
+### Step 2 — Install Ollama
+
+```bash
+curl -fsSL https://ollama.ai/install.sh | sh
+systemctl enable --now ollama
+ollama pull nomic-embed-text
+ollama pull qwen2:7b
+```
+
+### Step 3 — Create Docker network
+
+```bash
+docker network create rag-net
+```
+
+### Step 4 — Generate secrets
+
+```bash
+bash scripts/gen-secrets.sh
+# Secrets written to /opt/rag/.secrets
+# Edit /opt/rag/.secrets to add R2 credentials
+```
+
+### Step 5 — Start services
+
+```bash
+bash scripts/start-all.sh
+```
+
+### Step 6 — Setup reverse proxy
+
+```bash
+bash scripts/setup-nginx.sh your-domain.com
+```
+
+## Environment Variables
+
+All secrets stored in `/opt/rag/.secrets` (chmod 600):
+
+```env
+REDIS_PASSWORD=           # Redis auth password
+QDRANT_API_KEY=           # Qdrant service API key
+TYPESENSE_API_KEY=        # Typesense collection key
+FLOWER_USER=              # Flower dashboard user
+FLOWER_PASSWORD=          # Flower dashboard password
+CELERY_BROKER_URL=        # Auto-generated from REDIS_PASSWORD
+RAG_API_KEY=              # Public API key for queries
+ADMIN_KEY=                # Admin endpoint key
+JWT_SECRET=               # JWT signing secret (64 chars)
+R2_ENDPOINT=              # https://<account>.r2.cloudflarestorage.com
+R2_ACCESS_KEY=            # Cloudflare R2 access key ID
+R2_SECRET_KEY=            # Cloudflare R2 secret key
+R2_BUCKET=                # R2 bucket name (e.g. library)
+R2_PUBLIC_URL=            # CDN URL (e.g. https://cdn.yourdomain.com)
+```
+
+## PDF Upload Pipeline
+
+1. File saved to `/mnt/rag-media/pdfs/`
+2. Ghostscript compression (skipped if file < 200KB or compressed > original)
+3. Text extraction + chunking (~150 words/chunk, 200w max)
+4. Dense embeddings via `nomic-embed-text` → Qdrant
+5. Metadata extraction via `qwen2:7b` (title, author, category, description)
+6. Upload PDF + cover to Cloudflare R2
+7. Index in Typesense (Arabic full-text)
+8. BM25 index rebuild
+
+## RAG Query Pipeline
+
+1. Dense retrieval: top 15 chunks from Qdrant (threshold 0.20)
+2. BM25 retrieval: top 15 chunks (Arabic tokenization)
+3. RRF fusion (k=60)
+4. CrossEncoder rerank: `cross-encoder/mmarco-mMiniLMv2-L12-H384-v1`
+5. MMR diversity filter (λ=0.6, final 5 chunks)
+6. Answer generation: `qwen2:7b` with Arabic system prompt
+7. Numbered citations [١][٢] in response
+
+## API Endpoints
+
+```
+GET  /stats                          System statistics
+POST /query                          RAG query (requires API key)
+GET  /api/library                    List books
+POST /api/library/upload             Upload single PDF
+POST /api/library/bulk-upload        Bulk upload PDFs
+GET  /api/library/{id}/pdf           Serve/redirect PDF
+GET  /api/library/{id}/download      Download PDF
+GET  /library/{id}/cover             Book cover image
+GET  /admin/overview                 Admin dashboard stats
+GET  /admin/services                 Live service health
+GET  /admin/books                    Book management
+GET  /admin/r2-test                  Test R2 connection
+POST /admin/r2-config                Save R2 credentials
+```
+
+## Security
+
+- All internal service ports blocked from external access via iptables
+- Redis password protected
+- Qdrant API key enforced
+- FastAPI: docs/openapi endpoints disabled in production
+- CORS restricted to configured domain
+- Rate limiting: login 10/min, query 30/min, ingest 20/min
+- JWT fails fast if secret not set
+- File uploads: 50MB limit, path traversal protection
+
+## File Structure
+
+```
+/opt/rag/
+├── api.py              FastAPI app + admin endpoints
+├── routes.py           Library API router
+├── brain.py            Hybrid RAG pipeline
+├── ingest.py           PDF ingestion + chunking
+├── tasks.py            Celery task definitions
+├── celery_app.py       Celery app configuration
+├── compress.py         Ghostscript PDF compression
+├── storage.py          Cloudflare R2 file operations
+├── typesense_client.py Typesense search integration
+├── meta_extract.py     Ollama metadata extraction
+├── auth.py             JWT authentication
+├── worker.py           Legacy threading worker (fallback)
+├── .secrets            All credentials (chmod 600)
+└── static/
+    ├── index.html      Home page + RAG query interface
+    ├── library.html    Book library with grid/list view
+    ├── reader.html     PDF reader (redirects to R2)
+    └── admin.html      Admin dashboard
+
+/mnt/rag-media/
+├── pdfs/               PDF files (permanent, used for re-ingestion)
+└── covers/             Cover images extracted from PDFs
+
+scripts/
+├── setup.sh            Full automated setup
+├── gen-secrets.sh      Generate fresh credentials
+├── start-all.sh        Start all Docker containers
+└── setup-nginx.sh      Nginx + SSL configuration
+```
+
+## Version History
+
+- v1.7.4 — Celery for all uploads, R2 delivery, Typesense search
+- v1.7.3 — PDF compression via Ghostscript
+- v1.7.2 — Cloudflare R2 redirect for download/read
+- v1.7.0 — Typesense + Celery + MinIO (MinIO later disabled)
+- v1.5.x — Hybrid RAG (BM25 + dense + RRF + CrossEncoder + MMR)
+- v1.0.0 — Initial FastAPI + Qdrant + Ollama
+
+## Database Schema
+
+Key tables in PostgreSQL (`ragdb`):
+
+- `users` — user accounts with roles and permissions
+- `books` — book metadata (title, author, category, pdf_url, cover_url)
+- `chunks` — text chunks with book reference (used for BM25)
+- `conversations` — query history
+- `memory` — Mem0 conversation memory
+- `bulk_jobs` / `bulk_job_files` — Celery upload job tracking
+- `api_settings` — runtime configuration (Claude API key, etc.)
+- `favorites` — user favorite books
+
+## Notes for AI Assistants
+
+When helping install this system on a new server:
+
+1. Ollama runs on the HOST (not in Docker) — containers reach it via the Docker gateway IP (usually `172.17.0.1` or `172.16.x.1`)
+2. The Docker image `rag-api-img:secure` is pre-built with: Python 3.12, PyTorch, sentence-transformers, Ghostscript, Celery, Typesense client, boto3
+3. All inter-service communication uses container names (e.g., `rag-postgres`, `rag-redis`) within the Docker network
+4. The `R2_PUBLIC_URL` must be updated per deployment — point to your CDN domain for that server
+5. Qdrant vectors must be restored from snapshots — they cannot be regenerated without re-uploading all PDFs
+6. The BM25 index is rebuilt in-memory at startup from the `chunks` table — no migration needed
+7. Typesense collection `books` is auto-created on first startup — data restored via `index_all_books()`
+8. After import, run: `docker exec rag-api python3 -c "from typesense_client import index_all_books; index_all_books()"`