The All File Converter service is designed as a scalable, distributed SaaS system built on a modern asynchronous Python 3.12 stack. The architecture is divided into independent layers: network event intake, task orchestration, isolated execution of heavy binary processes, and the analytical loop.

1. General Technology Stack and System Architecture

The platform is built on the principles of high-throughput performance, minimal memory consumption, and fault tolerance:

  • Telegram Bot Framework: Aiogram 3.13 running in Webhook mode with secret token validation and custom message lifecycle filters.
  • Web Gateway and REST API: FastAPI based on the Uvicorn ASGI server with asynchronous binary file streaming (FileResponse) and background cleanup via BackgroundTasks.
  • Queue Broker and Cache: Redis 7 (Celery task management, race condition locks, brute force protection, session caching).
  • Background Executor (Task Queue): Celery 5.4 with a dedicated worker pool in an isolated converter_worker container.
  • Database: PostgreSQL 16 with the SQLAlchemy 2.0 (asyncpg) ORM layer, a persistent connection pool (20+10 overflow), and automatic retry for failed transactions (@db_retry).
  • Network Perimeter: Tunneling via Cloudflare Zero Trust with direct IP access to the server blocked via Middleware.

2. Asynchronous Queues and Isolation of Heavy Computations (Celery + Redis)

Media file and office document conversions create peak loads on CPU and RAM. To prevent incoming Telegram message processing from being blocked during heavy operations, strict isolation is implemented:

  • Task Delegation to Redis: Upon format selection, the Telegram handler registers the task in the database with the PROCESSING status and places the job in the tasks.execute_conversion queue via Celery.
  • Isolated Worker Container: Conversion utilities are executed within a separate Linux container with its own CPU time and memory limits.
  • Freeze Control and Timeouts: External utility calls are wrapped in an asynchronous context with strict time controls (conversion_timeout_sec = 180). If the limit is exceeded, the process is forcibly terminated via proc.kill(), freeing up resources.
  • Fault-Tolerant Fallback: If the Redis broker is temporarily unavailable, the task is automatically intercepted by the local asynchronous dispatcher and executed directly without disrupting the user.

3. Specialized Conversion Engine Pipeline

Highly specialized native utilities and libraries are used for each data type:

  • Documents and Spreadsheets (LibreOffice): A headless office suite (soffice --headless) for precise rendering of DOCX, XLSX, PPTX, RTF, and ODT into PDF or text files.
  • Streaming Audio and Video (FFmpeg): Multithreaded video codec transcoding (H.264), audio codecs (MP3, OGG Opus), audio track extraction, GIF generation (Lanczos filter), and square cropping (1:1) for Telegram video messages.
  • High-Speed PDF Processing (Poppler Utils): pdftotext utilities (instant extraction of formatted text in UTF-8) and pdftoppm (page-by-page rendering of PDFs into raster images without LibreOffice overhead).
  • Optical Character Recognition (Tesseract OCR): Neural network-based extraction of printed text from scans and photos in 40+ languages.
  • Raster and Vector Graphics: Pillow libraries (including HEIC and AVIF support), CairoSVG for vector images, and lottie for animated Telegram stickers (.TGS).
  • Books, Subtitles, and Fonts: The Calibre engine (ebook-convert), the pysubs2 subtitle parser (SRT, VTT, ASS, SSA), and the fonttools font compiler (Brotli compression into WOFF2).

4. Proactive Resource Protection (System Guard)

To prevent server crashes due to insufficient memory (OOM Killer), the proactive System Guard diagnostic service is implemented. Before accepting a file for processing, the system checks key host metrics:

  • Free RAM: Minimum 500 MB of free space (guard_min_free_ram_mb).
  • Disk Space: Minimum 2 GB of free space in the /tmp directory (guard_min_free_disk_mb).
  • Task Queue: Limitation on the Celery queue length (no more than 20 pending tasks).

If limits are exceeded, the service temporarily enables protection (HTTP 503 / chat message), preventing server overload and sending an instant alert to administrators in Telegram.

5. File Lifecycle and Security (GDPR)

The architecture is designed around a Zero-Data-Footprint model:

  • User files are uploaded to a secure tmp/conversions/ volume with unique prefixes based on task IDs.
  • Files remain accessible strictly within the working session — no more than 15 minutes (900 seconds).
  • The automatic garbage collector deletes source and ready files immediately after confirming successful delivery to the chat or upon session timeout.
  • The PostgreSQL database does not store binary files or personal document texts — tables only record anonymized technical metadata (formats, sizes in bytes, processing time, statuses).

6. Universal REST API for External Web Services

The service is originally designed as a multi-platform backend. Alongside the bot, a fully functional secure API operates for websites (e.g., built on Django) and third-party bots:

  • GET /api/v1/formats — a dynamic JSON matrix of available conversion directions, synchronized with Google Sheets settings.
  • POST /api/v1/convert — a universal endpoint that accepts multipart/form-data (file, target format, external client ID) and returns the ready byte stream as a direct download.
  • Authorization via Bearer tokens (WEBHOOK_REFRESH_TOKEN) with protection against timing attacks via secrets.compare_digest.

7. Control Panel and Observability (NiceGUI + AG Grid)

Monitoring of business metrics and system status is displayed in a native Single-Page control panel based on NiceGUI 2.x:

  • Interactive AG Grid (v32+) tables with custom Excel-style checkbox filters (aggrid_filters.js).
  • Real-time monitoring of queues, API provider latencies, and PostgreSQL / Redis pings.
  • End-to-end date filtering with seamless integration of the Metabase BI dashboard via signed JWT tokens.