TiempoLens AI by team Tiempo Legends
Link to open source: https://github.com/AMP0075/TiempoLegends_HackFest2
# 🔍 TiempoLens AI
**Intelligent Data Dictionary Agent**
Automatically generate comprehensive, AI-enhanced data dictionaries from enterprise databases.
---
## 🎯 Problem Statement
Enterprise organizations struggle with **tribal knowledge**, **stale documentation**, and **disconnected metadata** across their data estates. Data engineers waste hours manually documenting schemas, analysts can't find what data means, and data quality issues go undetected until they cause downstream failures.
**TiempoLens AI** solves this by connecting to your databases and automatically generating rich, AI-enhanced data dictionaries with quality profiling, lineage visualization, and an interactive chat interface for querying metadata in natural language.
---
## ✨ Key Features
Feature
Description
🔌 **Multi-Database Connectivity**
PostgreSQL, MySQL, SQL Server, Snowflake, MongoDB
📊 **Auto Schema Extraction**
Tables, columns, indexes, constraints, relationships
🤖 **DS-STAR AI Agents**
Planner → Coder → Verifier → Router pipeline for intelligent analysis
📈 **Data Quality Profiling**
Completeness, freshness, uniqueness, validity scoring with alerts
🔮 **Time Series Forecasting**
Prophet-powered trend analysis on quality metrics
💬 **Interactive AI Chat**
Natural language queries with streaming responses & voice I/O
🔗 **Data Lineage**
Visual table & column-level lineage graphs
📄 **Doc Generation**
Markdown & JSON data dictionary export
🧠 **RAG Pipeline**
pgvector-powered semantic search over schema metadata
🎨 **Professional UI**
Dark/light themes, responsive layout, real-time updates
---
## 🏗️ Architecture
```
┌──────────────┐ ┌──────────────────────────────────────────────┐│ Next.js │ │ FastAPI Backend ││ Frontend │────>│ ││ (React 18) │ │ ┌─────────┐ ┌──────────┐ ┌───────────┐ │└──────────────┘ │ │ REST │ │ WebSocket│ │ Celery │ │ │ │ │ API │ │ Streaming│ │ Workers │ │ │ │ └────┬────┘ └─────┬────┘ └─────┬─────┘ │ │ │ │ │ │ │ ┌────┴────┐ │ ┌────┴─────────────┴─────────────┴─────┐ │ │ Nginx │ │ │ Service Layer │ │ │ Reverse │ │ │ Connection │ Schema │ Quality │ RAG │ │ │ Proxy │ │ └────────────────┬─────────────────────┘ │ └─────────┘ │ │ │ │ ┌────────────────┴────────────────────┐ │ │ │ DS-STAR Agent Layer │ │ │ │ Planner → Coder → Verifier → Router│ │ │ └────────────────┬────────────────────┘ │ └───────────────────┼──────────────────────────┘ │ ┌───────────────────┼──────────────────────┐ │ PostgreSQL │ Redis │ │ + pgvector │ Cache / PubSub │ │ (Metadata DB) │ (Celery Broker) │ └───────────────────┴──────────────────────┘
```
### DS-STAR Agent Pattern
Based on the research paper *"Data Science with LLM and Agents"*, our multi-agent system follows the DS-STAR architecture:
1. **Router** — Classifies incoming queries and routes to the right agent
2. **Planner** — Decomposes complex questions into executable steps
3. **Coder** — Generates SQL, Python code, or analytical answers
4. **Verifier** — Validates outputs for correctness and completeness
5. **Analyzer** — Specialized metadata analysis with business-friendly explanations
---
## 🛠️ Tech Stack
### Backend
- **Python 3.12** + **FastAPI** (async, HATEOAS REST)
- **SQLAlchemy 2.0** (async ORM with AsyncAdaptedQueuePool)
- **LangChain / LangGraph** (agent orchestration)
- **pgvector** (vector similarity search for RAG)
- **Celery** + **Redis** (background task processing)
- **Prophet** (time series forecasting)
### Frontend
- **Next.js 14** + **React 18** + **TypeScript** (strict mode)
- **Shadcn UI** + **Tailwind CSS** (component library)
- **Zustand** (state management)
- **ECharts** / **React Flow** (visualizations)
- **Web Speech API** (voice input/output)
### Infrastructure
- **Docker Compose** (7 services)
- **Nginx** (reverse proxy, rate limiting, WebSocket)
- **PostgreSQL 16** + pgvector (metadata & vector store)
- **Redis 7** (cache, pub/sub, Celery broker)
---
- **Frontend**: [http://localhost:3000](http://localhost:3000)
- **Backend API**: [http://localhost:8000](http://localhost:8000)
- **API Docs**: [http://localhost:8000/api/docs](http://localhost:8000/api/docs)
---
## 📁 Project Structure
```
├── backend/│ ├── app/│ │ ├── agents/ # DS-STAR agent layer│ │ │ ├── graph.py # LangGraph state machine│ │ │ ├── planner.py # Query decomposition│ │ │ ├── coder.py # Code/SQL generation│ │ │ ├── verifier.py # Output validation│ │ │ ├── router_agent.py # State routing│ │ │ └── analyzer.py # Metadata analysis│ │ ├── api/v1/ # REST API routes│ │ ├── core/ # Cache, security, middleware│ │ ├── models/ # SQLAlchemy ORM models│ │ ├── schemas/ # Pydantic request/response DTOs│ │ ├── services/ # Business logic layer│ │ ├── workers/ # Celery tasks│ │ ├── config.py # App configuration│ │ ├── database.py # Async DB engine│ │ └── main.py # FastAPI app entry point│ ├── Dockerfile│ └── requirements.txt├── frontend/│ ├── src/│ │ ├── app/ # Next.js app router│ │ ├── components/│ │ │ ├── layout/ # Sidebar, header│ │ │ ├── ui/ # Shadcn primitives│ │ │ └── views/ # Dashboard, explorer, quality, chat, lineage, settings│ │ └── lib/ # Store, API client, utils│ ├── Dockerfile│ └── package.json├── nginx/│ └── nginx.conf├── scripts/│ └── init-db.sql # Database schema initialization├── docker-compose.yml├── .env.example└── README.md
```
---
## 📡 API Endpoints
### Connections
Method
Endpoint
Description
GET
`/api/v1/connections`
List all connections
POST
`/api/v1/connections`
Create connection
POST
`/api/v1/connections/test`
Test connection
POST
`/api/v1/connections/{id}/sync`
Sync schema
### Schema
Method
Endpoint
Description
GET
`/api/v1/schemas/{conn_id}/overview`
Schema overview
GET
`/api/v1/schemas/{conn_id}/tables/{table}`
Table detail
GET
`/api/v1/schemas/{conn_id}/relationships`
All relationships
### Data Quality
Method
Endpoint
Description
GET
`/api/v1/quality/{conn_id}/overview`
Quality dashboard
POST
`/api/v1/quality/{conn_id}/profile/{table}`
Profile table
GET
`/api/v1/quality/{conn_id}/alerts`
Quality alerts
POST
`/api/v1/quality/{conn_id}/timeseries`
Time series analysis
### Chat
Method
Endpoint
Description
POST
`/api/v1/chat/sessions`
Create chat session
POST
`/api/v1/chat/sessions/{id}/messages`
Send message
WS
`/api/v1/chat/ws/{session_id}`
Streaming WebSocket
### Export
Method
Endpoint
Description
POST
`/api/v1/export/{conn_id}/dictionary`
Generate data dictionary
POST
`/api/v1/export/{conn_id}/ai-docs`
Generate AI documentation
### Lineage
Method
Endpoint
Description
GET
`/api/v1/lineage/{conn_id}/graph`
Full lineage graph
GET
`/api/v1/lineage/{conn_id}/table/{table}`
Table lineage
---
## 🗄️ Database Schema
Four PostgreSQL schemas organize all metadata:
- **`metadata`** — connections, schemas, tables, columns, relationships, indexes, constraints
- **`quality`** — table_profiles, column_profiles, alerts, trend_metrics
- **`ai`** — embeddings (pgvector), documentation_artifacts, chat_sessions, chat_messages
- **`audit`** — schema_versions, activity_log
---
## 🧪 Recommended Test Datasets
Dataset
Source
Tables
Description
**Olist Brazilian E-Commerce**
Kaggle
9
Orders, customers, products, reviews
**Bike Store**
SQLServerTutorial
9
Sales, inventory, staff
**Chinook**
GitHub
11
Digital media store
---
## 👥 Team Tiempo Legends
Built for **HackFest 2.0** (February 21-22, 2026)
---
This build was uploaded as a hackathon project
