Kc/auto limit

.vscode/settings.json

@@ -2,8 +2,8 @@
   "editor.formatOnSave": true,
   "editor.defaultFormatter": "esbenp.prettier-vscode",
   "editor.codeActionsOnSave": {
-    "source.fixAll.eslint": true
-  },
+    "source.fixAll.eslint": "explicit"
+},
   "eslint.validate": [
     "javascript",
     "javascriptreact",

PAGINATION_IMPLEMENTATION.md

@@ -0,0 +1,187 @@
+# Pagination Implementation - DataPup
+
+## ✅ Implementation Complete
+
+This document outlines the comprehensive pagination feature that has been implemented in DataPup to automatically limit SELECT query results to 100 records by default and provide pagination controls.
+
+## 🚀 Features Implemented
+
+### Backend Implementation
+
+1. **Database Interface Updates** (`src/main/database/interface.ts`)
+   - Added `PaginationOptions` interface with `page` and `limit` parameters
+   - Added `PaginationInfo` interface with complete pagination metadata
+   - Extended `QueryResult` to include pagination information
+   - Updated `query()` method signature to accept optional pagination parameters
+
+2. **Base Database Manager** (`src/main/database/base.ts`)
+   - Smart SQL parsing to detect existing LIMIT clauses
+   - Automatic LIMIT/OFFSET injection for SELECT queries
+   - Logic to respect user-specified LIMIT clauses when smaller than page size
+   - Helper methods for pagination metadata generation
+
+3. **ClickHouse Implementation** (`src/main/database/clickhouse.ts`)
+   - Automatic pagination for SELECT queries only
+   - Total count calculation using `SELECT COUNT(*) FROM (query)`
+   - Efficient pagination with proper LIMIT/OFFSET syntax
+   - Enhanced result messages showing pagination status
+
+4. **IPC Layer** (`src/main/index.ts`)
+   - Updated `db:query` handler to accept pagination parameters
+   - Maintains full backward compatibility
+
+### Frontend Implementation
+
+5. **Type Definitions** (`src/renderer/types/tabs.ts`)
+   - Added `PaginationInfo` interface matching backend structure
+   - Extended `QueryExecutionResult` with pagination data
+
+6. **Pagination Component** (`src/renderer/components/Pagination/`)
+   - Reusable pagination component with full navigation controls
+   - Page size selector (25, 50, 100, 200, 500 rows)
+   - Current page, total pages, and record count display
+   - Proper disabled states and accessibility
+
+7. **Enhanced Export Functionality** (`src/renderer/components/ExportButton/`)
+   - Smart export component with pagination awareness
+   - Dropdown options for "Current Page" vs "All Data" export
+   - Automatic filename generation with pagination context
+   - Support for CSV, JSON, and SQL export formats
+
+8. **Query Workspace Integration** (`src/renderer/components/QueryWorkspace/`)
+   - Per-tab pagination state management
+   - Integrated pagination controls in results section
+   - Page navigation and size change handlers
+   - Export functionality for both current page and all data
+
+## 🔧 How It Works
+
+### Automatic Pagination Logic
+
+1. **Query Analysis**: The system detects if a query is a SELECT statement
+2. **LIMIT Detection**: Checks for existing LIMIT clauses in user queries
+3. **Smart Application**: 
+   - Applies pagination to SELECT queries without LIMIT
+   - Applies pagination to SELECT queries with LIMIT > page size
+   - Respects user LIMIT when ≤ page size
+   - Never applies pagination to INSERT, UPDATE, DELETE, or DDL queries
+
+### Pagination Workflow
+
+```sql
+-- User Query
+SELECT * FROM users WHERE age > 18
+
+-- Automatically becomes (page 1, 100 rows)
+SELECT * FROM users WHERE age > 18 LIMIT 100 OFFSET 0
+
+-- And generates count query for total pages
+SELECT COUNT(*) as total FROM (SELECT * FROM users WHERE age > 18)
+```
+
+### Export Options
+
+- **Current Page**: Exports only the currently displayed records (e.g., 100 records)
+- **All Data**: Re-executes the original query without pagination to export all records
+- **Smart Filenames**: 
+  - Current page: `query-results-page-1-of-5.csv`
+  - All data: `query-results-all-1250-records.csv`
+
+## 🧪 Testing Scenarios
+
+### Query Type Tests ✅
+
+| Query Type | Pagination Applied | Notes |
+|------------|-------------------|-------|
+| `SELECT * FROM table` | ✅ Yes | Default 100 records, page 1 |
+| `SELECT * FROM table LIMIT 50` | ❌ No | User limit ≤ page size |
+| `SELECT * FROM table LIMIT 500` | ✅ Yes | User limit > page size, override to 100 |
+| `INSERT INTO table...` | ❌ No | Non-SELECT query |
+| `UPDATE table SET...` | ❌ No | Non-SELECT query |
+| `CREATE TABLE...` | ❌ No | DDL query |
+| `SHOW TABLES` | ❌ No | System query |
+
+### Pagination Control Tests ✅
+
+| Page | Page Size | Expected OFFSET | Expected Records |
+|------|-----------|-----------------|------------------|
+| 1 | 100 | 0 | 1-100 |
+| 2 | 100 | 100 | 101-200 |
+| 3 | 50 | 100 | 101-150 |
+| 5 | 200 | 800 | 801-1000 |
+
+### Export Tests ✅
+
+| Scenario | Export Type | Expected Behavior |
+|----------|------------|-------------------|
+| Single page results | Simple buttons | CSV/JSON buttons only |
+| Multi-page results | Dropdown menu | Current Page + All Data options |
+| Current page export | Limited data | Exports ~100 records with page info |
+| All data export | Full dataset | Exports all records with total count |
+
+## 🎯 User Experience
+
+### Default Behavior
+- SELECT queries automatically show first 100 records
+- Pagination controls appear below results table
+- Page information shows "Page 1 of X" with record counts
+
+### User Controls
+- **Page Navigation**: First, Previous, Next, Last buttons
+- **Page Size**: Dropdown to select 25/50/100/200/500 rows per page
+- **Export Options**: Current page or all data via dropdown menus
+- **Per-Tab State**: Each query tab maintains independent pagination settings
+
+### Performance Benefits
+- Reduced memory usage (only current page in memory)
+- Faster initial query results
+- Efficient navigation with LIMIT/OFFSET
+- Background total count calculation
+
+## 📁 Files Modified
+
+### Backend Files
+- `src/main/database/interface.ts` - Core pagination interfaces
+- `src/main/database/base.ts` - Base pagination logic
+- `src/main/database/clickhouse.ts` - ClickHouse pagination implementation
+- `src/main/index.ts` - IPC handler updates
+
+### Frontend Files
+- `src/renderer/types/tabs.ts` - Frontend pagination types
+- `src/renderer/components/Pagination/` - New pagination component
+- `src/renderer/components/ExportButton/` - New enhanced export component
+- `src/renderer/components/QueryWorkspace/QueryWorkspace.tsx` - Integration
+- `src/renderer/components/QueryWorkspace/QueryWorkspace.css` - Styling
+
+### Test Files
+- `src/test/pagination.test.ts` - Comprehensive test scenarios
+
+## 🚦 Current Status
+
+✅ **All Features Implemented and Tested**
+- Backend pagination logic complete
+- Frontend pagination controls implemented
+- Enhanced export functionality working
+- Comprehensive test scenarios validated
+- Build successful with no critical errors
+
+## 🔄 Future Enhancements
+
+Potential future improvements:
+1. **Virtual Scrolling**: For very large result sets
+2. **Search Within Results**: Filter current page data
+3. **Bookmarkable Pages**: URL-based page navigation
+4. **Export Progress**: Progress indicators for large exports
+5. **Custom Page Sizes**: User-defined page size limits
+6. **Keyboard Shortcuts**: Ctrl+Left/Right for page navigation
+
+## 🏃‍♂️ Quick Test Guide
+
+1. **Connect to ClickHouse** with a database containing >100 records
+2. **Execute**: `SELECT * FROM [large_table]`
+3. **Verify**: Pagination controls appear, showing "Page 1 of X"
+4. **Test Navigation**: Click through different pages
+5. **Test Page Size**: Change from 100 to 50 rows per page
+6. **Test Export**: Try both "Current Page" and "All Data" export options
+
+The pagination implementation is now ready for production use! 🎉

src/main/database/base.ts

@@ -11,7 +11,9 @@ import {
   DatabaseCapabilities,
   TransactionHandle,
   BulkOperation,
-  BulkOperationResult
+  BulkOperationResult,
+  PaginationOptions,
+  PaginationInfo
 } from './interface'
 
 export abstract class BaseDatabaseManager implements DatabaseManagerInterface {
@@ -77,7 +79,8 @@ export abstract class BaseDatabaseManager implements DatabaseManagerInterface {
     data?: any[],
     error?: string,
     queryType?: QueryType,
-    affectedRows?: number
+    affectedRows?: number,
+    pagination?: PaginationInfo
   ): QueryResult {
     const isDDL = queryType === QueryType.DDL
     const isDML = [QueryType.INSERT, QueryType.UPDATE, QueryType.DELETE].includes(
@@ -92,11 +95,82 @@ export abstract class BaseDatabaseManager implements DatabaseManagerInterface {
       queryType,
       affectedRows,
       isDDL,
-      isDML
+      isDML,
+      pagination
     }
   }
 
-  abstract query(connectionId: string, sql: string): Promise<QueryResult>
+  protected parseSQLForPagination(sql: string): {
+    hasLimit: boolean
+    hasOffset: boolean
+    originalLimit?: number
+    originalOffset?: number
+  } {
+    const upperSql = sql.toUpperCase()
+
+    // Check for LIMIT clause
+    const limitMatch = upperSql.match(/\bLIMIT\s+(\d+)/i)
+    const offsetMatch = upperSql.match(/\bOFFSET\s+(\d+)/i)
+
+    return {
+      hasLimit: !!limitMatch,
+      hasOffset: !!offsetMatch,
+      originalLimit: limitMatch ? parseInt(limitMatch[1]) : undefined,
+      originalOffset: offsetMatch ? parseInt(offsetMatch[1]) : undefined
+    }
+  }
+
+  protected addPaginationToSQL(sql: string, pagination: PaginationOptions): string {
+    const { page = 1, limit = 100 } = pagination
+    const offset = (page - 1) * limit
+
+    // For databases that support LIMIT/OFFSET syntax (most SQL databases)
+    // Remove existing LIMIT/OFFSET clauses first
+    let cleanSql = sql.replace(/\s+LIMIT\s+\d+(\s+OFFSET\s+\d+)?/gi, '')
+
+    // Add new LIMIT and OFFSET
+    return `${cleanSql} LIMIT ${limit} OFFSET ${offset}`
+  }
+
+  protected createPaginationInfo(page: number, limit: number, totalCount?: number): PaginationInfo {
+    const totalPages = totalCount ? Math.ceil(totalCount / limit) : undefined
+
+    return {
+      currentPage: page,
+      pageSize: limit,
+      totalCount,
+      totalPages,
+      hasMore: totalPages ? page < totalPages : false,
+      hasPrevious: page > 1
+    }
+  }
+
+  protected shouldApplyPagination(sql: string, pagination?: PaginationOptions): boolean {
+    if (!pagination) return false
+
+    const queryType = this.detectQueryType(sql)
+    // Only apply pagination to SELECT queries
+    if (queryType !== QueryType.SELECT) return false
+
+    // Check if query already has a user-specified LIMIT that's smaller than our pagination
+    const sqlInfo = this.parseSQLForPagination(sql)
+    if (
+      sqlInfo.hasLimit &&
+      sqlInfo.originalLimit &&
+      sqlInfo.originalLimit <= (pagination.limit || 100)
+    ) {
+      return false // User has already specified a smaller limit
+    }
+
+    return true
+  }
+
+  abstract query(
+    connectionId: string,
+    sql: string,
+    sessionId?: string,
+    pagination?: PaginationOptions
+  ): Promise<QueryResult>
 
   async cancelQuery(
     connectionId: string,