Problem Statement
To develop a robust GenAI system for processing multiple documents using AWS Bedrock’s multi-modal LLM capabilities, with support for parallel processing and comprehensive error handling.
Pipeline Flow
- Input Processing
- Reads files from a specified input folder.
- Processes multiple prompts sequentially.
- Parallel Execution
- Creates a thread pool for concurrent processing.
- Distributes files across available threads for efficient execution.
- Output Management
- Creates an organized output directory structure.
- Generates individual CSV files for each document.
- Produces a consolidated final output file.
- Error Handling
- Captures and logs exceptions at multiple levels of the pipeline.
- Maintains processing continuity by bypassing failed tasks.
- Generates detailed exception logs for troubleshooting.
- Results
- Produces per-prompt CSV outputs for detailed analysis.
- Creates a consolidated final output for all processed documents.
- Provides exception logs for failed processes to ensure traceability.
