Create AI-ready files from site crawls
Workflow Description
Advanced automation that transforms crawled website data into AI-ready files through automatic classification, summarization, and content extraction. Processes large-scale web data for machine learning models using OpenAI integration and intelligent filtering.
How it works
- 1.Trigger workflow via form submission with crawled website data
- 2.Classify content automatically using text classifier and AI injection
- 3.Summarize important files and convert to standardized formats
- 4.Extract key data from documents and organize hierarchically
- 5.Apply filters to remove irrelevant or duplicate content
- 6.Generate final files optimized for AI model consumption
Use cases
- Preparing large website content repositories for AI model training and fine-tuning
- Extracting structured data from web pages for intelligent search systems
- Creating documented knowledge bases from manual and automated web crawling
Requirements
- Active OpenAI API key for classification and summarization services
- Configured source web form to receive crawled site data
Service Value
Ideal as a smart automation service combining integrations and AI to produce ready-to-use results.
Apps Used
Details
How to Use
- 1.Click "Download Template"
- 2.Open your n8n dashboard
- 3.Go to Workflows > Import from File
- 4.Select downloaded file and configure credentials
Nodes Used (23)
Sticky Note
Sticky Note
Sticky Note1
Sticky Note
Sticky Note2
Sticky Note
Set useful fields
Set
Text Classifier
If
OpenAI Chat Model
OpenAI
No Operation, do nothing
No Op
Sticky Note3
Sticky Note
Filter URLs
Filter
Sticky Note4
Sticky Note
Sticky Note5
Sticky Note
Sticky Note6
Sticky Note
Sticky Note7
Sticky Note
Sticky Note8
Sticky Note
Sticky Note9
Sticky Note
Summarize - Concatenate
Summarize
Set Fields - llms.txt Content
Set
upload file anywhere
No Op
Set Field - llms.txt Row
Set
Sticky Note10
Sticky Note
Form - Screaming frog internal_html.csv upload
Form Trigger
Extract data from Screaming Frog file
Extract From File
Generate llms.txt file
Convert To File