mirror of
https://github.com/ada-dmitry/osint_parser_ada.git
synced 2026-09-24 01:10:20 +00:00
feat: Add data models for disciplinary actions, organizations, and training centers
- Implemented DisciplinaryAction model for tracking disciplinary measures. - Created Organization model to represent auditing organizations with relevant details. - Developed TrainingCenter model for educational and methodological centers. - Added methods to convert models to dictionaries for easy export. feat: Implement parsers for auditors and organizations - Developed AuditorsParser to scrape and parse auditor registry data. - Created OrganizationsParser for scraping auditing organizations' registry. - Implemented GenericRegistryParser for handling generic tabular data. chore: Set up base parser functionality - Established BaseParser class with common methods for making requests and parsing HTML. - Added pagination handling and detailed page parsing capabilities. feat: Implement Excel export functionality - Created ExcelExporter class for exporting data to Excel files with formatting options. - Added support for exporting multiple sheets and organization data specifically. chore: Add logging utilities - Implemented setup_logger function for consistent logging across the application. - Configured logging to output to both console and file. chore: Update requirements and scripts - Added necessary dependencies for parsing, Excel handling, and logging. - Created run.sh and run.fish scripts for easy execution of the parser.
This commit is contained in:
@@ -0,0 +1,51 @@
|
||||
#!/usr/bin/env fish
|
||||
|
||||
# Скрипт быстрого запуска для Fish Shell
|
||||
# Использование: ./run.fish
|
||||
|
||||
echo "=============================================="
|
||||
echo " ПАРСЕР РЕЕСТРОВ СРО ААС"
|
||||
echo "=============================================="
|
||||
echo ""
|
||||
|
||||
# Проверка наличия Python
|
||||
if not command -v python3 &> /dev/null
|
||||
echo "❌ Python 3 не найден. Установите Python 3.8 или выше."
|
||||
exit 1
|
||||
end
|
||||
|
||||
echo "✅ Python найден: "(python3 --version)
|
||||
echo ""
|
||||
|
||||
# Проверка виртуального окружения
|
||||
if not test -d venv
|
||||
echo "📦 Виртуальное окружение не найдено. Создание..."
|
||||
python3 -m venv venv
|
||||
echo "✅ Виртуальное окружение создано."
|
||||
echo ""
|
||||
end
|
||||
|
||||
# Активация виртуального окружения
|
||||
echo "🔧 Активация виртуального окружения..."
|
||||
source venv/bin/activate.fish
|
||||
|
||||
# Проверка зависимостей
|
||||
if not python -c "import requests" 2>/dev/null
|
||||
echo "📥 Установка зависимостей..."
|
||||
pip install -q --upgrade pip
|
||||
pip install -q -r requirements.txt
|
||||
echo "✅ Зависимости установлены."
|
||||
echo ""
|
||||
end
|
||||
|
||||
# Создание необходимых директорий
|
||||
mkdir -p data/exports
|
||||
mkdir -p logs
|
||||
|
||||
# Запуск парсера
|
||||
echo "🚀 Запуск парсера..."
|
||||
echo ""
|
||||
python main.py
|
||||
|
||||
# Деактивация виртуального окружения
|
||||
deactivate
|
||||
Reference in New Issue
Block a user