mirror of
https://github.com/ada-dmitry/osint_parser_ada.git
synced 2026-09-24 01:10:20 +00:00
feat: Add data models for disciplinary actions, organizations, and training centers
- Implemented DisciplinaryAction model for tracking disciplinary measures. - Created Organization model to represent auditing organizations with relevant details. - Developed TrainingCenter model for educational and methodological centers. - Added methods to convert models to dictionaries for easy export. feat: Implement parsers for auditors and organizations - Developed AuditorsParser to scrape and parse auditor registry data. - Created OrganizationsParser for scraping auditing organizations' registry. - Implemented GenericRegistryParser for handling generic tabular data. chore: Set up base parser functionality - Established BaseParser class with common methods for making requests and parsing HTML. - Added pagination handling and detailed page parsing capabilities. feat: Implement Excel export functionality - Created ExcelExporter class for exporting data to Excel files with formatting options. - Added support for exporting multiple sheets and organization data specifically. chore: Add logging utilities - Implemented setup_logger function for consistent logging across the application. - Configured logging to output to both console and file. chore: Update requirements and scripts - Added necessary dependencies for parsing, Excel handling, and logging. - Created run.sh and run.fish scripts for easy execution of the parser.
This commit is contained in:
@@ -0,0 +1,18 @@
|
||||
# Основные библиотеки для парсинга
|
||||
requests>=2.31.0
|
||||
beautifulsoup4>=4.12.0
|
||||
lxml>=4.9.0
|
||||
|
||||
# Работа с Excel
|
||||
openpyxl>=3.1.0
|
||||
pandas>=2.0.0
|
||||
|
||||
# Selenium для динамического контента (опционально)
|
||||
selenium>=4.15.0
|
||||
|
||||
# Утилиты
|
||||
python-dateutil>=2.8.0
|
||||
tqdm>=4.66.0
|
||||
|
||||
# Логирование
|
||||
colorlog>=6.7.0
|
||||
Reference in New Issue
Block a user