feat: Add data models for disciplinary actions, organizations, and training centers

- Implemented DisciplinaryAction model for tracking disciplinary measures.
- Created Organization model to represent auditing organizations with relevant details.
- Developed TrainingCenter model for educational and methodological centers.
- Added methods to convert models to dictionaries for easy export.

feat: Implement parsers for auditors and organizations

- Developed AuditorsParser to scrape and parse auditor registry data.
- Created OrganizationsParser for scraping auditing organizations' registry.
- Implemented GenericRegistryParser for handling generic tabular data.

chore: Set up base parser functionality

- Established BaseParser class with common methods for making requests and parsing HTML.
- Added pagination handling and detailed page parsing capabilities.

feat: Implement Excel export functionality

- Created ExcelExporter class for exporting data to Excel files with formatting options.
- Added support for exporting multiple sheets and organization data specifically.

chore: Add logging utilities

- Implemented setup_logger function for consistent logging across the application.
- Configured logging to output to both console and file.

chore: Update requirements and scripts

- Added necessary dependencies for parsing, Excel handling, and logging.
- Created run.sh and run.fish scripts for easy execution of the parser.
This commit is contained in:
ada
2025-10-25 20:31:45 +03:00
commit 12d5c6eaff
23 changed files with 2210 additions and 0 deletions
+81
View File
@@ -0,0 +1,81 @@
"""
Модель данных для аудиторской организации
"""
from dataclasses import dataclass, field
from typing import Optional, List, Dict
from datetime import datetime
@dataclass
class Organization:
"""Модель аудиторской организации"""
# Основная информация
name: str
ornz: str # Основной регистрационный номер записи
inn: str
region: str
status: Optional[str] = None
# Дополнительная информация (заполняется при детальном парсе)
full_name: Optional[str] = None
ogrn: Optional[str] = None
kpp: Optional[str] = None
address: Optional[str] = None
phone: Optional[str] = None
email: Optional[str] = None
website: Optional[str] = None
director: Optional[str] = None
# Даты
registration_date: Optional[datetime] = None
membership_start_date: Optional[datetime] = None
membership_end_date: Optional[datetime] = None
# Дополнительные данные
auditors_count: Optional[int] = None
certificates: List[str] = field(default_factory=list)
networks: List[str] = field(default_factory=list)
# Метаинформация
source_url: Optional[str] = None
parsed_at: datetime = field(default_factory=datetime.now)
def to_dict(self) -> Dict:
"""Преобразование в словарь для экспорта"""
return {
"Наименование": self.name,
"Полное наименование": self.full_name or "",
"ОРНЗ": self.ornz,
"ИНН": self.inn,
"КПП": self.kpp or "",
"ОГРН": self.ogrn or "",
"Регион": self.region,
"Статус": self.status or "",
"Адрес": self.address or "",
"Телефон": self.phone or "",
"Email": self.email or "",
"Сайт": self.website or "",
"Руководитель": self.director or "",
"Дата регистрации": (
self.registration_date.strftime("%d.%m.%Y")
if self.registration_date
else ""
),
"Дата начала членства": (
self.membership_start_date.strftime("%d.%m.%Y")
if self.membership_start_date
else ""
),
"Дата окончания членства": (
self.membership_end_date.strftime("%d.%m.%Y")
if self.membership_end_date
else ""
),
"Количество аудиторов": self.auditors_count or "",
"Сертификаты": ", ".join(self.certificates),
"Сети": ", ".join(self.networks),
"URL источника": self.source_url or "",
"Дата сбора данных": self.parsed_at.strftime("%d.%m.%Y %H:%M:%S"),
}