mirror of
https://github.com/ada-dmitry/CourseWork_IRFM.git
synced 2026-09-24 01:10:18 +00:00
Dmitry
502f48a279
- Implemented a new crawler module to extract text and links from legal documents. - Added a loader module to handle HTML downloads with error handling and retries. - Created a main script to initiate the crawling process on a specified URL. - Defined functions for normalizing URLs, extracting links, checking terminal pages, and extracting text content. - Updated project metadata with dependencies and versioning in pyproject.toml and uv.lock.
9 lines
193 B
Python
9 lines
193 B
Python
from modules.crawler import crawl_document
|
|
|
|
def main():
|
|
url = "https://www.consultant.ru/document/cons_doc_LAW_10699/"
|
|
print(crawl_document(url))
|
|
|
|
if __name__ == "__main__":
|
|
main()
|