ada/CourseWork_IRFM
1
0
Fork 0
mirror of https://github.com/ada-dmitry/CourseWork_IRFM.git synced 2026-09-24 01:10:18 +00:00
Code Issues Packages Projects Releases Wiki Activity

Add web crawler for legal documents

Browse Source 
- Implemented a new crawler module to extract text and links from legal documents.
- Added a loader module to handle HTML downloads with error handling and retries.
- Created a main script to initiate the crawling process on a specified URL.
- Defined functions for normalizing URLs, extracting links, checking terminal pages, and extracting text content.
- Updated project metadata with dependencies and versioning in pyproject.toml and uv.lock.
This commit is contained in:
Dmitry
2026-04-16 18:44:27 +03:00
commit 502f48a279
9 changed files with 231 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
pyproject.toml
+9
View File
@@ -0,0 +1,9 @@
[project]
name = "coursework-irfm"
version = "0.1.0"
description = "Add your description here"
readme = "README.md"
requires-python = ">=3.13"
dependencies = [
"lxml>=6.0.4",
]