refactor: enable deep username scan for user_scan modules#355
refactor: enable deep username scan for user_scan modules#355kaifcodec wants to merge 3 commits into
Conversation
|
@VamatoHD You can add deep scan for other modules as well. |
ed1bd23 to
b8695ae
Compare
β¦ypass PoW for airliners.py
b8695ae to
379ce08
Compare
|
@kaifcodec After some testing, there is a slight problem: some extras, such as bios, are multi-line and mess up the visual, such as [β] Anilist (___): Found
βββ id: ___
βββ about: ~~~
β§ ____ β§
~!
βοΈβοΈβοΈ
on AniList since __.__.__
youtube(https://www.youtube.com/watch?v=___)
youtube(https://www.youtube.com/watch?v=___)
!~How should they be handled? |
|
Yeah, that is definitely an issue we need to catch. If a module returns data containing newlines ( Basically, we want subsequent lines to align perfectly directly below the first line of that key's block. This will keep multi-line details visually confined inside an "invisible square" layout so it doesn't break the CLI tree structure. |
user_scan/social/):anilist,bluesky,mastodon,openstreetmap: Scrapes GraphQL, NEXT_DATA, and HTML to pull detailed bio, location, followers count, and avatars.35photo,pinterest,snapchat: Extracts photographer stats, redux state JSON, and snapcode SVG profiles.user_scan/dev/):github: Direct profile scraping of names, bios, location, public emails, and social links using schemas (itemprop="image") to avoid sponsor avatar conflicts.gitlab,huggingface,dockerhub,cratesio: Switched to public APIs to capture user IDs, joined dates, active states, and avatar URLs.user_scan/music/):soundcloud,bandcamp: Extracts embedded hydration states and data-blobs containing full user details, followers, and play statistics.discogs,bandlab,freesound,lastfm,audiojungle,beatstars: Scrapes AJAX stats, public REST APIs, and profile headers to extract comprehensive artist registries and music metrics.TODO:
Refactor and mgrate rest of the modules
Test all the modules if any data extraction left behind or got ignored
Update docs