Some issues I noticed with latest commits

[milezzz]

So glad to see @Prefinem working on this again =]

1. I pulled the latest commit hoping it would fix torrent updates in loader.js but it seems the same behaviour to me. I don't see seed/leech being updated. Nothing shows up in loader-out.log, or loader.error.log with debug = true in index.js. When I run scraper.js I see in log:

Total Torrents: 785004
Torrents without Tracker: 781921
Torrents not in Search: 361

2. Another issue I found was that I kept noticing 0 byte files being scraped, or at least I thought they were 0byte! When I checked mysql I found they had a proper length field BUT they had a huge files field which was always truncated abruptly due to max length on TEXT type field in mysql. You can maybe see what I mean here: https://pastebin.com/STBrvykL

I did this to test (ALTER TABLE torrents MODIFY `files` MEDIUMTEXT;) and it seems no more broken file lists but this opens us up to 16mb max record sizes on this field. Might be worth it to truncate these instead in parser.js but I'm not sure the best approach :/

3. The last issue I noticed is I still cannot seem to filter out the invalid character � from scrapes, I've tried 0xFFFD, � - and a few others...but I'll keep trying. I haven't tested but I think this should work on all unexpected characters: dataStr.replace(/[\u{0080}-\u{FFFF}]/gu,"");

[Raxvis]

Did you do a pm2 restart?

I added a ticket to truncate file lists since I am working on updating the system as a whole.

Where does that invalid character come from?

[milezzz]

Did you do a pm2 restart?

Yes. pm2 restart all

Where does that invalid character come from?

Unsure! It appears exactly like that in the database. �

Could it be that his server blocks access to UDP or the tracker? would you get some error if so? it seems 90% of his torrents have a status of NULL in tacker updated in mysql

[Raxvis]

Maybe.

Can you set config.debug = true and see if there is any output?

Add a line to src/tracker.js on line 58 of console.log(records.length); and see if you are getting records

[milezzz]

Just tested with portquiz to make sure port 6881 isn't blocked:

$ curl portquiz.net:6881
Port 6881 test successful!
Your IP: xx.xx.xx.xx

$ wget -qO- portquiz.net:6881
Port 6881 test successful!
Your IP: xx.xx.xx.xx

Can you set config.debug = true and see if there is any output?

Yes it is.

Add a line to src/tracker.js on line 58 of console.log(records.length); and see if you are getting records

I see in the log for scraper now:
scraper > 75

[Raxvis]

The logs show scraper > 75?

[milezzz]

Ahhaaaa. So made some changes to see if maybe I am being rate limited or something:

frequency: 2,
host: 'udp://tracker.coppersurfer.tk:6969/announce',
limit: 55,

And now it works! I am seeing records like this now:

scraper > 004787ab3e6071b96b71fc18667156441ffd1b76 - 136:8

What a relief!

Q: Is there a way to just run the SE:LE updater without scraping more torrents? I tried only running loader.js but that doesn't seem to work.

Q: Is there a way to just run the SE:LE updater without scraping more torrents? I tried only running loader.js but that doesn't seem to work

I just changed the bind ip address in config/index.js from 0.0.0.0 to 1.1.1.1 to disable getting new torrents

[milezzz]

I just changed the bind ip address in config/index.js from 0.0.0.0 to 1.1.1.1 to disable getting new torrents

tyvm =]

[Raxvis]

I am splitting up the tracker and scraper in the current working branch which will allow this

im also now struggling with the tracker.

Total Torrents: 1983006                  
Torrents without Tracker: 208771

After checking it looks like the scraper seems to be scraping 75 every 1 minute give or take. Could it be that the tracker.js is querying MySQL for torrents with seeders info less than X age and this query is taking time and the tracker has to wait for this before it can update? @Prefinem

i can confirm this by checking the process list of mysql. and also running the command myself.

Showing rows 0 - 74 (75 total, Query took 96.7467 seconds.)
select * from torrents where trackerUpdated is null limit 75

maybe i need to tweak mysql lol

[Raxvis]

This is something I just ran into with reworking things. I think I am going to have to drop MySQL completely since it’s too slow.

…

On Thu, Feb 7, 2019 at 18:02 ash121121 ***@***.***> wrote: i can confirm this by checking the process list of mysql. and also running the command myself. Showing rows 0 - 74 (75 total, Query took 96.7467 seconds.) select * from torrents where trackerUpdated is null limit 75 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#52 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACnvZhjB5Y-nsVP0S8PC4h6Wj-b0a1knks5vLL6NgaJpZM4amw9r> .