Idea

[MrCyjaneK]

https://github.com/Miaosi001/JW-Library-macOS/blob/3f39b4a386ba00f52432607a745d7f0b9dcb9db1/JWLibrary/Utility/JWPubManager.swift#L45-51

Here you can get latest catalog version: https://app.jw-cdn.org/catalogs/publications/v4/manifest.json

and with that version go to https://app.jw-cdn.org/catalogs/publications/v4/ + version + /catalog.db.gz

Also, I've seen JWPUB manager.. have you figured out how to read them? (I'm not into swift, and I don't even own an 🍎 device)..

It took me couple of days with no results to get content out of the jwpub. MrCyjaneK/jwapi#1

[darioragusa]

Hi, thanks for the help.

I have not yet managed to find a way to read JWPub files. Even if I change the words in the db they don't change in the app. I noticed that if I change a word it does not change in the text but only in the text search function. I have no clue how it works.

[MrCyjaneK]

ah well. So I'm back to reverse engineering it again..

[darioragusa]

Me too. I started again this morning. This time I noticed some things that I didn't noticed before.

Now I arrived here:

UPDATE
I think I could manage to extract the text. But only the text... I can't find any info regarding punctuation and size.

[MrCyjaneK]

OH! If you can get the text that would he a huiegrnfdjksxfnwliekfncl (random letters of joy) help for me!

[MrCyjaneK]

I'm soo happy that somebody figured out how to use it :D <3

[darioragusa]

Well yes, but I keep believing that it is not the correct way to do it

[MrCyjaneK]

I'm actually out of luck and I have no idea what to do now.., So I just hope that you will figure out some method for it..

[darioragusa]

I think everything is in the 'Content' field of the 'Document' table but it is somehow encrypted and we cannot read it. Then two options remain:

1. Extract the text from the search table and invent up uppercase and punctuation;
2. Scraping of the text from the site.

And I would say that the first is not really the best. Even the second is not the best but I think it's the only way we have.

[MrCyjaneK]

How would you do the 1st thing? I believe that it is in fact stored in that table, and it is the correct way to extract the content, and with the leftover bytes we could possibly figure which stand for images/punctation/etc

[darioragusa]

Starting from the 'TextUnitIndices' column I take all the words that start with '80' (HEX). Then from the column 'PositionalList' the first word is the one that starts with '80', then the word with '81', etc... Reached 'FF' starts again with '00 81 ', '01 81', etc... After 'FF 81' I think there is '00 82' (but I have not yet tried). In the 'PositionalListIndex' column the first value (eg '85') indicates that the word is present 5 times so I could go to subtract that value until it reaches '80' and do not control that word for the current document (maybe I remove the '80' at the end of the cycle). Finally, in the 'TextUnitIndices' column I remove '80' and for all the rows decrease of 1 the next value (eg '81' -> '80', '85' -> '84', etc...). Then I start again the loop for the next document.

[MrCyjaneK]

Ooookay! I'll try to write a parser for that! Thanks for help :D

[darioragusa]

You're welcome. Let me know if you find a way to get the punctuation

[MrCyjaneK]

I will!

[MrCyjaneK]

words like god which probably are on every page, except for the table of content, have length of TextUnitIndices row equal to 18 (in publication with 18 topics + Table Of Content, which I believe is different than the rest.

I'm actually curious what those lss, lsr, sqr etc.. are, maybe formatting?

[MrCyjaneK]

yay! I think we have a bit of it here 🥳