Venho memory chat mixing up basic legal contract logic

[mannfredcom]

Describe the bug
I uploaded a very basic (made up) contract to Mind2 documents into its own memory context. A copy of the dummy contract used is available at https://casa.beige.org/jolla/test_mind2_contract_us.pdf

I asked a couple very simple questions about the contract just to confirm that the ingest had been successful, and I am pleased to report that that this worked as expected.

However, when I asked one level more complex of a question (which should still be answerable based on all information available in the PDF), namely "The contract was signed 50 days ago, but no monies have been paid yet. How much does which company owe whom?" the answer was mathematically entirely flawed (which is not surprising for small LLMs, to be fair) but what's worse is that the answer incorrectly claimed that the service provider is the entity owing money! Here is the Venho chat answer in its entirety:

Based on the provided contract excerpts, here's my analysis:

Since no payments have been made yet, and the due date for the first payment is not specified in the contract excerpts, we can infer that the monthly fee of $10,000 is not due until at least 15 days after the signing date.

To calculate the late charge, we need to determine how many days are left before the payment is due. Assuming the contract was signed 50 days ago, and considering the 15-day grace period for late payments:

Days remaining: 50 - 15 = 35 days

Since 15 days have already passed (since the contract was signed), we need to calculate the number of days left before the payment is due:

Remaining days after 15-day grace period: 35 - 15 = 20 days

According to the contract, a late charge of 1.5% per month will be incurred until paid. To calculate this:

Total late charge for 20 days:
    1.5% x 20 = 0.03 x 20 = $0.60
However, we need to check the contract's payment terms to see if a late fee is applied to the monthly fee. The excerpt doesn't explicitly state this, but it does mention "Payments not received within fifteen (15) days of the due date shall incur a late charge..."

Given the ambiguity, I'll stick with the calculation:

Total late charge for 20 days: $0.60

However, since no payment has been made yet and only 35 days have passed, it's possible that the Client is still within their grace period. In this case, there would be no late charges.

In summary:

The Service Provider owes the Client a monthly fee of $10,000.
There may not be any late charges applied if the payment has been made on or before the 15th day after signing.

Please note that without more specific information about when the first payment is due and whether it's been made, this answer is an educated guess."

(Note in particular the first line of the summary)

To Reproduce
Steps to reproduce the behavior:

1. Upload the pdf referenced above to Mind2 into its own context
2. Ask the question "The contract was signed 50 days ago, but no monies have been paid yet. How much does which company owe whom?"
3. See error (to the extent a non-deterministic/stochastic system is likely to reproduce problematic behaviour)

Expected behavior
I would have expected the LLM to parse that the monthly payments are due immediately. So Client would owe the Service Provider 20K after 50 days (plus around 250 dollars of interest if my math is correct, but admittedly some ambiguity could be argued re from which day the late interest should be accruing). But most importantly (given that maths are not the strong suite of LLMs) I would have expected even the limited model to at least be able to keep track of which company is paying and which company is receiving money (and under no circumstances mix up the two).

Screenshots

Environment(please complete the following information):

• OS: Sailfish 5.0.0.62
• Browser Firefox 135.0.1 (64-bit)
• Version Venho 0.3.1, llama 3.2:3b

Additional context
Add any other context about the problem here.