Fail early if RPKI fetch and parse don't output assignments#146
Fail early if RPKI fetch and parse don't output assignments#146jurraca wants to merge 2 commits into
Conversation
|
Concept ACK. I opened #147 as a possible follow-up to this (exit codes, stderr reasons, per-RIR counts), they should work independently. |
fjahr
left a comment
There was a problem hiding this comment.
I am ok to add this but it would be much more interesting to figure out what the errors really are that we are running into and try to catch these before the actual run, like in our pre-checks ideally.
For a OOM scenario I think the most likely place for this to be caught should be here: https://github.com/asmap/kartograf/pull/146/files#diff-5d8704566493265b100a183244c9429c1424306cb6f7d14ab32e76b19018e4acR173 If a batch doesn't return a result then it was likely killed. Maybe this could be addressed here as well.
But for the empty files case I don't think it makes sense that this could be an OOM case, there is no memory hungry operation running at that point. It's just rpki-client downloading stuff. So I am still wondering what the real cause there is.
Can you open a tracking issue for these cases so we can collect them in one place and try to coordinate reproducers?
| or (path.name == ".roa"))] | ||
|
|
||
| print(f"{len(files)} raw RKPI ROA files found.") | ||
| if len(files) == 0: |
There was a problem hiding this comment.
It doesn't really make sense to print the line above in this case, so I would move this above it.
There was a problem hiding this comment.
There is also a typo in the log above that you can fix while you are at it: "RKPI"
Looks like @jorisstrakeljahn is looking into this in #147 already |
RPKI can sometimes fail to write its cache data to disk, or return no valid assignments after validation. In both cases, we want to exit early, since RPKI is the basis upon which we try to build a more complete map.
For the parsing test, I modified the incompletes test as well to specifically test the incompletes output with at least one valid entry.