Credential Info being asked during the job running

[esdotzed]

The following credential was being asked during job running, and it interrupted the job which is supposed to run itself:

Username for 'https://bitbucket.org': Password for 'https://bitbucket.org':

I can tell this is not a standard Github link but not sure what's the best thing to bypass this automatically, so I am asking here first before digging too much into it.

Thanks much for such a great project!

[leonmoonen]

BitBucket is a GitHub alternative, and some of the repositories linked in CVEs are hosted on BitBucket. The fact that it asks for a valid username/password likely means that the repository changed from public to private, and we can no longer collect it. It is a bug that such a private repo breaks the collection of other CVEs, and I was under the impression that we removed such URLs. Do you see a repository name so we can investigate?

More generally, we're working towards a major revision of CVEfixes that uses NIST's new APIs. The plan is to have this new release available by the end of the year.

[esdotzed]

Thanks for your reply. Actually the collection process wasn't broken by it, it just means that it's not 100% unattended, it would ask for those credentials at some point, and after that's typed in, it continue to run, my apology for any confusions. I am re-running the job and I will reply back which repo it is if I see it again :)

Another thing, my previous run was killed several times due to lack of memory, and after it was all done finally, I noticed the method_change table was empty (all other tables seemed to be fine). I am guessing it might be because of the program was stopped too many times and that caused some data to be missing. I will report back after my new fresh run is done, which I hope will be done in one run since I have now allocated more than enough memory (144GB) :)

It's exiting to know that you guys are working on a major revision and we can't wait to try it! Best luck to it! 🤞
Thanks again for the reply and again this is amazing work!

[esdotzed]

FYI, I encountered this again so I am just posting the console output for your references:

10/12/2023 12:54:32 CVEfixes DEBUG Extracting commits for https://bitbucket.org/atlassian/atlasboard with 72 worker(s) looking for the following hashes: 10/12/2023 12:54:32 CVEfixes DEBUG https://bitbucket.org/atlassian/atlasboard/commit/9c03df09f09399e2601010466e8ba3a28236eb9c 10/12/2023 12:54:32 git.cmd DEBUG Popen(['git', 'clone', '-v', '--', 'https://bitbucket.org/atlassian/atlasboard', '/tmp/tmp44227br_/atlasboard'], cwd=/home/CVEfixes, universal_newlines=True, shell=None, istream=None) Username for 'https://bitbucket.org/':

Thank you!

[esdotzed]

hello @leonmoonen any updates about whether the major revision is coming soon? Thanks a lot!

[leonmoonen]

Hello @esdotzed,

I am currently on medical leave while recovering from being run over by a bus (I know, a literal bus factor). As a result, I am working at a reduced capacity, and our work on CVEfixes has seen some delays. We expect to deliver the major revision in August/September of this year. In the meantime, we have collected a new database that will be made available on Zenodo soon. Your understanding and patience during this time are greatly appreciated 🙏🏼

cheers,
Leon

[esdotzed]

Thank you @leonmoonen for the update!! I am sorry to hear about what had happened to you and I hope you will be fully recovered soon!! It's good to know that the team is still working on this project and we can expect for a revision soon! Thank you!

[CoffeeGeeker]

Hello @leonmoonen,

First of all, thank you for all your contributions to the CVEfix project, and I hope you're recovering well after such a serious accident! I'm sending my best wishes for your continued recovery and strength during this time.

Additionally, I wanted to inquire about the version number increase—perhaps we could plan to include this update in the next release?

Take care, and I’m looking forward to hearing from you!

[CoffeeGeeker]

Hello @leonmoonen,
May I ask how the progress on the new version is going? Is there any area where assistance is needed? If help is required, we can work on it together

[leonmoonen]

Hi @CoffeeGeeker,
Thanks for the wishes 🙏 The new version is shaping up nicely, although things are going a bit slower than we had estimated. We appreciate your offer for assistance, but the new version is actively being worked on as part of a master project, which makes contributing at this moment more difficult. Once the next version is released, we would absolutely welcome further improvements and extensions.

I'm afraid I did not completely understand your question about the version number increase, can you elaborate?

NB: I have started a new collection process (with the old version), and will update the Zenodo repository (see README) once that is finished, so that the latest CVEs will be available in the dataset.

[CoffeeGeeker]

Hi @leonmoonen,
Thank you very much for your hard work! Please take care of your health. Regarding the version number I mentioned, I am referring to the last version associated with the CVE vulnerability. Specifically, this is the version in which the commit that patches the CVE is included (or the most recent prior version, as some commits may not have a corresponding tag, and we need to find the previous tagged version). We can locate this version number using the relevant tags.
Below is a code example, where hashes refers to the current commit's hash:

from pydriller import Repository
from git import Repo
from tqdm import tqdm

# Function to retrieve all commits from the repository
def get_all_commits(repo_path):
   commits = []
   try:
       for commit in Repository(repo_path).traverse_commits():
           commits.append(commit.hash)
   except Exception as e:
       tqdm.write(f"Failed to retrieve commits for repository at {repo_path}: {str(e)}")
   return commits

# Function to retrieve all tags from the local Git repository
def get_local_tags(repo_path):
   tag_dict = {}
   try:
       repo = Repo(repo_path)
       tag_dict = {tag.commit.hexsha: tag.name for tag in repo.tags}
   except Exception as e:
       tqdm.write(f"Failed to retrieve tags for repository at {repo_path}: {str(e)}")
       raise e  # Raise an exception when tag retrieval fails for subsequent repository cleanup
   return tag_dict

# Function to map commit hashes to the closest previous version tag
def find_previous_version_for_hashes(hashes, all_commits, tag_dict):
   commit_to_version = {}

   for hash_value in hashes:
       try:
           if hash_value in all_commits:
               index = all_commits.index(hash_value)

               # Search backwards from the current commit until a commit with a version tag is found
               previous_version = None
               for i in range(index, -1, -1):
                   commit = all_commits[i]
                   if commit in tag_dict:
                       previous_version = tag_dict[commit]
                       break
               commit_to_version[hash_value] = previous_version
           else:
               commit_to_version[hash_value] = None
       except Exception as e:
           tqdm.write(f"Error processing hash {hash_value}: {str(e)}")
           commit_to_version[hash_value] = None  # Or use a specific error flag like "ERROR"

   return commit_to_version

Additionally, I’ve added an error skipping mechanism in my code, as some commits may not have tags.

Besides, I noticed your plan to add a license, as mentioned here. I was wondering if you're considering covering the entire project directly, or if the plan is to include only the software's own declared license? For example, in projects like FFmpeg, which call other open-source libraries, it can be challenging to include everything.

I’ve tried tools like LicenseFinder before, and it's very useful in analyzing each file. However, the final exported license information can be quite extensive. I'd love to know how you're planning to handle this. Looking forward to your response!

[esdotzed]

@leonmoonen Long time no see, hope this message find you well. May I kindly ask if there is going to be a major update coming up? Is there any plans for uploading up-to-date dataset for downloading since the latest data is from last year. Thank you so much again for this amazing project.

[leonmoonen]

Hi, I've tried to initiate a new collection process, but NIST has blocked access to the v1.1 JSON feeds that we rely on (e.g., https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2002.json.zip). I will review the necessary changes to support their v2.0 JSON scheme and, if feasible, update our "CVEfixes 1.0" collector. My collaborator on the major update of the CVEfixes collector also thinks that we're not too far from being able to release the new version.