Cleanup

CHANGELOG

@@ -0,0 +1,11 @@
+## [0.6] - 2022-12-05
+### Removed
+- Removed invalid docker code
+
+### Fixed
+- Moved doc files out of package
+- Cleaned up installation disorder
+
+### Added
+- `MANIFEST.in` for two files which are required by the package
+- Submodule `quick` for a quick download of the data as it appears on the site, which is what most folks want anyway

MANIFEST.in

@@ -0,0 +1,2 @@
+include fragalysis-api/fragalysis_api/xcglobalscripts/config.ini
+include fragalysis_api/xcimporter/non_ligs.json

README.md

@@ -4,50 +4,44 @@
 
 [![Language grade: Python](https://img.shields.io/lgtm/grade/python/g/xchem/fragalysis-api.svg?logo=lgtm&logoWidth=18)](https://lgtm.com/projects/g/xchem/fragalysis-api/context:python)
 
-Documentation: https://xchem.github.io/fragalysis-api/
+Documentation: [https://xchem.github.io/fragalysis-api/](https://xchem.github.io/fragalysis-api/)
 
-This api aims to allow any user to upload pdb files from the pdb or that they have created themselves,
+This API aims to allow any user to upload pdb files from the pdb or that they have created themselves,
 and analyse the ligand binding using the fragalysis webpage (https://fragalysis.diamond.ac.uk).
+Namely:
+
+* Upload data, i.e. import into Fragalysis -> **xcimporter** functionality (_vide infra_)
+* Download data, i.e. export from Fragalysis -> **xcexporter** functionality (_vide infra_)
+
+The full list of Fragalysis API endpoints can be found at [fragalysis.diamond.ac.uk/api](https://fragalysis.diamond.ac.uk/api/).
 
 ## Installation
 
-Starting out by initialising an environment and activating it.
-Clone the repository and cd to the relevant directory.
-Install rdkit via conda, and the other dependencies via the setup.py file:
+In order to manipulate the data for upload (e.g. aligning crystal maps), 
+some additional dependencies are required,
+namely [xchem/gemmi_pandda](https://github.com/xchem/gemmi_pandda) and [xchem/pandda_gemmi](https://github.com/xchem/pandda_gemmi).
 
 ```bash
-conda create -n fragalysis_env anaconda -y
-conda activate fragalysis_env
-conda install -c conda-forge rdkit -y
-
-# Install our-bespoke version of gemmi # Required
-git clone https://github.com/xchem/gemmi_pandda.git
-cd gemmi_pandda/
-pip install -U --force-reinstall .
-cd ..
-
-# Also Required
-git clone https://github.com/xchem/pandda_gemmi.git
-cd pandda_gemmi/
-pip install -e .
-cd ..
-
-# Finally install the api
-git clone "https://github.com/xchem/fragalysis-api.git"
-cd fragalysis-api/
-pip install -e .
-cd ..
-```
+# Install our-bespoke version of gemmi # Required for upload
+# Do note this is a drop-in replacement for gemmi, so will interfere with other packages that use gemmi.
+pip install -U --force-reinstall git+https://github.com/xchem/gemmi_pandda.git
 
-You can check if it has installed using: `conda list`
+# Also Required for upload
+pip install -e git+https://github.com/xchem/pandda_gemmi.git
+```
+The API itself can be installed via pypi or from the Git repo:
+```bash
+pip install fragalysis-api
+# or (for a later version if available)
+pip install git+https://github.com/xchem/fragalysis-api.git
+```
 
 ### How to use API
 
-1. Set up environment
-2. Download PDB files and query the PDB for structures of the same protein bound to the same or different ligands
-3. Submit PDB files - you will be given a query ID
-4. Push your files into fragalysis and view them online :construction:
-5. Analyse the binding of ligands to your target protein!
+1. Download PDB files and query the PDB for structures of the same protein bound to the same or different ligands
+2. Submit PDB files - you will be given a query ID
+3. Push your files into fragalysis and view them online :construction:
+4. Analyse the binding of ligands to your target protein!
 
 Other functionalities that are available:
 
@@ -59,9 +53,40 @@ Other functionalities that are available:
 
 ## Usage in Python
 
+### Export from Fragalysis
+> TL;DR: This is a Python package for accurately interacting with the Fragalysis API, 
+> if you simply want to download everything of a target,
+> please see the [quick download notes](quick_download.md).
+
+Download relevant data off Fragalysis. 
+
+
+See also [extractor notes](extractor.md).
+
+```python
+import fragalysis_api
+import os
+import pandas as pd
+
+hit_data: pd.DataFrame = fragalysis_api.xcextracter(target_name='NUDT5A')
+```
+The columns in the dataframe are:
+
+* `id`
+* `prot_id`: unique integer id per crystal, e.g. `protein_code`:`NUDT5A-x0114_1` and `protein_code`:`NUDT5A-x0114_2` have different `prot_id` but same `cmpd_id`.
+* `protein_code`, a string form of the above (e.g. `NUDT5A-x0114_1`)
+* `cmpd_id`, an integer, unique per ligand, but not per crystal
+* `lig_id`, the chemical compounent name, generally `LIG`
+* `chain_id`, the chain id of the ligand
+* `smiles` and `sdf_info`, the SMILE-String and the SDF block of the ligand
+* `molecule_protein`, the bound PDB file address
+* `mw`, `logp`, `tpsa`, `ha`, `hacc`, `hdon`, `rots`, `rings`, `velec`: inferred chemical properties of the ligand
+
+### Import into Fragalysis
 To prepare input data-files using python you api can import the `xcimporer` or `import_single_file` functions and then provide the necessary values to the functions.
+ See also [importer notes](importer.md).
 
-e.g
+Example:
 
 ```python
 from fragalysis_api import xcimporter, import_single_file
@@ -209,8 +234,8 @@ A description of the arguments are as follows:
 
 ### Enforced rules :scroll:
 
-- The pdb file shall not be greater than 5mb.
-- The pdb filename shall not contain non English language ascii characters
+- The pdb file shall not be greater than 5MB.
+- The pdb filename shall only contain non ASCII characters (e.g. Unicode characters such as é or ü).
   and shall be between 4 and 20 characters in length.
 - Each pdb file for alignment shall contain the same number of chains.
 - All pdb files to be aligned must be in the same directory.

fragalysis_api/__init__.py

@@ -1,15 +1,28 @@
-from .xcglobalscripts.set_config import ConfigSetup
-
-from .xcimporter.validate import Validate, ValidatePDB
-from .xcimporter.conversion_pdb_mol import set_up, convert_small_AA_chains, copy_extra_files
-from .xcimporter.align import Align
-from .xcimporter.xc_utils import to_fragalysis_dir
-from .xcimporter.sites import Sites, contextualize_crystal_ligands
-from .xcimporter.xcimporter import xcimporter
-from .xcimporter.single_import import import_single_file
+from .xcglobalscripts.set_config import ConfigSetup  # this requires `config.ini` in the working directory
+
+# import into Fragalysis requires gemmi
+try:
+    from .xcimporter.validate import Validate, ValidatePDB
+    from .xcimporter.conversion_pdb_mol import set_up, convert_small_AA_chains, copy_extra_files
+    from .xcimporter.xc_utils import to_fragalysis_dir
+    from .xcimporter.sites import Sites, contextualize_crystal_ligands
+    from .xcimporter.xcimporter import xcimporter
+    from .xcimporter.single_import import import_single_file
+    from .xcimporter.align import Align
+except ImportError:
+    import warnings
+    warnings.warn("The xcimporter functionality is unavailable without Gemmi", category=ImportWarning)
+
+# extract from Fragalysis does not require gemmi
 from .xcextracter.getdata import GetTargetsData, GetMoleculesData, GetPdbData, GetMolgroupData
 from .xcextracter.frag_web_live import can_connect
 from .xcextracter.xcextracter import xcextracter
 
 from .xcanalyser.graphcreator import GraphRequest, xcgraphcreator
 from .xcanalyser.xcanalyser import xcanalyser
+
+# this is not part of canonical Fragalysis wrapper, but a quick way to get the data.
+from .quick import QuickDownloader
+
+
+

fragalysis_api/quick.py

@@ -0,0 +1,185 @@
+"""
+A quick download functionality for the impatient.
+"""
+
+import warnings
+from typing import Optional, Tuple, List, Dict, Any
+
+import io
+import os
+import pandas as pd
+import requests
+import zipfile
+from rdkit.Chem import PandasTools
+
+
+class QuickDownloader:
+    """
+    This is simply a polished interface to the `api/download_structures` endpoint,
+    which is the same as the download button on the Fragalysis website.
+    Namely, it lacks the extended functionality of the `xcexporter` module.
+    Instantiating it with the target name will download the zip file of these and can be interactive with via different
+    methods.
+
+    :cvar: fragalysis_api_url: The URL of the Fragalysis API.
+    :cvar: api_data: The default options for the API endpoint. Can be also overridden in the constructor.
+    :ivar: target_name: The name of the target as on the main Fragalysis page, e.g. 'Mpro', case sensitive
+    :ivar: zf: The zip file object. see https://docs.python.org/3/library/zipfile.html
+
+    The contents of the ``zipfile.ZipFile`` object stored in the attribute ``.zf`` can be written to disk
+    with ``.write_all()``.
+
+    The contents of a file within the zipfile can be accessed by subscripting the ``QuickDownloader`` instance,
+    with a string that is contained in the filename, e.g. ``quick['metadata']``, will return the metadata.csv file
+    content without having to waste time with filepaths.
+
+    .. code-block:: python
+        from fragalysis_api import QuickDownloader
+        import pandas as pd
+        from typing import List
+        print(f'Default settings are: {QuickDownloader.api_data}')
+
+        # Check if the target name is right
+        target_names: List[str] = QuickDownloader.retrieve_target_names()
+        target_name='Mpro'
+        assert target_name in target_names, f'Target named "{target_name}" not found in the list of targets'
+
+        # Download the data
+        quick = QuickDownloader(target_name=target_name)
+        quick.write_all(directory='downloads')
+        hits: pd.DataFrame = quick.to_pandas(star_dummy=True)
+
+        # Not all files have the reference pdb block, so if it does not the template is returned:
+        reference_pdbblock: str = quick.reference_pdbblock
+
+    The class method ``QuickDownloader.retrieve_target_data`` will download all the metadata for the targets.
+    while ``QuickDownloader.retrieve_target_names`` will return their names.
+    """
+    fragalysis_api_url = 'fragalysis.diamond.ac.uk/api/download_structures/'
+    api_data = {
+        'proteins': '',
+        'event_info': False,
+        'sigmaa_info': False,
+        'diff_info': False,
+        'trans_matrix_info': False,
+        'NAN': False,
+        'mtz_info': False,
+        'cif_info': False,
+        'NAN2': False,
+        'map_info': False,
+        'single_sdf_file': True,
+        'sdf_info': False,
+        'pdb_info': False,
+        'bound_info': True,
+        'metadata_info': True,
+        'smiles_info': True,
+        'static_link': False,
+        'file_url': ''}
+
+    def __init__(self, target_name: str, **options):
+        """
+        Given a target download the zip file and store it in ``self.zf``.
+
+        :param target_name: The name of the target as on the main Fragalysis page, e.g. 'Mpro', case sensitive
+        :param options: Any of the options for the endpoint, e.g. ``event_info=True``, cf. ``cls.api_data``
+        """
+        self.target_name = target_name
+        self.api_data = {options.get(k, v) for k, v in self.api_data.items()}
+        url_response: requests.Response = requests.post(f'https://{self.fragalysis_api_url}',
+                                                        json={'target_name': self.target_name, **self.api_data})
+        url_response.raise_for_status()
+        self.file_url: str = url_response.json()['file_url']
+        response: requests.Response = requests.get(f"https://{self.fragalysis_api_url}?file_url={self.file_url}",
+                                                   allow_redirects=True)
+        response.raise_for_status()
+        self.zf = zipfile.ZipFile(io.BytesIO(response.content), "r")
+
+    def __getitem__(self, item: str) -> str:
+        """
+        Subscript via filename, e.g. ``quick['metadata']`` will return the metadata.csv file content.
+
+        :param item: Part of the filename whose contents will be returned
+        :return: The contents of the file whose name contains ``item``
+        """
+        for fileinfo in self.zf.infolist():
+            if item in fileinfo.filename:
+                return self.zf.read(fileinfo.filename).decode('utf8')
+        else:
+            raise KeyError(f'No file with {item} in the name found.')
+
+    def __iter__(self) -> Tuple[str, str]:
+        for fileinfo in self.zf.infolist():
+            yield fileinfo.filename, self.zf.read(fileinfo.filename).decode('utf8')
+
+    def __len__(self):
+        """
+        :return: The number of molecules-protein PDBs in the zip file.
+        """
+        return sum(['aligned/' in info.filename for info in self.zf.infolist()])
+
+    def write_all(self, directory: Optional[str] = None):
+        """
+        Writes all the files within the zip file to disk in ``directory``.
+        """
+        if directory is None:
+            directory = self.target_name
+        if not os.path.exists(directory):
+            os.makedirs(directory)
+        for fileinfo in self.zf.infolist():
+            if os.path.split(fileinfo.filename)[0] != '':
+                os.makedirs(os.path.join(directory, os.path.split(fileinfo.filename)[0]), exist_ok=True)
+            with open(os.path.join(directory, fileinfo.filename), 'w') as f:
+                f.write(self.zf.read(fileinfo.filename).decode('utf8'))
+
+    def to_pandas(self, star_dummy=True) -> pd.DataFrame:
+        """
+        Combine the metadata (``self['metadata]``) with sdf block (``self['combined.sdf']``),
+        into a single pandas DataFrame.
+        Peculiarly, Fragalysis stores dummy atoms as `Xe` instead of `*` in older SMILES, which is the standard.
+        """
+        # make a combined table
+        # Fragalysis does not give attributes in the sdf entries. This is instead stored in metadata.csv.
+        sdf_block = self['combined.sdf']
+        df = PandasTools.LoadSDF(io.StringIO(sdf_block)).set_index('ID')
+        try:
+            metadata_block = self['metadata.csv'].replace('Xe', '*') if star_dummy else self['metadata.csv']
+            df = pd.concat([df,
+                            pd.read_csv(io.StringIO(metadata_block), index_col=0).set_index('crystal_name')
+                            ], axis=1)
+        except KeyError:
+            warnings.warn('No metadata.csv found (legacy data). Returning only the sdf file.')
+        return df
+
+    @property
+    def reference_pdbblock(self) -> str:
+        """
+        Not all files have the reference pdb block, so if it does not the template is returned.
+
+        :return: The reference PDB for the target.
+        """
+        try:
+            return self['reference']
+        except KeyError:
+            first_response: requests.Response = requests.get(f'https://{self.fragalysis_api_url}/api/targets/')
+            first_response.raise_for_status()
+            template_url = first_response.json()['results'][0]['template_protein']
+            # /media/pdbs/ path:
+            second_response: requests.Response = requests.get(f'https://{self.fragalysis_api_url}/{template_url}')
+            second_response.raise_for_status()
+            return second_response.text
+
+    @classmethod
+    def retrieve_target_data(cls) -> List[Dict[str, Any]]:
+        """
+        :return: A list of all the target metadata available on the Fragalysis API.
+        """
+        response: requests.Response = requests.get(f'https://{cls.fragalysis_api_url}/api/targets/')
+        response.raise_for_status()
+        return response.json()['results']
+
+    @classmethod
+    def retrieve_target_names(cls) -> List[str]:
+        """
+        :return: A list of all the target names available on the Fragalysis API.
+        """
+        return [target['title'] for target in cls.retrieve_target_data()]

fragalysis_api/xcglobalscripts/set_config.py

@@ -4,6 +4,7 @@
 
 def ConfigSetup():
     # use config parser to get settings from config.ini
+    # todo Do not read package files this way, use importlib/pkg_resources
     settings_file = os.path.join(os.path.dirname(__file__), "config.ini")
     settings = configparser.ConfigParser()
     settings._interpolation = configparser.ExtendedInterpolation()