My system is AMD 3955W, 128 GB RAM, RTX A4000 16GB, Windows 10 WSL2, Ubuntu 20.04.
When I tried to predict a 156 a.a sequence, I got an error message "jaxlib.xla_extension.XlaRuntimeError: INTERNAL: RET_CHECK failure (external/org_tensorflow/tensorflow/compiler/xla/service/gpu/gpu_compiler.cc:641) dnn != nullptr".
Below is the log. Can someone help me to troubleshoot it?
(alphafold) whelanlab@CHEM-AG007D:~/alphafold$ exp/run_local.sh R1.fasta
/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/haiku/_src/data_structures.py:37: FutureWarning: jax.tree_structure is deprecated, and will be removed in a future release. Use jax.tree_util.tree_structure instead.
PyTreeDef = type(jax.tree_structure(None))
/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/absl/flags/_validators.py:203: UserWarning: Flag --output_dir has a non-None defau lt value; therefore, mark_flag_as_required will pass even if flag is not specified in the command line!
warnings.warn(
/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/absl/flags/_validators.py:203: UserWarning: Flag --model_names has a non-None defa ult value; therefore, mark_flag_as_required will pass even if flag is not specified in the command line!
warnings.warn(
/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/absl/flags/_validators.py:203: UserWarning: Flag --data_dir has a non-None default value; therefore, mark_flag_as_required will pass even if flag is not specified in the command line!
warnings.warn(
/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/absl/flags/_validators.py:203: UserWarning: Flag --preset has a non-None default v alue; therefore, mark_flag_as_required will pass even if flag is not specified in the command line!
warnings.warn(
/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/absl/flags/_validators.py:203: UserWarning: Flag --uniref90_database_path has a no n-None default value; therefore, mark_flag_as_required will pass even if flag is not specified in the command line!
warnings.warn(
/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/absl/flags/_validators.py:203: UserWarning: Flag --mgnify_database_path has a non- None default value; therefore, mark_flag_as_required will pass even if flag is not specified in the command line!
warnings.warn(
/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/absl/flags/_validators.py:203: UserWarning: Flag --uniclust30_database_path has a non-None default value; therefore, mark_flag_as_required will pass even if flag is not specified in the command line!
warnings.warn(
/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/absl/flags/_validators.py:203: UserWarning: Flag --bfd_database_path has a non-Non e default value; therefore, mark_flag_as_required will pass even if flag is not specified in the command line!
warnings.warn(
/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/absl/flags/_validators.py:203: UserWarning: Flag --pdb70_database_path has a non-N one default value; therefore, mark_flag_as_required will pass even if flag is not specified in the command line!
warnings.warn(
/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/absl/flags/_validators.py:203: UserWarning: Flag --template_mmcif_dir has a non-No ne default value; therefore, mark_flag_as_required will pass even if flag is not specified in the command line!
warnings.warn(
/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/absl/flags/_validators.py:203: UserWarning: Flag --max_template_date has a non-Non e default value; therefore, mark_flag_as_required will pass even if flag is not specified in the command line!
warnings.warn(
/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/absl/flags/validators.py:203: UserWarning: Flag --obsolete_pdbs_path has a non-No ne default value; therefore, mark_flag_as_required will pass even if flag is not specified in the command line!
warnings.warn(
I0320 18:45:41.543587 139960428058432 templates.py:880] Using precomputed obsolete pdbs /mnt/d/alphafold/database/pdb_mmcif/obsolete.dat.
I0320 18:45:43.169499 139960428058432 xla_bridge.py:166] Remote TPU is not linked into jax; skipping remote TPU.
I0320 18:45:43.169672 139960428058432 xla_bridge.py:413] Unable to initialize backend 'tpu_driver': Could not initialize backend 'tpu_driver'
I0320 18:45:43.511162 139960428058432 xla_bridge.py:413] Unable to initialize backend 'rocm': NOT_FOUND: Could not find registered platform with name: "rocm" . Available platform names are: Interpreter Host CUDA
I0320 18:45:43.511674 139960428058432 xla_bridge.py:413] Unable to initialize backend 'tpu': module 'jaxlib.xla_extension' has no attribute 'get_tpu_client'
I0320 18:45:43.511768 139960428058432 xla_bridge.py:413] Unable to initialize backend 'plugin': xla_extension has no attributes named get_plugin_device_clien t. Compile TensorFlow with //tensorflow/compiler/xla/python:enable_plugin_device set to true (defaults to false) to enable this.
I0320 18:45:51.545218 139960428058432 run_alphafold.py:293] Have 5 models: ['model_1', 'model_2', 'model_3', 'model_4', 'model_5']
I0320 18:45:51.545373 139960428058432 run_alphafold.py:306] Using random seed 4477881255564421494 for the data pipeline
I0320 18:45:51.553825 139960428058432 pipeline.py:130] query uniref90
I0320 18:45:51.554164 139960428058432 jackhmmer.py:119] Launching subprocess "jackhmmer -o /dev/null -A /tmp/tmpydz_0bbz/output.sto --noali --F1 0.0005 --F2 5e-05 --F3 5e-07 --incE 0.0001 -E 0.0001 --cpu 8 -N 1 R1.fasta /mnt/d/alphafold/database/uniref90/uniref90.fasta"
I0320 18:45:51.564941 139960428058432 utils.py:36] Started Jackhmmer (uniref90.fasta) query
I0320 19:45:55.761877 139960428058432 utils.py:40] Finished Jackhmmer (uniref90.fasta) query in 3604.197 seconds
I0320 19:45:55.808635 139960428058432 pipeline.py:141] query mgnify
I0320 19:45:55.809039 139960428058432 jackhmmer.py:119] Launching subprocess "jackhmmer -o /dev/null -A /tmp/tmp5t31w7bf/output.sto --noali --F1 0.0005 --F2 5e-05 --F3 5e-07 --incE 0.0001 -E 0.0001 --cpu 8 -N 1 R1.fasta /mnt/d/alphafold/database/mgnify/mgy_clusters.fa"
I0320 19:45:55.817553 139960428058432 utils.py:36] Started Jackhmmer (mgy_clusters.fa) query
I0320 20:44:21.103180 139960428058432 utils.py:40] Finished Jackhmmer (mgy_clusters.fa) query in 3505.285 seconds
I0320 20:44:21.107893 139960428058432 pipeline.py:153] query mgnify
I0320 20:44:21.108175 139960428058432 hhblits.py:128] Launching subprocess "hhblits -i R1.fasta -cpu 4 -oa3m /tmp/tmpasn145w/output.a3m -o /dev/null -n 3 -e 0.001 -maxseq 1000000 -realign_max 100000 -maxfilt 100000 -min_prefilter_hits 1000 -d /mnt/d/alphafold/database/bfd/bfd_metaclust_clu_complete_id30_c9 0_final_seq.sorted_opt -d /mnt/d/alphafold/database/uniclust30/uniclust30_2018_08/uniclust30_2018_08"
I0320 20:44:21.117413 139960428058432 utils.py:36] Started HHblits query
I0321 02:22:28.651200 139960428058432 utils.py:40] Finished HHblits query in 20287.534 seconds
I0321 02:22:28.785565 139960428058432 hhsearch.py:76] Launching subprocess "hhsearch -i /tmp/tmp5es22b99/query.a3m -o /tmp/tmp5es22b99/output.hhr -maxseq 1000000 -d /mnt/d/alphafold/database/pdb70/pdb70"
I0321 02:22:28.794022 139960428058432 utils.py:36] Started HHsearch query
I0321 02:54:02.204876 139960428058432 utils.py:40] Finished HHsearch query in 1893.411 seconds
I0321 02:54:02.476578 139960428058432 templates.py:892] Searching for template for: None
W0321 02:54:02.476918 139960428058432 templates.py:139] Template structure not in release dates dict: 1ivz
I0321 02:54:02.477041 139960428058432 templates.py:753] Reading PDB entry from /mnt/d/alphafold/database/pdb_mmcif/mmcif_files/1ivz.cif. Query: ATVPFMVPFT LNFTITNLQYEEDMRHPGSRKFNATERELQGLLKPLFRNSSLEYLYSGCRLASLRPEKDSSATAVDAICTHRPDPEDLGLDRERLYWELSNLTNGIQELGPYTLDRNSLYVNGFTHRSSMPTTSTPGTSTVDVGTSGTPSSSPSPT, templa te: SSSSQHFNLNFTITNLPYSQDIAQPSTTKYQQTKRSIENALNQLFRNSSIKSYFSDCQVLAFRSVSNNNNHTGVDSLCNFSPLARRVDRVAIYEEFLRMTHNGTQLLNFTLDRKSVFVDSGP
I0321 02:54:04.145149 139960428058432 templates.py:278] Found an exact template match 1ivz_A.
W0321 02:54:04.168012 139960428058432 templates.py:139] Template structure not in release dates dict: 2e7v
I0321 02:54:04.168263 139960428058432 templates.py:753] Reading PDB entry from /mnt/d/alphafold/database/pdb_mmcif/mmcif_files/2e7v.cif. Query: ATVPFMVPFT LNFTITNLQYEEDMRHPGSRKFNATERELQGLLKPLFRNSSLEYLYSGCRLASLRPEKDSSATAVDAICTHRPDPEDLGLDRERLYWELSNLTNGIQELGPYTLDRNSLYVNGFTHRSSMPTTSTPGTSTVDVGTSGTPSSSPSPT, templa te: KAYFYHSSFQILNVEYTEALNSPATHEYRTLSERIEAMITDEFRGSSLKSEFIRTHVVKLRKEGTGVVADVVMKFRSSKRNNRKVMKTRIQSVLRRLSSSGNLEIAPSNEI
I0321 02:54:04.227076 139960428058432 templates.py:278] Found an exact template match 2e7v_A.
W0321 02:54:04.230510 139960428058432 templates.py:139] Template structure not in release dates dict: 6bsb
I0321 02:54:04.230618 139960428058432 templates.py:753] Reading PDB entry from /mnt/d/alphafold/database/pdb_mmcif/mmcif_files/6bsb.cif. Query: ATVPFMVPFT LNFTITNLQYEEDMRHPGSRKFNATERELQGLLKPLFRNSSLEYLYSGCRLASLRPEKDSSATAVDAICTHRPDPEDLGLDRERLYWELSNLTNGIQELGPYTLDRNSLYVNGFTHRSSMPTTSTPGTSTVDVGTSGTPSSSPSPT, templa te: FFFLSFHISNLQFNSSLEDPSTDYYQELQRDISEMFLQIYKQGGFLGLSNIKFRPG
I0321 02:54:04.350127 139960428058432 templates.py:278] Found an exact template match 6bsb_A.
W0321 02:54:04.353108 139960428058432 templates.py:139] Template structure not in release dates dict: 6bsc
I0321 02:54:04.353233 139960428058432 templates.py:753] Reading PDB entry from /mnt/d/alphafold/database/pdb_mmcif/mmcif_files/6bsc.cif. Query: ATVPFMVPFT LNFTITNLQYEEDMRHPGSRKFNATERELQGLLKPLFRNSSLEYLYSGCRLASLRPEKDSSATAVDAICTHRPDPEDLGLDRERLYWELSNLTNGIQELGPYTLDRNSLYVNGFTHRSSMPTTSTPGTSTVDVGTSGTPSSSPSPT, templa te: FFFLSFHISNLQFNSSLEDPSTDYYQELQRDISEMFLQIYKQGGFLGLSNIKFRPG
I0321 02:54:04.616760 139960428058432 templates.py:278] Found an exact template match 6bsc_A.
W0321 02:54:04.619688 139960428058432 templates.py:139] Template structure not in release dates dict: 2acm
I0321 02:54:04.619805 139960428058432 templates.py:753] Reading PDB entry from /mnt/d/alphafold/database/pdb_mmcif/mmcif_files/2acm.cif. Query: ATVPFMVPFT LNFTITNLQYEEDMRHPGSRKFNATERELQGLLKPLFRNSSLEYLYSGCRLASLRPEKDSSATAVDAICTHRPDPEDLGLDRERLYWELSNLTNGIQELGPYTLDRNSLYVNGFTHRSSMPTTSTPGTSTVDVGTSGTPSSSPSPT, templa te: SFFFLSFHISNLQFNSSLEDPSTDYYQELQRDISEMFLQIYKQGGFLGLSNIKFRPG
I0321 02:54:05.585400 139960428058432 templates.py:278] Found an exact template match 2acm_A.
W0321 02:54:05.598183 139960428058432 templates.py:139] Template structure not in release dates dict: 5yl9
I0321 02:54:05.598412 139960428058432 templates.py:753] Reading PDB entry from /mnt/d/alphafold/database/pdb_mmcif/mmcif_files/5yl9.cif. Query: ATVPFMVPFT LNFTITNLQYEEDMRHPGSRKFNATERELQGLLKPLFRNSSLEYLYSGCRLASLRPEKDSSATAVDAICTHRPDPEDLGLDRERLYWELSNLTNGIQELGPYTLDRNSLYVNGFTHRSSMPTTSTPGTSTVDVGTSGTPSSSPSPT, templa te: VVEQYNQTILNLTSEISTLENKSAELNYTVQKLQTLIDNI
I0321 02:54:05.712731 139960428058432 templates.py:278] Found an exact template match 5yl9_B.
W0321 02:54:05.715018 139960428058432 templates.py:139] Template structure not in release dates dict: 1gvp
I0321 02:54:05.715114 139960428058432 templates.py:753] Reading PDB entry from /mnt/d/alphafold/database/pdb_mmcif/mmcif_files/1gvp.cif. Query: ATVPFMVPFT LNFTITNLQYEEDMRHPGSRKFNATERELQGLLKPLFRNSSLEYLYSGCRLASLRPEKDSSATAVDAICTHRPDPEDLGLDRERLYWELSNLTNGIQELGPYTLDRNSLYVNGFTHRSSMPTTSTPGTSTVDVGTSGTPSSSPSPT, templa te: APGLYTVHLSSFKVGQFGS
I0321 02:54:05.777992 139960428058432 templates.py:278] Found an exact template match 1gvp_A.
W0321 02:54:05.780884 139960428058432 templates.py:139] Template structure not in release dates dict: 6s8s
I0321 02:54:05.780977 139960428058432 templates.py:753] Reading PDB entry from /mnt/d/alphafold/database/pdb_mmcif/mmcif_files/6s8s.cif. Query: ATVPFMVPFT LNFTITNLQYEEDMRHPGSRKFNATERELQGLLKPLFRNSSLEYLYSGCRLASLRPEKDSSATAVDAICTHRPDPEDLGLDRERLYWELSNLTNGIQELGPYTLDRNSLYVNGFTHRSSMPTTSTPGTSTVDVGTSGTPSSSPSPT, templa te: FDFEGNLALFDKAAVFEEIDTY
I0321 02:54:06.046435 139960428058432 templates.py:278] Found an exact template match 6s8s_B.
W0321 02:54:06.049705 139960428058432 templates.py:139] Template structure not in release dates dict: 1ae3
I0321 02:54:06.049842 139960428058432 templates.py:753] Reading PDB entry from /mnt/d/alphafold/database/pdb_mmcif/mmcif_files/1ae3.cif. Query: ATVPFMVPFT LNFTITNLQYEEDMRHPGSRKFNATERELQGLLKPLFRNSSLEYLYSGCRLASLRPEKDSSATAVDAICTHRPDPEDLGLDRERLYWELSNLTNGIQELGPYTLDRNSLYVNGFTHRSSMPTTSTPGTSTVDVGTSGTPSSSPSPT, templa te: APGLYTVHLSSFKVGQFG
I0321 02:54:06.119893 139960428058432 templates.py:278] Found an exact template match 1ae3_A.
W0321 02:54:06.122766 139960428058432 templates.py:139] Template structure not in release dates dict: 4v23
I0321 02:54:06.122855 139960428058432 templates.py:753] Reading PDB entry from /mnt/d/alphafold/database/pdb_mmcif/mmcif_files/4v23.cif. Query: ATVPFMVPFT LNFTITNLQYEEDMRHPGSRKFNATERELQGLLKPLFRNSSLEYLYSGCRLASLRPEKDSSATAVDAICTHRPDPEDLGLDRERLYWELSNLTNGIQELGPYTLDRNSLYVNGFTHRSSMPTTSTPGTSTVDVGTSGTPSSSPSPT, templa te: AFKYIKPQSQFIVDLGAY
I0321 02:54:06.287062 139960428058432 templates.py:278] Found an exact template match 4v23_A.
I0321 02:54:06.426746 139960428058432 run_alphafold.py:185] Running model model_1
WARNING:tensorflow:From /home/whelanlab/alphafold/alphafold/model/tf/input_pipeline.py:151: calling map_fn (from tensorflow.python.ops.map_fn) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Use fn_output_signature instead
W0321 02:54:06.770174 139960428058432 deprecation.py:528] From /home/whelanlab/alphafold/alphafold/model/tf/input_pipeline.py:151: calling map_fn (from te nsorflow.python.ops.map_fn) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Use fn_output_signature instead
2023-03-21 02:54:08.766418: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:39] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_G ROWTH environment variable is set. Original config value was 0.
I0321 02:54:09.424343 139960428058432 model.py:132] Running predict with shape(feat) = {'aatype': (4, 156), 'residue_index': (4, 156), 'seq_length': (4,), 'template_aatype': (4, 4, 156), 'template_all_atom_masks': (4, 4, 156, 37), 'template_all_atom_positions': (4, 4, 156, 37, 3), 'template_sum_probs': (4, 4, 1), 'is_distillation': (4,), 'seq_mask': (4, 156), 'msa_mask': (4, 508, 156), 'msa_row_mask': (4, 508), 'random_crop_to_size_seed': (4, 2), 'template_m ask': (4, 4), 'template_pseudo_beta': (4, 4, 156, 3), 'template_pseudo_beta_mask': (4, 4, 156), 'atom14_atom_exists': (4, 156, 14), 'residx_atom14_to_atom 37': (4, 156, 14), 'residx_atom37_to_atom14': (4, 156, 37), 'atom37_atom_exists': (4, 156, 37), 'extra_msa': (4, 5120, 156), 'extra_msa_mask': (4, 5120, 1 56), 'extra_msa_row_mask': (4, 5120), 'bert_mask': (4, 508, 156), 'true_msa': (4, 508, 156), 'extra_has_deletion': (4, 5120, 156), 'extra_deletion_value': (4, 5120, 156), 'msa_feat': (4, 508, 156, 49), 'target_feat': (4, 156, 22)}
2023-03-21 02:54:09.995058: E external/org_tensorflow/tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:417] Loaded runtime CuDNN library: 8.2.1 bu t source was compiled with: 8.6.0. CuDNN library needs to have matching major version and equal or higher minor version. If using a binary install, upgra de your CuDNN library. If building from sources, make sure the library loaded at runtime is compatible with the version specified during compile configur ation.
2023-03-21 02:54:09.997237: E external/org_tensorflow/tensorflow/compiler/xla/status_macros.cc:54] INTERNAL: RET_CHECK failure (external/org_tensorflow/te nsorflow/compiler/xla/service/gpu/gpu_compiler.cc:641) dnn != nullptr
*** Begin stack trace ***
_PyObject_MakeTpCall
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyObject_Call
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyObject_Call
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyObject_Call
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyObject_Call
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
PyObject_Call
_PyObject_Call
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyObject_Call
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyObject_Call
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyObject_Call
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyEval_EvalCodeWithName
PyEval_EvalCodeEx
PyEval_EvalCode
PyRun_SimpleFileExFlags
Py_RunMain
Py_BytesMain
__libc_start_main
*** End stack trace ***
Traceback (most recent call last):
File "/home/whelanlab/alphafold/run_alphafold.py", line 338, in
app.run(main)
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/absl/app.py", line 312, in run
_run_main(main, args)
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/absl/app.py", line 258, in _run_main
sys.exit(main(argv))
File "/home/whelanlab/alphafold/run_alphafold.py", line 310, in main
predict_structure(
File "/home/whelanlab/alphafold/run_alphafold.py", line 192, in predict_structure
prediction_result = model_runner.predict(processed_feature_dict)
File "/home/whelanlab/alphafold/alphafold/model/model.py", line 134, in predict
result = self.apply(self.params, jax.random.PRNGKey(0), feat)
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/random.py", line 136, in PRNGKey
key = prng.seed_with_impl(impl, seed)
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/prng.py", line 269, in seed_with_impl
return random_seed(seed, impl=impl)
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/prng.py", line 581, in random_seed
return random_seed_p.bind(seeds_arr, impl=impl)
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/core.py", line 360, in bind
return self.bind_with_trace(find_top_trace(args), args, params)
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/core.py", line 363, in bind_with_trace
out = trace.process_primitive(self, map(trace.full_raise, args), params)
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/core.py", line 807, in process_primitive
return primitive.impl(*tracers, **params)
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/prng.py", line 593, in random_seed_impl
base_arr = random_seed_impl_base(seeds, impl=impl)
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/prng.py", line 598, in random_seed_impl_base
return seed(seeds)
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/prng.py", line 833, in threefry_seed
lax.shift_right_logical(seed, lax_internal._const(seed, 32)))
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/lax/lax.py", line 513, in shift_right_logical
return shift_right_logical_p.bind(x, y)
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/core.py", line 360, in bind
return self.bind_with_trace(find_top_trace(args), args, params)
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/core.py", line 363, in bind_with_trace
out = trace.process_primitive(self, map(trace.full_raise, args), params)
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/core.py", line 807, in process_primitive
return primitive.impl(*tracers, **params)
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/dispatch.py", line 122, in apply_primitive
compiled_fun = xla_primitive_callable(prim, *unsafe_map(arg_spec, args),
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/util.py", line 254, in wrapper
return cached(config._trace_context(), *args, **kwargs)
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/util.py", line 247, in cached
return f(*args, **kwargs)
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/dispatch.py", line 201, in xla_primitive_callable
compiled = _xla_callable_uncached(lu.wrap_init(prim_fun), device, None,
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/dispatch.py", line 357, in _xla_callable_uncached
return computation.compile(_allow_propagation_to_outputs=allow_prop).unsafe_call
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/interpreters/pxla.py", line 3373, in compile
executable = self._compile_unloaded(
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/interpreters/pxla.py", line 3344, in _compile_unloaded
return UnloadedMeshExecutable.from_hlo(
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/interpreters/pxla.py", line 3626, in from_hlo
xla_executable = dispatch.compile_or_get_cached(
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/dispatch.py", line 1091, in compile_or_get_cached
return backend_compile(backend, serialized_computation, compile_options,
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/profiler.py", line 314, in wrapper
return func(*args, **kwargs)
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/dispatch.py", line 1036, in backend_compile
return backend.compile(built_c, compile_options=options)
jaxlib.xla_extension.XlaRuntimeError: INTERNAL: RET_CHECK failure (external/org_tensorflow/tensorflow/compiler/xla/service/gpu/gpu_compiler.cc:641) dnn != nullptr
My system is AMD 3955W, 128 GB RAM, RTX A4000 16GB, Windows 10 WSL2, Ubuntu 20.04.
When I tried to predict a 156 a.a sequence, I got an error message "jaxlib.xla_extension.XlaRuntimeError: INTERNAL: RET_CHECK failure (external/org_tensorflow/tensorflow/compiler/xla/service/gpu/gpu_compiler.cc:641) dnn != nullptr".
Below is the log. Can someone help me to troubleshoot it?
(alphafold) whelanlab@CHEM-AG007D:~/alphafold$ exp/run_local.sh R1.fasta
/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/haiku/_src/data_structures.py:37: FutureWarning: jax.tree_structure is deprecated, and will be removed in a future release. Use jax.tree_util.tree_structure instead.
PyTreeDef = type(jax.tree_structure(None))
/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/absl/flags/_validators.py:203: UserWarning: Flag --output_dir has a non-None defau lt value; therefore, mark_flag_as_required will pass even if flag is not specified in the command line!
warnings.warn(
/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/absl/flags/_validators.py:203: UserWarning: Flag --model_names has a non-None defa ult value; therefore, mark_flag_as_required will pass even if flag is not specified in the command line!
warnings.warn(
/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/absl/flags/_validators.py:203: UserWarning: Flag --data_dir has a non-None default value; therefore, mark_flag_as_required will pass even if flag is not specified in the command line!
warnings.warn(
/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/absl/flags/_validators.py:203: UserWarning: Flag --preset has a non-None default v alue; therefore, mark_flag_as_required will pass even if flag is not specified in the command line!
warnings.warn(
/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/absl/flags/_validators.py:203: UserWarning: Flag --uniref90_database_path has a no n-None default value; therefore, mark_flag_as_required will pass even if flag is not specified in the command line!
warnings.warn(
/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/absl/flags/_validators.py:203: UserWarning: Flag --mgnify_database_path has a non- None default value; therefore, mark_flag_as_required will pass even if flag is not specified in the command line!
warnings.warn(
/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/absl/flags/_validators.py:203: UserWarning: Flag --uniclust30_database_path has a non-None default value; therefore, mark_flag_as_required will pass even if flag is not specified in the command line!
warnings.warn(
/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/absl/flags/_validators.py:203: UserWarning: Flag --bfd_database_path has a non-Non e default value; therefore, mark_flag_as_required will pass even if flag is not specified in the command line!
warnings.warn(
/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/absl/flags/_validators.py:203: UserWarning: Flag --pdb70_database_path has a non-N one default value; therefore, mark_flag_as_required will pass even if flag is not specified in the command line!
warnings.warn(
/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/absl/flags/_validators.py:203: UserWarning: Flag --template_mmcif_dir has a non-No ne default value; therefore, mark_flag_as_required will pass even if flag is not specified in the command line!
warnings.warn(
/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/absl/flags/_validators.py:203: UserWarning: Flag --max_template_date has a non-Non e default value; therefore, mark_flag_as_required will pass even if flag is not specified in the command line!
warnings.warn(
/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/absl/flags/validators.py:203: UserWarning: Flag --obsolete_pdbs_path has a non-No ne default value; therefore, mark_flag_as_required will pass even if flag is not specified in the command line!
warnings.warn(
I0320 18:45:41.543587 139960428058432 templates.py:880] Using precomputed obsolete pdbs /mnt/d/alphafold/database/pdb_mmcif/obsolete.dat.
I0320 18:45:43.169499 139960428058432 xla_bridge.py:166] Remote TPU is not linked into jax; skipping remote TPU.
I0320 18:45:43.169672 139960428058432 xla_bridge.py:413] Unable to initialize backend 'tpu_driver': Could not initialize backend 'tpu_driver'
I0320 18:45:43.511162 139960428058432 xla_bridge.py:413] Unable to initialize backend 'rocm': NOT_FOUND: Could not find registered platform with name: "rocm" . Available platform names are: Interpreter Host CUDA
I0320 18:45:43.511674 139960428058432 xla_bridge.py:413] Unable to initialize backend 'tpu': module 'jaxlib.xla_extension' has no attribute 'get_tpu_client'
I0320 18:45:43.511768 139960428058432 xla_bridge.py:413] Unable to initialize backend 'plugin': xla_extension has no attributes named get_plugin_device_clien t. Compile TensorFlow with //tensorflow/compiler/xla/python:enable_plugin_device set to true (defaults to false) to enable this.
I0320 18:45:51.545218 139960428058432 run_alphafold.py:293] Have 5 models: ['model_1', 'model_2', 'model_3', 'model_4', 'model_5']
I0320 18:45:51.545373 139960428058432 run_alphafold.py:306] Using random seed 4477881255564421494 for the data pipeline
I0320 18:45:51.553825 139960428058432 pipeline.py:130] query uniref90
I0320 18:45:51.554164 139960428058432 jackhmmer.py:119] Launching subprocess "jackhmmer -o /dev/null -A /tmp/tmpydz_0bbz/output.sto --noali --F1 0.0005 --F2 5e-05 --F3 5e-07 --incE 0.0001 -E 0.0001 --cpu 8 -N 1 R1.fasta /mnt/d/alphafold/database/uniref90/uniref90.fasta"
I0320 18:45:51.564941 139960428058432 utils.py:36] Started Jackhmmer (uniref90.fasta) query
I0320 19:45:55.761877 139960428058432 utils.py:40] Finished Jackhmmer (uniref90.fasta) query in 3604.197 seconds
I0320 19:45:55.808635 139960428058432 pipeline.py:141] query mgnify
I0320 19:45:55.809039 139960428058432 jackhmmer.py:119] Launching subprocess "jackhmmer -o /dev/null -A /tmp/tmp5t31w7bf/output.sto --noali --F1 0.0005 --F2 5e-05 --F3 5e-07 --incE 0.0001 -E 0.0001 --cpu 8 -N 1 R1.fasta /mnt/d/alphafold/database/mgnify/mgy_clusters.fa"
I0320 19:45:55.817553 139960428058432 utils.py:36] Started Jackhmmer (mgy_clusters.fa) query
I0320 20:44:21.103180 139960428058432 utils.py:40] Finished Jackhmmer (mgy_clusters.fa) query in 3505.285 seconds
I0320 20:44:21.107893 139960428058432 pipeline.py:153] query mgnify
I0320 20:44:21.108175 139960428058432 hhblits.py:128] Launching subprocess "hhblits -i R1.fasta -cpu 4 -oa3m /tmp/tmpasn145w/output.a3m -o /dev/null -n 3 -e 0.001 -maxseq 1000000 -realign_max 100000 -maxfilt 100000 -min_prefilter_hits 1000 -d /mnt/d/alphafold/database/bfd/bfd_metaclust_clu_complete_id30_c9 0_final_seq.sorted_opt -d /mnt/d/alphafold/database/uniclust30/uniclust30_2018_08/uniclust30_2018_08"
I0320 20:44:21.117413 139960428058432 utils.py:36] Started HHblits query
I0321 02:22:28.651200 139960428058432 utils.py:40] Finished HHblits query in 20287.534 seconds
I0321 02:22:28.785565 139960428058432 hhsearch.py:76] Launching subprocess "hhsearch -i /tmp/tmp5es22b99/query.a3m -o /tmp/tmp5es22b99/output.hhr -maxseq 1000000 -d /mnt/d/alphafold/database/pdb70/pdb70"
I0321 02:22:28.794022 139960428058432 utils.py:36] Started HHsearch query
I0321 02:54:02.204876 139960428058432 utils.py:40] Finished HHsearch query in 1893.411 seconds
I0321 02:54:02.476578 139960428058432 templates.py:892] Searching for template for: None
W0321 02:54:02.476918 139960428058432 templates.py:139] Template structure not in release dates dict: 1ivz
I0321 02:54:02.477041 139960428058432 templates.py:753] Reading PDB entry from /mnt/d/alphafold/database/pdb_mmcif/mmcif_files/1ivz.cif. Query: ATVPFMVPFT LNFTITNLQYEEDMRHPGSRKFNATERELQGLLKPLFRNSSLEYLYSGCRLASLRPEKDSSATAVDAICTHRPDPEDLGLDRERLYWELSNLTNGIQELGPYTLDRNSLYVNGFTHRSSMPTTSTPGTSTVDVGTSGTPSSSPSPT, templa te: SSSSQHFNLNFTITNLPYSQDIAQPSTTKYQQTKRSIENALNQLFRNSSIKSYFSDCQVLAFRSVSNNNNHTGVDSLCNFSPLARRVDRVAIYEEFLRMTHNGTQLLNFTLDRKSVFVDSGP
I0321 02:54:04.145149 139960428058432 templates.py:278] Found an exact template match 1ivz_A.
W0321 02:54:04.168012 139960428058432 templates.py:139] Template structure not in release dates dict: 2e7v
I0321 02:54:04.168263 139960428058432 templates.py:753] Reading PDB entry from /mnt/d/alphafold/database/pdb_mmcif/mmcif_files/2e7v.cif. Query: ATVPFMVPFT LNFTITNLQYEEDMRHPGSRKFNATERELQGLLKPLFRNSSLEYLYSGCRLASLRPEKDSSATAVDAICTHRPDPEDLGLDRERLYWELSNLTNGIQELGPYTLDRNSLYVNGFTHRSSMPTTSTPGTSTVDVGTSGTPSSSPSPT, templa te: KAYFYHSSFQILNVEYTEALNSPATHEYRTLSERIEAMITDEFRGSSLKSEFIRTHVVKLRKEGTGVVADVVMKFRSSKRNNRKVMKTRIQSVLRRLSSSGNLEIAPSNEI
I0321 02:54:04.227076 139960428058432 templates.py:278] Found an exact template match 2e7v_A.
W0321 02:54:04.230510 139960428058432 templates.py:139] Template structure not in release dates dict: 6bsb
I0321 02:54:04.230618 139960428058432 templates.py:753] Reading PDB entry from /mnt/d/alphafold/database/pdb_mmcif/mmcif_files/6bsb.cif. Query: ATVPFMVPFT LNFTITNLQYEEDMRHPGSRKFNATERELQGLLKPLFRNSSLEYLYSGCRLASLRPEKDSSATAVDAICTHRPDPEDLGLDRERLYWELSNLTNGIQELGPYTLDRNSLYVNGFTHRSSMPTTSTPGTSTVDVGTSGTPSSSPSPT, templa te: FFFLSFHISNLQFNSSLEDPSTDYYQELQRDISEMFLQIYKQGGFLGLSNIKFRPG
I0321 02:54:04.350127 139960428058432 templates.py:278] Found an exact template match 6bsb_A.
W0321 02:54:04.353108 139960428058432 templates.py:139] Template structure not in release dates dict: 6bsc
I0321 02:54:04.353233 139960428058432 templates.py:753] Reading PDB entry from /mnt/d/alphafold/database/pdb_mmcif/mmcif_files/6bsc.cif. Query: ATVPFMVPFT LNFTITNLQYEEDMRHPGSRKFNATERELQGLLKPLFRNSSLEYLYSGCRLASLRPEKDSSATAVDAICTHRPDPEDLGLDRERLYWELSNLTNGIQELGPYTLDRNSLYVNGFTHRSSMPTTSTPGTSTVDVGTSGTPSSSPSPT, templa te: FFFLSFHISNLQFNSSLEDPSTDYYQELQRDISEMFLQIYKQGGFLGLSNIKFRPG
I0321 02:54:04.616760 139960428058432 templates.py:278] Found an exact template match 6bsc_A.
W0321 02:54:04.619688 139960428058432 templates.py:139] Template structure not in release dates dict: 2acm
I0321 02:54:04.619805 139960428058432 templates.py:753] Reading PDB entry from /mnt/d/alphafold/database/pdb_mmcif/mmcif_files/2acm.cif. Query: ATVPFMVPFT LNFTITNLQYEEDMRHPGSRKFNATERELQGLLKPLFRNSSLEYLYSGCRLASLRPEKDSSATAVDAICTHRPDPEDLGLDRERLYWELSNLTNGIQELGPYTLDRNSLYVNGFTHRSSMPTTSTPGTSTVDVGTSGTPSSSPSPT, templa te: SFFFLSFHISNLQFNSSLEDPSTDYYQELQRDISEMFLQIYKQGGFLGLSNIKFRPG
I0321 02:54:05.585400 139960428058432 templates.py:278] Found an exact template match 2acm_A.
W0321 02:54:05.598183 139960428058432 templates.py:139] Template structure not in release dates dict: 5yl9
I0321 02:54:05.598412 139960428058432 templates.py:753] Reading PDB entry from /mnt/d/alphafold/database/pdb_mmcif/mmcif_files/5yl9.cif. Query: ATVPFMVPFT LNFTITNLQYEEDMRHPGSRKFNATERELQGLLKPLFRNSSLEYLYSGCRLASLRPEKDSSATAVDAICTHRPDPEDLGLDRERLYWELSNLTNGIQELGPYTLDRNSLYVNGFTHRSSMPTTSTPGTSTVDVGTSGTPSSSPSPT, templa te: VVEQYNQTILNLTSEISTLENKSAELNYTVQKLQTLIDNI
I0321 02:54:05.712731 139960428058432 templates.py:278] Found an exact template match 5yl9_B.
W0321 02:54:05.715018 139960428058432 templates.py:139] Template structure not in release dates dict: 1gvp
I0321 02:54:05.715114 139960428058432 templates.py:753] Reading PDB entry from /mnt/d/alphafold/database/pdb_mmcif/mmcif_files/1gvp.cif. Query: ATVPFMVPFT LNFTITNLQYEEDMRHPGSRKFNATERELQGLLKPLFRNSSLEYLYSGCRLASLRPEKDSSATAVDAICTHRPDPEDLGLDRERLYWELSNLTNGIQELGPYTLDRNSLYVNGFTHRSSMPTTSTPGTSTVDVGTSGTPSSSPSPT, templa te: APGLYTVHLSSFKVGQFGS
I0321 02:54:05.777992 139960428058432 templates.py:278] Found an exact template match 1gvp_A.
W0321 02:54:05.780884 139960428058432 templates.py:139] Template structure not in release dates dict: 6s8s
I0321 02:54:05.780977 139960428058432 templates.py:753] Reading PDB entry from /mnt/d/alphafold/database/pdb_mmcif/mmcif_files/6s8s.cif. Query: ATVPFMVPFT LNFTITNLQYEEDMRHPGSRKFNATERELQGLLKPLFRNSSLEYLYSGCRLASLRPEKDSSATAVDAICTHRPDPEDLGLDRERLYWELSNLTNGIQELGPYTLDRNSLYVNGFTHRSSMPTTSTPGTSTVDVGTSGTPSSSPSPT, templa te: FDFEGNLALFDKAAVFEEIDTY
I0321 02:54:06.046435 139960428058432 templates.py:278] Found an exact template match 6s8s_B.
W0321 02:54:06.049705 139960428058432 templates.py:139] Template structure not in release dates dict: 1ae3
I0321 02:54:06.049842 139960428058432 templates.py:753] Reading PDB entry from /mnt/d/alphafold/database/pdb_mmcif/mmcif_files/1ae3.cif. Query: ATVPFMVPFT LNFTITNLQYEEDMRHPGSRKFNATERELQGLLKPLFRNSSLEYLYSGCRLASLRPEKDSSATAVDAICTHRPDPEDLGLDRERLYWELSNLTNGIQELGPYTLDRNSLYVNGFTHRSSMPTTSTPGTSTVDVGTSGTPSSSPSPT, templa te: APGLYTVHLSSFKVGQFG
I0321 02:54:06.119893 139960428058432 templates.py:278] Found an exact template match 1ae3_A.
W0321 02:54:06.122766 139960428058432 templates.py:139] Template structure not in release dates dict: 4v23
I0321 02:54:06.122855 139960428058432 templates.py:753] Reading PDB entry from /mnt/d/alphafold/database/pdb_mmcif/mmcif_files/4v23.cif. Query: ATVPFMVPFT LNFTITNLQYEEDMRHPGSRKFNATERELQGLLKPLFRNSSLEYLYSGCRLASLRPEKDSSATAVDAICTHRPDPEDLGLDRERLYWELSNLTNGIQELGPYTLDRNSLYVNGFTHRSSMPTTSTPGTSTVDVGTSGTPSSSPSPT, templa te: AFKYIKPQSQFIVDLGAY
I0321 02:54:06.287062 139960428058432 templates.py:278] Found an exact template match 4v23_A.
I0321 02:54:06.426746 139960428058432 run_alphafold.py:185] Running model model_1
WARNING:tensorflow:From /home/whelanlab/alphafold/alphafold/model/tf/input_pipeline.py:151: calling map_fn (from tensorflow.python.ops.map_fn) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Use fn_output_signature instead
W0321 02:54:06.770174 139960428058432 deprecation.py:528] From /home/whelanlab/alphafold/alphafold/model/tf/input_pipeline.py:151: calling map_fn (from te nsorflow.python.ops.map_fn) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Use fn_output_signature instead
2023-03-21 02:54:08.766418: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:39] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_G ROWTH environment variable is set. Original config value was 0.
I0321 02:54:09.424343 139960428058432 model.py:132] Running predict with shape(feat) = {'aatype': (4, 156), 'residue_index': (4, 156), 'seq_length': (4,), 'template_aatype': (4, 4, 156), 'template_all_atom_masks': (4, 4, 156, 37), 'template_all_atom_positions': (4, 4, 156, 37, 3), 'template_sum_probs': (4, 4, 1), 'is_distillation': (4,), 'seq_mask': (4, 156), 'msa_mask': (4, 508, 156), 'msa_row_mask': (4, 508), 'random_crop_to_size_seed': (4, 2), 'template_m ask': (4, 4), 'template_pseudo_beta': (4, 4, 156, 3), 'template_pseudo_beta_mask': (4, 4, 156), 'atom14_atom_exists': (4, 156, 14), 'residx_atom14_to_atom 37': (4, 156, 14), 'residx_atom37_to_atom14': (4, 156, 37), 'atom37_atom_exists': (4, 156, 37), 'extra_msa': (4, 5120, 156), 'extra_msa_mask': (4, 5120, 1 56), 'extra_msa_row_mask': (4, 5120), 'bert_mask': (4, 508, 156), 'true_msa': (4, 508, 156), 'extra_has_deletion': (4, 5120, 156), 'extra_deletion_value': (4, 5120, 156), 'msa_feat': (4, 508, 156, 49), 'target_feat': (4, 156, 22)}
2023-03-21 02:54:09.995058: E external/org_tensorflow/tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:417] Loaded runtime CuDNN library: 8.2.1 bu t source was compiled with: 8.6.0. CuDNN library needs to have matching major version and equal or higher minor version. If using a binary install, upgra de your CuDNN library. If building from sources, make sure the library loaded at runtime is compatible with the version specified during compile configur ation.
2023-03-21 02:54:09.997237: E external/org_tensorflow/tensorflow/compiler/xla/status_macros.cc:54] INTERNAL: RET_CHECK failure (external/org_tensorflow/te nsorflow/compiler/xla/service/gpu/gpu_compiler.cc:641) dnn != nullptr
*** Begin stack trace ***
*** End stack trace ***
Traceback (most recent call last):
File "/home/whelanlab/alphafold/run_alphafold.py", line 338, in
app.run(main)
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/absl/app.py", line 312, in run
_run_main(main, args)
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/absl/app.py", line 258, in _run_main
sys.exit(main(argv))
File "/home/whelanlab/alphafold/run_alphafold.py", line 310, in main
predict_structure(
File "/home/whelanlab/alphafold/run_alphafold.py", line 192, in predict_structure
prediction_result = model_runner.predict(processed_feature_dict)
File "/home/whelanlab/alphafold/alphafold/model/model.py", line 134, in predict
result = self.apply(self.params, jax.random.PRNGKey(0), feat)
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/random.py", line 136, in PRNGKey
key = prng.seed_with_impl(impl, seed)
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/prng.py", line 269, in seed_with_impl
return random_seed(seed, impl=impl)
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/prng.py", line 581, in random_seed
return random_seed_p.bind(seeds_arr, impl=impl)
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/core.py", line 360, in bind
return self.bind_with_trace(find_top_trace(args), args, params)
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/core.py", line 363, in bind_with_trace
out = trace.process_primitive(self, map(trace.full_raise, args), params)
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/core.py", line 807, in process_primitive
return primitive.impl(*tracers, **params)
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/prng.py", line 593, in random_seed_impl
base_arr = random_seed_impl_base(seeds, impl=impl)
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/prng.py", line 598, in random_seed_impl_base
return seed(seeds)
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/prng.py", line 833, in threefry_seed
lax.shift_right_logical(seed, lax_internal._const(seed, 32)))
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/lax/lax.py", line 513, in shift_right_logical
return shift_right_logical_p.bind(x, y)
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/core.py", line 360, in bind
return self.bind_with_trace(find_top_trace(args), args, params)
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/core.py", line 363, in bind_with_trace
out = trace.process_primitive(self, map(trace.full_raise, args), params)
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/core.py", line 807, in process_primitive
return primitive.impl(*tracers, **params)
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/dispatch.py", line 122, in apply_primitive
compiled_fun = xla_primitive_callable(prim, *unsafe_map(arg_spec, args),
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/util.py", line 254, in wrapper
return cached(config._trace_context(), *args, **kwargs)
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/util.py", line 247, in cached
return f(*args, **kwargs)
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/dispatch.py", line 201, in xla_primitive_callable
compiled = _xla_callable_uncached(lu.wrap_init(prim_fun), device, None,
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/dispatch.py", line 357, in _xla_callable_uncached
return computation.compile(_allow_propagation_to_outputs=allow_prop).unsafe_call
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/interpreters/pxla.py", line 3373, in compile
executable = self._compile_unloaded(
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/interpreters/pxla.py", line 3344, in _compile_unloaded
return UnloadedMeshExecutable.from_hlo(
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/interpreters/pxla.py", line 3626, in from_hlo
xla_executable = dispatch.compile_or_get_cached(
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/dispatch.py", line 1091, in compile_or_get_cached
return backend_compile(backend, serialized_computation, compile_options,
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/profiler.py", line 314, in wrapper
return func(*args, **kwargs)
File "/home/whelanlab/localcolabfold/colabfold-conda/lib/python3.9/site-packages/jax/_src/dispatch.py", line 1036, in backend_compile
return backend.compile(built_c, compile_options=options)
jaxlib.xla_extension.XlaRuntimeError: INTERNAL: RET_CHECK failure (external/org_tensorflow/tensorflow/compiler/xla/service/gpu/gpu_compiler.cc:641) dnn != nullptr