SyntaxError: invalid character in identifier

[ericdill]

What in the world is going on here. @ocefpaf @CJ-Wright @tonyfast anyone have cycles to help me figure out what's going on?

1. clone jupyter-incubator/sparkmagic (hub clone jupyter-incubator/sparkmagic)
2. run depfinder over the hdijupyterutils subdir (depfinder ./hdijupyterutils)

/home/centos/miniconda/envs/depfinder/lib/python3.7/site-packages/depfinder/main.py:67: YAMLLoadWarning: calling yaml.lo
ad() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full
 details.
  pkgutil.get_data(__name__, 'pkg_data/pkg_data.yml').decode()
Traceback (most recent call last):                                                                                     
  File "/home/centos/miniconda/envs/depfinder/bin/depfinder", line 10, in <module>                                          sys.exit(cli())                                                                                                       File "/home/centos/miniconda/envs/depfinder/lib/python3.7/site-packages/depfinder/cli.py", line 191, in cli
    deps = simple_import_search(file_or_dir, remap=not args.no_remap)                                                  
  File "/home/centos/miniconda/envs/depfinder/lib/python3.7/site-packages/depfinder/main.py", line 332, in simple_import_search    
    for mod, path, catcher in catchers:                                                                                
  File "/home/centos/miniconda/envs/depfinder/lib/python3.7/site-packages/depfinder/main.py", line 293, in iterate_over_
library                                                                                                                
    yield parse_file(full_file_path)
  File "/home/centos/miniconda/envs/depfinder/lib/python3.7/site-packages/depfinder/main.py", line 262, in parse_file  
    catcher = get_imported_libs(code)
  File "/home/centos/miniconda/envs/depfinder/lib/python3.7/site-packages/depfinder/main.py", line 236, in get_imported_
libs                                                                                                                   
    tree = ast.parse(code)                                                                                                File "/home/centos/miniconda/envs/depfinder/lib/python3.7/ast.py", line 35, in parse
    return compile(source, filename, mode, PyCF_ONLY_AST)
  File "<unknown>", line 1                                                                                                  # Distributed under the terms of the Modified BSD License.
    ^                               
SyntaxError: invalid character in identifier                                                                           

I edited my local install of depfinder to print out the files it was parsing:

depfinder hdijupyterutils -v                      master :: 3d :: ⬢  
2019-07-22 15:58:52,115 - depfinder - DEBUG - Treating hdijupyterutils as a directory and recursively searching it for p
ython files                                                                                                            
DEBUG:depfinder:Treating hdijupyterutils as a directory and recursively searching it for python files                  
2019-07-22 15:58:52,115 - depfinder - DEBUG - Setting PACKAGE_NAME global variable to hdijupyterutils                  
DEBUG:depfinder:Setting PACKAGE_NAME global variable to hdijupyterutils                                                
2019-07-22 15:58:52,115 - depfinder - DEBUG - Reading file: hdijupyterutils/setup.py                                   
DEBUG:depfinder:Reading file: hdijupyterutils/setup.py                                                                 
2019-07-22 15:58:52,116 - depfinder - DEBUG - Reading file: hdijupyterutils/hdijupyterutils/events.py                  
DEBUG:depfinder:Reading file: hdijupyterutils/hdijupyterutils/events.py                                                
2019-07-22 15:58:52,117 - depfinder - DEBUG - Reading file: hdijupyterutils/hdijupyterutils/eventshandler.py           
DEBUG:depfinder:Reading file: hdijupyterutils/hdijupyterutils/eventshandler.py                                         
2019-07-22 15:58:52,117 - depfinder - DEBUG - Reading file: hdijupyterutils/hdijupyterutils/utils.py                   
DEBUG:depfinder:Reading file: hdijupyterutils/hdijupyterutils/utils.py                                                 
2019-07-22 15:58:52,117 - depfinder - ERROR - Error encountered reading file: hdijupyterutils/hdijupyterutils/utils.py 
Traceback (most recent call last):                                                                                     
  File "/home/ericdill/dev/depfinder/depfinder/main.py", line 297, in iterate_over_library                             
    yield parse_file(full_file_path)                                                                                   
  File "/home/ericdill/dev/depfinder/depfinder/main.py", line 265, in parse_file                                       
    catcher = get_imported_libs(code)                                                                                  
  File "/home/ericdill/dev/depfinder/depfinder/main.py", line 238, in get_imported_libs                                
    tree = ast.parse(code)                                                                                             
  File "/home/ericdill/miniconda/envs/depfinder/lib/python3.7/ast.py", line 35, in parse                               
    return compile(source, filename, mode, PyCF_ONLY_AST)                                                              
  File "<unknown>", line 1                                                                                             
    # Distributed under the terms of the Modified BSD License.                                                         
    ^                                                                                                                  
SyntaxError: invalid character in identifier                                           

So it seems like these files have some invalid characters in them, but i've got no idea how to find out which characters are "invalid". Also this seems insane because the library clearly works... help!

[ericdill]

After a bit more digging, it seems that some of the sparkmagic files have \ufeff characters at the beginning of them:

bad_file = '/home/ericdill/dev/sparkmagic/hdijupyterutils/hdijupyterutils/log.py'

with open(bad_file, 'r', encoding='utf-8') as f:
    content = f.read()

print(content)

Shows:

'\ufeff# Distributed under the terms of the Modified BSD License.\n\nfrom __future__ import print_function\nimport logging\nimport logging.config\n\nfrom .constants import LOGGING_CONFIG_CLASS_NAME\n\n\nclass Log(object):\n    """Logger for magics. A small wrapper class around the configured logger described in the configuration file"""\n    def __init__(self, logger_name, logging_config, caller_name):\n        logging.config.dictConfig(logging_config)\n        \n        assert caller_name is not None\n        self._caller_name = caller_name\n        self.logger_name = logger_name\n        self._getLogger()\n\n    def debug(self, message):\n        self.logger.debug(self._transform_log_message(message))\n\n    def error(self, message):\n        self.logger.error(self._transform_log_message(message))\n\n    def info(self, message):\n        self.logger.info(self._transform_log_message(message))\n\n    def _getLogger(self):\n        self.logger = logging.getLogger(self.logger_name)\n\n    def _transform_log_message(self, message):\n        return u\'{}\\t{}\'.format(self._caller_name, message)\n\n\ndef logging_config():\n    return {\n        u"version": 1,\n        u"formatters": {\n            u"magicsFormatter": {\n                u"format": u"%(asctime)s\\t%(levelname)s\\t%(message)s",\n                u"datefmt": u""\n            }\n        },\n        u"handlers": {\n            u"magicsHandler": {\n                u"class": LOGGING_CONFIG_CLASS_NAME,\n                u"formatter": u"magicsFormatter",\n                u"home_path": "~/.hdijupyterutils"\n            }\n        },\n        u"loggers": {\n            u"magicsLogger": {\n                u"handlers": [u"magicsHandler"],\n                u"level": u"DEBUG",\n                u"propagate": 0\n            }\n        }\n    }    \n'

Not quite sure yet how to fix this issue

[ericdill]

@CJ-Wright is this similar to that issue you encountered with adal?

[CJ-Wright]

Yes I think this is fixed now?

[ericdill]

Did we fix this in depfinder?

[CJ-Wright]

I think so, I think this was the same weird starting character that I was running into before. I did not run the test though, so reopening.