Tab-completion crashes

[rhaberkorn]

Tab-completions crash ILua on Ubuntu 20.04. It makes no difference whether I install it via Pip or from sources. To minimize possible interactions with installed Pip-packages (or even apt-installed packages), I tried to install ILua into its own virtual environment.

rhaberkorn@thinkpad-x270:~/working-copies/ilua$ python3 -m venv env
rhaberkorn@thinkpad-x270:~/working-copies/ilua$ . env/bin/activate
(env) rhaberkorn@thinkpad-x270:~/working-copies/ilua$ python3 -m pip install -e . 
Obtaining file:///home/rhaberkorn/working-copies/ilua
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting twisted
  Using cached twisted-23.8.0-py3-none-any.whl (3.1 MB)
Collecting pygments
  Using cached Pygments-2.16.1-py3-none-any.whl (1.2 MB)
Collecting termcolor
  Using cached termcolor-2.3.0-py3-none-any.whl (6.9 kB)
Collecting txzmq
  Using cached txZMQ-1.0.0-py3-none-any.whl (24 kB)
Collecting jupyter-console
  Using cached jupyter_console-6.6.3-py3-none-any.whl (24 kB)
Collecting jupyter-core
  Using cached jupyter_core-5.4.0-py3-none-any.whl (28 kB)
Collecting attrs>=21.3.0
  Using cached attrs-23.1.0-py3-none-any.whl (61 kB)
Collecting automat>=0.8.0
  Using cached Automat-22.10.0-py2.py3-none-any.whl (26 kB)
Collecting typing-extensions>=3.10.0
  Using cached typing_extensions-4.8.0-py3-none-any.whl (31 kB)
Collecting zope-interface>=5
  Using cached zope.interface-6.1-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (249 kB)
Collecting incremental>=22.10.0
  Using cached incremental-22.10.0-py2.py3-none-any.whl (16 kB)
Collecting constantly>=15.1
  Using cached constantly-15.1.0-py2.py3-none-any.whl (7.9 kB)
Collecting hyperlink>=17.1.1
  Using cached hyperlink-21.0.0-py2.py3-none-any.whl (74 kB)
Collecting pyzmq>=13
  Using cached pyzmq-25.1.1-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.1 MB)
Collecting prompt-toolkit>=3.0.30
  Using cached prompt_toolkit-3.0.39-py3-none-any.whl (385 kB)
Collecting jupyter-client>=7.0.0
  Using cached jupyter_client-8.4.0-py3-none-any.whl (103 kB)
Collecting traitlets>=5.4
  Using cached traitlets-5.11.2-py3-none-any.whl (83 kB)
Collecting ipykernel>=6.14
  Using cached ipykernel-6.25.2-py3-none-any.whl (154 kB)
Collecting ipython
  Using cached ipython-8.12.3-py3-none-any.whl (798 kB)
Collecting platformdirs>=2.5
  Using cached platformdirs-3.11.0-py3-none-any.whl (17 kB)
Collecting six
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Requirement already satisfied: setuptools in ./env/lib/python3.8/site-packages (from zope-interface>=5->twisted->ilua==0.2.2.dev18+g05eb181) (44.0.0)
Collecting idna>=2.5
  Using cached idna-3.4-py3-none-any.whl (61 kB)
Collecting wcwidth
  Using cached wcwidth-0.2.8-py2.py3-none-any.whl (31 kB)
Collecting python-dateutil>=2.8.2
  Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
Collecting tornado>=6.2
  Using cached tornado-6.3.3-cp38-abi3-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (427 kB)
Collecting importlib-metadata>=4.8.3; python_version < "3.10"
  Using cached importlib_metadata-6.8.0-py3-none-any.whl (22 kB)
Collecting debugpy>=1.6.5
  Using cached debugpy-1.8.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.5 MB)
Collecting nest-asyncio
  Using cached nest_asyncio-1.5.8-py3-none-any.whl (5.3 kB)
Collecting matplotlib-inline>=0.1
  Using cached matplotlib_inline-0.1.6-py3-none-any.whl (9.4 kB)
Collecting psutil
  Using cached psutil-5.9.6-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (283 kB)
Collecting comm>=0.1.1
  Using cached comm-0.1.4-py3-none-any.whl (6.6 kB)
Collecting packaging
  Using cached packaging-23.2-py3-none-any.whl (53 kB)
Collecting pexpect>4.3; sys_platform != "win32"
  Using cached pexpect-4.8.0-py2.py3-none-any.whl (59 kB)
Collecting stack-data
  Using cached stack_data-0.6.3-py3-none-any.whl (24 kB)
Collecting jedi>=0.16
  Using cached jedi-0.19.1-py2.py3-none-any.whl (1.6 MB)
Collecting pickleshare
  Using cached pickleshare-0.7.5-py2.py3-none-any.whl (6.9 kB)
Collecting decorator
  Using cached decorator-5.1.1-py3-none-any.whl (9.1 kB)
Collecting backcall
  Using cached backcall-0.2.0-py2.py3-none-any.whl (11 kB)
Collecting zipp>=0.5
  Using cached zipp-3.17.0-py3-none-any.whl (7.4 kB)
Collecting ptyprocess>=0.5
  Using cached ptyprocess-0.7.0-py2.py3-none-any.whl (13 kB)
Collecting asttokens>=2.1.0
  Using cached asttokens-2.4.0-py2.py3-none-any.whl (27 kB)
Collecting pure-eval
  Using cached pure_eval-0.2.2-py3-none-any.whl (11 kB)
Collecting executing>=1.2.0
  Using cached executing-2.0.0-py2.py3-none-any.whl (24 kB)
Collecting parso<0.9.0,>=0.8.3
  Using cached parso-0.8.3-py2.py3-none-any.whl (100 kB)
Installing collected packages: attrs, six, automat, typing-extensions, zope-interface, incremental, constantly, idna, hyperlink, twisted, pygments, termcolor, pyzmq, txzmq, wcwidth, prompt-toolkit, python-dateutil, traitlets, tornado, platformdirs, jupyter-core, zipp, importlib-metadata, jupyter-client, debugpy, nest-asyncio, matplotlib-inline, ptyprocess, pexpect, asttokens, pure-eval, executing, stack-data, parso, jedi, pickleshare, decorator, backcall, ipython, psutil, comm, packaging, ipykernel, jupyter-console, ilua
  Running setup.py develop for ilua
Successfully installed asttokens-2.4.0 attrs-23.1.0 automat-22.10.0 backcall-0.2.0 comm-0.1.4 constantly-15.1.0 debugpy-1.8.0 decorator-5.1.1 executing-2.0.0 hyperlink-21.0.0 idna-3.4 ilua importlib-metadata-6.8.0 incremental-22.10.0 ipykernel-6.25.2 ipython-8.12.3 jedi-0.19.1 jupyter-client-8.4.0 jupyter-console-6.6.3 jupyter-core-5.4.0 matplotlib-inline-0.1.6 nest-asyncio-1.5.8 packaging-23.2 parso-0.8.3 pexpect-4.8.0 pickleshare-0.7.5 platformdirs-3.11.0 prompt-toolkit-3.0.39 psutil-5.9.6 ptyprocess-0.7.0 pure-eval-0.2.2 pygments-2.16.1 python-dateutil-2.8.2 pyzmq-25.1.1 six-1.16.0 stack-data-0.6.3 termcolor-2.3.0 tornado-6.3.3 traitlets-5.11.2 twisted-23.8.0 txzmq-1.0.0 typing-extensions-4.8.0 wcwidth-0.2.8 zipp-3.17.0 zope-interface
(env) rhaberkorn@thinkpad-x270:~/working-copies/ilua$ python3 setup.py install_data
WARNING: The wheel package is not available.
WARNING: The wheel package is not available.
WARNING: The wheel package is not available.
/home/rhaberkorn/working-copies/ilua/.eggs/setuptools_scm-8.0.4-py3.8.egg/setuptools_scm/_integration/setuptools.py:30: RuntimeWarning: 
ERROR: setuptools==44.0.0 is used in combination with setuptools_scm>=8.x

Your build configuration is incomplete and previously worked by accident!
setuptools_scm requires setuptools>=61

Suggested workaround if applicable:
 - migrating from the deprecated setup_requires mechanism to pep517/518
   and using a pyproject.toml to declare build dependencies
   which are reliably pre-installed before running the build tools

  warnings.warn(
running install_data
creating /home/rhaberkorn/working-copies/ilua/env/share/jupyter/kernels/lua
copying defaultspec/logo-64x64.png -> /home/rhaberkorn/working-copies/ilua/env/share/jupyter/kernels/lua
copying defaultspec/logo-license.txt -> /home/rhaberkorn/working-copies/ilua/env/share/jupyter/kernels/lua
copying defaultspec/logo-32x32.png -> /home/rhaberkorn/working-copies/ilua/env/share/jupyter/kernels/lua
copying defaultspec/kernel.json -> /home/rhaberkorn/working-copies/ilua/env/share/jupyter/kernels/lua
(env) rhaberkorn@thinkpad-x270:~/working-copies/ilua$ ilua
Jupyter console 6.6.3

ILua 0.2.2.dev18+g05eb181
In [1]: print2023-10-22T01:27:26+0300 [ilua.kernel.ILuaKernel#critical] Uncought exception in message handler
	Traceback (most recent call last):
	  File "/home/rhaberkorn/working-copies/ilua/env/lib/python3.8/site-packages/twisted/internet/defer.py", line 1997, in _inlineCallbacks
	    result = context.run(gen.send, result)
	  File "/home/rhaberkorn/working-copies/ilua/ilua/kernelbase.py", line 199, in handle_message
	    content = yield self.do_complete(**msg['content'])
	  File "/home/rhaberkorn/working-copies/ilua/env/lib/python3.8/site-packages/twisted/internet/defer.py", line 2245, in unwindGenerator
	    return _cancellableInlineCallbacks(gen)
	  File "/home/rhaberkorn/working-copies/ilua/env/lib/python3.8/site-packages/twisted/internet/defer.py", line 2157, in _cancellableInlineCallbacks
	    _inlineCallbacks(None, gen, status, _copy_context())
	--- <exception caught here> ---
	  File "/home/rhaberkorn/working-copies/ilua/ilua/kernelbase.py", line 199, in handle_message
	    content = yield self.do_complete(**msg['content'])
	  File "/home/rhaberkorn/working-copies/ilua/env/lib/python3.8/site-packages/twisted/internet/defer.py", line 1997, in _inlineCallbacks
	    result = context.run(gen.send, result)
	  File "/home/rhaberkorn/working-copies/ilua/ilua/kernel.py", line 197, in do_complete
	    result = yield self.proto.sendRequest({
	builtins.AttributeError: 'ILuaKernel' object has no attribute 'proto'
	

Unhandled exception in event loop:
  File "/home/rhaberkorn/working-copies/ilua/env/lib/python3.8/site-packages/prompt_toolkit/buffer.py", line 1919, in new_coroutine
    await coroutine(*a, **kw)
  File "/home/rhaberkorn/working-copies/ilua/env/lib/python3.8/site-packages/prompt_toolkit/buffer.py", line 1743, in async_completer
    async for completion in async_generator:
  File "/home/rhaberkorn/working-copies/ilua/env/lib/python3.8/site-packages/prompt_toolkit/completion/base.py", line 323, in get_completions_async
    async for completion in completer.get_completions_async(
  File "/home/rhaberkorn/working-copies/ilua/env/lib/python3.8/site-packages/prompt_toolkit/completion/base.py", line 199, in get_completions_async
    for item in self.get_completions(document, complete_event):
  File "/home/rhaberkorn/working-copies/ilua/env/lib/python3.8/site-packages/jupyter_console/ptshell.py", line 150, in get_completions
    content = self.jup_completer.complete_request(
  File "/home/rhaberkorn/working-copies/ilua/env/lib/python3.8/site-packages/jupyter_console/completer.py", line 37, in complete_request
    msg = run_sync(self.client.shell_channel.get_msg)(timeout=self.timeout)
  File "/home/rhaberkorn/working-copies/ilua/env/lib/python3.8/site-packages/jupyter_client/channels.py", line 233, in get_msg
    raise Empty

Exception 
Press ENTER to continue...
In [1]:                                                                                                                                                                                       
Do you really want to exit ([y]/n)? y
Shutting down kernel
(env) rhaberkorn@thinkpad-x270:~/working-copies/ilua$ python3 --version
Python 3.8.10

The Python is quite old as you see, but Ubuntu 20.04 is still LTS. It also makes no difference which Lua interpreter I specify via -i.

[rhaberkorn]

Trying to execute 23:

In [1]: 232023-10-22T01:51:42+0300 [ilua.kernel.ILuaKernel#critical] Uncought exception in message handler
	Traceback (most recent call last):
	  File "/home/rhaberkorn/working-copies/ilua/env/lib/python3.8/site-packages/twisted/internet/defer.py", line 1997, in _inlineCallbacks
	    result = context.run(gen.send, result)
	  File "/home/rhaberkorn/working-copies/ilua/ilua/kernelbase.py", line 196, in handle_message
	    content = yield self.do_is_complete(**msg['content'])
	  File "/home/rhaberkorn/working-copies/ilua/env/lib/python3.8/site-packages/twisted/internet/defer.py", line 2245, in unwindGenerator
	    return _cancellableInlineCallbacks(gen)
	  File "/home/rhaberkorn/working-copies/ilua/env/lib/python3.8/site-packages/twisted/internet/defer.py", line 2157, in _cancellableInlineCallbacks
	    _inlineCallbacks(None, gen, status, _copy_context())
	--- <exception caught here> ---
	  File "/home/rhaberkorn/working-copies/ilua/ilua/kernelbase.py", line 196, in handle_message
	    content = yield self.do_is_complete(**msg['content'])
	  File "/home/rhaberkorn/working-copies/ilua/env/lib/python3.8/site-packages/twisted/internet/defer.py", line 1997, in _inlineCallbacks
	    result = context.run(gen.send, result)
	  File "/home/rhaberkorn/working-copies/ilua/ilua/kernel.py", line 184, in do_is_complete
	    result = yield self.proto.sendRequest({"type": "is_complete",
	builtins.AttributeError: 'ILuaKernel' object has no attribute 'proto'
	
/home/rhaberkorn/working-copies/ilua/env/lib/python3.8/site-packages/jupyter_console/ptshell.py:787: UserWarning: The kernel did not respond to an is_complete_request. Setting `use_kernel_is_complete` to False.
  warn('The kernel did not respond to an is_complete_request. '
In [1]: 23
2023-10-22T01:51:43+0300 [ilua.kernel.ILuaKernel#critical] Uncought exception in message handler
	Traceback (most recent call last):
	  File "/home/rhaberkorn/working-copies/ilua/env/lib/python3.8/site-packages/twisted/internet/defer.py", line 1997, in _inlineCallbacks
	    result = context.run(gen.send, result)
	  File "/home/rhaberkorn/working-copies/ilua/ilua/kernelbase.py", line 193, in handle_message
	    content = yield self.do_execute(**msg['content'])
	  File "/home/rhaberkorn/working-copies/ilua/env/lib/python3.8/site-packages/twisted/internet/defer.py", line 2245, in unwindGenerator
	    return _cancellableInlineCallbacks(gen)
	  File "/home/rhaberkorn/working-copies/ilua/env/lib/python3.8/site-packages/twisted/internet/defer.py", line 2157, in _cancellableInlineCallbacks
	    _inlineCallbacks(None, gen, status, _copy_context())
	--- <exception caught here> ---
	  File "/home/rhaberkorn/working-copies/ilua/ilua/kernelbase.py", line 193, in handle_message
	    content = yield self.do_execute(**msg['content'])
	  File "/home/rhaberkorn/working-copies/ilua/env/lib/python3.8/site-packages/twisted/internet/defer.py", line 1997, in _inlineCallbacks
	    result = context.run(gen.send, result)
	  File "/home/rhaberkorn/working-copies/ilua/ilua/kernel.py", line 131, in do_execute
	    result = yield self.proto.sendRequest({"type": "execute",
	builtins.AttributeError: 'ILuaKernel' object has no attribute 'proto'

[rhaberkorn]

@guysv What's the recommended environment to run this in? Perhaps I can try with Docker. However it won't be an option for me to run the Lua interpreter within Docker as well.

Also, will it be possible to use the web frontend as well?

[rhaberkorn]

Also, will it be possible to use the web frontend as well?

Just to explain this. I have a project based on a custom LuaJIT REPL loop and I am planning to add context sensitive auto-completions anyway. That's not really what I need ILua for. What really interests me is a way to cheaply add a Web interface. People nowadays are spoiled you know. In this context, it would also be interesting to find out whether it will be possible to render any markup or graphics.

[PsychedelicShayna]

EDIT: Should probably tag you, since this was initially addressing your comment @rhaberkorn

I'm running into the same problem, except I can't evaluate anything before it crashes.

I was/still am digging through the code, and traced the problem back to breaking changes in twisted, with the latest working version being v22.10.0, 23+ breaks ILua. I never used twisted, nor have I created a Jupyter kernel before, or interacted with its libraries, but I'm currently spending some time trying to make ILua work with twisted 23+

Until then, the only way to make ILua run is to use a version less than 23. Since Twisted may be used by various other Python packages on the system, globally downgrading it just for ILua is not a good idea.

Instead, if you just want to get it working for the moment, here are some overly methodical instructions for people who come across this issue:

# Pick somewhere to put it. I just placed it next to the repo for debugging purposes.
cd ~/Source/third-party/clones/ilua/

# Create and enter the virtual environment.
python -m venv vilua
cd vilua

# Activate the virtual environment by sourcing whichever activation script you nee to source. 
source ./bin/activate.fish

# Make sure it's activated by checking if python and or pip points to the virtual environment, not the system. 
which pip

# Specify any version < 23 for twisted. 22.10.0 is the latest compatible.
pip install twisted==22.10.0 ilua

# Success?
ilua 

Maybe a script to run it from anywhere conveniently:

#!/bin/sh

source ~/Source/third-party/ilua/vilua/bin/activate
ilua
deactivate

I placed it next to ILua

mv vilua ~/.local/bin/vilua
sudo chmod ug+x ~/.local/bin/vilua

This isn't a permanent solution, nor a complete one.

I think the dependency stems from ILua using it to re-implement existing parts of the ipython / jupyter libraries to work with Twisted, in order to have Python 2 async support?

From ilua/ilua/kernelbase.py

"""
Twisted-based Jupyter base kernel implementation
Pretty much a reimplementation of ipykernel.kernelbase,
but with Twisted, which allows async code in python 2,
and provides txzmq.
"""

Since Python2 is deprecated, and this re-write would need to be maintained long-term, I think the best solution would be to drop twisted entirely. It seems to be more trouble than it's worth, having broken ILua with v23 here, as well as unnecessarily broadening its attack surface (see CVE-2023-46137), emphasizing the importance of maintenance. In this case the CVE had nothing to do with the way ILua uses Twisted, but why even have an "in this case" and not just drop it entirely and not be forced to maintain it?

That's the ideal anyway, but the fastest fix would be to resolve the breaking changes from v22 to v23 of twisted, and write the v23 equivalent of the v22 code, which is what I'm investigating. That being said I don't have an awful lot of time, but I felt like I should at least share what I learned so far.

[rhaberkorn]

@PsychedelicShayna Will try that. Thanks! Using virtual environments is IMHO not hacky at all. Library conflicts are almost unavoidable in real life.
Somebody somewhere is not declaring the Twister-dependency on versions < 23 properly.

[rhaberkorn]

I can confirm that everything works for me once forcing Twister to the right version.

I can even run my custom Lua interpreter in a console - although, that's not terribly useful as I wrote.

It is indeed possible to run ILua in a notebook (Web UI) as well and it will even work with custom Lua interpreters. First you will have to install jupyter-notebook (into the same environment as everything else of course):

pip3 install notebook

Now just running jupyter notebook will start a web server and when creating a notebook from within the website you can manually choose a Lua kernel. It's probably also possible to make this the default choice, but I haven't yet figured out how that works.

Unfortunately, there does not seem to be a way to pass arguments to selected kernels via the jupyter notebook command line. That would be useful in order to set ILua's --lua-interpreter. If you know how, just tell me. It might be possible to just shadow lua in PATH of course and that might even be the best option if you need to pass any custom arguments to your interpreter as ILua does not support passing anything down to the interpreter so passing custom arguments requires a wrapper script anyway. Otherwise, you might just create a new kernel by copying ilua/env/share/jupyter/kernels/lua/ to some new directory, let's say ilua/env/share/jupyter/kernels/luajit/ and edit its kernel.json file. For instance to create an ILua-based kernel using luajit as the interpreter:

{
        "argv":
        [
                "python",
                "-m",
                "ilua.app",
                "-c",
                "{connection_file}",
                "--lua-interpreter", "luajit"
        ],
        "display_name": "LuaJIT",
        "language": "lua",
        "interrupt_mode": "message"
}

What I haven't found out yet is:

• How to interrupt a busy ILua kernel from a running notebook. At least, the interrupt button does not appear to send SIGINT to the corresponding Lua interpreter process.
• How to display some rich text or graphics in command output. From reading the IPython source code I get the impression that this must be supported by the kernel, ie. ILua in our case.

All of this should really be documented in some official place.

[guysv]

thanks for the heads up for the twisted version. working on a release pinning the version. I definitely can't keep up with latest twisted for this project so virtual envs are the way to go.