yamtl-m2m-service

[arturboronat]

Hi Will, I created a first attempt of a YAMTL M2M service at https://github.com/arturboronat/mdenet-yamtl

This service executes a basic M2M transformation where one model is received as input and another one is produced. The tool service is flexible: trafo, metamodels and models can be changed. The example is an activity that transforms a class diagram into a relational schema.

The unit test case in com.mde-network.ep.toolfunctions.yamtl_m2m_function is working. Please check that they are working for you locally too.

Some help is needed for configuring the service/activity. The configuration should be fairly similar to the ETL service/activity. In fact, I have followed their configuration but it crashes with this output log. The issues is with the dependencies of the Groovy plugin I am using (org.codehaus.gmavenplus:gmavenplus-plugin). If you look at the project pom, the configuration is fairly simple. It should work on docker too.

Some additional feedback regarding the example project template below:

• The project template could benefit from further improvement. Ideally, a template should include parameters and generate configuration files with the appropriate names. However, currently, a template is merely a copy, needing the user to manually search through different configuration files and replace names. This process can only be accomplished once the tool model is clearly understood (and yet there is a fair amount of detail). I believe this poses a challenge for the adoption of the tool.
• Not sure why we need two containers: The docker-compose.yml defines two containers one is the helloworld platform tool service and the other is a web server making the activity contents of the helloworld-example/ available.. Is this needed when using a Java-based tool?
• I found a bit of confusion around the terms tool and service and function (tool service kind of disambiguates the issue). The issue is that a tool (parent pom) seems to be an aggregate of services (function subprojects). Are they interchangeable? Perhaps use only one?
• I found a bit of confusion around the terms example and activity. tool-examples refer to activities.
• The helloworld example is java-based but uses the configuration based on eclipse, with two projects. Why not to use the OCL example to explain the Eclipse-based configuration instead? And the helloworld example for the Java-based one?
• In the helloworld example, RunConversionXToY is a template for RunHelloWorldFunction that is actually not used. It gives the impression that it needs to be included.

[barnettwilliam]

Great progress, I'll try it out and take a look at the error.

The reason for the separate activities webserver container is to replicate the intended environment that is the activities will be hosted separately somewhere e.g. github.

Thank you for the feedback I'll look at updating the example and documentation to clarify.

[barnettwilliam]

I found a couple of small things

• maven module name not matching folder name
• google cloud function maven plugin was missing from the pom. I added fixes for these on a docker-build-fix branch.

I haven't yet got to the bottom of the docker issue.

Causes of the [ERROR] No plugin found for prefix 'function' in the current project ... can be not having the maven plugin for google cloud functions (fixed) or not being in the correct directory when running mvn function run ...
I checked the docker configuration and the locations in the container image docker run -it --entrypoint /bin/bash mdenet-tool-yamtl and it seem the correct directories are being used.

It should be possible to run the services locally outside the container by

cd services/com.mde-network.ep.toolfunctions.yamtl_m2m_function
mvn function:run -Drun.functionTarget=com.mdenetnetwork.ep.toolfunctions.yamtlGroovy.RunYAMTL_m2m_groovy -Drun.port=8001

But I get

[ERROR] Failed to execute goal com.google.cloud.functions:function-maven-plugin ... : Could not invoke 
function: com.mdenetnetwork.ep.toolfunctions.yamtlGroovy.RunYAMTL_m2m_groovy

So the function isn't being found. Looking inside the compiled jar files in the local maven .m2 repository for the yamtl_m2m_function there are no class files. One key difference between the yamtl tool is that it is using groovy so @arturboronat can you configure groovy to create the classes? I don't have groovy installed locally but didn't get any errors building using maven so I assume that build was successful and isn't the reason no class files were generated.

It should be possible to use google cloud functions and groovy however it doesn't mention running them locally as we are doing. I suspect if the classes are generated then it will work.

[arturboronat]

Many thanks for this, @barnettwilliam .

This commit solves the issue of adding binaries to the jar.

The configuration of the service is not working yet but I guess that you will be able to make progress on your side now.

[barnettwilliam]

Perfect that helped a lot, the following commit arturboronat/mdenet-yamtl@daa28eb fixes the issue with the docker container failing to run by automatically downloading the dependencies. So you should be able to try out your activity now.

I've added an issue mdenet/platformtools#13 to apply the same to the mdenet repositories

[arturboronat]

Many thanks for the updates and encouraging words, @barnettwilliam

However, I'm not getting it up and running yet. Could you check the configuration in the last commit and let me know whether the URL I should be using is http://localhost:8080/?activities=http://localhost:8083/yamtl-cd2db-activity.json.

As an observation, I don't know why there are so many attempts to connect to localhost. See the output log.

[barnettwilliam]

Sure I'll try and run the activity, yes that is the correct url.

The yamtl tool port should be 8500 and the console is still part of the epsilon tool so you have to add http://127.0.0.1:8070/epsilon_tool.json to tools. There is an open issue to include common tools by default and remove them from epsilon.

I'll push some fixes for the activity config once I've tested.

I didn't get those Trying to connect to validate messages, how much spare much memory do you have available when running the platform, could it be running low?

[barnettwilliam]

There were a few things:

• not all panels being referenced in the activity config layout
• icon image file not matching the name in static.yamtlgroovy icons.css
• missing tools in activity config for metamodel and model
• activity config action mapping output was misnamed outModel
• types given as strings in yamtl tool config action function didn't match the panel so type checking / conversion wouldn't run the action function
• the mandatory 'languages' parameter that should be part of the tool function interface specification was missing. If you only have one function you don't have to do anything with it within the services function class.

Fixes in commits arturboronat/mdenet-yamtl@3c0173b and arturboronat/mdenet-yamtl@651f5cd

My output log after starting and invoking the yamtl function a few times just in case it's useful.

With these changes panels are displayed including the new YamTL one, the tool service is running when the action button is clicked, and the response is displayed. @arturboronat the response from YamTL contains an error about unextpected '\' but looks like it's from YamTL itself.

[arturboronat]

Hi @barnettwilliam ,

Fantastic job and congrats for the article!

Although I could not reproduce your error (I'll talk about this later), I managed to fix the issue that you mentioned.

The problem with the character encoding can be resolved in maven, by adding the following code to the root pom.xml:

  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
  </properties>

To keep a consistent character encoding in the docker container, I added the following line to the dockerfile: ENV LANG C.UTF-8

I have added additional output in the log to display the REQUEST object at the start of the execution of the function and the RESPONSE object after the transformation. The back-end service is processing the tranformation well after fixing the character encoding issues.

I have pushed the code to https://github.com/arturboronat/mdenet-yamtl to the branch docker-build-fix.

---

Remaining issues

Output not displayed

I couldn't find any information in the output model or log. When examining yamtl_tool.json and the activity configuration in activities/yamtl-cd2db-activity.json, I'm unsure how to associate the output values with the respective panels. Even after reviewing the Epsilon examples, I couldn't identify these connections. In your example, you showed the error message is displayed in the console panel but this is not the case for me.

Models not displayed graphically

The rendering of metamodels and models as graphical models is not working yet. Is there any pre-built service implementing this graphical rendering?

---

I have fiddled a little bit with the parameter names in the configuration trying to fix this second problem - I hope not to have broken anything. The last commit shows the changes.

Thanks!

[barnettwilliam]

Thank you and great that you have the YAMTL backend service processing information well, I will have to try it out.

The platform's response handling is an area that needs some work. Currently only text via a ConsolePanel class or a diagram via an OutputPanel class are supported, these are also mutually exclusive.

Output type conversion is also not implemented so tool functions have to generate the desired output as an svg image or text for it to be displayed by the platform. The aim is create conversion functions for common model types so the the output from tool functions can be automatically converted for display in a similar manner that is done for inputs to tool functions.

The relevant documentation
https://github.com/mdenet/educationplatform/wiki/Adding-a-Tool#output

Outstanding action for me to update documentation -
mdenet/educationplatform#4 (comment)

Sequence diagram of response handling -
https://github.com/mdenet/educationplatform/wiki/Design#user-activity

If the output of the YAMTL tool outputs a standard metamodel and model file? it would probably be worthwhile for me to implement the output type conversion and some display conversion functions for common model types. For quick testing, you could modify the YAMTL tool service to generate the desired svg images.

Also if you have any other particular display configuration in mind for an example activity that your creating which is currently unsupported let me know and I'll see if we can support it.

[arturboronat]

Many thanks for your prompt reply and useful pointers.

The transformation is now working in the sense that the target model is displayed in the panel-tmodel. Check the last commit on https://github.com/arturboronat/mdenet-yamtl (branch docker-build-fix).

As an idea for improvement, adding a column description to this table, would ensure that the field descriptions are consolidated, rather than being dispersed across various subsections. I believe this could improve clarity for users.

ConsolePanel vs OutputPanel

As far as I understand a tool function is defined with a single returnType (text in the case of yamtl_tool.json) and this corresponds to output. I tried to use the other response parameters as shown in the response specification but I could not make use of them in the function definition. So far output is working.

Having both output and console text would be helpful in YAMTL, to show how much time a transformation took on the server, for example.

Visualization of models and metamodels

For improved user experience, it might be beneficial to associate an action with the recycle icon, enabling it to convert XMI to an image and vice versa, similar to the functionality found in the Epsilon platform. Ideally, this action should operate independently from the transformation. In other words, while the transformation might output XMI, users should have the flexibility to select their preferred format. Creating separate activities for each output format introduces unnecessary redundancy.

Given a service that generates plantuml from a metamodel and I create the action that calls that service, how would I indicate that the output is of type puml? I would like to reuse an existing panel, which may be tagged with a different type (say xmi). Are there examples available that involve puml or dot?

Some questions

• What is the difference between output and generatedText?
• In the directory static.<toolname>, the directories dist and public contain duplicate information. For example the tool configuration. If I haven't overlooked it in the documentation, could you specify which directory should be the primary point of modification?
• In the response example makes use of the parameter "targetModelDiagram" but the response specification does not include this parameter, the key [*diagram*] appear instead.
• How is the parameter error used?
• In the user activity diagram,
  • How does the expression [response key contains diagram] in the user activity diagram translate to code? What specific key is being referred to, and how is its content verified to contain a diagram?
  • It appears that response.generatedText is interpreted as a boolean in the diagram, yet it's described as text in the response specification. Can you shed light on this?
  • I couldn't find a definition for outputType in the response specification. Could you provide further details?
• The following errors appear in the output console, which are likely to affect how the code is highlighted:

educationplatform-docker-mdenet-educationplatform-1  | 172.19.0.1 - - [22/Aug/2023:17:59:47 +0000] "GET /?activities=http://localhost:8083/yamtl-cd2db-activity.json HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36" "-"
educationplatform-docker-mdenet-tool-emf-1           | 2023/08/22 17:59:47 [error] 10#10: *1 open() "/usr/share/nginx/html/highlighting.js" failed (2: No such file or directory), client: 172.19.0.1, server: , request: "GET /highlighting.js HTTP/1.1", host: "127.0.0.1:8073", referrer: "http://localhost:8080/"
educationplatform-docker-mdenet-tool-emf-1           | 172.19.0.1 - - [22/Aug/2023:17:59:47 +0000] "GET /highlighting.js HTTP/1.1" 404 555 "http://localhost:8080/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36" "-"
educationplatform-docker-mdenet-educationplatform-1  | 172.19.0.1 - - [22/Aug/2023:18:06:40 +0000] "GET /?activities=http://localhost:8083/yamtl-cd2db-activity.json HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36" "-"
educationplatform-docker-mdenet-tool-emf-1           | 2023/08/22 18:06:41 [error] 11#11: *2 open() "/usr/share/nginx/html/highlighting.js" failed (2: No such file or directory), client: 172.19.0.1, server: , request: "GET /highlighting.js HTTP/1.1", host: "127.0.0.1:8073", referrer: "http://localhost:8080/"
educationplatform-docker-mdenet-tool-emf-1           | 172.19.0.1 - - [22/Aug/2023:18:06:41 +0000] "GET /highlighting.js HTTP/1.1" 404 555 "http://localhost:8080/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36" "-"

[arturboronat]

I left the demo working yesterday (https://github.com/arturboronat/mdenet-yamtl/tree/docker-build-fix) but I tried to execute it today and I get stuck with the login screen on http://localhost:8080/?activities=http://localhost:8083/yamtl-cd2db-activity.json

I have created a second branch with another activity that gets emfatic metamodels instead of ecore metamodels but I can’t test it for the same reason.

Do you know what may be happening? I'm a bit baffled since this was working yesterday.

Another question: what parameter types are supported when defining functions in https://github.com/mdenet/educationplatform/wiki/Adding-a-Tool#parameter?

[barnettwilliam]

For the activity not displaying

After taking a look at at your activity on branch cd2db-emfatic, there are a couple of typos:

• line 8, the emfatic tool should be http://127.0.0.1:8071/emfatic_tool.json
• line 51, the file 'CD' extension does not match the file in the activities dir
• line 57, the file 'relational' extension does not match the file in the activities dir

Browsers can cache the files so errors do not always immediately occur. Force refreshing e.g. ctrl + f5 after making changes can help show them.

At the moment there are no user friendly messages displayed and is definitely something that we need to add.

I still get this error reported back after trying to run the activity:

startup failed: 

CD2DB.groovy: 1: Unexpected character: '\' @ line 1, column 31.  

[barnettwilliam]

Additionally the last commit removed the epsilon tool "http://127.0.0.1:8070/epsilon_tool.json" under tools, which is needed for the console panel. This should fix the not loading issue. There is an open issue for the general purpose panels to be moved into the platform itself.

Please could you commit the missing CD and relational files for me to test the full example. Thanks

[mde-docs]

Done: https://github.com/arturboronat/mdenet-yamtl/tree/cd2db-emfatic/activities

I have reinstated the epsilon_tool line in the activity configuration file.

[barnettwilliam]

Thanks I have tested and its loading.

On my linux machine I still see the 'unexepected character' error #4 (comment) even with the settings committed in #4 (comment). Running linux2dos on the activity files resolves this and I get the yamtl generated xmi so it seems like the settings #4 (comment) aren't taking effect for groovy in the yamtl toolservivce?

For the display of models I'll fix the panel render button to support standard formats e.g. xmi mdenet/educationplatform#79 and fix the platform response handling so the output field can be displayed in a console with other fields mdenet/educationplatform#80