ffrsam-backend Keeps Restarting Every Second

[jonahyongNTU]

I have just installed FFRSAM on CentOS7.
I copy/paste the code in "database/rsam.sql" to create the default rsam database.

Currently, "docker ps" command shows that ffrsam-frontend and ffrsam-database are up and running.
However, ffrsam-backend is always in a state of restarting.

docker logs ffrsam-backend:
time="2022-12-12T03:21:01Z" level=info msg="read crontab: /app/crontab"
time="2022-12-12T03:21:01Z" level=fatal msg="open /app/crontab: permission denied"
time="2022-12-12T03:21:02Z" level=info msg="read crontab: /app/crontab"
time="2022-12-12T03:21:02Z" level=fatal msg="open /app/crontab: permission denied"
time="2022-12-12T03:21:03Z" level=info msg="read crontab: /app/crontab"
time="2022-12-12T03:21:03Z" level=fatal msg="open /app/crontab: permission denied"
time="2022-12-12T03:21:05Z" level=info msg="read crontab: /app/crontab"
time="2022-12-12T03:21:05Z" level=fatal msg="open /app/crontab: permission denied"

There is no such file "/app/crontab" on my server. I do not understand what ffrsam-backend is trying to do and why it keeps restarting.

Are you able to help me, please?

I have attached screenshot of my bash output.

Thanks,
Jonah

[dlnorgaard]

The backend process will get the data from the wave server, calculate the rsam, store in db, and then generate some images. The /app/crontab dir would be in the container. You should be able to connect with docker exec -it ffrsam-backend /bin/bash and have a look around. Also, from outside the container, do docker logs -f ffrsam-backend and let me know what you see.

[jonahyongNTU]

Thanks, dlnorgaard,

I have stopped and then restarted the ffrsam and then did the "docker logs" command. It shows only the same two lines of error over and over again.

Also, the "docker exec" command says it cannot do anything while the ffrsam-backend is restarting. And since it is always restarting, I cannot do anything.

Here is a screenshot of the output of both the above commands.

[dlnorgaard]

Oh right. Comment out the last line in backend. Dockerfile and then start.

[jonahyongNTU]

Hi dlnorgaard,

The last line in "backend/back.Dockerfile" is:
CMD [ "supercronic", "/app/crontab" ]

So, I commented that out by putting a hash at the beginning of the line.

Then I stopped ffrsam and started it.
Still the same errors, though.

Here attached is a screenshot.

[dlnorgaard]

I can't see your docker up command but did you rebuild the backend container? Run with -b option.

[jonahyongNTU]

Oops, I did not realize I had to rebuild after making any changes.

I have now stopped the ffrsam and built it again.

But I am getting the same results. The backend is just restarting every second and docker exec -it cannot work because it is restarting.

[dlnorgaard]

Also comment out the section under "Add Tini". What user are you installing this as? What is the id of the user you are using (id <username>.) Since it's a permissions denied problem, it could be couple of things: 1) doesn't have permission to run /app/crontab or 2) can't access the script the crontab is supposed to run (/opt/ffrsam/run.sh). The latter is a volume mount (see docker-compose.yml line 31) and may carry over permissions from your host system. I am wondering if the user you installed it as does not have user id 2020 and possibly throwing off permissions. I don't have access to a Linux machine where I can test this out on at the moment so sorry it's a bit of a guessing game for me. Please check permissions on those two things and let me know how they look. I was directly involved in installation for other people in the past so may have missed some steps in the instructions about ownership for the cloned repo.

[jonahyongNTU]

Hi, dlnorgaard,

Happy New Year. Sorry for the delayed response. I had completely forgot about this after the December holidays.

1. So, I have commented out the section under "Add Tini" in the backend/back.Dockerfile", and have done a rebuild.
   

2. The username I am installing as is "ffrsam". Previously, it had a user id of "790". However, I have now changed it to "2020".
   

3. There is no such directory named "/app". Likewise, there is no file named "/app/crontab".
   

4. ffrsam user certainly has permissions to read and execute "/home/ffrsam/ffrsam/backend/src/run.sh"
   

5. The docker ps still shows that the backend is restarting repeatedly.
   

6. "docker exec -it ffrsam-backend /bin/bash" still complains that container is restarting.
   

7. "docker logs ffrsam-backend" outputs nothing
   

Any ideas?
Also, we are going to migrate our server from CentOS7 to Rocky Linux8 later this year. Will the installation be easier on Rocky Linux 8?

Thanks,
Jonah

[dlnorgaard]

Hi & happy new year to you too! I will try to stand this up later in the week to see if I can figure out what's going on.

[dlnorgaard]

Actually got to it today. I used WSL (Ubuntu 22.04) though since I don't have easy access to CentOS 7. I was able to bring everything up in a matter of minutes. After cloning the repo on the Windows side these were the only command I ran in WSL:

95 cd ffrsam
96 cp template.env .env
97 vi .env
98 vi .env
99 ls
100 mkdir data
101 mkdir images
102 chmod a+w images
103 cd backend/src
104 cp config_template.py config.py
105 vi config.py
106 cd ../..
107 docker compose build
108 docker ps
109 docker compose up -d
110 docker ps

I did have to run dos2unix command on the backend scripts for the crons to work since I cloned in Windows, but this wouldn't apply to you.

All containers are up and no error in logs at the moment. The web frontend is also working.

I noticed your /home/ffrsam is under /mnt. What type of file system is being mounted? Btw, Is SELinux enabled? If so, try disabling.

[jonahyongNTU]

I have the following 2 lines in my fstab:

# THE DEFAULT HOME FOR LDAP USERS IS /mnt/home.  symlinks will cause SSH-KEYS to fail, so use bind instead.
/home /mnt/home none bind

This means "/home" and "/mnt/home" are the same. Both of them are the contents of /home.

The file system for "/home" is xfs.

SELinux is already disabled.

[dlnorgaard]

I'm unsure if user ffrsam has access to /app/crontab or not. If you comment out the USER ffrsam line, you will be able to look in the container as root and see if /app/crontab is there.

Another thing to try might be deleting the containers and images and rebuild.
docker compose down
docker container prune
docker image prune

[jonahyongNTU]

Hi, Diana.

"docker container prune" and "docker image prune" seemed to delete lots of stuff. But the it's still the same problem after that. ffrsam-backend is still perpertually restarting.

After commenting out the "USER ffrsam" line and recompiling, it's still the same problem.
After switching the user to root, I am unable to locate "/app/crontab". In fact, even "/app" does not exist. Or perhaps I am trying to access it the wrong way?

[dlnorgaard]

Try cat /app/crontab

[jonahyongNTU]

Hi Diana,

Here is the output from "cat /app/crontab":