Skip to content

getting_started.md

Latest commit

 

History

History
839 lines (589 loc) · 27.6 KB

getting_started.md

File metadata and controls

839 lines (589 loc) · 27.6 KB
Raw

Getting Started

Table of Contents

Quickstart

Make sure docker and other dependencies are installed. Then follow these steps:

Create a .env file (it can be left empty)

Then set a value within the string quotes.

Build the project. This can be done using

docker compose -f docker-compose-dev.yml build

or for mac / linux:

docker compose -f docker-compose-dev-mac.yml build

Build frontend using Bun

bun install
bun run build

Run the project

Run migrations

docker compose -f docker-compose-dev.yml run web python /dokuly_image/dokuly/manage.py migrate
docker compose -f docker-compose-dev.yml up

Access dokuly

In your web browser, dokuly can be accessed at

http://localhost:8000.

Logging in for the first time on localhost

The PLM comes with some default data. A user is created with the username "ossuser" and password: "oss_password". This can be used to login with for the first time to start using and configuring dokuly for your use-case.

Setup self hosting of Dokuly

To start hosting the service on your own machine you have to have all dependencies installed. This includes docker and Bun. See the table of contents for more information on installing these dependencies.

Nginx proxy

NOTE: The nginx proxy is not necessary for making the server accessible on your network from other machines. Your local host / network ip, (say 10.0.0.35) and the port is enough, so to access dokuly it would be: 10.0.0.35:8000.

Local Hosting Setup Guide

To enable access to your locally hosted project on your local network or VPN, follow these steps to configure an NGINX proxy. Directly using your machine's local IP address might not suffice, hence the need for these adjustments:

  1. Locate the NGINX Configuration

    • Navigate to the nginx folder where crucial configuration files reside.

  2. Edit the Configuration File

    • Within the nginx folder, find and open the nginx.conf file for editing.
      • Update IP Address: Search for a line starting with server_name, typically containing an IP like 10.0.0.21. Replace it with your machine's local network IP. To find your IP address:
        • On Linux/Mac: Open a terminal and enter ifconfig.
        • On Windows: Open Command Prompt and type ipconfig. Look for the IPv4 address under your network connection, usually formatted as 192.168.x.x.
      • Set Project Name: Locate the line proxy_set_header Host and change the adjacent value to your project's name. If you haven't set up a project name, run the self_host_create_new_tenant script.

  3. Save and Close

    • After making the necessary edits, save the changes and close the nginx.conf file.

  4. Restart NGINX

    • To apply your changes, execute the self_host_restart.ps1 script located in the self_hosting folder.

By following these steps, your locally hosted project should become accessible to others on the same network or VPN using your computer's local IP address.

SMTP Email

Dokuly has support for smtp email, and some basic functions via email like resetting your user's password. To setup the smtp you will need your own smtp server setup, and then in the .env file add the following fields:

EMAIL_HOST = "your_smtp_provider_address"
EMAIL_PORT = your_port
EMAIL_HOST_USER = "your_smtp_username"
EMAIL_HOST_PASSWORD = "your_smtp_password"
EMAIL_SENDER = 'your_smtp_sender_address'

Replace the values with your own information from your smtp provider.

Currency Exchange Rate API

Add currency conversion through a free currency API:

https://www.exchangerate-api.com/docs/pair-conversion-requests

Add the following to the .env.

CURRENCY_API = "https://v6.exchangerate-api.com/v6/__YOUR_API_KEY__/latest/"

Nexar Part API

dokuly supports using Nexar API.

Add the following to the .env.

NEXAR_CLIENT_ID=""
NEXAR_CLIENT_SECRET=""

To start developing on a new computer

Learning Resources

Docker

Dokuly can run in a Docker container to streamline the toolchain configuration.

A video on how Django with Postgres is set up in Docker can be found here.

If Docker desktop refuses to delete containers, the following command can help:

This will remove:

  • all stopped containers.
  • all networks not used by at least one container.
  • all images without at least one container associated to them.
  • all build cache.
docker system prune -a

If this doesn't help, a reboot might be necessary (on Windows).

Development with Docker (Windows)

  1. Install WSL2 and set it as default WSL version. Download from here.

    wsl --set-default-version 2

  2. Enable Hardware virtualization in BIOS. Follow this link.

  3. Install Docker Desktop. Download from here.

  4. Run ./localServer.ps1 script to build and run images. The localServer.ps1 script takes care of the migrations and has a cleanup function, clearing old images from cache, if you do not want to clear, run the following:

    docker compose -f docker-compose-dev.yml build

docker compose -f docker-compose-dev.yml run --rm web python /dokuly_image/dokuly/manage.py makemigrations

docker compose -f docker-compose-dev.yml run --rm web python /dokuly_image/dokuly/manage.py migrate

docker compose -f docker-compose-dev.yml up

    localServer.ps1 builds both Django and PostgreSQL services

  5. Run bun run dev to start the frontend development server with hot reloading. If Bun is not installed, install it from https://bun.sh (or run curl -fsSL https://bun.sh/install | bash on Linux/macOS).

    Local Django development server should run on localhost:8000

    Both db (postgres / psql) and Django are accessible through CLI in the docker desktop

Development with Docker (Mac)

When developing with Apple silicon (M1), a special Docker version must be installed.

Find the correct package here.

Development (Linux)

If working on Debian Linux. Run the script debain_install_dependencies, to install the necessary dependencies to run the dokuly Docker.

Build Django and PostgreSQL services

  1. Confirm Docker Compose-version and download Docker Compose for Linux.

    mkdir -p ~/.docker/cli-plugins/
curl -SL https://github.com/docker/compose/releases/download/v2.2.3/docker-compose-linux-x86_64 -o ~/.docker/cli-plugins/docker-compose

  2. Restart the machine.

    sudo systemctl reboot

  3. Set permission so that Docker Compose is executable.

    chmod +x ~/.docker/cli-plugins/docker-compose
sudo chown $USER /var/run/docker.sock

  4. Verify the installation.

    docker compose version

# output should be similar to this:
Docker Compose version v.2.2.3

  5. Run and build containers from root of dokuly project.

    docker compose -f docker-compose-dev.yml build
docker compose -f docker-compose-dev.yml run -u root web python /dokuly_image/dokuly/manage.py makemigrations
docker compose -f docker-compose-dev.yml run -u root web python /dokuly_image/dokuly/manage.py migrate
docker compose -f docker-compose-dev.yml up

    On Mac:

    docker compose -f docker-compose-dev-mac.yml build
docker compose -f docker-compose-dev-mac.yml run -u root web python /dokuly_image/dokuly/manage.py makemigrations
docker compose -f docker-compose-dev-mac.yml run -u root web python /dokuly_image/dokuly/manage.py migrate
docker compose -f docker-compose-dev-mac.yml up

    If you enconter the error:

    Status: cgroups: cgroup mountpoint does not exist: unknown, Code: 1

    Run the following workaround:

    sudo mkdir /sys/fs/cgroup/systemd
sudo mount -t cgroup -o none,name=systemd cgroup /sys/fs/cgroup/systemd

    This will build both Django and PostgreSQL, as they are built together in the same docker-compose file.

Build Front-End Main Entry Point

The Docker stack needs the compiled JSX from the Frontend Django Application. Building the entry point is done with Vite (run bun run build for production, or bun run dev for development with hot reloading). NOTE: Remember that the main entry point is built before pushing any changes to git.

  1. Install Bun if needed (see https://bun.sh).

  2. Install dependencies and start the development server with hot reloading.

    bun install
bun run dev

  3. Important: Use the Vite dev server URL for hot reloading:

    • ✅ Hot reloading: http://localhost:3000 (Vite dev server)
    • ❌ No hot reloading: http://localhost:8000 (direct Django)

Django: Dealing with files

Saving File in View

To save a file to DB from within a view, see the following example.

@api_view(('PUT', ))
@renderer_classes((JSONRenderer, ))
def save_doc_file(request, documentId):
    """Method for updating documents, and triggering regeneration of pdf documents.
    """
    # Extracts the data filled in the UI form. Assumes presence of relevant data.
    data = request.data
    document_obj = Document.objects.get(id=documentId)
    # Check if file is included in view data, and save it.
    if "document_file" in data:
        file = request.FILES['document_file']
        document_obj.document_file.save(file.name, file)

    return Response(serializer.data, status=status.HTTP_200_OK)

Passing File from a Link, Through a View

To enable secure sharing of files with clients, files have to be passed through a view.

The following example shows how a file can be accessed by using the view's URL.

The file is downloaded similarly to static files.

No axios query is necessary.

In urls.py:

urlpatterns = [
    path("api/pcbas/download/manufacture_pdf/<int:pcba_id>/", views.download_manufacture_pdf),
]

urlpatterns += router.urls

In views.py:

@api_view(('GET', ))
@renderer_classes((JSONRenderer, ))
def download_manufacture_pdf(request, pcba_id):
    """Return the actual file to the client.
    It fetches the file from the storage bucket and serves it over HTTP.
    """
    # Fetch file from DB
    pcba_obj = Pcba.objects.get(id=pcba_id)
    return  FileResponse(pcba_obj.manufacture_pdf.open('rb'), as_attachment=True)

Access the file in front-end like the following JSX example:

The URL is on the form: api/pcbas/download/manufacture_pdf/1/

// Download button
uri != "" ? (
  <a href={uri} download>
    {/* download icon (button) */)}
    <img width="25px" src="../../static/icons/file-download.svg" alt="icon" />
  </a>
) : (
  ""
);

Django: Entities (Apps), Fields, Migrations and Views

Django is the database and web server for dokuly.

Creating new entities / Apps

  1. Create the new entity using the manage.py file.

    python ./dokuly/manage.py startapp "appname"

  2. Create the files api.py and serializers.py in your new entity ./dokuly/"appname"/

  3. Example api.py

    from appname.models import AppnameModel
from rest_framework import viewsets, permissions
from rest_framework.permissions import IsAuthenticated
from knox.auth import TokenAuthentication
from .serializers import AppNameSerializer

# This class is used for the basic GET, PUT, POST, DELETE methods, using all objects stored in the database as the queryset.

class AppNameViewSet(viewsets.ModelViewSet):
    authetiation_classis = (TokenAuthentication,)
    permission_classes = (IsAuthenticated,)
    queryset = AppnameModel.objects.all()
    serializer_class = AppNameSerializer

    def perform_create(self, serializer):
        serializer.save()

  4. Example serializers.py:

    from rest_framework import serializers
from appname.models import AppnameModel
from rest_framework.fields import ListField

# A basic serializer

class AppnameSerializer(serializers.ModelSerializer):

    class Meta:
        model = AppnameModel
        fields = '__all__'

  5. Configure the urls.py file:

    from django.urls.conf import path
from rest_framework import routers
from .api import AppNameViewSet
from . import views

router = routers.DefaultRouter()
router.register('api/appname', AppNameViewSet, 'appname')

urlpatterns = [
    # Add additional views:
    path('api/appname/update/item/<int:id>/<str:string>/', views.update_item),
    path('api/appname/get/item/<int:id>/', views.fetch_data),
]

urlpatterns += router.urls

  6. Finally add the appname to installed apps in settings.py

    ./dokuly/dokuly/settings.py
# Application definition

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'pcbs',
    'rest_framework',
    'appname', # New app goes here
    ...
]

Adding fields to a model

  1. Locate the model to be altered inside ./system_platform

  2. Find the models.py file

  3. Add a field to the model, by adding a variable to the class:

    class Pcba(models.Model):

    # A standard integer field, can be used for ids, etc. All Django models come with a standard incrementing auto id field when saved to PostgreSQL.
    part_number = models.IntegerField(blank=True, null=True)
    # Blank=True, means the field is not required in forms for the model (PUT, POST)
    # null=True, means the field gets initialized as null, and we are allowed to set it to null after altercation

    # Adding a new char field to the model, with a max size (max_length) and blank=True
    new_field = models.CharField(max_length=100, blank=True, default='test')
    # default is for initializing with a value other than a blank string or null

  4. After the model is updated, run migrateDockerServer.ps1 if on Windows, or if on Linux run:

        docker compose -f docker-compose-dev.yml run -u root web python /dokuly_image/dokuly/manage.py makemigrations
    docker compose -f docker-compose-dev.yml run -u root web python /dokuly_image/dokuly/manage.py migrate

    This will migrate the changes made from Django and save them to the PostgreSQL server.

Adding default data to a model

To add data to a model, create a new view and follow the template below.

from django.db import migrations, models

def add_default_doc_types(apps, schema_editor):
    Document_Prefix = apps.get_model('documents', 'Document_Prefix')

    # Check if the field exits.
    tn_exists = Document_Prefix.objects.filter(prefix="TN")
    if not(tn_exists):
        prefix = Document_Prefix.objects.create(
            prefix="TN",
            display_name='Technical Note',
        )
        prefix.save()

class Migration(migrations.Migration):

    dependencies = [
        ('documents', '0023_auto_20220623_0800'),
    ]

    operations = [
        migrations.RunPython(add_default_doc_types),
    ]

Creating views

A Django view is a customizable function that takes a web response and returns a web response, much like a resolver or a custom express API call.

The request is sent from axios and the response can be anything, from a string or a queryset to XML or HTML. Example on how to write a view and connect it to the Django API:

  1. Create the function(s) in the views.py file.

    from rest_framework.response import Response
from rest_framework.decorators import api_view, renderer_classes
from rest_framework.renderers import JSONRenderer
from rest_framework import status
from .models import AppnameModel
from .serializers import AppnameSerializer

# Define the renderer classes (return types) and the allowed HTTP requests
@api_view(('GET', 'PUT', )) # GET and PUT are allowed
@renderer_classes((JSONRenderer, )) # Return JSON
def example_view(request, objectId): # Request is the HTTP request sent from axios, object id is parameter
    data = list(request.data.values()) # Access form data sent with the request
    method = request.method # Access the request method
    databaseObject = AppnameModel.objects.get(id=objectId) # Fetch a database object
    if method == 'GET': # Let's return the object if the request is GET
        # The serializer lets us return the databaseObject as a JSON Object, set many=True if the query set contains more than 1 entry.
        serializer = AppnameSerializer(databaseObject, many=False)
        return Response(serializer.data, status=status.HTTP_200_OK) # Return the serialized data
    elif method == 'PUT': # Update the object if the request is PUT
        someFormData = data[0] # Can also access request data with keys, data['name']
        AppnameModel.objects.filter(id=objectId).update(name=someFormData) # Single row and col update
        newObject = AppnameModel.objects.get(id=objectId) # Fetch updated object
        serializer = AppnameSerializer(newObject, many=False)
        return Response(serializer.data, status=status.HTTP_200_OK) # Return the serialized updated data
    else: # No request here, return error
        return Response("%s not allowed" % method, status=status.HTTP_400_BAD_REQUEST)

  2. Connect the view to the API in the urls.py file.

    from django.urls.conf import path
from rest_framework import routers
from .api import AppNameViewSet
from . import views

router = routers.DefaultRouter()
router.register('api/appname', AppNameViewSet, 'appname')

# View paths goes here
urlpatterns = [
    # The <int:objectId> need to correspond to the parameter set in the view function
    path('api/appname/customView/<int:objectId>/', views.example_view),
]

urlpatterns += router.urls

  3. Call it using axios from frontend, add a function to an existing queries.js file, create a new one if the component does not have one (usually found in the functions folder for a component).

import axios from 'axios'
import { tokenConfig } from '....../actions/auth'

export const callCustomView = (objectId) => {
    // URL used in the axios call needs to match the URL added to the Django API
    const promise = axios.get(`api/appname/customView/${objectId}/`)
    // Any data sent from the view saved in this const
    const dataPromise = axios.then((res) => res.data)
    // Catch errors
    const error = axois.catch((err) => err)
    if(error != null) {
        return error
    }
    return dataPromise
}

// From any component, to call the query:
import { callCustomView } from './functions/query'

callCustomView.
then((res) => {
    // Do anything with the data
})
.finally(() => {
    // Finish loading
})

Creating React Components (Example)

This is an empty example skeleton file of a React component. Components needs to return a value that is not null, so make sure the default return always returns a value that is not null, e.g. an empty div tag. JSX requires a parent element and tags should always have a corresponding closing tag.

import React, {useState, useEffect} from 'react' // Basic React imports
import { useSpring, animated, config } from 'react-spring' // React Spring Animation imports
import { useAlert } from 'react-alert' // Premade hook for React alerts

// Component name and parameters
const Example = (props) => {

    // Basic state hook. Has a value and a setter: [value, setter]
    const [refresh, setRefresh] = useState(false)
    // To change the value of "refresh" use:
    setRefresh(true)

    // To access props value ( Check the props below in Example Usage)
    // In a state
    const [propValue, setPropValue] = useState(props.data)
    // Or basic variable
    let value = props.value

    // Usage of custom hooks example
    const alertHook = useAlert()
    alertHook.show("This will create an alert!")

    // UseEffect = ComponentDidMount
    // All logic done inside will be applied to states before render of JSX
    useEffect(() => {
        if(refresh) {
            // Do something new
        }
    }, [props, refresh]) // Rerender array. If any of these update, run useEffect and rerender

    return (
        // See Bootstrap documentation for more className presets
        <div className="card-body bg-white m-3 shadow-sm rounded">
            {/* JSX elements here */}
        </div>
    )
}

export default Example // Export to grant other files access to this file

// To use the component in a different file
import Example from './exampleFile'
// The tags inside are the prop values, the attribute names define the prop value field names
<Example props={props} value={value} data={data}/>

Lifting up states in React (Example)

In React, it's easy to pass states down to different components, and here is the trick to pass states from a child component to a parent component, or in other words pass a state upward:

// In child component:
const ChildComponent = (props) => {
    const [state, setState] = useState(0) // Create a standard number state
    setState(1) // Set the new state
    props.liftState(state) // Call the function passed with the props
}
export default ChildComponent

// In parent component:
const ParentComponent = (props) => {
    const [childState, setChildState] = useState(0) // Create a state that will hold child state
    const liftState = (childData) => { // Create a function that takes the child data as parameter
        setChildState(childData) // if called we will set the parent state based on child state
    }

    return (
        // Send the liftState function down to the child component
        <ChildComponent liftState={liftState}>
    )
}

Production Tech Stack and server maintenance

Tech stack, running on production

Dokuly runs on the Django framework, where different entities are their own Django application, including the frontend. From top to bottom:

  1. Frontend Technologies: React (hooks and context) and Bootstrap CSS Package manager: Bun. Build tool: Vite

  2. Middleware Standard Django middleware, configured in the settings.py file, found at ./dokuly/dokuly/

  3. Backend Technologies: Django Package manager: pip and choco

  4. Database Technologies: PostgreSQL

Current Maintenance

  1. Github CI for main entry point errors, Django testing and commit linting

  2. git_fetch_script.sh and Crontab is setup server side to check for updates. Crontab is a Linux job scheduler and runs the git_fetch_script every minute. We run git fetch and if a change is detected on master we reset the head to the local branch, kill the running server and restart it.

Testing the Code

Testing of generic Python functions

A GitHub action for Python doctests is set up, as in the example under.def col_to_list(column_name, dict_reader_obj, skip_rows=0):

def col_to_list(column_name, dict_reader_obj, skip_rows=0):
    Extract a column from a DictReader object.
    This function iterates over the object and essentially consumates it.
    The function can only by tun once on a DataReader object.
    :param column_name: Name of the column.
    :type column_name: str
    :param dict_reader_obj: "dataframe"
    :type dict_reader_obj: DictReader
    :return: the column as a list.
    :rtype: [str]
    ## Example
    >>> csv_file = open('./tests/data/fusion360Bom.csv')
    >>> df=csv.DictReader(csv_file, delimiter=',')
    >>> mpn_list = col_to_list('MPN', df)
    >>> print(mpn_list)
    ['RCS06030000Z0EA', 'CRCW06031K00FKEAC', '', 'FCR7350B', 'FCR7350R', '5-146280-4', '', '', '', '', 'KPT-1608EC', 'PCB1221A']
    >>> csv_file.close()

  1. Create a doctest for a function.

  2. Add the following at the bottom of the file:

    if __name__ == "__main__":
    import doctest
    doctest.testmod()

  3. Add a run of the Python file in the shell script test/python_test.sh

Create a copy of the database on local server

docker exec -t sdp_db_OSS pg_dump -U postgres DokulyOSS > local_backup.sql

Update DB through PG Admin

If the IDs are reset, they can be corrected through this interface.

SELECT setval('nd.timetracking_employeetime_id_seq', (SELECT MAX(id) FROM nd.timetracking_employeetime)+1);

Set up PG Admin locally

image image