-
-
-
-
-
-
- back to the top -
- - -## Architecture - -``` -echo/ -├── .github/ # GitHub Actions workflows (CI/CD) -│ -├── apps/ -│ ├── api/ # Spring Boot backend (REST API, business logic, persistence) -│ │ ├── src/ # Application source code -│ │ ├── build.gradle # Application build configuration -│ │ └── ... -│ │ -│ └── client/ # Frontend web client (TypeScript, React, Vite) -│ ├── src/ # Application source code -│ ├── package.json # Application dependencies & scripts -│ └── ... -│ -├── docs/ # Documentation andrelated assets -│ ├── assets/ # Documentation images -│ ├── logo/ # Documentation logos -│ ├── architecture.md # Application architecture discussion -│ └── clerk-setup.md # Clerk configuration guide -│ -├── DEVELOPMENT.md # Developer setup guide -└── README.md # Project overview and quick start -``` - -- back to the top -
- -## Prerequisites - -The project requires a pre-configured Clerk application, as described in the [Clerk setup guide](./docs/clerk-setup.md). - -You will require [Java](https://www.oracle.com/europe/java/technologies/downloads/), [Node](https://docs.npmjs.com/downloading-and-installing-node-js-and-npm) and [Docker](https://docs.docker.com/get-started/get-docker/) installations. - -You should use a version manager (e.g. `sdkman` for Java, `nvm` for Node) so that you can easily switch to the required versions. - -### Minimum recommended versions - -- Java (JDK): **21 (LTS)** -- Node.js: **22.x (LTS, latest patch)** - -- back to the top -
- -## Environment setup - -### Clone the repository to your local machine - -```sh -git clone https://github.com/nednella/echo.git -cd echo -``` - -### Environment variables - -Each application’s root directory includes an `.env.*.template` file per environment. Copy the template file into a new environment file (as shown below), then fill in any missing values from the appropriate source. - -```sh -# Frontend -cp ./apps/client/.env.local.template ./apps/client/.env.local -``` - -```sh -# Backend -cp ./apps/api/.env.dev.template ./apps/api/.env.dev -``` - -- back to the top -
- -## Environment startup - -> [!TIP] -> Use multiple terminal tabs to run applications in parallel. - - -### Start backend - -**1. Navigate to the application root** - -```sh -cd apps/api -``` - -**2. Run the application with the local Gradle wrapper** - -> [!NOTE] -> Docker container lifecycles are [automatically managed](https://docs.spring.io/spring-boot/reference/features/dev-services.html) by Spring Boot. - -```sh -./gradlew bootRun -``` - -By default, the server will be available on port `8080`. - -### Start frontend - -**1. Navigate to the application root** - -```sh -cd apps/client -``` - -**2. Install dependencies** - -```sh -npm i -``` - -**3. Run the application** - -```sh -npm run dev -``` - -By default, the web client will be available on port [5173](http://localhost:5173/). - -- back to the top -
- -## Commands - -### Backend development - -```bash -./gradlew bootRun # start the development server (docker lifecycle management built-in) - -./gradlew clean # clean the /build directory - -./gradlew test # test the application - -./gradlew assemble # build the application without prior testing - -./gradlew build # test and build the application -``` - -### Frontend development - -```bash -npm i # Install dependencies - -npm run dev # Start the development server - -npm run build # build the application - -npm run format # run prettier to check for any formatting errors - -npm run format:fix # run prettier to fix any formatting errors - -npm run lint # run eslint to check for any linting errors - -npm run lint:fix # run eslint to fix any linting errors -``` - -- back to the top -
- -## The onboarding process - -> [!IMPORTANT] -> The server requires users to complete the onboarding step before any API calls are accepted. If onboarding is not completed, requests will return `403 Forbidden` with the message `User has not completed the onboarding process`. - -Depending on your area of local development, you may or may not need to complete this process manually. - -### Backend development - -If you are working with the Spring Boot server only, you must create a user in the Clerk dashboard and generate a long-lived session token for authentication. - -You can generate a token for a given `User ID` (found via the **Users** tab) using the [Clerk backend API](https://clerk.com/docs/reference/api/overview) from any HTTP client of your choice. - -Once you have a valid token, call `POST /v1/clerk/onboarding` once, immediately after registering the user in the dashboard. If the request succeeds, **generate a new session token** (required to pull in updated token claims) and use that token to authenticate subsequent API requests. - -### Frontend development - -If you register a new user through the web client’s authentication pages, onboarding will be handled automatically via the `/onboarding` page redirect after successful registration. - ---- - - -- back to the top -
+ +
+
+
+
+
+ back to the top +
+ +## Architecture + +``` +echo/ +├── .github/ # GitHub Actions workflows (CI/CD) +│ +├── apps/ +│ ├── api/ # Spring Boot backend (REST API, business logic, persistence) +│ │ ├── src/ # Application source code +│ │ ├── build.gradle # Application build configuration +│ │ └── ... +│ │ +│ └── client/ # Frontend web client (TypeScript, React, Vite) +│ ├── src/ # Application source code +│ ├── package.json # Application dependencies & scripts +│ └── ... +│ +├── docs/ # Documentation and related assets +│ ├── assets/ # Documentation images +│ ├── logo/ # Documentation logos +│ ├── architecture.md # Application architecture discussion +│ └── clerk-setup.md # Clerk configuration guide +│ +├── DEVELOPMENT.md # Developer setup guide +└── README.md # Project overview and quick start +``` + ++ back to the top +
+ +## Prerequisites + +The project requires a pre-configured Clerk application, as described in the [Clerk setup guide](./docs/clerk-setup.md). + +You will require [Java](https://www.oracle.com/europe/java/technologies/downloads/), [Node](https://docs.npmjs.com/downloading-and-installing-node-js-and-npm) and [Docker](https://docs.docker.com/get-started/get-docker/) installations. + +You should use a version manager (e.g. `sdkman` for Java, `nvm` for Node) so that you can easily switch to the required versions. + +### Minimum recommended versions + +- Java (JDK): **21 (LTS)** +- Node.js: **22.x (LTS, latest patch)** + ++ back to the top +
+ +## Environment setup + +### Clone the repository to your local machine + +```sh +git clone https://github.com/nednella/echo.git +cd echo +``` + +### Environment variables + +Each application’s root directory includes an `.env.*.template` file per environment. Copy the template file into a new environment file (as shown below), then fill in any missing values from the appropriate source. + +```sh +# Frontend +cp ./apps/client/.env.local.template ./apps/client/.env.local + +# Backend +cp ./apps/api/.env.dev.template ./apps/api/.env.dev +``` + ++ back to the top +
+ +## Environment startup + +> [!TIP] +> Use multiple terminal tabs to run applications in parallel. + +### Start backend + +**1. Navigate to the application root** + +```sh +cd apps/api +``` + +**2. Run the application with the local Gradle wrapper** + +> [!NOTE] +> Docker container lifecycles are [automatically managed](https://docs.spring.io/spring-boot/reference/features/dev-services.html) by Spring Boot. + +```sh +./gradlew bootRun +``` + +By default, the server will be available on port `8080`. + +### Start frontend + +**1. Navigate to the application root** + +```sh +cd apps/client +``` + +**2. Install dependencies** + +```sh +npm i +``` + +**3. Run the application** + +```sh +npm run dev +``` + +By default, the web client will be available on port [5173](http://localhost:5173/). + ++ back to the top +
+ +## Commands + +### Backend development + +```bash +./gradlew bootRun # start the development server (docker lifecycle management built-in) + +./gradlew clean # clean the /build directory + +./gradlew test # test the application + +./gradlew assemble # build the application without prior testing + +./gradlew build # test and build the application +``` + +### Frontend development + +```bash +npm i # Install dependencies + +npm run dev # Start the development server + +npm run build # build the application + +npm run format # run prettier to check for any formatting errors + +npm run format:fix # run prettier to fix any formatting errors + +npm run lint # run eslint to check for any linting errors + +npm run lint:fix # run eslint to fix any linting errors +``` + ++ back to the top +
+ +## The onboarding process + +> [!IMPORTANT] +> The server requires users to complete the onboarding step before any API calls are accepted. If onboarding is not completed, requests will return `403 Forbidden` with the message `User has not completed the onboarding process`. + +Depending on your area of local development, you may or may not need to complete this process manually. + +### Backend development + +If you are working with the Spring Boot server only, you must create a user in the Clerk dashboard and generate a long-lived session token for authentication. + +You can generate a token for a given `User ID` (found via the **Users** tab) using the [Clerk backend API](https://clerk.com/docs/reference/api/overview) from any HTTP client of your choice. + +Once you have a valid token, call `POST /v1/clerk/onboarding` once, immediately after registering the user in the dashboard. If the request succeeds, **generate a new session token** (required to pull in updated token claims) and use that token to authenticate subsequent API requests. + +### Frontend development + +If you register a new user through the web client’s authentication pages, onboarding will be handled automatically via the `/onboarding` page redirect after successful registration. + +--- + + ++ back to the top +
diff --git a/README.md b/README.md index 69b35998..fb3181c1 100644 --- a/README.md +++ b/README.md @@ -1,39 +1,27 @@ - -
-
- 
-
-
- Architecture - • - Development - • - Web Client - • - REST Service -
-- back to the top -
\ No newline at end of file + +
+
+
+
+
+ Architecture + • + Development + • + Web Client + • + REST Service +
+
-
- 
-
-
- 
- back to the top -
\ No newline at end of file + +
+
+ 
+
+
+
+ back to the top +
diff --git a/apps/client/README.md b/apps/client/README.md index 64d5923e..f3048a8e 100644 --- a/apps/client/README.md +++ b/apps/client/README.md @@ -2,7 +2,7 @@
-
- 
-
-
- back to the top -
- -## Backend - - - -### Settling on a language and framework - -Before starting out on this project, I had long since settled on building a Java Spring Boot backend service. I had previously covered Node Express backend services, and didn't like the lack of structure in the framework. - -Prior to this project, I was new to Java. I learned the language initially to cover the [Princeton Algorithms I](https://www.coursera.org/learn/algorithms-part1) online course content. The strictness of a statically typed language (minus the sheer quantity of boilerplate) was really refreshing, coming from JS. - -Java and Spring are widely used in industry, and I liked the opinionated structure that the framework itself seemed to offer when I was doing my research. So, I thought I'd pick it up and expand my knowledge base. - -### Structuring the codebase - -I initially started out on the application using a [package-by-layer](https://medium.com/@akintopbas96/spring-boot-code-structure-package-by-layer-vs-package-by-feature-5331a0c911fe) approach. It's fine for simplicity and pretty much any Spring Boot demo will use this approach. But once you start working on more and more features, you're touching every layer of the application, and it gets messy fast. I sat down and thought about how best to refactor the project. - -I researched best practices for monolithic applications and [landed on a feature-based approach](https://github.com/nednella/echo/pull/65), with some kind of by-layer folder organisation within. A bit of a hybrid. - -Feature-based layering should really be done by considering what components within the feature are public (API layer) and which parts can be kept internal (package-private). Unfortunately, after all this time, Java doesn't offer a take on sub-packaging, and the structure looked messy, so I combed each feature into its own layered structure. - -I refactored with the single-responsibility principle in mind, intending on using the package-private access modifier as the default option. Public access is only used where required (controllers, DTOs, service interfaces, shared components). It's not perfect, but it's good enough. Adding application complexity became much easier and that was my goal. - -### Defining the API contract vs. the domain model - -I started building out the application by considering what information I wanted to return to the client for a given domain. - -#### Working example - -Profile requests should return the information you'd expect when visiting a user's page. Their unique identifier, some personalised details specific to the application, and when they first joined. Contextual information like follower counts, post counts and relationship status with the viewing user should also be included. - -```javascript -{ - "id": String, - "username": String, - "name": String | null, - "bio": String | null, - "location": String | null, - "image_url": String | null, - "created_at": String, - "metrics": { ... }, - "relationship": { ... } | null -} -``` - -By starting with the public-facing contract and shaping the underlying entities to support, I kept the models clean and logical. - -### Defining a consistent error model - -I took inspiration from the [Spotify Web API](https://developer.spotify.com/documentation/web-api) on this one. Their API errors are simple, the user receives an error status code and a message describing the issue. I opted to replicate this, but also included `timestamp` and `path` properties for additional context. - -```java -public record ErrorResponse( - Instant timestamp, - int status, - String message, - String path -) {} -``` - -Starting with the desired client response makes the implementation straightforward. Any explicitly handled exceptions (using Spring's `@ControllerAdvice` and `@ExceptionHandler` annotations) are formatted into the standardised response object. - -Any given application exception should be able to be mapped into a particular HTTP status code and a descriptive error message, so I opted to error enums per-feature implemented against an `ErrorCode` interface. Each declared error should contain its own HTTP status code, a message template, and the number of arguments that the template expects. This way, a template can be formatted while avoiding runtime errors, and tested against accordingly. - -```java -public interface ErrorCode { - - HttpStatus getStatus(); - - String getMessageTemplate(); - - int getExpectedArgs(); - - default String formatMessage(Object... args) { - ... - } - - default ApplicationException buildAsException(Object... args) { - ... - } - -} -``` - -Now, the client can always expect errors in a predictable format, regardless of how they originated, simplifying both application debugging and client integration. - -### Ensuring data integrity through validation - -This was done using the recommended Spring approach — where possible, client-side inputs passed through the request body are validated at the controllers using bean validation annotations within the mapped request objects. This makes it simple to declare validation messages per-request object and keeps the validation logic tightly coupled to the request objects themselves. - -In some cases, simple controller input validation doesn't suffice. Taking the post replies feature for example, when a post is created in reply to another, the request includes the ID of the parent post. In this case, existence of records are also validated within the application business logic. This way, invalid data never finds its way into the database, even when the request shape looks good. - -Finally, the database schema itself acts as a last wall of defence. Constraints and indexes ensure that even if bad data splips through, it cannot be persisted. - -For cases where the request itself is invalid, and not the contained client inputs, Spring does a good job of raising exceptions. Cases like malformed JSON, unsupported HTTP methods, or mismatched argument types (`MethodArgumentTypeMismatch`, `HttpMessageNotReadable`, `HttpRequestMethodNotSupported`) are all thrown automatically. - -With the correct exception handlers in place, annotations for field-level checks, service logic for domain rules and framework exceptions for general request errors all flow back to the client in a predictable format. - -### Choosing a DBMS - -Not much to talk about here. I knew before starting out that a relational model would be best suited. The project has a rigid data structure, and most `GET` requests would involve stitching various structures together. During research into the various options, Postgres came up as the most widely recommended option. - -### Mixing data access methods - -Spring offers [various approaches](https://docs.spring.io/spring-framework/reference/data-access.html) to data access within your application. You have [full-feature ORMs](https://docs.spring.io/spring-framework/reference/data-access/orm.html) like Hibernate and JPA, and some barebones abstractions like [Spring JDBC](https://docs.spring.io/spring-framework/reference/data-access/jdbc.html). - -For simple CRUD operations, I leaned on Spring Data JPA's [repository interfaces](https://docs.spring.io/spring-data/jpa/reference/repositories/definition.html). The basics are covered very well with these, and it's easy to [define custom query methods](https://docs.spring.io/spring-data/jpa/reference/repositories/query-methods-details.html) if the defaults don't suffice. The best part is that, since JPA generates the query method implementations, you don't need to write any integration tests. - -Complexity grew when working on `GET` requests for profiles and posts. These endpoints required complex data transfer objects (DTO) that combined data from multiple tables. Taking a look at a standard representation of a user's post: - -```javascript -{ - "id": String, - "parent_id": String | null, - "conversation_id": String, - "author": { - "id": String, - "username": String, - "name": String | null, - "image_url": String | null, - "relationship": { ... } | null - }, - "text": String, - "created_at": String, - "metrics": { ... }, - "relationship": { ... }, - "entities": { ... } -} -``` - -It doesn't just expose a record in the `post` table; there's a lot of additional context such as author details, post metrics, relationships and entities to give the client full context. - -At the time, I couldn't figure out how to model these requirements through an ORM alone. Instead, I discovered that if I used Spring JDBC, I could write SQL directly and map the resulting rows into the required shape using a `RowMapper`. This way, I could focus on writing efficient SQL functions that covered my requirements. - -### The trade-offs of using Spring JDBC - -There were positives to draw from the decision. Writing raw SQL forced me to understand it better and expand my knowledge. I became more comfortable joining and aggregating data, and explored new-to-me features like views and CTEs. By writing the queries myself, I could also easily look at query performance using `EXPLAIN ANALYZE`. - -But, there are some significant hindrances. Refactoring data access is a headache, and debugging raw SQL is a nightmare. If I touch a base query function like `fetch_posts`, I will have to carefully update every higher-level function that consumes it. If I introduce a bug in a query, I don't have any IDE support, and I've found that SQL errors are not the most descriptive. - -It works, but it's very easy to make mistakes and frustrating to maintain. - -### Testing the application - -I kept the backend tests pragmatic. Fast unit tests for behaviour, and a smaller set of integration tests to prove the application wiring (filters → controllers → services → repositories). - -#### Unit testing - -At the unit level, services are tested in isolation with dependencies mocked with Mockito so I can focus on business rules and exception handling. Controllers are tested with `@MockMvcTest` to ensure request validation and error mapping. Smaller components like mappers and utility functions are also unit tested. Essentially, any code that can be isolated and tested against, is. - -#### Integration testing - -For integration testing, I use [Testcontainers](https://testcontainers.com/) to spin up fresh Postgres instances alongside either a full Spring context (`@SpringBootTest`), or a narrower repository slice (`@DataJpaTest`). HTTP endpoints are tested with Spring's [WebTestClient](https://docs.spring.io/spring-framework/reference/testing/webtestclient.html) by sending real valid/invalid requests through the filter chain, controllers, services and persistence layer, covering: - -- Authentication and security filters -- Input validation -- Business logic -- Database integrity - -- back to the top -
- -## Authentication & security - -### Starting with username-password form authentication - -The application revolves around users and interactions between those users, and the groundwork for those social interactions starts with authentication. Developing features before authentication didn't make sense when the app revolves around knowing *who* the authenticated user is. - -When I began working on the backend, I spent time learning the Spring framework, including Spring Security, a trusted and highly customisable authentication and access-control framework. It supports [numerous popular authentication mechanisms](https://docs.spring.io/spring-security/reference/servlet/authentication/index.html) with minimal setup required. - -I started with the simplest: username–password authentication. With minimal configuration, you can have: - -- A robust authentication implementation backed by a custom `user` table that stores usernames and BCrypt hashed passwords -- [In-memory session persistence](https://docs.spring.io/spring-security/reference/servlet/authentication/session-management.html) using configurable session cookies -- Custom authentication endpoints (e.g. `/auth/login`, `/auth/logout`) instead of the default Spring login page -- Options for external cache or database integration such as [Redis](https://docs.spring.io/spring-session/reference/http-session.html), for horizontal scaling - -This approach met my early development requirements and allowed me to start layering on application features. - -### "Why switch to Clerk, then?" - -The long-term goal was to support both username–password authentication with the usual email verification “forgot password” flows, and OAuth integration with providers like GitHub and LinkedIn. - -While I could have extended my Spring Security setup and built out additional security features, it's deceptively complex to go beyond the basics, and it's not recommended to do this from scratch. It's much more straightforward to integrate with 3rd party authentication libraries or providers, and rolling your own auth only once the application scale requires you to do so. - -I compared several popular solutions from libraries to hosted providers — including Auth.js, Better Auth, Lucia, Auth0, Clerk and Supabase — and considered the following points: - -- **Frontend integration**: Is there a plug-and-play TypeScript SDK? Are there pre-built UI components? -- **Database control**: Can I manage my own `user` table, and if not, how do I maintain database sync? -- **Developer experience**: Are the docs reliable? Is there support if I run into issues? What about the risk of vendor lock-in? -- **Cost**: Is there a free tier for development? What happens if the project grows unexpectedly? - -In the end, I chose [Clerk](https://clerk.com/). Their developer experience is second-to-none: the dashboard UI is one of the best, the React integration is seamless, the docs answer most questions, and I can personally vouch that their email support fills the gaps. Their free tier plan covers 10,000 monthly active users, with a unique definition of "active" — a user who comes back 24+ hours after registration, and not just uses the application once. Within a few minutes of setup, I had production-grade authentication running on the client, issuing session tokens that my API could validate against. - -### The challenge of synchronising databases - -The trade-off for integrating with Clerk was losing control of the `user` table — Echo's entire social model depends on referencing a local `user` table! - -Clerk currently does not offer exposure to their underlying database, nor do they provide triggers to synchronise state directly. What they do offer (at the time of writing) is asynchronous, non-guaranteed [webhook](https://clerk.com/docs/webhooks/sync-data) events. - -Webhooks are useful for maintaining sync of PUTs and DELETEs, but not for critical actions like user registration. A new user should immediately be able to access and navigate the application without error, *guaranteed*. This requires the user to be present in the local database immediately after the point of registration. - -Clerk discusses an ["onboarding flow"](https://clerk.com/blog/add-onboarding-flow-for-your-application-with-clerk) that you can use to collect key information from your user post-registration, and use that information to drive application state. [I chose to adapt this flow with a twist](https://github.com/nednella/echo/pull/61). Rather than collecting information from the user, an onboarding page will automatically send an authenticated request to the server as soon as registration completes. From that request, I can run a database upsert: - -- If a user doesn't exist with the given Clerk ID, create one, and update the Clerk user as having completed the onboarding process -- If a user already exists, update (or skip) - -Once the server responds OK, the client can safely redirect the user to the core application, and the server can rely on webhooks for post-registration updates. - -- back to the top -
- -## Frontend - -WIP - -- back to the top -
- -## DevOps & CI/CD - -### Git workflow - -The repository follows [trunk-based development](https://trunkbaseddevelopment.com/) using short-lived feature branches. - -- All new work starts from the latest version of `main` -- Branches are kept small and focussed on a specific change -- Once a change is ready, a PR is opened back into `main` - -This way, `main` acts as the single source of truth and is always in a deployable state. - -### Conventional commits - -To standardise commit messages, the repository uses the [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) spec. There are a couple key benefits: - -1. **Readable history** - the log explains clearly what changes were made -2. **Automation** - tools can parse commit messages to generate version bumps and changelogs - -### Branch protection rules - -The highlighted workflow is enforced with specific branch protection rules. - -- A pull request is required before merging -- A linear commit history is required (squash merges only -- required for automation tooling) -- Specific status checks must pass before permitting a merge - -Checks include a `check-pr-title` job that enforces the Conventional Commits spec, and CI workflows for each application. As a result, every change that lands on `main` is meaningful and production-ready. - -### Continuous integration - -CI runs via GitHub Actions for every PR into `main`. Each application has it's own dedicated CI workflow that involves formatting, linting and testing the source code. - -To save time, CI workflows are [conditionally executed](https://github.com/dorny/paths-filter) based on whether the relevant source code was touched. - -### Continuous deployment - -Application deployment workflows run via GitHub Actions on tag pushes. - -The deployment process is automated with [release-please](https://github.com/googleapis/release-please). The tool works by parsing commits on a given branch (in this case, `main`) and looks for messages that match the [Conventional Commits](#conventional-commits) spec. When qualifying changes are detected, it automatically opens a **"Release PR"** with an updated changelog and a [semantic version](https://semver.org/) bump. - -Upon merging a release PR, the tool executes the following tasks: - -- Updates the `CHANGELOG.md` file -- Updates the relevant versioned files (e.g., `package.json`, `build-gradle`) -- Tags the commit with the updated application version number -- Creates a GitHub release based on this tag - -The tool is configured to look at each application individually using the [manifest-driven configuration](https://github.com/googleapis/release-please/blob/main/docs/manifest-releaser.md). As a result, each application maintains its own changelog and version, allowing deployment on a per-application basis. - -From there, GitHub Actions picks up a pushed tag and runs the relevant deployment pipeline. The end result is that deployment for a given application within the project requires no manual intevention, except one click on the green button within the release PR. - -The process is entirely hands-off and reliable when matched with the afore mentioned branch protection ruleset. Implementing this tool was probably the best thing I did for myself throughout the entire project. - -### Deployment services - -Not much to discuss here. This is really just a learning project and I don't expect to generate any organic traffic, so it's hosted with services that either have a generous free tier, or at most, a cheap but reliable hobby tier. - -The web client is currently hosted with [Vercel](https://vercel.com/home), and the backend service is hosted with [Railway](https://railway.com/). - -- back to the top -
- -## Finishing up - -WIP - ---- - - -- back to the top -
\ No newline at end of file + +
+
+
+
+
+ back to the top +
+ +## Backend + + + +### Settling on a language and framework + +Before starting out on this project, I had long since settled on building a Java Spring Boot backend service. I had previously covered Node Express backend services, and didn't like the lack of structure in the framework. + +Prior to this project, I was new to Java. I learned the language initially to cover the [Princeton Algorithms I](https://www.coursera.org/learn/algorithms-part1) online course content. The strictness of a statically typed language (minus the sheer quantity of boilerplate) was really refreshing, coming from JS. + +Java and Spring are widely used in industry, and I liked the opinionated structure that the framework itself seemed to offer when I was doing my research. So, I thought I'd pick it up and expand my knowledge base. + +### Structuring the codebase + +I initially started out on the application using a [package-by-layer](https://medium.com/@akintopbas96/spring-boot-code-structure-package-by-layer-vs-package-by-feature-5331a0c911fe) approach. It's fine for simplicity and pretty much any Spring Boot demo will use this approach. But once you start working on more and more features, you're touching every layer of the application, and it gets messy fast. I sat down and thought about how best to refactor the project. + +I researched best practices for monolithic applications and [landed on a feature-based approach](https://github.com/nednella/echo/pull/65), with some kind of by-layer folder organisation within. A bit of a hybrid. + +Feature-based layering should really be done by considering what components within the feature are public (API layer) and which parts can be kept internal (package-private). Unfortunately, after all this time, Java doesn't offer a take on sub-packaging, and the structure looked messy, so I combed each feature into its own layered structure. + +I refactored with the single-responsibility principle in mind, intending on using the package-private access modifier as the default option. Public access is only used where required (controllers, DTOs, service interfaces, shared components). It's not perfect, but it's good enough. Adding application complexity became much easier and that was my goal. + +### Defining the API contract vs. the domain model + +I started building out the application by considering what information I wanted to return to the client for a given domain. + +#### Working example + +Profile requests should return the information you'd expect when visiting a user's page. Their unique identifier, some personalised details specific to the application, and when they first joined. Contextual information like follower counts, post counts and relationship status with the viewing user should also be included. + +```javascript +{ + "id": String, + "username": String, + "name": String | null, + "bio": String | null, + "location": String | null, + "image_url": String | null, + "created_at": String, + "metrics": { ... }, + "relationship": { ... } | null +} +``` + +By starting with the public-facing contract and shaping the underlying entities to support, I kept the models clean and logical. + +### Defining a consistent error model + +I took inspiration from the [Spotify Web API](https://developer.spotify.com/documentation/web-api) on this one. Their API errors are simple, the user receives an error status code and a message describing the issue. I opted to replicate this, but also included `timestamp` and `path` properties for additional context. + +```java +public record ErrorResponse( + Instant timestamp, + int status, + String message, + String path +) {} +``` + +Starting with the desired client response makes the implementation straightforward. Any explicitly handled exceptions (using Spring's `@ControllerAdvice` and `@ExceptionHandler` annotations) are formatted into the standardised response object. + +Any given application exception should be able to be mapped into a particular HTTP status code and a descriptive error message, so I opted to use error enums per-feature, implemented against an `ErrorCode` interface. Each declared error should contain its own HTTP status code, a message template, and the number of arguments that the template expects. This way, a template can be formatted while avoiding runtime errors, and tested against accordingly. + +```java +public interface ErrorCode { + + HttpStatus getStatus(); + + String getMessageTemplate(); + + int getExpectedArgs(); + + default String formatMessage(Object... args) { + ... + } + + default ApplicationException buildAsException(Object... args) { + ... + } + +} +``` + +Now, the client can always expect errors in a predictable format, regardless of how they originated, simplifying both application debugging and client integration. + +### Ensuring data integrity through validation + +This was done using the recommended Spring approach — where possible, client-side inputs passed through the request body are validated at the controllers using bean validation annotations within the mapped request objects. This makes it simple to declare validation messages per-request object and keeps the validation logic tightly coupled to the request objects themselves. + +In some cases, simple controller input validation doesn't suffice. Taking the post replies feature for example, when a post is created in reply to another, the request includes the ID of the parent post. In this case, existence of records are also validated within the application business logic. This way, invalid data never finds its way into the database, even when the request shape looks good. + +Finally, the database schema itself acts as a last wall of defence. Constraints and indexes ensure that even if bad data slips through, it cannot be persisted. + +For cases where the request itself is invalid, and not the contained client inputs, Spring does a good job of raising exceptions. Cases like malformed JSON, unsupported HTTP methods, or mismatched argument types (`MethodArgumentTypeMismatch`, `HttpMessageNotReadable`, `HttpRequestMethodNotSupported`) are all thrown automatically. + +With the correct exception handlers in place, annotations for field-level checks, service logic for domain rules and framework exceptions for general request errors all flow back to the client in a predictable format. + +### Choosing a DBMS + +Not much to talk about here. I knew before starting out that a relational model would be best suited. The project has a rigid data structure, and most `GET` requests would involve stitching various structures together. During research into the various options, Postgres came up as the most widely recommended option. + +### Mixing data access methods + +Spring offers [various approaches](https://docs.spring.io/spring-framework/reference/data-access.html) to data access within your application. You have [full-feature ORMs](https://docs.spring.io/spring-framework/reference/data-access/orm.html) like Hibernate and JPA, and some bare bones abstractions like [Spring JDBC](https://docs.spring.io/spring-framework/reference/data-access/jdbc.html). + +For simple CRUD operations, I leaned on Spring Data JPA's [repository interfaces](https://docs.spring.io/spring-data/jpa/reference/repositories/definition.html). The basics are covered very well with these, and it's easy to [define custom query methods](https://docs.spring.io/spring-data/jpa/reference/repositories/query-methods-details.html) if the defaults don't suffice. The best part is that, since JPA generates the query method implementations, you don't need to write any integration tests. + +Complexity grew when working on `GET` requests for profiles and posts. These endpoints required complex data transfer objects (DTO) that combined data from multiple tables. Taking a look at a standard representation of a user's post: + +```javascript +{ + "id": String, + "parent_id": String | null, + "conversation_id": String, + "author": { + "id": String, + "username": String, + "name": String | null, + "image_url": String | null, + "relationship": { ... } | null + }, + "text": String, + "created_at": String, + "metrics": { ... }, + "relationship": { ... }, + "entities": { ... } +} +``` + +It doesn't just expose a record in the `post` table; there's a lot of additional context such as author details, post metrics, relationships and entities to give the client full context. + +At the time, I couldn't figure out how to model these requirements through an ORM alone. Instead, I discovered that if I used Spring JDBC, I could write SQL directly and map the resulting rows into the required shape using a `RowMapper`. This way, I could focus on writing efficient SQL functions that covered my requirements. + +### The trade-offs of using Spring JDBC + +There were positives to draw from the decision. Writing raw SQL forced me to understand it better and expand my knowledge. I became more comfortable joining and aggregating data, and explored new-to-me features like views and CTEs. By writing the queries myself, I could also easily look at query performance using `EXPLAIN ANALYZE`. + +But, there are some significant hindrances. Refactoring data access is a headache, and debugging raw SQL is a nightmare. If I touch a base query function like `fetch_posts`, I will have to carefully update every higher-level function that consumes it. If I introduce a bug in a query, I don't have any IDE support, and I've found that SQL errors are not the most descriptive. + +It works, but it's very easy to make mistakes and frustrating to maintain. + +### Testing the application + +I kept the backend tests pragmatic. Fast unit tests for behaviour, and a smaller set of integration tests to prove the application wiring (filters → controllers → services → repositories). + +#### Unit testing + +At the unit level, services are tested in isolation with dependencies mocked with Mockito so I can focus on business rules and exception handling. Controllers are tested with `@MockMvcTest` to ensure request validation and error mapping. Smaller components like mappers and utility functions are also unit tested. Essentially, any code that can be isolated and tested against, is. + +#### Integration testing + +For integration testing, I use [Testcontainers](https://testcontainers.com/) to spin up fresh Postgres instances alongside either a full Spring context (`@SpringBootTest`), or a narrower repository slice (`@DataJpaTest`). HTTP endpoints are tested with Spring's [WebTestClient](https://docs.spring.io/spring-framework/reference/testing/webtestclient.html) by sending real valid/invalid requests through the filter chain, controllers, services and persistence layer, covering: + +- Authentication and security filters +- Input validation +- Business logic +- Database integrity + ++ back to the top +
+ +## Authentication & security + +### Starting with username-password form authentication + +The application revolves around users and interactions between those users, and the groundwork for those social interactions starts with authentication. Developing features before authentication didn't make sense when the app revolves around knowing _who_ the authenticated user is. + +When I began working on the backend, I spent time learning the Spring framework, including Spring Security, a trusted and highly customisable authentication and access-control framework. It supports [numerous popular authentication mechanisms](https://docs.spring.io/spring-security/reference/servlet/authentication/index.html) with minimal setup required. + +I started with the simplest: username–password authentication. With minimal configuration, you can have: + +- A robust authentication implementation backed by a custom `user` table that stores usernames and BCrypt hashed passwords +- [In-memory session persistence](https://docs.spring.io/spring-security/reference/servlet/authentication/session-management.html) using configurable session cookies +- Custom authentication endpoints (e.g. `/auth/login`, `/auth/logout`) instead of the default Spring login page +- Options for external cache or database integration such as [Redis](https://docs.spring.io/spring-session/reference/http-session.html), for horizontal scaling + +This approach met my early development requirements and allowed me to start layering on application features. + +### "Why switch to Clerk, then?" + +The long-term goal was to support both username–password authentication with the usual email verification “forgot password” flows, and OAuth integration with providers like GitHub and LinkedIn. + +While I could have extended my Spring Security setup and built out additional security features, it's deceptively complex to go beyond the basics, and it's not recommended to do this from scratch. It's much more straightforward to integrate with 3rd party authentication libraries or providers, and rolling your own auth only once the application scale requires you to do so. + +I compared several popular solutions from libraries to hosted providers — including Auth.js, Better Auth, Lucia, Auth0, Clerk and Supabase — and considered the following points: + +- **Frontend integration**: Is there a plug-and-play TypeScript SDK? Are there pre-built UI components? +- **Database control**: Can I manage my own `user` table, and if not, how do I maintain database sync? +- **Developer experience**: Are the docs reliable? Is there support if I run into issues? What about the risk of vendor lock-in? +- **Cost**: Is there a free tier for development? What happens if the project grows unexpectedly? + +In the end, I chose [Clerk](https://clerk.com/). Their developer experience is second-to-none: the dashboard UI is one of the best, the React integration is seamless, the docs answer most questions, and I can personally vouch that their email support fills the gaps. Their free tier plan covers 10,000 monthly active users, with a unique definition of "active" — a user who comes back 24+ hours after registration, and not just uses the application once. Within a few minutes of setup, I had production-grade authentication running on the client, issuing session tokens that my API could validate against. + +### The challenge of synchronising databases + +The trade-off for integrating with Clerk was losing control of the `user` table — Echo's entire social model depends on referencing a local `user` table! + +Clerk currently does not offer exposure to their underlying database, nor do they provide triggers to synchronise state directly. What they do offer (at the time of writing) is asynchronous, non-guaranteed [webhook](https://clerk.com/docs/webhooks/sync-data) events. + +Webhooks are useful for maintaining sync of PUTs and DELETEs, but not for critical actions like user registration. A new user should immediately be able to access and navigate the application without error, _guaranteed_. This requires the user to be present in the local database immediately after the point of registration. + +Clerk discusses an ["onboarding flow"](https://clerk.com/blog/add-onboarding-flow-for-your-application-with-clerk) that you can use to collect key information from your user post-registration, and use that information to drive application state. [I chose to adapt this flow with a twist](https://github.com/nednella/echo/pull/61). Rather than collecting information from the user, an onboarding page will automatically send an authenticated request to the server as soon as registration completes. From that request, I can run a database upsert: + +- If a user doesn't exist with the given Clerk ID, create one, and update the Clerk user as having completed the onboarding process +- If a user already exists, update (or skip) + +Once the server responds OK, the client can safely redirect the user to the core application, and the server can rely on webhooks for post-registration updates. + ++ back to the top +
+ +## Frontend + +WIP + ++ back to the top +
+ +## DevOps & CI/CD + +### Git workflow + +The repository follows [trunk-based development](https://trunkbaseddevelopment.com/) using short-lived feature branches. + +- All new work starts from the latest version of `main` +- Branches are kept small and focussed on a specific change +- Once a change is ready, a PR is opened back into `main` + +This way, `main` acts as the single source of truth and is always in a deployable state. + +### Conventional commits + +To standardise commit messages, the repository uses the [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) spec. There are a couple key benefits: + +1. **Readable history** - the log explains clearly what changes were made +2. **Automation** - tools can parse commit messages to generate version bumps and changelogs + +### Branch protection rules + +The highlighted workflow is enforced with specific branch protection rules. + +- A pull request is required before merging +- A linear commit history is required (squash merges only -- required for automation tooling) +- Specific status checks must pass before permitting a merge + +Checks include a `check-pr-title` job that enforces the Conventional Commits spec, and CI workflows for each application. As a result, every change that lands on `main` is meaningful and production-ready. + +### Continuous integration + +CI runs via GitHub Actions for every PR into `main`. Each application has it's own dedicated CI workflow that involves formatting, linting and testing the source code. + +To save time, CI workflows are [conditionally executed](https://github.com/dorny/paths-filter) based on whether the relevant source code was touched. + +### Continuous deployment + +Application deployment workflows run via GitHub Actions on tag pushes. + +The deployment process is automated with [release-please](https://github.com/googleapis/release-please). The tool works by parsing commits on a given branch (in this case, `main`) and looks for messages that match the [Conventional Commits](#conventional-commits) spec. When qualifying changes are detected, it automatically opens a **"Release PR"** with an updated changelog and a [semantic version](https://semver.org/) bump. + +Upon merging a release PR, the tool executes the following tasks: + +- Updates the `CHANGELOG.md` file +- Updates the relevant versioned files (e.g., `package.json`, `build-gradle`) +- Tags the commit with the updated application version number +- Creates a GitHub release based on this tag + +The tool is configured to look at each application individually using the [manifest-driven configuration](https://github.com/googleapis/release-please/blob/main/docs/manifest-releaser.md). As a result, each application maintains its own changelog and version, allowing deployment on a per-application basis. + +From there, GitHub Actions picks up a pushed tag and runs the relevant deployment pipeline. The end result is that deployment for a given application within the project requires no manual intervention, except one click on the green button within the release PR. + +The process is entirely hands-off and reliable when matched with the afore mentioned branch protection ruleset. Implementing this tool was probably the best thing I did for myself throughout the entire project. + +### Deployment services + +Not much to discuss here. This is really just a learning project and I don't expect to generate any organic traffic, so it's hosted with services that either have a generous free tier, or at most, a cheap but reliable hobby tier. + +The web client is currently hosted with [Vercel](https://vercel.com/home), and the backend service is hosted with [Railway](https://railway.com/). + ++ back to the top +
+ +## Finishing up + +WIP + +--- + + ++ back to the top +
diff --git a/docs/clerk-setup.md b/docs/clerk-setup.md index 08c212b5..91a8581e 100644 --- a/docs/clerk-setup.md +++ b/docs/clerk-setup.md @@ -1,169 +1,169 @@ - -
-
- 
-
-
-### Adding custom session token claims
-
-> [!CAUTION]
-> If the Clerk application is not configured to include the required custom claims on the bearer tokens, all authenticated HTTP requests sent to the REST API will return `403 Forbidden`.
-
-1. Head to the **Configure** tab within your newly created Clerk application
-2. Head to the **Session management → Sessions** section
-3. Customise the session token with the following JSON template
-
-
-
-- back to the top -
- -## Obtaining the API keys - -1. Head to the **Configure** tab within your newly created Clerk application -2. Head to the **Developers → API Keys** section - -For the React Web Client, you will need: - -- The **Publishable Key** → This is your **VITE_CLERK_PUBLISHABLE_KEY** - -For the Spring Boot REST API, you will need: - -- A **Secret Key** → This is your **CLERK_SECRET_KEY** -- The **Frontend API URL** → This is your **CLERK_ISSUER_URI** -- The **JWKS URL** → This is your **CLERK_JWK_SET_URI** - -
-
-- back to the top -
- -## OPTIONAL: Webhooks - -> [!IMPORTANT] -> You will require a tunneling service to expose your localhost port to the web. [Ngrok is the recommended option by Clerk](https://clerk.com/docs/webhooks/sync-data), but you may use any tunneling service of your choice. - -> [!IMPORTANT] -> The Echo REST API is currently only configured to accept `user` events; any other event type will be rejected. - -1. Spin up a tunnel pointing to `localhost:8080`, or the port the REST API will run on if you changed it from the default -2. Head to the **Configure** tab within your newly created Clerk application -3. Head to the **Developers → Webhooks** section -4. Add a new endpoint, set the **Webhook URL**, subscribe to all `user` events, then create -5. Locate your **Signing Secret** from the endpoint panel once created → This is your **CLERK_WEBHOOK_SIGNING_SECRET** - -
-
-- back to the top -
- -## OPTIONAL: Local integration testing - -As of **Sept. 2025**, Clerk does not currently support testing environments within the same Clerk application. The current workaround to run local integration tests is to create **a second identical Clerk application** (e.g., append the application name with **CI/CD** for clarity). - -Create the following JWT templates exactly in the CI/CD application. These templates simulate malformed or missing bearer token claims to test failure cases in the Spring Security filter chain. - -``` -Name: malformed-echo_id-claim -Token lifetime: 3600 - -{ - "echo_id": "not_a_valid_UUID", - "onboarded": true -} -``` - -``` -Name: malformed-onboarded-claim -Token lifetime: 3600 - -{ - "echo_id": "{{user.external_id || null}}", - "onboarded": "not_a_valid_boolean" -} -``` - -``` -Name: missing-echo_id-claim -Token lifetime: 3600 - -{ - "onboarded": "{{user.public_metadata.onboardingComplete || false}}" -} -``` - -``` -Name: missing-onboarded-claim -Token lifetime: 3600 - -{ - "echo_id": "{{user.external_id || null}}" -} -``` - ---- - - -- back to the top -
\ No newline at end of file + +
+
+
+
+
+### Adding custom session token claims
+
+> [!CAUTION]
+> If the Clerk application is not configured to include the required custom claims on the bearer tokens, all authenticated HTTP requests sent to the REST API will return `403 Forbidden`.
+
+1. Head to the **Configure** tab within your newly created Clerk application
+2. Head to the **Session management → Sessions** section
+3. Customise the session token with the following JSON template
+
+
+
++ back to the top +
+ +## Obtaining the API keys + +1. Head to the **Configure** tab within your newly created Clerk application +2. Head to the **Developers → API Keys** section + +For the React Web Client, you will need: + +- The **Publishable Key** → This is your **VITE_CLERK_PUBLISHABLE_KEY** + +For the Spring Boot REST API, you will need: + +- A **Secret Key** → This is your **CLERK_SECRET_KEY** +- The **Frontend API URL** → This is your **CLERK_ISSUER_URI** +- The **JWKS URL** → This is your **CLERK_JWK_SET_URI** + +
+
++ back to the top +
+ +## OPTIONAL: Webhooks + +> [!IMPORTANT] +> You will require a tunnelling service to expose your localhost port to the web. [Ngrok is the recommended option by Clerk](https://clerk.com/docs/webhooks/sync-data), but you may use any tunnelling service of your choice. + +> [!IMPORTANT] +> The Echo REST API is currently only configured to accept `user` events; any other event type will be rejected. + +1. Spin up a tunnel pointing to `localhost:8080`, or the port the REST API will run on if you changed it from the default +2. Head to the **Configure** tab within your newly created Clerk application +3. Head to the **Developers → Webhooks** section +4. Add a new endpoint, set the **Webhook URL**, subscribe to all `user` events, then create +5. Locate your **Signing Secret** from the endpoint panel once created → This is your **CLERK_WEBHOOK_SIGNING_SECRET** + +
+
++ back to the top +
+ +## OPTIONAL: Local integration testing + +As of **Sept. 2025**, Clerk does not currently support testing environments within the same Clerk application. The current workaround to run local integration tests is to create **a second identical Clerk application** (e.g., append the application name with **CI/CD** for clarity). + +Create the following JWT templates exactly in the CI/CD application. These templates simulate malformed or missing bearer token claims to test failure cases in the Spring Security filter chain. + +``` +Name: malformed-echo_id-claim +Token lifetime: 3600 + +{ + "echo_id": "not_a_valid_UUID", + "onboarded": true +} +``` + +``` +Name: malformed-onboarded-claim +Token lifetime: 3600 + +{ + "echo_id": "{{user.external_id || null}}", + "onboarded": "not_a_valid_boolean" +} +``` + +``` +Name: missing-echo_id-claim +Token lifetime: 3600 + +{ + "onboarded": "{{user.public_metadata.onboardingComplete || false}}" +} +``` + +``` +Name: missing-onboarded-claim +Token lifetime: 3600 + +{ + "echo_id": "{{user.external_id || null}}" +} +``` + +--- + + ++ back to the top +