Skip to content

Add idempotence-s3fs-nio module for S3 bucket test resource generation #113

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
Copilot wants to merge 4 commits into
base: master
Choose a base branch
from
Draft

Add idempotence-s3fs-nio module for S3 bucket test resource generation #113

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
36ac40c
Add idempotence-s3fs-nio module for S3 bucket test resource generation
Copilot Mar 3, 2026
46fc0ad
Rename system property to org.carlspring.testing.idempotence.s3fs.bas…
Copilot Mar 3, 2026
4de4179
Add S3BasicFunctionalityTest with MinIO Testcontainer and lazy S3Reso…
Copilot Mar 3, 2026
90422b0
Rename S3BasicFunctionalityTest to MinIO variant and add S3BasicFunct…
Copilot Mar 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 55 additions & 0 deletions docs/content/developers-guide/modules/idempotence-s3fs-nio.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
# `idempotence-s3fs-nio`

This Maven module adds support for generating test files in S3 buckets using the
[carlspring/s3fs-nio](https://github.com/carlspring/s3fs-nio) library.

## Directory Structure (S3)

Test resources are copied to an S3 URI of the form:

```
s3:///[bucket-name]/[prefix]/[TestClass]-[testMethod]/[resource-path]
```

For example, with the default configuration:

```
s3:///idempotence-test-resources/MyTest-testSingleFile/foo.txt
```

## Configuration

Set the `org.carlspring.testing.idempotence.s3fs.basedir` system property to an S3 URI:

| Example | Description |
|---------|-------------|
| `s3:///my-bucket` | Default AWS S3 endpoint, bucket `my-bucket` |
| `s3:///my-bucket/prefix` | Default AWS S3 endpoint, bucket `my-bucket`, key prefix `prefix` |
| `s3://localhost:9090/my-bucket` | Custom endpoint (e.g. MinIO), bucket `my-bucket` |

If no system property is set, the default base directory is `s3:///idempotence-test-resources`.

## Usage

Use the `S3TestResourceExtension` JUnit Jupiter extension instead of (or in addition to) the
standard `TestResourceExtension`:

```java
@ExtendWith(S3TestResourceExtension.class)
class MyTest {

@Test
@TestResources(@TestResource(source = "classpath:/foo.txt"))
void testSingleFile() {
// The file is available in S3 at:
// s3:///idempotence-test-resources/MyTest-testSingleFile/foo.txt
}

}
```

## S3 Credentials

S3 credentials and region are resolved by the AWS SDK's default credential provider chain.
For local development with MinIO, configure the endpoint via the system property and supply
credentials via the standard AWS environment variables or `~/.aws/credentials`.
164 changes: 164 additions & 0 deletions idempotence-s3fs-nio/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,164 @@
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">

<modelVersion>4.0.0</modelVersion>

<parent>
<groupId>org.carlspring.testing.idempotence</groupId>
<artifactId>idempotence-parent</artifactId>
<version>1.0.0-SNAPSHOT</version>
<relativePath>../idempotence-parent</relativePath>
</parent>

<groupId>org.carlspring.testing.idempotence</groupId>
<artifactId>idempotence-s3fs-nio</artifactId>
<version>1.0.0-SNAPSHOT</version>
<packaging>jar</packaging>

<name>Idempotence S3FS NIO</name>
<description>S3 extension for the Idempotence project using s3fs-nio</description>
<url>https://carlspring.github.io/idempotence/</url>

<properties>
<version.s3fs-nio>3.0.0</version.s3fs-nio>
<version.testcontainers>1.21.3</version.testcontainers>
</properties>

<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
</plugin>

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-javadoc-plugin</artifactId>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-source-plugin</artifactId>
</plugin>

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
</plugin>
</plugins>
</build>

<dependencies>
<dependency>
<groupId>org.carlspring.testing.idempotence</groupId>
<artifactId>idempotence-core</artifactId>
<version>${project.version}</version>
</dependency>

<dependency>
<groupId>org.carlspring.cloud.aws</groupId>
<artifactId>s3fs-nio</artifactId>
<version>${version.s3fs-nio}</version>
</dependency>

<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-api</artifactId>
</dependency>
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-engine</artifactId>
</dependency>

<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
</dependency>

<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-core</artifactId>
</dependency>
<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
</dependency>

<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>

<dependency>
<groupId>org.testcontainers</groupId>
<artifactId>testcontainers</artifactId>
<version>${version.testcontainers}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.testcontainers</groupId>
<artifactId>junit-jupiter</artifactId>
<version>${version.testcontainers}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.testcontainers</groupId>
<artifactId>minio</artifactId>
<version>${version.testcontainers}</version>
<scope>test</scope>
</dependency>
</dependencies>

<licenses>
<license>
<name>The Apache License, Version 2.0</name>
<url>https://www.apache.org/licenses/LICENSE-2.0.txt</url>
<distribution>repo</distribution>
</license>
</licenses>

<developers>
<developer>
<id>carlspring</id>
<name>Martin Todorov</name>
<email>carlspring@gmail.com</email>
<url>https://github.com/carlspring</url>
</developer>
</developers>

<scm>
<url>https://github.com/carlspring/idempotence</url>
<connection>scm:git:git://github.com/carlspring/idempotence.git</connection>
<developerConnection>scm:git:ssh://git@github.com:carlspring/idempotence.git</developerConnection>
<tag>${project.version}</tag>
</scm>

<issueManagement>
<system>GitHub Issues</system>
<url>https://github.com/carlspring/idempotence/issues</url>
</issueManagement>

<ciManagement>
<system>GitHub Actions</system>
<url>https://github.com/carlspring/idempotence/actions</url>
</ciManagement>

<distributionManagement>
<repository>
<id>central</id>
<name>Maven Central Release Repository</name>
<url>https://s01.oss.sonatype.org/service/local/staging/deploy/maven2/</url>
</repository>
<snapshotRepository>
<id>central</id>
<name>Maven Central Snapshot Repository</name>
<url>https://s01.oss.sonatype.org/content/repositories/snapshots/</url>
</snapshotRepository>
</distributionManagement>

</project>
46 changes: 46 additions & 0 deletions ...src/main/java/org/carlspring/testing/idempotence/config/S3fsNioIdempotenceProperties.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
package org.carlspring.testing.idempotence.config;

import org.springframework.stereotype.Service;

/**
* S3-specific implementation of {@link AbstractIdempotenceProperties} that uses
* an S3 URI as the base directory for test resources.
* <p>
* The default base directory is {@code s3:///idempotence-test-resources}, which points
* to a bucket named {@code idempotence-test-resources} on the default AWS S3 endpoint.
* Override the {@code org.carlspring.testing.idempotence.s3fs.basedir} system property to
* specify a different S3 URI (e.g. {@code s3://localhost:9090/my-bucket/prefix} for
* a local S3-compatible server such as MinIO).
* </p>
*
* @author carlspring
*/
@Service
public class S3fsNioIdempotenceProperties
extends AbstractIdempotenceProperties
{

private String basedir = System.getProperty("org.carlspring.testing.idempotence.s3fs.basedir") != null ?
System.getProperty("org.carlspring.testing.idempotence.s3fs.basedir") :
"s3:///idempotence-test-resources";

/**
* Creates a new instance of {@link S3fsNioIdempotenceProperties}.
*/
public S3fsNioIdempotenceProperties()
{
}

@Override
public String getBasedir()
{
return basedir;
}

@Override
public void setBasedir(String basedir)
{
this.basedir = basedir;
}

}
Loading
Loading