SeaweedFS - The On-Prem S3 Bucket

What is SeaweedFS? Link to heading

SeaweedFS is a distributed object storage system designed to store and serve billions of files quickly. It’s S3-compatible, making it an excellent choice for on-premises deployments that need to integrate with cloud-native tooling. Unlike traditional distributed file systems, SeaweedFS focuses on simplicity and performance, offering a lightweight alternative to solutions like MinIO or Ceph.

The key advantage? You get AWS S3-compatible APIs without the cloud costs or latency, perfect for edge computing, hybrid cloud architectures, and scenarios where data sovereignty matters.

Installation using Docker Compose Link to heading

The best way to set up SeaweedFS is through Docker. I wrote a Docker Compose YAML file to create the SeaweedFS container. I am using a Raspberry Pi to host this Docker container, but feel free to host it on any device.

services:
  seaweedfs:
    image: chrislusf/seaweedfs:latest
    command: >
      server
      -s3
      -s3.port=8333
      -filer
      -s3.config=/config/s3.json
      -dir=/data
      -ip=seaweedfs
    volumes:
      - ./data:/data
      - ./s3config:/config   # directory containing s3.json
    ports:
      - "8333:8333"
      - "9333:9333"
      - "8080:8080"
      - "8888:8888"

This command starts SeaweedFS with:

  • Port 9333: Master server
  • Port 8333: S3 API endpoint server
  • Port 8080: S3 WebUI
  • Persistent storage mounted at /data/seaweedfs

We also need to create a directory called s3config and a file inside called s3.json. Inside our s3.json file, we define the access key and secret key of the identity we are creating. An example would be the following:

{
  "identities": [
    {
      "name": "seaweedfs",
      "credentials": [
        {
          "accessKey": "seaweedfs",
          "secretKey": "seaweedfs"
        }
      ],
      "actions": [
        "Read",
        "Write",
        "List",
        "Tagging",
        "Admin"
      ],
      "resources": [
        "arn:seaweedfs:::*"
      ]
    }
  ]
}

Once we set up our SeaweedFS integration, we can run the command docker compose up -d to run the container in detached mode. Let’s add Tailscale integration to the Raspberry Pi so we can connect to it remotely.

Simply install Tailscale on your Docker host and configure SeaweedFS to bind to the Tailscale interface. Your storage cluster becomes accessible via your private Tailscale network, eliminating the need for complex VPN configurations. Make sure you copy the IP Address of the Tailscale Node.

AWS Configuration Link to heading

Now that SeaweedFS is installed, we need to configure it with aws. To do that, we need to run the command:

aws configure --profile <PROFILE NAME>

I am going to use this command for SeaweedFS: aws configure --profile seaweedfs. You will be asked for the Access Key, Secret Key, Region, and Format. The two keys should match what is in the s3.json file. For the region, use us-east-1 and for the format use json.

Now you can test it out with the following command:

aws --profile <PROFILE NAME> --endpoint http://<TAILSCALE IP>:8333 s3 mb s3://my-tf-backend-seaweedfs
aws --profile <PROFILE NAME> --endpoint http://<TAILSCALE IP>:8333 s3 ls

You should see the bucket named my-tf-backend-seaweedfs.

OpenTofu with SeaweedFS Backend Link to heading

As I was performing research on SeaweedFS and how similar it is to S3, I stumbled upon a question: Can I use it as an OpenTofu backend the same way I would use S3? The answer to this is YES! …with a caveat. You see, this method only works with creating on-premises resources, which makes sense. If I was going to provision cloud resources whether it be in AWS, Azure, GCP, or OCI, I would use the CSP (Cloud Service Provider) as the backend. But for on-premises infrastructure creation, this method works great!

Can I apply Object Lock and Versioning to the SeaweedFS bucket? Link to heading

Yes! SeaweedFS supports both versioning and object locking on S3 buckets. This was successfully tested and works as expected:

# Enable versioning
aws --profile <PROFILE NAME> --endpoint http://<TAILSCALE IP>:8333 s3api put-bucket-versioning --bucket <S3 BUCKET> --versioning-configuration Status=Enabled

aws --profile <PROFILE NAME> --endpoint http://<TAILSCALE IP>:8333 s3api put-object-lock-configuration --bucket <S3 BUCKET> --object-lock-configuration '{
    "ObjectLockEnabled": "Enabled",
    "Rule": {
        "DefaultRetention": {
            "Mode": "GOVERNANCE",
            "Days": 30
        }
    }
}'

This makes SeaweedFS production-ready for critical infrastructure state management.

OpenTofu Backend File Link to heading

I created a file called backend.tf that consists of the OpenTofu backend configuration. It looks like this:

terraform {
  backend "s3" {
    bucket       = "my-tf-backend-seaweedfs" # bucket you created in SeaweedFS
    key          = "opentofu.tfstate"
    region       = "us-east-1" # any valid string, SeaweedFS won't care
    endpoint     = "http://<TAILSCALE IP>:8333"
    access_key   = "seaweedfs"
    secret_key   = "seaweedfs"
    encrypt      = true
    use_lockfile = true

    # Important when using non-AWS S3:
    skip_credentials_validation = true
    skip_metadata_api_check     = true
    skip_region_validation      = true
    use_path_style              = true
  }
}

NOTE: The file above doesn’t do a good job at masking the access_key and secret_key. Your best bet is to use something like OpenBao or Environment Variables

OpenTofu Main File Link to heading

Remember how I said this method works only with On-Premise Infrastructure Creation? Well, let’s create a simple local-exec resource that runs echo Hello World anytime we run tofu apply.

resource "null_resource" "hello_world" {
  triggers = {
    always_run = timestamp() # forces execution every apply
  }

  provisioner "local-exec" {
    command = "echo Hello World"
  }
}

OpenTofu Commands Link to heading

  1. First we need to initialize our OpenTofu file. Run the command:
tofu init
  1. Next run the command to apply the changes:
tofu apply

DO NOT TYPE YES JUST YET!

Type this command to see the Object Lock working on your SeaweedFS bucket:

aws --profile <PROFILE NAME> --endpoint http://<TAILSCALE IP>:8333 s3 ls s3://my-tf-backend-seaweedfs/

You should see a file called opentofu.tfstate.tflock which means Object Lock works!

  1. Now you can type yes

  2. Now if you retype the aws s3 ls command on the bucket, you should see your OpenTofu statefile

aws --profile <PROFILE NAME> --endpoint http://<TAILSCALE IP>:8333 s3 ls s3://my-tf-backend-seaweedfs/

Best Use Cases for OpenTofu Backend Link to heading

SeaweedFS works best for on-premises resources rather than cloud-oriented infrastructure. Ideal scenarios include:

  • VMware vSphere infrastructure
  • Kubernetes clusters (on-prem or edge)
  • Local or remote physical resources
  • Hybrid environments where state needs to remain on-premises for compliance

For pure cloud infrastructure, native cloud backends (AWS S3, Azure Blob, GCS) are still the better choice due to lower latency and tighter integration.

Conclusion and Next Steps Link to heading

Throughout this exploration, we’ve covered the fundamentals of SeaweedFS as an on-premises S3-compatible storage solution. We walked through Docker-based installation, integrated it with Tailscale for secure remote access, and successfully demonstrated its capability as an OpenTofu backend with full support for versioning and object locking. The key takeaway? SeaweedFS provides a production-ready alternative to cloud storage for on-premises infrastructure management, particularly for VMware, Kubernetes, and edge computing scenarios.

But we’re just scratching the surface. The next logical steps involve two critical areas:

S3 Replication with rclone Link to heading

The first priority is implementing automated S3 replication between SeaweedFS and AWS S3 using rclone. This will enable:

  • Scheduled backups from edge devices to the cloud
  • Disaster recovery capabilities
  • Hybrid storage strategies that balance cost and performance

We’ll explore both push-based (on-prem cron jobs) and pull-based (AWS EventBridge + Lambda) replication patterns to determine which approach best fits different use cases. The goal is to create a resilient data pipeline that keeps mission-critical data synchronized across on-premises and cloud environments.

SeaweedFS CLI Wrapper Link to heading

The second initiative is building a custom CLI wrapper that simplifies switching between multiple S3 buckets and endpoints. Instead of constantly specifying --profile and --endpoint flags, imagine a tool that:

  • Stores endpoint configurations in OpenBao
  • Automatically injects credentials as environment variables
  • Provides a unified interface: seaweedfs s3 ls instead of the verbose AWS CLI syntax
  • Enables seamless context switching between SeaweedFS, AWS S3, and other S3-compatible providers

This wrapper would dramatically improve developer experience and operational efficiency, especially in environments managing multiple storage backends.

Stay tuned for follow-up articles diving deep into these implementations. The combination of SeaweedFS, Rclone automation, and intelligent CLI tooling creates a powerful foundation for modern hybrid cloud architectures.