How to Mount an Amazon S3 Bucket as a Local Drive using cunoFS

Amazon S3 object storage is very popular due to its relatively low cost and its scalability. However, working with S3 isn’t as straightforward as working with a local filesystem. Object storage is conceptually different from files on a filesystem. And if you already have a set of tools and workflows you use that rely on files and folders, changing them to use object storage, through AWS CLI commands or the S3 SDK, is painful and requires replacing quite a bit of code.

With cunoFS, you can mount S3 buckets as folders and access objects inside them as files. This makes it possible to use almost any file-based application, including custom software but also machine learning tools like PyTorch, media tools like FFmpeg, and even high-performance computing workloads, with S3 storage.

Read on to find out how cunoFS works and how to use it to mount S3 buckets on Linux, on Windows with WSL, and on Mac with Docker.

Why mount an S3 bucket as a local drive

Editing a single small file stored on S3 is easy — you can just grab it from S3 via the AWS CLI, edit the file, and then upload the new version. Changing a large volume of small and large files isn’t practical with this manual process, so you would need to update scripts to use the AWS CLI to download and re-upload each file, or use the S3 SDK and integrate it into your code.

All this is not usually as simple as switching one filesystem command for its S3 equivalent. As a result, you might end up rewriting big chunks of your application code. For some applications, especially ones that read and write large files, changing to object storage might be way too complicated and time-consuming.

A solution to this problem is to make your S3 bucket available as a local drive, or “mount” it (to use the UNIX term) as a local drive. This way, you won’t need to change your applications, and the filesystem will transparently manage the connection between your local “drive” representation of the bucket and the S3 bucket itself.

Many tools can mount a bucket, but they are slow

There are many tools that can mount S3 storage as a local drive, including s3fs, mount-s3, and goofys. The problem with these tools is that, while they do work, their performance is very slow, making them unsuitable for regular use with file-based applications.

s3fs and goofys are slow, for example, because they are based on FUSE. FUSE introduces significant overheads – each time an application does an operation on the virtual file system, there are typically at least four context switches involved between user-space and the kernel to service the request. Read more in this article if you’re interested in the details.

In contrast, cunoFS provides native-filesystem access to S3 storage, without FUSE, and is much faster than the alternatives, including enterprise solutions. Here is an example comparison of upload duration versus other solutions; in this case we’re writing Linux kernel source files (74999 files, 1.21 GiB) to S3:

  • s3fs – cunoFS is 61x faster
  • goofys – cunoFS is 38x faster
  • s3ql – cunoFS is 12x faster
  • FSx Lustre – cunoFS is 2x faster (18x faster on read operations)
  • EFS – cunoFS is 2.7x faster (30x faster on read operations)
Bar graph comparing read and write times for cunoFS versus other solutions.
cunoFS is faster than other tools at reading and writing S3 objects.

In addition to the speed advantages, you can use cunoFS with multiple buckets—for example, for copying items objects between buckets—and even with different object stores, for example, S3, Azure Blob Storage, Google Cloud Storage, and on-premises object storage like minIO.

cunoFS also has the ability to preserve POSIX metadata in a bucket, such as file and folder permissions, timestamps, symlinks, and hardlinks, which otherwise would be lost during standard upload from local filesystem to S3.

cunoFS currently works natively on Linux, is available under WSL in Windows, and works via virtualization on macOS. Native Windows and macOS apps are coming soon.

Steps to mount an S3 bucket as a local drive with cunoFS

Here’s how you can mount an S3 bucket as a local drive with cunoFS.

  1. Download and install cunoFS: Go to the cunoFS Downloads page and pick your platform, then follow installation instructions.
  2. Add your AWS credentials to cunoFS: Save your AWS credentials to a text file, for example, credentials.txt, with the following format:
aws_access_key_id = xxxxxxxxxxxxxxxxxx
aws_secret_access_key = xxxxxxxxxxxxxxxxxx

Then import these credentials into cunoFS:

$ cuno creds import credentials.txt

- Examining credential file..
- Importing credential file into CUNO's credentials store...
- Detecting available buckets/containers..
Detected buckets: 1    
- Attempting to pair 1 buckets/containers..
Paired 1/1 buckets:
[+] s3://cunofs-examples

For advanced ways of authenticating to AWS, as well as authentication for other cloud storage platforms, check out the Authentication documentation page.

  1. Access your bucket as a local drive: If your S3 bucket is called cunofs-examples, by default cunoFS will make it available under /cuno/s3/cunofs-examples. Launch cunoFS and try accessing the bucket by listing its contents:
$ cuno
(cuno) $ ls /cuno/s3/cunofs-examples
index.html
  1. Launch applications and access files: You can now work with the bucket using any application. For example, you could add a few lines of text using Vim:
$ vim /cuno/s3/cunofs-examples/index.html

The file looks “normal” to the operating system, so you could also access it from your code. Here’s an example in Python:

# cuno_example.py
import os
path = "/cuno/s3/cunofs-examples"
dir_list = os.listdir(path)
print("Files and directories in '", path, "' :")
print(dir_list)

You can then run this Python script:

(cuno) $ python cuno_example.py
Files and directories in ' /cuno/s3/cuno-examples ' :
['index.html']

FAQ

Is it possible to mount multiple buckets?

Yes, all S3 buckets will be available under /cuno/s3/ in the above example.

If you have two buckets, you can use cunoFS to move objects between them:

(cuno) $ ls  /cuno/s3/cunofs-examples
index.html
(cuno) $ ls /cuno/s3/media-bucket/hero-image.jpg
/cuno/s3/media-bucket/index.html
(cuno) $ cp  /cuno/s3/media-bucket/hero-image.jpg /cuno/s3/cunofs-examples/
(cuno) $ ls /cuno/s3/cunofs-examples
index.html hero-image.jpg

Is it possible to mount other object storage types?

Yes, cunoFS supports Amazon S3, Azure Blob Storage, Google Cloud Storage, and any other object storage, both cloud-based and local, such as minIO. All available buckets will be auto-mounted under the /cuno/ directory by default:

(cuno) $ ls /cuno/
az gs s3

What kinds of files does cunoFS work with on S3?

cunoFS supports any file or directory structure and contents that can be stored on S3, including specialized workloads such as HPC applications, scientific computing, and machine learning applications. If it’s required for your workflow, cunoFS can retain POSIX metadata as well.

For machine learning specifically, cunoFS has an additional tool that makes data access with cunoFS even faster. Check out the PyTorch accelerator package for more details.

Get started with cunoFS today

To get started with cunoFS, head to the Download page and follow the installation instructions.

Related Posts

Is POSIX Outdated in the Cloud Era?
POSIX — the Portable Operating System Interface — is the standard API used by UNIX and conforming non-UNIX (including Windows) operating systems to maintain compatibility with each other. The POSIX standard allows applications written for one operating system to work
Read More »
Thumbnail illustration for the How to mount an Amazon S3 bucket as a local drive using cunoFS article
Table of Contents