mirror of
https://github.com/jesseduffield/lazygit.git
synced 2025-05-16 06:22:58 +02:00
7b302d8c29
Afero is a package that lets you mock out a filesystem with an in-memory filesystem. It allows us to easily create the files required for a given test without worrying about a cleanup step or different tests tripping on eachother when run in parallel. Later on I'll standardise on using afero over the vanilla os package
442 lines
14 KiB
Markdown
442 lines
14 KiB
Markdown
|
||
|
||
A FileSystem Abstraction System for Go
|
||
|
||
[](https://github.com/spf13/afero/actions/workflows/test.yml) [](https://godoc.org/github.com/spf13/afero) [](https://gitter.im/spf13/afero?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
|
||
|
||
# Overview
|
||
|
||
Afero is a filesystem framework providing a simple, uniform and universal API
|
||
interacting with any filesystem, as an abstraction layer providing interfaces,
|
||
types and methods. Afero has an exceptionally clean interface and simple design
|
||
without needless constructors or initialization methods.
|
||
|
||
Afero is also a library providing a base set of interoperable backend
|
||
filesystems that make it easy to work with afero while retaining all the power
|
||
and benefit of the os and ioutil packages.
|
||
|
||
Afero provides significant improvements over using the os package alone, most
|
||
notably the ability to create mock and testing filesystems without relying on the disk.
|
||
|
||
It is suitable for use in any situation where you would consider using the OS
|
||
package as it provides an additional abstraction that makes it easy to use a
|
||
memory backed file system during testing. It also adds support for the http
|
||
filesystem for full interoperability.
|
||
|
||
|
||
## Afero Features
|
||
|
||
* A single consistent API for accessing a variety of filesystems
|
||
* Interoperation between a variety of file system types
|
||
* A set of interfaces to encourage and enforce interoperability between backends
|
||
* An atomic cross platform memory backed file system
|
||
* Support for compositional (union) file systems by combining multiple file systems acting as one
|
||
* Specialized backends which modify existing filesystems (Read Only, Regexp filtered)
|
||
* A set of utility functions ported from io, ioutil & hugo to be afero aware
|
||
* Wrapper for go 1.16 filesystem abstraction `io/fs.FS`
|
||
|
||
# Using Afero
|
||
|
||
Afero is easy to use and easier to adopt.
|
||
|
||
A few different ways you could use Afero:
|
||
|
||
* Use the interfaces alone to define your own file system.
|
||
* Wrapper for the OS packages.
|
||
* Define different filesystems for different parts of your application.
|
||
* Use Afero for mock filesystems while testing
|
||
|
||
## Step 1: Install Afero
|
||
|
||
First use go get to install the latest version of the library.
|
||
|
||
$ go get github.com/spf13/afero
|
||
|
||
Next include Afero in your application.
|
||
```go
|
||
import "github.com/spf13/afero"
|
||
```
|
||
|
||
## Step 2: Declare a backend
|
||
|
||
First define a package variable and set it to a pointer to a filesystem.
|
||
```go
|
||
var AppFs = afero.NewMemMapFs()
|
||
|
||
or
|
||
|
||
var AppFs = afero.NewOsFs()
|
||
```
|
||
It is important to note that if you repeat the composite literal you
|
||
will be using a completely new and isolated filesystem. In the case of
|
||
OsFs it will still use the same underlying filesystem but will reduce
|
||
the ability to drop in other filesystems as desired.
|
||
|
||
## Step 3: Use it like you would the OS package
|
||
|
||
Throughout your application use any function and method like you normally
|
||
would.
|
||
|
||
So if my application before had:
|
||
```go
|
||
os.Open("/tmp/foo")
|
||
```
|
||
We would replace it with:
|
||
```go
|
||
AppFs.Open("/tmp/foo")
|
||
```
|
||
|
||
`AppFs` being the variable we defined above.
|
||
|
||
|
||
## List of all available functions
|
||
|
||
File System Methods Available:
|
||
```go
|
||
Chmod(name string, mode os.FileMode) : error
|
||
Chown(name string, uid, gid int) : error
|
||
Chtimes(name string, atime time.Time, mtime time.Time) : error
|
||
Create(name string) : File, error
|
||
Mkdir(name string, perm os.FileMode) : error
|
||
MkdirAll(path string, perm os.FileMode) : error
|
||
Name() : string
|
||
Open(name string) : File, error
|
||
OpenFile(name string, flag int, perm os.FileMode) : File, error
|
||
Remove(name string) : error
|
||
RemoveAll(path string) : error
|
||
Rename(oldname, newname string) : error
|
||
Stat(name string) : os.FileInfo, error
|
||
```
|
||
File Interfaces and Methods Available:
|
||
```go
|
||
io.Closer
|
||
io.Reader
|
||
io.ReaderAt
|
||
io.Seeker
|
||
io.Writer
|
||
io.WriterAt
|
||
|
||
Name() : string
|
||
Readdir(count int) : []os.FileInfo, error
|
||
Readdirnames(n int) : []string, error
|
||
Stat() : os.FileInfo, error
|
||
Sync() : error
|
||
Truncate(size int64) : error
|
||
WriteString(s string) : ret int, err error
|
||
```
|
||
In some applications it may make sense to define a new package that
|
||
simply exports the file system variable for easy access from anywhere.
|
||
|
||
## Using Afero's utility functions
|
||
|
||
Afero provides a set of functions to make it easier to use the underlying file systems.
|
||
These functions have been primarily ported from io & ioutil with some developed for Hugo.
|
||
|
||
The afero utilities support all afero compatible backends.
|
||
|
||
The list of utilities includes:
|
||
|
||
```go
|
||
DirExists(path string) (bool, error)
|
||
Exists(path string) (bool, error)
|
||
FileContainsBytes(filename string, subslice []byte) (bool, error)
|
||
GetTempDir(subPath string) string
|
||
IsDir(path string) (bool, error)
|
||
IsEmpty(path string) (bool, error)
|
||
ReadDir(dirname string) ([]os.FileInfo, error)
|
||
ReadFile(filename string) ([]byte, error)
|
||
SafeWriteReader(path string, r io.Reader) (err error)
|
||
TempDir(dir, prefix string) (name string, err error)
|
||
TempFile(dir, prefix string) (f File, err error)
|
||
Walk(root string, walkFn filepath.WalkFunc) error
|
||
WriteFile(filename string, data []byte, perm os.FileMode) error
|
||
WriteReader(path string, r io.Reader) (err error)
|
||
```
|
||
For a complete list see [Afero's GoDoc](https://godoc.org/github.com/spf13/afero)
|
||
|
||
They are available under two different approaches to use. You can either call
|
||
them directly where the first parameter of each function will be the file
|
||
system, or you can declare a new `Afero`, a custom type used to bind these
|
||
functions as methods to a given filesystem.
|
||
|
||
### Calling utilities directly
|
||
|
||
```go
|
||
fs := new(afero.MemMapFs)
|
||
f, err := afero.TempFile(fs,"", "ioutil-test")
|
||
|
||
```
|
||
|
||
### Calling via Afero
|
||
|
||
```go
|
||
fs := afero.NewMemMapFs()
|
||
afs := &afero.Afero{Fs: fs}
|
||
f, err := afs.TempFile("", "ioutil-test")
|
||
```
|
||
|
||
## Using Afero for Testing
|
||
|
||
There is a large benefit to using a mock filesystem for testing. It has a
|
||
completely blank state every time it is initialized and can be easily
|
||
reproducible regardless of OS. You could create files to your heart’s content
|
||
and the file access would be fast while also saving you from all the annoying
|
||
issues with deleting temporary files, Windows file locking, etc. The MemMapFs
|
||
backend is perfect for testing.
|
||
|
||
* Much faster than performing I/O operations on disk
|
||
* Avoid security issues and permissions
|
||
* Far more control. 'rm -rf /' with confidence
|
||
* Test setup is far more easier to do
|
||
* No test cleanup needed
|
||
|
||
One way to accomplish this is to define a variable as mentioned above.
|
||
In your application this will be set to afero.NewOsFs() during testing you
|
||
can set it to afero.NewMemMapFs().
|
||
|
||
It wouldn't be uncommon to have each test initialize a blank slate memory
|
||
backend. To do this I would define my `appFS = afero.NewOsFs()` somewhere
|
||
appropriate in my application code. This approach ensures that Tests are order
|
||
independent, with no test relying on the state left by an earlier test.
|
||
|
||
Then in my tests I would initialize a new MemMapFs for each test:
|
||
```go
|
||
func TestExist(t *testing.T) {
|
||
appFS := afero.NewMemMapFs()
|
||
// create test files and directories
|
||
appFS.MkdirAll("src/a", 0755)
|
||
afero.WriteFile(appFS, "src/a/b", []byte("file b"), 0644)
|
||
afero.WriteFile(appFS, "src/c", []byte("file c"), 0644)
|
||
name := "src/c"
|
||
_, err := appFS.Stat(name)
|
||
if os.IsNotExist(err) {
|
||
t.Errorf("file \"%s\" does not exist.\n", name)
|
||
}
|
||
}
|
||
```
|
||
|
||
# Available Backends
|
||
|
||
## Operating System Native
|
||
|
||
### OsFs
|
||
|
||
The first is simply a wrapper around the native OS calls. This makes it
|
||
very easy to use as all of the calls are the same as the existing OS
|
||
calls. It also makes it trivial to have your code use the OS during
|
||
operation and a mock filesystem during testing or as needed.
|
||
|
||
```go
|
||
appfs := afero.NewOsFs()
|
||
appfs.MkdirAll("src/a", 0755)
|
||
```
|
||
|
||
## Memory Backed Storage
|
||
|
||
### MemMapFs
|
||
|
||
Afero also provides a fully atomic memory backed filesystem perfect for use in
|
||
mocking and to speed up unnecessary disk io when persistence isn’t
|
||
necessary. It is fully concurrent and will work within go routines
|
||
safely.
|
||
|
||
```go
|
||
mm := afero.NewMemMapFs()
|
||
mm.MkdirAll("src/a", 0755)
|
||
```
|
||
|
||
#### InMemoryFile
|
||
|
||
As part of MemMapFs, Afero also provides an atomic, fully concurrent memory
|
||
backed file implementation. This can be used in other memory backed file
|
||
systems with ease. Plans are to add a radix tree memory stored file
|
||
system using InMemoryFile.
|
||
|
||
## Network Interfaces
|
||
|
||
### SftpFs
|
||
|
||
Afero has experimental support for secure file transfer protocol (sftp). Which can
|
||
be used to perform file operations over a encrypted channel.
|
||
|
||
### GCSFs
|
||
|
||
Afero has experimental support for Google Cloud Storage (GCS). You can either set the
|
||
`GOOGLE_APPLICATION_CREDENTIALS_JSON` env variable to your JSON credentials or use `opts` in
|
||
`NewGcsFS` to configure access to your GCS bucket.
|
||
|
||
Some known limitations of the existing implementation:
|
||
* No Chmod support - The GCS ACL could probably be mapped to *nix style permissions but that would add another level of complexity and is ignored in this version.
|
||
* No Chtimes support - Could be simulated with attributes (gcs a/m-times are set implicitly) but that's is left for another version.
|
||
* Not thread safe - Also assumes all file operations are done through the same instance of the GcsFs. File operations between different GcsFs instances are not guaranteed to be consistent.
|
||
|
||
|
||
## Filtering Backends
|
||
|
||
### BasePathFs
|
||
|
||
The BasePathFs restricts all operations to a given path within an Fs.
|
||
The given file name to the operations on this Fs will be prepended with
|
||
the base path before calling the source Fs.
|
||
|
||
```go
|
||
bp := afero.NewBasePathFs(afero.NewOsFs(), "/base/path")
|
||
```
|
||
|
||
### ReadOnlyFs
|
||
|
||
A thin wrapper around the source Fs providing a read only view.
|
||
|
||
```go
|
||
fs := afero.NewReadOnlyFs(afero.NewOsFs())
|
||
_, err := fs.Create("/file.txt")
|
||
// err = syscall.EPERM
|
||
```
|
||
|
||
# RegexpFs
|
||
|
||
A filtered view on file names, any file NOT matching
|
||
the passed regexp will be treated as non-existing.
|
||
Files not matching the regexp provided will not be created.
|
||
Directories are not filtered.
|
||
|
||
```go
|
||
fs := afero.NewRegexpFs(afero.NewMemMapFs(), regexp.MustCompile(`\.txt$`))
|
||
_, err := fs.Create("/file.html")
|
||
// err = syscall.ENOENT
|
||
```
|
||
|
||
### HttpFs
|
||
|
||
Afero provides an http compatible backend which can wrap any of the existing
|
||
backends.
|
||
|
||
The Http package requires a slightly specific version of Open which
|
||
returns an http.File type.
|
||
|
||
Afero provides an httpFs file system which satisfies this requirement.
|
||
Any Afero FileSystem can be used as an httpFs.
|
||
|
||
```go
|
||
httpFs := afero.NewHttpFs(<ExistingFS>)
|
||
fileserver := http.FileServer(httpFs.Dir(<PATH>))
|
||
http.Handle("/", fileserver)
|
||
```
|
||
|
||
## Composite Backends
|
||
|
||
Afero provides the ability have two filesystems (or more) act as a single
|
||
file system.
|
||
|
||
### CacheOnReadFs
|
||
|
||
The CacheOnReadFs will lazily make copies of any accessed files from the base
|
||
layer into the overlay. Subsequent reads will be pulled from the overlay
|
||
directly permitting the request is within the cache duration of when it was
|
||
created in the overlay.
|
||
|
||
If the base filesystem is writeable, any changes to files will be
|
||
done first to the base, then to the overlay layer. Write calls to open file
|
||
handles like `Write()` or `Truncate()` to the overlay first.
|
||
|
||
To writing files to the overlay only, you can use the overlay Fs directly (not
|
||
via the union Fs).
|
||
|
||
Cache files in the layer for the given time.Duration, a cache duration of 0
|
||
means "forever" meaning the file will not be re-requested from the base ever.
|
||
|
||
A read-only base will make the overlay also read-only but still copy files
|
||
from the base to the overlay when they're not present (or outdated) in the
|
||
caching layer.
|
||
|
||
```go
|
||
base := afero.NewOsFs()
|
||
layer := afero.NewMemMapFs()
|
||
ufs := afero.NewCacheOnReadFs(base, layer, 100 * time.Second)
|
||
```
|
||
|
||
### CopyOnWriteFs()
|
||
|
||
The CopyOnWriteFs is a read only base file system with a potentially
|
||
writeable layer on top.
|
||
|
||
Read operations will first look in the overlay and if not found there, will
|
||
serve the file from the base.
|
||
|
||
Changes to the file system will only be made in the overlay.
|
||
|
||
Any attempt to modify a file found only in the base will copy the file to the
|
||
overlay layer before modification (including opening a file with a writable
|
||
handle).
|
||
|
||
Removing and Renaming files present only in the base layer is not currently
|
||
permitted. If a file is present in the base layer and the overlay, only the
|
||
overlay will be removed/renamed.
|
||
|
||
```go
|
||
base := afero.NewOsFs()
|
||
roBase := afero.NewReadOnlyFs(base)
|
||
ufs := afero.NewCopyOnWriteFs(roBase, afero.NewMemMapFs())
|
||
|
||
fh, _ = ufs.Create("/home/test/file2.txt")
|
||
fh.WriteString("This is a test")
|
||
fh.Close()
|
||
```
|
||
|
||
In this example all write operations will only occur in memory (MemMapFs)
|
||
leaving the base filesystem (OsFs) untouched.
|
||
|
||
|
||
## Desired/possible backends
|
||
|
||
The following is a short list of possible backends we hope someone will
|
||
implement:
|
||
|
||
* SSH
|
||
* S3
|
||
|
||
# About the project
|
||
|
||
## What's in the name
|
||
|
||
Afero comes from the latin roots Ad-Facere.
|
||
|
||
**"Ad"** is a prefix meaning "to".
|
||
|
||
**"Facere"** is a form of the root "faciō" making "make or do".
|
||
|
||
The literal meaning of afero is "to make" or "to do" which seems very fitting
|
||
for a library that allows one to make files and directories and do things with them.
|
||
|
||
The English word that shares the same roots as Afero is "affair". Affair shares
|
||
the same concept but as a noun it means "something that is made or done" or "an
|
||
object of a particular type".
|
||
|
||
It's also nice that unlike some of my other libraries (hugo, cobra, viper) it
|
||
Googles very well.
|
||
|
||
## Release Notes
|
||
|
||
See the [Releases Page](https://github.com/spf13/afero/releases).
|
||
|
||
## Contributing
|
||
|
||
1. Fork it
|
||
2. Create your feature branch (`git checkout -b my-new-feature`)
|
||
3. Commit your changes (`git commit -am 'Add some feature'`)
|
||
4. Push to the branch (`git push origin my-new-feature`)
|
||
5. Create new Pull Request
|
||
|
||
## Contributors
|
||
|
||
Names in no particular order:
|
||
|
||
* [spf13](https://github.com/spf13)
|
||
* [jaqx0r](https://github.com/jaqx0r)
|
||
* [mbertschler](https://github.com/mbertschler)
|
||
* [xor-gate](https://github.com/xor-gate)
|
||
|
||
## License
|
||
|
||
Afero is released under the Apache 2.0 license. See
|
||
[LICENSE.txt](https://github.com/spf13/afero/blob/master/LICENSE.txt)
|