This release adds WAL support and we've done a ton of hardening & stability work. Checksumming has been revamped so it is not so error prone which should help stability quite a bit.

What's Changed

docs: disconnected fixup in store by @ryanrussell in https://github.com/superfly/litefs/pull/111
Prefix FUSE debug logs by @benbjohnson in https://github.com/superfly/litefs/pull/113
Use uppercase hex for node ID by @benbjohnson in https://github.com/superfly/litefs/pull/114
WAL support by @benbjohnson in https://github.com/superfly/litefs/pull/120
Add WAL checkpoint test by @benbjohnson in https://github.com/superfly/litefs/pull/121
Fix return lock type for SHM lock by @benbjohnson in https://github.com/superfly/litefs/pull/123
List journal filenames in root handle by @dangra in https://github.com/superfly/litefs/pull/125
Updates workflows to support external PRs by @dangra in https://github.com/superfly/litefs/pull/126
Force WAL fsync() by @benbjohnson in https://github.com/superfly/litefs/pull/127
Remove dead code for WAL locking by @benbjohnson in https://github.com/superfly/litefs/pull/129
Consolidate locks & guards by @benbjohnson in https://github.com/superfly/litefs/pull/130
Fix snapshot in WAL mode by @benbjohnson in https://github.com/superfly/litefs/pull/131
Continue running on error by @benbjohnson in https://github.com/superfly/litefs/pull/137
Reduce logging by @benbjohnson in https://github.com/superfly/litefs/pull/143
Rollback journal on startup by @benbjohnson in https://github.com/superfly/litefs/pull/144
Validate that mount & data directories are different by @benbjohnson in https://github.com/superfly/litefs/pull/145
Ensure mount directory exists before mounting by @benbjohnson in https://github.com/superfly/litefs/pull/146
refactor: replace Readdir(-1) with os.ReadDir by @Juneezee in https://github.com/superfly/litefs/pull/147
Fix locking when applying LTX file to replica in WAL mode by @benbjohnson in https://github.com/superfly/litefs/pull/148
Retain OS page cache on open(2) by @benbjohnson in https://github.com/superfly/litefs/pull/150
Attempt unmount for unclean exit by @benbjohnson in https://github.com/superfly/litefs/pull/151
Specify 'exec' command from CLI by @benbjohnson in https://github.com/superfly/litefs/pull/152
Add SHM locks to /debug/vars by @benbjohnson in https://github.com/superfly/litefs/pull/154
Add endpoint for dynamically setting debug logging by @benbjohnson in https://github.com/superfly/litefs/pull/156
Replay last LTX on recovery by @benbjohnson in https://github.com/superfly/litefs/pull/157
Fix recovery on replication position mismatch by @benbjohnson in https://github.com/superfly/litefs/pull/158
Clear LTX files on snapshot received by @benbjohnson in https://github.com/superfly/litefs/pull/160
Remove pre-exec sleep by @benbjohnson in https://github.com/superfly/litefs/pull/161
Print version & commit info on startup by @benbjohnson in https://github.com/superfly/litefs/pull/162
Initialize DB.mode on open by @benbjohnson in https://github.com/superfly/litefs/pull/163
Add support for PERSIST journal mode by @benbjohnson in https://github.com/superfly/litefs/pull/168
Add support for TRUNCATE journal mode by @benbjohnson in https://github.com/superfly/litefs/pull/169
Show position file in readdir output by @benbjohnson in https://github.com/superfly/litefs/pull/175
Allow simple expressions in env var expansion by @benbjohnson in https://github.com/superfly/litefs/pull/177
Reduce Consul lock delay by @benbjohnson in https://github.com/superfly/litefs/pull/180
Fix FUSE connection close deadlock by @benbjohnson in https://github.com/superfly/litefs/pull/181
Add END frame to HTTP stream by @benbjohnson in https://github.com/superfly/litefs/pull/183
Recover from invalid database on startup by @benbjohnson in https://github.com/superfly/litefs/pull/184
Improve snapshot logging by @benbjohnson in https://github.com/superfly/litefs/pull/185
Acquire all SHM read locks on snapshot by @benbjohnson in https://github.com/superfly/litefs/pull/186
Read page count after verify by @benbjohnson in https://github.com/superfly/litefs/pull/187
Move functionality to 'mount' subcommand by @benbjohnson in https://github.com/superfly/litefs/pull/188
Update page count on apply by @benbjohnson in https://github.com/superfly/litefs/pull/189
Implement StrictVerify for WAL by @benbjohnson in https://github.com/superfly/litefs/pull/190
Add 'strict-verify' to YAML config by @benbjohnson in https://github.com/superfly/litefs/pull/191
Fix compilation on Darwin by @benbjohnson in https://github.com/superfly/litefs/pull/193
Fix JournalNode.Read() by @benbjohnson in https://github.com/superfly/litefs/pull/194
Restrict dirty pgno to commit size by @benbjohnson in https://github.com/superfly/litefs/pull/195
Allow retention enforcement to be disabled by @benbjohnson in https://github.com/superfly/litefs/pull/196
Add litefs-bench test program by @benbjohnson in https://github.com/superfly/litefs/pull/198
Fix frame count calculation on journal recovery by @benbjohnson in https://github.com/superfly/litefs/pull/197
Upgrade to LTX v0.2.7 by @benbjohnson in https://github.com/superfly/litefs/pull/199
Check for journal header magic after first segment by @benbjohnson in https://github.com/superfly/litefs/pull/200
Set journal mode in litefs-bench by @benbjohnson in https://github.com/superfly/litefs/pull/201
Skip dirty page tracking in WAL mode by @benbjohnson in https://github.com/superfly/litefs/pull/202
Add -debug CLI flag to mount command by @benbjohnson in https://github.com/superfly/litefs/pull/203
Move WAL commit to after WAL_WRITE_LOCK release by @benbjohnson in https://github.com/superfly/litefs/pull/205
Fix Pos() deadlock with file system invalidation by @benbjohnson in https://github.com/superfly/litefs/pull/206
Fix WriteSnapshotTo() lock in WAL mode by @benbjohnson in https://github.com/superfly/litefs/pull/207
Add 'query' command to litefs-bench by @benbjohnson in https://github.com/superfly/litefs/pull/208
Add latency metrics by @benbjohnson in https://github.com/superfly/litefs/pull/209
Add skip-sync config flag by @benbjohnson in https://github.com/superfly/litefs/pull/210
Acquire shared CKPT & RECOVER lock during snapshot by @benbjohnson in https://github.com/superfly/litefs/pull/211
Add 'import' command by @benbjohnson in https://github.com/superfly/litefs/pull/212
Disable fail-fast on CI by @benbjohnson in https://github.com/superfly/litefs/pull/214
Remove journal on import by @benbjohnson in https://github.com/superfly/litefs/pull/213
Make FUSE layer thinner by @benbjohnson in https://github.com/superfly/litefs/pull/215
Fix journal checksum validation by @benbjohnson in https://github.com/superfly/litefs/pull/216
Consolidate Remaining DB API by @benbjohnson in https://github.com/superfly/litefs/pull/218
Add trace logging by @benbjohnson in https://github.com/superfly/litefs/pull/219
Implement page-level checksum tracking by @benbjohnson in https://github.com/superfly/litefs/pull/221
Sync WAL to LTX on startup by @benbjohnson in https://github.com/superfly/litefs/pull/222
Rewrite SHM after replica LTX apply by @benbjohnson in https://github.com/superfly/litefs/pull/224
Move debug flag to FUSE config section by @benbjohnson in https://github.com/superfly/litefs/pull/225
Implement rolling trace log by @benbjohnson in https://github.com/superfly/litefs/pull/226
Skip lock page on checksum by @benbjohnson in https://github.com/superfly/litefs/pull/227
Skip lock page on snapshot by @benbjohnson in https://github.com/superfly/litefs/pull/228
Checkpoint on primary/replica state change by @benbjohnson in https://github.com/superfly/litefs/pull/229
Remove FUSE debugging on CI testing by @benbjohnson in https://github.com/superfly/litefs/pull/231
Implicit CKPT lock on WRITE lock by @benbjohnson in https://github.com/superfly/litefs/pull/230
Add additional trace logging to CommitWAL() by @benbjohnson in https://github.com/superfly/litefs/pull/232
Prevent partial WAL rewind by @benbjohnson in https://github.com/superfly/litefs/pull/233
Continue init on database early EOF by @benbjohnson in https://github.com/superfly/litefs/pull/235
Increase default retention by @benbjohnson in https://github.com/superfly/litefs/pull/236
Require explicit Consul key by @benbjohnson in https://github.com/superfly/litefs/pull/237
Commit WAL on Handle Close by @benbjohnson in https://github.com/superfly/litefs/pull/238
Move mount directory to FUSE config by @benbjohnson in https://github.com/superfly/litefs/pull/239
Move data-dir to data section by @benbjohnson in https://github.com/superfly/litefs/pull/240
Consolidate consul & static lease configs by @benbjohnson in https://github.com/superfly/litefs/pull/241
Fail unless lease type is valid by @dangra in https://github.com/superfly/litefs/pull/242
Add ref_name to Docker builds by @benbjohnson in https://github.com/superfly/litefs/pull/243
Remove static.primary config field by @benbjohnson in https://github.com/superfly/litefs/pull/244
Strict config type checking by @benbjohnson in https://github.com/superfly/litefs/pull/245
Minor updates to litefs.yml reference by @benbjohnson in https://github.com/superfly/litefs/pull/246
Refactor config comments by @benbjohnson in https://github.com/superfly/litefs/pull/247
Add test for rollback on database creation by @benbjohnson in https://github.com/superfly/litefs/pull/248

New Contributors

@ryanrussell made their first contribution in https://github.com/superfly/litefs/pull/111
@dangra made their first contribution in https://github.com/superfly/litefs/pull/125
@Juneezee made their first contribution in https://github.com/superfly/litefs/pull/147

Full Changelog: https://github.com/superfly/litefs/compare/v0.2.0...v0.3.0

Sep 21, 2022

The v0.2.0 release primarily focuses on stability and hardening, however, it also includes a few important features. This release is also marks the end of the alpha phase of LiteFS and it is now considered "beta" software. It should generally run well, however, please report bugs to the GitHub repository.

Major features

Candidacy

By default, every node in the cluster can become the primary node. However, moving a primary from one region to another can change performance characteristics significantly so it may be useful to only allow certain nodes to become primary. You can now set the candidate flag in the config to false for any nodes which should not become primary:

candidate: false

Static Lease

LiteFS uses Consul to dynamically change the current primary in the cluster on the fly. However, for users that do not wish to run Consul, they can set a single fixed primary node. This is similar to how Litestream's read replication was meant to work. Please see https://github.com/superfly/litefs/pull/47 for more details.

Pull Requests

Upgrade to LTX v0.2.0 in https://github.com/superfly/litefs/pull/45
Implement candidacy in https://github.com/superfly/litefs/pull/46
Implement static lease in https://github.com/superfly/litefs/pull/47
Return ENOSYS for xattr requests by @darthShadow in https://github.com/superfly/litefs/pull/51
Implement database snapshot to LTX in https://github.com/superfly/litefs/pull/53
Update contribution policy in https://github.com/superfly/litefs/pull/54
Retention enforcement in https://github.com/superfly/litefs/pull/57
Implement statfs() callback in https://github.com/superfly/litefs/pull/58
Support non-default page size in https://github.com/superfly/litefs/pull/59
Expose DB position via FUSE in https://github.com/superfly/litefs/pull/60
Optional registration of Consul node in https://github.com/superfly/litefs/pull/63
Use default consul key, if not specified in https://github.com/superfly/litefs/pull/65
Return hostname in primary file in https://github.com/superfly/litefs/pull/66
Metrics in https://github.com/superfly/litefs/pull/67
Update lock delay to 5s in https://github.com/superfly/litefs/pull/68
Set hostname env variable, if unset in https://github.com/superfly/litefs/pull/69
Default Advertise URL in https://github.com/superfly/litefs/pull/70
Add configurable data directory in https://github.com/superfly/litefs/pull/71
Rename pprof import in https://github.com/superfly/litefs/pull/72
Fix database locks to use LockOwner in https://github.com/superfly/litefs/pull/74
Atomically write LTX files on commit in https://github.com/superfly/litefs/pull/78
Add unique, persistent node ID in https://github.com/superfly/litefs/pull/79
Disconnect streams when primary status changes in https://github.com/superfly/litefs/pull/80
Add DB nil check on HTTP server in https://github.com/superfly/litefs/pull/81
Remove GITHUB_TOKEN & Makefile in https://github.com/superfly/litefs/pull/82
Add staticcheck CI in https://github.com/superfly/litefs/pull/83
Add errcheck to CI in https://github.com/superfly/litefs/pull/84
Wait for ready state on startup in https://github.com/superfly/litefs/pull/85
Ensure node does not startup if LTX files are invalid in https://github.com/superfly/litefs/pull/86
Minor CI tweaks in https://github.com/superfly/litefs/pull/87
Disable poll() in FUSE layer in https://github.com/superfly/litefs/pull/88
Verify all LTX files on startup in https://github.com/superfly/litefs/pull/89
Verify database checksum on startup in https://github.com/superfly/litefs/pull/90
Fix locking on LTX apply in https://github.com/superfly/litefs/pull/91
Verify txid/checksum on LTX frame in https://github.com/superfly/litefs/pull/92
Add long-running, functional test in https://github.com/superfly/litefs/pull/93
Fix http log formatting in https://github.com/superfly/litefs/pull/94
Skip LTX checksum check on snapshot in https://github.com/superfly/litefs/pull/95
Document 'exec' field in config in https://github.com/superfly/litefs/pull/97
Fix LTX checksum on VACUUM in https://github.com/superfly/litefs/pull/98
Refactor GitHub Actions in https://github.com/superfly/litefs/pull/99
Mark ready after initial replication set in https://github.com/superfly/litefs/pull/100
Fix FUSE return so replicas report a SQLITE_READONLY error in https://github.com/superfly/litefs/pull/101
Remove DBID in https://github.com/superfly/litefs/pull/102
Refactor RWMutex & RWMutexGuard in https://github.com/superfly/litefs/pull/103
Move CI timeout to 'go test' command in https://github.com/superfly/litefs/pull/104
Use HTTP/2 for internal endpoints in https://github.com/superfly/litefs/pull/106
Require explicit data directory in https://github.com/superfly/litefs/pull/109
Only release amd64 and arm64 in https://github.com/superfly/litefs/pull/110

New Contributors

@darthShadow made their first contribution in https://github.com/superfly/litefs/pull/51

Full Changelog: https://github.com/superfly/litefs/compare/v0.1.1...v0.2.0

Aug 5, 2022

v0.1.1

↗

This version of LiteFS replaces the FUSE implementation from github.com/hanwen/go-fuse to bazil.org/fuse. The new implementation seems to be much more stable on a variety of operating systems (Fedora, Alpine, GH Actions, etc).

Use of PRAGMA journal_mode = wal now returns an error since LiteFS does not yet support WAL mode. Also, additional test coverage has been added and increased testing will continue.

Jul 26, 2022

v0.1.0

↗

This is the initial release of LiteFS. It is still in an alpha stage but the build is intended to make it easier to test.

From other products

Last Checked

3h ago

Latest

v0.5.14

Source

@superfly/litefs4.8k

Tracking since Jul 26, 2022

.json·.md·.atom

LiteFS

What's Changed

What's Changed

What's Changed

New Contributors

What's Changed

What's Changed

What's Changed

New Contributors

What's Changed

What's Changed

What's Changed

What's Changed

What's Changed

What's Changed

What's Changed

What's Changed

What's Changed

New Contributors

What's Changed

New Contributors

What's Changed

New Contributors

Overview

Major features

Candidacy

Static Lease

Pull Requests

New Contributors

More from Fly.io

From other products

More from Fly.io

From other products