Lightwhale

Getting Started

The fastest way to get started is to boot Lightwhale off a USB flash drive on the actual server hardware.

The easiest way to experiment with Lightwhale is use a virtual machine like QEMU on your workstation. This gives faster turnaround when rebooting while testing e.g. immutability and persistence.

This guide uses QEMU and provides Linux commands that can be copied and pasted directly into the shell for convenience. Alternative instructions are available elsewhere.

Download Lightwhale

Start by downloading a Lightwhale image from the release repository.

For QEMU and classic machines, get the latest BIOS image:

Or, for newer computers with UEFI, get the latest EFI image instead:

Booting into QEMU

1. Install

Install QEMU on your workstation, or host:

sudo pacman -S qemu-system-x86 # On Arch/Endavour sudo apt-get install qemu-system-x86 # On Debian/Ubuntu

2. Boot

Boot Lightwhale in QEMU, setup port-forwarding from the host to QEMU port 10022 and 10080 that serve Lightwhale's SSH and (eventually) HTTP services respectively:

qemu-system-x86_64 \ -enable-kvm \ -m 2G \ -drive file=lightwhale-2.0.4.bios.iso,format=raw,if=virtio \ -device virtio-rng-pci \ -net nic,model=virtio \ -net user,hostfwd=tcp::10022-:22,hostfwd=tcp::10080-:80

Booting from USB Flash Drive

To instead boot on real hardware, write the Lightwhale image file to a USB flash drive and boot it. Almost any drive will suffice since Lightwhale is small, and once booted, no further reads or writes will occur with that device. Note that, as always, any existing data on the devices will be permanently lost when writing the image.

1. Insert

Insert the USB flash drive and identify what device it was assigned, e.g. /dev/sdx.

lsblk

2. Write

First unmount the device for safety, then write the image to it:

sudo umount /dev/sdx? sudo dd bs=4M conv=fsync status=progress \ if=lightwhale-2.0.4.bios.iso of=/dev/sdx

3. Boot

Boot the computer off the USB flash drive, adjust computer boot order as necessary.

Congratulations!

Now your Lightwhale server is up and ready for use.

Using Lightwhale

The default Lightwhale account is op with password opsecret.

After logging in, it is Docker business as usual; docker and docker compose are conveniently aliased by d and d-c respectively — for your immersive Docker-centric pleasure.

At this point Lightwhale is immutable, allowing safe experimentation; nothing is stored permanently, and rebooting will completely restore the system to factory settings.

Following are some recommended first steps and simple use-cases.

Again, the commands are intended to be copy and pasted into a Linux shell on the host, targeting a Lightwhale server running in QEMU.

Log in using SSH

Using QEMU port-forwarding, connect to the host port 10022 which forwards to Lightwhale port 22 where SSH is served:

For security reasons, it is important to change the op password before connecting Lightwhale to the internet, because now everyone knows your password.

Or even better…

Disable SSH Password Login

To harden security and increase convenience, disable SSH password logins and only allow authentication using SSH keys.

1. Generate SSH key pair on host machine

Chances are these already exist, but otherwise run:

ssh-keygen -t ed25519 -b 4096 \ -C "This Key belongs to $USER@$HOSTNAME"

2. Copy SSH key from host machine to Lightwhale

ssh-copy-id -p 10022 op@localhost

This will prompt for a password for the login.

If ssh warns about remote host identification changed, it is most likely because Lightwhale was rebooted and its SSH server auto-generated a new key during startup, which the host machine does not know about. Simply delete the old identification entry on the host in ~/.ssh/known_hosts and then retry ssh-copy-id:

ssh-keygen -R [localhost]:10022

3. Log into Lightwhale using host SSH key

This should not (and never again) prompt for password:

ssh -p 10022 op@localhost

4. Reconfigure Lightwhale SSH service to disable password logins

Now, on Lightwhale, modify /etc/ssh/sshd_config to disable SSH password logins by adding PasswordAuthentication no to the configuration.

sudo sed '/^#PasswordAuthentication/a\ PasswordAuthentication no # Disable, ie. only allow pubkey to auth.' \ -i.orig /etc/ssh/sshd_config nohup sudo /etc/init.d/S50sshd restart

When restarting sshd for the changes to take effect, notice how this cuts the branch you're sitting on ;)

Connect to a WiFi Network

Lightwhale automatically attempts to connect to wired LAN using DHCP, but wifi requires manual setup.

Use setup-wifi to connect to a wifi network given its SSID and password, e.g.:

Alternatively, do it from the boot loader prompt, or press F1 for detailed help:

Start a Web Server

A nice way to take the Docker Engine of Lightwhale for a quick spin, is to start a container with a web server and request its contents.

1. Create some simple content to serve

echo "Hello, World! This is Lightwhale!" | \ sudo tee /var/www/index.html

2. Start the web server container

d run -d -p 80:80 -v /var/www:/public \ --restart unless-stopped iorivur/darkhttpd

In the event that Docker fails to lookup the registry, simply retry the command. For some reason, this happens regularly within QEMU.

3. Query the web server

Either do the query locally, or like here from the host machine against Lightwhale in QEMU:

curl http://localhost:10080/

Reboot

This concludes our tour.

Now would be a good time to reboot your system and experience firsthand how the immutable system reverts all changes and starts up fully restored.

Read on to learn why Lightwhale is immutable, how it works, and later how to enable persistence.

Immutability by Design

Ah, yes!  This is what distinguishes Lightwhale from other server operating systems.

A server without the ability to any install software or retain data across reboots is useless. This is why in practice, one always combines the two core principles of Lightwhale: immutability and persistence. They are quite contradicting concepts, but work wonders when combined properly — like ☯.

Advantages of Immuta­bility

Immutability is implemented by using a read-only root file system, or rootfs. And this alone instantly brings a number of advantages into the system:

Zero installation

Since the immutable rootfs cannot have anything added on it, all necessary software must be baked into it with sensible defaults from the start, primarily docker and sshd. This results in a fully self-contained image that can simply be written to a boot media and then live-boot, similar to a video game cartridge in principle.

Gone are the tedious and error-prone installation sittings that involve deciding on a partition scheme, formatting file systems, selecting and install software, the bootloader too, and finally rebooting into a system that still requires configuration before it can be used. And that is just the installation…

Zero maintenance

With a fully functioning rootfs, there is no need to install any extra software, nor is there any need to update what already works.

No more package managers, package dependencies, or race of staying up to date. The dreadful operation of reinstalling is effectively accomplished by a simple reboot.

Docker, SSH, and the Linux kernel are the most critical moving part of Lightwhale. And in all fairness, they could eventually need updating. But it is only very rarely required. When that happens, just write a new Lightwhale image on the boot device.

Reduced file system attack surface

The rootfs is inherintly resilient to both unintentional and malicious modification. One cannot accidentally delete something, nor can a virus modify any files here.

In contrast, on a traditional system all files are exposed including every little part of the underlying operating system, e.g /bin/sh, /lib/libc.so.6, and even /usr/bin/[.

As an extreme but interesting example, this command harmless to Lightwhale:

[ -f /etc/lightwhale-release ] && sudo rm -frv /bin /lib /usr /boot

No junk

A common issue with long-living installations is the accumulation of unused files. Junk take up disk space, pollute backups, and may even slow down the system, but is fortunately impossible on this rootfs.

It's just a copy

Remember that the entire system is just a static image written to a boot device. That device and system holds no information itself. So if the boot device breaks, toss it and write a new one.

Reduced power usage

A sustainable system and architecture starts with a sustainable platform. And Lightwhale has taken measures to be more environmentally friendly.

First if all, it only runs a bare minimum of system services, which reduces CPU load and thereby power consumption.

Futhermore, once the rootfs has been loaded into memory, the boot device is never accessed again until next reboot. This greatly reduces wear and tear of the physical boot device, especially because no writes ever occur. But it also allows the boot device to spin down, enter low-power mode, even power off, or be completely removed. If many Lightwhale instances are network-booted, it may lower power consumption even further because no boot media is required in any of these servers.

Free to experiment

Don't hold back, feel free to explore and experiment, have fun, you can't break it.

Booting into Immutability

When Lightwhale is booted off the boot device, first the Linux kernel loaded and started, then the rootfs is loaded and mounted into memory, and everything continues to initialize and eventually run off here. The rootfs is a squashfs which is compressed to save memory, and files are decompresses at run-time when read and cached.

Fallback tmpfs Writability Strategy

The immutable rootfs clearly has some advantages, but in order to be useful some means of writability is required. Because without it, the SSH server has nowhere to save its auto-generated host keys, network configuration and logs cannot be saved, and no containers can be installed.

If no persistence is active, Lightwhale employs its fallback tmpfs-writability strategy that uses tmpfs to add an in‑memory overlay on top of the immutable rootfs. This provides the writable file system, that allows the system to function as expected.

The overlaying of tmpfs works exactly like the overlaying of persistence, and is infact dynamically interchangeable during startup and will be explained in detail in the next chapter.

Lightwhale can now seamlessly both read and write, but with the tmpfs living in memory, all changes will unfortunately be lost at reboot. This is where Lightwhale persistence comes into play.

Persis­tence

Lightwhale handles persistence in its own special way. In order for custom configuration, containers, and data to be retained across reboots, a physical persistence device is required. The persistence device cannot be the same as the boot device, rather it must be a separate and dedicated device that Lightwhale has full disposal of.

In general, persistence should always be enabled to make Lightwhale useful.

The main reason behind persistence not being enforced, is because Lightwhale, being a headless zero-installation live-bootable server OS, has no natural way to pause during startup and prompt the user to prepare the persistence device, as traditional installers do. Additionally, it provides a convenient way to try out Lightwhale before commiting to using a persistence device.

When persistence is active, it completely sets aside the fallback tmpfs-writablity strategy.

Lightwhale fully automates persistence device detection, partitioning, formatting, mounting, and making sure only data and modified configuration is persisted.

The only manual step required is to actually enable persistence.

Enable Persistence

Simply write the magic header to the desired storage device to be used as persistence device. Writing the magic header is easy, and can be done from Lightwhale itself once the device has been identified, e.g. /dev/sdx:

Now reboot, and Lightwhale will automatically claim the device for persistence.

Claim Persistence

Early during system initialization, Lightwhale will attempt to claim persistence through a series of steps, implemented in the setup-persistence script. This also activates the fallback tmpfs-writability if no persistence device was mounted in step 3.

1. Scan for magic header

Scan all available block devices for a magic header, specifically this sequence of bytes at the very start of the device: lightwhale-please-format-me

If found, it is consider a magic device.

2. Create persistence partition

If a magic device was found, it will be partitioned, formatted with ext4, and labeled lightwhale-data.

This becomes the actual persistence partition.

3. Mount persistence partition or tmpfs

Scan for the persistence partition by its label: lightwhale‑data.

If found, mount it read-write under /mnt/lightwhale‑data.

If not found, employ the fallback tmpfs-writability strategy and mount a tmpfs instead. Note that the tmpfs is mounted at the same mount point: /mnt/lightwhale‑data.

At this point, /mnt/lightwhale‑data is a read-writable directory intended to essentially contain all customizations. If persistence is active, any files saved here will be retained during reboot.

4. Create overlay file system

If the persistence partition (or tmpfs) was mounted on /mnt/lightwhale‑data, then merge it into an overlay file system, and thereby ensure persistence (or tmpfs writability) of selected directories.

Overlay File System

Lightwhale adds persistence (or tmpfs) on a per-directory basis by mounting an overlay file system on top of the rootfs. The overlay virtually merges layers of existing directories, while keeping their individual content separate.

Effectively, the overlay mirrors the original immutable rootfs, while modifications, new files, and even deletions made to it, will override the original content, but be kept isolated on the persistence file system (or tmpfs).

Selected Per‑Directory Persistence

Persistence (or tmpfs) does not cover the entire rootfs. Instead it is limited to the following essential directories only:

/etc/

All custom system configuration. The persisted /etc is overlaid early in the boot process for custom configuration and init scripts to take effect.

/home/

All user home directories.

/var/www/

Standard web root. Not strictly necessary, but convenient for running an ad-hoc web server.

/var/lib/docker/

All Docker runtime files including downloaded images, containers and their state, configured networks, and managed volumes.

There is a technically that prevents this directory to be overlaid like the others, because Docker requires it for an internal overlay itself. So instead, Lightwhale automatically reconfigures Docker to use /mnt/lightwhale-data/docker for its data root.

Directories excluded

Directories like /bin, /usr, and /lib are deliberately not persisted. Installing software directly on the rootfs is considered "abuse" of the system because it would defeat the very purpose of immutability and of using Lightwhale in general. One should add software though Docker containers, which can easily include system tools like an updated git client.

Absolute Data Segregation

The strict separation of the system on the boot device and custom configuration, containers, and data on the persistence device, virtually merged by an overlay, gives Lightwhale its amazing perk of absolute data segregation.

It also ensure that it is safe to rewrite Lightwhale or replace the boot device anytime.

A more classic approach would be to mount, for example /etc, on a different disk to separate config from system. However, remember that this requires a fully populated directory structure to be available on the that mount point, including files that were never changed from the original system. Lightwhale only saves the modifications.

Testing Persistence in QEMU

This is easy, interesting, heck, you might even find it fun!

1. Create a persistence device file

Create an empty file for the virtual persistence device:

dd if=/dev/zero of=persistence.img bs=1M count=700

2. Write magic header

Write magic header to persistence device file so it will automatically be claimed for persistence at startup:

echo "lightwhale-please-format-me" |\ dd conv=notrunc of=persistence.img

3. Start QEMU

Start QEMU with the virtual persistence device file attached:

qemu-system-x86_64 \ -enable-kvm \ -m 2G \ -drive file=persistence.img,index=0,format=raw,media=disk \ -drive file=lightwhale-2.0.4.bios.iso,index=2,format=raw,media=cdrom -boot d \ -device virtio-rng-pci \ -net nic,model=virtio \ -net user,hostfwd=tcp::10022-:22,hostfwd=tcp::10080-:80

During startup, notice how Lightwhale claims the device and creates the lightwhale-data persistence partition, and overlays the essential directories.

4. Copy SSH key from host machine to Lightwhale

Copy SSH user key from host to Lightwhale in QEMU and enter password, after removing any old offending SSH key:

ssh-keygen -R [localhost]:10022 ssh-copy-id -p 10022 op@localhost

5. Log into Lightwhale using host SSH key

This should not prompt for password:

ssh -p 10022 op@localhost

6. Start a Web Server

Start a web server and test it locally:

echo "Hello, World! Lightwhale has persistence enabled!" | \ sudo tee /var/www/index.html d run -d --restart unless-stopped -p 80:80 -v /var/www:/public iorivur/darkhttpd curl http://localhost

7. Reboot

sudo reboot

Now, normally this would completely reset Lightwhale, and the SSH server host keys, the authorized SSH user key, and the web server would be erased from memory. But we have persistence enabled…

8. Test Web Server

Query the web server from the host:

curl http://localhost:10080/

This returns the index.html, which confirms that both data and container were persistend.

9. Test SSH

Connect with SSH from the host:

ssh -p 10022 op@localhost

This logs in with out password prompt, which confirms that the SSH server host keys and user SSH key were persisted.

FAQ

Frequently asked questions, or future question that someone could be asking a some point.

How do I login on Lightwhale?

Please refer to the guide.

Wait, what? Does Lightwhale save data or not?

By default, Lightwhale only saves changes to memory, which means you will lose all your work at shutdown. To preserve configuration, containers, and data across reboots and have Lightwhale restart as you intend, you must first enable persistence. This is a very simple process that requires a dedicated persistence device.

How can I connect Lightwhale to wifi?

Please refer to the guide.

What hardware is supported?

Only x86-64, both BIOS and EFI.

Can I run Lightwhale on Raspberry Pi?

No, not currently. It's on the backlog.

Can I run Lightwhale on Macintosh M1 or M2?

No.

Can I run Lightwhale in the cloud/Proxmox/Ventoy/etc?

Probably. Lightwhale is just Linux, so everything is possible. However, Lightwhale is intended to run as the main OS on your own, real, physical computer, and that is currently what this guide covers.

Does Lightwhale come with a firewall/anti-virus?

No. Always take adequate security measures. Run servers behind a firewall. And for the love of all that's good in life, at least change the default password!

How am I covered if Lightwhale crashes, breaks something, makes me loose money/customers/reputation/hair/sleep/etc?

Think, take responsibility of your own actions, use at your own risk. There's no warrenty.

Should I use Lightwhale at home, school, or work?

Yes.

How do I update Lightwhale?

The underlying concept is that you don't.

This is based on the premise that critically necessary updates of the OS are in fact extremely rare. Lightwhale primarily consists of Docker, sshd, and Linux, running off an immutable rootfs using containerized execution. This significantly reduces the likelihood of requiring such updates.

That said, updated Lightwhale image will occur, either due to a critical bug-fix or must-have features. In that case, you have two options:

1. First and foremost: You can choose not to update! If the update doesn't include anything important to you, just leave your system running as it is.

2. Or, if you must: Simply download the image, write it to a boot device, and reboot into an updated Lightwhale.

Remember to read the CHANGELOG, as it may contain important information.

A running Lightwhale instance can conveniently be updated in-place from itself. Because the boot device is not in use at this point, simply download the latest BIOS or EFI image, and directly overwrite the old version on the boot device, e.g. /dev/sdx, then reboot into the updated Lightwhale:

curl -sL https://lightwhale.asklandd.dk/download/latest-bios | \ sudo dd bs=4M conv=fsync of=/dev/sdx

What's the TL;DR on how to run Lightwhale off a USB flash drive with persistence enabled?

Fair enough. First of all Lightwhale requires separate boot and persistence devices. Assuming your USB flash drive for the boot device is /dev/sdx, and your harddisk for the persistence device is /dev/sdy:

curl -sL https://lightwhale.asklandd.dk/download/latest-bios | \ sudo dd bs=4M conv=fsync of=/dev/sdx echo "lightwhale-please-format-me" | \ sudo dd conv=notrunc of=/dev/sdy

Can you please add wget, nano, $my_fav_app_omg_i_love_it to the rootfs?

No, not likely.

I understand that it's disappointing not to have all the tools at your disposal, that you're accustomed to on mainstream Linux distros. Maybe you're used to nano, and now you feel forced to learn vi. But remember, this is a minimalistic purpose-specific server OS, and with that comes some compromises and limitations; you get one editor, one http client, and other preselected essentials. Everything needed is there, but perhaps not the shapes and sizes you prefer. If every person's preferred apps were added, Lightwhale would quickly turn into Bloatwhale, and that is not the goal of this project.