From dfd06264e1744a19bd1e648c137550bf6ee8ae67 Mon Sep 17 00:00:00 2001 From: Asahi Lina Date: Oct 10 2024 12:39:22 +0000 Subject: x86-support.adoc: New page --- diff --git a/modules/ROOT/nav.adoc b/modules/ROOT/nav.adoc index e7d4788..f656698 100644 --- a/modules/ROOT/nav.adoc +++ b/modules/ROOT/nav.adoc @@ -2,5 +2,6 @@ * xref:troubleshooting.adoc[Troubleshooting] * xref:deviations.adoc[Deviations] * xref:how-its-made.adoc[How it's made] +* xref:x86-support.adoc[x86 and x86-64 application support] * xref:talks.adoc[Conference talks] * xref:faq.adoc[FAQ] diff --git a/modules/ROOT/pages/faq.adoc b/modules/ROOT/pages/faq.adoc index 85b7bbd..f9a65f9 100644 --- a/modules/ROOT/pages/faq.adoc +++ b/modules/ROOT/pages/faq.adoc @@ -59,11 +59,9 @@ We currently do not have a supported process to install to external storage. The In the future, we expect to have a supported process for external-storage installs that does not require re-partitioning the internal disk (taking advantage of the same mechanism used for macOS installs to external storage), but this is not ready yet. Note that such external installs will still be tied to the machine they were installed on (since some components would still be installed to the internal disk, as the platform requires this to boot). -=== Can I run software built for x86-64 (Intel/AMD)? +=== Can I run software built for x86-64 (Intel/AMD)? [[x86]] -We are currently working on integrating x86 and x86-64 emulation to support running proprietary software that is not available for aarch64 (similar to Rosetta for macOS), including Windows games via Steam/Proton and Wine. However, this isn't quite ready yet, so stay tuned for updates! - -`box64` is available in the Fedora repositories and can be used to run some native x86-64 Linux software, including some games. Note that compatibility is limited to a subset of 64-bit native Linux software (32-bit software will not run, nor can `box64` be used to run Windows apps with Wine at this time). +{variant-name} now includes support for emulating x86 and x86-64 software, initially targeted at gaming. For more details, please see xref:x86-support.adoc[Running x86/x86-64 applications on {variant-name}]. For Free/Open Source Software, it is always preferable to make a build for aarch64 (ARM64) instead of trying to emulate an x86-64 build. If your favorite software package is not yet available for this architecture, please request it from its developers! Users comfortable with building software may want to try building it themselves. If you do so, consider xref:package-maintainers::Joining_the_Package_Maintainers.adoc[becoming a package maintainer] to help bring the software to all Fedora users. diff --git a/modules/ROOT/pages/index.adoc b/modules/ROOT/pages/index.adoc index 25e6354..c302c61 100644 --- a/modules/ROOT/pages/index.adoc +++ b/modules/ROOT/pages/index.adoc @@ -20,6 +20,7 @@ This guide covers those areas where {variant-name} differs from a standard Fedor * xref:installation.adoc[OS installation] * xref:faq.adoc[FAQ] * xref:troubleshooting.adoc[Troubleshooting steps] +* xref:x86-support.adoc[x86 and x86-64 application support] [[known-bugs]] == Known bugs diff --git a/modules/ROOT/pages/x86-support.adoc b/modules/ROOT/pages/x86-support.adoc new file mode 100644 index 0000000..41d9a9e --- /dev/null +++ b/modules/ROOT/pages/x86-support.adoc @@ -0,0 +1,136 @@ += Running x86/x86-64 applications on {variant-name} + +There are lots of lots of legacy x86/x86_64 applications that users want to run on arm64 platforms, including Windows applications and games. To support this in {variant-name}, we have integrated a stack of existing and bespoke components to make it possible to transparently run x86/x86_64 apps directly on arm64 Linux. + +Since Apple platforms use a 16K page size natively and x86/x86_64 processors use a 4K page size, this is especially tricky, as x86/x86_64 applications generally do not work when presented with a host kernel that requires 16K page alignment. To bridge this gap, we are using a microVM to run an entirely separate guest Linux kernel in 4K page size mode. To keep it as seamless as possible, the guest environment is designed to be as close as possible to the host environment, and we use native context GPU passthrough to have high-performance graphics inside the guest. + +The stack consists of these components: + +- https://github.com/AsahiLinux/muvm[muvm] (package: `muvm`), our bespoke microVM runner based on https://github.com/containers/libkrun[libkrun]. +- https://fex-emu.com[FEX-emu] (package: `fex-emu`), a fast userspace x86/x86_64 emulator focused on correctness. +- The https://src.fedoraproject.org/rpms/fex-emu-rootfs-fedora[Fedora FEX RootFS] (package: `fex-emu-rootfs-fedora`), which provides common x86/x86_64 library dependencies to be used by emulated applications. +- https://gitlab.freedesktop.org/asahi/mesa[mesa] (packages: `mesa-fex-emu-overlay-i386` and `mesa-fex-emu-overlay-x86_64`), built for the x86/x86_64 architectures and packaged as a FEX RootFS overlay. This provides the OpenGL/OpenCL/Vulkan support for Apple GPUs. +- https://chromium.googlesource.com/chromiumos/platform2/+/HEAD/vm_tools/sommelier/README.md[sommelier] (package: `sommelier`), a forwarding Wayland compositor for cross-domain window passthrough (will likely be replaced with a native X11 passthrough solution in the future). +- https://github.com/WhatAmISupposedToPutHere/hidpipe[hidpipe] (package: `hidpipe`), a proxy for HID input devices to allow gamepads to work within the VM. + +We also have our own https://pagure.io/fedora-asahi/steam[Steam wrapper] that automates the process of installing and launching Steam inside the microVM stack. When running Windows games using Steam, these open-source components are used behind the scenes: + +- https://github.com/ValveSoftware/Proton[Proton], a Wine distribution aimed at gaming. +- https://github.com/doitsujin/dxvk[dxvk], a translation layer that converts the Windows DirectX 8 - DirectX 11 APIs to Vulkan. +- https://github.com/HansKristian-Work/vkd3d-proton[vkd3d-proton], a translation layer that converts the Windows DirectX 12 API to Vulkan. + +This technology stack is currently primarily aimed at running x86 and x86-64 games, although in the future it should also be useful to run non-game productivity applications. When possible, you should prefer native alternatives over emulation. Please read xref:faq.adoc#x86[this FAQ entry] for more information. + +== Usage + +Just use `dnf install steam` to install our Steam wrapper, and then run Steam from your desktop's launcher (or the `steam` command) to download and install Steam. This will install all necessary dependencies automatically. + +To run applications within a microVM manually, just prefix the command with `muvm --`. For example, `muvm \-- sh` will give you a shell within the microVM environment. In this environment, the kernel's binfmt support is already configured to use FEX to run x86/x86_64 applications, so you should be able to just run them. Note that we do not officially support applications outside of Steam right now, as there are a number of issues that can make the experience quite frustrating. This will change in the future. + +== How it works + +`muvm` creates a virtual machine that shares as much with the host OS as possible. Within the VM, the root filesystem is _the same as the host root filesystem_, with the following exceptions: + +* `/dev`, `/sys`, and `/proc` are guest-private. +* `/run` is also private to the guest +* The FEX-emu rootfs and overlay images are mounted under `/run/fex-emu/`, with the combined overlay rootfs available at `/run/fex-emu/rootfs`. +* `/usr/share/fex-emu` and `/usr/local/share/fex-emu` are overmounted with a tmpfs to inject a FEX `Config.json` suitable for use within the VM +* A tmpfs is also mounted on `/tmp/.X11-unix`, so X11 server sockets are private to the VM +* The entire host filesystem view is available at `/run/muvm-host`, including any overlaid mounts. For example, you can access the host's `/run` at `/run/muvm-host/run`. (Note: /run/muvm-host/dev exists but will not do what you might hope it does. Host devices are not available in the guest.) + +This means that `/usr`, `/home`, `/etc`, `/opt`, `/var`, `/tmp`, and any other directories in your filesystem root are _shared between the guest and the host_. The arm64 guest OS does not run its own root filesystem, but rather _runs exactly the same binaries as your host OS does_. + +Additionally, FEX itself uses the filesystem mounted at `/run/fex-emu/rootfs` as its virtual RootFS. This means that x86/x86_64 applications (and only those) will see the contents of that directory overlaid on top of the root filesystem. This is how we make x86/x86_64 libraries available to those applications, while still sharing most of the filesystem contents. + +When muvm starts, it registers FEX as a binfmt provider, so x86/x86_64 applications will be transparently run through it. On startup, FEX will detect that TSO support is available on the Apple Silicon platform (even within the VM), and automatically enable it for faster accurate emulation. + +== Known issues + +=== Performance isn't very good + +As this project is still in its early stages, we aimed for correctness for the initial release. Performance optimization will happen over time. We're aware of several changes that should bring significant performance improvements, and we're actively working on it! + +For Windows DX8-DX11 games under Proton in particular, you might want to try WineD3D instead of DXVK. WineD3D uses OpenGL instead of Vulkan as its backend, and it _may_ have better performance thanks to optimizations in our OpenGL driver that are not available on Vulkan. To enable it, change the Steam launch options to `PROTON_USE_WINED3D=1 %command%`. Note that DXVK tends to have better compatibility, so this is a trade-off. Let us know what games work better using either backend! + +=== Window management is weird (titlebar problems, popup problems, DPI problems, etc.) + +These are known issues caused by bugs/limitations in Sommelier. We recommend using our x86/x86_64 stack only to run fullscreen applications (games, Steam in Big Picture mode) at this time. We're working on a solution to this issue by replacing Sommelier with a more lightweight alternative. + +=== The VM uses a lot of RAM + +To allow guest apps to use a large amount of RAM (as some modern games require), by default `muvm` allows the guest to use up to 80% of the system RAM. This also means that some of that will be taken up by guest page cache. Especially on lower RAM size machines (16GB or lower), we recommend not running any heavy host applications while the VM is in use. We don't recommend gaming on 8GB machines. + +You can configure the guest RAM allocation with the `muvm --mem=SIZE` parameter. + +This will be less of an issue in the future, when we enable virtiofs-DAX in `muvm`. This directly maps host filesystem cache pages into the guest, and therefore relieves memory pressure. + +=== I can't access media mounted under `/run/media` within the VM + +This does not work (not even via `/run/muvm-host/run/media`) due to missing POSIX ACL support in libkrun at this time. You must manually mount any disks that you wish to use within the VM, e.g. under `/mnt`. + +== FAQ + +=== Is this like Rosetta on macOS? + +This is as close to Rosetta as we can get! The main difference is that Rosetta side-steps the page size issue by instead relying on the XNU kernel's multiple page size support for user processes, so it doesn't need a VM. While making Linux support mixed page sizes would not be completely impossible in theory, it would be an enormous project that would likely take years to complete, and it isn't at all clear whether such a change would be accepted upstream (Linux doesn't even have boot-time page size selection within a single kernel yet!). + +Other than the page size issue, FEX and Rosetta are comparable technologies (both are emulators, despite what Apple marketing might have you believe). Both FEX and Rosetta use the unique Apple Silicon CPU feature that is most important for x86/x86_64 emulation performance: TSO mode. Thanks to this feature, FEX can offer fast _and_ accurate x86/x86_64 emulation on Apple Silicon systems. + +=== Why not just use a 4K host kernel? + +While Apple Silicon systems support 4K CPU pages, the rest of the hardware (IOMMUs, GPU) runs with 16K pages only. The Linux kernel does not play nicely in this environment, as it generally assumes that the CPU page size is at least as large or larger than the IOMMU page size. In the past we had some kernel patches to make this partially work, but they were buggy and incomplete, so we abandoned the approach. Even if it did work well, running the whole system using 4K pages has a measurable performance impact, so we would never ship 4K kernels by default. Therefore, running x86/x86_64 would require that users manually change their kernel and reboot, which is quite cumbersome. + +=== Why not box64? + +box64 and FEX-Emu have different approaches to emulation, with FEX-Emu aiming for better correctness by default (but requiring a more complex setup) while box64 aims to cover more "out of the box" use cases (like running a subset of applications directly on a 16K kernel without a VM using some tricks). We have chosen FEX-Emu for our stack because we believe it will have higher compatibility with its approach, but both have their uses. box64 is https://src.fedoraproject.org/rpms/box64[packaged in Fedora], so we encourage users to try it (both natively and within muvm) and let us know how it compares! + +=== Steam says steamwebhelper crashed, what do I do? + +Just let it restart, and it should work on the second try. Steam has a timeout for steamwebhelper, and when running under emulation, startup is slow enough that the timeout expires. This usually only happens on a cold startup. + +=== Can I run Windows applications outside of Steam? + +At this point, we do not support running Windows apps outside of Steam for two reasons: + +- Non-Proton Wine does not yet work on Fedora + FEX due to https://github.com/FEX-Emu/FEX/pull/3831[FEX issues] that we are working on. +- Sommelier (the cross-vm compositor that we use currently) is https://github.com/AsahiLinux/muvm/issues/73[quite buggy] and not really suitable for non-fullscreen applications. We are working on a new solution that directly proxies the X11 protocol cross-domain to the host XWayland/compositor, which fixes all of these problems and makes the applications act as native X11 apps running on the host OS. + +Once these issues are resolved, you will be able to use wine under muvm to run Windows apps. + +=== Can I run x86_64/x86 Linux applications? + +Native Linux games should generally work under muvm, as long as they are self-contained and do not depend on complex host system libraries (we ship a large selection of common dependencies, but not everything under the sun). + +Non-game productivity apps may work, but window management will probably be quite broken due to Sommelier bugs (see the previous answer). + +=== Is Wayland supported? + +Wayland is not supported inside the VM at this time (Sommelier is a Wayland compositor, but it is used in an XWayland-exclusive mode). As most of the legacy x86/x86_64 applications people want to run are X11 applications, we are focusing on X11 support first. This means that you cannot run native Wayland apps inside the VM at this time. Of course, the host desktop is still a Wayland desktop, and X11 support is provided by XWayland. + +=== Can I access hardware from applications running within the microVM? + +As the VM does not pass through any host hardware other than the GPU and the virtual filesystem, you will not be able to use applications that require direct hardware access. We use software passthrough for the following interfaces: + +- X11 protocol (display, keyboard, mouse) +- Gamepads via hid/uinput passthrough +- Sound I/O via the PulseAudio socket protocol footnote:[This works with PipeWire running on the host with pipewire-pulse, as installed by default. You do *not* need to and should not install PulseAudio proper, as that will break your speaker support!] + +=== Is this like a Qemu/libvirt/UTM/Parallels/VMWare/VirtualBox/etc. VM? + +No, muvm does not work like a traditional whole-system VM. While it does also use KVM as a backend for efficient virtualization, the concept is very different to traditional VMs running entirely separate guest operating systems. The guest kernel is a https://github.com/containers/libkrunfw[special kernel] optimized to start up in a fraction of a second, and the VM monitor passes through the host filesystem mostly as-is. There is no low-level hardware passthrough (USB, etc.) and instead we focus on higher-level software protocol passthrough, like X11/Wayland. The VM does not run its own standalone init system, only some minimal startup code. This means that the environment within the VM should "feel" the same as the host OS from the point of view of applications, just with a 4K page size instead of a 16K page size. + +=== Can I sudo inside the VM? + +Since the VM monitor runs as your own user identity, it cannot gain root privileges. "root" inside the VM still only has the privileges of your own user, so sudo doesn't make much sense (and in fact doesn't work). We recommend installing software that you want to use with muvm+FEX under your home directory. For software that is designed to be installed under /opt or similar, we recommend performing the installation steps manually on the host OS, and then just running the app under muvm. + +=== Can applications within the VM communicate with applications outside the VM? + +Communication is mostly limited to the host filesystem. The VM shares your home directory (and in fact most of the filesystem) with the host, so any files you create on one side will be visible on the other. + +In the future, we will enable virtiofs-DAX, which will allow for shared memory communication (/dev/shm) between guest apps and host apps, but this isn't quite ready yet. + +It is also possible to share audio between host and guest apps by using the PulseAudio forwarding support. For example, you can record guest audio by using a recording app on the host and recording from the system "Monitor" device. You can also configure virtual sinks/sources in the host using the normal PipeWire mechanisms, and direct guest apps to use those for audio I/O to have custom audio routing and processing. Note that the native PipeWire protocol is not passed through, only the PulseAudio protocol which is more limited (but more commonly used by applications). ALSA applications are supported via the `pulse` plug-in. + +=== Why do I have fewer CPU cores inside the VM? + +By default, `muvm` passes through as many CPUs as there are performance cores on your host machine, and pins those vCPUs to the physical performance cores. Since the host CPU scheduler has no visibility into the guest CPU scheduler, this ensures that performance is consistent. You can modify this behavior with the `muvm --cpu-list=CPU_LIST` option.