i2p/haiku-beta6
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Generate developer docs with Sphinx

Browse Source 
An effort was started some time ago to consolidate all internal
documentation in the git tree. However, this was just an accumulation of
files in various formats without any strucutre or way to browse it,
which results in no one even knowing that we have docs here.

This converts most of the files to restructuredtext and uses Sphinx to
generate an HTML browsable user manual (with a table of content and a
first attempt to put things in a global hierarchy).

There are almost no changes to the documentation content in this commit
(some obviously obsolete things were removed). The plan is to get the
toolchain up and running to make these docs easily available, and only
then see about improving the content. We can migrate some things off the
wiki and website, and rework the table of contents to have some more
hierarchy levels because currently it's a bit messy.

Change-Id: I924ac9dc6e753887ab56f18a09bdb0a1e1793bfd
Reviewed-on: https://review.haiku-os.org/c/haiku/+/4370
Reviewed-by: Niels Sascha Reedijk <[email protected]>
This commit is contained in:
Adrien Destugues
2021-08-27 11:41:17 +00:00
committed by Adrien Destugues
parent 7f8195344a
commit a5061ecec5
146 changed files with 10937 additions and 12069 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/develop/kernel/boot/Debugging_Bootloaders_GEF.md
-93
View File
@@ -1,93 +0,0 @@
# Bootloader debugging with GEF
When Haiku's early boot process is experiencing unknown crashes or faults, it can be extremely
difficult to troubleshoot (especially when serial, video, or other i/o devices are non-functional)
It **is** possible to step through the boot of any architecture of Haiku in a debugger if the system
boots and the issue can be reproduced in qemu.
> This works for any architecture and is _extremely_ helpful to trouble early platforms. Linux or Mac OS
> are requirements. You need a full POSIX environment.
## Building Haiku
On most non-x86 platforms, you will need a "kernel" (haiku_loader) and an "initrd" (haiku_floppyboot).
For arm/arm64: ```jam -q @minimum-mmc```
## Launching Haiku in QEMU
In the example below, we will prepare Haiku arm in QEMU for debugging.
```
qemu-system-arm -M raspi2 -kernel haiku_loader.u-boot -initrd haiku-floppyboot.tgz.u-boot -serial stdio -m 2G -dtb rpi2.dtb -s -S
```
**Key Flags:**
* **-s**
* Shorthand for -gdb tcp::1234, i.e. open a gdbserver on TCP port 1234.
* **-S**
* Do not start CPU at startup (you must type 'c' in the monitor).
These simple flags will make qemu listen for a debugger connection on localhost:1234 and have the VM not start until you tell it to.
> In the example above, we are Emulating a Raspberry Pi 2, and using our Raspberry Pi 2 dtb. If you don't have a dtb for the machine
> you're emulating, you can dump qemu's internal dtb by adding ```-M dumpdtb=myboard.dtb``` to the end of your qemu command.
## Attaching GEF
[GEF](https://github.com/hugsy/gef) is an enhanced debugger which works extremely well for debugging code running in virtual machines.
It piggy-backs on gdb and offers a lot of valueable insight at a glance without requiring to know every gdb command.
Once GEF is installed, we can step through the process to attach gdb to qemu.
### Open gdb with our symbols.
First we run gdb pointed at our boot loader. We use the native ELF binary as that seems to give gdb/gef the most accurate knowledge
of our symbols. (the haiku_loader.u-boot is wrapped by u-boot's mkimage, your milage may vary based on platform)
```gdb objects/haiku/arm/release/system/boot/u-boot/boot_loader_u-boot```
### Set the architecture
This may not be required, but re-enforces to gef/gdb that we're working on arm.
```set architecture arm```
### Connect to QEMU
Now we tell gdb/gef about out running (but paused) QEMU instance.
```gef-remote -q localhost:1234```
A successful connection should occur.
### Step into debugging
Before you begin execution, it's handy to set a *breakpoint*. A *breakpoint* tells gdb/gef where it should pause execution to begin
the debugging process. All of our bootloaders start in a ```start_gen``` function, so this is a good place to start.
```breakpoint start_gen```
Now that a breakpoint is defined, lets run the virtual machine.
In gef, type ```continue```.
If everything is working as expected, you should now be "paused" at the ```start_gen``` function (hopefully showing the C/C++ code).
Now, you have a few commands to leverage:
* **step**
* Take a single step forward and execute the code listed.
* Does **not** step "into" functions, just over them getting the return from the code.
* Alias: s
* **stepi**
* step forward "into" the next code.
* If you're on a function it will enter the function and show the code executed.
* **break**
* add additional "breakpoints" where you can step through the code execution.
* **continue**
* Resume execution.
* If you have no additional breakpoints the code will "go do what it's supposed to"
* Alias: c
docs/develop/kernel/boot/Debugging_Bootloaders_GEF.rst
+137
View File
@@ -0,0 +1,137 @@
Bootloader debugging with GEF
=============================
When Haiku’s early boot process is experiencing unknown crashes or
faults, it can be extremely difficult to troubleshoot (especially when
serial, video, or other i/o devices are non-functional)
It **is** possible to step through the boot of any architecture of Haiku
in a debugger if the system boots and the issue can be reproduced in
qemu.
This works for any architecture and is *extremely* helpful to trouble
early platforms. Linux or Mac OS are requirements. You need a full
POSIX environment.
Building Haiku
--------------
On most non-x86 platforms, you will need a “kernel” (haiku_loader) and
an “initrd” (haiku_floppyboot).
For arm/arm64: ``jam -q @minimum-mmc``
Launching Haiku in QEMU
-----------------------
In the example below, we will prepare Haiku arm in QEMU for debugging.
::
qemu-system-arm -M raspi2 -kernel haiku_loader.u-boot -initrd haiku-floppyboot.tgz.u-boot -serial stdio -m 2G -dtb rpi2.dtb -s -S
**Key Flags:**
- **-s**
- Shorthand for -gdb tcp::1234, i.e. open a gdbserver on TCP port
1234.
- **-S**
- Do not start CPU at startup (you must type ‘c’ in the monitor).
These simple flags will make qemu listen for a debugger connection on
localhost:1234 and have the VM not start until you tell it to.
In the example above, we are Emulating a Raspberry Pi 2, and using
our Raspberry Pi 2 dtb. If you don’t have a dtb for the machine
you’re emulating, you can dump qemu’s internal dtb by adding
``-M dumpdtb=myboard.dtb`` to the end of your qemu command.
Attaching GEF
-------------
`GEF <https://github.com/hugsy/gef>`__ is an enhanced debugger which
works extremely well for debugging code running in virtual machines. It
piggy-backs on gdb and offers a lot of valueable insight at a glance
without requiring to know every gdb command.
Once GEF is installed, we can step through the process to attach gdb to
qemu.
Open gdb with our symbols.
~~~~~~~~~~~~~~~~~~~~~~~~~~
First we run gdb pointed at our boot loader. We use the native ELF
binary as that seems to give gdb/gef the most accurate knowledge of our
symbols. (the haiku_loader.u-boot is wrapped by u-boot’s mkimage, your
milage may vary based on platform)
``gdb objects/haiku/arm/release/system/boot/u-boot/boot_loader_u-boot``
Set the architecture
~~~~~~~~~~~~~~~~~~~~
This may not be required, but re-enforces to gef/gdb that we’re working
on arm.
``set architecture arm``
Connect to QEMU
~~~~~~~~~~~~~~~
Now we tell gdb/gef about out running (but paused) QEMU instance.
``gef-remote -q localhost:1234``
A successful connection should occur.
Step into debugging
~~~~~~~~~~~~~~~~~~~
Before you begin execution, it’s handy to set a *breakpoint*. A
*breakpoint* tells gdb/gef where it should pause execution to begin the
debugging process. All of our bootloaders start in a ``start_gen``
function, so this is a good place to start.
``breakpoint start_gen``
Now that a breakpoint is defined, lets run the virtual machine.
In gef, type ``continue``.
If everything is working as expected, you should now be “paused” at the
``start_gen`` function (hopefully showing the C/C++ code).
Now, you have a few commands to leverage:
- **step**
- Take a single step forward and execute the code listed.
- Does **not** step “into” functions, just over them getting the
return from the code.
- Alias: s
- **stepi**
- step forward “into” the next code.
- If you’re on a function it will enter the function and show the
code executed.
- **break**
- add additional “breakpoints” where you can step through the code
execution.
- **continue**
- Resume execution.
- If you have no additional breakpoints the code will “go do what
it’s supposed to”
- Alias: c
- **next**
- Resume execution until it reaches the next line of code.
- Useful for example to run until a loop is completed, and stop at the first line after that loop.
docs/develop/kernel/boot/boot_process_specs.html
-165
View File
@@ -1,165 +0,0 @@
<body bgcolor=white>
<h1>Haiku boot process specification</h1>
<h6>
Creation Date: November 23, 2002<br>
Version: 2.0 (Jan 22, 2021)<br>
Status: documenting the current state of things<br>
Author(s): Axel D&ouml;rfler, Adrien Destugues
</h6>
<h2>Overview</h2>
<p>Unlike other systems, Haiku comes with its own user-friendly bootloader. The main task of
the bootloader is to load and start the kernel. We don't have a concept of an initramfs as
Linux does, instead our bootloader is able to find the kernel and modules in a BFS partition,
and even extract them from packages as needed. It also provides an early boot menu that can
be used to change settings, boot older versions of Haiku that were snapshotted by the package
system, and write boot logs to USB mass storage.</p>
<h2>Booting from BIOS</h2>
<p>
Haiku BIOS boot loader process is split into 3 different stages. Since the second
stage is bound tightly to both other stages (which are independent from each other),
it is referred to as stage 1.5, whereas the other stages are referred to as stage 1
and 2. This architecture is used because the BIOS booting process only loads a very
small piece of code from disk for booting, insufficient for the needs outlined above.</p>
<p>The following will explain all stages in detail.</p>
<h3>Stage 1</h3>
<p>
The first stage is responsible for loading the real boot loader from a BFS disk. It
will be loaded by the Master Boot Record (MBR) and executed in the x86 real mode.
It is only used if the system will be booted directly from a BFS partition, it won't
be used at all if it is booted from a floppy disk or CD-ROM (in this case, stage
1.5 is in charge immediately).
</p>
<p>
It resides in the first 1024 bytes of a BFS disk which usually refers to the
first two sectors of the partition in question. Since the BFS superblock is located
at byte offset 512, and about 170 bytes large, this section is already reserved,
and thus cannot be used by the loader itself.<br>
The MBR only loads the first sector of a partition into memory, so it has to load
the superblock (and the rest of its implementation) by itself.
</p>
<p>
The loader must be able to load the real boot loader from a certain path, and
execute it. In BeOS this boot loader would be in "/boot/beos/system/zbeos",
in Haiku this is haiku_loader.bios_ia32 found in the haiku_loader package.<br>
Theoretically, it is enough to load the first few blocks from the loader, and
let the next stage then load the whole thing (which it has to do anyway if it
has been written on a floppy). This would be one possible optimization
if the 850 bytes of space are filled too early, but would require that "zbeos"
is written in one sequential block (which should be always the case anyway).
</p>
<h3>haiku_loader.bios_ia32</h3>
<p>
Contains both the stage 1.5 boot loader, and the compressed stage 2 loader.
It's not an ELF executable file; i.e. it can be directly written to a floppy
disk which would cause the BIOS to load the first 512 bytes of that file and
execute it.
</p>
<p>
Therefore, it will start with the stage 1.5 boot loader which will be loaded
either by the BIOS when it directly resides on the disk (for example when
loaded from a floppy disk), or the stage 1 boot loader, although this one
could have a different entry point than the BIOS.
</p>
<h3>Stage 1.5</h3>
<p>
Will have to load the rest of haiku_loader into memory (if not already done by the
stage 1 loader in case it has been loaded from a BFS disk), set up the global
descriptor table, switch to x86 protected mode, uncompress stage 2, and execute it.
</p>
<p>
This part is very similar to the stage 1 boot loader from NewOS.
</p>
<h3>Stage 2</h3>
<p>
This is the most complex part of the boot loader. In short, it has to load
any modules and devices the kernel needs to access the boot device, set up
the system, load the kernel, and execute it.
</p>
<p>
The kernel, and the modules and drivers needed are loaded from the boot
disk - therefore the loader has to be able to access BFS disks. It also
has to be able to load and parse the settings of these drivers (and the
kernel) from the boot disk, some of them are already important for the
boot loader itself (like "don't call the BIOS"). Since this stage is already
executed in protected mode, it has to use the virtual-86 mode to call the
BIOS and access any disk.
</p>
<p>
Before loading those files from the boot disk, it should look for additional
files located on a specific disk location after the "zbeos" file (on floppy disk
or CD-ROM). This way, it could access disks that cannot be accessed by the
BIOS itself.
</p>
<p>
Setting up the system for the kernel also means initalizing PCI devices needed
during the boot process before the kernel is up. It must be able to do so since
the BIOS might not have set up those devices correctly or at all.
</p>
<p>
It also must calculate a check sum for the boot device which the kernel can then
use to identify the boot volume and partition with - there is no other reliable
way to map BIOS disk IDs to the /dev/disk/... tree the system itself is using.
</p>
<p>
After having loaded and relocated the kernel, it executes it by passing a special
structure which tells the kernel things like the boot device check sum, which
modules are already loaded and where they are.
</p>
<p>
The stage 2 boot loader also includes user interaction. If the user presses a
special key during the boot process (like the space key, or some others as well),
a menu will be presented where the user can select the boot device (if several,
the loader has to scan for options), safe mode options, VESA mode, etc.
</p>
<p>
This menu may also come up if an error occured during the execution of the stage
2 loader.
</p>
<h2>Open Firmware</h2>
<p>On Open Firmware based systems, there is no need for a stage 1.5 because the firmware
does not give us as many constraints. Instead, the stage 2 is loaded directly by the firmware.
This requires converting the haiku_loader executable to the appropriate executable format
(a.out on sparc, pef on powerpc). The conversion is done using custom tools because binutils
does not support these formats anymore.</p>
<p>There is no notion of real and protected mode on non-x86 architectures, and the bootloader
is able to easily call Open Firmware methods to perform most tasks (disk access, network booting,
setting up the framebuffer) in a largely hardware-independent way.</p>
<h2>U-Boot</h2>
<p>U-Boot is able to load the stage2 loader directly from an ELF file. However, it does not
provide any other features. It is not possible for the bootloader to call into U-Boot APIs
for disk access, displaying messages on screen etc (while possible in theory, these features
are often disabled in U-Boot). This means haiku_loader would need to parse the FDT (describing
the available hardware) and bundle its own drivers for using the hardware. This approach is
not easy to set up, and it is recommended to instead use the UEFI support in U-Boot where
possible.</p>
<h2>EFI</h2>
<p>On EFI systems, there is no need for a stage1 loader as there is for BIOS. Instead, our stage2
loader (haiku_loader) can be executed directly from the EFI firmware.</p>
<p>The EFI firmware only knows how to run executables in the PE format
(as used by Windows) because Microsoft was involved in specifying it.
On x86_64, we can use binutils to output a PE file directly. But on other platforms, this is not
supported by binutils. So, what we do is generate a "fake" PE header and wrap our elf file inside
it. The bootloader then parses the embedded ELF header and relocates itself, so the other parts
of the code can be run.</p>
<p>After this initial loading phase, the process is very similar to the Open Firmware one. EFI
provides us with all the tools we need to do disk access and both text mode and framebuffer
output.</p>
</body>
docs/develop/kernel/boot/boot_process_specs.rst
+174
View File
@@ -0,0 +1,174 @@
Haiku boot process specification
================================
Creation Date: November 23, 2002
Version: 2.0 (Jan 22, 2021)
Status: documenting the current state of things
Author(s): Axel Dörfler, Adrien Destugues
Overview
--------
Unlike other systems, Haiku comes with its own user-friendly bootloader.
The main task of the bootloader is to load and start the kernel. We
don't have a concept of an initramfs as Linux does, instead our
bootloader is able to find the kernel and modules in a BFS partition,
and even extract them from packages as needed. It also provides an early
boot menu that can be used to change settings, boot older versions of
Haiku that were snapshotted by the package system, and write boot logs
to USB mass storage.
Booting from BIOS
-----------------
Haiku BIOS boot loader process is split into 3 different stages. Since
the second stage is bound tightly to both other stages (which are
independent from each other), it is referred to as stage 1.5, whereas
the other stages are referred to as stage 1 and 2. This architecture is
used because the BIOS booting process only loads a very small piece of
code from disk for booting, insufficient for the needs outlined above.
The following will explain all stages in detail.
Stage 1
~~~~~~~
The first stage is responsible for loading the real boot loader from a
BFS disk. It will be loaded by the Master Boot Record (MBR) and executed
in the x86 real mode. It is only used if the system will be booted
directly from a BFS partition, it won't be used at all if it is booted
from a floppy disk or CD-ROM (in this case, stage 1.5 is in charge
immediately).
| It resides in the first 1024 bytes of a BFS disk which usually refers
to the first two sectors of the partition in question. Since the BFS
superblock is located at byte offset 512, and about 170 bytes large,
this section is already reserved, and thus cannot be used by the
loader itself.
| The MBR only loads the first sector of a partition into memory, so it
has to load the superblock (and the rest of its implementation) by
itself.
| The loader must be able to load the real boot loader from a certain
path, and execute it. In BeOS this boot loader would be in
"/boot/beos/system/zbeos", in Haiku this is haiku_loader.bios_ia32
found in the haiku_loader package.
| Theoretically, it is enough to load the first few blocks from the
loader, and let the next stage then load the whole thing (which it has
to do anyway if it has been written on a floppy). This would be one
possible optimization if the 850 bytes of space are filled too early,
but would require that "zbeos" is written in one sequential block
(which should be always the case anyway).
haiku_loader.bios_ia32
~~~~~~~~~~~~~~~~~~~~~~
Contains both the stage 1.5 boot loader, and the compressed stage 2
loader. It's not an ELF executable file; i.e. it can be directly written
to a floppy disk which would cause the BIOS to load the first 512 bytes
of that file and execute it.
Therefore, it will start with the stage 1.5 boot loader which will be
loaded either by the BIOS when it directly resides on the disk (for
example when loaded from a floppy disk), or the stage 1 boot loader,
although this one could have a different entry point than the BIOS.
Stage 1.5
~~~~~~~~~
Will have to load the rest of haiku_loader into memory (if not already
done by the stage 1 loader in case it has been loaded from a BFS disk),
set up the global descriptor table, switch to x86 protected mode,
uncompress stage 2, and execute it.
This part is very similar to the stage 1 boot loader from NewOS.
Stage 2
~~~~~~~
This is the most complex part of the boot loader. In short, it has to
load any modules and devices the kernel needs to access the boot device,
set up the system, load the kernel, and execute it.
The kernel, and the modules and drivers needed are loaded from the boot
disk - therefore the loader has to be able to access BFS disks. It also
has to be able to load and parse the settings of these drivers (and the
kernel) from the boot disk, some of them are already important for the
boot loader itself (like "don't call the BIOS"). Since this stage is
already executed in protected mode, it has to use the virtual-86 mode to
call the BIOS and access any disk.
Before loading those files from the boot disk, it should look for
additional files located on a specific disk location after the "zbeos"
file (on floppy disk or CD-ROM). This way, it could access disks that
cannot be accessed by the BIOS itself.
Setting up the system for the kernel also means initalizing PCI devices
needed during the boot process before the kernel is up. It must be able
to do so since the BIOS might not have set up those devices correctly or
at all.
It also must calculate a check sum for the boot device which the kernel
can then use to identify the boot volume and partition with - there is
no other reliable way to map BIOS disk IDs to the /dev/disk/... tree the
system itself is using.
After having loaded and relocated the kernel, it executes it by passing
a special structure which tells the kernel things like the boot device
check sum, which modules are already loaded and where they are.
The stage 2 boot loader also includes user interaction. If the user
presses a special key during the boot process (like the space key, or
some others as well), a menu will be presented where the user can select
the boot device (if several, the loader has to scan for options), safe
mode options, VESA mode, etc.
This menu may also come up if an error occured during the execution of
the stage 2 loader.
Open Firmware
-------------
On Open Firmware based systems, there is no need for a stage 1.5 because
the firmware does not give us as many constraints. Instead, the stage 2
is loaded directly by the firmware. This requires converting the
haiku_loader executable to the appropriate executable format (a.out on
sparc, pef on powerpc). The conversion is done using custom tools
because binutils does not support these formats anymore.
There is no notion of real and protected mode on non-x86 architectures,
and the bootloader is able to easily call Open Firmware methods to
perform most tasks (disk access, network booting, setting up the
framebuffer) in a largely hardware-independent way.
U-Boot
------
U-Boot is able to load the stage2 loader directly from an ELF file.
However, it does not provide any other features. It is not possible for
the bootloader to call into U-Boot APIs for disk access, displaying
messages on screen etc (while possible in theory, these features are
often disabled in U-Boot). This means haiku_loader would need to parse
the FDT (describing the available hardware) and bundle its own drivers
for using the hardware. This approach is not easy to set up, and it is
recommended to instead use the UEFI support in U-Boot where possible.
EFI
---
On EFI systems, there is no need for a stage1 loader as there is for
BIOS. Instead, our stage2 loader (haiku_loader) can be executed directly
from the EFI firmware.
The EFI firmware only knows how to run executables in the PE format (as
used by Windows) because Microsoft was involved in specifying it. On
x86_64, we can use binutils to output a PE file directly. But on other
platforms, this is not supported by binutils. So, what we do is generate
a "fake" PE header and wrap our elf file inside it. The bootloader then
parses the embedded ELF header and relocates itself, so the other parts
of the code can be run.
After this initial loading phase, the process is very similar to the
Open Firmware one. EFI provides us with all the tools we need to do disk
access and both text mode and framebuffer output.