Get Linux kernel arm64 v5.8 to boot after the entry point offset was changed

Description

Patch: https://gem5-review.googlesource.com/c/public/gem5/+/35076

First, we already know that you need the following two unmerged changes to have a chance of booting:

The kernel commit is: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?h=v5.8&id=cfa7ede20f133cc81cef01dc3a516dda3a9721ee

Previously, the entry addresses ended in:

-TEXT_OFFSET := 0x00080000

and now the end in 0:

+TEXT_OFFSET := 0x0

In gem5 fs.py arm, we always relocate the kernel to be near 2GiB (0x80000000). This is done by masking the actual kernel addresses (which are way above 2GiB) and moving them close to the 2GiB with an offset.

Then we hardcode the 0x80000 offset in our bootloader: https://github.com/gem5/gem5/blob/fa70478413e4650d0058cbfe81fd5ce362101994/system/arm/bootloader/arm64/boot.S#L184

So the question is now is: how to have a single bootloader that boots both 5.7 and 5.8 which have different offsets.

Solution attempt 1: always relocate the first kernel instruction to be exactly at 0x80000000

https://gem5-review.googlesource.com/c/public/gem5/+/35078

I.e. move the previous v5.7 0x80080000 to that, and always jump to that address from bootloader.

This feels like the right approach but TODO breaks boot at:

so we see that the kernel appears to do a non-relocatable LDR and jump to the fetched value with BR (which is RIP relative).

Without the offset, the simulation goes past fine, a good offset if fetched:


Solution attempt 2: detect this offset in gem5 and inform bootloader

https://gem5-review.googlesource.com/c/public/gem5/+/35076

We can detect it by detect this offset by reading the _kernel_offset_le_lo32 kernel symbol and inform the bootloader by patching a memory location in the bootloader (_kernel_offset) from gem5 with the correct value.

Brain dump of key logs and code locations:

kernel v5.4.3 gem5 no patches

Kernel entry:

```
13500: system.cpu: A0 T0 : 0x80080000 : add x13, x18, #22 : IntAlu : D=0x0000000000000016 flags=(IsInteger)
14000: system.cpu: A0 T0 : 0x80080004 : b <__crc_crypto_alg_tested+278109> : IntAlu : flags=(IsControl|IsDirectControl|IsUncondControl)
```

Message:

```
info: Using kernel entry physical address at 0x80080000
```

FsWorkload::FsWorkload:

`kernelObj->entryPoint`(): 0xffffffc010080000
`loadAddrMask()`: 0xffffff
`loadAddrOffset()`: 0x80000000

`objdump -D`:

```
Disassembly of section .head.text:

ffffffc010080000 <__efistub__text>:
ffffffc010080000: 91005a4d add x13, x18, #0x16
ffffffc010080004: 1422bfff b ffffffc010930000 <stext>
ffffffc010080008: 00080000 .inst 0x00080000 ; undefined
ffffffc01008000c: 00000000 .inst 0x00000000 ; undefined
```

kernel v5.8 gem5 patched

Kernel entry:

```
13500: system.cpu: A0 T0 : 0x80000000 : add x13, x18, #22 : IntAlu : D=0x0000000000000016 flags=(IsInteger)
14000: system.cpu: A0 T0 : 0x80000004 : b <__crc_netdev_emerg+206220> : IntAlu : flags=(IsControl|IsDirectControl|IsUncondControl)
```

Message:

```
info: Using kernel entry physical address at 0x80000000
```

FsWorkload::FsWorkload:

`kernelObj->entryPoint()`: 0xffffffc010000000
`loadAddrMask()`: 0xffffff
`loadAddrOffset()`: 0x80000000

`objdump -D`:

```
Disassembly of section .head.text:

ffffffc010000000 <_text>:
ffffffc010000000: 91005a4d add x13, x18, #0x16
ffffffc010000004: 1424ffff b ffffffc010940000 <primary_entry>
ffffffc010000008: 00000000 .inst 0x00000000 ; undefined
ffffffc01000000c: 00000000 .inst 0x00000000 ; undefined
```

load_addr_offset and load_addr_mask

load_addr_offset comes directly from VExpress_GEM5_Base:

```
src/dev/arm/RealView.py:631: cur_sys.workload.load_addr_offset = load_offset
src/sim/Workload.py:50: load_addr_offset = Param.UInt64(0, "Address to offset the kernel with")
src/sim/kernel_workload.cc:35: _loadAddrMask(p.load_addr_mask), _loadAddrOffset(p.load_addr_offset),
```

```
class VExpress_GEM5_Base(RealView):
def setupBootLoader(self, cur_sys, boot_loader):
super(VExpress_GEM5_Base, self).setupBootLoader(
cur_sys, boot_loader, 0x8000000, 0x80000000)
```

```
FsWorkload::FsWorkload(Params *p) : KernelWorkload(*p)
{
if (kernelObj) {
kernelEntry = (kernelObj->entryPoint() & loadAddrMask()) +
loadAddrOffset();
}
```

load_addr_mask is set to 0 only on arm, and then autofixed in the constructor:

```
KernelWorkload::KernelWorkload(Params *p) : KernelWorkload(*p)
{
_start = image.minAddr();
_end = image.maxAddr()
if (_loadAddrMask == 0)
_loadAddrMask = mask(findMsbSet(_end - _start) + 1);
```

```
class KernelWorkload(Workload):
load_addr_mask = Param.UInt64(0xffffffffffffffff,
"Mask to apply to kernel addresses. If zero, "
"auto-calculated to be the most restrictive.")
```

```
class ArmFsLinux(ArmFsWorkload):
load_addr_mask = 0
```

kernelObj->entryPoint() is taken from the ELF entry point attribute:

```
class ObjectFile : public ImageFile
Addr entryPoint() const { return entry; }
```

```
class ElfObject : public ObjectFile

ElfObject::ElfObject(ImageFileDataPtr ifd) : ObjectFile(ifd)
{
entry = ehdr.e_entry;
```

Memory writing

```
class ElfObject : public ObjectFile
MemoryImage buildImage() const override { return image; }
```

```
bool
MemoryImage::write(const PortProxy &proxy) const
{
for (auto &seg: _segments)
if (!writeSegment(seg, proxy))
return false;
return true;
}
```

```
void
KernelWorkload::initState()
{
auto &phys_mem = system->physProxy;
/**

  • Load the kernel code into memory.
    */
    if (params().object_file != "") {
    image.write(phys_mem);
    ```

On kernel v5.8, `base = 0x80000000` for the segment, so at write time the offset has already been applied.

The changes appear to be done via:

```
KernelWorkload::KernelWorkload(const Params &p
image.move([this](Addr a) {
return (a & _loadAddrMask) + _loadAddrOffset;
})
```

Environment

None
Done

Assignee

Unassigned

Reporter

Ciro Santilli

Priority

Medium

Affects versions

None

Fix versions

None

Epic Link

None

Components

Labels

None