ubuntu22 커널디버깅 환경구성

Cute_Security15·2023년 8월 26일

커널

목록 보기

8/10

환경

vmware workstation 17 pro
2개의 ubuntu22 vm (1개는 gdbserver 용, 다른 하나는 target)

목적

crash 세션 및 gdb pwndbg 세션이 붙는걸 확인

커널 자동업데이트 hold

커널심볼 크기가 크므로, 커널 변경안되게 자동업데이트 hold

gdbserver 쪽과 target 쪽 vm 에 둘다 적용
https://connectwww.com/how-to-prevent-ubuntu-kernel-update-disable-or-stop-linux-kernel-update/62052

sudo apt-mark hold linux-image-generic linux-headers-generic

커널소스

커널소스 git tag 확인 : 6.2.0-26.26~22.04.1-generic

root@user-virtual-machine:~/jammy# cat /proc/version_signature
Ubuntu 6.2.0-26.26~22.04.1-generic 6.2.13
root@user-virtual-machine:~/jammy#

codename 확인 : jammy

root@user-virtual-machine:/boot# lsb_release -cs
jammy

git clone 후 tag로 checkout

hwe? : 신규 hardware support 목적의 커널 (ga 커널보다 최신)
https://askubuntu.com/questions/248914/what-is-hardware-enablement-hwe

git clone https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/jammy
cd jammy

git tag -l | grep 6.2.0-26.26
Ubuntu-hwe-6.2-6.2.0-26.26_22.04.1

git checkout Ubuntu-hwe-6.2-6.2.0-26.26_22.04.1

커널심볼

ddebs.ubuntu.com 에서 다운로드 가능

가이드 : https://wiki.ubuntu.com/Debug%20Symbol%20Packages

ddebs.ubuntu.com jammy, jammy-updates, jammy-proposed repository 추가

echo "deb http://ddebs.ubuntu.com $(lsb_release -cs) main restricted universe multiverse
deb http://ddebs.ubuntu.com $(lsb_release -cs)-updates main restricted universe multiverse
deb http://ddebs.ubuntu.com $(lsb_release -cs)-proposed main restricted universe multiverse" | \
sudo tee -a /etc/apt/sources.list.d/ddebs.list

signed key 설치후 업데이트

sudo apt install ubuntu-dbgsym-keyring
sudo apt-get update

커널심볼 설치 : "약 1시간" 소요

apt-get install linux-image-unsigned-6.2.0-26-generic-dbgsym linux-image-6.2.0-26-generic-dbgsym

The following NEW packages will be installed:
  linux-image-6.2.0-26-generic-dbgsym linux-image-unsigned-6.2.0-26-generic-dbgsym
0 upgraded, 2 newly installed, 0 to remove and 20 not upgraded.
Need to get 1,109 MB of archives.
After this operation, 7,521 MB of additional disk space will be used.
Get:1 http://ddebs.ubuntu.com jammy-updates/main amd64 linux-image-unsigned-6.2.0-26-generic-dbgsym amd64 6.2.0-26.26~22.0                                                    4.1 [1,109 MB]
Get:2 http://ddebs.ubuntu.com jammy-updates/main amd64 linux-image-6.2.0-26-generic-dbgsym amd64 6.2.0-26.26~22.04.1 [18.8 kB]
Fetched 1,109 MB in 1h 21min 10s (228 kB/s)
Selecting previously unselected package linux-image-unsigned-6.2.0-26-generic-dbgsym.
(Reading database ... 174100 files and directories currently installed.)
Preparing to unpack .../linux-image-unsigned-6.2.0-26-generic-dbgsym_6.2.0-26.26~22.04.1_amd64.ddeb ...
Unpacking linux-image-unsigned-6.2.0-26-generic-dbgsym (6.2.0-26.26~22.04.1) ...
Selecting previously unselected package linux-image-6.2.0-26-generic-dbgsym.
Preparing to unpack .../linux-image-6.2.0-26-generic-dbgsym_6.2.0-26.26~22.04.1_amd64.ddeb ...
Unpacking linux-image-6.2.0-26-generic-dbgsym (6.2.0-26.26~22.04.1) ...
Setting up linux-image-unsigned-6.2.0-26-generic-dbgsym (6.2.0-26.26~22.04.1) ...
Setting up linux-image-6.2.0-26-generic-dbgsym (6.2.0-26.26~22.04.1) ...

root@user-virtual-machine:/usr/lib/debug/boot# ls -lh
total 765M
-rw-r--r-- 1 root root 764M  7월 13 22:22 vmlinux-6.2.0-26-generic
root@user-virtual-machine:/usr/lib/debug/boot#

설치된 패키지의 상세정보는 ddebs.ubuntu.com 의 Package 인덱스파일에서 확인가능
http://ddebs.ubuntu.com/dists/jammy-updates/main/binary-amd64/Packages

Packages 인덱스파일 포맷설명 : https://wiki.debian.org/DebianRepository/Format?action=show&redirect=RepositoryFormat#A.22Packages.22_Indices

Package: linux-image-6.2.0-26-generic-dbgsym
Architecture: amd64
Version: 6.2.0-26.26~22.04.1
Priority: optional
Section: devel
Source: linux-signed-hwe-6.2
Maintainer: Canonical Kernel Team <kernel-team@lists.ubuntu.com>
Installed-Size: 25
Depends: linux-image-unsigned-6.2.0-26-generic-dbgsym
Filename: pool/main/l/linux-signed-hwe-6.2/linux-image-6.2.0-26-generic-dbgsym_6.2.0-26.26~22.04.1_amd64.ddeb
Size: 18802
MD5sum: 280a10f6b8ab3c14bb53399be00dcef2
SHA1: 3d898d0cebf43cc0c04c3760e479c865a0d2ffc0
SHA256: a639fd262962bc590c51fb8242b3168707d5077427f62d097d042a6eb155fb1f
SHA512: bdd2ec3e15b3691fc7750f80fa40a076515c9cbf7f76079c966f3cd3eef8b7e17c61ef4887d9cfe84922ee6b9fc3c7082f3bc0cfd61e8f2341b550b7991ae12c
Description: Signed kernel image generic
 A link to the debugging symbols for the generic signed kernel.

crash 업데이트

crash 는 struct kmem_cache 내용을 읽을때 cpu_slab 필드에 의존하는데, 이 필드는 linux 5.17-rc1 에서 사라졌으므로, 다음 에러가 발생한다.

crash: invalid structure member offset: kmem_cache_s_num
       FILE: memory.c  LINE: 9619  FUNCTION: kmem_cache_init()

[/usr/bin/crash] error trace: 5608e26a269e => 5608e26762f4 => 5608e274411b => 5608e274409c

ubuntu22 엔 아직 업데이트가 되지 않았으므로, crash 8.0.2 를 수동빌드하여 사용

debian changelog : https://tracker.debian.org/media/packages/c/crash/changelog-8.0.2-1
ubuntu22 bug report : https://bugs.launchpad.net/ubuntu/+source/crash/+bug/2012060

# crash 빌드전에, 의존성 패키지 설치
apt-get install g++ texinfo bison zlib1g-dev ncurses-dev

git clone https://github.com/crash-utility/crash.git
cd crash
git checkout 8.0.2
make
make install

root@user-virtual-machine:~/crash# crash --version
crash 8.0.2

crash 세션 확인

      KERNEL: /usr/lib/debug/boot/vmlinux-6.2.0-26-generic  [TAINTED]
    DUMPFILE: /proc/kcore
        CPUS: 1 [OFFLINE: -1]
        DATE: Sun Aug 27 05:43:49 KST 2023
      UPTIME: 00:10:02
LOAD AVERAGE: 0.61, 0.83, 0.72
       TASKS: 464
    NODENAME: user-virtual-machine
     RELEASE: 6.2.0-26-generic
     VERSION: #26~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Jul 13 16:27:29 UTC 2
     MACHINE: x86_64  (2611 Mhz)
      MEMORY: 4 GB
         PID: 2505
     COMMAND: "crash"
        TASK: ffff95dd18c44bc0  [THREAD_INFO: ffff95dd18c44bc0]
         CPU: 0
       STATE: TASK_RUNNING (ACTIVE)

crash>

pwndbg 설치

주요 기능들 : https://github.com/pwndbg/pwndbg/blob/dev/FEATURES.md

git clone https://github.com/pwndbg/pwndbg
cd pwndbg
./setup.sh

target vm 에 vmware debugStub 설정추가

vm 설정파일(.vmx) 에 4줄추가 후 재부팅

가이드 : https://wiki.osdev.org/VMware#Guest_debugging

debugStub.listen.guest64 ="TRUE"

# Allows debugging from a different computer / VM instead of localhost. 
# The IP for remote debugging will be that of the host.
debugStub.listen.guest64.remote = "TRUE"

# Enables the use of hardware breakpoints instead of software (INT3) breakpoints
debugStub.hideBreakpoints = "TRUE"

# "VMware Workstation is listening for debug connection on port 55555." into the vmware.log file
debugStub.port.guest64 = "55555"

# Breaks into debug stub on first instruction (warning: in BIOS!) # This will halt the VM at the very first instruction at 0xFFFF0, you could set the next breakpoint to break *0x7c00 to break when the bootloader is loaded by the BIOS
monitor.debugOnStartGuest64 = "TRUE"

gdb 세션확인

gdbserver vm 에서 gdb 실행

gdb /usr/lib/debug/boot/vmlinux-6.2.0-26-generic

# xxx 는 vmnet8 nat network 설정을 확인하거나 ifconfig 로 확인가능 
# (ex. 192.168.171.1)
target remote 192.168.xxx.1:55555

break(CTRL+C) 후 bt 확인

주의사항

Hyper-V 가 활성화 되어있는 window 에서는 vmware workstation 에 내장된 gdb stub 이 동작하지 않는 문제가 있었다.
구체적으로는, bp 를 설치하고 c 하면 target vm 이 종료되는 문제이다.

(gdb) b ckltm_get_process_path
Breakpoint 1 at 0xffffffffc0803f40: file /root/ckl/Src/tools/ckltm/ckltm_network.c, line 30.
(gdb) c
Continuing.
Remote communication error.  Target disconnected.: Connection reset by peer.

vmware.log 를 보면 NOT_IMPLEMENTED 로그가 찍혀있다.

2024-01-10T16:30:22.047+09:00| vcpu-1| E001: PANIC: NOT_IMPLEMENTED D:\build\ob\bora-16894299\bora\vmcore\monitor\common\main\vmm\debugReg.c:287

vmware 커뮤니티를 검색해보니 Hyper-V 를 disable 한 후에 재부팅후 해보라고 권고하고 있다.
그대로 실행한 후 잘되는걸 확인

https://communities.vmware.com/t5/VMware-Workstation-Pro/Crash-after-resuming-execution-from-gdb-stub/td-p/2824667

(gdb) b ckltm_get_process_path
Breakpoint 1 at 0xffffffffc0b73f40: file /root/ckl/Src/tools/ckltm/ckltm_network.c, line 30.
(gdb) c
Continuing.

Cute_Security15

관심분야 : Filesystem, Data structure, user/kernel IPC

이전 포스트

bpftrace

다음 포스트

vmlinux-gdb.py

1개의 댓글

Cute_Security15

2023년 9월 11일

커널 자동업데이트 삭제
https://connectwww.com/how-to-prevent-ubuntu-kernel-update-disable-or-stop-linux-kernel-update/62052/
커널 버전 고정
https://unix.stackexchange.com/a/668763

답글 달기