meta-rust

markyang92·2022년 10월 18일

yocto

목록 보기

42/53

solana build

meta-rust
- rust 관련 build를 위한 레이어
- yocto 4.0 (kirkstone)에서, openembedded-core에 정식 합류

solana 관련 component를 bitbake에서 빌드시, dependency error 이슈
- solana: https://github.com/solana-labs/solana

solana prerequisite (github)
$ sudo apt-get install libssl-dev libudev-dev pkg-config zlib1g-dev llvm clang cmake make libprotobuf-dev protobuf-compiler

solana 관련 레시피 작성

solana clone

$ git clone https://github.com/solana-labs/solana

target: solana/cli, solana/keygen
git uri: git://github.com/solana-labs/solana.git
commit: 05ffda9b16bfa9c784b22bbb0483f5a567373ae5

solana.inc + solana-cli_${PV}.bb

현재까지 문제

[RFC,1/2] cargo-update-recipe-crates.bbclass: add a class to generate SRC_URI crate lists from Cargo.lock

crate://의 문제

https://patchwork.yoctoproject.org/project/oe-core/patch/20220930175520.2850089-1-alex@linutronix.de/
rust는 dependency를 tar, git 으로 가져오는 경향 보다는, cargo를 사용하여 dependencies로 fetch
yocto의 bitbake는 위의 것을 SRC_URI에 crate://에 명시해 가져오는데, 이는 cargo를 사용하지 않는다.
- wget 을 사용한다.
- 우선 이러한 '리스트'를 만듦으로서, 레시피가 업데이트 될 때, 리스트된 것들을 업데이트하는데 문제를 일으킨다.

cargo-update-recipe-crates.bbclass는 이러한 업데이트를 지원케 한다.
- python3-bcrypt
- python 스크립트는 tomllib library와 관련있으며, 이는 Python 3.11에 있음
  - 따라서, oe-core에서 python3.11-rc2로 먼저 업데이트해야한다.

1. 레시피의 ${S}를 가져와
2. Cargo.lock을 발견하면, (아래 예시는 solana의 Cargo.lock)['package'] 를 가져와서 source, source 내부에 crates.io가 있으면
crate://crates.io/<name>/<version> 를 줄줄이 단다.

[RFC] python3: update 3.10.6 -> 3.11.0rc2

위에서 설명한대로, cargo-update-recipe-crates.bbclass를 사용하기 위해,
python 3.11.0rc2로 업그레이드한 컴포넌트를 넣는다.

patch: https://patchwork.yoctoproject.org/project/oe-core/patch/20220914120421.551923-1-alex@linutronix.de/

+) cargo-update-recipe-crates.bbclass

diff: https://git.openembedded.org/openembedded-core-contrib/commit/?h=jansa/master&id=73236b0fbcdb72c817666238f85dfeb9819cd2e2

use 4 spaces
avoid trailing space
add CARGO_LOC_SRC_DIR to allow searching outside ${S}
use ${BPN} in output file name

diff --git a/meta/classes-recipe/cargo-update-recipe-crates.bbclass b/meta/classes-recipe/cargo-update-recipe-crates.bbclass
index f90938c734..c25fe8ef3e 100644
--- a/meta/classes-recipe/cargo-update-recipe-crates.bbclass
+++ b/meta/classes-recipe/cargo-update-recipe-crates.bbclass
@@ -16,26 +16,30 @@
 addtask do_update_crates after do_patch
 do_update_crates[depends] = "python3-native:do_populate_sysroot"
 
+# The directory where to search for Cargo.lock files relative to the root directory ${S}
+CARGO_LOCK_SRC_DIR ??= ""
+
 do_update_crates() {
     nativepython3 - <<EOF
 
 def get_crates(f):
     import tomllib
-    c_list = 'SRC_URI += " \\ \n'
+    c_list = '# from %s' % os.path.relpath(f, '${S}')
+    c_list += '\nSRC_URI += " \\\'
     crates = tomllib.load(open(f, 'rb'))
     for c in crates['package']:
         if 'source' in c and 'crates.io' in c['source']:
-            c_list += "        crate://crates.io/{}/{} \\ \n".format(c['name'], c['version'])
-    c_list += '"\n'
+            c_list += '\n    crate://crates.io/%s/%s \\\' % (c['name'], c['version'])
+    c_list += '\n"\n'
     return c_list
 
 import os
 crates = "# Autogenerated with 'bitbake -c update_crates ${PN}'\n\n"
-for root, dirs, files in os.walk('${S}'):
+for root, dirs, files in os.walk('${S}/${CARGO_LOCK_SRC_DIR}'):
     for file in files:
         if file == 'Cargo.lock':
             crates += get_crates(os.path.join(root, file))
-open(os.path.join('${THISDIR}', '${PN}'+"-crates.inc"), 'w').write(crates)
+open(os.path.join('${THISDIR}', '${BPN}'+"-crates.inc"), 'w').write(crates)
 
 EOF
 }

CARGO_LOCK_SRC_DIR

do_compile[network] = "1"
- 링크 참고

solana recipes

recipes-A/solana/

solana-program-library recipes

recipes-A/solana-program-library/

Error

DEBUG: Executing shell function do_compile
NOTE: cargo build -v --target aarch64-unknown-linux --release --manifest-path=TOPDIR/BUILD/work/raspberrypi4_64-unknown-linux/solana-program-library/2.1.1-r0/git/token/cli//Cargo.toml 
error: failed to load manifest for workspace member `TOPDIR/BUILD/work/raspberrypi4_64-webos-linux/solana-program-library/2.1.1-r0/git/token-upgrade/cli`

Caused by:
  failed to load manifest for dependency `spl-token-client`

Caused by:
  failed to parse manifest at `TOPDIR/BUILD/work/raspberrypi4_64-unknown-linux/solana-program-library/2.1.1-r0/git/token/client/Cargo.toml`

Caused by:
  namespaced features with the `dep:` prefix are only allowed on the nightly channel and requires the `-Z namespaced-features` flag on the command-line
WARNING: exit code 101 from a shell command.

Older cargo isn't compatible!

또한, python3_pyo3.bbclass를 backport 해야한다.
- build 과정 중 python3-cryptography에서 Error발생
  - error: Error loading target specification: Could not find specification for target "aarch64-unknown-linux". Run rustc --print target-list for a list of built-in targets
- PyO3: https://github.com/PyO3/pyo3
  - rust에서 python을 사용 혹은 그 반대

rust-target-config

yocto 4.1의 rust-target-config.bbclass
BUILD HOST(native)를 x86-64로 사용
TUNE_FEATURES는 항상 target을 참조한다. 따라서 BUILD HOST(native)에서는 사용하길 원치않는다.TUNE_FEATURES에 vfpv4가 있으면, f에 +vfp4를 추가한다.
....

rust-common.bbclass

yocto 4.1의 rust-common.bbclassrustlibdir = "${libdir}/rustlib/${RUST_HOST_SYS}/lib
- FILES:${PN} += "${rustlibdir}/*.so"
- FILES:${PN}-dev += "${rustlibdir}/*.rlib ${rustlibdir}/*.rmeta "
- FILES:${PN}-dbg += "${rustlibdir}/.debug"

RUSTLIB = "-L ${STAGING_DIR_HOST}${rustlibdir}"
RUST_DEBUG_REMAP = "--remap-path-prefix=${WORKDIR}=/usr/src/debug/${PN}/${EXTENDPE}${PV}-${PR}"
RUSTFALGS += "${RUSTLIB} ${RUST_DEBUG_REMAP}"
RUSTLIB_DEP ?= "libstd-rs"
RUST_PANIC_STRATEGY ?= "unwind"

RUST_LIBC는 제거 되었음, backport된 레시피는 필요치 않음. 하지만 overlayed bbclass는 kirkstone의 원래 rust-crosssdk 레시피 parsing에 실패하게 만든다.
- 이 레이어의 1.63.0 버전이 PREFERRED 될 것이기 때문에 결국엔 사용되지는 않을 것임
- rust-crosssdk_1.59.0.bb

rust_base_triple
- Mangle bitbake's *_SYS into something that rust might support (see rust/mk/cfg/* for a list
- Note that os is assumed to be some linux form

RUST_BUILD_ARCH = "x86_64"

Naming explanation
- Yocto
  - BUILD_SYS: Yocto triple of the build environment
  - HOST_SYS: What we're building for in Yocto
  - TARGET_SYS: What we're building for in Yocto
  - So when building -native
    - BUILD_SYS == HOST_SYS == TARGET_SYS
  - 일반적인 Component
    - HOST_SYS == TARGET_SYS
- Rust
  - TARGET: the system where the binary will run
  - HOST : the system whrere the binary is being built

`build-rust-cc`

#!/usr/bin/env python3
import os, sys
orig_binary = "gcc "
extras = ""
binary = orig_binary.split()[0]
args = orig_binary.split() + sys.argv[1:]
if extras:
	args.append(extras)
os.execvp(binary, args)

`target-rust-cc`

#!/usr/bin/env python3
import os, sys
orig_binary = "aarch64-webos-linux-gcc  -mcpu=cortex-a72 -march=armv8-a+crc -fstack-protector-strong  -O2 -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -Werror=format-security -Werror=return-type --sysroot=/home/dhyang/workspace/build-webos/BUILD/work/raspberrypi4_64-webos-linux/solana-cli/1.14.5-r0/recipe-sysroot -Wl,-O1 -Wl,--hash-style=gnu -Wl,--as-needed -fmacro-prefix-map=/home/dhyang/workspace/build-webos/BUILD/work/raspberrypi4_64-webos-linux/solana-cli/1.14.5-r0=/usr/src/debug/solana-cli/1.14.5-r0                      -fdebug-prefix-map=/home/dhyang/workspace/build-webos/BUILD/work/raspberrypi4_64-webos-linux/solana-cli/1.14.5-r0=/usr/src/debug/solana-cli/1.14.5-r0                      -fdebug-prefix-map=/home/dhyang/workspace/build-webos/BUILD/work/raspberrypi4_64-webos-linux/solana-cli/1.14.5-r0/recipe-sysroot=                      -fdebug-prefix-map=/home/dhyang/workspace/build-webos/BUILD/work/raspberrypi4_64-webos-linux/solana-cli/1.14.5-r0/recipe-sysroot-native=  -Wl,-z,relro,-z,now"
extras = ""
binary = orig_binary.split()[0]
args = orig_binary.split() + sys.argv[1:]
if extras:
    args.append(extras)
os.execvp(binary, args)

solana workdir

`${CARGO_HOME}/config`

# EXTRA_OECARGO_PATHS
paths = [
]

# Local mirror vendored by bitbake
[source.bitbake]
directory = "/home/dhyang/workspace/build-webos/BUILD/work/raspberrypi4_64-<os>-linux/solana-cli/1.14.5-r0/cargo_home/bitbake"

[source.crates-io]
replace-with = "bitbake"
local-registry = "/nonexistant"

[http]
# Multiplexing can't be enabled because http2 can't be enabled
# in curl-native without dependency loops
multiplexing = false

# Ignore the hard coded and incorrect path to certificates
cainfo = "/home/dhyang/workspace/build-webos/BUILD/work/raspberrypi4_64-<os>-linux/solana-cli/1.14.5-r0/recipe-sysroot-native/etc/ssl/certs/ca-certificates.crt"


# HOST_SYS
[target.aarch64-webos-linux-gnu]
linker = "/home/dhyang/workspace/build-webos/BUILD/work/raspberrypi4_64-<os>-linux/solana-cli/1.14.5-r0/wrapper/target-rust-ccld"

# BUILD_SYS
[target.x86_64-linux-gnu]
linker = "/home/dhyang/workspace/build-webos/BUILD/work/raspberrypi4_64-<os>-linux/solana-cli/1.14.5-r0/wrapper/build-rust-ccld"

[build]
# Use out of tree build destination to avoid poluting the source tree
target-dir = "/home/dhyang/workspace/build-webos/BUILD/work/raspberrypi4_64-<os>-linux/solana-cli/1.14.5-r0/build/target"

[term]
progress.when = 'always'
progress.width = 80

`openssl-sys` 문제

solana-program-library를 빌드할 때,
error: failed to run custom build command for 'openssl-sys v0.9.60가 뜬다.
https://github.com/sfackler/rust-openssl/issues/1430 (rust-openssl)

thread 'main' panicked at 'don't know how to configure OpenSSL for \
aarch64-webos-linux-gnu', \
TOPDIR/BUILD/work/raspberrypi4_64-<os>-linux/solana-program-library/2.1.1-r0/\
cargo_home/bitbake/openssl-src-111.22.0+1.1.1q/src/lib.rs:294:18
  stack backtrace:
     0: rust_begin_unwind

solana-program-library에 DEPENDS에 openssl을 추가하고, export OPENSSL_NO_VENDOR = "1" 문장 추가.

markyang92

pllpokko@alumni.kaist.ac.kr

이전 포스트

meta/classes

다음 포스트

meta-rust

yocto

solana build

solana 관련 레시피 작성

solana.inc + solana-cli_${PV}.bb

현재까지 문제

[RFC,1/2] cargo-update-recipe-crates.bbclass: add a class to generate SRC_URI crate lists from Cargo.lock

[RFC] python3: update 3.10.6 -> 3.11.0rc2

+) cargo-update-recipe-crates.bbclass

solana recipes

solana-program-library recipes

rust-target-config

rust-common.bbclass

`build-rust-cc`

`target-rust-cc`

solana workdir

`${CARGO_HOME}/config`

`openssl-sys` 문제

meta/classes

meta-rust-bin, fakeroot

0개의 댓글

meta-rust

yocto

solana build

solana 관련 레시피 작성

solana.inc + solana-cli_${PV}.bb

현재까지 문제

[RFC,1/2] cargo-update-recipe-crates.bbclass: add a class to generate SRC_URI crate lists from Cargo.lock

[RFC] python3: update 3.10.6 -> 3.11.0rc2

+) cargo-update-recipe-crates.bbclass

solana recipes

solana-program-library recipes

rust-target-config

rust-common.bbclass

build-rust-cc

target-rust-cc

solana workdir

${CARGO_HOME}/config

openssl-sys 문제

meta/classes

meta-rust-bin, fakeroot

0개의 댓글

`build-rust-cc`

`target-rust-cc`

`${CARGO_HOME}/config`

`openssl-sys` 문제