Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[WIP] feat: add basic FreeBSD support on Kubelet #115870

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

[WIP] feat: add basic FreeBSD support on Kubelet #115870

wants to merge 1 commit into from
+1,579 −487

Conversation

nerzhul
Copy link

@nerzhul nerzhul commented Feb 18, 2023 •
edited

What type of PR is this?

/kind feature

What this PR does / why we need it:

This PR as the first bricks to make kubelet start on FreeBSD and talk with CRI runtime (in my case, containerd+runj) to run (for now) FreeBSD containers.

Special notes for your reviewer:

There is some limitations:

  • i don't find a way from Kubelet to run platform=linux containers, whereas talking directly to containerd permit to run any linux container on freebsd
  • only hostNetwork=true containers are supported for now, as there is no CRI on FreeBSD with network namespacing for now, as i tested
  • i didn't tested yet pod scheduled from Kubernetes control plane, just static pods, but it should not be an issue

Here is the logging startup of the kubelet

root@freebsd-13:~/kubernetes # ./kubelet --container-runtime-endpoint=unix:///run/containerd/containerd.sock --pod-manifest-path /usr/local/etc/kubernetes/manifests/
Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --pod-manifest-path has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
I0218 14:12:32.305939    6881 server.go:412] "Kubelet version" kubeletVersion="v0.0.0-master+$Format:%H$"
I0218 14:12:32.306036    6881 server.go:414] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
I0218 14:12:32.306465    6881 server.go:575] "Standalone mode, no API client"
I0218 14:12:32.306589    6881 server.go:463] "No api server defined - no events will be sent to API server"
I0218 14:12:32.306628    6881 server.go:659] "--cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /"
I0218 14:12:32.306738    6881 fake_topology_manager.go:33] "NewFakeManager"
I0218 14:12:32.306768    6881 container_manager_freebsd.go:116] "Creating device plugin manager"
I0218 14:12:32.306876    6881 fake_topology_manager.go:68] "AddHintProvider" hintProvider=&{checkpointdir:/var/lib/kubelet/device-plugins/ endpoints:map[] mutex:{state:0 sema:0} server:0x8585f0e00 activePods:0x5852d20 sourcesReady:0x7378ee0 allDevices:map[] healthyDevices:map[] unhealthyDevices:map[] allocatedDevices:map[] podDevices:0x858129560 checkpointManager:0x858bec1e0 numaNodes:[] topologyAffinityStore:0x85829f3b0 devicesToReuse:map[] pendingAdmissionPod:<nil>}
I0218 14:12:32.306930    6881 server.go:775] "Failed to ApplyOOMScoreAdj" err="setting OOM scores is unsupported in this build"
I0218 14:12:32.311433    6881 kubelet.go:401] "Kubelet is running in standalone mode, will skip API server sync"
I0218 14:12:32.311473    6881 kubelet.go:287] "Adding static pod path" path="/usr/local/etc/kubernetes/manifests/"
E0218 14:12:32.311519    6881 file_unsupported.go:29] "Watching source file is unsupported in this build"
I0218 14:12:32.311623    6881 state_mem.go:36] "Initialized new in-memory state store"
I0218 14:12:32.311643    6881 state_mem.go:35] "Initializing new in-memory state store"
I0218 14:12:32.311658    6881 fake_topology_manager.go:33] "NewFakeManager"
I0218 14:12:32.312266    6881 kuberuntime_manager.go:244] "Container runtime initialized" containerRuntime="containerd" version="1.7.0-beta.4" apiVersion="v1"
I0218 14:12:32.312440    6881 volume_host.go:75] "KubeClient is nil. Skip initialization of CSIDriverLister"
W0218 14:12:32.312863    6881 csi_plugin.go:189] kubernetes.io/csi: kubeclient not set, assuming standalone kubelet
W0218 14:12:32.312891    6881 csi_plugin.go:266] Skipping CSINode initialization, kubelet running in standalone mode
I0218 14:12:32.313094    6881 server.go:1186] "Started kubelet"
I0218 14:12:32.313125    6881 kubelet.go:1499] "No API server defined - no node status update will be sent"
I0218 14:12:32.313171    6881 server.go:193] "Starting to listen read-only" address="0.0.0.0" port=10255
I0218 14:12:32.313961    6881 server.go:161] "Starting to listen" address="0.0.0.0" port=10250
I0218 14:12:32.316062    6881 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
I0218 14:12:32.316240    6881 status_manager.go:172] "Kubernetes client is nil, not starting status manager"
I0218 14:12:32.316321    6881 kubelet.go:2110] "Starting kubelet main sync loop"
E0218 14:12:32.316528    6881 kubelet.go:2134] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
E0218 14:12:32.316555    6881 processstarttime.go:40] Could not get process start time, stat /proc/6881: no such file or directory
I0218 14:12:32.316742    6881 desired_state_of_world_populator.go:145] "Desired state populator starts to run"
I0218 14:12:32.316880    6881 server.go:451] "Adding debug handlers to kubelet server"
I0218 14:12:32.316703    6881 volume_manager.go:284] "Starting Kubelet Volume Manager"
E0218 14:12:32.318584    6881 processstarttime.go:40] Could not get process start time, stat /proc/6881: no such file or directory
I0218 14:12:32.321718    6881 manager.go:455] "Failed to read data from checkpoint" checkpoint="kubelet_internal_checkpoint" err="checkpoint is not found"
I0218 14:12:32.322345    6881 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
E0218 14:12:32.324506    6881 eviction_manager.go:261] "Eviction manager: failed to get summary stats" err="failed to get root cgroup stats: failed to get cgroup stats for \"/\": unexpected number of containers: 0"
I0218 14:12:32.517040    6881 desired_state_of_world_populator.go:153] "Finished populating initial desired state of world"
I0218 14:12:32.522958    6881 reconciler.go:41] "Reconciler: start to sync state"
I0218 14:12:32.632885    6881 fake_cpu_manager.go:51] "AddContainer" pod="default/static-web-freebsd-13.1" containerName="main" containerID="f2f62c1a2934922e94545b717298389e84e392cdc2fba1115d648af48069f8b6"
I0218 14:12:32.632909    6881 fake_memory_manager.go:50] "Add container" pod="default/static-web-freebsd-13.1" containerName="main" containerID="f2f62c1a2934922e94545b717298389e84e392cdc2fba1115d648af48069f8b6"
I0218 14:12:32.632920    6881 fake_topology_manager.go:72] "AddContainer" pod="default/static-web-freebsd-13.1" containerName="main" containerID="f2f62c1a2934922e94545b717298389e84e392cdc2fba1115d648af48069f8b6"
E0218 14:12:42.326275    6881 eviction_manager.go:261] "Eviction manager: failed to get summary stats" err="failed to get root cgroup stats: failed to get cgroup stats for \"/\": unexpected number of containers: 0"

Does this PR introduce a user-facing change?

NONE

add kubelet FreeBSD vanilla support

@k8s-ci-robot k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Feb 18, 2023
@k8s-ci-robot
Copy link
Contributor

Welcome @nerzhul!

It looks like this is your first PR to kubernetes/kubernetes 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/kubernetes has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

@k8s-ci-robot k8s-ci-robot added do-not-merge/needs-kind Indicates a PR lacks a `kind/foo` label and requires one. do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Feb 18, 2023
@k8s-ci-robot
Copy link
Contributor

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added needs-priority Indicates a PR lacks a `priority/foo` label and requires one. kind/feature Categorizes issue or PR as related to a new feature. labels Feb 18, 2023
@k8s-ci-robot k8s-ci-robot added sig/node Categorizes an issue or PR as relevant to SIG Node. sig/storage Categorizes an issue or PR as relevant to SIG Storage. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. and removed do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Feb 18, 2023
@k8s-ci-robot
Copy link
Contributor

Hi @nerzhul. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot removed the do-not-merge/needs-kind Indicates a PR lacks a `kind/foo` label and requires one. label Feb 18, 2023
@aojea
Copy link
Member

aojea commented Feb 18, 2023

Adding support for a new OS seems something that should be discussed and properly documented in a KEP so it can be feature gated

https://github.com/kubernetes/enhancements

@nerzhul
Copy link
Author

nerzhul commented Feb 18, 2023

@aojea i'm not familiar with KEP, i'm first trying to have all working. For now we miss many functions, and i didn't suceed yet to have a fully working container, i suspect it's somewhere in runc, but kubelet sounds to do the job with containerd to create a hostNetwork container.

@k8s-ci-robot k8s-ci-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Feb 19, 2023
@linux-foundation-easycla
Copy link

linux-foundation-easycla bot commented Feb 19, 2023 •
edited

CLA Signed

The committers listed above are authorized under a signed CLA.

@k8s-ci-robot k8s-ci-robot added cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. and removed cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Feb 19, 2023
@bart0sh bart0sh added this to Triage in SIG Node PR Triage Feb 19, 2023
@bart0sh bart0sh moved this from Triage to WIP in SIG Node PR Triage Feb 19, 2023
@aojea
Copy link
Member

aojea commented Feb 19, 2023

@aojea i'm not familiar with KEP, i'm first trying to have all workin

ok, is just to set expectations, if you are planning to get this change into this repository you should talk first with SIG-Node members to give you guidance and also to verify that this is aligned with their roadmap ...

@nerzhul
Copy link
Author

nerzhul commented Feb 19, 2023

as it's totally random PR from a people using massively Kubernetes on Linux at work, i think it's not on the roadmap, i just know that FreeBSD wants to enhance their containerization to be more modern, and Kubernetes is a de-facto standard in the industry. For now, let's keep it as a WIP i'll enhance and rebase on free time, for reseach (but it's working, for the first commit, i'm working on the next ones)

@k8s-ci-robot k8s-ci-robot added area/apiserver area/dependency Issues or PRs related to dependency changes area/test sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. sig/testing Categorizes an issue or PR as relevant to SIG Testing. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Feb 19, 2023
@nerzhul
feat(porting/freebsd): make kubelet start & interact with CRI on FreeBSD
602a187 
This is a vanilla porting making basic parts of the kubelet interact
with FreeBSD and containerd 1.7.0b4 (for now) on FreeBSD
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: nerzhul
Once this PR has been reviewed and has the lgtm label, please assign smarterclayton for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@cici37
Copy link
Contributor

cici37 commented Feb 23, 2023

/remove-sig api-machinery

@k8s-ci-robot k8s-ci-robot removed the sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. label Feb 23, 2023
@SergeyKanzhelev SergeyKanzhelev moved this from Triage to PRs Waiting on Author in SIG Node CI/Test Board Mar 1, 2023
@k8s-ci-robot k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 17, 2023
@k8s-ci-robot
Copy link
Contributor

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gnufied gnufied

@liggitt liggitt

Assignees
No one assigned
Labels
area/apiserver area/dependency Issues or PRs related to dependency changes area/kubelet area/test cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. kind/feature Categorizes issue or PR as related to a new feature. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/node Categorizes an issue or PR as relevant to SIG Node. sig/storage Categorizes an issue or PR as relevant to SIG Storage. sig/testing Categorizes an issue or PR as relevant to SIG Testing. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.
Projects
SIG Node CI/Test Board
PRs Waiting on Author
SIG Node PR Triage
WIP
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@nerzhul @k8s-ci-robot @aojea @cici37