Developer Guide

Jenkins Operator for developers

This document explains how to setup your development environment.

Prerequisites

Clone repository and download dependencies

git clone git@github.com:jenkinsci/kubernetes-operator.git
cd kubernetes-operator
make go-dependencies

Build and run with a minikube

Start minikube instance configured for Jenkins Operator. Appropriate minikube version will be downloaded to bin folder.

make minikube-start

Next run Jenkins Operator locally.

make run

Console output indicating readiness of this phase:

+ build
+ run
kubectl config use-context minikube
Switched to context "minikube".
Watching 'default' namespace
bin/manager --jenkins-api-hostname=192.168.99.252 --jenkins-api-port=0 --jenkins-api-use-nodeport=true --cluster-domain=cluster.local 
2021-02-08T14:14:45.263+0100    INFO    cmd     Version: v0.5.0
2021-02-08T14:14:45.263+0100    INFO    cmd     Git commit: 305dbeda-dirty-dirty
2021-02-08T14:14:45.264+0100    INFO    cmd     Go Version: go1.15.6
2021-02-08T14:14:45.264+0100    INFO    cmd     Go OS/Arch: darwin/amd64
2021-02-08T14:14:45.264+0100    INFO    cmd     Watch namespace: default
2021-02-08T14:14:45.592+0100    INFO    controller-runtime.metrics      metrics server is starting to listen    {"addr": "0.0.0.0:8383"}
2021-02-08T14:14:45.599+0100    INFO    cmd     starting manager
2021-02-08T14:14:45.599+0100    INFO    controller-runtime.manager      starting metrics server {"path": "/metrics"}
2021-02-08T14:14:45.599+0100    INFO    controller-runtime.manager.controller.jenkins   Starting EventSource    {"reconciler group": "jenkins.io", "reconciler kind": "Jenkins", "source": "kind source: jenkins.io/v1alpha2, Kind=Jenkins"}
2021-02-08T14:14:45.700+0100    INFO    controller-runtime.manager.controller.jenkins   Starting EventSource    {"reconciler group": "jenkins.io", "reconciler kind": "Jenkins", "source": "kind source: /, Kind="}
2021-02-08T14:14:45.800+0100    INFO    controller-runtime.manager.controller.jenkins   Starting EventSource    {"reconciler group": "jenkins.io", "reconciler kind": "Jenkins", "source": "kind source: /, Kind="}
2021-02-08T14:14:45.901+0100    INFO    controller-runtime.manager.controller.jenkins   Starting EventSource    {"reconciler group": "jenkins.io", "reconciler kind": "Jenkins", "source": "kind source: /, Kind="}
2021-02-08T14:14:46.003+0100    INFO    controller-runtime.manager.controller.jenkins   Starting EventSource    {"reconciler group": "jenkins.io", "reconciler kind": "Jenkins", "source": "kind source: core/v1, Kind=Secret"}
2021-02-08T14:14:46.004+0100    INFO    controller-runtime.manager.controller.jenkins   Starting EventSource    {"reconciler group": "jenkins.io", "reconciler kind": "Jenkins", "source": "kind source: core/v1, Kind=ConfigMap"}
2021-02-08T14:14:46.004+0100    INFO    controller-runtime.manager.controller.jenkins   Starting EventSource    {"reconciler group": "jenkins.io", "reconciler kind": "Jenkins", "source": "kind source: jenkins.io/v1alpha2, Kind=Jenkins"}
2021-02-08T14:14:46.004+0100    INFO    controller-runtime.manager.controller.jenkins   Starting Controller     {"reconciler group": "jenkins.io", "reconciler kind": "Jenkins"}
2021-02-08T14:14:46.004+0100    INFO    controller-runtime.manager.controller.jenkins   Starting workers        {"reconciler group": "jenkins.io", "reconciler kind": "Jenkins", "worker count": 1}

Lastly apply Jenkins Custom Resource to minikube cluster:

kubectl apply -f config/samples/jenkins.io_v1alpha2_jenkins.yaml

{"level":"info","ts":1612790690.875426,"logger":"controller-jenkins","msg":"Setting default Jenkins container command","cr":"jenkins-example"}
{"level":"info","ts":1612790690.8754492,"logger":"controller-jenkins","msg":"Setting default Jenkins container JAVA_OPTS environment variable","cr":"jenkins-example"}
{"level":"info","ts":1612790690.875456,"logger":"controller-jenkins","msg":"Setting default operator plugins","cr":"jenkins-example"}
{"level":"info","ts":1612790690.875463,"logger":"controller-jenkins","msg":"Setting default Jenkins master service","cr":"jenkins-example"}
{"level":"info","ts":1612790690.875467,"logger":"controller-jenkins","msg":"Setting default Jenkins slave service","cr":"jenkins-example"}
{"level":"info","ts":1612790690.881811,"logger":"controller-jenkins","msg":"*v1alpha2.Jenkins/jenkins-example has been updated","cr":"jenkins-example"}
{"level":"info","ts":1612790691.252834,"logger":"controller-jenkins","msg":"Creating a new Jenkins Master Pod default/jenkins-jenkins-example","cr":"jenkins-example"}
{"level":"info","ts":1612790691.322793,"logger":"controller-jenkins","msg":"Jenkins master pod restarted by operator:","cr":"jenkins-example"}
{"level":"info","ts":1612790691.322817,"logger":"controller-jenkins","msg":"Jenkins Operator version has changed, actual '' new 'v0.5.0'","cr":"jenkins-example"}
{"level":"info","ts":1612790691.3228202,"logger":"controller-jenkins","msg":"Jenkins CR has been replaced","cr":"jenkins-example"}
{"level":"info","ts":1612790695.8789551,"logger":"controller-jenkins","msg":"Creating a new Jenkins Master Pod default/jenkins-jenkins-example","cr":"jenkins-example"}
{"level":"warn","ts":1612790817.9423082,"logger":"controller-jenkins","msg":"Reconcile loop failed: couldn't init Jenkins API client: Get \"http://192.168.99.254:31998/api/json\": dial tcp 192.168.99.254:31998: connect: connection refused","cr":"jenkins-example"}
{"level":"warn","ts":1612790817.9998221,"logger":"controller-jenkins","msg":"Reconcile loop failed: couldn't init Jenkins API client: Get \"http://192.168.99.254:31998/api/json\": dial tcp 192.168.99.254:31998: connect: connection refused","cr":"jenkins-example"}
{"level":"info","ts":1612790818.581316,"logger":"controller-jenkins","msg":"base-groovy ConfigMap 'jenkins-operator-base-configuration-jenkins-example' name '1-basic-settings.groovy' running groovy script","cr":"jenkins-example"}
...
{"level":"info","ts":1612790820.9473379,"logger":"controller-jenkins","msg":"base-groovy ConfigMap 'jenkins-operator-base-configuration-jenkins-example' name '8-disable-job-dsl-script-approval.groovy' running groovy script","cr":"jenkins-example"}
{"level":"info","ts":1612790821.244055,"logger":"controller-jenkins","msg":"Base configuration phase is complete, took 2m6s","cr":"jenkins-example"}
{"level":"info","ts":1612790821.7953842,"logger":"controller-jenkins","msg":"Waiting for Seed Job Agent `seed-job-agent`...","cr":"jenkins-example"}
...

{"level":"info","ts":1612790851.843638,"logger":"controller-jenkins","msg":"Waiting for Seed Job Agent `seed-job-agent`...","cr":"jenkins-example"}
{"level":"info","ts":1612790853.489524,"logger":"controller-jenkins","msg":"User configuration phase is complete, took 2m38s","cr":"jenkins-example"}

Two log lines says that Jenkins Operator works correctly:
 
* `Base configuration phase is complete` - ensures manifests, Jenkins pod, Jenkins configuration and Jenkins API token  
* `User configuration phase is complete` - ensures Jenkins restore, backup and seed jobs along with user configuration 

> Details about base and user phase can be found [here](https://jenkinsci.github.io/kubernetes-operator/docs/how-it-works/architecture-and-design/).
kubectl get jenkins -o yaml

apiVersion: v1
items:
- apiVersion: jenkins.io/v1alpha2
  kind: Jenkins
  metadata:
  ...
  spec:
    backup:
      action: {}
      containerName: ""
      interval: 0
      makeBackupBeforePodDeletion: false
    configurationAsCode:
      configurations: []
      secret:
        name: ""
    groovyScripts:
      configurations: []
      secret:
        name: ""
    jenkinsAPISettings:
      authorizationStrategy: createUser
    master:
      basePlugins:
      ...
      containers:
      - command:
        - bash
        - -c
        - /var/jenkins/scripts/init.sh && exec /usr/bin/tini -s -- /usr/local/bin/jenkins.sh
        env:
        - name: JAVA_OPTS
          value: -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap
            -XX:MaxRAMFraction=1 -Djenkins.install.runSetupWizard=false -Djava.awt.headless=true
        image: jenkins/jenkins:2.263.3-lts-alpine
        imagePullPolicy: Always
        livenessProbe:
        ...
        readinessProbe:
        ...
        resources:
          limits:
            cpu: 1500m
            memory: 3Gi
          requests:
            cpu: "1"
            memory: 500Mi
      disableCSRFProtection: false
    restore:
      action: {}
      containerName: ""
      getLatestAction: {}
    seedJobs:
    - additionalClasspath: ""
      bitbucketPushTrigger: false
      buildPeriodically: ""
      description: Jenkins Operator repository
      failOnMissingPlugin: false
      githubPushTrigger: false
      id: jenkins-operator
      ignoreMissingFiles: false
      pollSCM: ""
      repositoryBranch: master
      repositoryUrl: https://github.com/jenkinsci/kubernetes-operator.git
      targets: cicd/jobs/*.jenkins
      unstableOnDeprecation: false
    service:
      port: 8080
      type: NodePort
    serviceAccount: {}
    slaveService:
      port: 50000
      type: ClusterIP
  status:
    appliedGroovyScripts:
    - configurationType: base-groovy
      hash: 2ownqpRyBjQYmzTRttUx7axok3CKe2E45frI5iRwH0w=
      name: 1-basic-settings.groovy
      source: jenkins-operator-base-configuration-jenkins-example
    ...
    baseConfigurationCompletedTime: "2021-02-08T13:27:01Z"
    createdSeedJobs:
    - jenkins-operator
    operatorVersion: v0.5.0
    provisionStartTime: "2021-02-08T13:24:55Z"
    userAndPasswordHash: nnfZsWmFfAYlYyVYeKhWW2KB4L8mE61JUfetAsr9IMM=
    userConfigurationCompletedTime: "2021-02-08T13:27:33Z"
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""
kubectl get po

NAME                                              READY   STATUS              RESTARTS   AGE
jenkins-jenkins-example                           1/1     Running             0          23m
seed-job-agent-jenkins-example-758cc7cc5c-82hbl   1/1     Running             0          21m

Build and run with Docker Desktop

Install Docker Desktop. If you are using Docker Desktop for Windows, you will also need to install WSL or WSL2. Ensure that Docker Desktop is currently running, and that you have enabled Kubernetes in it.

Run Jenkins Operator locally.

make config="config.docker-desktop.env" run

From this point on, Docker Desktop usage is identical to minikube usage.

Debug Jenkins Operator

make run OPERATOR_EXTRA_ARGS="--debug"

Stop or delete minikube cluster

To stop Kubernetes cluster running locally on minikube:

minikube stop

To delete the cluster altogether:

minikube delete

Build and run with a remote Kubernetes cluster

You can also run the controller locally and make it listen to a remote Kubernetes server.

make run NAMESPACE=default KUBECTL_CONTEXT=remote-k8s EXTRA_ARGS='--kubeconfig ~/.kube/config'

Once Jenkins Operator are up and running, apply Jenkins custom resource:

kubectl --context remote-k8s --namespace default apply -f deploy/crds/jenkins_v1alpha2_jenkins_cr.yaml
kubectl --context remote-k8s --namespace default get jenkins -o yaml
kubectl --context remote-k8s --namespace default get po

Testing

Tests are written using Ginkgo with Gomega.

Run unit tests with go fmt, lint, staticcheck, vet:

make verify

Run unit tests only:

make test

Running E2E tests

Run e2e tests with minikube:

make minikube-start
make e2e

Run Helm e2e tests:

eval $(bin/minikube docker-env)
make helm-e2e

Run the specific e2e test:

make e2e E2E_TEST_SELECTOR='^TestConfiguration$'

Building docker image on minikube

To be able to work with the docker daemon on minikube machine run the following command before building an image:

eval $(bin/minikube docker-env)

When api/v1alpha2/jenkins_types.go has changed

Run:

make manifests

Getting the Jenkins URL and basic credentials

minikube service jenkins-operator-http-<cr_name> --url
kubectl get secret jenkins-operator-credentials-<cr_name> -o 'jsonpath={.data.user}' | base64 -d
kubectl get secret jenkins-operator-credentials-<cr_name> -o 'jsonpath={.data.password}' | base64 -d

Webhook

To deploy the operator along with webhook, run :

eval $(minikube docker-env)
make deploy-webhook

It uses cert-manager as an external dependency.

Self-learning

Last modified January 16, 2023