notes

Containers & Kubernetes

Containers & their orchestration using Kubernetes offers a relatively portable, flexible approach for both production use cases, and for encapsulating demos and tutorials.

So much is possible that we should discuss what we have, what we could have, and what (if any!) value ideas bring.

Containers

The following are currently provided on quay.io & docker.io

• Egeria
• Jupyter - had hoped to remove, but some issues with permissions in loading from github has left us with an image
• configure - should replace with generic image
• UIs - should be to UBI8 for security
• Postgres, SAS connectors - include connector with Egeria - not needed?

Kubernetes

The most tested environment is RedHat OpenShift (IBM cloud), with Rancher Desktop being favoured for local linux/mac/windows.

Helm Charts

Recent additions include

• Ensuring connector jars can be downloaded
• improved security (containers, passwords etc
• Separating out the jupyter notebooks into their own github repo

lab

• recently changed to jupyter charts downloaded on demand. Master only - sufficient?
• odpi/egeria-charts#191 Jupyter self-cert see here
• odpi/egeria-charts#195 New UI v4
• odpi/egeria-charts#179 continual resetting of auto gen password in Jupyter
• odpi/egeria-charts#126 ssh between containers (ie copying files, demos)
• What needs adding?

base

• odpi/egeria-charts#195 Business UI?
• odpi/egeria-charts#182 content packs
• odpi/egeria-charts#133 Atruvia/IBM suggests to use in real scenario
• fixed configuration
• Some improvements have been submitted to add flexibility

cts

• odpi/egeria#197 issue getting cts results

pts

• not regularly run

Note on XTDB

• odpi/egeria-charts#125 XTDB in base/lab charts - work with chris important

Additional helm charts

• Scenario to include postgres or database connector?
• Open lineage - this looks valuable to drive more discussion
• Apache Atlas (see here - a stakeholder wants this

All charts

• odpi/egeria-charts#5 certificate/CA configuration
• odpi/egeria-charts#166 Open kafka configuration (ie for event streams)
• odpi/egeria-charts#166 Open kafka configuration (ie for event streams)
• odpi/egeria-charts#104 health checks .. some other minor issues see https://github.com/odpi/egeria-charts/issues?page=1&q=is%3Aissue+is%3Aopen
• odpi/egeria-docs#392 - docs on persistence in charts

Operator

• Go based
• can be built and deployed via CLI
• not in catalog
• Deploys config as-is to server (so endpoints must be correct)
• At point of testing replication/with Crux - helps to validate other production-oriented scenarios
• no specific consumer identified & needs review
• egeria-k8s-operator#12 - publish to marketplace
• egeria-k8s-operator#18 - automated testing

Issues

These issues are broader than k8s and apply to other deployment environments, but k8s work tends to draw attention to them, plus provide a useful approach to reproducing, or sharing good/bad patterns

Scalability/Management

• 6813 - kafka - unavailability?
• 5471 checking connector status
• 6809 review jvm mem
• 6804 smaller footprint containers
• 6732 image support for more architectures (s390?)

• 6728 shutdown hook

• 5918 container environment for ui server chassis
• 5912 UBI8 image for configure container
• 1070 mixed cohort testing
• 229 out of reach repos
• egeria-charts#17 - prometheus
• egeria-charts#180 - java heap dump

Security

• 6005 omvs log incomplete
• 5956 configure image error checking
• 5955 docker tag for latest release
• 5944 container metadata

Development, distribution & tidying up

• 5514 performance - use slf4jh message formatter
• 6702 egeria parms passed to kafka
• 1734 PR build of docker container for testing
• 6340 cross egeria container image <- proposal for uber image

Questions

• What else is needed for production
• What is needed for demos
• What is needed for tutorials