StorageCluster
The Portworx cluster configuration is specified by a Kubernetes CRD (CustomResourceDefinition) called StorageCluster. The StorageCluster object acts as the definition of the Portworx Cluster.
The StorageCluster
object provides a Kubernetes native experience. You can manage your Portworx cluster just like any other application running on Kubernetes. That is, if you create or edit the StorageCluster
object, the operator will create or edit the Portworx cluster in the background.
To generate a StorageCluster
spec customized for your environment, point your browser to PX-Central, and click “Install and Run” to start the Portworx spec generator. Then, the wizard will walk you through all the necessary steps to create a StorageCluster
spec customized for your environment.
Note that using the Portworx spec generator is the recommended way of generating a StorageCluster
spec. However, if you want to generate the StorageCluster
spec manually, you can refer to the StorageCluster Examples and StorageCluster Schema sections.
StorageCluster Examples
This section provides a few examples of common Portworx configurations you can use for manually configuring your Portworx cluster. Update the default values in these files to match your environment.
- Portworx with internal KVdb, configured to use all unused devices on the system.
apiVersion: core.libopenstorage.org/v1
kind: StorageCluster
metadata:
name: portworx
namespace: kube-system
spec:
image: portworx/oci-monitor:2.6
kvdb:
internal: true
storage:
useAll: true
- Portworx with external ETCD, Stork, and Lighthouse.
apiVersion: core.libopenstorage.org/v1
kind: StorageCluster
metadata:
name: portworx
namespace: kube-system
spec:
image: portworx/oci-monitor:2.6
kvdb:
endpoints:
- etcd:http://etcd-1.net:2379
- etcd:http://etcd-2.net:2379
- etcd:http://etcd-3.net:2379
authSecret: px-etcd-auth
stork:
enabled: true
args:
health-monitor-interval: "100"
userInterface:
enabled: true
- Portworx with Security enabled.
apiVersion: core.libopenstorage.org/v1
kind: StorageCluster
metadata:
name: portworx
namespace: kube-system
spec:
image: portworx/oci-monitor:2.6
security:
enabled: true
- Portworx with Security enabled, guest access disabled, a custom self signed issuer/secret location, and five day token lifetime.
apiVersion: core.libopenstorage.org/v1
kind: StorageCluster
metadata:
name: portworx
namespace: kube-system
spec:
image: portworx/oci-monitor:2.6
security:
enabled: true
auth:
guestAccess: 'Disabled'
selfSigned:
issuer: 'openstorage.io'
sharedSecret: 'my-k8s-secret'
tokenLifetime: '5d'
- Portworx with update and delete strategies, and placement rules.
apiVersion: core.libopenstorage.org/v1
kind: StorageCluster
metadata:
name: portworx
namespace: kube-system
spec:
image: portworx/oci-monitor:2.6
updateStrategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 20%
deleteStrategy:
type: UninstallAndWipe
placement:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: px/enabled
operator: NotIn
values:
- "false"
- key: node-role.kubernetes.io/master
operator: DoesNotExist
tolerations:
- key: infra/node
operator: Equal
value: "true"
effect: NoExecute
- Portworx with custom image registry, network interfaces, and miscellaneous options
apiVersion: core.libopenstorage.org/v1
kind: StorageCluster
metadata:
name: portworx
namespace: kube-system
spec:
image: portworx/oci-monitor:2.6
imagePullPolicy: Always
imagePullSecret: regsecret
customImageRegistry: docker.private.io/repo
network:
dataInterface: eth1
mgmtInterface: eth1
secretsProvider: vault
runtimeOptions:
num_io_threads: "10"
env:
- name: VAULT_ADDRESS
value: "http://10.0.0.1:8200"
- Portworx with node specific overrides. Use different devices or no devices on different set of nodes.
apiVersion: core.libopenstorage.org/v1
kind: StorageCluster
metadata:
name: portworx
namespace: kube-system
spec:
image: portworx/oci-monitor:2.6
storage:
devices:
- /dev/sda
- /dev/sdb
nodes:
- selector:
labelSelector:
matchLabels:
px/storage: "nvme"
storage:
devices:
- /dev/nvme1
- /dev/nvme2
- selector:
labelSelector:
matchLabels:
px/storage: "false"
storage:
devices: []
StorageCluster Schema
This section explains the fields used to configure the StorageCluster
object.
Field | Description | Type | Default |
---|---|---|---|
spec. image |
Specifies the Portworx monitor image. | string |
None |
spec. imagePullPolicy |
Specifies the image pull policy for all the images deployed by the operator. It can take one of the following values: Always or IfNotPresent |
string |
Always |
spec. imagePullSecret |
If Portworx pulls images from a secure repository, you can use this field to pass it the name of the secret. Note that the secret should be in the same namespace as the StorageCluster object. |
string |
None |
spec. customImageRegistry |
The custom container registry server Portworx uses to fetch the Docker images. You may include the repository as well. | string |
None |
spec. secretsProvider |
The name of the secrets provider Portworx uses to store your credentials. To use features like cloud snapshots or volume encryption, you must configure a secret store provider. Refer to the Secret store management page for more details. | string |
None |
spec. runtimeOptions |
A collection of key-value pairs that overwrites the runtime options. | map[string]string |
None |
spec. security |
An object for specifying Portworx Security configurations. Refer to the Operator Security page for more details. | object |
None |
spec. featureGates |
A collection of key-value pairs specifying which Portworx features should be enabled or disabled. 1 | map[string]string |
None |
spec. env[] |
A list of Kubernetes like environment variables. Similar to how environment variables are provided in Kubernetes, you can directly provide values to Portworx or import them from a source like a Secret , ConfigMap , etc. |
[]object |
None |
KVdb configuration
This section explains the fields used to configure Portworx with a KVdb. Note that, if you don’t specify the endpoints, the operator starts Portworx with the internal KVdb.
Field | Description | Type | Default |
---|---|---|---|
spec. kvdb. internal |
Specifies if Portworx starts with the internal KVdb. | boolean |
true |
spec. kvdb.endpoints[] |
A list of endpoints for your external key-value database like ETCD or Consul. This field takes precedence over the spec.kvdb.internal field. That is, if you specify the endpoints, Portworx ignores the spec.kvdb.internal field and it uses the external KVdb. |
[]string |
None |
spec. kvdb. authSecret |
Indicates the name of the secret Portworx uses to authenticate against your KVdb. The secret must be placed in the same namespace as the StorageCluster object. The secret should provide the following information: - The username and password stored under the username and password keys, if you’re using a username/password authentication schema. - The CA certificate stored under the kvdb-ca.crt key and the certificate key stored under the kvdb.key , if you’re using certificates for authentication. - The ACL token stored under the acl-token key, if you’re using ACL tokens for authentication. |
string |
None |
Storage configuration
This section provides details about the fields used to configure the storage for your Portworx cluster. If you don’t specify a device, the operator sets the spec.storage.useAll
field to true
.
Field | Description | Type | Default |
---|---|---|---|
spec. storage. useAll |
If set to true , Portworx uses all available, unformatted, and unpartitioned devices. 2 |
boolean |
true |
spec. storage. useAllWithPartitions |
If set to true , Portworx uses all the available and unformatted devices. 2 |
boolean |
false |
spec. storage. forceUseDisks |
If set to true , Portworx uses a device even if there’s a file system on it. Note that Portworx may wipe the drive before using it. |
boolean |
false |
spec. storage. devices[] |
Specifies the list of devices Portworx should use. | []string |
None |
spec. storage. journalDevice |
Specifies the device Portworx uses for journaling. | string |
None |
spec. storage. systemMetadataDevice |
Indicates the device Portworx uses to store metadata. For better performance, specify a system metadata device when using Portworx with the internal KVdb. | string |
None |
spec. storage. kvdbDevice |
Specifies the device Portworx uses to store internal KVDB data. | string |
None |
Cloud storage configuration
This section explains the fields used to configure Portworx with cloud storage. Once the cloud storage is configured, Portworx manages the cloud disks automatically based on the provided specs. Note that the spec.storage
fields take precedence over the fields presented in this section. Make sure the spec.storage
fields are empty when configuring Portworx with cloud storage.
Field | Description | Type | Default |
---|---|---|---|
spec. cloudStorage. deviceSpecs[] |
A list of the specs for your cloud storage devices. Portworx creates a cloud disk for every device. | []string |
None |
spec. cloudStorage. journalDeviceSpec |
Specifies the cloud device Portworx uses for journaling. | string |
None |
spec. cloudStorage. systemMetadataDeviceSpec |
Indicates the cloud device Portworx uses for metadata. For performance, specify a system metadata device when using Portworx with the internal KVdb. | string |
None |
spec. cloudStorage. kvdbDeviceSpec |
Specifies the cloud device Portworx uses for an internal KVDB. | string |
None |
spec. cloudStorage. maxStorageNodesPerZone |
Indicates the maximum number of storage nodes per zone. If this number is reached, and a new node is added to the zone, Portworx doesn’t provision drives for the new node. Instead, Portworx starts the node as a compute-only node. | uint32 |
None |
spec. cloudStorage. maxStorageNodes |
Specifies the maximum number of storage nodes. If this number is reached, and a new node is added, Portworx doesn’t provision drives for the new node. Instead, Portworx starts the node as a compute-only node. As a best practice, it is recommended to use the maxStorageNodesPerZone field. |
uint32 |
None |
Network configuration
This section describes the fields used to configure the network settings. If these fields are not specified, Portworx auto-detects the network interfaces.
Field | Description | Type | Default |
---|---|---|---|
spec. network. dataInterface |
Specifies the network interface Portworx uses for data traffic. | string |
None |
spec. network. mgmtInterface |
Indicates the network interface Portworx uses for control plane traffic. | string |
None |
Volume configuration
This section describes the fields used to configure custom volume mounts for Portworx pods.
Field | Description | Type | Default |
---|---|---|---|
spec. volumes[]. name |
Unique name for the volume. | string |
None |
spec. volumes[]. mountPath |
Path within the Portworx container at which the volume should be mounted. Must not contain the ‘:’ character. | string |
None |
spec. volumes[]. mountPropagation |
Determines how mounts are propagated from the host to container and the other way around. | string |
None |
spec. volumes[]. readOnly |
Volume is mounted read-only if true, read-write otherwise. | boolean |
false |
spec. volumes[]. [secret|configMap|hostPath] |
Specifies the location and type of the mounted volume. This is similar to the VolumeSource schema of a Kubernetes pod volume. | object |
None |
Placement rules
You can use the placement rules to specify where Portworx should be deployed. By default, the operator deploys Portworx on all worker nodes.
Field | Description | Type | Default |
---|---|---|---|
spec. placement. nodeAffinity |
Use this field to restrict Portwox on certain nodes. It works similarly to the Kubernetes node affinity feature. | object |
None |
spec. placement. tolerations[] |
Specifies a list of tolerations that will be applied to Portworx pods so that they can run on nodes with matching taints. | []object |
None |
Update strategy
This section provides details on how to specify an update strategy.
Field | Description | Type | Default |
---|---|---|---|
spec. updateStrategy. type |
Indicates the update strategy. Currently, Portworx supports the following update strategies- RollingUpdate and OnDelete . |
object |
RollingUpdate |
spec. updateStrategy. rollingUpdate. maxUnavailable |
Similarly to how Kubernetes rolling update strategies work, this field specifies how many nodes can be down at any given time. Note that you can specify this as a number or percentage. | int or string |
1 |
Delete/Uninstall strategy
This section provides details on how to specify an uninstall strategy for your Portworx cluster.
Field | Description | Type | Default |
---|---|---|---|
spec. deleteStrategy. type |
Indicates what happens when the Portworx StorageCluster object is deleted. By default, there is no delete strategy, which means only the Kubernetes components deployed by the operator are removed. The Portworx systemd service continues to run, and the Kubernetes applications using the Portworx volumes are not affected. Portworx supports the following delete strategies: - Uninstall - Removes all Portworx components from the system and leaves the devices and KVdb intact. - UninstallAndWipe - Removes all Portworx components from the system and wipes the devices and metadata from KVdb. |
string |
None |
Monitoring configuration
This section provides details on how to enable monitoring for Portworx.
Field | Description | Type | Default |
---|---|---|---|
spec. monitoring. prometheus. enabled |
Enables or disables a Prometheus cluster. | boolean |
false |
spec. monitoring. prometheus. exportMetrics |
Expose the Portworx metrics to an external or operator deployed Prometheus. | boolean |
false |
Stork configuration
This section describes the fields used to manage the Stork deployment through the Portworx operator.
Field | Description | Type | Default |
---|---|---|---|
spec. stork. enabled |
Enables or disables Stork at any given time. | boolean |
true |
spec. stork. image |
Specifies the Stork image. | string |
None |
spec. stork. lockImage |
Enables locking Stork to the given image. When set to false, the Portworx Operator will overwrite the Stork image to a recommended image for given Portworx version. | boolean |
false |
spec. stork. args |
A collection of key-value pairs that overrides the default Stork arguments or adds new arguments. | map[string]string |
None |
spec. stork. env[] |
A list of Kubernetes like environment variables passed to Stork. | []object |
None |
spec. stork. volumes[] |
A list of volumes passed to Stork pods. The schema is similar to the top-level volumes. | []object |
None |
Lighthouse configuration
This section provides details on how to deploy and manage Lighthouse.
Field | Description | Type | Default |
---|---|---|---|
spec. userInterface. enabled |
Enables or disables Lighthouse at any given time. | boolean |
false |
spec. userInterface. image |
Specifies the Lighthouse image. | string |
None |
spec. userInterface. lockImage |
Enables locking Lighthouse to the given image. When set to false, the Portworx Operator will overwrite the Lighthouse image to a recommended image for given Portworx version. | boolean |
false |
Autopilot configuration
This section provides details on how to deploy and manage Autopilot.
Field | Description | Type | Default |
---|---|---|---|
spec. autopilot. enabled |
Enables or disables Autopilot at any given time. | boolean |
false |
spec. autopilot. image |
Specifies the Autopilot image. | string |
None |
spec. autopilot. lockImage |
Enables locking Autopilot to the given image. When set to false, the Portworx Operator will overwrite the Autopilot image to a recommended image for given Portworx version. | boolean |
false |
spec. autopilot. providers[] |
List of data providers for Autopilot. | []object |
None |
spec. autopilot. providers[]. name |
Unique name of the data provider. | string |
None |
spec. autopilot. providers[]. type |
Type of the data provider. For instance, prometheus . |
string |
None |
spec. autopilot. providers[]. params |
Map of key-value params for the provider. | map[string]string |
None |
spec. autopilot. args |
A collection of key-value pairs that overrides the default Autopilot arguments or adds new arguments. | map[string]string |
None |
spec. autopilot. env[] |
A list of Kubernetes like environment variables passed to Autopilot. | []object |
None |
spec. autopilot. volumes[] |
A list of volumes passed to Autopilot pods. The schema is similar to the top-level volumes. | []object |
None |
Node specific configuration
This section provides details on how to override certain cluster level configuration for individual or group of nodes.
Field | Description | Type | Default |
---|---|---|---|
spec. nodes[] |
A list of node specific configurations. | []object |
None |
spec. nodes[]. selector |
Selector for the node(s) to which the configuration in this section will be applied. | object |
None |
spec. nodes[]. selector. .nodeName |
Name of the node to which this configuration will be applied. Node name takes precendence over selector.labelSelector . |
string |
None |
spec. nodes[]. selector. .labelSelector |
Kubernetes style label selector for nodes to which this configuration will be applied. | object |
None |
spec. nodes[]. network |
Specify network configuration for the selected nodes, similar to the one specified at cluster level. If this network configuration is empty, then cluster level values are used. | object |
None |
spec. nodes[]. storage |
Specify storage configuration for the selected nodes, similar to the one specified at cluster level. If some of the config is left empty, the cluster level storage values are passed to the nodes. If you don’t want to use a cluster level value and set the field to empty, then explicitly set an empty value for it so no value is passed to the nodes. For instance, set spec.nodes[0].storage.kvdbDevice: "" , to not use kvdb device for the selected nodes. |
object |
None |
spec. nodes[]. env |
Specify extra environment variables for the selected nodes. Cluster level environment variables are combined with these and sent to the selected nodes. If same variable is present at cluster level, then the node level variable takes precedence. | object |
None |
spec. nodes[]. runtimeOptions |
Specify runtime options for the selected nodes. If specified, cluster level options are ignored and only these runtime options are passed to the nodes. | object |
None |
Security Configuration
Field | Description | Type | Default |
---|---|---|---|
spec. security. enabled |
Enables or disables Security at any given time. | boolean |
false |
spec. security. auth. guestAccess |
Determines how the guest role will be updated in your cluster. The options are Enabled, Disabled, or Managed. Managed will cause the operator to ignore updating the system.guest role. Enabled and Disabled will allow or disable guest role access, respectively. | string |
Enabled |
spec. security. auth. selfSigned. tokenLifetime |
The length operator-generated tokens will be alive until being refreshed. | string |
24h |
spec. security. auth. selfSigned. issuer |
The issuer name to be used when configuring Portworx Security. This field maps to the PORTWORX_AUTH_JWT_ISSUER environment variable in the Portworx Daemonset. |
string |
operator.portworx.io |
spec. security. auth. selfSigned. sharedSecret |
The Kubernetes secret name for retrieving and storing your shared secret. This field can be used to add a pre-existing shared secret or for customizing which secret name the operator will use for its auto-generated shared secret key. This field maps to the PORTWORX_AUTH_JWT_SHAREDSECRET environment variable in the Portworx Daemonset. |
string |
px-shared-secret |