Security is all around us, just the other day, I blogged about a new feature of CloudIQ, called “CloudIQ CyberSecurity” , now, let’s talk about encryption of the actual data..

Keeping data safe in a traditional data center is one of the biggest challenge’s security administrators are facing these days. Ensuring data safety requires security controls, and system checks built layer by layer into the structure of a data center. Customers are turning to data encryption to help secure their deployments from on-premise and throughout the cloud.

Modern enterprises are heading toward embracing hybrid cloud models for deployment flexibility, infrastructure scalability, and cost-effective use of IT resources. However, because cloud computing is based on a shared, multitenant compute, network, and storage architecture, traditional security controls are not sufficient. Data owners have many reasons for encrypting their data, from securing sensitive data that reside on the cloud, addressing regulatory compliance to protecting against theft of customer data and sensitive intellectual property.

Encryption enciphers the original data into an unrecognizable format. This format of the data is different from the original data. Encryption is done using a key algorithm. These encryption and decryption keys must be stored in a secure location. Unauthorized access to the keys or losing them may lead to data inaccessibility and data security issues.

There are two methods of encryption- symmetric and asymmetric encryption.

• Symmetric encryption, pertains to the sender and the recipient holding the same keys to encrypt and decrypt a message.
• Asymmetric encryption, uses a key pair—a public key for encrypting a message, and a private key to decrypt it.

  Challenges in Data Encryption
  

  Datacenters require that data must be secured both at the hardware and software layers that is on-premise and on the cloud.
  

  Cloud computing is based on a shared, multi-tenant, and software-defined compute, network, and storage architecture. Data owners are responsible for securing sensitive data across public and private clouds, but traditional security controls no longer apply. New solutions must address privacy, regulatory, and data remanence (residual data) requirements. They must also provide the flexibility to support various encryption approaches for diverse use cases.
  

  Storage infrastructure-level encryption provides a convenient way to secure data in the private data center that is transparent to the applications deployed on the physical and virtual infrastructures that consume the storage.
  

  Virtual machine-level encryption offers an infrastructure-agnostic approach that is portable across private and public clouds and keeps VMs secure during the migration process.
  

  Some of the challenges that might be encountered when encrypting data:
  

• Key Management – Encrypted volumes in virtual machines require externally stored keys. But they do not have direct access to physical hardware in cloud environments. Also, for every volume in a virtual machine that we deploy, another key must be secured and managed. Managing these keys can be difficult and time-consuming, and if we do not do it consistently, we risk losing data.
• VM movement – Moving a VM from one host to another in the encrypted state requires a flexible solution that keeps the data encrypted regardless of where and how it moves.
• Native OS Data Encryption – We can secure data using native operating system encryption capabilities (Microsoft Windows uses BitLocker for Windows and various Linux distributions use dm-crypt), but these solutions are difficult to configure, manage, and apply consistently throughout a deployment.

  CloudLink Solution
  

  
  

  CloudLink deployed across multiple sites
  

  CloudLink is a flexible data encryption and key management solution for data encryption in public, private, and hybrid cloud environments. CloudLink encrypts data with unique keys that enterprise security administrators control. Neither data center administrators nor other tenants in the cloud have access to the keys, offering protection against tampering and data spillage.
  

  CloudLink can secure sensitive information within machines on-premise, cloud, and multi-cloud environments. It provides encryption for volumes with pre-startup authorization by leveraging native operating system encryption features – Microsoft BitLocker for Windows or dm-crypt for Linux. Pre-startup authorization is provided via key release policies which help to validate that the encrypted machine has not been compromised. Policy-based key release is available for VM, bare metal, VxFlex platform encryption, and SED (Self-Encrypting Drive) key management.
  

  CloudLink Features
  

  Data Center Encryption
  

  Let us look at the encryption types supported by CloudLink at different data center layers. At the bottom hardware layer, encryption can be provided for SEDs (Self-Encrypting Drives) or Controller-based encryption. Moving up the stack, encryption can be done at the software-defined storage level, at virtual machines, or even in Kubernetes containers (next release v7.0).
  

  CloudLink can provide external key management over KMIP (Key Management Interoperability Protocol). KMIP is an open standard defined and developed by OASIS, and it governs the communication between the third-party encryptors and key managers. CloudLink supports KMIP versions 1.1 through 1.4.
  

  CloudLink Components

  CloudLink provides policy-based key management and supports multiple data encryption options for VxFlex OS devices, SEDs, virtual, and bare-metal machines.
  

  The image describes the CloudLink solution architecture. Click each component for more details.
  
  

  CloudLink Center and Agent
  

  
  

  CloudLink Center: A significant component of the CloudLink solution is the CloudLink Center. CloudLink Center is a policy-based key management server. It is packaged as a virtual appliance and can be deployed on VMware ESXi, Microsoft Hyper-V, Microsoft Azure, or Amazon Web Services infrastructure.
  

  CloudLink Center provides a web-based user interface for managing the CloudLink environment. The GUI uses the underlying REST-API and allows an administrator to manage agents, users, and encryption keys. An administrator can also configure roles, policies, and keystores, as well as monitor events, alarms, and logs.
  

  Agent: The CloudLink agent runs on individual machines. It communicates with CloudLink Center for pre-startup authorization and decryption of BitLocker or dm-crypt encryption keys.
  

  
  

  Encryption Keys
  

  
  

  CloudLink uses the following types of encryption keys to secure virtual machines:
  

• A device/volume encryption key that is used by native OS based encryption. A unique device encryption key is generated for each encrypted device.
• A device/volume key encryption key (VKEK) that is generated by CloudLink. CloudLink generates a VKEK for each device or encryption key.

  When the CloudLink agent initially sends a volume encryption key to the CloudLink Center, the CloudLink Center uses its own encryption key to encrypt the volume key. The VKEKs protect volume keys from being copied and used to decrypt volumes.
  

  CloudLink Center stores and protects keys in a location that is called a keystore.
  

  Keystores
  

  
  

  CloudLink Center stores and protects keys in a location called a keystore. The keystore is made up of two components, the Key Location which stores the encryption keys and a Key Protector which is the protection mechanism that is used to encrypt and protect the encryption keys.
  

• Local Keystore
  
• Remote Keystore
  

  During installation, CloudLink creates an internal key location – the CloudLink Vault to store volume encryption keys and VKEKs.
  

  The Vault is used as the internal key protector which encrypts or protects the key location.
  

  CloudLink can also use other external Key Protectors such as Microsoft Azure Key Vault, SafeNet Network HSM, and KMIP-compliant servers
  

  CloudLink supports the use of remote keystores, such as Microsoft Active Directory or Amazon S3. The keys are stored in a location external to CloudLink Vault, and secured with an external key protector.
  

  Key Location
  

  
  

  CloudLink Center supports several options for the key location used to store encryption keys.
  

  
  

  
  

  
  

  Key Protectors
  

  A key protector is the protection mechanism used to encrypt and protect the device encryption keys.
  

  
  

  
  

  
  

  
  

  KMIP Server
  

  
  

  A KMIP (Key Management Interop Protocol) server is used to store public and private keys for encrypted machines. CloudLink Center supports KMIP to allow applications supporting that protocol to securely store keys and certificates.
  

  The applications, or KMIP clients, are given access to a single KMIP partition. A KMIP partition is a logical container for keys that are created by the client. Multiple clients can be assigned to the same partition. All objects within a partition are encrypted using a key that is saved to the partition’s keystore and are stored in the CloudLink Center database.
  

  CloudLink is a certified VMware Ready KMS (Key Management Server), giving VMware customers complete flexibility for their data encryption needs. With VMware vCenter connected, CloudLink can perform as a KMIP server to support vSAN and vSphere encryption without deploying any agents.
  

  CloudLink Supported Encryption

  
  

  SED Encryption
  

  
  

  Self-encrypting drive (SED) is an intelligent hardware security solution that automatically encrypts the data that is written to it without any user interaction. SED offloads encryption from the host CPU, freeing cycles that software-based encryption would otherwise consume. The encryption process in SED is done by using a unique and random Data Encryption Key (DEK) which the drive uses to both encrypt and decrypt data. The Authentication Key in the SED manages access to the encrypted data.
  

  Click to watch video on What is SED.
  

  Securing data with SEDs requires a key management service that stores, manages, and serves the appropriate authentications to these drives. CloudLink provides support and management capabilities that allow users to safely secure their data-at-rest.
  

  CloudLink provides two key management options:
  

• Local Key Management: CloudLink agent can provide direct SED key management via the HBA controller. The CloudLink Agent provided by the CloudLink Center acts as the keystore manager. SED encryption keys are stored on the disk controller which ensures that data cannot be accessed if the disk is removed from the server for purposes like maintenance, decommissioning and repurposing.
• External Key Management: SED encryption keys are stored in an external key manager via KMIP. This protects against cases like theft of the server.

  PowerFlex Integration with CloudLink
  

  
  

  PowerFlex is a software-defined storage solution that uses the nodes’ local disks and LAN to create a virtual SAN. It is designed for a massively parallel I/O system with high resiliency and can scale to thousands of nodes. VxFlex OS pools together local storage devices from each node. Volumes are then created from the pooled storage and used as shared storage among all nodes. PowerFlex is inherently elastic and can be deployed in various architectures to support various use cases.
  

  Storage Data Server (SDS) manages the capacity of a single node and acts as a backend for data access. The SDS is installed on all nodes contributing storage devices to the VxFlex OS cluster. SDS could be running on Bare metal Linux (only supported on Linux) nodes or a Linux VM on supported hypervisors. CloudLink Agent is installed on SVM or Linux running on SO nodes.
  

  Similar to VM encryption, agents leverage dm-crypt, a Linux OS native encryption capability to encrypt the SDS devices. Encryption is enabled on raw devices before they can be added to the storage pool.
  

  Enterprise security administrators can control these keys with CloudLink Center. The CloudLink Center provides a centralized, policy-based key release, enabling single-screen security monitoring and management across one or more PowerFlex deployments, including storage-only, hyperconverged, and those on ESXi.
  

  CloudLink operates directly on SDS devices so that data at rest encryption is transparent to applications. There is no impact to VxFlex OS features because encryption is performed before data is written to SDS devices. This ensures that the enterprise-grade data protection and resiliency that is provided by PowerFlex are uninterrupted by the encryption process.
  

  PowerFlex Encryption Overview
  

  
  

  In a PowerFlex environment, storage is provided by multiple nodes that are running the Storage Data Server (SDS) process. Applications reside on nodes that are running the Storage Data Client (SDC) process. When an application requires data, the SDC contacts the SDS to retrieve the information. Nodes can run both the SDC and SDS processes and can, therefore, support applications and provide storage.
  

  CloudLink agents are installed on the SDS nodes only, and work with the operating system to encrypt local devices. The agents communicate with CloudLink Center using ports 80 and 1194. Although both CloudLink and PowerFlex support multiple operating systems, currently the solution only works with SDS nodes running a CloudLink agent on Linux.
  

  Encrypting SDS Device
  

  CloudLink encrypts raw devices so the encrypted device can be added to VxFlex OS SDS. A raw device does not contain partitions or a file system. If the device is already added to VxFlex OS, the device should be removed before you encrypt it because the encrypted device no longer contains the original data. An encrypted device’s data can be erased so it can be used for another purpose, but this process destroys all data.
  

  In CloudLink Center, you can encrypt a VxFlex OS SDS device when the machine is in the connected state. Encrypting a device is a quick operation because the device contains no data. Only one device can be encrypted at a time.
  

  In the image here, we have encrypted three devices through the CLI and this is reflected in the CloudLink Center.
  

  
  

  CloudLink and PowerFlex Manager
  

  
  

  CloudLink can be integrated into PowerFlex Manager as a discoverable resource in PowerFlex integrated rack and appliance.
  

  PowerFlex Manager can provide life-cycle management, monitoring, upgrades, and alerts for the CloudLink Center. These operational tasks are configured automatically in VxFlex Manager during the integration (discovery process) of the CloudLink Center.
  

  CloudLink Capabilities

  Users and Roles
  

  Each person who must work with CloudLink must be defined as a user in CloudLink Center. User accounts can be created locally in CloudLink Center or can be defined by integrating with a Microsoft Windows domain.
  

  Each user is assigned a role that determines their permissions and specifies which administrative functions or tasks are permitted in CloudLink Center. A user can be assigned multiple roles, giving them a combined set of permissions.
  

  CloudLink Center provides three built-in roles:
  

• SecAdmin – Full access to all objects and functionality and provided by default.
• Admin – Limited access to CloudLink Center functionality, primarily for managing user accounts and backups, and viewing event logs.
• Observer – View configuration options and event logs.

  
  

  Custom roles can also be created. The roles are assigned to users and when a user logs in to CloudLink Center, he or she receives the permissions that are assigned to the role.
  

  Machine and Machine Groups
  

  
  

  When a CloudLink agent registers with CloudLink Center, a machine object is created. Each machine must be assigned to a machine group, which is also an object defined in the CloudLink Center and used to combine machines into manageable units.
  

  Machine groups link various objects, like approved networks, policies, approved locations, and keystores to all the machines in the group. Machine groups are used to combine machines for administrative or operational purposes.
  

  For example, you might group machines by department, where each department may have a different set of requirements. Another example could be to group machines by function or location, where development has different requirements than production. Each machine group can have its own administrator as well.
  

  A machine is assigned to a machine group during deployment. If you do not specify a group during deployment, the machine is assigned to the built-in machine group named Default. You can change the machine group that a machine belongs to after deployment.
  

  Policies
  

  
  

  Policies are created on the CloudLink Center and are used to determine which volumes are encrypted and which agent is authorized to have a key.
  

  All machines in a group use the same:
  

• Key Release Policy
  
• Volume Encryption Policy
  
• Keystore
• Approved Location
  
• Approved Network
  

  Authentication
  

  
  

  To access CloudLink Center, a user must provide both a username and password. For increased security, CloudLink provides an option to enable two-factor authentication provided by RSA.
  

  CloudLink supports the creation of local user accounts with REST-API and also supports users and groups from Microsoft Windows Domain.
  

  An extra layer of security can be added using two-factor authentication, provided by Google Authenticator.
  

  Licensing
  

  Licenses come in different flavors for CloudLink:
  

• Capacity license defines how much total storage will be protected.
• Instance license defines the number of virtual or bare-metal machines that will be protected in a CloudLink environment.
• KMIP license defines how many KMIP clients can be managed.
• Socket-based license is available for VMware platforms and is assigned to ESXi hosts by the number of sockets.
• SED license enables the management of keys for self-encrypted drives. A single SED license is used per physical machine regardless of the number of SEDs connected to that machine.

  
  

  CloudLink Deployment and Management

  
  

  CloudLink agents can be deployed on various versions of Windows and Linux operating systems. Refer to the latest Deployment guide for detailed information. A network connection between the agents and the CloudLink Center is required for virtual machine startup, key requests, and continuous monitoring. Network port 80 is used for the agent installation and port 1194 is for normal agent communication. These ports must be opened for any firewalls that exist between the two endpoints. Port 443 must be open for users to access the web UI of the CloudLink Center.
  

  Clustered CloudLink Center
  

  
  

  A CloudLink Center cluster provides for high availability if one CloudLink Center server in the cluster becomes unavailable, whether due to planned maintenance or an unexpected issue. High availability is achieved by deploying the CloudLink Center appliance in an active/active cluster of 2-4 nodes. A CloudLink Center cluster consists of up to four CloudLink Centers. These additional nodes should be dispersed across multiple physical servers to increase availability. Configuration information is replicated across nodes. Replicated items include licenses, policies, user accounts, locally stored passcodes, and keys, actions, alarms, and events. Network ports 80 and 443 must be open between the clustered CloudLink Centers to enable all communication traffic.
  

  CloudLink Cluster Node Failure
  

  
  

  When a CloudLink cluster is established, agents can be connected to and controlled by any node in the cluster. Under normal operating conditions, an agent communicates with one node. If that node fails, the agents that were connected to that node can connect and communicate to any of the other nodes.
  

  Deployment in Multiple Clouds
  

  
  

  CloudLink Center can be deployed in a private cloud running various hypervisors and can secure virtual machines within that cloud. CloudLink can also secure virtual machines in other clouds, as long as a network connection exists between the CloudLink Center and agents and ports 80 and 1194 are open. CloudLink Agent is deployed to individual virtual machines hosted in the private cloud or to virtual machine instances in a supported public or hybrid cloud environment. For public cloud, CloudLink Center can be deployed in Microsoft Azure, or Amazon Web Services.
  

  Backup and Recovery in CloudLink
  

  System issues such as power interruptions or hardware failures may cause problems in CloudLink Center, such as data loss or database corruption. If problems occur, it is important to have a backup of CloudLink Center so that you can deploy a new server and restore CloudLink Center from the backup.
  

  A backup file includes all critical information to get CloudLink Center up and running, including keystore configuration, user accounts, machine registrations and policies, and events. You can create backup files manually or automatically. You can create a backup store where CloudLink Center stores automatic backups.
  

  
  

  A backup is stored in a file that is encrypted using a FIPS 140 encryption library. The private key from the RSA key pair must be secured by the administrator and will be required to perform CloudLink restore operation. Backups are stored locally by default but can be redirected to a remote site using FTP, SFTP, or FTPS protocols.
  

  You generate the RSA-2048 key pair. The public key is stored in CloudLink Center. Download and save the private key when the key pair is generated. To restore CloudLink Center from a backup file, both the backup
  

  CloudLink, can be downloaded by clicking the screenshot below
  

  and the documentation, can be accessed from this url https://www.dell.com/support/home/en-us/product-support/product/cloudlink-securevm/docs

The post Dell EMC CloudLink – Part1, Introduction. appeared first on Itzikr's Blog.

In the previous post, Dell EMC CloudLink – Part1, Introduction, we have covered the Dell EMC CloudLink overview and its key features.

One of the hottest topics right now around containers, is ‘Security’, you see, containers changed the game re architecture and as such, customers are looking for various options to protect, secure and encrypt their containers workloads. In fact, let’s take a look at one (out of many..) surveys done by a company called “StackRox”

Security tops the list of concerns with container strategies

“Inadequate investment in security leads the list of concerns users cite about their company’s container strategy (34%). When combined with not taking threats seriously (15%) and not accounting for compliance needs (17%), two-thirds of respondents identify security and compliance as their biggest source of concern.”

Now that we understand how CloudLink works, it’s only natural to go to the next step which is the integration with Kubernetes, Containers, and Cloud native storage.

CloudLink provides seamless security – from edge to core to cloud – with unmatched flexibility, superior reliability, and highly efficient operations. CloudLink provides multiple options for data encryption and key management across a broad spectrum of operating platforms including bare-metal, virtualized, and containerized workloads across public and private clouds, simplifying and streamlining security workflows.

Encryption at the Workload

While most of the storage solutions these days provides DARE (Data at Rest Encryption), encryption at rest though may not be sufficient as workloads move out through the cloud to the edge. To keep data safe in these environments, customers look to implement encryption at the workload as well. In these multi-level encryption environments customers can ensure that their data is encrypted right at the source, remains encrypted as it traverses to storage, and is securely encrypted at rest. Encrypting at the source also gives customers even more control over their data, allowing for cryptographic erasure once the data is no longer needed.

As the datacenter evolves, so does its needs for data encryption. Although Kubernetes a phenomenal platform for container orchestration and management, Kubernetes does not address one of the most critical components of an overall security strategy – data protection. Kubernetes does not provide robust mechanisms to encrypt, manage, and share secrets across a Kubernetes cluster. initially, you will probably leverage secrets management solutions like Vault, but you’ll quickly discover they do not provide a full solution in a container environment.

To keep up with this evolution, CloudLink not only offers Encryption for Machines (VMs and bare metal), we have now added Encryption for Containers to our toolbox. CloudLink supports encrypting Kubernetes container volumes by leveraging the CSI (Container Storage Interface) and we validated this encryption with PowerScale’s and PowerFlex’s own CSI implementation. Encryption for Containers is licensed per CloudLink Center cluster, allowing for the management of up to 25,000 keys for container volume encryption.

How does CloudLink’s encryption for containers work?

CloudLink 7.1 supports data encryption in a Kubernetes containerized environment. CloudLink encryption for containers enables you to encrypt shared volumes in a Kubernetes cluster. This functionality leverages Kubernetes 1.14 to 1.19 Container Storage Interface (CSI), which is customizable to the user environment, and features a quick, easy setup with the UI or REST-API. Encryption of Containers Agents sits between the Application and the CSI Storage Plugin encrypting the application data before it is sent to storage-thus providing both Data at Rest and Data in Motion. One CloudLink Center instance can support multiple Kubernetes clusters. Each Kubernetes cluster node can have multiple Container agents running on it, which includes one Encryption for Containers agent for each driver.

To encrypt data both in flight and at rest in a containerized environment, CloudLink deploys an agent on the container itself. The agent sits between the application and the CSI Storage Plugin. The CSI (Container Storage Interface) is a community standard that abstracts the underlying storage interface away from the application, allowing for multiple back end storage support options without having to change the application itself. Since our agent is deployed directly on the data path, we can ensure that the data is encrypted immediately when the application saves it.

Deploying and managing a container environment can be challenging. Kubernetes strives to keep it as simple as possible but there are a lot of moving parts. The addition of functionality like encryption just makes it even more complicated. To counter this, CloudLink has simplified the deployment process for the agent as much as possible. We have boiled it down to:

• A configuration file to customize per deployment
  
• 2-click initial setup for Kubernetes cluster preparation and authentication
  
• 1-click deployment for the CloudLink agent in each Kubernetes cluster
  

The above allows for dynamic provisioning of encryption for volumes in the environment for fast and easy container data encryption.

Support:

CloudLink 7.1 supports the following:

● Kubernetes version 1.14 to 1.19

● Tanzu Kubernetes version 1.1 or later

● Storage types that support Container Storage Interface (CSI):

○ Generic NFS storage

○ PowerScale (NFS) storage

○ PowerFlex block storage

● Volume types that support CSI:

○ File System provisioning for all storage types

○ Raw Block Volume provisioning for PowerFlex block storage

● FIPS validated dm-crypt crypto module for container block volume encryption

These are the high-level configuration steps for Encryption for Kubernetes using CloudLink 7.1:

1. Deploy CloudLink Center.

2. Create a Kubernetes or a Tanzu Kubernetes cluster as per your requirement.

3. Add a Kubernetes cluster entry to CloudLink Center.

4. The cluster_name_secret.yaml file is downloaded to the Downloads folder.

5. Upload the cluster_name_secret.yaml file as a secret to the Kubernetes cluster or the Tanzu Kubernetes cluster.

6. Build the node and controller Docker images using the Dockerfile in the Kubernetes node plug-in package

7. Push the node and controller Docker images to the Docker registry.

8. Push the NFS plug-in image to the Docker registry. Ensure that you have the NFS plug-in image handy as per your requirement.

9. Use Helm to deploy Encryption for containers in the Kubernetes or the Tanzu Kubernetes cluster.

10. Map the volumes to workloads.

11. Create workload container configuration files that reference the volume claims

How to Configure Encryption for Containers?

• Login to CloudLink Center and access the Kubernetes Clusters page.
  
• Add a new cluster by entering required information.
  
• Save the downloaded <cluster_name>_secret.yaml file.
  
• Download the CL k8s Node plugin and Docker file from CLCin order to create agent images.
  
• Apply the secret to Kubernetes cluster withplanned namespace.
  
• Create Encryption for container agent and push to docker server
  
• Deploy Encryption for containers agent (Dell EMC CloudLink CSI+ Dell EMC POWERSCALE CSI”) using helm
  

Helm install –namespace <name> -f defaultvaues.yaml demo http://<CLC-IP&gt;:/cloudlink/kubernetes/sdp/helm/cloudlink-sdp-helmx.x.x.tgz

• During the process, New encryption of containers agent pods gets created in Kubernetes within the provided namespace
  

•    After EOC agent gets installed, K8S nodes will be shown under Kubernetes Nodes

•    Deploy PV/PVC and the workload in order to mount volume on workload running on K8S nodes.

•    By navigating back to Cloud Link Center, you’ll see that the volumes count increased to 1

Additional Features:

Raw Block Volume

CloudLink 7.1 supports raw block volume encryption

• RBV feature allows persistent volumes to be exposed inside containers as a block device instead of as a mounted file system
  
• Block devices enable random access to data in fixed-size blocks.
  
• Block devices include whole disks, disk partitions, and LUNs from a storage area network (SAN) device.
  
• Raw block volumes require cryptsetup (v1 or v2) to be installed on every worker node.
  

Re-keying

CloudLink 7.1 supports shallow Rekeying (KEK) or Key rotation for existing encrypted volumes which means data is not re-encrypted, only KEK is rotated.

A per-volume action “Re-key volume” is added which will initialize re-keying for the selected volume.

• For a raw block volume re-keying, only the ReadWriteOnce mode is supported.
  
• For a file system volume re-keying, multiple agents may report the same volume. In this case, the operation should be initialized in any one of these agents. The agent which has connected first.
  

In conclusion, whether you need external key management with Data at Rest Encryption, or workload encryption for Kubernetes CSI deployments, PowerScale, PowerFlex, and CloudLink offer up a reliable, easy to use, flexible solutions that fits your data center.

Here you can see a demo of CloudLink’s encryption for Kubernetes workloads:

A guest post by Tomer Nahumi

The post Dell EMC CloudLink – Part2, Encrypting Kubernetes, Containers, and Cloud Native Storage Workloads appeared first on Itzikr's Blog.

The Dell EMC Flex family of products is powered by PowerFlex– a scale-out block storage service that enables customers to create a resilient server SAN or hyper-converged infrastructure on x86 server hardware. The Flex family accommodates a wide variety of deployment options, with multiple OS and hypervisor capabilities, and is ideal for applications requiring high performance and ease of management as one scales from small to large.

Back in June 2020, I blogged about the 3.5 release, which you can read all about here & here. That release, added a new, HTML5 based UI and native replication capabilities.

Today we are excited to announce PowerFlex version 3.6. this release helps you execute AI/ML and VDI workloads flawlessly with expanded GPU options, while offering superior automation for cloud-native applications with an updated CSI Driver and all-new Ansible Modules. This release also enhances security, compliance, and data protection with scalable replication and VMware SRM support, AppSync integrated copy-data management, and FIPS-140-2 compliance. You can operate your data center confidently with innovations including CloudIQ’s AI-based infrastructure insights, network resiliency enhancements, and new guided upgrade paths within PowerFlex Manager. Supercharge your data center with PowerFlex!

With this release, PowerFlex helps you execute workloads and operations flawlessly with expanded GPU options. Superior performance, automation, and integration for cloud-native applications is made possible by an updated CSI Driver and all-new Ansible Modules. Achieve superior outcomes for cloud-native and analytics workloads with PowerFlex!

PowerFlex’s newest release ensures effortless compliance with expanded security and data protection. Newly expanded replication options with SRA for VMware SRM combine to provide business continuity. Achieve compliance and regulatory requirements with ease using FIPS-140-2 and protect your applications with CDM (copy data management). Safeguard your critical assets with PowerFlex!

Operate your data center confidently, with new integrations including CloudIQ’s AI-based infrastructure insights, network resiliency enhancements, and new guided upgrade paths within PowerFlex Manager. Streamline your data center operations with PowerFlex!

Now, let’s review some of the great features we’ve added to this release:

CloudIQ support for PowerFlex

CloudIQ is Cloud-based proactive monitoring and predictive analytics for core, edge and cloud. CloudIQ combines monitoring, machine learning and predictive analytics so you can take quick action and simplify operations of your on-premises infrastructure and data protection in the cloud.

The integration between CloudIQ and PowerFlex includes system level monitoring of Configuration / Inventory, Capacity Usage, and Performance as we as health score based on PowerFlex alerts and anomaly detection and proactive monitoring for performance impacts.

Network Resiliency Improvements

In two-layer systems, where the SDC is external to the PowerFlex cluster, there can be a network path disconnection/failure between an SDC and an SDS that is not a general network failure in the cluster.

In version 3.6, if a path to one SDS fails, the SDC will retry I/Os (after a 10s cool-off period) using another SDS in the Protection Domain as a proxy for the requests.

Proactive Network Disqualification

A port or socket experiencing frequent disconnects (flapping), can cause system performance issues in PowerFlex’s fully distributed architecture.

PowerFlex 3.6 detects this and proactively disqualifies the path, preventing general disruption across the system.

Protected Maintenance Mode – Auto-abort

• PMM was introduced in version 3.5 to create a very safe alternative for putting a node into maintenance
  
  • Entering PMM: creates a new, temporary copy of the data by leveraging the many-to-many rebalance algorithm
    
    • We leave the data on the node being maintained in place
      
    • This makes for three copies, but only two are available
      
  • However, the process of entering PMM can be long, and the state of a system can change
    
  • PMM now checks every minute if the system still has the resources to support entering PMM.*
    
  • If it does not, the PMM is aborted, and the user is notified.
    

PowerFlex Replication Enhancements

The PowerFlex journal-based replication architecture allows for very low RPOs. In the initial release, we supported only as short as 30 seconds. In 3.6, we cut that in half to 15 seconds.

In 3.6, we’ve added support for replication in VMware HCI environments, this feature is available vailable only for systems managed by PowerFlex Manager (appliance and rack)

In 3.5.x a user has no way to maintain a configuration of RCG and Pairs without the actual replication happening and resources consumed. Also, in case the system is incapable to continue the replication of an RCG, it can only delete it completely from the system.

The ‘Activate-Deactivate RCG’ feature introduces a new INACTIVE state of RCG, when no replication actually happens, no resources are consumed and I/O to volumes is served directly by SDS. However, the RCG configuration, which includes, name, cycle, all the Pairs – is stored, agreed between the peers. The RCG can be created in INACTIVE state and can be requested by user to move to the normal, ACTIVE, state, and then back again. Also, in case when the system is incapable to continue the replication of the RCG, it will automatically move the RCG to the INACTIVE state.

PowerFlex SRA for VMware SRM

vCenter Site Recovery Manager is a disaster recovery and business continuity solution with the following characteristics:

• Operates as an extension of vCenter Server
  
• Automates the recovery or migration of virtual machines between a protected site and a recovery site
  
• Uses array-based replication solutions from other vendors, in our case, it’s PowerFlex
  

PowerFlex SRA enables the use of PowerFlex native asynchronous replication as the replication engine for protecting VMs on vSphere datastores
PowerFlex SRA for use with Site Recovery Manager 8.2* or 8.3* on the SRM Server appliance
PowerFlex replication must be configured and working
At least one Replication Consistency Group containing volumes that are:

• Mapped to ESXi hosts
  
• Used to create the datastores hosting the VMs that SRM will protect
  

New Scalability Limits

• Max SDCs per system = 2048
  
  • Increase from 1024
    
• Max SDR count = 128
  
  • Increase from 64
    
• Max volumes per SDC = 1024
  
  • Decrease from 8192
    
• Maximum replicated volume size = 64TB
  
  • Increase from 32TB
    
• PowerFlex Manager supports 320 drives per Storage Pool
  
  • Increase from 300*
    

PowerFlex Container Storage Interface (CSI) Plugin enhancements

PowerFlex CSI v1.5 update (Coming June 28th)

• Consistency groups support
  
• Custom FS format options
  
• User can set specific disk format command parameters when provisioning a volume
  
• Support matrix updates
  
  • Kubernetes 1.21
    
  • RHEL 8.4
    
  • OpenShift 4.6 and 4.7
    
  • Rancher RKE II
    

PowerFlex CSI v1.4 update

• Multi-array Support
  
• Allow mounting RWO volume on multiple PODs
  
• Supports CSI Ephemeral Inline Volumes
  
• Support Matrix updates
  
  • PowerFlex 3.6
    
  • Kubernetes 1.20
    
  • OpenShift 4.7
    
  • RHEL 8.3
    
  • Fedora CoreOS
    

PowerFlex CSI v1.3 update

• Containerized SDC deployment
  
• Support for Red Hat OpenShift 4.6 and RH CoreOS 4.6
  
• Single-Click deployment of CSI on OpenShift Cluster
  
• Support for Bare Metal & Virtualized Deployment
  

Ansible Modules for Dell EMC PowerFlex

The Ansible Modules for Dell EMC PowerFlex allow Data Center and IT administrators to use RedHat Ansible to automate and orchestrate the provisioning and management of Dell EMC PowerFlex storage systems.

The capabilities of the Ansible modules are managing SDCs, volumes, snapshots and storage pools; and to gather high level facts from the storage system. The options available for each are list, show, create, modify and delete. These tasks can be executed by running simple playbooks written in yaml syntax. The modules are written so that all the operations are idempotent, so making multiple identical requests has the same effect as making a single request.

Ansible Modules for PowerFlex release 1.0 supports the following features:

• The following are the features of the gatherfacts module:
  
  • Get the API details of a PowerFlex storage device.
    
  • Get the list of SDCs.
    
  • Get the list of SDSs.
    
  • Get the list of volumes.
    
  • Get the list of snapshots.
    
  • Get the list of storage pools.
    
  • Get list of protection domains.
    
  • Get list of snapshot policies.
    
• The following are the features of the volume module:
  
  • Get the details of a volume.
    
  • Create a volume.
    
  • Modify details of a volume.
    
  • Delete a volume.
    
• The following are the features of the snapshot module:
  
  • Get the details of a snapshot.
    
  • Create a snapshot.
    
  • Modify details of a snapshot.
    
  • Delete a snapshot.
    
• The following are the features of the storage pools module:
  
  • Get the details of a storage pool.
    
  • Create a storage pool.
    
  • Modify details of a storage pool.
    
• The following are the features of the SDCs module:
  
  • Get the details of the SDC.
    
  • Rename an SDC.
    

Hardware updates

New acceleration card offerings:

• Next-gen NVIDIA acceleration cards now available for customers requiring specialized, high-performance computing and analytics applications – Quadro RTX 6000, Quadro RTX 8000, A40, and A100
  
• The first small form factor acceleration card supported by R640 – NVIDIA T4
  
• Replacement to V100 now available – NVIDIA A100
  

Other new offerings of note:

• Replacement to ConnectX-5 100Gb now available – Mellanox ConnectX-6 100Gb
  
• Factory installed fiber channel HBA cards* – Emulex LPE 31002
  

NSX-T Ready for PowerFlex appliance

NSX-T is VMware’s software-defined-network infrastructure that addresses cross-cloud needs for VM-centric network services and security

An “NSX-T Ready” PowerFlex rack or appliance includes all necessary hardware components needed to deploy and configure NSX-T software. The required hardware components are installed in the factory. Software deployment and lifecycle management are user responsibility

Customer provides NSX-T software and deploy with assistance from VMware or Dell Professional Service

New enabling hardware components

• 4-node PowerFlex management cluster available to host NSX-T controller VMs
  
• 2 to 8 NSX-T edge nodes for NSX-T edge services
  
• High level NSX-T topologies and consideration in Appliance Network Planning Guide
  
• PowerFlex Manager Lifecyle mode
  

Dell EMC AppSync for PowerFlex

AppSync is a software that enables Integrated Copy Data Management (iCDM) with Dell EMC’s primary storage systems.

The solution provides:

• Automated copy creation and consumption – no more manual steps or custom scripts
  
• Tight integration with host environments, applications and Dell EMC PowerFlex storage and replication
  
• Applications owners and storage admins are on the same page with a transparent copy workflow
  

AppSync simplifies and automates the process of generating and consuming copies of production data. By abstracting the underlying storage and replication technologies, and through deep application integration, AppSync empowers application owners to satisfy copy demand for operational recovery and data repurposing on their own. In turn, storage administrators need only be concerned with initial setup and policy management, resulting in an agile, frictionless environment.

AppSync automatically discovers application databases, learns the database structure, and maps it through the Virtualization Layer to the underlying storage LUN. It then orchestrates all the activities required from copy creation and validation through mounting at the target host and launching or recovering the application. Supported workflows also include refresh, expire, and restore production.

AppSync simplifies and automates the process of generating and consuming copies of production data. AppSync enables application owners to satisfy copy demand for data repurposing, operational recovery and disaster recovery across multiple Dell EMC arrays and applications with a single user interface.

With PowerFlex 3.6, AppSync supports Bronze Service Plans and local repurposing as well as Silver Service Plans and remote repurposing which utilizes Asynchronous replication

Other 3.6 Improvements

• Inclusive Language changes in UI
  
  • New MDM cluster component naming
    
  • Replace Master/Slave with Primary/Secondary
    
• Improved/simplified layout in WebUI
  
  • Consolidated menus
    
  • Dedicated section for Snapshots and Snapshot Policies
    
• Support for Oracle Linux KVM
  
  • In addition to pre-existing support for OEL
    
  • Enables Oracle database sales plays, where users need to run Oracle software at all layers
    
• MG layout write improvements
  
• 
  Improved first writes to new data layout chunks for volumes and snapshots
  
• Hardware Details in UI
  

You can download PowerFlex 3.6, from the link below

The documentation for the release, can be found at the following link: https://www.dell.com/support/home/en-il/product-support/product/scaleio/docs#sort=relevancy&f:versionFacet=%5B50723%5D&f:lang=%5Ben%5D

A guest post by Tomer Nahumi

The post Dell EMC PowerFlex 3.6 – part1, An high level overview appeared first on Itzikr's Blog.

in the first part of the series, we provided an high-level overview of the PowerFlex 3.6 release. you can read all about it here.

now, let’s dive deeper into the integration of PowerFlex with VMware SRM (Site Recovery Manager).

VMware Site Recovery Manager is a business continuity and disaster recovery solution that helps you plan, test, and run the recovery of virtual machines between a protected vCenter Server site and a recovery vCenter Server site.

You can use Site Recovery Manager to implement different types of recovery from the protected site to the recovery site.

Planned migration is the orderly migration of virtual machines from the protected site to the recovery site. Planned migration prevents data loss when migrating workloads in an orderly fashion. For planned migration to succeed, both sites must be running and fully functional.

Disaster recovery does not require that both sites be up and running, and it can be initiated in the event of the protected site going offline unexpectedly. During a disaster recovery operation, failure of operations on the protected site is reported, but otherwise ignored.

Site Recovery Manager orchestrates the recovery process with the replication mechanisms, to minimize data loss and system downtime.

• Operates as an extension of vCenter Server
  
• Automates the recovery or migration of virtual machines between a protected site and a recovery site
  
• Uses array-based replication solutions from other vendors, in our case, it’s PowerFlex
  

Planned Migration

Planned migration is a vCenter Site Recovery Manager workflow that enables application-consistent migration of virtual machine workloads with zero data loss to the recovery site

Automated Failback

Automated failback provides an automated workflow to migrate virtual machines back to the protected site after the recovery site is restored.

About Storage Replication Adapters and Array Managers

An SRA is a program provided by a storage array vendor that enables vCenter Site Recovery Manager to work with the vendor’s array.

Functions performed by SRAs:

• Array discovery
  
• Replicated logical unit number (LUN)
  recognition
  
• Initiation of recovery plan tests and
  execution
  

An SRA is used by an array manager, a vCenter Site Recovery Manager component that helps to identify available arrays and replicated LUNs.

• PowerFlex SRA enables the use of PowerFlex native asynchronous replication as the replication engine for protecting VMs on vSphere datastores
  
• PowerFlex SRA for use with Site Recovery Manager 8.2* or 8.3* on the SRM Server appliance
  
• PowerFlex replication must be configured and working
  
• At least one Replication Consistency Group containing volumes that are:
  
  • Mapped to ESXi hosts
    
  • Used to create the datastores hosting the VMs that SRM will protect
    

PowerFlex SRA – Architecture

Failover and Planned Migration with vCenter Site Recovery Manager Recovery Plans

Unplanned Failover – Recover from unexpected site failure:

• Full or partial site failure

The most critical but least frequent use case:

• Unexpected site failures do not happen often.
• When site failures happen, fast recovery is critical to the business.

Preventive Failover – Anticipate potential data center outages:

• Examples: hurricane, floods, forced evacuation, and so on

Initiate preventive failover for smooth migration:

• Graceful shutdown of virtual machines at protected site
• Using vCenter Site Recovery Manager planned migration capabilities to ensure no data loss
• Planned Migration

Most frequent vCenter Site Recovery Manager use case:

• Planned data center maintenance
• Global load balancing

Ensure smooth site migrations:

• Test to minimize risk.
• Execute partial failovers.
• Use vCenter Site Recovery Manager planned migration to minimize data loss.
• Use automated failback, which enables bidirectional migrations.

Planned Migration and Application Consistency

Planned migration ensures application consistency and no data loss during migration by providing the following:

• Graceful shutdown of production virtual machines in an application-consistent state
• Data sync to complete replication of virtual machines
• Recovery of fully replicated virtual machines

Recovery Testing and SRM

The ability to non-disruptively test a recovery plan is one of the most powerful features of VMware Site Recovery Manager (SRM). Frequent testing reduces risk by reducing drift between plans and desired behavior. It also provides the confidence that if a disaster strikes you are ready, knowing how SRM and your applications will respond.
An SRM recovery plan test is designed to be as close as possible to running an actual recovery plan while not disrupting the protected VMs and not impacting replication.
From a storage perspective, SRM will utilize the PowerStore snapshots at the designated testing site so replication will still keep going on and from a network perspective, you can either designate your own dedicated vSwitches that needs to be isolated from your production network or you can use the built in SRM “auto-network”, while it’s easier to go with the latter, networking will only work between the VMs running on that host during the test since, its not connected to any physical nic so really, the latter option is more of a safety belt mechanism rather than a true test scenario so you really want to designate a real, isolated network vSwitches for the test.

PowerFlex SRA Compatibility Matrix

The following table presents the Dell EMC PowerFlex SRA Compatibility Matrix; the latest versions are recommended:

Setting Up the PowerFlex Native Replication

To use the VMware Site Recovery Manager, the PowerFlex clusters must be configured for native replication. Refer to the to Configure and Customize PowerFlex Guide for a detailed explanation of setting up the PowerFlex clusters for native replication.

Installation

• Verify that PowerFlex clusters are configured to be fully operational, including remote systems, volume groups, and replication sessions.
• Download the Dell EMC PowerFlex SRA file:
  Dell_EMC_PowerFlex_SRA_Linux_v.1.0.0
  
• Go to SRM WEB UI https://<FQDN>/, and click Launch SRM Appliance Management.

The Welcome to VMware Appliance Management dialog box appears.

Enter your credentials (default administrator user is ‘admin’), and click LOG IN.

In the SRM Appliance Management dialog box, click Storage Replication Adapters.

Click NEW ADAPTER; the New Adapter dialog box appears.

Click UPLOAD, and in the resulting file browser, navigate to and select the Dell EMC PowerFlex SRA.tar.gz file; the upload process commences.

Close the resulting Success window, confirming a successful SRA upload procedure.

Advanced Configuration

Here you can see a demo of VMware SRM Failover with PowerFlex 3.6.

A guest post by Tomer Nahumi

The post DELL EMC PowerFlex 3.6 – Part2, PowerFlex SRA for VMware SRM appeared first on Itzikr's Blog.

On the previous blog post, we covered the SyncIQ Jobs:

• The three phases of SyncIQ
• Configuring a SyncIQ policy
• Impacts of modifying SyncIQ policies
• SyncIQ performance rules

And on this blog post we will cover the following topics

• SyncIQ design considerations
  
• Failover and failback
  
• Superna Eyeglass DR Edition
  
• SyncIQ security
  

SyncIQ design considerations

Prior to configuring data replication policies with SyncIQ, it is recommended to map out how policies align with IT administration requirements. Data replication between clusters is configured based on either entire cluster replication or directory-based replication. Designing the policy to align with departmental requirements ensures policies satisfy requirements at the onset, minimizing policy reconfiguration. When creating policies, Disaster Recovery (DR) plans must be considered, in the event of an actual DR event. DR readiness is a key factor to success during a DR event.

As policies are created for new departments, it is important to consider policy overlap. Although the overlap does not impact the policy running, the concerns include managing many cumbersome policies and resource consumption. If the directory structure in policies overlap, data is being replicated multiple times impacting cluster and network resources. During a failover, time is a critical asset. Minimizing the number of policies allows administrators to focus on other failover activities during an actual DR event. Additionally, RPO times may be impacted by overlapping policies.

During the policy configuration stage, select options that have been tested in a lab environment. For example, for a synchronize policy configured to run anytime the source is modified, consider the time delay for the policy to run. If this is set to zero, every time a client modifies the dataset, a replication job is triggered. Although this may be required to meet RPO and RTO requirements, administrators must consider if the cluster resources and network bandwidth can meet the aggressive replication policy.

Considering cluster resources with data replication

As the overall architecture of SyncIQ Policies is designed, other factors to consider are the number of policies running together. Depending on how policies are configured, the cluster may have many policies running at once. If many policies are running together, cluster resources and network bandwidth must be considered. Under standard running conditions, the cluster resources are also providing client connectivity with an array of services running. It is imperative to consider the cluster and network utilization when the policies are running.

Source and target cluster replication performance

During the design phase, consider the node types on the source and target cluster impacting the overall data replication performance. When a performance node on the source cluster is replicating to archive nodes on the target cluster, this causes the overall data replication performance to be compromised based on the limited performance of the target cluster’s nodes. For example, if a source cluster is composed of F800 nodes and the target cluster is composed of A200 nodes, the replication performance reaches a threshold, as the A200 CPUs cannot perform at the same level as the F800 CPUs.

Snapshots and SyncIQ policies

As snapshots and SyncIQ policies are configured, it is important to consider the scheduled time. As a best practice, it is recommended to stagger the scheduled times for snapshots and SyncIQ policies. Staggering snapshots and SyncIQ policies at different times ensures the dataset is not interacting with snapshots while SyncIQ jobs are running, or vice versa. Additionally, if snapshots and SyncIQ policies have exclusive scheduled times, this ensures the maximum system resources are available, minimizing overall run times. However, system resources are also dependent on any Performance Rules configured, as stated in Section 8, SyncIQ performance rules.

Jobs targeting a single directory tree

Creating SyncIQ policies for the same directory tree on the same target location is not supported. For example, consider the source directory /ifs/data/users. Creating two separate policies on this source to the same target cluster is not supported:

• one policy excludes /ifs/data/users/ceo and replicates all other data in the source directory

• one policy includes only /ifs/data/users/ceo and excludes all other data in the source directory

Splitting the policy with this format is not supported with the same target location. It would only be supported with different target locations. However, consider the associated increase in complexity required in the event of a failover or otherwise restoring data.

Authentication integration

UID/GID information is replicated, via SID numbers, with the metadata to the target cluster. It does not require to be separately restored on failover.

Small File Storage Efficiency (SFSE) and SyncIQ

OneFS’ Small File Storage Efficiency (SFSE) provides a feature for small files in OneFS, packing them into larger files, resulting in increased storage efficiency. If a SyncIQ policy is configured for a SFSE dataset, the data is replicated to the target cluster. However, the SFSE dataset is unpacked on the source cluster prior to replication. If the target cluster has SFSE enabled, the dataset is packed when the next SmartPools job runs on the target cluster. If the target cluster does not have SFSE enabled, the dataset remains unpacked.

Failover and failback

Under normal operation, SyncIQ target directories can be written to only by the SyncIQ job itself – all client writes to any target directory are disabled, this is referred to as a protected replication domain. In a protected replication domain, files cannot be modified, created, deleted or moved within the target path of a SyncIQ job.

SyncIQ provides built-in recovery to the target cluster with minimal interruption to clients. By default, the RPO (recovery point objective) is to the last completed SyncIQ replication point. Optionally, with the use of SnapshotIQ, multiple recovery points can be made available.

Failover

In the event of a planned or unplanned outage to the source cluster, a failover is the process of directing client traffic from the source cluster to the target cluster. An unplanned outage of the source cluster could be a disaster recovery scenario where the source cluster no longer exists, or it could be unavailable if the cluster is not reachable.

On the contrary, a planned outage is a coordinated failover, where an administrator knowingly makes a source cluster unavailable for disaster readiness testing, cluster maintenance, or other planned event. Prior to performing a coordinated failover, ensure a final replication is completed prior to starting, ensuring the dataset on the target matches the source.

To perform a failover, set the target cluster or directory to Allow Writes.

Failover while a SyncIQ job is running

It is important to note that if the replication policy is running at the time when a failover is initiated, the replication job will fail, allowing the failover to proceed successfully. The data on the target cluster is restored to its previous state before the replication policy ran. The restore completes by utilizing the snapshot taken by the replication job after the last successful replication job.

Target cluster dataset

If for any reason the source cluster is entirely unavailable, for example, under a disaster scenario, the data on the target cluster will be in the state after the last successful replication job completed. Any updates to the data since the last successful replication job are not available on the target cluster.

Failback

Users continue to read and write to the target cluster while the source cluster is repaired. Once the source cluster becomes available again, the administrator decides when to revert client I/O back to it. To achieve this, the administrator initiates a SyncIQ failback, which synchronizes any incremental changes made to the target cluster during failover back to the source. When complete, the administrator redirects client I/O back to the original cluster again.

Failback may occur almost immediately, in the event of a functional test, or more likely, after some elapsed time during which the issue which prompted the failover can be resolved. Updates to the dataset while in the failover state will almost certainly have occurred. Therefore, the failback process must include propagation of these back to the source.

Resync-prep

Run the preparation phase (resync-prep) on the source cluster to prepare it to receive intervening changes from the target cluster. This phase creates a read-only replication domain with the following steps:

• The last known good snapshot is restored on the source cluster.

• A SyncIQ policy is created on the target policy appended with ‘_mirror’. This policy is used to failback the dataset with any modification that has occurred since the last snapshot on the source cluster. During this phase, clients are still connected to the target.

Mirror policy

Run the mirror policy created in the previous step to sync the most recent data to the source cluster.

Verify

Verify that the failback has completed, via the replication policy report, and redirect clients back to the source cluster again. At this time, the target cluster is automatically relegated back to its role as a target.

Allow-writes compared to break association

Once a SyncIQ policy is configured between a source and target cluster, an association is formed between the two clusters. OneFS associates a policy with its specified target directory by placing a cookie on the source cluster when the job runs for the first time. The cookie allows the association to persist, even if the target cluster’s name or IP address is modified. SyncIQ provides two options for making a target cluster writable after a policy is configured between the two clusters. The first option is to ‘Allow-Writes’, as stated previously in this section. The second option to make the target cluster writeable, is to break a target association.

If the target association is broken, the target dataset will become writable, and the policy must be reset before the policy can run again. A full or differential replication will occur the next time the policy runs. During this full resynchronization, SyncIQ creates a new association between the source and its specified target.

In order to perform a Break Association, from the target cluster’s CLI, execute the following command:

isi sync target break –policy=[Policy Name]

To perform this from the target cluster’s web interface, select Data Protection > SyncIQ and select the “Local Targets” tab. Then click “More” under the “Actions” column for the appropriate policy, and click “Break Association”.

Superna Eyeglass DR Edition

Many SyncIQ failover and failback functions can be automated with additional features through Superna Eyeglass® DR Edition. Superna provides software that integrates with PowerScale, delivering disaster recovery automation, security, and configuration management. Dell EMC sells Superna software as a Select partner.

Superna’s DR Edition supports PowerScale SyncIQ by automating the failover process. Without Superna the failover process requires manual administrator intervention. Complexity is minimized with DR Edition as it provides one-button failover, but also updates Active Directory, DNS, and client data access.

Once DR Edition is configured it continually monitors the PowerScale cluster for DR readiness through auditing, SyncIQ configuration, and several other cluster metrics. The monitoring process includes alerts and steps to rectify discovered issues. In addition to alerts, DR edition also provides options for DR testing, which is highly recommended, ensuring IT administrators are prepared for DR events. The DR testing can be configured to run on a schedule. For example, depending on the IT requirements, DR testing can be configured to run on a nightly basis, ensuring DR readiness.

As DR Edition collects data, it provides continuous reports on RPO compliance, ensuring data on the target cluster is current and relevant.

Superna Eyeglass DR Edition is recommended as an integration with PowerScale SyncIQ, providing a simplified DR process and further administrator insight into the SyncIQ configuration. For more information about Superna Eyeglass DR Edition, visit https://www.supernaeyeglass.com/dr-edition.

SyncIQ security

By default, SyncIQ starts replication to a target PowerScale cluster specified without any configuration necessary on the target cluster. The replication policy is configured on the source cluster only, and if network connectivity is available through the front-end ports, the replication policy is initiated.

Depending on the network architecture hierarchy and where the PowerScale clusters are placed in the hierarchy, this could be a concern. For instance, a cluster could receive many replication policies from a source cluster that could overwhelm its resources. In environments where several PowerScale clusters are active, an administrator may inadvertently specify the IP address of another cluster rather than the intended target cluster.

Securing a PowerScale cluster from unauthorized replication of data is performed through two available options. As a best practice and per DSA-2020-039, Dell EMC PowerScale OneFS Security Update for a SyncIQ Vulnerability, enabling SyncIQ encryption, is recommended, preventing man-in-the-middle attacks and alleviating security concerns. SyncIQ encryption was introduced in OneFS 8.2.

SyncIQ is disabled by default on greenfield OneFS release 9.1 clusters. Once SyncIQ is enabled, the global encryption flag is enabled, requiring all SyncIQ policies to be encrypted. For PowerScale clusters upgraded to OneFS 9.1, the global encryption flag is also enabled. However, the global encryption flag is not enabled on PowerScale clusters upgraded to OneFS 9.1 with an existing SyncIQ policy.

As an alternative for PowerScale clusters running a release prior to OneFS 8.2, a SyncIQ pre-shared key (PSK) can be configured, protecting a cluster from unauthorized replication policies without the PSK.

SyncIQ encryption

OneFS release 8.2 introduced over-the-wire, end-to-end encryption for SyncIQ data replication, protecting and securing in-flight data between clusters. A global setting is available enforcing encryption on all incoming and outgoing SyncIQ policies.

SyncIQ provides encryption through the use of X.509 certificates paired with TLS version 1.2 and OpenSSL version 1.0.2o. The certificates are stored and managed in the source and target cluster’s certificate stores, as illustrated in Figure 30. Encryption between clusters is enforced by each cluster, storing its certificate and its peer’s certificate. Therefore, the source cluster is required to store the target cluster’s certificate, and vice versa. Storing the peer’s certificate essentially creates a white list of approved clusters for data replication. SyncIQ encryption also supports certificate revocation through the use of an external OCSP responder.

The post Dell EMC PowerScale OneFS 9.2, Part7 – SyncIQ Considerations appeared first on Itzikr's Blog.

Previously, we released the 1.3 CSI driver for the PowerFlex array, you can read all about it here And the PowerFlex 1.4 CSI version, which you can read all bout here

CSI Driver for Dell EMC PowerFlex v1.5 is aligned with CSI Spec 1.3 and supports the following new features:

• Added support for PowerFlex 3.6
  
• Added support for Kubernetes v1.21
  
  
• Added support for OpenShift 4.6 EUS and 4.7 with RHEL and CoreOS worker nodes
  
  
• Added support for Red Hat Enterprise Linux (RHEL) 8.4
  
  
• Added support for Mirantis Kubernetes Engine (FKA Docker EE 3.4)
  
  
• Added support for Rancher RKE v1.2.8
  
  
• Added support for MKFS format option
  
  
• Added support for dynamic log config
  
• Config.yaml support
  
  
• CSM Volume Group Snapshotter
  

The following describes the support strategy for CSI Driver for Dell EMC PowerFlex:

• The CSI Driver for Dell EMC PowerFlex image, which is the built driver code, is available on
  https://hub.docker.com/r/dellemc/csi-vxflexos and is officially supported by Dell EMC.
  
  
• The source code available on Github https://github.com/dell/csi-powerflex
  is unsupported and provided solely under the terms of the license attached to the source code. For clarity, Dell EMC does not provide support for any source code modifications.
  
  
• A Dell EMC Storage Automation and Developer Resources page is available and includes developer resources, case studies, forums, and other materials. It is anticipated that members of this page will be excellent resources in addressing product issues and concerns. It is recommended this community page be the first page customers use for their product questions prior to contacting Dell EMC Support. For any setup, configuration issues, questions or feedback, join the Dell EMC Container community at
  Dell EMC Storage Automation and Developer Resources .
  

Documentation and Downloads

CSI Driver for Dell EMC PowerFlex v1.5 downloads and documentation are available on:

https://github.com/dell/csi-powerflex

https://dell.github.io/storage-plugin-docs/docs/features/powerflex/

New Features in v1.5

Volume Snapshot Feature

Many stateful Kubernetes applications use several persistent volumes to store data. To create recoverable snapshots of volumes for these applications, it is necessary to have capability to create consistent snapshots across all volumes of the application at the same time.
Dell CSM Volume Group Snapshotter is an operator which extends Kubernetes API to support crash-consistent snapshots of groups of volumes. This operator consists of VolumeGroupSnapshot CRD and csi-volumegroupsnapshotter controller. The csi-volumegroupsnapshotter is a sidecar container, which runs in the controller pod of CSI driver. The csi-volumegroupsnapshotter uses CSI extension, implemented by Dell EMC CSI drivers, to manage volume group snapshots on backend arrays.

CSM Volume Group Snapshotter is currently in a Technical Preview Phase, and should be considered alpha software. We are actively seeking feedback from users about its features. Please provide feedback using . We will take that input, along with our own results from doing extensive testing, and incrementally improve the software. We do not recommend or support it for production use at this time.

The Volume Snapshot feature was introduced in alpha (v1alpha1) in Kubernetes 1.13 and then moved to beta (v1beta1) in Kubernetes 1.17 and is generally available (v1) in Kubernetes version >=1.20.

The CSI PowerFlex driver version 1.5 supports v1beta1 snapshots on Kubernetes 1.19 and v1 snapshots on Kubernetes 1.20 and 1.21.

In order to use Volume Snapshots, ensure the following components are deployed to your cluster:

• Kubernetes Volume Snaphshot CRDs
  
• Volume Snapshot Controller
  

Before PowerFlex driver v1.5, the installation of the driver created default instance of VolumeSnapshotClass. API version of this VolumeSnapshotClass instance was defined based on Kubernetes version, as below:

Following is the manifest for the Volume Snapshot Class created during installation, prior to PowerFlex driver v1.5:

{{- if eq .Values.kubeversion “v1.20” }}

apiVersion: snapshot.storage.k8s.io/v1

{{- else }}

apiVersion: snapshot.storage.k8s.io/v1beta1

{{- end}}

kind: VolumeSnapshotClass

metadata:

name: vxflexos-snapclass

driver: csi-vxflexos.dellemc.com

deletionPolicy: Delete

Note: Installation of PowerFlex driver v1.5 does not create VolumeSnapshotClass. You can find samples of default v1beta1 and v1 VolumeSnapshotClass instances in helm/samples/volumesnapshotclass directory. There are two samples, one for v1beta1 version and the other for v1 snapshot version. If needed, install appropriate default sample, based on the version of snapshot CRDs in your cluster.

Create Consistent Snapshot of Group of Volumes

This feature extends CSI specification to add the capability to create crash-consistent snapshots of a group of volumes. PowerFlex driver implements this extension in v1.5. This feature is currently in Technical Preview. To use this feature users have to deploy csi-volumegroupsnapshotter side-car as part of the PowerFlex driver.

Volume group snapshotter support added this release only for helm.
Dell CSM Volume Group Snapshotter is an operator which extends Kubernetes API to support crash-consistent snapshots of groups of volumes.
The csi-volumegroupsnapshotter is a sidecar container, which runs in the controller pod of CSI driver
CSM Volume Group Snapshotter is currently in a Technical Preview Phase and should be considered alpha software.

Custom File System Format Options

The CSI PowerFlex driver version 1.5 supports additional mkfs format options. A user is able to specify additional format options as needed for the driver. Format options are specified in storageclass yaml under mkfsFormatOption as in the following example:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: vxflexos
annotations:
storageclass.kubernetes.io/is-default-class: “true”
provisioner: csi-vxflexos.dellemc.com
reclaimPolicy: Delete
allowVolumeExpansion: true
parameters:
storagepool: <STORAGE_POOL> # Insert Storage pool
systemID: <SYSTEM_ID> # Insert System ID
mkfsFormatOption: “<mkfs_format_option>” # Insert file system format option
volumeBindingMode: WaitForFirstConsumer
allowedTopologies:
– matchLabelExpressions:
– key: csi-vxflexos.dellemc.com/<SYSTEM_ID> # Insert System ID
values:
– csi-vxflexos.dellemc.com

Multiarray Support

The CSI PowerFlex driver version 1.5 adds support for managing multiple PowerFlex arrays from the single driver instance. This feature is enabled by default and integrated to even single instance installations.

To manage multiple arrays you need to create an array connection configuration that lists multiple arrays.

Creating array configuration

There is a sample yaml file under the top directory named config.yaml with the following content:

– username: “admin” # username for connecting to API
password: “password” # password for connecting to API
systemID: “ID1” # system ID for system
endpoint: “https://127.0.0.1&#8221; # full URL path to the PowerFlex API
skipCertificateValidation: true # skip array certificate validation or not
isDefault: true # treat current array as default (would be used by storage class without arrayIP parameter)
mdm: “10.0.0.1,10.0.0.2” # MDM IPs for the system
– username: “admin”
password: “Password123”
systemID: “ID2”
endpoint: “https://127.0.0.2&#8221;
skipCertificateValidation: true
mdm: “10.0.0.3,10.0.0.4”

Here we specify that we want the CSI driver to manage two arrays: one with an IP 127.0.0.1 and the other with an IP 127.0.0.2.

To use this config we need to create a Kubernetes secret from it. To do so, run the following command:

kubectl create secret generic vxflexos-config -n vxflexos –from-file=config=config.yaml

Creating storage classes

To be able to provision Kubernetes volumes using a specific array we need to create corresponding storage classes.

Find the sample yaml files under helm/samples/storageclass. Edit storageclass.yaml if you want ext4 filesystem, and use storageclass-xfs.yaml if you want xfs filesystem. Replace <STORAGE_POOL> with the storage pool you have, and replace <SYSTEM_ID> with the system ID you have.

Then we need to apply storage classes to Kubernetes using kubectl:

kubectl create -f storageclass.yaml

After that, you can use the storage class for the corresponding array.

Dynamic Logging Configuration

This feature is introduced in version 1.5, CSI Driver for PowerFlex now supports dynamic logging configuration.

To accomplish this, we utilize two fields in logConfig.yaml: LOG_LEVEL and LOG_FORMAT.

LOG_LEVEL: minimum level that the driver will log
LOG_FORMAT: format the driver should log in, either text or JSON

If the configmap does not exist yet, simply edit logConfig.yaml, changing the values for LOG_LEVEL and LOG_FORMAT as you see fit.

If the configmap already exists, you can use this command to edit the configmap:
kubectl edit configmap -n vxflexos driver-config

or you could edit logConfig.yaml, and use this command:
kubectl apply -f logConfig.yaml

and then make the necessary adjustments for LOG_LEVEL and LOG_FORMAT.
If LOG_LEVEL or LOG_FORMAT are set to options outside of what is supported, the driver will use the default values of “info” and “text” .

Here you can see a demo of the new CSM Volume Group Snapshotter feature:

A guest post by Tomer Nahumi

The post CSI Driver for DELL EMC PowerFlex v1.5 appeared first on Itzikr's Blog.

On June 10th, 2021, Dell Technologies released PowerStore OS version 2.0. This release brought many new features, enhancements, and capabilities to the fast growing PowerStore user base. One of those new features is a Disaster Recovery (DR) testing feature that is seamlessly integrated into PowerStore Manager (the HTML5 GUI).

In the image above you can see the addition of a new button that allows an administrator to initiate a “Failover Test” which allows for seamless DR testing. This feature exists on the target array which is the destination array for a replication agreement. So literally by the click-of-a-button a full DR test can performed.

Key Concepts of the DR Test Feature

1. It is a completely seamless and integrated capability of PowerStore’s native asynchronous replication
2. Simulates a fully featured failover to the Disaster Recovery site
3. During the test the DR location has full read-write access to the mapped volumes at the DR site
4. The testing period has no time restrictions meaning the testing period can be as long or as short as you want
5. Production data and replication session is NOT impacted by performing a DR Test. Your production replication and protection continues while the DR Failover Test is taking place.
6. The DR Failover Test can be performed with the most recently synchronized data or any available snapshot

Defining Source and Destination

As I mentioned, this is predicated on having a replication agreement setup with what we call a source array and a destination array.

The source array, as shown in the image above, is the production array that houses your production volumes. The destination or target is another PowerStore array that in a replication agreement and is the receiving target for your replicated production data. Note that the destination systems volumes are in a “Read-Only” state while they are actively part of an active replication agreement. This protects the destination from being overwritten and preserves the integrity of the data within the DR destination system.

The source and destination setup is extremely simple and can be implemented via an intuitive wizard and protection policies. In fact, adding a volume to an pre-defined protection policy at the point of creation will automagically establish the replication agreement and schedule for you.

Note in the image above, during the volume creation process, you can create a single volume or hundreds of volumes and have the replication schedules applied instantly via the Volume Protection Policy. The Volume Protection Policy allows you to customize snapshot and replication schedules based on your needs and then apply the policy instantly to a volume or volume group. There is no need to manage replication sessions for each and every volume… click… click… done! It is that simple.

Once the DR Failure Test is initiated (click the button) the destination array makes the target data read-writeable while isolating the production continuous replication stream (see below image).

Let’s Make it Happen

The steps to making a DR Failover Test happen are extremely simple, so let’s make it happen in 1…2…3…!

1. On the Destination array go to the volume properties and select the Protection tab. Once in Protection tab select the Replication tab to view the replication status. It is here you will see the START FAILOVER TEST button.

2. Press the START FAILOVER TEST button

3. Next you are prompted to use the most recently synchronized data OR you can choose to use any pre-existing snapshot.

Let’s use the most current data for this example… Select the START FAILOVER TEST button to confirm. Once this step is completed you are done! All that is left to do is mount the DR Test Volume to the DR hosts which will now have read-write access.

You can tell that you are in DR Test mode by the refreshed Replication properties screen (see below image):

Once you have completed your DR testing scenario and validated your test process and data you can STOP FAILOVER TEST from the same screen. A very nice feature to the STOP FAILOVER TEST process is the ability to preserve your testing data if desired. When you press the STOP FAILOVER TEST button you are immediately greeted with a confirmation with the ability to preserve your testing data. If you check the box, a snapshot will be automatically created preserving all of your testing data (new writes to the volume).

Clicking STOP FAILOVER TEST on this confirmation screen instantly stops the DR FAILOVER TEST and returns the destination volume back to its synchronized state with the original production volume.

Note in the image above the volume replication screen provides the following notification: The failover test has been stopped. The read/write mode has been disabled on the destination resource.

Summary

Simplicity is the key. PowerStore creates replication sessions for you automatically based on the protection policy you define for volumes or volume groups. Once a replication session has been initiated the DESTINATION system will have the DR FAILOVER TEST option available under the protection/replication properties tab. And that is DR Testing made simple!

More details about DR FAILOVER TEST can be found in the PowerStore Protecting Your Data guide.

You can also view a video based demo of the DR FAILOVER TEST option on the Dell EMC YouTube Channel.

A guest post by Jodey Hogeland

The post Dell EMC PowerStore, DR Testing Made Simple appeared first on Itzikr's Blog.

The virtual machine snapshot is a commonly used feature in the VMware vSphere environment, before upgrading/patching virtual machines, and during backup processes.

One of the most common questions we get is, “What is the performance impact of VM snapshot creation/deletion on the performance of guest applications running inside the VMs?”

In this post, we will explore these performance aspects in VMFS, and vVOL datastores running on PowerStore.

A VMware snapshot preserves the state and data of a VM at a specific point in time.

• The state includes the VM’s power state (for example, powered on, powered off, suspended).
  
• The data includes all the files that make up the VM. This includes disks, memory, and other devices, such as virtual network interface cards.
  

When taking a VM snapshot of a VMFS-based VM, the state of the virtual disk is preserved, the guest stops writing to it, and a delta or child disk is created.as a result, there is a performance degradation. That’s because when a snapshot is taken, what really happens is that a delta VMDK is created. VMware uses SEsparse
format for all delta disks on VMFS6 datastores In this scenario, all the new writes from that virtual machine go to the delta VMDK, instead of the base VMDK, while the reads go to the base VMDK . This makes that base VMDK store the point-in-time state of the virtual disk. This mechanism is called “redirect-on-write”.

With VMware virtual volumes (vVols), data services such as snapshot and clone operations are offloaded to the storage array. With vVols, PowerStore uses native snapshot facilities, hence vSphere snapshots can operate at a base disk performance level.

When you right-click on a VM and choose take snapshot, VMware does not create the performance-impacting delta VMDK files that were traditionally used, but instead VMware entirely offloads this process to the array. So, the array creates the snapshots and VMware just tracks them.

A major benefit for VVols with Powerstore is that snapshots are now Powerstore volume copies, which are globally data reduced , metadata based (so their creation, restore from and deletion is instantaneous) and are not copy-on-write (so there’s no performance impact during backup operations for example).

Dell EMC PowerStore provides a simple but powerful approach to local data protection using snapshots. PowerStore uses the same snapshot technology across all the resources within the system, including volumes, volume groups, file systems, virtual machines, and thin clones. Snapshots use thin, redirect-on-write technology to ensure that system space is used optimally and reduces the management burden by never requiring administrators to designate protection space.

PowerStore supports the VMware vSphere Virtual Volumes (vVols) framework through the VASA 3.0 protocol. This feature enables VM-granular data services and Storage Policy Based Management (SPBM). In traditional storage environments, volumes or file systems are formatted as VMFS or NFS datastores for VMs. Data services are applied at the volume or file-system level, which means all VMs that reside on that datastore are also affected.

With vVols, VM data is stored on dedicated storage objects that are called storage containers, which become vVol datastores in vSphere. A VM consists of multiple vVols depending on its configuration and status. PowerStore works with vSphere to track which vVols belong to which VM.

Data services, such as VM snapshots and clones, can be applied at a VM-level of granularity since they are only applied to the relevant vVols. These data services are offloaded to PowerStore to maximize efficiency. Policies and profiles can be leveraged to ensure VMs are provisioned with the required storage capabilities.

VM snapshots are visible in both PowerStore Manager and vCenter, regardless of where they are created. You can view information about VM snapshots in the Manage Snapshots page in vCenter. You can also initiate a revert operation from here in order to restore the VM using the snapshot. You can revert to any snapshot in the snapshot tree.

Performance Impact Comparison

There are three primary performance-impacting events when taking a VM snapshot of a VMFS-based VM:

• Creation of a snapshot – The process basically pauses all I/O (if the option is chosen to quiesce the virtual machine state).
  
• Existence of a snapshot – In this case the redirection happens, and all workload is impacted during this period. This is experienced as lower throughput and IOps and higher latency.
  
• Deletion of a snapshot – a VM snapshot of a VMFS-based VM is not just deleted like a storage array-based snapshot. It writes all the changes from the delta VMDK into the original VMDK, so this merge process can be quite impactful, especially if there are a lot of changes to write.
  

My experiment included 2 Windows 2019 virtual machines running 100% of random 8KB I/Os and 70% reads, 30% writes on a 200GB drive, WIN_VMFS virtual machine running on a VMFS6 PowerStore datastore, WIN_VVOL virtual machine running on a PowerStore vVOL datastore.

The main focus of this experiment is to understand the impact on the guest application performance with snapshots. The screenshots below have been taken from vROPS and PowerStore Manager.

Before taking the snapshots, the WIN_VMFS virtual machine was generating 90K IOps and 700 MBps while the WIN_VVOL virtual machine was generating 93K IOps and 730 MBps, the difference is due to the VMFS file system overhead.

Immediately during the snapshot creation, we see a big performance impact on the WIN_VMFS virtual machine, the IOps dropped to 20K while the bandwidth dropped to MBps

You can see the performance continues to suffer as long as the snapshot still exists on the WIN_VMFS virtual machine.

The guest performance impact in the presence of snapshots on VMFS is due to the nature of the SEsparse redo logs. When I/O is issued from a VM with a snapshot, vSphere determines whether the data resides in the base VMDK (data written prior to a VM snapshot operation) or if it resides in the redo-log (data written after the VM snapshot operation) and the I/O is serviced accordingly.

Taking a snapshot of the VMFS_VVOL virtual machine didn’t cause any performance impact on the virtual machine at all, the virtual machine continued generating the same amount if IOps and bandwidth as before.

With VMware virtual volumes (vVols), data services such as snapshot and clone operations are offloaded to the storage array. PowerStore uses native snapshot facilities, hence vSphere snapshots can operate at a base disk performance level.

A major benefit for VVols with Powerstore is that snapshots are now Powerstore volume copies, which are globally data reduced , metadata based (so their creation, restore from and deletion is instantaneous) and are not copy-on-write (so there’s no performance impact during backup operations for example).

In comparison to VMFS, we observe the presence of VM snapshots on the vVOL datastore has zero impact on the guest application performance.

Impact of snapshot removal on guest performance

Deleting a snapshot on VMFS filesystem consolidates the changes between snapshots and writes all the data from the delta disk to the parent snapshot. When you delete the base parent snapshot, all the changes merge with the base VM disk.

With PowerStore vVOLs, snapshot deletion is an instantaneous operation with zero performance impact on the guest operation system virtual machine.

It took almost 15 minutes to delete the snapshot on the VMFS virtual machine compared to 2 seconds for the vVOL virtual machine, while during the snapshot deletion, the VMFS virtual machine performance continued to suffer until the snapshot has been deleted completely.

Conclusion

• The presence of snapshots can have a significant impact on guest performance in a VMFS environment, for I/O intensive workloads. Hence, the use of snapshots only on a temporary basis is strongly recommended.
  
• Using PowerStore vVOLs, which uses native snapshot technology, is highly recommended to avoid any guest impact during snapshots creation, existence, or deletion.
  
• Our findings show that performance degradation on VMFS datastores is higher as the snapshot chain length increases. Keeping the snapshot chain length smaller whenever possible is recommended to minimize the guest performance impact when using VMFS datastores.
  

PowerStore provides additional data protection with array-based snapshots. Snapshots are point-in-time space-efficient copies of data and take seconds to create with zero impact during creation, existence, or deletion. With tight integration with VMware vSphere, PowerStore can take vVol-based snapshots directly from the PowerStore manager using a protection schedule or on-demand. VM snapshot information is seen in both PowerStore and in vCenter.

A guest post by Tomer Nahumi

The post Dell EMC PowerStore Performance Comparison for VMware Snapshots with vVOLs and VMFS appeared first on Itzikr's Blog.

A guest post by Marco Abela

When we designed PowerStore, we wanted integrations with VMware to be in the DNA of the product, and boy have we delivered on that. In this blog post we will cover what’s new in regards to these integrations.

AppsON:

One of the most differentiated features of PowerStore is the AppsON capability available in PowerStore X models, where we load the ESXi hypervisor on each of the two internal PowerStore nodes to allow users to run Virtual Machines directly on PowerStore. You can get a detailed refresher of what AppsON is in this past blog post, but in summary PowerStore with AppsON is:

• The only purpose-built array with a built-in VMware ESXi hypervisor
• It provides two active-active HA nodes with shared storage without limitations (such as needing an external witness)…. All in a small 2U form factor
• Each node exposes local compute for users to run their VMs
• All the same storage services available from the PowerStore OS
• Tightly integrated with VMware

AppsON Use Cases

PowerStore with AppsON was designed to target infrastructure consolidation use cases, and data intensive workloads. AppsON provides small to extra large storage resources, with small to medium compute resources.

From an infrastructure consolidation standpoint, this is the use case AppsON is currently dominating. We are seeing customers extremely attracted by the small 2U footprint, and how it provides two active-active HA ESXi nodes without limitations that typically come in 2 node deployments such as needing an external witness, making it ideal for Edge and ROBO workloads. We are seeing customers transition these sites with a couple of different servers + SAN to a single PowerStoreX appliance resulting in significant infrastructure footprint and higher availability.

• One particular manufacturer with over 30 manufacturing plants transitioned their current infrastructure of 3 ESXi servers, 2 switches, and a SAN to a single PowerStore X appliance to run their mission critical manufacturing app which consists of 4 to 8 VMs. PowerStore with AppsON was selected because of the size and integration simplicity of deploying a single 2U device to run their manufacturing application at each plant. The customer also liked the dual personality nature of PowerStore with AppsON, that apart from allowing VMs to run locally also acts as a regular SAN. This allows them to repurpose their existing servers by attaching them to the PowerStore, protecting their existing investment while modernizing.
• Another Law firm transitioned their satellite offices from 4 ESXi hosts and a VNX SAN to a PowerStore X using AppsON to run VMs that provide local AD services, Windows File services, Windows Print Server, and a SQL data database at these sites. AppsON was selected in order to reduce footprint with an easy to manage, highly available solution that simplified their infrastructure needs at their remote offices​, allowing them to consolidate to a single 2U solution
• Another home appliance retail chain, previously running 3 servers with local direct-attached-storage at each of their 10 stores consolidated to PowerStore X at each site to run their Point-of-Sale system after an outage had occurred with their previous implementation, resulting in sales being halted for multiple days while the system was brought back up.

Customers are also attracted to the standardization on vSphere, allowing them to manage everything with a common skillset.

• In the last example above, the customer selected PowerStore with AppsON for each of their stores in order to give them a reliable system to run their POS system in a highly available way with as small of a form factor as possible. In addition, because of the standardization of vSphere, it allowed their small IT team to use vSphere skillset to manage everything with same experience. ​
• Another example of this is a global software company was looking to standardize on vSphere and move away from 3-tier infrastructure. In their previous state, their edge sites would vary from 3 to 6 ESXi servers with a SAN or using direct-attached-storage to run about 16-20 VMs at each edge site. The infrastructure administration was straining their 4 person systems engineering team due to lack to technology standardization and administration across disparate infrastructures​. By standardizing on VMware based modern technology, it enableed single-pane administration across the infrastructure, using vCenter to manage for all core + edge locations; the Core datacenter is standardized on VxRail, while their edge sites will standardize on PowerStore X with AppsON.

We also have customers using PowerStore X with AppsON for some very interesting data intensive workloads, that don’t huge amount of compute, but rather either lots of storage capacity and/or a mixture of lots of IOPS and low latency.

• A global engineering company standardized their entire datacenter on VxRail (VCF with VxRail, along with regular VxRail). The customer needed a reliable, high performing and consolidated platform to complement VxRail, to run a data intensive application with low compute requirements, but extra large capacity requirements (6 VMs with 26 TB each!) with sub-ms access time requirements. The customer selected PowerStore with AppsON as it could meet these requirements with a single 2U chasis while allowing them to scale up their storage as the solution continues to grow, while being managed from a vSphere environment similar to VxRail.
• A bank wanted to move Some T1 and T2/T3 applications to midrange storage and HCI products so that they can balance performance with cost per TB. ​ As the bank has a big VMware footprint, AppsON  was attractive to optimizes system performance, scalability, and storage efficiency to support any workload without compromise.​ Because of the standardization on vSphere, AppsON allowed the customer to move VMs between VxRail and PowerStore, with those VMs which require more IOPS moved onto PowerStore with AppsON.
• A world renowned sciences university was looking for high-performance storage for a chemistry research project using artificial intelligence, and a healthcare research project in partnership with the hospital to investigate the COVID-19 genome that could be integrated with their existing VMware environment in an easy way. PowerStore with AppsON is now a key factor for their virtual HPC environment, providing 6x array performance improvements while consolidating on 1/3 of the existing space.

New Features:

Lets talk about some of the new features made available in recent code upgrades, including the recent 2.0 release. Note that some of these features are applicable to PowerStore in general, and not specific to PowerStore with AppsON.

AppsON Scale out

Scale-out is now supported with AppsON, allowing more compute and storage within a PowerStore X cluster, allowing for more workloads to be consolidated with AppsON. Up to 4 appliances are supported within a cluster, and the beauty of PowerStore’s scale-out is that the cluster can be made up of completely different model appliances with completely different capacities. In addition, our uniquely awesome vVol design is such that a single stretched vVol container spans across all appliances, presenting a single pool of storage accessible to all nodes simplifying management and load balancing.

Automated VASA Registration
Previously only available with PowerStore X models, PowerStore T now supports automated VASA registration directly from the PowerStore manager, making vVol setup as easy as it can get. Simply add in your vCenter information and press connect, automating VASA provider registration for your PowerStore.

After doing so, status of the connection is displayed:

This makes the PowerStore manager VMware aware, displaying vVol VMs directly in the PowerStore manager, along with the VM details, capacity utilization, VM and storage performance (including at the individual vVol level), alerting, and protection.

You can see your VM compute performance for the VM:

Storage performance for the VM:

And even performance for a specific individual vVol:

As a reminder, PowerStore’s vVol integration was built into the very DNA of the product:

vSphere host awareness
A new capability in PowerStore 2.0 (applies to both T and X models) is that the PowerStore manager becomes vSphere host aware, showing the correlation between PowerStore objects and their corresponding vSphere host. Lets take for example that I have an ESXi host that I have added that I named “My_ESXi” host. When talking to my vSphere admin, or if I am also a vSphere admin, I want to know what this host is known as in vSphere so that I know they are the same.

In the PowerStore manager, I can see the “My_ESXi” is known as “10.245.11.107” in vSphere and is running VMware ESXi 7.0.2, 17867351.

Now lets say I want to see which hosts my VMs that are using up PowerStore vVol storage are running. In the Virtual Machines view in the PowerStore manager, I can see exactly see that my VM called “Windows2019-Base” is running on an ESXi host known as “10.245.17.252” in vSphere.

Multi vCenter support for vVols

We have documented two methods for supporting multiple vCenters with the same VASA provider for using vVols across multiple vCenters. Although not a product limitation, and instead a VMware design (we are working hard with them to improve this) limitation, we have two approved methods for supporting this.

Option 1 is to use Enhanced Linked Mode. The second option is a procedure that involves the manual sharing of vCenter certificates across vCenters. You can find more details here: https://www.dell.com/support/kbdoc/en-us/000186239/powerstore-how-to-register-vasa-provider-with-vcenters-in-different-sso-domains

vSphere ROBO support:

Introduced in PowerStore OS version 1.0.3, in addition to vSphere Enterprise Plus, AppsON now supports vSphere ROBO (Advanced and Enterprise) licenses. vSphere ROBO licenses support a maximum of 25 virtual machines. While it is still recommended to use a vSphere Enterprise Plus license, ROBO is perfect for small environments. However, note that some features are not supported by vSphere ROBO Advanced and Enterprise licenses, such as DRS:

• vSphere ROBO Advanced does not supported DRS, and vSphere ROBO Enterprise only supports DRS for entering maintenance mode.

Bringing in an External Host into the PowerStore X vSphere Cluster

One of the unique capabilities of PowerStore X with AppsON is that it is dual personality: It allows VMs to run locally on the local ESXi nodes while at the same time also acting like a regular SAN in providing storage to external servers. Need a more compute in your vSphere AppsON cluster? Want to re-use existing servers from previous investments? No problem! As you can see here I have a two appliance PowerStore X cluster (4 hosts; 10.245.17.120-121, 10.245.17.252-253), and I have brought in my existing ESXi server called 10.245.11.107. This now provides additional compute to my vSphere cluster, and providing additional load balancing and HA if needed. This previously required an RPQ, but this is no longer needed. See the Virtualization Infrastructure Guide for more details.

Registering a host from vCenter
One of my favorite new features in the VSI plugin, is the ability to automagically “push” and register ESXi host from vCenter to the PowerStore manager. When dealing with many hosts, this is a HUGE time saver. Go to the VSI plugin in vCenter, click Storage Systems>Hosts>Add. This is applicable to all PowerStore models (T and X).

In this example I am registered my ESXi host known as 10.245.11.107 in vSphere to the PowerStore manager.

And can now see it in the PowerStore manager:

The post Dell EMC PowerStore, Integrating with VMware vSphere appeared first on Itzikr's Blog.

In the first part of the series, we provided a high-level overview of the PowerFlex 3.6 release. You can read all about it here.

and in the second post, we covered the integration with VMware SRM

Now, let’s dive deeper into the integration of PowerFlex 3.6 with AppSync

Data centers these days, contain a lot of redundant data, IDC forecast that 60% of the data is made of copies.

To improve organizational agility and competitiveness, more copies, more frequently, with more operational self-service across the process cycles are required. Indeed, IDC estimates >60% of enterprise storage is used by such non-production copies.

So, what do customers use data copies for? The three most commons use cases are:

AppSync is a software that enables Integrated Copy Data Management (iCDM) with Dell EMC’s primary storage systems.
The solution provides:

• Automated copy creation and consumption – no more manual steps or custom scripts
  
  
• Tight integration with host environments, applications and Dell EMC PowerFlex storage and replication
  
  
• Applications owners and storage admins are on the same page with a transparent copy workflow
  

AppSync simplifies and automates the process of generating and consuming copies of production data. By abstracting the underlying storage and replication technologies, and through deep application integration, AppSync empowers application owners to satisfy copy demand for operational recovery and data repurposing on their own. In turn, storage administrators need only be concerned with initial setup and policy management, resulting in an agile, frictionless environment.

AppSync automatically discovers application databases, learns the database structure, and maps it through the Virtualization Layer to the underlying storage LUN. It then orchestrates all the activities required from copy creation and validation through mounting at the target host and launching or recovering the application. Supported workflows also include refresh, expire, and restore production.

AppSync simplifies and automates the process of generating and consuming copies of production data. AppSync enables application owners to satisfy copy demand for data repurposing, operational recovery and disaster recovery across multiple Dell EMC arrays and applications with a single user interface.

Dell EMC AppSync for PowerFlex provides a single user interface that simplifies, orchestrates and automates the process of generating and consuming DevOps data across all enterprise database applications deployed on PowerFlex.

AppSync for PowerFlex provides simple automated copy creation and consumption, eliminating manual steps or custom scripts. AppSync integrates tightly with host environments and database applications including, but not limited to, Oracle and SQL Server. With AppSync, applications owners, database administrators, and storage administrators get on – and stay on – the same page through a transparent copy workflow.

Dell EMC AppSync for PowerFlex allows you to protect, restore and repurpose application data, satisfying any DevOps requirements.

AppSync version 4.3 enables support for the PowerFlex family – rack, appliance, and ready node consumption options.

AppSync architecture

The architecture of AppSync has three major components:

• AppSync server is deployed on a Windows server system, either physical or virtual. It controls all workflow activities, manages the alerting and monitoring aspects, and persists internal data in a PostgreSQL database.
  

• AppSync host plug-ins are installed on all source and mount hosts. They provide integration with the operating systems and the applications that are hosted on the hosts. These applications include Microsoft Exchange, Microsoft SQL Server®, Oracle®, SAP HANA, and VMware datastores or other file systems. With VMware datastore replication, there is no host plug-in because AppSync communicates directly with the VMware vCenter® server.
  

• AppSync user interface is the web-based UI for the AppSync copy-management feature. AppSync can also be managed using the vSphere VSI plug-in, REST API, or command-line interface (CLI).
  

Registering the PowerFlex system with AppSync

AppSync interacts with the PowerFlex system by communicating with PowerFlex Gateway using API calls:

On the AppSync console, select Settings > Infrastructure Resources > STORAGE SYSTEMS.  Click ADD SYSTEMS.

Under Select System Type, choose PowerFlex.

Enter the PowerFlex Gateway IP and credentials to configure the storage system.

Review the configurations in the Summary page and click FINISH to register the PowerFlex system.

AppSync service plans

AppSync provides intuitive workflows to set up protection and repurposing jobs (called Service Plans) that provide end-to-end automation of all the steps from application discovery and storage mapping to mounting copies to the target hosts. Service plans can be scheduled with alert emails to easily track their status. AppSync also provides an application protection monitoring and reporting service that generates alerts if SLAs are not met or if a service plan fails.

AppSync supports three types of service plans:

• Bronze — You can use the Bronze service plan to create local copies of your applications data
  
• Silver — You can use the Silver service plan to create remote copies of your applications data
  
• Gold — You can use the Gold service plan to create both local and remote copies of your applications data
  

AppSync features

AppSync Protect

AppSync enables application owners and DBAs to protect, restore. and repurpose their data to satisfy their unique copy requirements. This accelerates and improves processes like test and dev by providing the latest production data for high quality product releases. AppSync’s support for second generation copies (a copy of a copy) allows for required data masking, filtering and obfuscation by DBAs so that end-users of data have access to only the data that they need. At any given point of time, storage admins can get a complete picture of the copy landscape so that they are aware of capacity utilization and the scope for optimization.

AppSync repurposing 

AppSync allows you to create copies of your database and file systems for application testing and validation, test and development, reporting, data masking, and data analytics. AppSync identifies copies that are created from a repurpose action as first-generation and second-generation copies. The source of a second-generation copy is a first-generation copy. You can create multiple second-generation copies from a first-generation copy.

AppSync support for PowerFlex

• Discovery of PowerFlex Nodes and clusters
  
  • 3.x versions of code supported
    
  • AppSync works with any PowerFlex hardware (rack, appliance)
    
  • AppSync server can be located on any Windows physical box or virtual machine
    

• AppSync supports both Bronze Service Plans and local repurposing
  
• AppSync supports Silver Service Plans and remote repurposing
  
  • Utilizes Asynchronous replication
    
  • Not supported for Microsoft applications
    

PowerFlex licensing in AppSync

• Advanced license only, tied to PowerFlex installation ID
  
• Based on total raw capacity of PowerFlex system
  
• Warning and non-compliance alerts shown if licensed capacity is lower than total raw capacity of the cluster
  
• AppSync 90-day free trial can be used before purchase
  

Dell EMC PowerFlex and AppSync integration Demo

Integrated copy data management with AppSync for SQL Server running on PowerFlex

Integrated copy data management with AppSync for Oracle running on PowerFlex

Integrated copy data management with AppSync for File Systems running on PowerFlex

You can access the AppSync downloads by, clicking the screenshot below

https://dl.dell.com/downloads/DL105036_AppSync-4.3.0.0-(Build-Number%C2%A0PR-4.3.0.0-38)-Software.zip

A guest post by Tomer Nahumi

The post DELL EMC PowerFlex 3.6 – Part3, copy data management with AppSync for applications running on Dell EMC PowerFlex appeared first on Itzikr's Blog.

With the release of vCenter 7.0 U2a, VMware has introduced VM Service. VM Service runs on top of vSphere with Tanzu and allows developers to deploy Virtual Machines using kubectl declarative object configuration while simultaneously allowing the IT administrator to govern resource consumption and service availability.

This blog post will be focusing on the new vSphere with Tanzu VM Service feature and what benefits do we get from running it on PowerStore VVOLs

VM Service allows DevOps engineers to deploy and run Virtual Machines in a shared Kubernetes environment. Virtual Machines are deployed using kubectl, like containers, directly to Supervisor Namespaces.

How does it work?

VM Service is made up of two components, a vSphere side component and a Kubernetes side component.

The vSphere side is built right into vCenter, it allows you to manage VM Images (Content Libraries) and VM Classes (VM sizing), the Kubernetes side is called VM Operator which creates and services the Kubernetes Custom Resources (CRs/CRDs), which we’ll get into later, and tells K8s how to talk to vSphere.

One of the new features that was added in vSphere 7.0 is the ability to provision Virtual Volumes (vVols) to back Kubernetes Persistent Volumes (PVs) via the updated version of the vSphere Container Storage Interface (CSI) driver.

PowerStore vVols leverage Storage Policy Based Management (SPBM) to ensure VMs have the appropriate storage capabilities through their entire life cycle. VM storage policies can be optionally created after the storage provider is registered. These policies are used to determine the desired storage capabilities when a VM is being provisioned.

vVol datatores can be added to the vSphere Kubernetes namespace as SPBM policies by clicking on the edit storage button

Like pods and persistent volumes provisioning, these Storage Policy Based Management that represent storage classes at the Kubernetes level, can be used for virtual machines deployment using the VM service.

The vSphere side of the VM Service is very straight forward, it is created as part of your vSphere with Tanzu namespaces once you enable the service in the new Workload Management “Services”

Inside the VM Service, you’ll find where you can manage the components, namely the VM Images (Content Libraries) and VM Classes (t-shirt sizes).

From Kubernetes perspective, we have the new VM Operator, The VM Operator creates Kubernetes CRDs for a few different types that allows K8s users to discover and define the resources they need from the Operator, the most important types they would be interested in are VirtualMachine, VirtualMachineImage, VirtualMachineClasses and VirtualMachineServices.

Virtual Machines in vSphere with Tanzu Namespaces the templates have to be deployed from a content library. VMware provides supported Images in their Marketplace. So we can upload their to the vCenter content library

As I’ve mentioned before, in vSphere with Tanzu, VMware SPBM policies represent K8s storage classes which can be used to create persistent volumes using the VMware CNS.

To create virtual machines using the Kubernetes API, we use, of course, yaml files –

As for the storage class name, we specify the VVOL SPBM policy, which automatically creates virtual volumes for each virtual machine file on the PowerStore array.

To deploy Virtual Machines, it required to know in which networking mode the Supervisor Cluster is running. VM Service supports both modes, NSX-T and Distributed Portgroups. When using NSX-T, VM Operator will automatically create the required network segments and attach virtual machine interfaces. In that case, you just set the networkType to nsx-t.

We create the VM by running the kubectl apply command and specifying the YAML files, within a few seconds, we can find the new VM up and running in the vSphere web client.

If we navigate to PowerStore manager and click on the virtual machines tab, we can see that PowerStore automatically detected the newly created virtual machines that we’ve just deploying using the new Tanzu VM service.

We can see useful information about the VM such as the managing ESXi host, the guest OS, the capacity and the protection policy, and it doesn’t stop there, PowerStore collects and displays both history and live performance metrics about the virtual machine, allowing the storage admin to get full visibility of the virtual machines running on the PowerStore storage array.

The type of vVol provisioned depends on the type of data that is being stored:

• Config: Stores standard VM configuration data such as .vmx files, logs, and NVRAM

• Data vvols: Stores data such as VMDKs, snapshots, full clones, and fast clones

• Swap: Stores a copy of the VM memory pages when the VM is powered on. Swap vVols are automatically created and deleted when VMs are powered on and off.

In addition to the compute performance, we provide storage performance metrics showing the total IOPS, bandwidth, latency and IO size of the VM, and if that’s not enough, because each vmdk file is a virtual volume at the array level, we have a more granular view of the performance of each and every disk of the VM.

If we navigate to a specific VM and select one of its disks, we can get live performance metrics of that disk, allowing the storage administrator go get full visibility of the workloads running in his data center, no matter if these are regular virtual machines, virtual machines managed by the VM service, Supervisor Kubernetes pods, and Tanzu guest cluster workloads, all, from a single pane of glass.

Below, you can see a demo showing how it all work:

A guest post by Tomer Nahumi

The post VMware vSphere with Tanzu VM service on DELL EMC PowerStore vVOLs appeared first on Itzikr's Blog.

I blogged about RP4VMs many times in the past, if you are new to this software based replication engine, I highly suggest you start by reading about it here and here

RecoverPoint for VMs is a hypervisor-based, software-only data protection solution for protecting VMware® virtual machines.

RecoverPoint for VMs provides both local and remote replication that enables recovery to any point in time. RecoverPoint for

VMs also supports replication to the Amazon cloud. RecoverPoint for VMs consists of the following:

• RecoverPoint for VMs write-splitter that is embedded in the ESXi hypervisor and enables replication from any storage type to any type of storage.
  
  
• Virtual RecoverPoint appliances (vRPAs) that manage data replication. A vRPA cluster is a group of vRPAs that works together to replicate and protect data.
  
  
• A journal that comprises one or more volumes that are dedicated on the storage at each copy in a system to hold the production data history.
  
  
• vSphere Client plugin that is used for managing VM replication:
  
  
  • The HTML5 plugin is introduced in RecoverPoint for VMs 5.3 to support vSphere 6.7 U1 and later environments. The HTML5 plugin communicates with vRPA clusters through a separate, dedicated HTML5 plugin server.
    
  • The Flex plugin continues to support environments that use vSphere 6.7 and earlier versions. The Flex plugin communicates directly with the vRPA clusters.
    

What’s new in the 5.3 SP2 release:

• Secure boot support

• Support for UEFI secure boot is introduced in RP4VMs 5.3.2
• Prior to RP4VMs 5.3.2, secure boot must be disabled before RP4VMs deployment and must not be enabled post deployment
• Splitter and JAM VIBs are signed in RP4VMs 5.3.2 and later
• Splitter and JAM VIBs must be at 5.3.2 or later before secure boot can be enabled
• The 5.3.2 JAM and splitter VIBs feature reduced footprint on /bootbank and /opt
• H5 plugin enhancements
  • Enhanced dashboard

  • System limits

  • Monitoring – Event log and components

  • Ability to use an existing replica VM when adding a copy
  • Control transfer upon failover for group sets
• New RESTful API Enhancements
  • System limits
  • Existing replica for add copy
  • Monitoring
• vSphere 7.0U2 Support and NSX-T 3.1 Support
• Plugin Server NTP Configuration

  New option in the plugin server OVA deployment to configure NTP server

  Simplifies configuration of time sync via NTP

  Reduces certificate and other issues due to time difference between vRPAs, plugin server and vCenter

• Splitter Upgrade Tool enhancements

You can see a demo of the new version, below

and you can download the new version, by clicking the screenshot below

The post Dell EMC RecoverPoint for VMs 5.3.2 Is Here – What’s New appeared first on Itzikr's Blog.

@August the 2^nd, 2021, we had the pleasure to discuss all sort of modern data centers topics @Tech Field Day, here’s the link to the first day recording.

Modern Infrastructure for the Data Era with Dell Technologies

Dell Technologies empowers your IT organization to evolve – so you can become an innovation engine for your business. In this session, Shannon introduces our industry-leading and award-winning primary storage and hyperconverged infrastructure (HCI). Presented by Shannon Champion, VP Product Marketing, Dell Technologies.

Dell EMC PowerStore Overview

This session will present an overview of PowerStore – including the three pillars of data centric architecture, intelligence and adaptability. We will introduce the main topics and go into the details in following sessions. Presented by Avishek Kumar, Director, Product Management, Midrange & Entry Storage, Dell Technologies.

Dell EMC PowerStoreOS 2.0

Overview of the enterprise innovation, increased intelligence and cost effective entry solution that were made available as part of the PowerStoreOS 2.0 release. Presented by Avishek Kumar, Director, Product Management, Midrange & Entry Storage, Dell Technologies.

VMware DNA at the Heart of a Storage Solution: Dell EMC PowerStore

In this session, we will review how PowerStore provides a modern truly integrated VMware storage platform that provides end-to-end visibility with VMware environments, a turn-key vVol experience, and VM mobility between HCI, external servers, and storage. We will showcase how PowerStore is the most integrated purpose built array with VMware and drives customer realized improvements in application performance, efficiency and simplifies management. Presented by Marco Abela, Consultant, Product Manager, Dell Technologies.

Dell EMC PowerStore Data Mobility

In this session we will cover the data mobility capabilities of PowerStore that enable disaster recovery and business continuity, such as snapshots and replication. In addition, we will demonstrate the advanced data mobility capabilities of PowerStore metro node for the highest levels of availability, while discussing typical use cases to move data within a data center or between sites without application interruption. Presented by Chuck Farah, Consultant, Product Management, and Marco Abela, Consultant, Product Manager, Dell Technologies.

Multi-Appliance Dell EMC PowerStore Clusters: Interactive Demo

In this session we will showcase the intelligent scale-out and import capabilities of PowerStore. An interactive demonstration will highlight the simplicity and ease-of-use built into PowerStore. Presented by Ethan Stokes, Senior Engineering Technologist, Dell Technologies.

Microsoft SQL Server 2019 Big Data Clusters on Kubernetes with Dell EMC PowerStore

Deploying Microsoft SQL Server 2019 Big Data Clusters on Kubernetes can be a daunting experience for even the most seasoned SQL Server professional.  Learn how Dell Technologies is leveraging PowerStore and partnering with Microsoft to smooth the transition to the modern data estate. Presented by Doug Bernhardt, Sr. Principal Engineering Technologist, Dell Technologie

Dell Technologies APEX: Block Data Storage Services

APEX Data Storage Services is an as-a-Service portfolio of scalable and elastic storage resources that is focused on business outcomes. In this session we will highlight its capabilities and demonstrate how the APEX cloud console makes achieving these outcomes simple. Presented by Rob Hunsaker, Senior Advisor Product Manager, Dell Technologies.

Dell EMC CloudIQ Presentation and Demo

This session will show how CloudIQ enables you to proactively reduce risk, intelligently plan ahead, and improve productivity.  The session will include a demonstration, highlighting the advanced analytics powered by Machine Learning, and features recently added to CloudIQ to expand its capabilities and value. Presented by Susan Sharpe, Senior Consultant Product Manager, Dell Technologies.

Mission Critical Applications and Their Requirements on Dell EMC PowerMax

During the session what it means to be mission critical and the different types of businesses these applications are found in will be presented. The key requirements of these applications, resiliency/availability, consistent performance, and data security at scale will also be discussed. Presented by John Madden, Engineering Technologist, Dell Technologies.

Dell EMC PowerMax Technology Highlights

This session will present some of the key technical differentiators that create value for PowerMax customers. Features reviewed will include low response times, global cache, non-disruptive upgrades, D@RE, data reduction, snaps and advanced snap use cases, SRDF, and simple management. Presented by Vince Westin, Technical Evangelist, Dell Technologies.

Dell EMC PowerMax SRDF MetroDR Demonstration

SRDF/MetroDR is a two-region high available (HA) disaster recovery (DR) solution that integrates SRDF/Metro and SRDF/A. The demo will show how with a few clicks the user can protect a storage group. Presented by Drew Tonnesen, Technical Staff, Dell Technologies.

Ansible Automation for Dell EMC PowerMax Demonstration

Demonstrating Ansible Playbooks for Dell EMC PowerMax: This demo will show how Ansible modules check for the desired final state and only perform those actions which have not been completed making it an ideal solution for automation. Presented by Drew Tonnesen, Technical Staff, Dell Technologies.

Dell EMC PowerMax Cloud Mobility Demonstration

The demo will create policy-based, automated snapshots for archiving and long-term retention to both the public and private cloud.  Plus show how to recover individual files and entire volumes from cloud snapshots with Cloud Mobility for Dell EMC Storage vApp. Presented by Kevin Vaillancourt, Sr. Principal Engineering Technologist, Dell Technologies.

The post The Dell EMC Sessions At Tech Field Day 2021. Day1 – PowerStore, PowerMax & CloudIQ appeared first on Itzikr's Blog.

@August the 3^nd, 2021, we had the pleasure to discuss all sort of modern data centers topics @Tech Field Day, here’s the link to the first day recording.

The Dell Technologies VxRail Advantage

VxRail HCI System Software delivers a seamless, automated, operational experience to keep your infrastructure in a continuously validated state. Presented by Ash McCarty, Sr. Consultant, Product Management, Dell Technologies.

Dell Technologies VxRail LCM and vLCM: Better Together

Dive into how VxRail continues to differentiate our unique lifecycle management experience. Presented by Daniel Chiu, Sr. Principal Engineer, VxRail Technical Marketing, Dell Technologies.

Dell Technologies VxRail Configuration and Deployment

Deploy on your schedule. See the VxRail configuration portal up close, to validate, orchestrate and automate cluster deployment. Presented by Curtis Edwards, Technical Staff – VxRail Engineering Technologist, and Paul Galjan, Sr. Consultant Product Manager, Dell Technologies.

Dell Technologies VxRail Management Overview

Learn more about the management tools that makes managing your infrastructure easier. Presented by Jeremy Merrill, Technical Staff – VxRail Engineering Technologist, and Michael Gordon, Sr. Consultant Product Manager, Dell Technologies.

Dell Technologies VxRail Customer Presentation: New Belgium Brewing

Adam Little from New Belgium Brewing presents his search for a new infrastructure solution and the reasons his company selected Dell Technologies VxRail.

Dell EMC VxRail Dynamic Nodes: Flexibility for the Future

Find out what all the buzz is about for the NEW VxRail dynamic nodes. Presented by Don Pisinski, Consultant Product Manager, and Daniel Chiu, Sr. Principal Engineer, VxRail Technical Marketing, Dell Technologies.

What Does Seamless Technology Integration Mean for Dell EMC VxRail?

A look at the testing, engineering and validation enabling users to easily move between generations of hardware and integrate the latest technology on VxRail. Presented by David Glynn, Sr. Principal Engineer, VxRail Technical Marketing, and Bill Leslie, Sr. Manager, Technical Marketing

Engineering, Dell Technologies.

NVIDIA-Certified Dell EMC VxRail for GPU-Intensive Workloads

Learn how NVIDIA Certified Systems on VxRail HCI makes adopting and deploying GPU’s easier. Presented by Ash McCarty, Sr. Consultant, Product Management, Dell Technologies, and Bob Crovella, Solution Architect, NVIDIA.

Get the Most Out Of Your K8S with Tanzu on Dell EMC VxRail

Tanzu on VxRail helps make developing your cloud native strategy easy by leveraging consistent infrastructure and operations to support faster application development, scalability, and lifecycle management to ensure you are using the latest Kubernetes tools and features. In this session Jason Marques, Dell Technologies Sr. Principal Engineering Technologist, and Chris Catano, VMware Technical Program Director, discuss the unique benefits of Tanzu on VxRail and solutions to accelerate your Kubernetes journey across core, edge, and cloud. Presented by Jason Marques, Sr. Principal Engineer, VxRail Technical Marketing, Dell Technologies and Chris Catano, DSAT Director Technical Enablement and Activation, VMware.

VMware Cloud Foundation and Tanzu on Dell EMC VxRail

VxRail is the only jointly engineered HCI system with deep VMware Cloud Foundation integration, delivering a simple and direct path to modern apps and the hybrid cloud with one, complete automated platform. Watch Jason Marques, Sr. Principal Engineering Technologist, explain the benefits of VMware Cloud Foundation on VxRail and how it supports native Kubernetes workloads and management as well as traditional VM-based workloads. Next, Chris Catano, Vmware Technical Program Director, shows you how easy it is to get started with Tanzu on VxRail. Presented by Jason Marques, Sr. Principal Engineer, VxRail Technical Marketing, Dell Technologies and Chris Catano, DSAT Director Technical Enablement and Activation, VMware.

The post The Dell EMC Sessions At Tech Field Day 2021. Day2 – vXRail, LCM, GPUs, Tanzu & VCF appeared first on Itzikr's Blog.

In the previous post, we provided an overview of the PowerFlex integration with AppSync 4.3. You can read all about it here.

This post will be focusing on the AppSync repurposing feature for database and file systems.

Dell EMC AppSync enables integrated copy data management (iCDM) with Dell EMC primary storage systems. AppSync simplifies and automates the process of generating and consuming copies of production data. By abstracting the underlying storage and replication technologies, and through deep application integration, AppSync empowers application owners to satisfy copy demands for operational recovery and data repurposing. In turn, storage administrators must only be concerned with initial setup and policy definition management, resulting in an agile, frictionless environment. AppSync automatically discovers the application, analyzes the layout structure, and maps it through the virtualization layer to the underlying storage device. AppSync orchestrates all the activities required from copy creation and validation through mounting, at the target host, and launching, or recovering, the application copy. The supported workflows also include refresh, expire, and restore to production

AppSync allows you to create copies of your database and file systems for application testing and validation, test and development, reporting, data masking, and data analytics. AppSync identifies copies that are created from a repurpose action as first generation and second-generation copies. The source of a second-generation copy is a first-generation copy. You can create multiple second-generation copies from a first-generation copy. AppSync supports repurposing on File systems, SQL Server and Oracle databases.

There are two types of repurposing:

● Native array repurposing – The first-generation copy is a copy of the source database. For example, in the case of an PowerFlex array, snapshot of the source is the first-generation copy.

● RecoverPoint bookmark repurposing – The first-generation copy is a copy of the LUNs at the local or remote replication sites in the RecoverPoint consistency groups.

AppSync repurposing supports several types of workflows, such as creating application-consistent local or remote copies, automating mounting and application-recovery scenarios, and scheduling these operations.

Repurposing workflows focus on a single application at a time, and do not use a copy-count-rotation policy, which occurs with service plan workflows. Repurposing workflows are generally not used to protect the application or are considered a backup solution. Repurposing is most often used to replicate production environments and use the copies for quality assurance testing, development, offloading reporting, and patch management. For more details about the repurposing workflow, see the AppSync User and Administration Guide.

Repurposing workflows offer multigeneration copies. A first-generation copy is one copy removed from the source, and it can be optionally copied to a second-generation copy, or a copy that is twice removed from the source.

First-generation copy:

• Copy of production
  
• Can be application consistent
  
• Can be used as a restore point in time
  

Second-generation copy:

• Copy of a copy with first generation as the source
  
• Fast creation time
  
• No application integration
  
• Cannot be used as a restore point
  

Repurposing use cases:

The following are typical repurposing use cases with AppSync. This is not an exhaustive list, but it can help you identify some possible workflows.

• On-demand copies: This is a copy of a single application that is used for an extended time and then discarded, maintaining copy retention. The discarded copy is not used for backup purposes to restore from. Copies can be used for performing patch-management testing, performance tuning against nonproduction environments, or offloading reporting. This practice reduces the amount of I/O that is performed against a production environment.
  
• Data masking: A first-generation copy is created and mounted for sensitive data to be masked. When the sensitive data is masked, the copy is unmounted to create a second-generation copy. This second-generation copy, which has the sensitive data removed, can then be used.
  
• Remote copy retention: This provides long-term retention on a remote appliance, sometimes identified as a disaster recovery copy, which can be accomplished using repurposing. Remote copies can be created using the array native replication sessions. AppSync supports using the silver and gold service plans with native replication for long-term retention on the remote appliance.
  
• Copy-of-copy: This is similar to a data-masking requirement. Repurposing supports creating a first generation, application-consistent copy of a single application, which is used as the source for multiple second-generation copies. These second-generation copies can be used for purposes such as the following:
  

  – Providing multiple copies of the same point-in-time (PIT) to developers; these are identical copies for training purposes or as a baseline for collaboration efforts
  

  – Alleviating the need to quiesce the production environment unnecessarily for many copies.
  

  – Refreshing the second-generation copy while not having to change the PIT
  

Repurposing considerations

The following are typical repurposing considerations to make with AppSync:

• Repurposed copies are used primarily for testing or development for extended periods of time and are discarded or expired when the process is complete.
  
• Repurposed copies do not figure into RPO calculations (refer to the AppSync User and Administration Guide for more details regarding RPO alerts).
  
• Restores are supported from the first-generation copies only.
  
• Second-generation copies are not application consistent. There is no application discovery, mapping, or application integration such as freeze and thaw of a database, and callouts are supported for unmounting only. See the AppSync User and Administration Guide for more details about callout scripts.
  

Protecting Microsoft SQL Server Databases

Microsoft SQL Server 2012 and beyond includes native capabilities for creating sync and async replicas via AlwaysOn Availability Groups. Availability replicas provide high availability, scale out readable copies, disaster recovery and migration and upgrade capabilities. The SQL Server availability features do not replace the requirement to have a robust, well tested backup and restore strategy, the most fundamental building block of any availability solution.

While many methods of data protection exist, one of the simplest and most-effective methods involves using snapshots. Snapshots allow recovery of data by rolling back to an older point-in-time or copying select data from the snapshot. Snapshots continue to be an essential data-protection mechanism that is used across a wide variety of industries and use cases. Snapshots can preserve the most important mission-critical production data, sometimes integrating with other data protection technologies. For versions of Microsoft SQL Server running on Microsoft Windows Server, Dell EMC AppSync can leverage the Volume Shadow Copy Service (VSS) to take an application consistent snapshot of the databases. This method of snapshot mimics the process of a full backup and can be used as such.

Automating & Orchestrating Protection

Integrated copy data management (iCDM) strategy is important to the overall return on investment for any storage solution. Dell EMC PowerFlex data services provide the foundation for the ability to protect the production Microsoft SQL Server workloads while also easily repurposing the database copies.

iCDM allows the DBA and Application owners peace of mind with policy driven protection and ad-hoc copies as needed without needing to involve or wait for the storage admin.

Service plans within Dell EMC AppSync, our Integrated Copy Data Management (iCDM) software, provide a copy management workflow for protecting applications and are able to make copies locally, remotely, or both. In addition, repurposing workflows provide multi-generational copy processes.

Copies on demand

Integrated Copy Data Management enables rich application workflows making a development copy, instantly refreshing it to the latest prod data, pushing the dev copy to QA, pushing the QA copy to a scalability test bed, and rolling the output back into prod.

For analytics processes, production data can be extracted and pushed to all downstream analytics applications on demand. For operations, copies from production can be made when needed to pre-production testing, run simulations or test the latest patches. iCDM operations do not affect other copies or risk their SLAs.

Here you can see a demo showing how it works:

In conclusion, AppSync integration enables PowerFlex users to protect, restore and repurpose their data to satisfy their unique copy requirements for their enterprise applications users.

AppSync and PowerFlex appliances can integrate to create copies of application data. Coupled together, they simplify and automate the process of generating and consuming copies of production data. This ability can significantly improve overall efficiency in enterprise environments.

A guest post by Tomer Nahumi

The post Repurposing Application Copies with AppSync 4.3 for Applications Running on Dell EMC PowerFlex appeared first on Itzikr's Blog.

We have just released our latest (9.0) version of our VMware vCenter plugin.
What is VSI?
Dell EMC Virtual Storage Integrator is a plug-in that extends the VMware vSphere Client, allowing users of the vSphere Client to perform Dell EMC storage management tasks within the same tool they use to manage their VMware virtualized environment.
If you are new to the VSI plugin, i suggest you start by reading about it here What is Dell Technologies PowerStore – Part 17, Managing the array via VMware vCenter with the VSI 8.4 Plugin | Itzikr’s Blog (volumes.blog)
here Manage your Dell PowerStore array from VMware vCenter, with VSI 8.5 | Itzikr’s Blog (volumes.blog) and here Virtual Storage Integrator (VSI) 8.6 is Here! – Itzikr’s Blog (volumes.blog)
What’s new in 9.0?

Multiple Storage Owners

– Allow Multiple Owners of Storage Systems.

Under a resource the Owners are denoted by the (Managed By) tag.

– All owners can be viewed under system details or main storage system
grid by selected the ” ” icon.

By selecting a storage system and clicking the ” ” icon you can open
the edit storage system dialog and add multiple owners.
– Select or unselect owners of the storage system and hit the “Save”
button to update the storage owners.

Ÿ
Upgrade VSI through UI – Upload VSI Upgrade Packages

2. Execute VSI Upgrade Packages
3. Remove VSI Upgrade Packages
(Will be applicable for 9.0 to 9.x upgrades)

Login to the VSI Management Interface at “https://<vsi-ip>&#8221;
– Click on the “Upgrade” tab to view, import execute and delete upgrade
packages.
– Older/previously installed packages will be show with a warning icon
and will not be able to execute upgrades using these packages again

Click the import button ” ” to open the import dialog.
– Click the “Browse” button and select the upgrade package.
– Click the “Import” button and wait for the file to upload to complete.
This can take several minutes to upload based on network bandwidth.

VSI 9.0 Upgrade VSI through UI: Execute Upgrade

– Select the upgrade package and click the “Execute” button ” ” to
open the execute dialog.
– Click the “Upgrade” button to kick off the upgrade with the selected
package.

Ÿ
PowerFlex Registration and Management – View Registered PowerFlex Systems
2. Register PowerFlex System
3. Un-Register PowerFlex System
4. Edit PowerFlex System Registration
5. View PowerFlex pool
6. Register Storage Pools to VSI
7. Unregister Storage Pools from VSI
8. Create VMFS Datastore(s)
9. View VMFS Datastore Storage Details
10. Edit VMFS Datastore
11. Delete VMFS Datastore
12. Host and Cluster Storage Viewers
13. Create RDM Disk
14. View RDM Disk Back End Storage Details
15. Delete RDM Disk
16. Host Management

Ÿ
Unity Hosts Management – Add and Remove Unity Hosts.
2. Support Add/Remove host for linked mode vCenters.
3. Add/Remove Hosts and register VASA
provider during storage registration.
4. Edit Vasa Provider Registration for Unity.

Ÿ
PowerStore NVMe/FC Support – Add/View/Remove NVMe hosts
2. Create VMFS datastore with NVMe hosts
3. Clone/Refresh/Restore datastore with NVMe hosts
4. RDM disk is not supported on NVMe hosts

Below, you can see a demo, showing the new, PowerFlex magement capabilities

You can download the 9.0, OVA package by clicking, the screenshot below

The post Virtual Storage Integrator (VSI) 9.0 is here appeared first on Itzikr's Blog.

Dell EMC PowerProtect Data Manager provides software defined data protection, automated discovery, deduplication, operational agility, self-service and IT governance for physical, virtual and cloud environments.

• Orchestrate protection directly through an intuitive interface or empower data owners to perform self-service backup and restore operations from their native applications
  
• Unique VMware protection: Protect VMware VMs without business disruption
  
• Ensure compliance and meet even the strictest of service level objectives
  
• Leverage your existing Dell EMC PowerProtect appliances
  

PowerProtect Data Manager gives you valuable insight into protected on-premises and in-cloud workloads, applications, file systems, and virtual machines. Designed with operational simplicity and agility in mind, Data Manager enables the protection of traditional workloads including Oracle, Exchange, SQL, SAP HANA and file systems as well as Kubernetes containers and virtual environments.

Network attached storage (NAS) is an IP-based file-sharing storage device which is attached to a local area network (LAN). NAS can serve a variety of clients and servers over an IP network. A NAS device uses its own operating system and integrated hardware and software to deliver a range of file-service needs. NAS is widely used for its simplicity, ease of use and outstanding performance. With its simple use comes challenges regarding data protection. For years, NAS and backup vendors have used the NDMP protocol to protect NAS data. The NDMP protocol has its own limitations, such as manual slicing of a NAS share to achieve multi-stream backup, limited parallel streams, and periodic full backups. Customers also face
challenges to protect their growing amounts of data and back up this data within their specified backup windows.
PowerProtect Data Manager for NAS protection addresses today’s customer challenges of protecting evolving NAS environments. Unlike NDMP-based solutions, dynamic NAS protection is a NAS-vendor-agnostic solution. With dynamic NAS protection, customers can overcome the challenges with the NDMP protocol. Protecting NAS assets with Data Manager is a non-NDMP solution. Dynamic NAS protection uses the NAS Protection Engine for backup and recovery orchestration. This solution is easy to use, providing automatic discovery, orchestration, and management through the Data Manager UI. With its snapshot technology and intelligent slicing, Data Manager protects NAS data efficiently within the required backup window.

This solution addresses some of the challenges to dynamic NAS protection with the following capabilities:
• Vendor-agnostic solution for NAS protection
• Forever incremental backup and no periodic full
• High number of parallel streams and multiple virtual containers to address scale and performance
• Index, search, and restore
• Restore to any NAS device, such as NFS/CIFS

With PowerProtect Data Manager 19.9 , You have the ability to restore data on-premises or in the cloud, and you can do that from native interfaces. And, centralized governance control ensures IT compliance, making even the strictest service level objectives obtainable. You are enabled with additional operational efficiencies such as self-service, automated discovery, SaaS-monitoring and reporting, and easy deployments, upgrades and scale. And only with PowerProtect Data Manager can you ensure availability of all your VMs at scale without business disruption.

In this post, I’ll introduce Dell EMC PoweProtect 19.9, Part1 – Dynamic NAS Protection, a new feature within Dell EMC PowerProtect Data Manager 19.9, which changes the way we protect our NAS infrastructure to ensure availability of all your NAS systems at scale without business disruption.

A new feature of PowerProtect Data Manager is Dynamic NAS Protection. Dynamic NAS Protection automates and optimizes the protection of your NAS infrastructure, delivering a simple, modern way to protect your NAS systems.

Protect and recover any NAS that supports NFS or CIFS, including Dell EMC PowerStore, PowerScale and Unity. NAS assets are automatically discovered through communication with the NAS array.

Dynamic NAS Protection efficiently runs backups in parallel multi-streams by automatically and dynamically slicing NAS assets with load balancing for movement to protection storage. The slicer can be used to partition a NAS share, a filesystem, or a volume with multiple files and folders. Slicing is tunable from the PowerProtect Data Manager UI and the Data Manager REST API with support for both full and incremental backups.

The Slicer partitions NAS assets dynamically, with slices being reassessed prior to each backup, with slices added, removed or rebalanced based on backup history and on changes in the content of the NAS asset being partitioned. Periodically, unbalanced trees are automatically managed as content changes over time. No manual reconfiguration is required.

Dynamic NAS Protection intelligently and automatically scales to optimize performance. Multiple proxy hosts are deployed with Dynamic NAS Protection and as load requires, proxies can be spun up and torn down efficiently. Because proxies are virtual, reconfiguration happens without the need for manual intervention.

The virtual proxies are bundled within a proxy host so that all services needed for backup are provided at deployment. Further, agents are containerized on the proxy host, and the setup and teardown of proxies happens dynamically, spinning up and tearing down NAS containers as needed. This in turn minimizes backup job queues, while keeping backup resources productive and eliminates the need for separate agents to accommodate different share types.

By providing up to 3x faster backups and up to 2x faster restores, Dynamic NAS helps you achieve improved SLAs through simple, efficient management of NAS backup and recovery.

What are the key Differentiators?

• Snapshots for every backup with Intelligent crawl
  
  • Snapshot for Unity, PowerStore, PowerScale
    
• Scale beyond the NDMP backup stream limitation e.g. 8 max streams
  
• Intelligent slicing / directory slicing for better efficiency
  
• Multiple proxies support
  
• A single share can be backed up with multiple proxies / streams
  
• Existing Avamar supports forever incremental but at the cost of high compute resource
  
• Higher number of parallel streams and multiple virtual compute nodes to address scale and performance
  

Backup Workflow

• NAS Backup is initiated from PPDM Server
  

  – Each task is executed in Proxy host(s)
  

• Backup “Pre-Task”
  

  – Create Snapshot and “Slice” NAS Share
  

• Backup data task
  

  – Compute Proxies and Parallel streams
  

  – Backup “Slices” in parallel from multiple Proxies
  

• Backup “Post-Task”
  

  – Register backup with PPDM
  

  – Initiate Index and Search
  

  – Collect logs
  

  – Clean-up
  

• Backup job Monitoring during complete backup cycle
  
  

Restore Workflow

• User Selection for Recovery
  

  – Share Level
  

  – Search and Recovery
  

• Recovery initiated from PPDM Server
  

  – vPod identifies the Proxy host Recovery
  

  – vPod sends Recovery job to Proxy host
  

• Recover Tasks on Proxy host
  

  – Parse backup catalog
  

  – Create input list to restore selected backup
  

  – Recover to user selected destination path
  

• Recover “Post Task”
  

  – Collect logs
  

  – Cleanup
  

Here you can see a video showing how to configure and run Dynamic NAS Protection using PowerProtect Data Manager 19.9

And below, you can go through, an interactive demo

And, some media coverage

A guest post by Tomer Nahumi

The post Dell EMC PoweProtect 19.9, Part1 – Dynamic NAS Protection appeared first on Itzikr's Blog.

A few months ago, I wrote a post about the performance impact of VMware Snapshots with vVOLs and VMFS datastores and explored these performance aspects of VM snapshot creation/deletion on the performance of guest applications running inside the VMs. In this post, I’ll introduce Transparent Snapshots, a feature within Dell EMC PowerProtect Data Manager 19.9, which changes the way we protect VMware virtual machines to ensure availability of all your VMs at scale without business disruption.

Before we start, if you are new to PowerProtect 19.9, I suggest you start reading about another feature, I blogged about here: Dell EMC PowerProtect 19.9, Part1 – Dynamic NAS Protection

Organizations are continuously expanding their virtualized environments, which are becoming the norm for hosting applications, and their associated VM datasets are constantly growing.

A natural outcome of increased adoption of virtualization is that virtualized environments host more mission-critical applications that cannot experience downtime. These applications demand high, transaction rates, stable IOPS, and low read/write latency during backup.

The technology traditionally used for backing up VMware assets is VMWare’s vStorage APIs for Data Protection – VADP. Though widely used, VADP-based backup solutions, and other similar solutions such as storage array snaps and journaling, introduce challenges that lead to application disruption, ultimately compromising day-to-day operation.

These challenges include:

• Slow determination of deltas in the VM dataset, making the halt time for an application longer than desired, prior to each backup
• Undesirable need for resource-intensive external agents to move VM snapshot data to secondary storage
• Longer-than-acceptable application read and write latencies during backup
• Overly impactful application IOPS reduction during backup

The result is “stunned” mission-critical applications, impacting business operations.

Even as VADP evolves to address some of its performance issues, such as by adding Change Block Tracking, VADP-based solutions continue to struggle with latency and IOPS issues due to their need for external data movers, outside the ESX host.

The answer is to modernize how you protect your VMs. Transparent Snapshots technology is a new VM image method that eliminates application disruption during VM backups, all without compromise.

Transparent Snapshots is a new feature available with Dell EMC PowerProtect Data Manager 19.9 that delivers unique VMware VM protection.  Transparent Snapshots simplifies and automates VM image-level backups and enables backing up VMs WITHOUT the need to pause the VM during the backup process.  The result is significantly reduced impact on VMs, especially large, high-change-rate VMs.

Additionally, the simplified backup process via Transparent Snapshots reduces infrastructure by removing the reliance on proxies for data movement.

The bottom line is that with Transparent Snapshots you can effectively and efficiently backup up your VMs via a process that requires fewer steps, leading to less impact to your entire VM environment, thus ensuring availability of all your VMs without business disruption.

So how does it work?

PowerProtect Data Manager uses Transparent Snapshots by using a variant of the vSphere API for I/O (VAI/O) Filtering framework with version 7.0u3. It uses an internal component (TSDM) that is created through the PowerProtect Data Manager VIB to write the copies to the protection storage (Dell EMC PowerProtect appliance). During a restore, these same components use the backup copies.

PowerProtect Data Manager assumes the role of an orchestrator where it identifies the VM assets in the VMware environment and provides scheduling capabilities.

• PowerProtect Data Manager uses VM Direct Engine to communicate with the VMware vCenter level APIs provided by VMware. The VM Direct Engine connects to a component provided by VMware called Data Protection Service (DPS) which is a vCenter level service and has two key responsibilities in various scenarios:

• − Creates and tracks the progress of the vCenter level tasks that are visible to the end users such as sync, restore, and snapshot operations
  

• − Responsible for locating the relevant ESXi host on which the operation (backup or restore) is to be performed, based on the placement of the VM asset to be protected
  

• On each ESXi host, there is a Data Protection Daemon (DPD) provided by VMware that facilitates the protection-related APIs and workflows from VMware.
• The DPD also communicates to the Dell EMC Transparent Snapshot Data Mover (TSDM) component, which is responsible for the VM-backup data movement.
• The backup and restore processes transfer the Transparent Snapshots respectively to and from the PowerProtect Appliance.
• TSDM also consists of the PowerProtect Appliance SDK (DDBoost Library) which helps the framework access the storage units on the PowerProtect Appliance. It also helps write and read data from those storage units.

Look Mom! No vProxies!

Most prominent in the Transparent Snapshots method is the use of a new plug-in that embeds DD Boost on the ESX host, allowing direct data transfer to PowerProtect appliances without the need for external proxies to move either full-image or incremental data at backup time. The new plug-in is installed and deployed once on the ESX host itself as part of VM Image protection workflow.

There are three simple steps of the VM image backup lifecycle with Transparent Snapshots:

• Change monitoring, which tracks the VM deltas since the last snapshot was taken
• Snapshot processing, during which the VM deltas are accessed in memory.  Data is then transferred directly to PowerProtect appliances
• Snapshot release, at which time delta tables and temporary data blocks are removed.

Because it does not require external proxies or temporary copies to be used as part of the VM image backup lifecycle, Transparent Snapshots is superior to traditional backups at each step of the lifecycle.

Transparent Snapshots is a simple and efficient method to backup VMs and offers significant advantages.

• Auto-scaling: An ESX-based plugin – which is a VMware certified VMware Installation Bundle (VIB) – gets deployed real-time, as needed, eliminating the need to use proxies (that is needed with traditional methods) and allowing for scaling by leveraging additional ESXi hosts where the data mover resides, versus needing manual intervention.
• Continuous in-memory deltas: Monitors the deltas of full VMs, resulting in no impact to VM or ESX host – versus full real-time continuous data protection with traditional methods that requires extensive resources and does not scale for hundreds of VMs.
• Storage agnostic: Transparent Snapshots deploys seamlessly into your existing storage environment. It does not require hardware-specific snapshots to be used in order to perform a single VM backup. Traditional methods utilize vendor-specific storage hardware that supports, and is specialized for, that storage type. Therefore, Transparent Snapshots eliminates the need for expensive, specialized storage. No new storage arrays or upgrades are required.
• Direct data movement to PowerProtect appliances: Direct access to data eliminates the need for working copies. Traditional options require intermediary copies (for example, clones) that use additional compute and storage resources and impact backup processes.

• VADP is the standard. It’s a low-cost snapshot method that is not optimized for large or performant VMs and requires proxies to move the data off the ESX.
• Custom VM Tools is a way to help address the VM latency by impacting the performance of the ESX and adds the additional overhead of custom agents.
• Storage Array Snaps leverages the built-in hardware snapshot capability to address the VM latency but is only available on Storage Array enabled appliances.
• Journaling is another method for backing up the image but requires additional resources to monitor and track IO changes between backups. This solution is expensive to scale because of the increased resources.

With Transparent Snapshots, the benefits are huge:

• Increase backup performance
  • Near zero impact to VMs or environment
  • Up to 5X faster backups
  • Up to 5X reduction in VM latency
• Lower costs & simplify management
  • Zero proxies for data movement
  • Storage agnostic
  • Environment auto-scales via automation

In conclusion, Transparent Snapshots offers significant benefits to backup performance, costs and simplified management as well as reducing risk of data loss.

Backup performance is increased since there is no need to pause VMs during backup processes and the compute and network resources used to perform the backup are separate from the VMs. The result is near-zero impact to VMs or the environment. Restore performance is also faster and VM latency is reduced.

Transparent Snapshots does not require proxies for data movement, which lowers infrastructure, costs, and simplifies management.  Transparent Snapshots is also storage agnostic, supporting NFS and vSAN, and automatically scales to support the number of ESX hosts you want to backup without additional impact to the host.

Here you can see a video comparing a VMware Snapshot with Transparent Snapshot and showing how to configure and run a Transparent Snapshot backup using PowerProtect Data Manager 19.9

An interactive demo (click the screenshot below)

White Paper is available here (click the screenshot below)

Some other coverage by the leading industry publications:

A guest post by Tomer Nahumi

The post Dell EMC PowerProtect 19.9, Part 2, Protecting VMware Virtual Machines, Using Transparent Snapshots appeared first on Itzikr's Blog.

The OpenManage Integration for VMware vCenter is a virtual appliance that streamlines tools and tasks associated with management and deployment of Dell servers in the virtual environment.

To effectively run today’s data centers, you need to manage both physical and virtual infrastructure. This means using multiple disconnected tools and processes to manage your environment, wasting valuable time and resources.
Dell has created a solution to address this problem while increasing task automation right from within the virtualization management console that you use most, VMware vCenter. The OpenManage Integration for VMware vCenter is a virtual appliance that streamlines tools and tasks associated with the management and deployment of Dell servers in your virtual environment.
OpenManage Integration for VMware vCenter (OMIVV) is designed to streamline the management processes in your data center environment by allowing you to use VMware vCenter to manage your entire server infrastructure – both physical and virtual. From monitoring system level information, bubbling up system alerts for action in vCenter, rolling out firmware updates to an ESXi cluster, to bare metal deployment, the OpenManage Integration will expand and enrich your data center management experience with Dell PowerEdge servers.

If you are new to the OpenManage Integration for VMware vCenter, I highly suggest you start by, reading about it here

The release notes:

Dell EMC OpenManage Integration for VMware vCenter, v5.4 | Driver Details | Dell US

Fixes:
1. PowerEdge R820 host image is not displayed on the OMIVV Host Information page after successful inventory
2. When you perform the cluster or host level firmware update on ESXi 7.0 U2 host having drivers, driver installation status information in missing in the OMIVV logs.
3. IDSDM component firmware version is not reflecting in inventory after firmware update.
4. After backup and restore or RPM upgrade operation, Memory Page Retire (MPR) alarm is enabled on the Alarm Definitions page of vCenter.
5. vSphere Lifecycle Manager shows firmware version for PCIe SSD or NVMe SSD as blank for vSAN cluster. When the Hardware compatibility is triggered, vSphere Lifecycle Manager does not validate PCIe SSD or NVMe SSD. If these components exist in the server, firmware version is shown as blank because vSphere Lifecycle Manager is not considering OMIVV output for these components
6. In vCenter linked mode environment, cluster profile entry remains in OMIVV UI and API even after unregistering
vCenter and re-registering the same vCenter.
7. The IDSDM component is not detected under Installation Target section in OMIVV during OS deployment. However, iDRAC can detect the IDSDM.
8. Fan and temperature sensor details of FX chassis are not displayed on the Hardware Overview page of OMIVV.

Enhancement:
1. Support for additional RESTful APIs.
2. Support for vSphere 7.0 U3.
3. Support for PowerEdge R750xs, R650xs, R550, R450, XR11, and XR12 servers.
4. Support for R750, R650, and R7525 vSAN Ready Nodes.
5. Enhancement in cluster profile to display cluster profile version

Upgrading it, is very easy, if you have a network interface of the OpenManage OVA that is connected to the internet (or via proxy), simply browse to the OpenManage IP -> Appliance Management

And click -> Update

Alternatively, for new deployments, You can download it, by clicking the screenshot below

Also, the Dell ESXi 7.0 Update 3, can be downloaded from the VMware web site

Changes there, are

bnxtnet–218.0.62.0-1OEM.700.1.0.15843807–BCM–VMwareCertified–2021-09-27
bnxtroce–218.0.21.0-1OEM.700.1.0.15843807–BCM–VMwareCertified–2021-09-27
dell-shared-perc8–06.806.92.00-1OEM.700.1.0.15843807–BCM–VMwareCertified–2021-09-27
dell-configuration-vib–7.0.0-A00–DEL–PartnerSupported–2021-09-27
dellemc-osname-idrac–7.0.0-A00–DEL–PartnerSupported–2021-09-27
i40en-ens–1.4.1.0-1OEM.700.1.0.15843807–INT–VMwareCertified–2021-09-27
i40en–1.14.1.0-1OEM.700.1.0.15843807–INT–VMwareCertified–2021-09-27
icen–1.6.2.0-1OEM.700.1.0.15843807–INT–VMwareCertified–2021-09-27
igbn–1.5.2.0-1OEM.700.1.0.15843807–INT–VMwareCertified–2021-09-27
irdman–1.3.4.23-1OEM.700.1.0.15843807–INT–VMwareCertified–2021-09-27
ixgben-ens–1.5.1.0-1OEM.700.1.0.15843807–INT–VMwareCertified–2021-09-27
ixgben–1.10.3.0-1OEM.700.1.0.15843807–INT–VMwareCertified–2021-09-27
nmlx5-core–4.21.71.101-1OEM.702.0.0.17630552–MEL–VMwareCertified–2021-09-27
nmlx5-rdma–4.21.71.101-1OEM.702.0.0.17630552–MEL–VMwareCertified–2021-09-27
qlnativefc–4.1.44.0-1OEM.700.1.0.15843807–Marvell–VMwareCertified–2021-09-27
qcnic–2.0.59.0-1OEM.700.1.0.15843807–QLC–VMwareCertified–2021-09-27
qedentv–3.40.36.0-1OEM.700.1.0.15843807–QLC–VMwareCertified–2021-09-27
qedf–2.2.62.0-1OEM.700.1.0.15843807–QLC–VMwareCertified–2021-09-27
qedi–2.19.59.0-1OEM.700.1.0.15843807–QLC–VMwareCertified–2021-09-27
qedrntv–3.40.35.0-1OEM.700.1.0.15843807–QLC–VMwareCertified–2021-09-27
qfle3–1.4.17.0-1OEM.700.1.0.15843807–QLC–VMwareCertified–2021-09-27
qfle3f–2.1.25.0-1OEM.700.1.0.15843807–QLC–VMwareCertified–2021-09-27
qfle3i–2.1.8.0-1OEM.700.1.0.15843807–QLC–VMwareCertified–2021-09-27

The post Dell OpenManage Integration for VMware vCenter 5.4 and the Dell ESXi 7.0 Update 3 ISO, are now available. appeared first on Itzikr's Blog.

Dell Tech World 2019, yea, the days of actual in-person conferences, Michael Dell is on stage and during his keynote, he says “we are fully embracing Kubernetes”. My session is the next one where I explain our upcoming integration of storage arrays with the Kubernetes CSI (Container Storage Interface) API. Now, don’t get me wrong, CSI is awesome! But at the end of my session, I’m getting a lot of people coming to me and ask very similar questions, the theme was around ‘how do I still keeping track of what’s going to happen in the storage array’, you see, CSI doesn’t have role-based access to the storage array, not to even mention things like quota management. At a very high level, think about storage admins that want to embrace Kubernetes but are afraid to lose control of their storage arrays. If ‘CSI’ feels like a name of a TV show, I encourage you to stop here and go ahead and have some previous reads in my blog about it: https://volumes.blog/?s=csi Back to 2019. Post my session, I gathered a team of product managers and we started to think about upcoming customer’s needs, we didn’t have to use a crystal ball but rather, as the largest storage company in the world, started to interview customers about their upcoming needs re K8s. Now, let’s take a step back and discuss the emergence of cloud-native apps and Kubernetes In the past, companies would rely on Waterfall development and ITIL change management operational practices. This meant organizations had to plan for:

• Long Development cycles before handing an application to ops
• IT ops often resisting change and slow innovation

Now companies want to take advantage of a new development cycle called Agile along with DevOps operational practices. This new foundation for IT accelerates innovation through:

• Rapid iteration and quick releases
• Collaboration via involving the IT ops teams throughout the process

Operational practices aren’t the only evolving element in today’s enterprises; application architectures are quickly changing as well. For years, monolithic architectures were the standard for application architectures. These types of applications had great power and efficiency and run on virtual machines. However, these applications have proven costly to reconfigure, update, and take a long time to load. In cloud-native applications, components of the app are segmented into microservices, which are then bundled and deployed via containers. This container/microservice relationship allows cloud-native apps to be updated and scaled independently. To manage these containerized workloads, organizations use an open-source management platform called Kubernetes. To give a real-world example, imagine monolithic apps like a freight train – there is a lot of power and capacity but it takes a long time to load and is not easy to reconfigure. Whereas cloud-native apps function more like a fleet of delivery vehicles with reduced capacity but resilient and flexible in changing the payload or adapting capacity as needed. A fleet of delivery vehicles needs a conductor to schedule and coordinate the service, and that is the role that Kubernetes plays for containers in a cloud-native environment. Both approaches are present in today’s modern apps but the speed and flexibility of cloud-native apps shifting priorities everywhere. Let’s dig more into this shift in software development and delivery. Leading this shift is the use of microservices, which are loosely coupled components that are self-contained, highly available, and easily deployable, and with containers that provide these microservices with lightweight packages capable of resource utilization efficiencies, enable those microservices patterns. They provide a ‘build once, run anywhere flexibility with the scale that developers are embracing. Then came Kubernetes. Kubernetes is an open-source system for automating deployment, scaling, and management of containerized applications. It has become the industry “go-to” for more service discovery, load balancing, storage orchestration. With agile development comes the need for speed and continuous delivery which, with the right tools and infrastructure can create the right business outcomes as demands increase. With the advent of flexible cloud-native applications; DevOps teams formed and created their own agile frameworks that in addition to increasing delivery of code with less dysfunction and overhead of traditional models whereby intentionally or unintentionally bypassing IT Operations’ best practices and the opportunity to build modern IT infrastructures to support their development initiatives, as well as enhance them.

As traditional models for software development evolve, so does the infrastructure that supports it. IT Operations’ best practices can be applied to these new models through the Enterprise level data management tools that Dell Technologies’ provides. DevOps teams require seamless, non-disruptive, and reliable mechanisms to continue to meet business demands with agility and scale. With Dell Technologies” broad portfolio designed for modern and flexible IT growth, customers can employ end-to-end storage, data protection, compute and open networking solutions that support accelerated container adoption. Developers can create and integrate modern applications by relying on accessible open-source integrated frameworks and tools across bare metal, virtual, and containerized platforms. Dell enables support for DevOps elasticity and real-time benefits for container and Kubernetes platforms’ applying best practices based on their own design and needs.

Dell Technologies aligns developers and IT operations, empowering them to design and operate cloud-native organizations while achieving business demands and increasing quality outputs. With the support of industry standards built on containers such as Containers’ storage interfaces, Plug-ins with container storage modules, PowerProtect data manager can Availability is the most important aspect of data that customers and different levels of business ultimately care about from about every angle; especially securely accessed data whether it be on-premises, in the cloud. Though developers seem to claim they understand Kubernetes inside and out, they miss out on features at the IT operations level that we can provide.  With a big portfolio such as ours, we must understand what maturity level the customer is in. For the storage administrator, they will defer using their PowerMax or VxRail; if they want to continue to purchase these products, they would appreciate built-in containers/Kubernetes support that is easy to onboard without disrupting their developers. At the application layer, you may be employing Kubernetes or OpenShift well into the software-defined journey and PowerFlex would be an optional choice. GitHub CSI downloads exceed 1million downloads. Kubernetes developers know nothing about storage except local storage servers and drives; whereby their operational partners care about resiliency, snapshot, restore, replication, compression, and security. With the variety of storage solutions, having CSI plug-ins and Container Storage Modules simplifies deployment choices, emphasis on applying operational best practices. Build :

• Cloud-Native Computing Foundation (CNCF) SIG contributor
• Complete E2E integrated industry-standard APIs
• Self-service CSI driver workflows
• GitHub repository for developers
• Partner integrations with VMware Tanzu, Red Hat OpenShift,  Google Anthos, Rancher, others
• DevOps and IaC integration with Ansible, Terraform, Puppet, ServiceNow, vRO, Python, Powershell, etc.
• Kubernetes Certified Service Provider (KCSP) Consultant Services

Automate & Manage:

• Container storage modules (CSM)
  • Data replication across data centers
  • RBAC authorization
  • Observability
  • Resiliency (disaster recovery & avoidance)
• Single platform, Kubernetes & application-aware data protection
• Application consistent backups
  • MySQL, MongoDB, Cassandra, Postgres, etc.
• Infrastructure Automation & Lifecycle Management
  • API driven software-defined infrastructure with automated lifecycle management
• Policy-based protection
  • Replication, retention, tiering to S3-compatible object storage, SLA reporting
• Provide in-cloud options for developers with support for AWS, Azure backup policies

Scale & Secure:

• Provisioning and automating policies
• Extract data value in real-time through open networking and server/compute
• Deploy data protection backup and restores via PowerProtect Data Manager
• Integrated Systems; VxBlock, VxRail, PowerFlex, Azure Stack 
• Manage Kubernetes with PowerScale in multi-cloud environments
• Accelerate with edge / bare metal via Kubespray / Streaming Data Platform (SDP) w/ Ready Stack for Red Hat OpenShift platforms
• Obtain seamless security and secure critical data via CloudLink

Ok, let’s talk Kubernetes

Kubernetes is really starting to pick up, as you can see in the above graphs, by 2025, it is expected that up to 70% of the enterprises out there, will be using Kubernetes AND that, 54% will be deployed primarily in their production environments! Yep, that means, we are way beyond the ‘Kicking the tires’ phase. A few weeks ago, I talked with my manager about these trends which you can see below

BUT, it’s not all rosy, Kubernetes provides a lot of challenges, to name a few:
Lack of internal alignment…shadow IT results… which leads to a harder job for the IT admins with lack of visibility and monitoring, and meeting security and compliance requirements. Kubernetes also cannot automatically guarantee that resources are properly allocated between different workloads running in a cluster. To set that up, you need to set up resource quotas manually. The opportunity is to align developers and IT operations by empowering them to design and operate cloud-native organizations while achieving business demands and increasing quality outputs.

In the next post, I will share the ‘What’ are we releasing to tackle these challenges..

The post Introducing Dell Container Storage Modules (CSM), Part1 – The ‘Why’ appeared first on Itzikr's Blog.