ONTAP S3 on Local Cluster v Remote Cluster

on

Scenario

I have couple of ONTAP clusters.

Cluster 2 was originally setup with internal FabricPool like:

How to set up ONTAP S3 as object store for FabricPool on local cluster using HTTP - NetApp Knowledge Base

That is, a 4-node cluster, with AFF A-Series tiering to an S3 bucket on C-Series.

But now we need to connect an A-Series in a different cluster to a new S3 bucket on the same C-Series. If it was a new setup, we would be following this KB:

How to set up ONTAP S3 as object store for FabricPool on remote cluster using HTTPS - NetApp Knowledge Base


Q: What are the differences in the set up?

Local / HTTP Workflow:

  1. Create the S3 server and bucket on a local SVM.
  2. Generate S3 access keys.
  3. Add the object store tier to FabricPool using the local S3 LIF IP address, specifying HTTP and port 80 (or your custom HTTP port).
  4. Attach the object store to your local aggregates.


Remote / HTTPS Workflow:

  1. Create the S3 server and bucket on the remote cluster's SVM.
  2. Install/Configure an SSL certificate for the remote S3 server.
  3. Export the public CA certificate from the remote cluster (or your enterprise CA).
  4. Import the CA certificate into the source cluster so it trusts the remote S3 server.
  5. Generate S3 access keys on the remote cluster.
  6. Add the object store tier to FabricPool on the source cluster using the remote S3 server's FQDN, specifying HTTPS and port 443.
  7. Attach the object store to your local aggregates.


Comparing the ONTAP CLI Procedures for differences

From the links above, here we look at the Remote/HTTPS workflow and see which steps are the same as the Local/HTTP workflow and which are different. I use colour coding:

  • Green for SAME
  • Cyan for what was done DIFFERENT in the Local/HTTP version
  • Yellow for DIFFERENT setp in the Remote/HTTPS version
After the first 3 steps, the workflows diverge. There are similarities in the remaining tasks, but it would make sense to create a new Data SVM for the remote access, rather than trying to modify the local access one you've already created.


Local/HTTP Workflow

ONTAP_S3 server configuration and bucket creation:

1) Verify if ONTAP >=9.8RC1
::> version

2) Add ONTAP_S3 license code
::> system license add XXXXXXXXXXXXXXXXXXXXXXXXXXXX

3) Verify ONTAP_S3 license code application
ONTAP_S3_cluster::> system license show -package s3
 
4) Configure ONTAP_S3 server on Cluster SVM with HTTP enabled
::*> vserver object-store-server create -vserver Cluster -object-store-server S3server.local -is-http-enabled true -is-https-enabled false -listener-port 80  -status-admin up

5) Check the Cluster SVM is listening on ports configured above which is port 80 and cluster lif traffic is via port 7700
::*> network connections listening show -vserver Cluster

6) Create a data SVM to hold the ONTAP S3 bucket
::*> vserver create -vserver svm1 -subtype default -rootvolume svm1_root -rootvolume-security-style unix -language C.UTF-8 -aggregate aggr1

NOTE: By default the local S3 configuration does not require data-services to be enabled. If the S3 SVM also needs to serve data (S3/CFIS/NFS etc), include the data-services in the configuration.

7) Create an ONTAP_S3 bucket on a data SVM
::*> vserver object-store-server bucket create -vserver svm1 -bucket ontaps3bucket -comment "Test ONTAPS3 Bucket" -size 1TB -aggr-list aggr1

NOTE: A Flexgroup volume is created. Minimal space of 100GB required. Bucket name should be in all lower case.

::*> vserver object-store-server bucket show
::*> vol show -is-constituent true

8) Create a non-root user on data SVM that can access the bucket with access keys and secret keys
::*> vserver object-store-server user create -vserver svm1 -user OntapS3User
::*> vserver object-store-server user show -vserver svm1 -user OntapS3User -instance

9) Set access permissions on the bucket and its objects for the above user 
::*> vserver object-store-server bucket policy add-statement -vserver svm1 -bucket ontaps3bucket -effect allow -action GetObject,PutObject,DeleteObject,ListBucket,GetBucketAcl,GetObjectAcl,ListBucketMultipartUploads,ListMultipartUploadParts  -principal OntapS3User -resource ontaps3bucket,ontaps3bucket/*

NOTE: Above command allows full access to bucket and its object for specific user and is customizable

FabricPool configuration on the same local cluster as ONTAP_S3 server and bucket:

1) Create a FabricPool object store configuration on Cluster SVM:
::*> storage aggregate object-store config create -object-store-name LocalFPwithS3 -provider-type ONTAP_S3 -server S3server.local -is-ssl-enabled false -port 80 -container-name ontaps3bucket -ipspace Cluster -use-http-proxy false -access-key OntapS3UserAccessKey -secret-password OntapS3UserSecretkey

2) Verify the created local FabricPool object store
::*> storage aggregate object-store config show -instance

3) Attach object store to aggregate
::*> storage aggregate object-store attach -aggr YouLocalAggregate -object-store-name LocalFPwithS3

Remote/HTTPS Workflow

ONTAP_S3 server configuration and bucket creation:

Example Cluster Name: ONTAP_S3_cluster


1) Verify if version >=9.8RC1

ONTAP_S3_cluster::> version


2) Add ONTAP_S3 license code

ONTAP_S3_cluster::> system license add XXXXXXXXXXXXXXXXXXXXXXXXXXXX


3) Verify ONTAP_S3 license code application

ONTAP_S3_cluster::> system license show -package s3


4) Create a data SVM with -data-services data-s3-server  for ONTAP_S3 server

ONTAP_S3_cluster::> vserver create -vserver ONTAPS3SVM -subtype default -rootvolume ONTAPS3SVM_root -rootvolume-security-style unix -language C.UTF-8 -snapshot-policy default -data-services data-s3-server


5) Create a service policy to be associated with the network interface to access the ONTAP_S3 server.

ONTAP_S3_cluster::> network interface service-policy create -vserver ONTAPS3SVM -policy  ServicePolicyONTAPS3 -services data-core,data-s3-server


6) Create a network interface to access the ONTAP_S3 server and associate the service policy created for object store access

ONTAP_S3_cluster::*> network interface create -vserver ONTAPS3SVM -lif ONTAPS3Lif -service-policy ServicePolicyONTAPS3 -address 10.x.x.25 -netmask 255.255.255.0 -home-node nodename -home-port portname


NOTE: If DNS name resolution of  ONTAP_S3 server FQDN is available, add DNS to ONTAP_S3 vserver


7) Create a route for the SVM used for ONTAP_S3 server

ONTAP_S3_cluster::*> network route create -vserver ONTAPS3SVM -destination 0.0.0.0/0 -gateway  10.x.x.1


8) In case of no DNS, add a static IP (ONTAP_S3 server lif created in Step#6) to host entry to the name of the ONTAPS3 server that you are planning to create.

ONTAP_S3_cluster::*> vserver services name-service dns hosts create -vserver ONTAPS3SVM -address 10.x.x.25 -hostname S3server.domain.com 


NOTE: FQDN is needed for SSL certificate create as well as to configure ONTAPS3 server and object-store configuration for Fabric Pool later.


9) Choose between two options (creating SSL certificate):

a) Option 1 - Using a self-signed certificate (easy)(we don't include Option 2 here):

ONTAP_S3_cluster::*> security certificate create -vserver ONTAPS3SVM -common-name S3server.domain.com -type server  -expire-days 365 -cert-name cert_ONTAP_S3


ONTAP_S3_cluster::*> security certificate show -vserver ONTAPS3SVM -type server  -cert-name cert_ONTAP_S3 -instance


NOTE: copy the key for use to install on the client.


NOTE: By default during vserver create, a single self-signed server type certificate is created. You can not use that one, as it has not set the FQDN of the ONTAP_S3 data LIF as Common Name (CN). Adjust -expire-days to your requirement. Above command is a minimal version - you can populate all other certificate parameters, as example -country, -email-addr, -organization, etc. - these have no technical relevance.


10) Copy the root-ca certificate for later installation within the client (FabricPool cluster) certificate trust store later in correct format:

ONTAP_S3_cluster::*> security certificate show -vserver ONTAPS3SVM -cert-name cert_RootCA -type root-ca -instance

 

11) Configure ONTAP_S3 server using the following: 

a) -object-store-server = FQDN or Custom Common Name: from above output

b) -certificate-name =  give it a certificate name

ONTAP_S3_cluster::*> vserver object-store-server create -vserver ONTAPS3SVM -object-store-server S3server.domain.com -is-http-enabled true -is-https-enabled true -listener-port 80 -secure-listener-port 443 -status-admin up -comment "" -certificate-name cert_ONTAP_S3


NOTE: Here both HTTPs and HTTP is configured.


12) Check if ONTAP_S3 server is listening on ports configured above

ONTAP_S3_cluster::*> network connections listening show -vserver ONTAPS3SVM                                        


13) Create an ONTAP_S3 bucket on the ONTAP_S3 vserver where ONTAP_S3 server is configured 

ONTAP_S3_cluster::*> vserver object-store-server bucket create -vserver ONTAPS3SVM -bucket ontaps3bucket -comment "Test ONTAPS3 Bucket" -size 1TB


NOTE: A Flexgroup volume is created. Minimal space of 100GB required. Bucket name should be in all lower case.


ONTAP_S3_cluster::*> vserver object-store-server bucket show


14) Create a non-root user that can access the bucket with access keys and secret keys

ONTAP_S3_cluster::*> vserver object-store-server user create -vserver ONTAPS3SVM -user OntapS3User

ONTAP_S3_cluster::*> vserver object-store-server user show -vserver ONTAPS3SVM -user OntapS3User -instance


15) Set access permissions on the bucket and is objects for the above user 

ONTAP_S3_cluster::*> vserver object-store-server bucket policy add-statement -vserver ONTAPS3SVM -bucket ontaps3bucket -effect allow -action GetObject,PutObject,DeleteObject,ListBucket,GetBucketAcl,GetObjectAcl,ListBucketMultipartUploads,ListMultipartUploadParts  -principal OntapS3User -resource ontaps3bucket,ontaps3bucket/*


NOTE: Above command allows full access to bucket and its object for specific user and is customizable


16) Create a new group containing the above user and assign desired policy

ONTAP_S3_cluster::*> vserver object-store-server group create -vserver ONTAPS3SVM -name Ontaps3Group -users OntapS3User


FabricPool configuration to use the created ONTAP_S3 server and bucket

Example Cluster Name: FabricPool_cluster


1) Install FabricPool license NetApp License File

2) Create IC LIF homed each node on cluster , cluster peering not required.

FabricPool_cluster::*> network interface create -vserver FabricPool_cluster -lif FabricPool_cluster-01_IClif -service-policy default-intercluster -address 10.x.x.51 -netmask 255.255.255.0 -home-node FabricPool_cluster-01

FabricPool_cluster::*> network interface create -vserver FabricPool_cluster -lif FabricPool_cluster-02_IClif -service-policy default-intercluster -address 10.x.x.52 -netmask 255.255.255.0 -home-node FabricPool_cluster-02


3) In case of no DNS, add a static IP (ONTAP_S3 server lif created in step6 on ONTAP_S3_cluster ) to host entry to the name of the ONTAPS3 server that you are planning to create, to admin vserver (remote access via Intercluster lif) 

FabricPool_cluster::*> vserver services name-service dns hosts create -vserver FabricPool_cluster -address 10.x.x.25 -hostname S3server.domain.com 


NOTE: If DNS name resolution available of ONTAP_S3 server FQDN, add DNS to admin vserver


4) Install -type server-ca root-ca or self-signed certificate used in ONTAP_S3 into the truststore of the Fabricpool cluster, you can choose any cert-name. 

FabricPool_cluster::*> security certificate install -vserver FabricPool_cluster -type server-ca -cert-name cert_ONTAP_S3


5) Configure object store with all the details 

FabricPool_cluster::*> storage aggregate object-store config create -object-store-name ONTAPS3store -provider-type ONTAP_S3 -server S3server.domain.com -is-ssl-enabled true -port 443 -container-name ontaps3bucket -ipspace Default -use-http-proxy false -access-key OntapS3UserAccessKey -secret-password OntapS3UserSecretKey

 

FabricPool_cluster::*>  storage aggregate object-store config show -instance


6) Verify connectivity using Object store profiler

​​FabricPool_cluster::*> storage aggregate object-store profiler start -node FabricPool_cluster-01 -object-store-name ONTAPS3store


FabricPool_cluster::*> storage aggregate object-store profiler show


Comments

Post a Comment