Adding a new Notary organization in R3 Corda Enterprise¶
Corda Enterprise Network Map (CENM) 1.2 does not allow dynamic addition of new Notaries to an existing network via API Call. This process is manual and involves few steps as described in the Corda Official Documentation here.
To overcome this, we have created an Ansible playbook. The playbook will update the Networkmap service so that a networkparameter update is submitted. But the run flagDay command has to be manual, as it is not possible to login to each Network Participant and accept the new parameters. Also, whenever the parameter update happens, it will trigger a node shutdown. Hence, the run flagDay command must be executed when no transactions are happening in the network.
run flagDay command must be run after the network parameters update deadline is over (+10 minutes by default). And this command must be run during downtime as it will trigger Corda node restart.
Prerequisites¶
To add a new Notary organization, Corda Idman and Networkmap services should already be running. The public certificates and NetworkTrustStore from Idman and Networkmap should be available and specified in the configuration file.
Note
Addition of a new Notary organization has been tested on an existing network which is created by Bevel. Networks created using other methods may be suitable but this has not been tested by Bevel team.
Deploy new Notary Service¶
Deploy the additional notary/notaries as separate organizations by following the guidance on how to add new organizations here. A sample network.yaml for adding new notary orgs can be found here.
# Copyright Accenture. All Rights Reserved.
#
# SPDX-License-Identifier: Apache-2.0
##############################################################################################
---
# yaml-language-server: $schema=../../../../platforms/network-schema.json
# This is a sample configuration file for Corda Enterprise network to add a New Notary organization to existing network.
# For multiple K8s clusters, there should be multiple configuration files or refer to configuration guide on the below link.
# https://hyperledger-bevel.readthedocs.io/en/latest/operations/corda_networkyaml.html
network:
# Network level configuration specifies the attributes required for each organization
# to join an existing network.
type: corda-enterprise
version: "4.7" # Hyperledger Bevel deployment supports node and notary enterprise 4.7 version (use older tag for other version supports)
frontend: enabled #Flag for frontend to enabled for nodes/peers
#Environment section for Kubernetes setup
env:
type: "dev" # tag for the environment. Important to run multiple flux on single cluster
proxy: ambassador # value has to be 'ambassador' as 'haproxy' has not been implemented for Corda
proxy_namespace: "ambassador" # Namespace required for Ambassador Edge-Stack deployment
ambassadorPorts: # Any additional Ambassador ports can be given here, this is valid only if proxy='ambassador'
portRange: # For a range of ports
from: 15005
to: 15051
# ports: 15020,15021 # For specific ports
loadBalancerSourceRanges: # (Optional) Default value is '0.0.0.0/0', this value can be changed to any other IP adres or list (comma-separated without spaces) of IP adresses, this is valid only if proxy='ambassador'
retry_count: 20 # Retry count for the checks
external_dns: enabled # Should be enabled if using external-dns for automatic route configuration
# Docker registry details where images are stored. This will be used to create k8s secrets
# Please ensure all required images are built and stored in this registry.
# Do not check-in docker_password.
docker:
url: "adopblockchaincloud0502.azurecr.io"
username: "docker_username"
password: "docker_password"
# Remote connection information for idman and networkmap (will be blank or removed for hosting organization)
network_services:
- service:
name: idman
type: idman
uri: https://idman.test.corda.blockchaincloudpoc.com
certificate: home_dir/platforms/r3-corda-ent/configuration/build/ambassador/idman/ambassador.pem
crlissuer_subject: "CN=Corda TLS CRL Authority,OU=Corda UAT,O=R3 HoldCo LLC,L=New York,C=US"
- service:
name: networkmap
type: networkmap
uri: https://networkmap.test.corda.blockchaincloudpoc.com
certificate: home_dir/platforms/r3-corda-ent/configuration/build/ambassador/networkmap/ambassador.pem
truststore: home_dir/platforms/r3-corda-ent/configuration/build/networkroottruststore.jks #Certificate should be encoded in base64 format
truststore_pass: rootpassword
# Allows specification of one or many organizations that will be connecting to a network.
# If an organization is also hosting the root of the network (e.g. idman, membership service, etc),
# then these services should be listed in this section as well.
organizations:
# The first worganization should be the one hosting the networkmap and should already exist
- organization:
name: supplychain
country: UK
state: London
location: London
subject: "CN=DLT Root CA,OU=DLT,O=DLT,L=London,C=GB"
subordinate_ca_subject: "CN=Test Subordinate CA Certificate, OU=HQ, O=HoldCo LLC, L=New York, C=US"
type: cenm
version: 1.5
external_url_suffix: test.corda.blockchaincloudpoc.com
cloud_provider: aws # Options: aws, azure, gcp
aws:
access_key: "aws_access_key" # AWS Access key, only used when cloud_provider=aws
secret_key: "aws_secret_key" # AWS Secret key, only used when cloud_provider=aws
# Kubernetes cluster deployment variables. The config file path and name has to be provided in case
# the cluster has already been created.
k8s:
context: "cluster_context"
config_file: "cluster_config"
# Hashicorp Vault server address and root-token. Vault should be unsealed.
# Do not check-in root_token
vault:
url: "vault_addr"
root_token: "vault_root_token"
secret_path: "secretsv2"
# Git Repo details which will be used by GitOps/Flux.
# Do not check-in git_access_token
gitops:
git_protocol: "https" # Option for git over https or ssh
git_url: "https://github.com/<username>/bevel.git" # Gitops https or ssh url for flux value files
branch: "develop" # Git branch where release is being made
release_dir: "platforms/r3-corda-ent/releases/dev" # Relative Path in the Git repo for flux sync per environment.
chart_source: "platforms/r3-corda-ent/charts" # Relative Path where the Helm charts are stored in Git repo
username: "git_username" # Git Service user who has rights to check-in in all branches
password: "git_access_token" # Git Server user password/token (Optional for ssh; Required for https)
email: "git@email.com" # Email to use in git config
git_repo: "github.com/<username>/bevel.git" # Gitops git repository URL for git push
private_key: "path_to_private_key" # Path to private key file which has write-access to the git repo (Optional for https; Required for ssh)
# Cordapps Repository details (optional if cordapps jar are store in a repository)
cordapps:
jars:
- jar:
# e.g https://maven.pkg.github.com/hyperledger/bevel/com.supplychain.bcc.cordapp-supply-chain/4.7/cordapp-supply-chain-4.7.jar
url: "https://repo/path/cordapp1.jar"
- jar:
# e.g https://maven.pkg.github.com/hyperledger/bevel/com.supplychain.bcc.cordapp-contracts-states/4.7/cordapp-contracts-states-4.7.jar
url: "https://repo/path/cordapp2.jar"
username: "git_username"
password: "git_access_token"
credentials:
keystore:
keystore: cordacadevpass
idman: password
networkmap: password
subordinateca: password
rootca: password
tlscrlsigner: password
truststore:
truststore: trustpass
rootca: rootpassword
ssl: password
ssl:
networkmap: password
idman: password
signer: password
root: password
auth: password
# Services maps to the pods that will be deployed on the k8s cluster
# This sample has idman, networkmap, signer and notary on one cluster but different namespaces
services:
zone:
name: zone
type: cenm-zone
ports:
enm: 25000
admin: 12345
auth:
name: auth
subject: "CN=Test TLS Auth Service Certificate, OU=HQ, O=HoldCo LLC, L=New York, C=US"
type: cenm-auth
port: 8081
username: admin
userpwd: p4ssWord
gateway:
name: gateway
subject: "CN=Test TLS Gateway Certificate, OU=HQ, O=HoldCo LLC, L=New York, C=US"
type: cenm-gateway
port: 8080
idman:
name: idman
subject: "CN=Test Identity Manager Service Certificate, OU=HQ, O=HoldCo LLC, L=New York, C=US"
crlissuer_subject: "CN=Corda TLS CRL Authority,OU=Corda UAT,O=R3 HoldCo LLC,L=New York,C=US"
type: cenm-idman
port: 10000
networkmap:
name: networkmap
subject: "CN=Test Network Map Service Certificate, OU=HQ, O=HoldCo LLC, L=New York, C=US"
type: cenm-networkmap
ports:
servicePort: 10000
targetPort: 10000
signer:
name: signer
subject: "CN=Test TLS Signer Certificate, OU=HQ, O=HoldCo LLC, L=New York, C=US"
type: cenm-signer
ports:
servicePort: 8080
targetPort: 8080
notaries:
- notary:
name: notary-1
subject: "O=Notary,OU=Notary1,L=London,C=GB"
serviceName: "O=Notary Service,OU=Notary1,L=London,C=GB"
type: notary
validating: true
emailAddress: "dev@bevel.com"
p2p:
port: 10002
targetPort: 10002
ambassador: 15005 #Port for ambassador service (must be from env.ambassadorPorts above)
rpc:
port: 30000
targetPort: 10003
rpcadmin:
port: 30009
targetPort: 10005
dbtcp:
port: 9101
targetPort: 1521
dbweb:
port: 8080
targetPort: 81
- notary:
name: notary-2
subject: "O=Notary,OU=Notary2,L=London,C=GB"
serviceName: "O=Notary Service,OU=Notary2,L=London,C=GB"
type: notary
validating: true
emailAddress: "dev@bevel.com"
p2p:
port: 10002
targetPort: 10002
ambassador: 15010 #Port for ambassador service (must be from env.ambassadorPorts above)
rpc:
port: 30000
targetPort: 10003
rpcadmin:
port: 30009
targetPort: 10005
dbtcp:
port: 9101
targetPort: 1521
dbweb:
port: 8080
targetPort: 81
# The second organization should be the new organization hosting new notary service.
# There can be multiple new notaries or orgs added
- organization:
name: add-notary
version: 4.7
cenm_version: 1.5
firewall_version: 4.4 # Supports 4.4
country: CH
state: Zurich
location: Zurich
subject: "O=NotaryAdd,OU=NotaryAdd,L=Zurich,C=CH"
type: notary
external_url_suffix: test.corda.blockchaincloudpoc.com
cloud_provider: aws # Options: aws, azure, gcp
aws:
access_key: "aws_access_key" # AWS Access key, only used when cloud_provider=aws
secret_key: "aws_secret_key" # AWS Secret key, only used when cloud_provider=aws
# Kubernetes cluster deployment variables. The config file path and name has to be provided.
k8s:
context: "cluster_context"
config_file: "cluster_config"
# Hashicorp Vault server address and root-token. Vault should be unsealed.
# Do not check-in root_token
vault:
url: "vault_addr"
root_token: "vault_root_token"
secret_path: "secretsv2"
# Git Repo details which will be used by GitOps/Flux.
# Do not check-in git_access_token
gitops:
git_protocol: "https" # Option for git over https or ssh
git_url: "https://github.com/<username>/bevel.git" # Gitops https url for flux value files
branch: "develop" # Git branch where release is being made
release_dir: "platforms/r3-corda-ent/releases/dev" # Relative Path in the Git repo for flux sync per environment.
chart_source: "platforms/r3-corda-ent/charts" # Relative Path where the Helm charts are stored in Git repo
username: "git_username" # Git Service user who has rights to check-in in all branches
password: "git_access_token" # Git Server user password/token (Optional for ssh; Required for https)
email: "git@email.com" # Email to use in git config
git_repo: "github.com/<username>/bevel.git" # Gitops git repository URL for git push
private_key: "path_to_private_key" # Path to private key file which has write-access to the git repo (Optional for https; Required for ssh)
# Cordapps Repository details (optional if cordapps jar are store in a repository)
cordapps:
jars:
- jar:
# e.g https://maven.pkg.github.com/hyperledger/bevel/com.supplychain.bcc.cordapp-supply-chain/4.7/cordapp-supply-chain-4.7.jar
url: "https://repo/path/cordapp1.jar"
- jar:
# e.g https://maven.pkg.github.com/hyperledger/bevel/com.supplychain.bcc.cordapp-contracts-states/4.7/cordapp-contracts-states-4.7.jar
url: "https://repo/path/cordapp2.jar"
username: "git_username"
password: "git_access_token"
# The participating nodes are named as peers
services:
notaries:
- notary:
name: notary3
subject: "O=Notary3,OU=Notary3,L=Zurich,C=CH"
serviceName: "O=NotaryAdd,OU=NotaryAdd,L=Zurich,C=CH"
type: notary
validating: true
emailAddress: "dev@bevel.com"
credentials:
truststore: trustpass
keystore: cordacadevpass
p2p:
port: 10002
targetPort: 10002
ambassador: 15010 #Port for ambassador service (must be from env.ambassadorPorts above)
rpc:
port: 10003
targetPort: 10003
rpcadmin:
port: 10003
targetPort: 10005
dbtcp:
port: 9101
targetPort: 1521
dbweb:
port: 8080
targetPort: 81
Run Playbook¶
After the new notary is running, execute the playbook [add-notaries.yaml] (https://github.com/hyperledger/bevel/blob/main/platforms/r3-corda-ent/configuration/add-notaries.yaml) with the same configuration file as used in previous step. This can be done using the following command
ansible-playbook platforms/r3-corda-ent/configuration/add-notaries.yaml --extra-vars "@path-to-new-network.yaml"
Run Parameter Update¶
The default networkparameters update timeout is 10 minutes, so wait for 10 minutes and then login to the networkmap ssh shell from the networkmap pod by running the commands below
#Login to networkmap pod
kubectl exec -it networkmap-0 -n <cenm-namespace> -c main -- bash
root@networkmap-0:/opt/corda# ssh nmap@localhost -p 2222 # say yes for hostkey message
Password authentication
Password: # Enter password at prompt
_ __ __ __ ___
/ | / /__ / /_/ |/ /___ _____
/ |/ / _ \/ __/ /|_/ / __ `/ __ \
/ /| / __/ /_/ / / / /_/ / /_/ /
/_/ |_/\___/\__/_/ /_/\__,_/ .___/
/_/
Welcome to the Network Map interactive shell.
Type 'help' to see what commands are available.
Thu Dec 03 17:40:37 GMT 2020>>> view notaries # to view current notaries
# Run the following commands to execute flagday so that latest network parameters update is accepted
Thu Dec 03 17:43:04 GMT 2020>>> view networkParametersUpdate # to check the current update (will be empty if no updates are in progress)
Thu Dec 03 17:43:57 GMT 2020>>> run flagDay # to initiate flagDay which will apply the networkParameters update only if the deadline has passed
# If you want to cancel the update, run following
Thu Dec 03 17:45:17 GMT 2020>>> run cancelUpdate
Ensure that the Corda Node users know that the network parameters have changed which will trigger node restart automatically.