Service Restart Notes

• Production Services
  • Notification Broker
  • Messenger
  • Host Broker
  • Subscription Manager
  • LEAD Portal
  • MsgBox
  • Resource Catalog
  • MyLEAD Agent
  • XMC Cat/MyLEAD Server
  • Dynamic Service Creator (DSC)
  • GFac
  • DaMN
  • Calder
  • Adaptation Manager
  • MyLEAD Publisher
  • Resource Broker
  • WCS
  • Query Service
  • Geo GUI
  • Data Catalog
  • SCMS
  • XRegistry
  • WorkflowMonitoring
• Development Services
  • Notification Broker
  • MessageBox
  • Messenger
  • XMC Cat/MyLEAD Server
  • MyLEAD Agent
  • Resource Catalog
  • Dynamic Service Creator (DSC) (Secure)
  • GFac
  • GFac (Secure)
  • XPola
  • DaMN
  • WCS (Secure)
  • Host Broker
  • Data Catalog
  • XRegistry
  • SCMS
  • Query Mediator
  • Karma

Notification Broker Old production

Previous broker production version has two decoupled service components

Notification Broker server

Messenger

Broker and Messagebox Production

Broker production version has two decoupled service components

Notification Broker server

Messenger

MessageBox

Host Broker Service (Production)

ssh drlead@tyr14
export CATALINA_HOME=/u/drlead/deployment/production/hss/apache-tomcat-5.5.20
Shutdown: /u/drlead/deployment/production/hss/apache-tomcat-5.5.20/bin/shutdown.sh
Check if no tomcat process are hanging
Startup: /u/drlead/deployment/production/hss/apache-tomcat-5.5.20/bin/startup.sh

Database: tyr09: hostdbwithproperties
        
        

Host Broker Service (Development)

ssh drlead@tyr13
export CATALINA_HOME=/u/drlead/deployment/development/hss/apache-tomcat-5.5.20
Shutdown: /u/drlead/deployment/production/hss/apache-tomcat-5.5.20/bin/shutdown.sh
Check if no tomcat process are hanging
Startup: /u/drlead/deployment/development/hss/apache-tomcat-5.5.20/bin/startup.sh

        

Subscription Manager

Dynamic Service Creator (DSC) - Production

Dynamic Service Creator (DSC) (Secure) - Development

LEAD Portal (Production)

To restart the lead portal, log into tyr09 as user drlead:

  ssh drlead@tyr09

  cd /data/lead-portal
  source bashrc
    

to shutdown:

  $CATALINA_HOME/bin/shutdown.sh
    

to start:

  $CATALINA_HOME/bin/startup.sh
    

restarting apache requires root privileges

  sudo /etc/init.d/lead-portal-httpd {stop|start|restart|status}
    

And since you have root privileges, you can also use the init scripts to start/stop/restart the portal

  sudo /etc/init.d/lead-portal {stop|start|restart|status} 
    

MsgBox (Production)

To start MsgBox, log into tyr10 as user drlead:

  ssh drlead@tyr10
    

Then run:

  cd ~/production-msgbox
  nohup msgBox.sh &
    

to shutdown, run the ps command to find the PID of the process, then kill it:

  ps wwaux | grep 13333
  kill <PID>
    

to restart just shut it down and then start it.

Resource Catalog (Production)

To restart the resource catalog on everest:
- Log onto everest using an admin account (in ADS domain, ysimmhan, machrist, smarru, gannon, bramley, CS System Staff)
- Go to Start->Administrative Tools->Services
- From the list of services, double click "LEAD ResCat"
- Click on the Stop button to stop the service
- Click on the Start button to start the service

To see the log messages:
- Open two command shells (Start->Run, cmd <enter>)
- Change to c:\data\extreme\repos\codes\resource_catalog
- On one window, type "tail -f wrapper.log"
- On the other window, type "tail -f rescat.out"

To see errors or other windows events published by the ResCat service:
- Go to Start->Administrative Tools->Event Viewer
- Double click on "Application" to see application events
- Sort the table on "Source" and look for events from the "rescat" source

To test the rescat service from a client:
- Run either of the test scripts from the rescat installation checked out from CVS on a client machine

test_service.sh:
#!/bin/sh set RESCAT_URL=http://everest.extreme.indiana.edu:22000/resource_catalog?wsdl echo ===== testing service catalog at $RESCAT_URL ====== ./run.sh service_shell -url $RESCAT_URL -lsAwsdl ./run.sh service_shell -url $RESCAT_URL -lsCwsdl ./run.sh service_shell -url $RESCAT_URL -lsSm ./run.sh service_shell -url $RESCAT_URL -lsApp ./run.sh service_shell -url $RESCAT_URL -lsHost echo ===== testing data catalog at $RESCAT_URL$ ====== ./run.sh data_shell -url $RESCAT_URL -lsThreddsCols ./run.sh data_shell -url $RESCAT_URL -listCats ./run.sh data_shell -url $RESCAT_URL -queryTimeFalls -from -4 -queryKeywords "grib"

test_service.bat:
set RESCAT_URL=http://everest.extreme.indiana.edu:22000/resource_catalog?wsdl echo ===== testing service catalog at %RESCAT_URL% ====== call run service_shell -url %RESCAT_URL% -lsAwsdl call run service_shell -url %RESCAT_URL% -lsCwsdl call run service_shell -url %RESCAT_URL% -lsSm call run service_shell -url %RESCAT_URL% -lsApp call run service_shell -url %RESCAT_URL% -lsHost echo ===== testing data catalog at %RESCAT_URL% ====== call run data_shell -url %RESCAT_URL% -lsThreddsCols call run data_shell -url %RESCAT_URL% -listCats call run data_shell -url %RESCAT_URL% -queryTimeFalls -from -4 -queryKeywords "grib"

- Check that no exceptions are seen
- verify that the last index time of the different catalogs is within the past 1 hour (UTC)
- Check that the last query returns about 15 results (XML documents) or more.
  
  

Resource Catalog (development)

1. Log onto whitney using drlead (localhost domain) account
2. To kill existing rescat service, go to the open command windows running the rescat command line and press Ctrl-C
3. Change dir to C:\extreme\repository\codes\resource_catalog-0.4.7 (if not already in that dir)
4. Run: run.bat rescat restart
5. To remove existing info (registrations/catalogs) from rescat and do a clean start, instead of #4, do:
run.bat rescat start
6. DON'T log off! Just lock the computer.
7. If you did a clean start, you will have to register the THREDDS catalog for crawling. From the rescat CVS checkout on
a client, type:
run[.sh|.bat] data_shell 
-url "http://whitney.extreme.indiana.edu:22002/resource_catalog?wsdl"
-addThredds 
-threddsUrl "http://lead4.unidata.ucar.edu:8080/thredds/lead/leadcatalog.xml"
-desc "LEAD UniData Root Catalog"
-freq 60

MyLEAD Agent (Production)

NOTE: Production agent is current on tyr01 using port 10243
Dev agent is on tyr03 using port 20243

1. cd to deployment/production/myleadagent-3.xxx

2. nohup ./run.sh agent 10243 &

  

MyLEAD Agent (Development)

1. cd to deployment/development/myleadagent-3.xxx

2. nohup ./run.sh agent 20243 &

  
  

XMC Cat/MyLEAD Server (Production)

0. Set env var MYSQL_HOME to /usr/local/mysql
1. Before running the myLEAD server, we need to make sure the mySQL server 
   is running. If not, we can start it by doing the following:
     1) cd /usr/local/mysql/bin
     2) nohup sudo ./mysqld_safe --user=mysql &
2. go to /usr/local/xmccat (as drlead)
3. source the file named mylead-production-bash-env.sh
4. go to the subdirectory: /usr/local/xmccat/apache-tomcat-5.5.26
5.  run bin/startup.sh

Resource Broker

To start the service on dante:

1. ssh drlead@tyr11.cs.indiana.edu
2. cd /u/drlead/Resourcebroker.Service/resourcebroker
3. ./runResourceBroker.sh >> iu.log & 
   

Workflow Configuration Service (WCS) (Production)

Restart Instructions:

ssh drlead@tyr10
cd services/wcs
./wcs.sh &

To restart
ps -ef | grep wcs
If service is running kill it and then run shell script wcs.sh &
   

Workflow Configuration Service (WCS) (Development)

to start
DrLEAD@tyr12 [~]
$ cd services/dev-wcs/

DrLEAD@tyr12 [~/services/dev-wcs]
$ ./secure_wcs_service.sh&

To restart
ps -ef | grep wcs
If service is running kill it and then run shell script wcs.sh &
   

Calder (Development)

   
PORTS to be used for persistent planner service : 9982
PORT for the channer server is set inside setenv.sh : 20000

First:
ssh drlead@tyr04
cd CalderTest

To start all Calder services:
./start_demo.sh

The outputs of the services are store in /tmp/XXX in the respective
nodes as specified in start_demo.sh. Each run overwrites the previous log.
This is to make sure we don't run out of memory until crontab is setup.

To stop all Calder services:
./stop_demo.sh


Optionally, you can also start services in different terminals for
testing purposes, see ~/CalderTest/RUNNING_LEAD_DEMO for more details.
   

GFac (Development)

Location https://tyr12.cs.indiana.edu:23443/ 
directory:/u/drlead/deployment/development/sgfac2/target/dist-bin
logfile:/u/drlead/deployment/development/sgfac2/target/dist-bin/factory.log

How to restart

   1. Kill the running factory ps -ef|grep gfac
   2. cd /u/drlead/deployment/development/sgfac2/target/dist-bin
   3. export GFAC_HOME=.
   4. nohup ./factory.sh factory.properties > factory.log &

GFac (Production)

Location https://tyr09.cs.indiana.edu:23443/ 
directory:/u/drlead/deployment/production/sgfac2/target/dist-bin
logfile:/u/drlead/deployment/production/sgfac2/target/dist-bin/factory.log

How to restart

   1. Kill the running factory ps -ef|grep gfac
   2. cd /u/drlead/deployment/production/sgfac2/target/dist-bin
   3. export GFAC_HOME=.
   4. nohup ./factory.sh factory.properties > factory.log &

MyLEAD Publisher (Production)

To run mylead publisher service, please follow these steps:
 
1. log into tyr01 as drlead
 
2. go into the directory where the code resides:
   cd deployment/production/myleadpublisher-1.xxx
 
3. nohup ./run.sh publisher 11243 &
  

DaMN (Production)

Startup instructions:
1) login to drlead@tyr12

2) Go to the current installation of the DaMN service
cd ~drlead/deployment/production/damn-1.5

3) Start the service:
./run.sh damn_service &

Shutdown instructions:
1) Log onto drlead@tyr12

2) Search for the Damn Service process ID
ps -Aef | grep java | grep -i damn_service | grep -i 55443

3) Kill the process for the java process
kill -9 <process_id>

  

DaMN (Development)

Startup instructions:
1) Log onto kilimanjaro.extreme.indiana.edu in the drlead account
ssh -l drlead kilimanjaro.extreme.indiana.edu

2) Go to the current installation of the DaMN service
cd ~drlead/damn_service/development

3) Start the service:
./run.sh damn_service_secure

Shutdown instructions:
1) Log onto kilimanjaro.extreme.indiana.edu in the drlead account
ssh -l drlead kilimanjaro.extreme.indiana.edu

2) Search for the DamnService process
ps -Aef | grep java | grep -i damnservice | grep -i 50443

3) Kill the process for the java process
kill -9 <process_id>

  

Adaptation Manager (Development)

On tyr11:
  
cd /u/drlead/Resourcebroker.Service/adaptationManager
./manager.sh >> log &
  
  

Query Service (Production)

    It is on tyr02.
    cd /u/drlead/queryService/queryService-0.0.5
    ./run.sh qserv 
  

Geo GUI (Production)

Location: /u/drlead/deployment/production/geogui/

Restart instructions:
1) Shutdown: /u/drlead/deployment/production/geogui/apache-tomcat-5.5.20/bin/./shutdown.sh

2) Startup: /u/drlead/deployment/production/geogui/apache-tomcat-5.5.20/bin/./startup.sh 
  

Geo GUI (Development)

Location: /u/drlead/deployment/development/geogui-qm

Restart instructions:
1) Shutdown: /u/drlead/deployment/development/geogui-qm/apache-tomcat-5.5.20/bin/./shutdown.sh

2) Startup: /u/drlead/deployment/development/geogui-qm/apache-tomcat-5.5.20/bin/./startup.sh 
  

XPola (Development)

Url: http://tyr13.cs.indiana.edu:3456/
directory:/u/drlead/deployment/development/xpola/target/dist-bin

How to restart|
1. kill running xpola by ps-ef|grep xpola and kill $
2. cd  /u/drlead/deployment/development/xpola/target/dist-bin and $ nohup run.sh > xpola.log &
  
  

Data Catalog (Production)

1. cd deployment/production/data_catalog
2. nohup ./run.sh datacat_secure
3. ./run.sh add_new_uni
4. ./run.sh add_iu

Data Catalog (Development)

ssh -l drlead tyr12

*) To stop service:
ps -Af | grep 20443
kill -9 <process_id>

*) To start service:
cd rescat/data_catalog/development

nohup ./run.sh datacat restart -port 20443 -secure 2>&1 | /u/drlead/Linux/sbin/cronolog
/u/drlead/logs/log-data_catalog-dev/log-data_catalog_sec_dev-%Y-%m-%d.log &

  
  

XMC Cat/MyLEAD Server (Development)

The development MySQL must be running, but it is set to start when the server is started, so you should not need to start it.

1. go to ~/deployment/experiment/xmccat (as drlead on pagodatree)

2. source the file named mylead-experiment-bash-env.sh
3. go to the subdirectory: ~/deployment/experiment/xmccat/apache-tomcat-5.5.26/bin
4. run ./startup.sh

XRegistry (Development)

service URL: https://tyr14.cs.indiana.edu:6666/xregistry
wsdl URL: https://tyr14.cs.indiana.edu:6666/xregistry?wsdl
installation dir: /u/drlead/deployment/development/xregistry/target/dist-bin

Restart instructions:
1. kill running service by ~/bin/zap xregistry
2. cd to /u/drlead/deployment/development/xregistry/target/dist-bin
3. ./xregistry.sh
  

SCMS (Production)

service URL: https://tyr12.cs.indiana.edu:60443/SCMS
wsdl URL: https://tyr12.cs.indiana.edu:60443/SCMS?wsdl
installation dir: /u/drlead/deployment/production/scms/default

Restart instructions:
1. 'ps wwwaux | grep ScmsService | grep 60443' then kill that process id, e.g., "kill PID",
   don't do a "kill -9" or else you'll need to manually remove the database
   lock file in /data/scms/scmsdb_production.lck
2.  cd to /u/drlead/deployment/production/scms/default
3.  ./run_service.sh
  

SCMS (Development)

service URL: https://tyr12.cs.indiana.edu:62443/SCMS
wsdl URL: https://tyr12.cs.indiana.edu:62443/SCMS?wsdl
installation dir: /u/drlead/deployment/development/scms/scms-TRUNK

Restart instructions:
1. 'ps wwwaux | grep ScmsService | grep 62443' then kill that process id, e.g., "kill PID",
   don't do a "kill -9" or else you'll need to manually remove the database
   lock file in /data/scms/scmsdb.lck
2.  cd to /u/drlead/deployment/development/scms/scms-TRUNK
3.  ./run_service.sh
  

Karma (Development)

ssh drlead@tyr10
cd karma2/development
nohup ./run.sh edu.indiana.extreme.karma.impl.ProvenanceService -port 33003 -broker_url
http://tyr10.cs.indiana.edu:12346/ -info_model_ver 2.6 -msgbox_url http://tyr10.cs.indiana.edu:13333/MsgBox -mysql_url
jdbc:mysql://tyr11.cs.indiana.edu/karma2_dev -mysql_user karma -mysql_pass karma2007 2<&1 |
/u/drlead/Linux/sbin/cronolog /u/drlead/logs/log-dev-karma/log-dev-karma-%Y-%m-%d.log &

To stop the service:
ssh drlead@tyr10
ps -Af | grep karma  | grep 33003
kill -9 <process_id>  
  

XRegistry (Production)

service URL: https://tyr11.cs.indiana.edu:6666/xregistry
wsdl URL: https://tyr11.cs.indiana.edu:6666/xregistry?wsdl
installation dir: /u/drlead/deployment/production/xregistry/target/dist-bin
database: serviceReg in MySQL on tyr09

Restart instructions:
1. kill running service by ~/bin/zap xregistry
2. cd to /u/drlead/deployment/production/xregistry/target/dist-bin
3. ./xregistry.sh
  

GPEL (Production)

Restart GPEL

Host  tyr10.cs.indiana.edu, tyr13.cs.indiana.edu
Account  drlead
location  /u/drlead/deployment/production/gpel
Pre-conditions:
  1. on tyr13 kill the previous instance of pound, if any. (ps xww | grep
pound)
  2. on tyr10 shutdown mysqld (/u/drlead/deployment/production/gpel/stop-db.sh)
  3. on tyr10 kill all java JVM that run gpel (ps -ef|grep java|grep gpel)
  4. on tyr10 Startup script  nohup ./start-all.sh &
Post-condition  Logs will be written to /u/drlead/logs/gpel-*
The service is running on port 7443 (tyr13). Mysql is using port 7306 (tyr10)
  

Cleanup GPEL

    mysql -u -p 
    mysql> show tables
    mysql> select * from ge_state;
  

Query Mediator (Development)

    

  ssh drlead@pagodatree

  ps -ax|grep Query
  //kill that process
  
  cd /u/drlead/lead-prod-deployment/QueryMediator 
  nohup ./run.sh quemed 18874 &
  
 
 

Workflow Monitoring

Starting Monitoring Services

ssh tyr11 -l drlead
location  /u/drlead/deployment/production/workflowMonitor/codes/workflowMonitor/
nohup startworkflowmonitor.sh &
This should start up all the workflow monitoring services