sysadmin
@
extreme.indiana.edu


Home
Machines
CSG
    Support
    Packages
Lab
    User's Guide
    Sysadmin's Guide
    Packages
Projects
    Portal
    LEAD
    Grid Info

Service Restart Notes

Notification Broker Old production

Previous broker production version has two decoupled service components

Notification Broker server

Host tyr10.cs.indiana.edu
Account drlead
location /u/drlead/deployment/development/broker/apache-tomcat-5.5.16
Pre-conditions export CATALINA_HOME=/u/drlead/deployment/development/broker/apache-tomcat-5.5.16
MySQL server in tyr09.cs.indiana.edu is up and running
Startup script /u/drlead/deployment/development/broker/apache-tomcat-5.5.16/bin/startup.sh
Post-condition A HTTP server start running on tyr10.cs.indiana.edu:12346

Messenger

Host tyr10.cs.indiana.edu
Account drlead
location /u/drlead/deployment/development/broker/codes/xsul/messenger/java
Pre-conditions MySQL server in tyr09.cs.indiana.edu is up and running
Startup script nohup ./startMessenger.sh > messenger.log
Post-condition Logs will be writtent to messenger.txt


Broker and Messagebox Production

Broker production version has two decoupled service components

Notification Broker server

Host tyr11.cs.indiana.edu
Account drlead
location /u/drlead/broker_production/apache-tomcat-5.5.16
Pre-conditions export CATALINA_HOME=/u/drlead/broker_production/apache-tomcat-5.5.16
MySQL server in tyr09.cs.indiana.edu is up and running
Startup script /u/drlead/broker_production/apache-tomcat-5.5.16/bin/startup.sh
Post-condition A HTTP server start running on tyr11.cs.indiana.edu:12346
Logs will be writtent to the catalina.out of the above Tomcat Location

Messenger

Host tyr11.cs.indiana.edu
Account drlead
location /u/drlead/deployment/production/broker/codes/xsul/messenger/java
Pre-conditions MySQL server in tyr09.cs.indiana.edu is up and running
Startup script nohup ./startMessenger.sh > messenger.log
Post-condition Logs will be written to   /u/drlead/logs/messenger/log-messenger<date and time>.log
A HTTP server will run in 12347 port.

MessageBox

Host tyr11.cs.indiana.edu
Account drlead
location /u/drlead/msgbox-production/codes/xsul/java
Pre-conditions MySQL server in tyr09.cs.indiana.edu is up and running(databaseis msgbox)
Startup script nohup /u/drlead/msgbox-production/codes/xsul/java/startMsgBox.sh
Post-condition Logs will be written to /u/drlead/services/logs/log-dev-messagebox/log-dev-messagebox--<date and time>.log
A HTTP server will run in 13333 port.

Host Broker Service (Production)

ssh drlead@tyr14
export CATALINA_HOME=/u/drlead/deployment/production/hss/apache-tomcat-5.5.20
Shutdown: /u/drlead/deployment/production/hss/apache-tomcat-5.5.20/bin/shutdown.sh
Check if no tomcat process are hanging
Startup: /u/drlead/deployment/production/hss/apache-tomcat-5.5.20/bin/startup.sh

Database: tyr09: hostdbwithproperties
        
        

Host Broker Service (Development)

ssh drlead@tyr13
export CATALINA_HOME=/u/drlead/deployment/development/hss/apache-tomcat-5.5.20
Shutdown: /u/drlead/deployment/production/hss/apache-tomcat-5.5.20/bin/shutdown.sh
Check if no tomcat process are hanging
Startup: /u/drlead/deployment/development/hss/apache-tomcat-5.5.20/bin/startup.sh

        

 

Subscription Manager

Host rainier.extreme.indiana.edu
Account drlead
location /u/drlead/subscriptionManager/jakarta-tomcat-4.1.18/bin
Pre-conditions export CATALINA_HOME=/u/drlead/subscriptionManager/jakarta-tomcat-4.1.18/bin
MySQL server in tyr09.cs.indiana.edu is up and running
Startup script /u/drlead/subscriptionManager/jakarta-tomcat-4.1.18/bin/startup.sh
Post-condition A HTTP server will run in 16666 port.

Dynamic Service Creator (DSC) - Production

Host tyr09.cs.indiana.edu
Account drlead
location /u/drlead/deployment/production/dsc
Pre-conditions Kill the previous instance, if any. (ps xww | grep DSCServer)
Startup script nohup ./secure_dsc_on_tyr09.sh &
Post-condition Logs will be written to /u/drlead/logs/log-production-dsc/log-production-dsc-*.log. The service is runnin on port 25443.

Dynamic Service Creator (DSC) (Secure) - Development

Host tyr12.cs.indiana.edu
Account drlead
location /u/drlead/deployment/development/dsc
Pre-conditions Kill the previous instance, if any. (ps xww | grep DSCServer)
Startup script nohup ./secure_dsc_on_tyr12.sh &
Post-condition Logs will be written to /u/drlead/logs/log-development-dsc/log-development-dsc-*.log The service is running on port 25443.

LEAD Portal (Production)

To restart the lead portal, log into tyr09 as user drlead:

  ssh drlead@tyr09

  cd /data/lead-portal
  source bashrc
    

to shutdown:

  $CATALINA_HOME/bin/shutdown.sh
    

to start:

  $CATALINA_HOME/bin/startup.sh
    

restarting apache requires root privileges

  sudo /etc/init.d/lead-portal-httpd {stop|start|restart|status}
    

And since you have root privileges, you can also use the init scripts to start/stop/restart the portal

  sudo /etc/init.d/lead-portal {stop|start|restart|status} 
    

MsgBox (Production)

To start MsgBox, log into tyr10 as user drlead:

  ssh drlead@tyr10
    

Then run:

  cd ~/production-msgbox
  nohup msgBox.sh &
    

to shutdown, run the ps command to find the PID of the process, then kill it:

  ps wwaux | grep 13333
  kill <PID>
    

to restart just shut it down and then start it.

Resource Catalog (Production)

To restart the resource catalog on everest:
- Log onto everest using an admin account (in ADS domain, ysimmhan, machrist, smarru, gannon, bramley, CS System Staff)
- Go to Start->Administrative Tools->Services
- From the list of services, double click "LEAD ResCat"
- Click on the Stop button to stop the service
- Click on the Start button to start the service

To see the log messages:
- Open two command shells (Start->Run, cmd <enter>)
- Change to c:\data\extreme\repos\codes\resource_catalog
- On one window, type "tail -f wrapper.log"
- On the other window, type "tail -f rescat.out"

To see errors or other windows events published by the ResCat service:
- Go to Start->Administrative Tools->Event Viewer
- Double click on "Application" to see application events
- Sort the table on "Source" and look for events from the "rescat" source

To test the rescat service from a client:
- Run either of the test scripts from the rescat installation checked out from CVS on a client machine

test_service.sh:

test_service.bat:

- Check that no exceptions are seen
- verify that the last index time of the different catalogs is within the past 1 hour (UTC)
- Check that the last query returns about 15 results (XML documents) or more.
  
  

Resource Catalog (development)

1. Log onto whitney using drlead (localhost domain) account
2. To kill existing rescat service, go to the open command windows running the rescat command line and press Ctrl-C
3. Change dir to C:\extreme\repository\codes\resource_catalog-0.4.7 (if not already in that dir)
4. Run: run.bat rescat restart
5. To remove existing info (registrations/catalogs) from rescat and do a clean start, instead of #4, do:
run.bat rescat start
6. DON'T log off! Just lock the computer.
7. If you did a clean start, you will have to register the THREDDS catalog for crawling. From the rescat CVS checkout on
a client, type:
run[.sh|.bat] data_shell 
-url "http://whitney.extreme.indiana.edu:22002/resource_catalog?wsdl"
-addThredds 
-threddsUrl "http://lead4.unidata.ucar.edu:8080/thredds/lead/leadcatalog.xml"
-desc "LEAD UniData Root Catalog"
-freq 60

MyLEAD Agent (Production)

NOTE: Production agent is current on tyr01 using port 10243
Dev agent is on tyr03 using port 20243

1. cd to deployment/production/myleadagent-3.xxx

2. nohup ./run.sh agent 10243 &

  

MyLEAD Agent (Development)

1. cd to deployment/development/myleadagent-3.xxx

2. nohup ./run.sh agent 20243 &

  
  

XMC Cat/MyLEAD Server (Production)

On tyr01...
0. Set env var MYSQL_HOME to /usr/local/mysql
1. Before running the myLEAD server, we need to make sure the mySQL server 
   is running. If not, we can start it by doing the following:
     1) cd /usr/local/mysql/bin
     2) nohup sudo ./mysqld_safe --user=mysql &
2. go to /usr/local/xmccat (as drlead)
3. source the file named mylead-production-bash-env.sh
4. go to the subdirectory: /usr/local/xmccat/apache-tomcat-5.5.26
5.  run bin/startup.sh

Resource Broker

To start the service on dante:

1. ssh drlead@tyr11.cs.indiana.edu
2. cd /u/drlead/Resourcebroker.Service/resourcebroker
3. ./runResourceBroker.sh >> iu.log & 
   

Workflow Configuration Service (WCS) (Production)

Restart Instructions:

ssh drlead@tyr10
cd services/wcs
./wcs.sh &

To restart
ps -ef | grep wcs
If service is running kill it and then run shell script wcs.sh &
   

Workflow Configuration Service (WCS) (Development)

to start
DrLEAD@tyr12 [~]
$ cd services/dev-wcs/

DrLEAD@tyr12 [~/services/dev-wcs]
$ ./secure_wcs_service.sh&

To restart
ps -ef | grep wcs
If service is running kill it and then run shell script wcs.sh &
   

Calder (Development)

   
PORTS to be used for persistent planner service : 9982
PORT for the channer server is set inside setenv.sh : 20000

First:
ssh drlead@tyr04
cd CalderTest

To start all Calder services:
./start_demo.sh

The outputs of the services are store in /tmp/XXX in the respective
nodes as specified in start_demo.sh. Each run overwrites the previous log.
This is to make sure we don't run out of memory until crontab is setup.

To stop all Calder services:
./stop_demo.sh


Optionally, you can also start services in different terminals for
testing purposes, see ~/CalderTest/RUNNING_LEAD_DEMO for more details.
   

GFac (Development)

Location https://tyr12.cs.indiana.edu:23443/ 
directory:/u/drlead/deployment/development/sgfac2/target/dist-bin
logfile:/u/drlead/deployment/development/sgfac2/target/dist-bin/factory.log

How to restart

   1. Kill the running factory ps -ef|grep gfac
   2. cd /u/drlead/deployment/development/sgfac2/target/dist-bin
   3. export GFAC_HOME=.
   4. nohup ./factory.sh factory.properties > factory.log &

GFac (Production)

Location https://tyr09.cs.indiana.edu:23443/ 
directory:/u/drlead/deployment/production/sgfac2/target/dist-bin
logfile:/u/drlead/deployment/production/sgfac2/target/dist-bin/factory.log

How to restart

   1. Kill the running factory ps -ef|grep gfac
   2. cd /u/drlead/deployment/production/sgfac2/target/dist-bin
   3. export GFAC_HOME=.
   4. nohup ./factory.sh factory.properties > factory.log &

MyLEAD Publisher (Production)

To run mylead publisher service, please follow these steps:
 
1. log into tyr01 as drlead
 
2. go into the directory where the code resides:
   cd deployment/production/myleadpublisher-1.xxx
 
3. nohup ./run.sh publisher 11243 &
  

DaMN (Production)

Startup instructions:
1) login to drlead@tyr12

2) Go to the current installation of the DaMN service
cd ~drlead/deployment/production/damn-1.5

3) Start the service:
./run.sh damn_service &

Shutdown instructions:
1) Log onto drlead@tyr12

2) Search for the Damn Service process ID
ps -Aef | grep java | grep -i damn_service | grep -i 55443

3) Kill the process for the java process
kill -9 <process_id>

  

DaMN (Development)

Startup instructions:
1) Log onto kilimanjaro.extreme.indiana.edu in the drlead account
ssh -l drlead kilimanjaro.extreme.indiana.edu

2) Go to the current installation of the DaMN service
cd ~drlead/damn_service/development

3) Start the service:
./run.sh damn_service_secure

Shutdown instructions:
1) Log onto kilimanjaro.extreme.indiana.edu in the drlead account
ssh -l drlead kilimanjaro.extreme.indiana.edu

2) Search for the DamnService process
ps -Aef | grep java | grep -i damnservice | grep -i 50443

3) Kill the process for the java process
kill -9 <process_id>

  

Adaptation Manager (Development)

On tyr11:
  
cd /u/drlead/Resourcebroker.Service/adaptationManager
./manager.sh >> log &
  
  

Query Service (Production)

    It is on tyr02.
    cd /u/drlead/queryService/queryService-0.0.5
    ./run.sh qserv 
  

Geo GUI (Production)

Location: /u/drlead/deployment/production/geogui/

Restart instructions:
1) Shutdown: /u/drlead/deployment/production/geogui/apache-tomcat-5.5.20/bin/./shutdown.sh

2) Startup: /u/drlead/deployment/production/geogui/apache-tomcat-5.5.20/bin/./startup.sh 
  

Geo GUI (Development)

Location: /u/drlead/deployment/development/geogui-qm

Restart instructions:
1) Shutdown: /u/drlead/deployment/development/geogui-qm/apache-tomcat-5.5.20/bin/./shutdown.sh

2) Startup: /u/drlead/deployment/development/geogui-qm/apache-tomcat-5.5.20/bin/./startup.sh 
  

XPola (Development)

Url: http://tyr13.cs.indiana.edu:3456/
directory:/u/drlead/deployment/development/xpola/target/dist-bin

How to restart|
1. kill running xpola by ps-ef|grep xpola and kill $
2. cd  /u/drlead/deployment/development/xpola/target/dist-bin and $ nohup run.sh > xpola.log &
  
  

Data Catalog (Production)

1. cd deployment/production/data_catalog
2. nohup ./run.sh datacat_secure
3. ./run.sh add_new_uni
4. ./run.sh add_iu

Data Catalog (Development)

ssh -l drlead tyr12

*) To stop service:
ps -Af | grep 20443
kill -9 <process_id>

*) To start service:
cd rescat/data_catalog/development

nohup ./run.sh datacat restart -port 20443 -secure 2>&1 | /u/drlead/Linux/sbin/cronolog
/u/drlead/logs/log-data_catalog-dev/log-data_catalog_sec_dev-%Y-%m-%d.log &

  
  

XMC Cat/MyLEAD Server (Development)

The development MySQL must be running, but it is set to start when the server is started, so you should not need to start it.

1. go to ~/deployment/experiment/xmccat (as drlead on pagodatree)

2. source the file named mylead-experiment-bash-env.sh
3. go to the subdirectory: ~/deployment/experiment/xmccat/apache-tomcat-5.5.26/bin
4. run ./startup.sh

XRegistry (Development)

service URL: https://tyr14.cs.indiana.edu:6666/xregistry
wsdl URL: https://tyr14.cs.indiana.edu:6666/xregistry?wsdl
installation dir: /u/drlead/deployment/development/xregistry/target/dist-bin

Restart instructions:
1. kill running service by ~/bin/zap xregistry
2. cd to /u/drlead/deployment/development/xregistry/target/dist-bin
3. ./xregistry.sh
  

SCMS (Production)

service URL: https://tyr12.cs.indiana.edu:60443/SCMS
wsdl URL: https://tyr12.cs.indiana.edu:60443/SCMS?wsdl
installation dir: /u/drlead/deployment/production/scms/default

Restart instructions:
1. 'ps wwwaux | grep ScmsService | grep 60443' then kill that process id, e.g., "kill PID",
   don't do a "kill -9" or else you'll need to manually remove the database
   lock file in /data/scms/scmsdb_production.lck
2.  cd to /u/drlead/deployment/production/scms/default
3.  ./run_service.sh
  

SCMS (Development)

service URL: https://tyr12.cs.indiana.edu:62443/SCMS
wsdl URL: https://tyr12.cs.indiana.edu:62443/SCMS?wsdl
installation dir: /u/drlead/deployment/development/scms/scms-TRUNK

Restart instructions:
1. 'ps wwwaux | grep ScmsService | grep 62443' then kill that process id, e.g., "kill PID",
   don't do a "kill -9" or else you'll need to manually remove the database
   lock file in /data/scms/scmsdb.lck
2.  cd to /u/drlead/deployment/development/scms/scms-TRUNK
3.  ./run_service.sh
  

Karma (Development)

ssh drlead@tyr10
cd karma2/development
nohup ./run.sh edu.indiana.extreme.karma.impl.ProvenanceService -port 33003 -broker_url
http://tyr10.cs.indiana.edu:12346/ -info_model_ver 2.6 -msgbox_url http://tyr10.cs.indiana.edu:13333/MsgBox -mysql_url
jdbc:mysql://tyr11.cs.indiana.edu/karma2_dev -mysql_user karma -mysql_pass karma2007 2<&1 |
/u/drlead/Linux/sbin/cronolog /u/drlead/logs/log-dev-karma/log-dev-karma-%Y-%m-%d.log &

To stop the service:
ssh drlead@tyr10
ps -Af | grep karma  | grep 33003
kill -9 <process_id>  
  

XRegistry (Production)

service URL: https://tyr11.cs.indiana.edu:6666/xregistry
wsdl URL: https://tyr11.cs.indiana.edu:6666/xregistry?wsdl
installation dir: /u/drlead/deployment/production/xregistry/target/dist-bin
database: serviceReg in MySQL on tyr09

Restart instructions:
1. kill running service by ~/bin/zap xregistry
2. cd to /u/drlead/deployment/production/xregistry/target/dist-bin
3. ./xregistry.sh
  

GPEL (Production)

Restart GPEL

Host  tyr10.cs.indiana.edu, tyr13.cs.indiana.edu
Account  drlead
location  /u/drlead/deployment/production/gpel
Pre-conditions:
  1. on tyr13 kill the previous instance of pound, if any. (ps xww | grep
pound)
  2. on tyr10 shutdown mysqld (/u/drlead/deployment/production/gpel/stop-db.sh)
  3. on tyr10 kill all java JVM that run gpel (ps -ef|grep java|grep gpel)
  4. on tyr10 Startup script  nohup ./start-all.sh &
Post-condition  Logs will be written to /u/drlead/logs/gpel-*
The service is running on port 7443 (tyr13). Mysql is using port 7306 (tyr10)
  

Cleanup GPEL

Here are instructions on how I have deleted instances from the gpel db
    mysql -u -p 
    mysql> show tables
    mysql> select * from ge_state;
  
You can then delete the instance based on the ids you see. I automate it by using timestamp to find instances older than X hours. however it may not help that much as mysql database became very slow (executing of even simple select queries may take several seconds) - i am not sure why but it may be because it is using SAN directory --Alek

Query Mediator (Development)

    

  ssh drlead@pagodatree

  ps -ax|grep Query
  //kill that process
  
  cd /u/drlead/lead-prod-deployment/QueryMediator 
  nohup ./run.sh quemed 18874 &
  
 
 

Workflow Monitoring

Starting Monitoring Services

ssh tyr11 -l drlead
location  /u/drlead/deployment/production/workflowMonitor/codes/workflowMonitor/
nohup startworkflowmonitor.sh &
This should start up all the workflow monitoring services
  

Last updated 31 Mar 2006 by machrist@cs.indiana.edu