PSOFT : Issues/errors while booting PeopleSoft Application Server

We often face trouble while booting application server and we have to troubleshoot/search/try-catch to find workaround.
This is frequent & error prone task for any psadmin,
Hence I am putting my workarounds here and some of the cases which I found helpful on oracle support here hoping to reduce your time in solving these issues.

I'll be glad to help you for your specific errors which are not covered in this article.


Possible errors & workarounds while booting PeopleSoft Application Server.

---------------------------------------------->>>>>>> ISSUE #01 <<<<<<<-----------------------------------------------------

The application server fails to start due to errors such as

Database Signon: Could not sign on to database HRPRD with user PSAPPS or
Invalid User ID or Password for SignOn ; logon denied. or
Invalid Access ID or Password for Signon - see your security administrator.

Steps to check : 

a) Check Connect ID and Connect Password Values are correct one.(default : people/peop1e)
    Connect ID and Connect Password are valid Oracle login & id is not locked out
    (You should be able to login with connectid/connectpass to database)
    Connect ID login has SELECT privilege on PSOPRDEFN,PSACCESSPRFL,PSSTATUS.

b) Make sure OPRID/password that is used to boot app server is correct.
    Make sure OPRID is not locked in PSOPRDEFN table. ( Select ACCTLOCK from PSOPRDEFN )
    The SymbolicID field in PSOPRDEFN & PSACCESSPRFL is same.

c) Make sure AccessID/Access password in PSACCESSPRFL are correct and the values are encrypted.
    Make sure You are able to login to Database with Accessid/pass
    Pls make sure sysadm(accessid for oracle) is not locked out or it's passwd is not expired

d) Analyse appsrv/tuxlog files.

---------------------------------------------->>>>>>> ISSUE #02 <<<<<<<-----------------------------------------------------

Trying to boot an application server domain, it fails to start. TuxLog shows the following messages

JOLT_CAT:1079: ERROR: Error starting minimum number of handlers
LIBTUX_CAT:250: ERROR: tpsvrinit() failed

Solution :
First make sure there is no port conflicts of your JSL and JSH ports.
You can issue a 'netstat -an' and grep for the port that you are trying to use.
netstat -an | grep portnumber)
Keep in mind that JSH ports always start right after the JSL port.
This error will occur when a JSL or JSH process is not able to capture it's assigned port.
If you have multiple domains running on the same box, make sure that the ports are
not already being used by another process.
You will need to ensure that the number of free ports after the JSL port
is more than the JSL port + Max Handler value set in the psappsrv.cfg file in the Jolt section.

You can use the command 'netstat -an' to see if a port is currently in use.

The OS kernel parameter settings is the next thing to check.
If you bring down one of the existing domains,attempt to boot the domain that originally failed,
and it works, most likely you need to tune the kernel parameters.

---------------------------------------------->>>>>>> ISSUE #03 <<<<<<<-----------------------------------------------------

Unable to boot the appserver domain. The error seen in APPSRV_MMDD.LOG file is:

Database Signon: Invalid access ID or password for database signon.
Make sure your ODBC data source is configured under the same user as the tuxipc service.
Consult the PeopleSoft Install/Admin guide for more information."

Solution : 

CASE 1->
1. Start the Tuxedo service BEA Proc Manager as the account you used to install Tuxedo.
This should be the same account the boots the application server domain.
2. The ODBC entry must be defined under 'System DSN'.

CASE 2->
This issue is caused by how Microsoft implemented the Windows registry virtualization
on 64-bit Windows platform to support running 32-bit and 64-bit code on the same system.


Since PeopleTools and Tuxedo are 32-bit applications,
you need to configure the ODBC data source using the 32-bit Windows control panel.

1. Open the 64-bit Control Panel and launch "View 32-bit Control Panel Items".
2. Start the "ODBC Data Sources" configuration tool from the 32-bit control panel items.
3. Continue to configure to setup the ODBC data source using the System DSN.

---------------------------------------------->>>>>>> ISSUE #04 <<<<<<<-----------------------------------------------------

On new install of PplTools 8.49,unable to boot the application server domain on AIX Unix system

Errors : PSAPPSRV LOG shows :
PS General SQL Routines: Missing or invalid version of SQL library libpsora (200,0) 
Database Signon: Could not sign on to database FTST with user PSAPPS.

StdErr shows :
open in libpscompat failed for 'libpsora.so':/oracle/10.2.0.3/lib32/libclntsh.so.9.0:file too short

Cause :

The cause of the problem is the incorrect size of Oracle library in use.

<xxxxxxxxxxx: test></oracle/10.2.0.3/lib32> ls -l /oracle/10.2.0.3/lib32/libclnt*
lrwxrwxrwx 1 oracle dba 85   Feb 11 10:56 /oracle/10.2.0.3/lib32/libclntsh.so -> libclntsh.so.10.1
-rwxrwxr-x 1 oracle dba 0    Feb 01 02:18 /oracle/10.2.0.3/lib32/libclntsh.so.10.1

Here, -rwxrwxr-x 1 oracle dba 0    Feb 01 02:18 /oracle/10.2.0.3/lib32/libclntsh.so.10.1
shows that the file everything is pointed to has 0 bytes

Solution :

1. Run the following as the UNIX account that boots the Application Server:

ls -l $ORACLE_HOME/lib32/libclnt*

Review the listing results to determine if the libclntsh soft links are accurate
and that the files/links returned have some size value.

2. Output should show files with a size greater than 0 bytes and not like below:

<xxxxxxxxx: test></oracle/10.2.0.3/lib32> ls -l /oracle/10.2.0.3/lib32/libclnt*
lrwxrwxr-- 1 oracle dba 85   Feb 11 10:56 /oracle/10.2.0.3/lib32/libclntsh.so -> libclntsh.so.10.1
-rwxrwxr-- 1 oracle dba 0    Feb 01 02:18 /oracle/10.2.0.3/lib32/libclntsh.so.10.1

3. If the Oracle client lib has a size of '0' then you will need to re-create it
using the Oracle DB delivered '$ORACLE_HOME/bin/genclntsh' script using the UNIX Oracle account.


4 .  To resolve the issue install the 32 bit Oracle client on your application server machine
and setup your environment to only find the 32 bit libraries.
Make sure ORACLE_HOME, PATH, LD_LIBRARY_PATH are all configured to find 32 bit Oracle libraries.

---------------------------------------------->>>>>>> ISSUE #05 <<<<<<<-----------------------------------------------------

while trying to logon to Application Designer in 3-tier mode getting below error message:

Could not connect to application server FSDEV. 
Make sure the PeopleTools authentication server (PSAUTH) is booted.

Cause :
Not having the tuxedo folder under the <PSHOME> where
Application Designer is launched from will cause this issue.

Solution :

The tuxedo client needs to be under the <PSHOME> where you launch the executable for App Designer,
which is usually your file server.
The error is related to files missing on file server where Application Designer was started.
Make sure the tuxedo folder exists under this <PSHOME>.
Also,you must have the following four files installed -
PS_HOME/tuxedo/udataobj/Lic.txt, SysRegiiop.rdp, SysRegtgiop.rdp, System.rdp.

For PeopleTools 8.50 and higher the only files you will find under PS_HOME/tuxedo/udataobj are
SysRegiiop.rdp, and System.rdp, which is all that are needed.

Some users copy the <PSHOME> to their PC to have the executables local on their machine.
If you do this make sure you include the tuxedo folder and all files that reside under it.

You can also get this error if your Domain Connection Password in Configuration Manager
does not match the password for DomainConnectionPwd on your web server in the
configuration.properties file

---------------------------------------------->>>>>>> ISSUE #06 <<<<<<<-----------------------------------------------------

When you start the Application Server, the PSRENSRV process fails to start.
The following error is logged to APPSRV_MMDD.log:

REN configuration file not found.

Also, the following error may be logged to PSRENSRV_mmdd.LOG

(FATAL) nsmain: configuration file empty or missing ns/servers section (162,2076)
(ERROR) Server failed to start

Steps
1. Start Application Server (with 'Event Notification' enabled)
2. Application Server fails to start with message:

exec PSRENSRV -A -- -C psappsrv.cfg -D TESTSERV -S PSRENSRV :
CMDTUX_CAT:1685: ERROR: Application initialization failure
tmboot: CMDTUX_CAT:827: ERROR: Fatal error encountered; initiating user error handler
Cause

The REN configuration file (psrenconfig.txt) is missing or corrupted.

Solution :

Follow these steps in order to correct problems with the REN configuration file
in your PeopleTools environment:

1. A default REN configuration file is delivered when PeopleTools is installed.
   Whenever a new application server domain is created, it is populated with this file.
   So we need to make sure the file exists, and is not corrupted, as follows:
  a. Go to <PS_HOME>/appserv/ren.
  b. Within this directory, there should be a file called psrenconfig.txt
  c. If the file exists, make sure if it not corrupted, as follows:
       -Open file and it should be readable (this is a text file).
       -Verify that the last line is '#### END OF FILE #####'.
  d. If file does not exist, or is corrupted, then get a new copy from your DEMO system.

2. Next, place a new copy of psrenconfig.txt into the application server domain
   where you are getting the PSRENSRV error, as follows:
   a. Copy psrenconfig.txt
      From: <PS_HOME>/appserv/ren
      To:     <PS_CFG_HOME>/appserv/<DOMAIN_NAME>.
   b.  If you had made any customizations to the previous psrenconfig.txt file,
       you'll need to remake the changes.

3. After making the above changes, restart the application server.

---------------------------------------------->>>>>>> ISSUE #07 <<<<<<<-----------------------------------------------------
Applicable to : PT 8.51 and later
App server domain is having an issue starting up & the TUXLOG file is showing the following errors:

GWADM.10104.1.0: 12-09-2010: Tuxedo Version 10.3.0.0, 64-bit
GWADM.10104.1.0: LIBTUX_CAT:262: INFO: Standard main starting
DMADM.10103.1.0: CMDGW_CAT:3250: ERROR: Cannot send config to GWADM group=GWTGROUP
GWADM.10104.1.0: LIBTUX_CAT:250: ERROR: tpsvrinit() failed
tmboot.10102.1.-2: CMDTUX_CAT:825: ERROR: Process GWADM at g3u0181c failed with /T tperrno
(TPESVCFAIL - application level service failure)
tmboot: CMDTUX_CAT:827: ERROR: Fatal error encountered; initiating user error handler

Cause :
Upon checking the configuration template via PSADMIN.
The Domain Gateway was found out to be enabled = "YES"
which will cause the server to search for a remote server
to which this domain will send search requests.

Solution :

To resolve the issue, refer to the details below:

1. open PSADMIN
2. go to Configure this Domain
3. enter 12 to disable the Domain Gateway
4. enter 13 to configure the domain and save the changes made

After the changes, the Application Server domain booted without any problem

---------------------------------------------->>>>>>> ISSUE #08 <<<<<<<-----------------------------------------------------

trying to boot application server domain or process scheduler receive errors like the following:
PSAPPSRV LOG
GenMessageBox(200, 0, M): SQL Access Manager: SQL error. Stmt #: 2 Error Position: 0 
Return: 12541 - ORA-12541: TNS:no listener
GenMessageBox(0, 0, M): Database Signon: Could not sign on to database FSYS with user PSAPPS.

Solution :

ORA-12541: TNS:no listener means the Oracle database listener is not started.
Check to ensure this is started with the Oracle DBA.
(you can check listner service using lsnrctl on solaris)
Also make sure tnsnames.ora has correct entries.

---------------------------------------------->>>>>>> ISSUE #09 <<<<<<<-----------------------------------------------------

PSAPPSRV Fails to Boot - Return: 8097 - Oracle initialization error

Errors  :Database Signon: Could not sign on to database HCMV with  
     user PSAPPS.
    Return: 8097 - Oracle initialization error

Solution: 
Case #1- Same error message. Tnsping was able to connect to the DB OK.
The application server user on the Unix box needs read accesss to all files
under $ORACLE_HOME.Please make sure that.

Case #2 - if Appserver is on 32bit Windows & Oracle client installed is 11.2.
This Oracle client version was never certified for 32bit Windows OS.
Installing Oracle client 11.1 or lower resolves the problem.

Oracle Client 11.2 is certified for Windows 64bit OS only.

---------------------------------------------->>>>>>> ISSUE #10 <<<<<<<-----------------------------------------------------

Platform : AIX
Getting the following error messages in the APPSRV_MMDD.LOG log on PeopleTools 8.48+:

PSPAL: Abort: Unrecoverable exception 
Memory allocation failed (SQLSTATE PS118)
CMgrImp::WriteCache : Cache write failed

Solution :

Check the output of 'ulimit -a' command while logged into your AIX system
as the user that you use to boot the domain.

It may give you results like below, which are the older AIX defaults:

time(seconds)         unlimited
file(blocks)             2097151
data(kbytes)           131072
stack(kbytes)          32768
memory(kbytes)      32768
coredump(blocks)   2097151
nofiles(descriptors)  2000

From the 'ulimit -a' output above, it the limitations are set too low for PeopleTools.

The out of memory problems are due to the low setting for the 'data' limit -
This should be set to unlimited.
Additionally, the 'nofiles' limit should be raised to at least 4000.

Increase these limits, logout, log back in, verify the limits,
as the user that will boot the application server domain,
have been raised by running 'ulimit -a', then restart the domain.

Note : this setting also helps if you are being thrown out of PIA multiple times & if you are getting below error in app logs

/vob/peopletools/src/pssys/crmget.cppSQL error. Stmt #: 1066  Error Position: 28  Return: 8007 - Insufficient memory available

You can also change the ulimit settings to unlimited except for codedump parameter.

After changing the limits you need to log out and back into your Unix account and restart the process scheduler

---------------------------------------------->>>>>>> ISSUE #11 <<<<<<<-----------------------------------------------------

(Specific to AIX) When trying to configure a large app server domain,
the following error is received in one of the last JSL processes:"

ERROR: Error starting mininum number of handlers"
LIBTUX_CAT:250: ERROR: tpsvrinit() failed
tmboot: CMDTUX_CAT:827: ERROR: Fatal error encountered; initiating user error handler
NLS:6: Message not found, catalog CMDTUX_CAT, set 1, num 724
PSADMIN.741496: End boot attempt on domain FSPRD

There are no issues if building a smaller domain.

Cause :
The AIX operating system has a system parameter that limits
the maximum number of processes allowed per user, to 128.
This default value may not be sufficient for large domains.

Solution :
As root, run "smit chgsys" and
increase the "Maximum number of PROCESSES allowed per user" parameter to 512 or higher.

---------------------------------------------->>>>>>> ISSUE #12 <<<<<<<-----------------------------------------------------

After an upgrade to PeopleTools 8.48 or higher, the 3-tier signon is failing.
There are no problems logging into 2 tier Application Designer.

Error receiving:
Network API can not connect to server-
Could not connect to application server FSPRD (//NJserver:7060).
Possible causes are: the server name/IP address and port for the app server alias are incorrect,
the application server is not booted, or the network is unreachable.
Contact your system administrator or check the Tuxedo log for more information.

Solution :

There is a new option starting with PeopleTools 8.48 to start the WSL process to allow 3-tier access.
If you are sure the Configuration Manager is rightly configured,
check your setup using psadmin for WSL as follows:
Do you see a Yes on WSL? (If not, please change this to Yes)
And does the Configuration Manager show the same port for WSL as on PSADMIN?

WSL port is needed for the 3 tier connection, which needs to be up and running and the correct one.

Features                       Settings
                       
1) Pub/Sub Servers  : No      15) DBNAME     :[FSPRD]
2) Quick Server       : No      16) DBTYPE       :[ORACLE]
3) Query Servers     : No      17) UserId            :[PSOPER1]
4) Jolt                      : Yes     18) UserPswd       :[ydhjdn]
5) Jolt Relay            : No      19) DomainID        :[FSPRD]
6) WSL                   : No      20) AddToPATH  :[E:\Oracle\ora920\bin]
7) PC Debugger       : No     21) ConnectID       :[people]
8) Event Notification: No      22) ConnectPswd:[peop1e]
9) MCF Servers       : No     23) ServerName :[]
10) Perf Collator       : Yes   24) WSL Port   :[7060]

Once this was done, the 3-tier login should be successful.

---------------------------------------------->>>>>>> ISSUE #13 <<<<<<<-----------------------------------------------------

When attempting to boot an application server domain with PUBSUB enabled
the following error occurs in the appserv_mmdd.log file:

(JNIUTIL): Java exception thrown: java.lang.OutOfMemoryError

NOTE: The error can occur not only for PSPUBDSP_dflt but for any of the PUBSUB
services such as
PSPUBDSP_dflt, PSPUBHND_dflt, PSBRKDSP_dflt, PSBRKHND_dflt, PSSUBDSP_dflt, PSSUBHND_dflt

Cause :
The problem is caused by insufficient Java memory space to boot the services.
as described by the Java error:  (JNIUTIL): Java exception thrown: java.lang.OutOfMemoryError.

This memory refers to the amount of java memory,
as allocated to the application server java heap setting.

Solution :

Increase the java heap setting in the psappsrv.cfg file

Steps:

1) Open the psappsrv.cfg file on the application server

2) Find the JavaVM Options setting in following section:

[PSTOOLS]
; General settings for PSTOOLS
...
;To pass java options to the jvm loaded via JNI, uncomment JavaVM Options=
;and set it equal to the list of parameters you would like to use.
;The options need to be a blank space separated list of valid java options.
;Consult your JRE vendor for valid java options.
;For example, JavaVM Options=-Xmx256m -Xms128m
JavaVM Options=-Xrs -Dxdo.ConfigFile=../xdo.cfg

3) Append the parameteres -Xmx512m -Xms256m   to the line JavaVM Options

For example: JavaVM Options=-Xrs -Dxdo.ConfigFile=%PS_HOME%/xdo.cfg -Xmx512m -Xms256m

NOTE: -Xmx512m -Xms256m is usually a good starting point for the java heap size.
This can be increased as needed (e.g: -Xmx1024m -Xms512m) or reduced (e.g: -Xmx256m -Xms128m)
as this is a tunable parameter. However, 512/256 will work in most situations.

4) Reconfigure and boot the application server domain with the new settings in place.

Notes on tuning the java heap:

A) The java heap setting may need to be "tuned" in order to work properly.
If the following error occurs after changing the java heap,try a lower value like -Xmx256m -Xms128m.

"Error occurred during initialization of VM. Could not reserve enough space for object heap"

Also note, some earlier versions of windows will not handle a large heap size, and
the value -Xmx32m -Xms32m must be used.  However,
this is not a problem with current releases of Windows server and PeopleTools 8.49.18 and higher.

B) If the error Java exception thrown: java.lang.OutOfMemoryError persists,
try a larger value such as -Xmx1024m -Xms512m or even -Xmx2048m -Xms1024m

C) Typically, Xmx512m -Xms256m is a solid value for the Java heap to handle app server functions.

---------------------------------------------->>>>>>> ISSUE #14 <<<<<<<-----------------------------------------------------

Issue : Unable to boot Peoplesoft Application server due to tuxedo error (PT8.52,Linux)

Error :

LIBTUX_CAT:681: ERROR: Failure to create message queue
171351.abc.nirajsrvr.com!PSMONITORSRV.18464.996141072.-2: LIBTUX_CAT:248: ERROR: System init function failed, Uunixerr = : msgget: No such file or directory
171351.abc.nirajsrvr.com!tmboot.18369.334269536.-2: CMDTUX_CAT:825: ERROR: Process PSMONITORSRV at abc.nirajsrvr.com failed with /T tperrno (TPEOS - operating system error)171351.abc.nirajsrvr.com!tmboot.18369.334269536.-2: tmboot: CMDTUX_CAT:827: ERROR: Fatal error encountered; initiating user error handler
171355.abc.nirajsrvr.com!BBL.18365.469684640.0: CMDTUX_CAT:26: INFO: The BBL is exiting system
171358.abc.nirajsrvr.com!PSADMIN.18349: End boot attempt on domain FINDMO92

Solution :
It means that you have run out of msg queue identifiers or memory allocated to your message queues.
To fix it increase the msgmni parameter for your IPC resources. (NOTE:  There may be other settings applicable according to your platform)

Changed below parameters to given values in /etc/sysctl.conf and reloaded with sysctl -p
You need to be logged in as root to be able to do it.
Old values
kernel.msgmni = 1024
kernel.msgmax = 131072
kernel.msgmnb = 131072

New values
kernel.msgmni = 8092
kernel.msgmax = 1048576
kernel.msgmnb = 1048576

after reconfiguring app server the services came up fine.

Note :
In Solaris, change the values in /etc/system.
In HP, this value is tunned using the "sam" utility.
In IBM/AIX, you will use "smit".

---------------------------------------------->>>>>>> ISSUE #15 <<<<<<<-----------------------------------------------------

Issue : Unable to boot app server due to below error

PSAPPSRV.20316230 (0) [06/14/13 09:40:25](3) File: /vob/peopletools/src/pssys/stmget.cppSQL error. Stmt

#: 2509  Error Position: 70  Return: 942 - ORA-00942: table or view does not exist

Failed SQL stmt:SELECT VERSION, OPERPSWD, ENCRYPTED, SYMBOLICID, ACCTLOCK FROM SYSADM.PSOPRDEFN WHERE OP

RID = :1

PSAPPSRV.20316230 (0) [06/14/13 09:40:25](0) Database Options: override_connection_reset: 0

PSAPPSRV.20316230 (0) [06/14/13 09:40:25](1) GenMessageBox(0, 0, M): Database Signon: Could not sign on

to database FIPSFRP1 with user PSAPPS.

Solution : when checked logging in through people id to Database.

it didn't have select access on below security tables. Somehow SELECT privilege had been lost

So granted select on PSOPRDEFN , PSACCESSPRFL & PSSTATUS to people.

This resolved the issue and apps booted fine.

---------------------------------------------->>>>>>> ISSUE #16 <<<<<<<-------------------------------------------

Issue : App Engine failing with below error
File: SQL Access ManagerSQL error. Stmt #: 2  Error Position: 0  Return: 404 - ORA-01017: invalid username/password; logon denied
Could not setup user NPATIL's credentials for this AE program.  Perhaps
this AE request is running outside the user's allowed signon times??
Process request aborted.

Solution : Restart the failure App Engine process to re-run the program.
   If issue still persists then restart AESRV using below commands in tuxedo commandline in psadmin utility.
   stop -g AESRV
   boot -g AESRV
   If issue still there then try restarting process scheduler or reconfigure it if needed.

---------------------------------------------->>>>>>> @ <<<<<<<-----------------------------------------------------

Bounce PeopleSoft services manually from tuxedo command prompt : 

1. Boot the Tuxedo Bulletin Board Liaison process.
    #boot  the  Tuxedo administrative processes
    boot  -A

2. Bounce the PeopleSoft Application Server processes
    (specifed the -w parameter so that they don't wait as they start)
    stop -g APPSRV
    boot -g APPSRV -w

3. Stop entire app server without waiting for each server process to stop
    stop -y -c

3. Boot entire App server
    boot -y -w

3. Bounce PUSBUB without affecting other processes
    stop -g PUBSUB
    boot -g PUBSUB

3. Boot the JREPSRV process (which maps Java Classes to Tuxedo Services).
    boot  -g  JREPGRP

4. When all the other processes have booted, boot the WSL and JSL processes.
    boot  -g  BASE
    boot  -g  JSLGRP