Search This Blog

Sunday, April 14, 2013

Self-tuned Thread Pool Count


-->
Thread Pool count will give you the idea about the WebLogic Server instance throughput. First let us see how to monitor a server instance with WLST. if you provide the instance name the script will search the corresponding listen address, listen port for that instances then forms a URL using t3 protocal(WebLogic specific protocol) which is used to connect the instance and get the serverRuntime MBean which will contains that servers ThreadPoolRuntime.

Hogging Threads
Hogging Threads that have taken the too much time and we can assume that they are never going to come back. Hogging threads help us take some decisions, lets say many threads are hogging, we may take a decision to create new threads for next cycle.
My understanding about Thread States in WebLogic Server:
  1. ACTIVE
  2. STUCK
  3. STANDBY
A live thread which is ready to process the request, which is known as ACTIVE state. That is indicated when therad newly created. WebLogic Server start the server instance with 1 ACTIVE thread and the thread count grows as per the min size if specified other wise it will do self-tune as per the request.

Threads might wait for other thread to release resource. This might happen due to application varialbles. The variables are 2 types thread-safe other is risk for thread. All local variables in the methods are thread-safe. The variable defined in class level are unsafe. which causes memory leak, this state of threads are known as hogging. WebLogic identify a thread as hog by the time interval. If thread is waiting more than 600 sec will be treated as hog. STUCKthread interval we can tune as per the project need.

animations
If the number of HoggingThreadCount increases then the server health is in dangerous. That time you can take the ThreadDump
After Threads increase to a max utilization then the thread will be in STANDBY state.

Let us see the following script will get the Thread statistics for a given Weblogic server instance.


# This script is for single instance thread statistics
# You can enhance it further to take thread dump as per your env.

import sys
from java.util import Date

ucf='scriptpath/userConfigFile.sec'
ukf='scriptpath/userKeyFile.sec'
ECODE='\n \033[0m' # ending of color code

def ThreadCnt():
    try:
        print 'Connecting to Admin server....'
        connect(userConfigFile=ucf, userKeyFile=ukf, url='t3://admindns:port')
    except:
        print 'Admin Server NOT in RUNNING state....'
    urldict={}
    serverlist=getRunningServerNames()  # Getting Serverlist
    for svr in serverlist:
        cd("/Servers/"+svr.getName())
        urldict[svr.getName()]='t3://'+get('ListenAddress')+':'+str(get('ListenPort'))

    x = raw_input('Enter a server instance name : ')
    try:
        connect(userConfigFile=ucf, userKeyFile=ukf,url=urldict[x])
        serverRuntime()
        openSocks = cmo.getOpenSocketsCurrentCount();
        print('Open Sockets:: ' + str(openSocks));
        cd('serverRuntime:/ThreadPoolRuntime/ThreadPoolRuntime/')
        compReq = cmo.getCompletedRequestCount()
        status = cmo.getHealthState()
        hoggingThreads = cmo.getHoggingThreadCount()
        totalThreads = cmo.getExecuteThreadTotalCount()
        idleThrds = cmo.getExecuteThreadIdleCount()
        pending = cmo.getPendingUserRequestCount()
        qLen = cmo.getQueueLength()
        thruput = cmo.getThroughput()
        if idleThrds == 0:
            pstr='\033[1;47;31m'    # RED color
        else:
            pstr='\033[1;40;32m'    # GREEN color
        print(pstr+'Status of the Server: ' + str(status)  +ECODE
            +'The completed Requests: ' + str(compReq) +ECODE'
            +'Total the threads no s: ' + str(totalThreads)+ECODE
            +'The Idle threads: ' + str(idleThrds)+ECODE
            +'Hogging threads : ' + str(hoggingThreads)+ECODE
            +'Pending : ' + str(pending)+ECODE
            +'ThreadPool QueueLength: ' + str(qLen)+ECODE
            +'Server (Throughput): ' +str(thruput)+ECODE)
    except:
        print 'Exception... Unable to connect to given Server', x
        pass
    quit()

def quit():
    d = Date() # now
    print  d
    print '\033[1;40;32mHit any key to Re-RUN this script ...'+ECODE
    Ans = raw_input("Are you sure Quit from WLST... (y/n)")
    if (Ans == 'y'):
         disconnect()
        stopRedirect()
        exit()
    else:
        ThreadCnt()

def getRunningServerNames():
    domainConfig()
    return cmo.getServers()

if __name__== "main":
    redirect('./logs/ThreadCntwlst.log', 'false')
    ThreadCnt()
    print 'done'




WebLogic Server Health Status can be one of the following:

  1. HEALTH_OK
  2. HEALTH_WARN
  3. HEALTH_FAILED
  4. HEALTH_CRITICAL
  5. LOW_MEMORY_REASON
  6. HEALTH_OVERLOADED

OK is indicates everything fine, no worries!!

WARN raised when there is few stuck threads in the server instance.

LOW_MEMORY_REASON is going to tell you about JVM crash expected. You can configure to 'Exit' the managed server on low memory conditions with the help of NodeManager and WorkManager.

CRITICAL when multiple number of stuck threads happening and the threadpool count reaching unsual number. This case you need to suspect Network, JDBC or back-end connectivity has trouble.
FAILED happen when the new deployments fails. The NodeManager should not restart this managed server.

OVERLOADED Change the server health state to OVERLOADED on overload. The Nodemanager need to work at this state and bounce such WebLogic instance. This is a new feature of WebLogic 9.x and later versions, for detecting, avoiding and recovering from an overload condition of a WebLogic managed server. Overload protection can be used to throttle Work Managers and thread pools for performance. You can configure Shutdown the Work Manager or application on stuck threads when it crosss more than 5 or you can set threshold.
-->

Good Reference links:

http://forums.oracle.com/forums/thread.jspa?threadID=683752

Thursday, April 11, 2013

Home

Do you know this fun thing about Python scripting sing a poem of quotations just by by importing 'this' module.
wls:/offline> import this
The Zen of Python, by Tim Peters

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!
wls:/offline>

Content

You are invited for the contribution of WLST scripts and articles.

Technorati Claim token code JDU3DCFE66EH

Popular Posts