Sunday, June 5, 2011

Poor man's approach in practice

One day we ran into problem when an application was randomly starting using more CPU resources as usual. It continued picking more and more CPU resources from time to time. Thread dump showed many threads though we couldn't understand wich of them is problematic...and I remembered about poor man's profiler and its approach: the stack that's all we need to!

Htop is awesome
At first we simply caught problematic thread using htop (you just need to remember about tree process view, type t-button, or anyway you need to see all forked java processes with its identifiers and CPU usage). We just obtained a java thread wich took more CPU than others. We got an identifier of native OS thread, let it be 3256, but what's next?

I like jstack, it helps me all the time I deal with spontaneous problems. If you look thru the content of jstack output you'll see some thread identifiers and all of them are hexadecimal. Thus we converted our native thread id into hex representation 0xcb8. Lazy guys can use Integer.toHexString ;)

The final step
We searched by our hexadecimal string in jstack output (don't forget about leading zeros, therefore just search by value, i.e. cb8 in our case). Matched nid thread attribute value (suppose stands for native id) was our problematic thread with its stack. In fact we were getting infinite loop. The problem is localized and will be fixed soon.

Sunday, May 22, 2011

MySQL: ON DUPLICATE KEY UPDATE breaks generated key list of extended INSERT

When extended INSERT (insert with several tuples in one query) is used with ON DUPLICATE KEY UPDATE clause and there're unique key conflict the Connector/J's Statement will return incorrect list of generated identifiers that cannot be matched with tuples from INSERT query at all. There are steps to reproduce this issue:
  1. create a table with auto increment primary key and unique key:
    create table TEST_TABLE(
            id int auto_increment, 
            test_field varchar(255), 
            primary key (id), 
            unique key(test_field)
    ) engine=InnoDB;

  2. insert the following row:
    insert into TEST_TABLE(id, test_field) values(100, 'test1');

  3. perform an extended insert with ON DUPLICATE KEY UPDATE clause via JDBC driver:
    insert into TABLE(test_field) values('test2'),('test1'),('test3') 
    on duplicate key update id=LAST_INSERT_ID(id);
and check returned generated keys:
  • at first, you'll get 4 values instead of 3: [1, 2, 3, 4]
  • at second, there's no correct id for test1 row in returned list
Thus only for the first row returned id will be correct, if test1 will be first in a query whole list will be incorrect and cannot be matched with inserted rows. The only workaround is to use separate query to retrive generated identifiers to reduce count of Data Base access and avoid the split of this extended insert in separate queries. I've looked thru the code in com.mysql.jdbc.StatementImpl and found that It retrieves count of affected rows and just increments the value of LAST_INSERT_ID for each update and gets sequential list of identifiers. But in fact it can be nonsequential: for my case I expect the result set: [1, 100, 3] <-- 3 is here because of the 2nd value, or even [1, NULL, 3] will be normal. But current result: [1, 2, 3, 4] is very confused and cannot be matched with inserted rows.

In fact MySql returns unexpected count of updated rows in such case because of this behavior:
For INSERT ... ON DUPLICATE KEY UPDATE statements, the affected-rows value is 1 if the row is inserted as a new row and 2 if an existing row is updated.
I've reported new #61213 issue for MySQL Connector/J and according to Mark Matthews the fix raises a modification in MySQL server protocol because there's currently no way to get generated keys for whole data set though LAST_INSERT_ID returns only the first one. Thus be aware of this issue.

Saturday, April 30, 2011

How does JVM handle locks

As we are talking about the latest version of Sun Hotspot Java Virtual Machine 1.6 there're the following three types of locks performed by JVM when you try to acquire lock from java.util.concurrent.locks.Lock implementation or enter synchronized block:
  • biased: sometimes even in concurrent systems there's no contention and JVM shouldn't borrow mutex from OS for perform locking in this case. Hotspot can operate with its own internal data structures to simulate locking in more effective way. For example, if synchronized part of code is not executed concurrently in real time, JVM assigns an owner thread id to an object used as mutex in your Java code using CAS operation and additionally stores the reentrancy count if CAS is passed. It is biased lock - the 'lightest' type of locks done by JVM. Reentrancy count will be updated by lock owner thread just as usual local variable without CAS. If CAS fails it means another thread already gets this lock and in this case JVM stops the mutex owner thread, flushes thread context into the main memory and checks the reentrancy count. If it's 0 JVM escalates lock to thin type otherwise to fat (I assume the main purpose is the wait time, it should be very small if lock is thin). Note Hotspot uses the same field for storing owner thread id in mutex object as for caching identity hash code. Thus if you retrieve identity hash code on you mutex once then it will be unavailable for biased locking even it was already used as biased. More info about biased locks is described in David Dice's blog.
  • thin: it's a simple spin lock. It helps to save time for thread context switching when time for spinning is quite small. When one thread tries to acquire an occupied mutex it spins some time until the lock will be freed. The count of spins is based on internal JVM resolution and may depend from different factors: statistics gathered by JVM about your application, count of used threads, CPU and so forth. JVM determines when thin lock becomes inefficient and escalates it to fat lock.
  • fat: the 'strongest' type of lock when JVM requests for an OS mutex and uses OS scheduler engine for threads parkings and wake ups. It is much costly than previous types because in this case JVM should directly interact with OS every time when thread acquires and frees the lock.

Wednesday, February 2, 2011

The GrinderStone is nominated for Eclipse Community Awards

The GrinderStone, an IDE for Grinder load testing scripts development, is nominated for Eclipse Community Awards 2011 in Best Developer Tool section. Any comments will be appreciated on GrinderStone nominee page. We thank our users and give them unique Grinder scripts development features based on Eclipse platform.

Tuesday, January 25, 2011

MySQL 5.5 won't start on Mac OS 10.6

Modify the mysql.server script file using the following command :
sudo /Applications/ /usr/local/mysql/support-files/mysql.server 
Locate the configuration defining the basedir and set the following :
Save the file and start the mysql server in the system preferences or terminal.

from MySQL Developer Zone

Sunday, November 7, 2010

InnoDB: strange deadlock

One time I've noticed that some concurrent threads that perform inserts into one simple table can cause deadlock in MySQL. I don't have auto increment primary key in a table and generate id on application level. So let concurrent inserts are performed into this table in separate transactions with default isolation level (repeatable-read):

insert into TEST values(...);
insert into TEST values(...);
insert into TEST values(...);

I got deadlock! Why?! As I understand according to this discussion if two different concurrent inserts try to lock the same gap at the end of a table at the same time (i.e. concurrently set X lock) we will get deadlock (I think it should be livelock but not's a bit strange). In a case if primary key is auto increment we will have lock by auto_inc index and all inserts will wait for granting lock - so no deadlock here can happen, only livelock and InnoDB successfully handles this case.

I think InnoDB should lock the primary key index during insert in a case if it is not auto increment. I think there's a problem with locks granularity in InnoDB - I'll try to research this problem and submit it to Oracle.


One guy told me the reason of this deadlock. It was the reason in inserts performed from separate threads that became not ordered and InnoDB locked some necessary gaps in indexes - I missed that I had an index in a table - for record insertions (some threads, some monitors, not ordered actions...potential's classic). See detailed explanation here. Thus I was wrong when told you that InnoDB has a problem with locks granularity...

Monday, October 18, 2010

JBPM concurrency issue: Action altered id

I've been working for several days on problem when some concurrent threads create process instances from one process definition. In my environment I deploy one process definition on application startup and use this instance during whole application lifecycle. It means that this instance becomes detached and is used in all JBPM related places. The error was:

org.hibernate.HibernateException: identifier of an instance of org.jbpm.graph.def.Action was altered from 532 to 533

After data base dumps analysis I've obtained a problematic place. It was in jobs processing (or in other definitions - timers) persistence. JBPM performs manual cascade persistence of Action from ProcessDefinition called by this timer. The following code snippet from org.jbpm.db.JobSession is problematic:

public void saveJob(Job job) {

    if (job instanceof Timer) {
        Timer timer = (Timer) job;
        Action action = timer.getAction();

        if (action != null && !session.contains(action)) {

As process definition is detached - linked Action also is detached so Session will always persist Action as new instance (note: they call save and not merge). I don't know why JBMP do that because we cannot create timer without process instance and we cannot create process instance without fully persisted process definition (may be process definition can be changed on the fly? don't think that it's correct behavior). I created JBPM-2950 but hadn't got any response. The same with HHH-4309 - we created it more than one year ago but anyone responded. I don't know why these problems stay ranked as Major and Critical - may be JBoss doesn't wanna see them? Comparing this approach with Spring Issue tracking where I always get a responses in a few days or even hours.

In any case I would like to share the patch fixing this problem.
At first we should correct check Action existence in Persistence Context: I replaced it with id nullabillity check, because when Action is persisted and Session is flushed we will get non zero id and we don't need to update this Action because we don't change it.

public class JbpmJobSession
    extends JobSession {

    private Session session;

    public JbpmJobSession(Session session) {
        this.session = session;

    public void saveJob(Job job) {

        if (job instanceof Timer) {
            Timer timer = (Timer) job;
            Action action = timer.getAction();

            // action already persisted? 
            if ((action != null) && (action.getId() == 0L)) {

Then we should correct JBPM DB persistence service to allow saving jobs via our modified session:
public class JbpmDbPersistenceService
    extends DbPersistenceService {

    public JbpmDbPersistenceService(
           DbPersistenceServiceFactory persistenceServiceFactory) {

    public JobSession getJobSession() {

        if (jobSession == null) {
            Session session = getSession();

            if (session != null) {
                jobSession = new JbpmJobSession(session);

        return jobSession;
Then create corresponding factory:
public class JbpmPersistenceServiceFactory
    extends DbPersistenceServiceFactory {

    public Service openService() {
        return new JbpmDbPersistenceService(this);

And one step to finish: modify persistence service to yours in jbpm.cfg.xml


That's it. Technology Blogs