[p4dti-discussion] Perforce\Bugzilla sync problem

[Nick Barnes]

At 2007-12-13 20:06:29+0000, "Hamid Ghassemi" writes:
> Good morning,
> 
> A few days ago, I upgraded our Bugzilla from 3.0 to 3.0.2.  After the
> upgrade I noticed P4DTI is slowing the Bugzilla down significantly,
> specially during a query page when we are looking for a specific bug.
> Also when I looked at the P4DTI log, I noticed that it is trying to
> replicate all the bugs in the perforce and some are reporting the
> following;
> 
> 2007-12-13 17:59:30 UTC  (P4DTI-8046)  Replicating issue '1412' to job
> 'bug1412'.
> 2007-12-13 17:59:30 UTC  (P4DTI-8137)  -- No issue fields were
> replicated.
> 
> Bug 1412 is a very old bug and should not be replicated according to the
> configuration.  I set the configuration date to the following and still
> I got messages like the above.  The date is for last night at 23:59:59
> 
> start_date = "2007-12-12 23:59:59"
> 
> I have stopped the process on the bugzilla server, so we can use
> Bugzilla.
> 
> Is there  a compatibility problem with Bugzilla 3.0.2 and P4DTI?  I am
> wondering if I should start the replication tonight and see if it
> finishes. 
> 
> By the way, I did run python refresh.py and it did not seem to help.
> 
> Any ideas?

Try to find out how this bug is being selected for replication.  Edit
config.py to set log_level to message.DEBUG, then restart the
replicator.  This will generate a huge amount of log output, including
logs of all the SQL traffic (P4DTI-1003 and P4DTI-1014).  Find the
SELECT statements which choose which bugs to replicate. (there should be lines like this:

2006-07-23 15:34:02 UTC  (P4DTI-1003)  Executing SQL command 'select bugs.bug_id from bugs   left join p4dti_bugs using (bug_id)   where bugs.creation_ts >= '2006-07-23 16:34:01'     and bugs.creation_ts < '2006-07-23 16:34:02'     and (p4dti_bugs.rid is null          or (p4dti_bugs.rid = 'replicator0'              and p4dti_bugs.sid = 'perforce0'              and p4dti_bugs.migrated is null))'.

2006-07-23 15:34:02 UTC  (P4DTI-1003)  Executing SQL command 'select bugs.bug_id from bugs   left join p4dti_bugs using (bug_id)   left join bugs_activity     on (bugs_activity.bug_when >= '2006-07-23 16:34:01' and         bugs_activity.bug_when < '2006-07-23 16:34:02' and         bugs.bug_id = bugs_activity.bug_id)   where bugs.delta_ts >= '2006-07-23 16:34:01'     and bugs.delta_ts < '2006-07-23 16:34:02'     and creation_ts < '2006-07-23 16:34:01'     and bugs_activity.fieldid is null    and (p4dti_bugs.rid is null          or (p4dti_bugs.rid = 'replicator0'              and p4dti_bugs.sid = 'perforce0'))     and (p4dti_bugs.migrated is null          or p4dti_bugs.migrated < '2006-07-23 16:34:01') '.

2006-07-23 15:34:02 UTC  (P4DTI-1003)  Executing SQL command 'select bugs.bug_id from bugs, bugs_activity ba left join p4dti_bugs using (bug_id) left join p4dti_bugs_activity pba   on (ba.bug_id = pba.bug_id and       ba.bug_when = pba.bug_when and       ba.who = pba.who and       ba.fieldid = pba.fieldid and       ba.removed = pba.oldvalue and       ba.added = pba.newvalue and       pba.rid = 'replicator0' and       pba.sid = 'perforce0')   where ba.bug_when >= '2006-07-23 16:34:01'     and ba.bug_when < '2006-07-23 16:34:02'     and bugs.bug_id = ba.bug_id     and pba.rid is null     and (p4dti_bugs.rid is null          or (p4dti_bugs.rid = 'replicator0'              and p4dti_bugs.sid =  'perforce0'))    and (bugs.creation_ts < '2006-07-23 16:34:01' or          p4dti_bugs.migrated is not null)   group by bugs.bug_id '.

Identify which one of these selects is incorrectly returning your
unwanted bug, and think through the logic of the select to figure out
why it is going wrong.

I'm not aware of changes in 3.0.2 which could affect this, but I
haven't paid very close attention to the 3.0.1 and 3.0.2 patch
releases.

Nick Barnes
P4DTI Project
Ravenbrook Limited