[p4dti-discussion] UnicodeDecodeError: 'ascii' codec can't decode byte 0xe0 in position 123: ordinal not in range(128)

[Lukas Kucera]

Hello,
I have problem with changelist with unicode character in description. 
The replicator raised exception. Please help me to solve this. Detailed 
info is below.
Thank you in advance.

Best Regards
Lukas Kucera


The original description looks: "FIX: correct word sarà in finish 
dialog" - replicator reports "FIX: correct word sar\xe0 in finish dialog\n"

SETUP:

python 2.5
p4dti 2.4.2
bugzilla 3.0.2


P4DTI log:

(P4DTI-8658)  This is an automatically generated e-mail from the Perforce Defect
Tracking Integration replicator 'replicator0'.

(P4DTI-8647)  The replicator failed to poll successfully, because of the
following problem:

(P4DTI-891X)  Error (<type 'exceptions.UnicodeDecodeError'>): 'ascii' codec
can't decode byte 0xe0 in position 123: ordinal not in range(128)

(P4DTI-8523)  Here's a full Python traceback:

Exception:
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe0 in position 123: ordinal not in range(128)
Traceback (innermost last):
  File "/opt/p4dti/replicator.py", line 861, in carefully_poll_databases
    self.poll_databases()
    locals:
      self: <replicator.replicator instance at 0x2b0fe7586878>
  File "/opt/p4dti/replicator.py", line 1220, in poll_databases
    self.replicate_changelist_p4_to_dt(c)
    locals:
      p4_marker: 4459
      c: {'Status': 'submitted', 'code': 'stat', 'Description': 'FIX: correct word sar\xe0 in finish dialog\n', 'Client': 'computer1', 'User': 'john.doe', 'Date': '2008/01/04 09:00:39', 'Change': '43280'}
      changed_issues: <replicator.list_cursor instance at 0x2b0fe7586b00>
      self: <replicator.replicator instance at 0x2b0fe7586878>
      dt_marker: '2008-01-04 09:36:11'
      changed_jobs: {}
      changelists: [{'Status': 'submitted', 'code': 'stat', 'Description': 'FIX: correct word sar\xe0 in finish dialog\n', 'Client': 'computer1', 'User': 'john.doe', 'Date': '2008/01/04 09:00:39', 'Change': '43280'}, {'Status': 'submitted', 'code': 'stat', 'Description': 'FIX: add dot behind sentence\n', 'Client': 'computer1', 'User': 'john.doe', 'Date': '2008/01/04 09:18:24', 'Change': '43281'}]
      _: {}
  File "/opt/p4dti/replicator.py", line 1268, in replicate_changelist_p4_to_dt
    description, status, user):
    locals:
      status: 'submitted'
      description: 'FIX: correct word sar\xe0 in finish dialog'
      changelist: {'Status': 'submitted', 'code': 'stat', 'Description': 'FIX: correct word sar\xe0 in finish dialog\n', 'Client': 'computer1', 'User': 'john.doe', 'Date': '2008/01/04 09:00:39', 'Change': '43280'}
      client: 'computer01'
      user: 12
      date: '2008-01-04 09:00:39'
      self: <replicator.replicator instance at 0x2b0fe7586878>
      change: 43280
  File "/opt/p4dti/dt_bugzilla.py", line 925, in replicate_changelist
    self.bugzilla.add_changelist(dt_changelist)
    locals:
      status: 'submitted'
      dt_changelists: []
      description: 'FIX: correct word sar\xe0 in finish dialog'
      self: <dt_bugzilla.dt_bugzilla instance at 0x2b0fe7586758>
      change: 43280
      client: 'computer01'
      user: 12
      date: '2008-01-04 09:00:39'
      dt_changelist: {'description': 'FIX: correct word sar\xe0 in finish dialog', 'changelist': 43280, 'client': 'computer01', 'flags': 1, 'user': 12, 'sid': 'perforce0', 'rid': 'replicator0', 'p4date': '2008-01-04 09:00:39'}
  File "/opt/p4dti/bugzilla.py", line 1867, in add_changelist
    self.insert_row_rid_sid('p4dti_changelists', dict)
    locals:
      self: <bugzilla.bugzilla instance at 0x2b0fe757e680>
      dict: {'description': 'FIX: correct word sar\xe0 in finish dialog', 'changelist': 43280, 'client': 'computer01', 'flags': 1, 'user': 12, 'sid': 'perforce0', 'rid': 'replicator0', 'p4date': '2008-01-04 09:00:39'}
  File "/opt/p4dti/bugzilla.py", line 615, in insert_row_rid_sid
    self.insert_row(table, dict)
    locals:
      table: 'p4dti_changelists'
      self: <bugzilla.bugzilla instance at 0x2b0fe757e680>
      dict: {'description': 'FIX: correct word sar\xe0 in finish dialog', 'changelist': 43280, 'client': 'computer01', 'flags': 1, 'user': 12, 'sid': 'perforce0', 'rid': 'replicator0', 'p4date': '2008-01-04 09:00:39'}
  File "/opt/p4dti/bugzilla.py", line 604, in insert_row
    rows = self.execute(command)
    locals:
      self: <bugzilla.bugzilla instance at 0x2b0fe757e680>
      value: "'2008-01-04 09:00:39'"
      values: "'FIX: correct word sar\xe0 in finish dialog', 43280, 'computer01', 1, 12, 'perforce0', 'replicator0', '2008-01-04 09:00:39'"
      key: 'p4date'
      table: 'p4dti_changelists'
      command: "insert p4dti_changelists ( description, changelist, client, flags, user, sid, rid, p4date ) values ( 'FIX: correct word sar\xe0 in finish dialog', 43280, 'computer01', 1, 12, 'perforce0', 'replicator0', '2008-01-04 09:00:39' );"
      columns: 'description, changelist, client, flags, user, sid, rid, p4date'
      dict: {'description': 'FIX: correct word sar\xe0 in finish dialog', 'changelist': 43280, 'client': 'computer01', 'flags': 1, 'user': 12, 'sid': 'perforce0', 'rid': 'replicator0', 'p4date': '2008-01-04 09:00:39'}
  File "/opt/p4dti/bugzilla.py", line 102, in execute
    self.cursor.execute(sql)
    locals:
      self: <bugzilla.bugzilla instance at 0x2b0fe757e680>
      sql: "insert p4dti_changelists ( description, changelist, client, flags, user, sid, rid, p4date ) values ( 'FIX: correct word sar\xe0 in finish dialog', 43280, 'computer01', 1, 12, 'perforce0', 'replicator0', '2008-01-04 09:00:39' );"
  File "/usr/lib64/python2.5/site-packages/MySQLdb/cursors.py", line 146, in execute
    query = query.encode(charset)
    locals:
      args: None
      self: <MySQLdb.cursors.Cursor object at 0x2b0fe7576d10>
      charset: 'latin1'
      db: <weakproxy at 0x2b0fe7564ec0 to Connection at 0x796a00>
      query: "insert p4dti_changelists ( description, changelist, client, flags, user, sid, rid, p4date ) values ( 'FIX: correct word sar\xe0 in finish dialog', 43280, 'computer01', 1, 12, 'perforce0', 'replicator0', '2008-01-04 09:00:39' );"
      ListType: <type 'list'>
      exc_info: <built-in function exc_info>
      TupleType: <type 'tuple'>