Issue with Websphere logs

Saying this up front so no one is confused: I don’t think this is a fluentd issue. Still, I wonder if anyone has run into this issue.

Seems to me the websphere cannot write Chinese char in the log file and the websphere log contains boxes and question marks. I know that at least on Linux, there are no issues with Chinese char in fluentd.

I cannot share you the message as it contains PII, but I believe this websphere log message causes TD-agent converting it to hex value starting with \x and this hex encoding cannot support by elastic unless escaping like [\x](file://x).

This is the ultimate error message from Elastic:

“error” : {

"root_cause" : [

{ “type” : “mapper_parsing_exception”, “reason” : “failed to parse field [message] of type [text] in document with id ‘RViQMXoBcX4lFx6VquxI’. Preview of field’s value: ‘’” }


"type" : "mapper_parsing_exception",

"reason" : "failed to parse field [message] of type [text] in document with id 'RViQMXoBcX4lFx6VquxI'. Preview of field's value: ''",

"caused_by" : {

  "type" : "json_parse_exception",

  "reason" : "Unrecognized character escape 'x' (code 120)\n at [Source: (byte[])\"{ \"message\":\"[6/8/21 14:37:39:438 CST] 000000f6 SystemOut     O

{call db_aadmin…po_aa_app_new(‘A’,‘01’,null,‘HAR’,‘06/08/2021 14:37:39’,‘AHR’,‘06/08/2021 14:37:39’,‘05/28/2021’,‘2AP202’,null,null,null,‘HA’,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,‘48’,1000000.00,null,null,null,null,null,null,‘Y052’,‘Y052’,‘WONG’,‘JUNG’,null,‘WONG ‘,‘WONG’,‘KUN’,’\xF\\xC4\\xEF’,‘xF9’,‘xF9’,‘xEF’,‘10989’,‘M’,null,‘01’,'jimmy@g"[truncated 1012 bytes]; line: 1, column: 436]" }


“status” : 400


I think this may be the fix, but I am not on the Websphere team, so I can’t just try it. IBM Docs

Any thoughts? Specifically, even though fluentd is not the cause of this, is there anything we can do in fluentd?