I've successfully created BigQuery subscription for Pub/Sub topic.
Subscription schema options used:
- Use table schema
- Write metadata
- Drop unknown fields
Topic does not have a schema. I assumed that is not required but I might be wrong.
Now I'm publishing messages to the topic however subscription shows an error saying "The subscription cannot receive messages because of resource errors."
Subscription state is: table schema mismatch.
I'm not able to find anything in the logs about specific mismatch.
[
{
"name": "subscription_name",
"type": "STRING",
"mode": "REQUIRED",
"description": "Pub/Sub subscription name"
},
{
"name": "message_id",
"type": "STRING",
"mode": "REQUIRED",
"description": "Pub/Sub message identifier"
},
{
"name": "publish_time",
"type": "TIMESTAMP",
"mode": "REQUIRED",
"description": "Pub/Sub publishing time"
},
{
"name": "attributes",
"type": "JSON",
"mode": "REQUIRED",
"description": "Pub/Sub message attributes"
},
{
"name": "id",
"type": "STRING",
"mode": "REQUIRED",
"description": "Message identifier"
},
{
"name": "vendor_id",
"type": "STRING",
"mode": "REQUIRED",
"description": "Message vendor identifier"
},
{
"name": "account",
"type": "string",
"mode": "REQUIRED",
"description": "EmailEngine account identifier"
},
{
"name": "path",
"type": "STRING",
"mode": "REQUIRED",
"description": "Mailbox path"
},
{
"name": "created_on",
"type": "STRING",
"mode": "REQUIRED",
"description": "Creation date"
},
{
"name": "from",
"type": "STRING",
"mode": "REQUIRED",
"description": "Message sender"
},
{
"name": "subject",
"type": "STRING",
"mode": "NULLABLE",
"description": "Message subject"
},
{
"name": "body",
"type": "STRING",
"mode": "NULLABLE",
"description": "Message body"
},
{
"name": "body_type",
"type": "STRING",
"mode": "NULLABLE",
"description": "Message body type (plain, html or markdown)"
},
{
"name": "to",
"type": "STRING",
"mode": "NULLABLE"
},
{
"name": "cc",
"type": "STRING",
"mode": "NULLABLE"
},
{
"name": "bcc",
"type": "STRING",
"mode": "NULLABLE"
},
{
"name": "reply_to",
"type": "STRING",
"mode": "NULLABLE"
},
{
"name": "in_reply_to",
"type": "STRING",
"mode": "NULLABLE"
},
{
"name": "is_auto_reply",
"type": "BOOLEAN",
"mode": "NULLABLE"
},
{
"name": "labels",
"type": "STRING",
"mode": "NULLABLE"
},
{
"name": "flags",
"type": "STRING",
"mode": "NULLABLE"
},
{
"name": "sender_role",
"type": "STRING",
"mode": "NULLABLE",
"description": "Classified sender role"
},
{
"name": "message_category",
"type": "STRING",
"mode": "NULLABLE",
"description": "Classified message category"
},
{
"name": "duration",
"type": "INTEGER",
"mode": "NULLABLE",
"description": "Classification duration"
},
{
"name": "prompt_tokens",
"type": "INTEGER",
"mode": "NULLABLE"
},
{
"name": "completion_tokens",
"type": "INTEGER",
"mode": "NULLABLE"
},
{
"name": "total_tokens",
"type": "INTEGER",
"mode": "NULLABLE",
"description": "Total tokens used for the interaction with LLM"
},
{
"name": "payload",
"type": "STRING",
"mode": "NULLABLE",
"description": "Classification payload from LLM"
}
]
PubSub message:
{
"id": "<id.com>",
"vendor_id": "AAAAAQAAAyY",
"account": "naxj4rkbqadgqppi",
"path": "INBOX",
"created_on": "2024-02-08T15:15:53+00:00",
"from": "John Doe <[email protected]>",
"subject": "Subject",
"body": "Message body",
"body_type": "md",
"to": "Mark Doe <[email protected]>",
"cc": null,
"bcc": null,
"reply_to": null,
"in_reply_to": null,
"is_auto_reply": false,
"labels": "",
"flags": "",
"sender_role": "internal",
"message_category": null,
"duration": 12,
"prompt_tokens": 1637,
"completion_tokens": 7,
"total_tokens": 1644,
"payload": "{"json": "payload"}"
}
Question is what is wrong with the schema or how to find error details?
