data-pipeline

Environment

• klio* version(s):
  • klio-cli: v1.0.5

Description of the bug

When

We're using marshmallow to parse whylogs config from YAML

However, Pydantic is much more powerful as it allows users to set config via various mechanims, from YAML, JSON to Environment settings.

We should consider moving to pydantic

Write unit test coverage for SafeDataset and SafeDataLoader, along with the functions in utils.py.

The following packages must be updated in package.json:

• @aws-cdk/aws-dynamodb
• @aws-cdk/aws-events
• @aws-cdk/aws-events-targets
• @aws-cdk/aws-iam
• @aws-cdk/aws-lambda
• @aws-cdk/aws-lambda-event-sources
• @aws-cdk/core

The latest published version on NPM is 1.109.0 - all of these packages should be updated to at least 1.100.0 if possible.

Don't worry about event types, just have it forward messages over HTTP in the SSE protocol.

Read more: https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events