Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Need better debug logging for patch operations #89874

Open
MikeSpreitzer opened this issue Apr 6, 2020 · 19 comments
Open

Need better debug logging for patch operations #89874

MikeSpreitzer opened this issue Apr 6, 2020 · 19 comments

Comments

@MikeSpreitzer
Copy link
Member

@MikeSpreitzer MikeSpreitzer commented Apr 6, 2020

What would you like to be added:
I would like debug logging to show the heart of the processing of a patch request. For example, showing the inputs and outputs of https://github.com/kubernetes/apiserver/blob/fa3c8ad6d7aff4ba13bf7296583ed2815db1be44/pkg/endpoints/handlers/patch.go#L620 .

Why is this needed:
It is way too easy to botch a PATCH request. Due to the way JSON decoding happily and silently discards unexpected contents, it is very easy to accidentally write buggy code that provokes no error. There is no existing debug logging to help a developer see the error in their ways.
I spent a lot of time chasing a bug that was immediately evident when I put in debug logging about the line mentioned above.

@MikeSpreitzer
Copy link
Member Author

@MikeSpreitzer MikeSpreitzer commented Apr 6, 2020

@MikeSpreitzer
Copy link
Member Author

@MikeSpreitzer MikeSpreitzer commented Apr 6, 2020

/good-first-issue

@k8s-ci-robot
Copy link
Contributor

@k8s-ci-robot k8s-ci-robot commented Apr 6, 2020

@MikeSpreitzer:
This request has been marked as suitable for new contributors.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-good-first-issue command.

In response to this:

/good-first-issue

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@MikeSpreitzer
Copy link
Member Author

@MikeSpreitzer MikeSpreitzer commented Apr 6, 2020

This may be particularly helpful to someone struggling with #89875

@tanjunchen
Copy link
Member

@tanjunchen tanjunchen commented Apr 7, 2020

/assign

@MikeSpreitzer
Copy link
Member Author

@MikeSpreitzer MikeSpreitzer commented Apr 7, 2020

FYI, for a comparison of similar debug logging you can try setting -v=5 and seeing what gets logged. You can also try some higher settings. You will see request bodies logged at the site where the call goes out. I did not see request bodies logged on the receiving side.

@MikeSpreitzer
Copy link
Member Author

@MikeSpreitzer MikeSpreitzer commented Apr 7, 2020

FYI, here is the debug logging statement that I used:

	klog.V(8).Infof("Applied %#+v to %#+v to produce %#+v, err=%#+v", patchMap, originalMap, patchedObjMap, err)

That was right after the call to strategicpatch.StrategicMergeMapPatch.

@MikeSpreitzer
Copy link
Member Author

@MikeSpreitzer MikeSpreitzer commented Apr 7, 2020

It would be appropriate to do the same for the case of JSON Patch and JSON Merge Patch as well as the stategic merge patch.

@liggitt
Copy link
Member

@liggitt liggitt commented Apr 7, 2020

I did not see request bodies logged on the receiving side.

That's somewhat intentional. Logging of API server request bodies without regard to whether the resource is a confidential one (e.g. secrets) would expose confidential content. If we add debug output of Request bodies to the API server log, we'll need to do so in a way that is not considered a vulnerability.

@MikeSpreitzer
Copy link
Member Author

@MikeSpreitzer MikeSpreitzer commented Apr 7, 2020

Good point. So would it suffice to carve out an exception for objects of the core Secret kind?

@liggitt
Copy link
Member

@liggitt liggitt commented Apr 7, 2020

So would it suffice to carve out an exception for objects of the core Secret kind?

The audit logging and storage encryption mechanisms do not special-case secret resources, I would not expect this to.

@fedebongio
Copy link
Contributor

@fedebongio fedebongio commented Apr 7, 2020

/assign @jennybuckley

@sresthas
Copy link

@sresthas sresthas commented Apr 16, 2020

Can I take this up as my first issue? Any pointers would be helpful.

@MikeSpreitzer
Copy link
Member Author

@MikeSpreitzer MikeSpreitzer commented Apr 16, 2020

@twisted-sres : yes.
@liggitt : your last remark looks like a complete rejection of this issue. Is there something you think can be done here? Remember, the motivating problem is patch requests that DO NOT provoke errors yet fail to accomplish what the client developer expected. In extreme cases (the ones that actually provoked this issue), the non-erring patch request produced no change at all and I thought I had botched something server-side.

@liggitt
Copy link
Member

@liggitt liggitt commented Apr 17, 2020

your last remark looks like a complete rejection of this issue. Is there something you think can be done here?

In general, I don't want to increase the number of ways confidential data can leak into server logs. Are there other ways we could ease testing patches (like simpler use of patching functions in a library/unit test) or surface potential problems (like returning warnings, similar to the mechanism proposed in #73032)?

@MikeSpreitzer
Copy link
Member Author

@MikeSpreitzer MikeSpreitzer commented Apr 21, 2020

The particularly difficult thing about patching is that it is very easy for a client developer to make a mistake that produces an unexpected result and NO error NOR warning message on the server. Just goof up a field name or a bit of structure, and the patch has little or no effect on the target object.

I suppose it should be possible --- for JSON merge and for strategic merge patches --- to add a check on the server side, comparing the patch with the patched object to see if the patch contains material that does not appear in the patched object. It should be possible to also do a similar check for a JSON patch, looking for patch material that had no effect. Would there be a way to plumb a warning from these out to a reply header?

@liggitt
Copy link
Member

@liggitt liggitt commented Apr 22, 2020

Would there be a way to plumb a warning from these out to a reply header?

I actually just started a proposal for server -> client warnings (https://github.com/kubernetes/enhancements/tree/master/keps/sig-api-machinery/1693-warnings) which could surface something like that in the future, though getting the "field X in patch didn't exist so was ignored" plumbed out of the patch/json stack without erroring would be non-trivial as well.

@lavalamp
Copy link
Member

@lavalamp lavalamp commented Apr 22, 2020

Note that users can and should use dry run to verify their change will have the desired effect.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
9 participants
You can’t perform that action at this time.