Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Root Nameserver support #434

Open
wants to merge 29 commits into
base: master
from
Open

Root Nameserver support #434

wants to merge 29 commits into from

Conversation

@joschi36
Copy link
Contributor

joschi36 commented Dec 11, 2019

This PR adds optional support for root NS records.
You can enable root NS records per provider with the parameter managed_root_ns for the providers that support it.
Support is added for this providers:

  • AWS Route53
  • NS1
  • YAML
@@ -1349,6 +1350,10 @@ def _apply(self, plan):
mod_type = getattr(self, '_mod_{}'.format(c.__class__.__name__))
mods = mod_type(c, zone_id, existing_rrsets)

# Rewrite NS Record to UPSERT because it always exists.

This comment has been minimized.

@ross

ross Dec 11, 2019

Contributor

I believe this may only be true in cases where there are 4 before and after. You might try updating it to 8 and then dropping it back down to 4 and seeing what that looks like. IIRC you can actually delete the original Route53 root NS records (with no way to find out what they were 😦.)

Our main domains run with 8, 4 from each of our two active providers:

otter:octodns ross$
otter:octodns ross$ dig +short NS github.com
ns1.p16.dynect.net.
ns3.p16.dynect.net.
ns-520.awsdns-01.net.
ns-421.awsdns-52.com.
ns2.p16.dynect.net.
ns-1707.awsdns-21.co.uk.
ns-1283.awsdns-32.org.
ns4.p16.dynect.net.

This comment has been minimized.

@joschi36

joschi36 Jan 9, 2020

Author Contributor

A HostedZone must contain at least one NS record for the zone itself.

You can't delete the Root NS records in AWS. But i will make it optional to use root NS per provider.

This comment has been minimized.

@ross

ross Jan 16, 2020

Contributor

What would happen here if you started with the default 4, added 4 more with octoDNS, and then dropped things back down to just 4. I don't believe this would allow you to delete the added NS records.

A HostedZone must contain at least one NS record for the zone itself.

This reads like you can delete all but 1. Seems like deletes are possible, but it won't let you delete the "last" one?

Copy link
Contributor

ross left a comment

Looks good thus far.

@joschi36
Copy link
Contributor Author

joschi36 commented Dec 30, 2019

hey @ross I need some help with the tests.
I've fixed some tests by just updating the asserts for the route53 provider (f31806e) I'm not sure if this is the right way to go.
I also fixed some tests for other providers here c6f7ef7 where I'm confident that it is right but maybe you want to look at it too.

I've also just tested to downgrade from eight NS records to four. And yeah you were right: plan: No changes
Maybe you or someone else have an idea to implement this better?

@joschi36 joschi36 force-pushed the joschi36:support-root-ns branch from caf8ab0 to c3d32f7 Jan 9, 2020
joschi36 added 6 commits Jan 9, 2020
@joschi36 joschi36 marked this pull request as ready for review Jan 9, 2020
@joschi36
Copy link
Contributor Author

joschi36 commented Jan 9, 2020

Can somebody please check my work as I am a newby to bigger Python projects with testing/coverage/linting.
I did my best but at some parts I was just frustrated.

@ross
Copy link
Contributor

ross commented Jan 16, 2020

I think I got the conflicts w/master resolved. Looking more closely at this now.

Copy link
Contributor

ross left a comment

Looks pretty solid. Comments/questions inline. This is a big enough change that it could use a testing plan. Something like the following, though you have probably thought about it more than me and may have further ideas:

  1. Create a new zone in the provider w manage_root_ns: false
  2. Copy the root NS records into the zone's config file, plan, make sure there are no changes
  3. Change manage_root_ns: true, plan, make sure there are no changes
  4. Change one of the root NS record's values, plan, see the change, apply verify change happened, plan, should se no changes
  5. Change the other 3 root NS record's values, repeat the above
  6. Delete one of the NS records, plan, verify it will go away, apply, verify it's gone, plan should show no changes
  7. Add the NS record back, repeat the above
  8. Add 4 new NS records, plan, verify they would be created, apply, verify they exist, plan should show no changes
  9. Remove the original 4 NS records leaving the 4 new ones, plan, verify the original ones would be deleted, apply, verify they're gone, plan should show no changes.
@@ -60,7 +72,8 @@ def plan(self, desired):

# allow the provider to filter out false positives
before = len(changes)
changes = [c for c in changes if self._include_change(c)]
changes = [c for c in changes if self._include_change(c) and
self._check_root_ns(c)]

This comment has been minimized.

@ross

ross Jan 16, 2020

Contributor

I think I'd prefer to add _check_root_ns logic to _include_change since that's the sort of thing it is intended to check, just didn't have anything to do in the base provider until now. Looks like the two existing places where _include_change is overridden don't currently call up to super, but they actually should, return super(XXX, self)._include_change(change) rather than return True.

@@ -600,6 +600,9 @@ class Route53Provider(BaseProvider):
# The AWS session token (optional)
# Only needed if using temporary security credentials
session_token:
# Needed if you want to manage your root NS records with octodns
# When you enable this you MUST specify a root NS.
manage_root_ns:

This comment has been minimized.

@ross

ross Jan 16, 2020

Contributor
Suggested change
manage_root_ns:
manage_root_ns: true
@@ -46,6 +47,7 @@ def __init__(self, id, directory, default_ttl=3600, enforce_order=True,
self.directory = directory
self.default_ttl = default_ttl
self.enforce_order = enforce_order
self.manage_root_ns = True

This comment has been minimized.

@ross

ross Jan 16, 2020

Contributor

Would prefer that this be passed in as a parameter to super rather than setting the parent class's property here.

SUPPORTS = set(('A', 'CNAME', 'MX', 'NS', 'TXT', 'AAAA'))

split_re = re.compile(r':+')

def __init__(self, id, default_ttl=3600):
super(TinyDnsBaseSource, self).__init__(id)
self.default_ttl = default_ttl
self.manage_root_ns = True

This comment has been minimized.

@ross

ross Jan 16, 2020

Contributor

Would prefer this be passed in as a parameter to super instead of setting the parent class's property here in the child.

@@ -175,9 +175,13 @@ class Ns1Provider(BaseProvider):
ns1:
class: octodns.provider.ns1.Ns1Provider
api_key: env/NS1_API_KEY
# Needed if you want to manage your root NS records with octodns
# When you enable this you MUST specify a root NS.
manage_root_ns:

This comment has been minimized.

@ross

ross Jan 16, 2020

Contributor
Suggested change
manage_root_ns:
manage_root_ns: true
@@ -224,11 +228,13 @@ class Ns1Provider(BaseProvider):
'NA': ('US-CENTRAL', 'US-EAST', 'US-WEST'),
}

def __init__(self, id, api_key, retry_count=4, *args, **kwargs):
def __init__(self, id, api_key, retry_count=4, manage_root_ns=False, *args,
**kwargs):

This comment has been minimized.

@ross

ross Jan 16, 2020

Contributor

I believe there's no need to include manage_root_ns here as an explicit parameter and pass it through to the super call, it should be in **kwargs. This happens in a few other places as well.

@@ -618,7 +622,7 @@ class Route53Provider(BaseProvider):

def __init__(self, id, access_key_id=None, secret_access_key=None,
max_changes=1000, client_max_attempts=None,
session_token=None, *args, **kwargs):
session_token=None, manage_root_ns=False, *args, **kwargs):

This comment has been minimized.

@ross

ross Jan 16, 2020

Contributor

Same here, **kwargs should hopefully cover it.

@@ -1349,6 +1350,10 @@ def _apply(self, plan):
mod_type = getattr(self, '_mod_{}'.format(c.__class__.__name__))
mods = mod_type(c, zone_id, existing_rrsets)

# Rewrite NS Record to UPSERT because it always exists.

This comment has been minimized.

@ross

ross Jan 16, 2020

Contributor

What would happen here if you started with the default 4, added 4 more with octoDNS, and then dropped things back down to just 4. I don't believe this would allow you to delete the added NS records.

A HostedZone must contain at least one NS record for the zone itself.

This reads like you can delete all but 1. Seems like deletes are possible, but it won't let you delete the "last" one?

@@ -153,7 +153,7 @@ def test_populate(self):

changes = self.expected.changes(zone, provider)

self.assertEquals(0, len(changes))
self.assertEquals(1, len(changes))

This comment has been minimized.

@ross

ross Jan 16, 2020

Contributor

The intention of this test appears to be that 0 changes happen. It should be possible to get that to be the case by adding the root NS record to the fixture and/or in-code records setup. This appears in a few other providers too.

@joschi36
Copy link
Contributor Author

joschi36 commented Jan 16, 2020

The problem is this is more a PoC at the moment and to merge it some testing and some beauty misses. I'm not a developer and its very hard to follow up on this.

@dchauran I've read your message in the other PR and wanted to ask if you or someone from your team is able to catch up on this?

@ross
Copy link
Contributor

ross commented Jan 16, 2020

I'm not a developer and its very hard to follow up on this.

Heh. Don't sell yourself short. Both of your PRs are pretty solid 😁

I wouldn't mind having this support, it just hasn't been a super high priority so I never looked at it. I can throw this onto my TODO list, but it might take a while before I get enough time for it to do all the detailed testing, and I assume handle the edge cases.

@dchauran
Copy link

dchauran commented Jan 17, 2020

@dchauran I've read your message in the other PR and wanted to ask if you or someone from your team is able to catch up on this?

@basirjamil is looking at this functionality for our team. Basir - can you please take a look at this PR and see if you think you can take it forward?

@johnlane johnlane mentioned this pull request Jul 6, 2020
phelpsw added a commit to phelpsw/octodns that referenced this pull request Jul 14, 2020
Root NS modification support is being developed on
github#434

Removing attempts to validate this functionality for now.
@acm1 acm1 mentioned this pull request Jul 23, 2020
@acm1
Copy link
Contributor

acm1 commented Jul 23, 2020

We're interested in this functionality so I've opened joschi36#6 to try to push this across the finish line. I've opened #590 with the merged, up-to-date version to verify that it passes tests.

@ross and @joschi36, could you review my changes? Thanks!

Root NS record support
@joschi36
Copy link
Contributor Author

joschi36 commented Jul 23, 2020

@acm1 I have reviewed and merged your changes to my branch. We have checked the code on our setup and It works like a charm.
After we get a review from @ross and everything is alright I think we can get this merged.

@joschi36 joschi36 requested a review from ross Jul 23, 2020
@joschi36 joschi36 changed the title WIP: Root Nameserver support Root Nameserver support Aug 3, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

4 participants
You can’t perform that action at this time.