Not sure if this is a bug in Composer or Finder\Glob::toRegex or just a borked .gitignore rule that should have no place being there in the first place, but if someone can fix it I guess why not. Doesn't seem very high prio though.

I've had a look at this issue and my conclusion would be that it is a borked .gitignore rule that should have no place being there in the first place.

The test case graciously provided by @mborne shows that the issue is still reproducible on the current master.

The only adjustment to Finder\Glob::toRegex which could be made IMO, would be a check that any opening brace also has a close brace and if not, escape the brace instead of changing it.
Aside from the fact that that check would be quite complicated as it would need to account for already escaped braces, I'm not sure that the method should account for invalid glob patterns.

As for Composer, a simple solution could be to prefix the filter preg_match() with the error silencing operator.
preg_match() would return false anyway, so the value of $exclude would not be changed and the tar file would be created as expected.