apache / spark Public

…eferences

### What changes were proposed in this pull request?
Retry `ClassSymbol.selfType` in case we receive a cyclic reference exception.

### Why are the changes needed?
`ClassSymbol.selfType` fails on certain classes like `HiveGenericUDF` due to cyclic reference. This issue is due to [bug#12190 in Scala](scala/bug#12190) which does not handle cyclic references in Java annotations correctly. The cyclic reference in this case comes from `InterfaceAudience` annotation which [annotates itself](https://github.com/apache/hadoop/blob/db8ae4b65448c506c9234641b2c1f9b8e894dc18/hadoop-common-project/hadoop-annotations/src/main/java/org/apache/hadoop/classification/InterfaceAudience.java#L45). This annotation class is present in the type hierarchy of `HiveGenericUDF`.

To workaround the issue, we can just retry the operation and it will succeed on the retry probably because the annotation is partially resolved from the previous attempt.

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
Added a unit test. This unit test prefers to be run a separate JVM as the error for cyclic references may not be thrown if the annotation class is previously loaded by some other test and so may be dependent on test execution order.

Closes #35852 from shardulm94/spark-38510.

Authored-by: Shardul Mahadik <smahadik@linkedin.com>
Signed-off-by: Sean Owen <srowen@gmail.com>