Feedback Request: Limitations on Data Classes
While M13 is approaching, we are planning a little ahead. This is a request for feedback on some future changes in Kotlin.
We want to deliver Kotlin 1.0 rather sooner than later, and this makes us postpone some design choices we don’t have enough confidence about. Today let’s discuss data classes.
Introduction
The concept of data classes has proven very useful when it comes to simply storing data. All you need is say:
data class Foo(val a: A, val b: B)
and you get equals()/hashCode(), toString(), copy() and component functions for free.
The most common use case works like a charm, but interaction of data classes with other language features may lead to surprising results.
Issues
For example, what if I want to extend a data class? What if the derived class is also a data class?
open data class Base(val a: A, val b: B) data class Derived(a: A, b: B, val c: C) : Base(a, b)
Now, how does equals() or copy() work in Derived? All the well-known issues arise at once:
- should an instance of
Basebe equal to an instance ofDerivedif they have the same values foraandb? - what about transitivity of
equals()? - what if I copy an instance of
Derivedthrough a reference of typeBase?
And what about component functions that enable multi-declarations? It seems more or less logical that c simply becomes the third component in Derived in this basic case:
val (a, b, c) = Derived(...)
But nothing prevents us from writing something like this:
data class Derived(b: B, a: A, val c: C) : Base(a, b)
Note that the parameter order is reversed: first b, than a. Now it’s not that clear any more. And it may get worse:
data class Derived(val c: C, b: B, a: A) : Base(a, b)
Now c comes first, and the inherited component1(): A is simply a conflict, it is not an override, but such an overload is not legal either.
And these are only some examples, there’re many more issues, big and small.
Our strategy
On the one hand, we are not sure whether there is an elegant design for inheritance involving data classes. We have some sketches, but none of them looks promising enough.
On the other hand, we want to finalize the language design now, to be able to ship 1.0.
So, we decided to restrict data classes quite a bit to rule out all the problematic cases in 1.0, so that we can get back to them later and maybe lift some of the restrictions.
Proposed restrictions
We are going to do the following:
- allow to inherit data classes from interfaces
- forbid to inherit data classes from other classes
- forbid open data classes (i.e. other classes can not extend data classes)
- forbid inner data classes (not clear how
equals()/hashCode()should treat the outer reference) - allow local data classes (the closure is not structured, so it’s OK for
equals()/hashCode()to ignore it) - require
val/varon all primary constructor parameters for data classes - require at least one primary constructor parameter for data classes
- allow private primary constructor parameters for data classes
var’s are as good asval’s in all respects (they participate inequals()/hashCode()etc)- forbid
varargsin primary constructor parameters for data classes
Again, some of the restrictions in this list may be lifted later, but for now we don’t want to deal with these cases.
Appendix. Comparing arrays
It’s a long-standing well-known issue on the JVM: equals() works differently for arrays and collections. Collections are compared structurally, while arrays are not, equals() for them simply resorts to referential equality: this === other.
Currently, Kotlin data classes are ill-behaved with respect to this issue:
- if you declare a component to be an array, it will be compared structurally,
- but if it is a multidimensional array (array of arrays), the subarrays will be compared referentially (through
equals()on arrays), - and if the declared type of a component is
AnyorT, but at runtime it happens to be an array,equals()will be called too.
This behavior is inconsistent, and we decided to fix it following the path of least resistance:
- arrays are always compared using
equals(), as all other objects
So, whenever you say
arr1 == arr2arr in setOfArraysDataClass(arr1) == DataClass(arr2)- or anything else along these lines,
you get the arrays compared through equals(), i.e. referentially.
We’d love to fix the inconsistency with collections, but the only sane way of fixing it seems to be fixing it in Java first, which is beyond anybody’s power, AFAIK 🙂
Call for feedback
Please share your opinion on the proposed changes. We are more or less sure about arrays, and pretty confident about limitations on data classes too, but it’s always a good idea to double-check with a wider range of use cases.
Thanks for your help!
Artem Zinnatullin says:
September 9, 2015LGTM!
I’d also suggest you to look at AutoValue’s README to their design principles, AutoValue brings immutable value classes to Java with help of annotation processing.
Christian says:
September 9, 2015require val/var on all primary constructor parameters for data classes
Why? Normal parameters can be pretty useful. I’d love this to be kept allowed.
Andrey Breslav says:
September 9, 2015I’m not saying they are not useful. I’m saying that we are not ready to decide on the intricacies that they bring at the moment.
Paul says:
September 9, 2015Why can’t we forbid to reorder parameters in data classes? It can be checked in compiletime.
Andrey Breslav says:
September 10, 2015We can. It’s a tiny part of a possible design. There are too many such parts for us to be confident about arranging them the right way under the time pressure.
Paul says:
September 10, 2015One else semi-obvious decision is just skip parent constructor argemunts in child cpnstructor while data class inheritance.
Thing that frustrates me is that we loose beatiful hibernate inheritance things 🙁
Rob Bygrave says:
September 9, 2015“Composition” of data classes is still available right. So inheritance is not an option but composition was probably my preferred option anyway. Yes, happy with those restrictions on data classes for my use cases.
So with var’s I’d expect the hashCode() value can change. Mutable data classes are very handy so I’m happy there but that is going to catch the uninitiated with use in Set’s etc. No difference to Java here but good documentation on var’s / changing hashCode() values might be good.
My 2c says “fine by me”. For me the need/use of “structural equals for array” is extremely rare (for what it’s worth I don’t remember fussing on this in 17 years of Java coding) so for me this is fine.
Mikael Gueck says:
September 9, 2015Is it a priority to be able to use Kotlin 1.0 easily with JPA and Spring Boot?
Andrey Breslav says:
September 9, 2015I’d say yes, why?
Mikael Gueck says:
September 9, 2015The continuing lack of Serializable in various Kotlin classes typically used in JPA @Id fields is a major roadblock in this kind of basic usage.
Andrey Breslav says:
September 9, 2015We are working on this. Unfortunately, it turned out to be a lot more work than we anticipated, but we’ll finish it by 1.0
Peter Niederwieser says:
September 10, 2015How about using Kotlin with JUnit? Not being able to declare public instance fields (correct me if I’m wrong, but I couldn’t find a way) means missing out on one of JUnit’s most important features (@Rule). To work around this, I had to introduce Java base classes for my tests. JUnit isn’t the only library with this requirement; without a way to declare public instance fields, the Java interop story isn’t complete.
Andrey Breslav says:
September 10, 2015This is an important case. We have partly addressed it in M13 (coming soon), and will provide some more support a little later
Rob Bygrave says:
September 9, 2015I’m curious. What Kotlin classes are you using in JPA @Id fields?
Mikael Gueck says:
September 10, 2015See this-SHOULD-work vs. master branches:
https://github.com/mikaelhg/kotlin-spring-boot-data-rest/tree/this-SHOULD-work/src/main/kotlin/io/mikael/app
vs
https://github.com/mikaelhg/kotlin-spring-boot-data-rest/tree/master/src/main/kotlin/io/mikael/app
Rob Bygrave says:
September 10, 2015So I had a quick look and it is not clear to me. The branches seem to be swapping between @Id of Long and Key and both should work fine and both of those types are nothing to do with Kotlin per say so I’m missing your point/issue.
Certainly there are no issues with Kotlin and Ebean ORM (which uses JPA mapping and entity bean enhancement that would be similar to Eclipselink).
It seems that you are expecting to use Kotlin data classes as JPA entity beans which is interesting. As the author of Ebean ORM I’m not going to be recommending that approach to anyone as most commonly it is good practice to use inheritance with entity beans and have a @MappedSuperclass bean with common properties such as @Id, @Version, @WhoCreated, @WhoModified, @WhenCreated, @WhenModified etc.
Data classes implement hashCode()/equals() so that could conflict with JPA vendor enhancement/weaving (when using data classes as @EmbddedId for example).
Sorry, probably no helpful comments there.
Cheers, Rob.
Adel says:
September 9, 2015so basically we should avoid Arrays in data classes or provide our own equals() every time?
Andrey Breslav says:
September 9, 2015I’d say you should avoid arrays everywhere (including your pure Java projects) unless you are doing some low-level optimizations.
And yes, if you want structural equality for arrays anywhere (including pure Java code), you have to provide custom implementations for
equals()/hashCode()M Platvoet says:
September 9, 2015For now this restrictions seems logical to me. Rather restrictive now and expand later then the other way around.
I believe equals (and hashcode) should work something along the lines like this
example. So a data class should only equal the exact same type, everything else should be nonsense. But you could use propertiesEquals to test properties of unequal types.
Peter Gromov says:
September 9, 2015Maybe I’m too big a fan of purity, but allowing mutable properties to participate in equals/hashCode by default sounds like a heresy to me. Such a nice opportunity to shoot oneself in the foot! OTOH silently excluding them from equals/hashCode would probably also be unexpected by the code authors 🙁
Andrey Breslav says:
September 9, 2015This is exactly the concern we had, but simply disallowing var’s would be overly restrictive in some use cases. This is a discouraged practice, as having vars anywhere else, basically.
Gordon Tyler says:
September 10, 2015I too am disturbed by the prospect of mutable vars in a data class.
I was wondering if it would be possible to declare a data class as mutable or immutable and then assert on the use of mutable data classes in places whether they would cause problems, for example anywhere that depends on a stable hashCode.
Andrey Breslav says:
September 11, 2015I think the best we can do for you is have an opt-in inspection that would warn you on the declaration site that
var‘s in data classes require careful treatmentRob Bygrave says:
September 11, 2015I think the issue here is that we only know this (places where we want stable hashCode) based on knowing how specific implementations (HashMap, HashSet etc) actually work – hence Andrey’s answer of an inspection warning.
Aaron Sarazan says:
September 9, 2015+1. I think every one of the data class restrictions is completely reasonable.
I throw up my hands on arrays because that’s a pretty messed up situation to begin with.
Rodrigo Quesada says:
September 9, 2015I mostly like the data annotation (I know it’s a modifier, but I’m gonna call it like that here) because it allows me to have equals/hashCode/toString methods automatically generated, I don’t care too much about the other functions, though.
I think a good way of making everyone happy would be to allow for various configurations on the data annotation, kind of like project Lombok (https://projectlombok.org/features/index.html) allows you to configure certain annotations and fine-tune what you get in Java (such as the callSuper option of the EqualsAndHashCode annotation https://projectlombok.org/features/EqualsAndHashCode.html).
The only issue with that would be that you could get pretty lengthy data class declarations when specifying multiple options, so it would be definitely great that besides that you also provide a way to create custom data annotations (that would be allowing to annotate your annotations with the data annotation or being able to alias it on a per-module basis), that way you get your own data annotations that behave exactly as you want/prefer. Something like this maybe:
data(sameClassEquals=true, transitive=true) class Foo(val a: A, var b: B)
Regarding the restriction of only being able to use val for data classes, I’m glad you will not be doing that, I definitely like freedom when modeling my classes (such as aggregates/entities when using DDD).
Also, something else I would like to add is that I would certainly love being able to annotate non-primary-constructor properties and get them included in equals/hashCode/toString calculation for the data class. Something like what I commented on this ticket: https://youtrack.jetbrains.com/issue/KT-8466
class A() {
data val a: Int
get() = 123*321 //I don't know, do something cooler here instead
}
In this particular edge case (i.e. a class with data annotations only on its properties), though, you would not get component or copy methods generated, but you would get the rest for free (which I would certainly hate having to implement manually).
Rodrigo Quesada says:
September 9, 2015Sorry, I screwed-up the text formatting but I’m unable to edit the comment.
Andrey Breslav says:
September 10, 2015Fine-grained annotations along the lines of what you are proposing are under consideration for future versions. Thanks!
Oliver Plohmann says:
September 10, 2015I think it’s a good idea you are aiming for delivering Kotlin 1.0 rather sooner than later. One more year and Scala gets more and more ground and people waiting for Kotlin might get tired of waiting. Once KT-3029 is implemented real Kotlin life will start :-).
Andrey Breslav says:
September 10, 2015I don’t think KT-3029 will be implemented per se. Most likely, we’ll just forbid protected in interfaces on the JVM
Oliver Plohmann says:
September 10, 2015Do you mean it won’t be implemented for Kotlin 1.0 or do you mean it won’t be implemented at all? Thanks.
Andrey Breslav says:
September 10, 2015Since Java does not allow
protectedin interfaces on the class file level, there’s no way to implement this properly on the JVM. So, unless we find a really clever trick, it won’t happen in any version of Kotlin/JVM.Oliver Plohmann says:
September 10, 2015Traits may have protected vars and methods in Scala. I just tried it out to be sure. Think also Ceylon has that. So they did find some trick. I really hope Kotlin will have protected methods in traits. This is for modelling purposes very important to make sure encapsulation of the class extending the trait is not broken. Really hoping it can be done :-).
Alexander Udalov says:
September 14, 2015Protected vars and methods in Scala are compiled down to public methods, which is unfortunately not very “protected”.
Oliver Plohmann says:
September 14, 2015@Alexander: Thanks for sharing this. A trait might be in a different package than the class extending it. So methods/vars in a trait have to be public. Painful situation, really ;-).
Andrey Breslav says:
September 15, 2015Everything in an interface must be public (private static members are allowed since Java 8), there’s no way around it on the JVM.
Jacob Zimmerman says:
September 10, 2015Those data class restrictions seem reasonable. Personally, I don’t think I’ll ever use vars on mine, but I don’t mind you leaving that possibility open.
Karl Leopold says:
September 10, 2015For beginners, it may be useful to get a compiler warning if they use data classes with var properties as keys in maps or sets.
I love these restrictions, because they make data classes very easy and intuitive to use. For more complex cases, there are still regular classes.
Array handling sounds good.
Michael Rush says:
September 10, 2015We’ve got about 60 data classes in our current codebase and all conform to the proposed restrictions. I guess this is because they were created according to the “spirit” of how data classes are intended to be used. The restrictions sound reasonable to me. Looking forward to 1.0!
Jon Schneider says:
September 24, 2015Final-by-default is the most inconvenient feature of Kotlin to me at the moment. Trying to pair Spring with Kotlin, I often find myself opening classes and methods as I add Spring features to allow Spring to generate proxies as necessary. Final-without-recourse really scares me.
The status quo feels like being engrossed in a great film only to see the sound man suddenly walk into the scene. It is jarring, and shatters the magic of the moment.
Is it possible to make final-by-default a compile-time only constraint? After all, in the case of a Spring RestController class that I need to open for Spring to do its magic, I have no intention of ever inheriting from the type and would be perfectly happy to have the compiler enforce this so long as the final modifier didn’t make it into the bytecodes.
Chris Kent says:
October 1, 2015I know I’m late to the party but I was on holiday and only just saw this post.
In my day job I have to use primitive arrays quite often (unfortunately). If data classes compare arrays by reference it would make them a lot less useful for cases like mine. I would have to wrap them in a type that correctly implements
hashCode()andequals()or manually writehashCode()andequals()methods for any data classes containing arrays. That would remove a lot of the benefits of using a data class.I think it would be better for data classes to use
Arrays.deepEquals()for comparing arrays. Using reference equality will inevitably lead to subtle bugs. The caller would have to check the types of all the fields in the data class to know how equality works for that particular class.I understand that arrays aren’t very popular, and for good reason, but when you need them there is nothing else that will do. It would be a shame if data classes were broken WRT arrays when it shouldn’t be hard to support them.
I don’t think it’s a problem that the
equals()method of a data class containing an array won’t have the same behaviour as using==on two array references. If you’re using arrays you already need to be aware of the pitfalls and you have the option of usingArrays.deepEquals()to get the correct behaviour. But if reference equality checking is baked into data classes there is no workaround.Andrey Breslav says:
October 1, 2015I can’t agree. If in a rare case you can’t use a data class and have to resort to manually implementing equals, it’s not very much a problem, IMO.
Chris Kent says:
October 1, 2015What are the downsides of supporting arrays properly in data classes?
Andrey Breslav says:
October 2, 2015Inconsistency and unpredictability. It’s better to learn that arrays are compared by identity everywhere, once and for all than debug weird behaviours with special-case behaviours. Scala has been there, and it’s very hard
Zdenek says:
October 23, 2015Allowing data classes to inherit from interface with combination of non-val/var parameters – Does it allow you something more than using tag interfaces on data classes?
The reason is why I’m asking is this simle case:
interface Pet {
val name: String
}
// I'm lost here as I cannot have non-val/var in constructor and cannot override
data class Puppy(name: String) : Pet {
override val name = name
}
Did I miss something or you simply cannot make use of “reasonable” interface usage?
Andrey Breslav says:
October 23, 2015Zdenek says:
October 26, 2015Oops, thank you. I tried something similar but with wrong syntax. Maybe it is worth noting in data class documentation