List of objects or parallel arrays of properties?
The question is, in principle: which would be preferable, both in terms of performance and design, to have a list of Python class objects, or to have multiple lists of numeric properties?
I am writing some kind of scientific simulation that involves a fairly large system of interacting particles. For simplicity, let's say we have a set of balls bouncing inside a box, so that each ball has a number of numerical properties like xyz coordinates, diameter, mass, velocity vector, etc. What is the best way to store the system? I can think of two main options:
to create a "Ball" class with these properties and some methods, and then store a list of class objects, e. d. [b1, b2, b3, ... bn, ...] where for each bn we can access bn.x, bn.y, bn.mass, etc.
to make an array of numbers for each property, then for each i-th "ball" we can access the "x" coordinate as xs [i], 'y' coordinate as ys [i], 'mass' as mass [i] etc.;
It seems to me that the first option represents the best design. The second option looks a little ugly, but it might be better from a performance standpoint and it would be easier to use it with numpy and scipy, which I try to use as much as possible.
I'm still not sure if Python is going to be fast enough, so it might need to be rewritten in C ++ or whatever after initial prototyping in Python. Will the choice of data representation be different for C / C ++? What about the hybrid approach, for example. Python with C ++ extension?
Update: I never expected any performance gain from parallel arrays per se, but in a mixed environment like Python + Numpy (or something like SlowScriptingLanguage + FastNativeLibrary) using them can (or not)? you're moving more work out of the slow script code and into the fast native library.
a source to share
Having an object for each ball in this example is by far the best design. Parallel arrays are actually a workaround for languages that don't support matching objects. I wouldn't use them in an OO-capable language unless it's a tiny case that fits into a function (and maybe not even then), or if I run out of all other optimization options and the profiler shows that property access is the culprit. This is twice as true for Python as compared to C ++, as there has been a lot of emphasis on readability and elegance in the past.
a source to share
I agree that parallel arrays are almost always a bad idea, but don't forget that you can use views to a numpy array when you're setting things up, but ... (Yes, I know this is effectively using parallel arrays, but I think what is the best option in this case ...)
It's great if you know the number of "balls" you are going to create in advance, since you can allocate an array for the coordinates and store a representation in that array for each ball object.
You have to be a little careful to do in-place operations on the coords array, but it makes updating coordinates for many "balls" much, much, much faster.
For instance...
import numpy as np
class Ball(object):
def __init__(self, coords):
self.coords = coords
def _set_coord(self, i, value):
self.coords[i] = value
x = property(lambda self: self.coords[0],
lambda self, value: self._set_coord(0, value))
y = property(lambda self: self.coords[1],
lambda self, value: self._set_coord(1, value))
def move(self, dx, dy):
self.x += dx
self.y += dy
def main():
n_balls = 10
n_dims = 2
coords = np.zeros((n_balls, n_dims))
balls = [Ball(coords[i,:]) for i in range(n_balls)]
# Just to illustrate that that the coords are updating
ball = balls[0]
# Random walk by updating coords array
print 'Moving all the balls randomly by updating coords'
for step in xrange(5):
# Add a random value to all coordinates
coords += 0.5 - np.random.random((n_balls, n_dims))
# Display the coords for a particular ball and the
# corresponding row of the coords array
print ' Value of ball.x, ball.y:', ball.x, ball.y
print ' Value of coords[0,:]:', coords[0,:]
# Move an individual ball object
print 'Moving a ball individually through Ball.move()'
ball.move(0.5, 0.5)
print ' Value of ball.x, ball.y:', ball.x, ball.y
print ' Value of coords[0,:]:', coords[0,:]
main()
To illustrate this, you end up with something like:
Moving all the balls randomly by updating coords
Value of ball.x, ball.y: -0.125713650677 0.301692195466
Value of coords[0,:]: [-0.12571365 0.3016922 ]
Value of ball.x, ball.y: -0.304516863495 -0.0447543559805
Value of coords[0,:]: [-0.30451686 -0.04475436]
Value of ball.x, ball.y: -0.171589457954 0.334844443821
Value of coords[0,:]: [-0.17158946 0.33484444]
Value of ball.x, ball.y: -0.0452864552743 -0.0297552313656
Value of coords[0,:]: [-0.04528646 -0.02975523]
Value of ball.x, ball.y: -0.163829876915 0.0153203173857
Value of coords[0,:]: [-0.16382988 0.01532032]
Moving a ball individually through Ball.move()
Value of ball.x, ball.y: 0.336170123085 0.515320317386
Value of coords[0,:]: [ 0.33617012 0.51532032]
The advantage here is that updating a single numpy array will be much, much faster than iterating over all of your ball objects, but you keep a more object oriented approach.
Just my thoughts on this, anyway.
EDIT: To give some idea of the difference in speed, with 1,000,000 balls:
In [104]: %timeit coords[:,0] += 1.0
100 loops, best of 3: 11.8 ms per loop
In [105]: %timeit [item.x + 1.0 for item in balls]
1 loops, best of 3: 1.69 s per loop
So, updating coordinates directly with numpy is about 2 orders of magnitude faster when using a lot of balls. (the difference is less when using 10 balls, as an example, about 2 times, not 150x)
a source to share
I think it depends on what you are going to do with them and how often you will work (all attributes of one particle) vs (one attribute of all particles). The former is better suited for an object approach; the latter is better for the array approach.
I ran into a similar problem (albeit on a different domain) a couple of years ago. The project was canceled before I actually implemented this step, but I was leaning towards a hybrid approach where, in addition to the Ball class, I would have an Ensemble class. The ensemble would not be a list or other simple container of balls, but it would have its own attributes (which would be arrays) and its own methods. Whether you create an ensemble of balls or balls from an ensemble depends on how you are going to create them.
One of my colleagues argued about a solution where the main object was an ensemble that could only hold one ball, so that no caller would ever know if you were only running on one ball (do you ever do this for your application? ) or many.
a source to share
Will you have any forces between the balls (hard sphere / collision, gravity, electromagnet)? I think so. Will you have enough balloons to want to use Barnes-Hut ideas ? If so, then you should definitely use the idea of the Ball class so that you can easily store them in octets or whatever along those lines. In addition, using Barnes-Hut simulation will reduce simulation complexity to O (N log N) from O (N ^ 2).
Indeed, if you don’t have forces between balls, or you don’t use many balls, you don’t need the speed gains from using parallel arrays, and you should also follow the idea of the Ball class.
a source to share