I Tried to Break My Own Demo

A complete wiring map of a fruit fly brain was published recently, and for a few weeks people have been building demos on it. Beat Saber. Blackjack. Expense approval, for some reason.

I built one too. A webcam demo where a fly dodges your hand, using the real escape circuit from the published data. Camera to muscle, published connectivity, nothing trained.

It looks great. Then I spent a week trying to break it, and the result was not what I wanted.

The first question was simple. The fly dodges, but is the wiring doing that, or would anything shaped roughly like a brain produce a dodge?

So I scrambled it. Every neuron kept its exact number of connections in and out. Same neurons, same synapse count. Only who connects to whom was permuted.

Escape rate went from 1.00 to 0.00.

Good. The specific wiring was doing the work.

Then the question that actually matters. If this carries real visual structure, can I use it for object detection?

I froze the network, trained a small classifier on its activity, and compared against the dumbest possible baseline. Raw downsampled pixels. Same number of features, held-out backgrounds, identical classifier on both.

The pixels won two of the three tasks. On the third the connectome came out three points ahead, which at that sample size is a tie.

So it never clearly beat downsampled pixels. That is my own demo, measured honestly, failing to beat the dumbest baseline available.

The part that stung was motion. I had predicted motion would be the one task it won, because the fly optic lobe is a motion machine. Motion is where it lost worst.

I checked it was not just a weak classifier. A bigger one made everything worse, real wiring included.

And both things are true at once. Real against shuffled is significant on all three tasks. The wiring carries real structure that scrambling destroys, and it is still not better than the pixels it started from. I do not think there is a way to resolve that.

One correction I had to make. On an earlier dataset I measured that removing the two loom-sensitive cell types dropped escape to zero, and I had been repeating that. Re-running on the current data gives 0.33, not zero. Some of it arrives through other cells. The new number is the one I am posting.

The reason I am writing this up is not the fly.

The scramble test took an afternoon. If you are building on top of something you did not build yourself, run the version where you break the part you are claiming credit for, and see whether the output changes.

Mine did change, which was a relief. Then the second test said the whole thing was not useful anyway, which was not.

To be precise: one visual system’s published wiring, simulated. Not an organism, not validated against recordings. The looming signal is injected in software, so what is tested is the visual-to-motor step, not the detection.