Friday, 2:30 AM, Hilton Odawara
Family trip. A proper Friday night.
And yet there I was, MacBook open on a lobby sofa. A BigQuery batch job refusing to finish. Estimated time remaining: 100 hours.
100 hours. More than four days. Starting over would be faster.
The problem was clear enough. Parallelism tuning, JOIN pattern optimization, clustered table utilization. I could see every possible move. Which one to play, in what order — that’s what I was working through.
Two hours of circling at the same altitude.
Five Minutes
I threw the problem at AI. Fed it the codebase, explained the situation.
The structural analysis came back clean.
Nine areas running in parallel, but one of them had orders-of-magnitude more data than the rest. Heavy queries on that area took tens of times longer than the others.
Parallelize all nine evenly, and that single area becomes the bottleneck dragging everything down.
The answer was simple. Isolate the heavy area.
Run it as a separate job with time-range partitioning. The remaining eight finish in hours. Query rewrites can wait until next week.
Five minutes. A problem I’d been buried in for two hours dissolved in a five-minute structural review.
Seeing Too Many Veins
Why did I stay buried for two hours?
Because I knew that rewriting a query could cut processing by 80%. I knew that redesigning a table would stabilize the pipeline. I understood resource contention patterns at higher parallelism, the platform’s rate limits — all of it was already in my head.
When you can see every vein in every leaf, you optimize inside the leaf.
Rewrite one query for a 10% speedup. Adjust parallelism for 15%. Redesign a table for 80% cost reduction. All correct. All “inside moves.”
The question “should we restructure the processing architecture itself?” doesn’t emerge from inside the leaf.
No Separation Between “Strategy Person” and “Implementation Person”
In most organizations, there are two kinds of people. Those who see the forest, and those who work on the veins.
A manager says “think about the big picture.” An engineer says “I’ll fix this query.” The perspective shift happens naturally — between people.
When you do everything solo, that shift lands entirely on you.
The brain writing SQL thinks “how do I fix this line.” The brain doing strategy thinks “do we even need this job.” Same person, two modes of thinking that must be manually switched.
And here’s the treacherous part — working in the veins feels good. Moving your hands means making progress. Every query fixed delivers a small hit of accomplishment.
Returning to the forest means cutting off that dopamine yourself.
Build a System That Pulls You Back
Willpower-based perspective shifting doesn’t scale. If an exhausted brain at 2:30 AM could “step back and think big,” it wouldn’t have been buried for two hours in the first place.
So build a system that forces the pullback.
That night, the system was AI. It read the codebase in five minutes, organized the structure, and returned “what if you just isolate the heavy area?”
The important thing isn’t that AI produced the right answer. It’s that a pulled-back perspective was forcibly injected into my process.
What someone buried in the veins lacks isn’t knowledge or skill. It’s a trigger for the zoomed-out view.
A sparring partner, a coach, an AI — anything works. Systematize “when you’re deep, throw it out.” Elevating perspective isn’t a talent. It’s a habit.
Shave 10% with Tactics, or Eliminate 90% with Strategy
Facing a 100-hour job, I was debating whether to raise parallelism from 4 to 9. Best case, a 40% reduction. Still 60 hours.
Isolating the heavy area alone cuts the remaining work to hours. Run the heavy one separately with time-range partitioning. Total processing time drops by over 90%.
If you’re going to spend two hours either way, spend them changing the strategy, not polishing the tactics.
The ability to dive into veins is a weapon. Touching every domain solo is an irreplaceable strength. But that weapon can also become the chain that pins you to the leaf.
Being able to dive and staying submerged are not the same thing.
When you dive, surface once. Build the surfacing mechanism before you need it.
The forest is invisible from inside the veins.