Raw conversation logs become unreadable at scale. After a few months of heavy use, you are staring at hundreds of thousands of lines of back-and-forth. Searching them is slow; reading them is impossible. An Obsidian vault, by contrast, gives you structured long-term memory: bi-directional links, tags, and a graph view that surfaces connections. The gap between the two is obvious. Closing it requires automation.
I built a small pipeline that bridges that gap every night. It runs three steps in sequence, with no manual intervention. First, it harvests the latest conversation exports and refreshes them to their current state. Second, it distills the prior 28 hours of raw dialogue through Claude, appending structured summaries directly into the Obsidian vault. That 28-hour window is deliberate; it covers the full previous day plus a small buffer for delays. Third, it backs everything up by committing and pushing the changes to a private repository, which acts as both offline storage and accidental-deletion insurance. The entire chain has to be boring and reliable. Boring automation is the goal.
When macOS Silently Kills Your Script
The biggest enemy of a nightly job is not a dramatic crash. It is silence.
On macOS, the Documents folder lives under the protection of TCC, the privacy subsystem that gatekeeps access to sensitive locations. If you schedule your script through launchd, the system may simply block the process without throwing an error you will notice. Your log file stays empty. Your vault never updates. The job failed, but you only discover the gap days later when you realize your notes are stale.
I handle this with a strict preflight routine. Before the main work begins, the script attempts a harmless write operation inside the target directory. If macOS denies it, the failure is caught immediately and a loud desktop notification fires. There is no quiet exit. Next, the shell interpreter itself needs Full Disk Access granted inside System Settings. That permission is blunt, so I minimize exposure by keeping all automation scripts outside the protected Documents folder tree entirely. They live in a dedicated directory elsewhere, and the vault path is referenced explicitly. That combination of early detection, noisy alerts, and careful folder placement fixes the silent death scenario.
Designing for Sleep, Timeouts, and Frozen Jobs
Laptops sleep. Networks hiccup. API calls hang. A single frozen job should not stall the entire system indefinitely.
I use four practical mechanisms to keep the pipeline stable.
Multiple time slots. Instead of betting everything on a single early morning run, the scheduler tries again at different points throughout the day. If the machine was closed at dawn, the afternoon window still catches it.
Done markers. When a run succeeds, it leaves a clear breadcrumb. The next scheduled attempt checks for that marker first. If the work is already complete, it exits cleanly within seconds. This prevents unnecessary API calls and redundant processing.
File locks through mkdir. To avoid overlapping executions, I use a directory-based lock. Creating a directory is atomic on local filesystems, so two simultaneous jobs cannot claim the same lock path. If the directory already exists, the second instance aborts immediately. When the first finishes, it removes the directory. This is lighter than PID files and harder