Google Docs Auto-Upload for Scan Analysis¶

This notebook documents and exercises the automatic uploading of summary figures into the daily Google Docs scan log (issue #286).

Analyzers with gdoc_slot set will have their last summary figure inserted into the corresponding cell of a 2×2 display table in the scan entry after each analysis completes. See the live_watch notebook for the full live-running workflow.

Enabling slot-based uploads in an analyzer config¶

Add gdoc_slot to any analyzer entry in your experiment YAML:

analyzers:
  - type: array2d
    device_name: UC_GaiaMode
    priority: 0
    gdoc_slot: 0        # 0 → row 1 col 0 | 1 → row 1 col 1
                        # 2 → row 2 col 0 | 3 → row 2 col 1
    image_analyzer:
      analyzer_class: image_analysis.offline_analyzers.beam_analyzer.BeamAnalyzer
      camera_config_name: UC_GaiaMode

Analyzers without gdoc_slot will have their display files uploaded to the per-day Drive folder and appended as hyperlinks (coming in the next PR).

Persistent Drive folder (recommended)¶

By default images are uploaded to a shared staging folder that is periodically purged. To store images permanently, add ImageParentFolderID to your experiment INI (e.g. HTUparameters.ini):

[DEFAULT]
LogID               = <today's doc id, updated by createExperimentLog>
ImageParentFolderID = <drive folder id for permanent image storage>

A per-day subfolder (YYYY-MM-DD) is created automatically inside that parent folder.

In [1]:

Copied!





import logging

from geecs_data_utils import ScanPaths, ScanTag
from geecs_data_utils.config_roots import image_analysis_config, scan_analysis_config

# Show only gdoc-related logs by default; increase verbosity as needed
for name in ("image_analysis", "scan_analysis", "geecs_data_utils"):
    logging.getLogger(name).setLevel(logging.ERROR)
logging.getLogger("scan_analysis.gdoc_upload").setLevel(logging.INFO)
logging.getLogger("scan_analysis.task_queue").setLevel(logging.INFO)

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s  %(levelname)-8s  %(name)s \u2014 %(message)s",
)

image_analysis_config.set_base_dir(ScanPaths.paths_config.image_analysis_configs_path)
scan_analysis_config.set_base_dir(ScanPaths.paths_config.scan_analysis_configs_path)
import logging

from geecs_data_utils import ScanPaths, ScanTag
from geecs_data_utils.config_roots import image_analysis_config, scan_analysis_config

# Show only gdoc-related logs by default; increase verbosity as needed
for name in ("image_analysis", "scan_analysis", "geecs_data_utils"):
    logging.getLogger(name).setLevel(logging.ERROR)
logging.getLogger("scan_analysis.gdoc_upload").setLevel(logging.INFO)
logging.getLogger("scan_analysis.task_queue").setLevel(logging.INFO)

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s  %(levelname)-8s  %(name)s \u2014 %(message)s",
)

image_analysis_config.set_base_dir(ScanPaths.paths_config.image_analysis_configs_path)
scan_analysis_config.set_base_dir(ScanPaths.paths_config.scan_analysis_configs_path)

Out[1]:

WindowsPath('//131.243.169.248/HDNA2/software/control-all-loasis/HTU/Active Version/GEECS-Plugins-Configs/scan_analysis_configs')

Mode 1 — Direct upload (fastest test)¶

Upload a single known image to a specific table cell. Useful for verifying that Drive upload and insertImageToTableCell work before running any analysis.

In [ ]:

Copied!





from scan_analysis.gdoc_upload import upload_summary_to_gdoc

# ── edit these ──────────────────────────────────────────────────────────────────────────
scan_tag = ScanTag(year=2026, month=3, day=23, number=7, experiment="Undulator")

image_path = "Z:\\data\\Undulator\\Y2026\\03-Mar\\26_0323\\analysis\\Scan007\\UC_DonDSampleArea\\Array2DScanAnalyzer\\UC_DonDSampleArea_average_processed_visual.png"  # any PNG on disk

# Paste from the Google Doc URL: https://docs.google.com/document/d/<DOC_ID>/edit
# Leave as None to use the ID stored in HTUparameters.ini
document_id = "1IOzeFGcsy2gZ6Hp6hEC9CDYD9bfcDtsXx0mn5NJpWXI"

gdoc_slot = 0  # 0–3
# ────────────────────────────────────────────────────────────────────────────

ok = upload_summary_to_gdoc(
    scan_tag=scan_tag,
    display_files=[image_path],
    gdoc_slot=gdoc_slot,
    document_id=document_id,
)
print("Upload succeeded" if ok else "Upload failed or skipped (check logs)")
from scan_analysis.gdoc_upload import upload_summary_to_gdoc

# ── edit these ──────────────────────────────────────────────────────────────────────────
scan_tag = ScanTag(year=2026, month=3, day=23, number=7, experiment="Undulator")

image_path = "Z:\\data\\Undulator\\Y2026\\03-Mar\\26_0323\\analysis\\Scan007\\UC_DonDSampleArea\\Array2DScanAnalyzer\\UC_DonDSampleArea_average_processed_visual.png"  # any PNG on disk

# Paste from the Google Doc URL: https://docs.google.com/document/d/<DOC_ID>/edit
# Leave as None to use the ID stored in HTUparameters.ini
document_id = "1IOzeFGcsy2gZ6Hp6hEC9CDYD9bfcDtsXx0mn5NJpWXI"

gdoc_slot = 0  # 0–3
# ────────────────────────────────────────────────────────────────────────────

ok = upload_summary_to_gdoc(
    scan_tag=scan_tag,
    display_files=[image_path],
    gdoc_slot=gdoc_slot,
    document_id=document_id,
)
print("Upload succeeded" if ok else "Upload failed or skipped (check logs)")

Mode 2 — Full pipeline replay on a historical date¶

Uses LiveTaskRunner against a past date exactly as it would run live. rerun_completed=True resets any previously-done analyses back to queued so they execute again, and display_files are written to the status YAMLs as they complete. By default, gdoc uploading is disabled in teh LiveTaskRunner and must be explicitly enabled using gdoc_enabled=True

Note on the document ID: By default LiveTaskRunner reads LogID from the experiment INI on every upload, which is the correct behaviour for live running (the INI is updated to today's doc each morning). For a historical backtest, pass document_id="<old_doc_id>" directly to LiveTaskRunner so you don't have to edit the INI file. Find the ID in the doc URL: https://docs.google.com/document/d/<DOC_ID>/edit.

In [ ]:

Copied!





import time

from scan_analysis.live_task_runner import LiveTaskRunner

logging.getLogger("scan_analysis.live_task_runner").setLevel(logging.INFO)

# ── edit these ──────────────────────────────────────────────────────────────────────────
date_tag = ScanTag(year=2026, month=2, day=12, number=0, experiment="Undulator")
analyzer_group = "Undulator_test"  # name of the experiment config to load
# ────────────────────────────────────────────────────────────────────────────

# Paste the ID from the historical Google Doc URL, or leave None to read from
# HTUparameters.ini (correct for live running; must be set manually for backtest).
document_id = "1s69sjSeKmk_OGPRKwd-iNUSa3FflV8YP9xEuHG6C5Vo"  # e.g. "1BxiMVs0XRA5nFMdKvBdBZjgmUUqptlbs74OgVE2upms"
# ────────────────────────────────────────────────────────────────────────────

runner = LiveTaskRunner(
    analyzer_group=analyzer_group,
    date_tag=date_tag,
    config_dir=scan_analysis_config.base_dir,
    image_config_dir=image_analysis_config.base_dir,
    document_id=document_id,
    gdoc_enabled=True,
)

runner.start()
try:
    while True:
        runner.process_new(
            base_directory=ScanPaths.paths_config.base_path,
            max_items=1,
            dry_run=False,
            rerun_completed=True,  # re-run analyses already marked done
            rerun_failed=True,
        )
        time.sleep(1)
finally:
    runner.stop()
import time

from scan_analysis.live_task_runner import LiveTaskRunner

logging.getLogger("scan_analysis.live_task_runner").setLevel(logging.INFO)

# ── edit these ──────────────────────────────────────────────────────────────────────────
date_tag = ScanTag(year=2026, month=2, day=12, number=0, experiment="Undulator")
analyzer_group = "Undulator_test"  # name of the experiment config to load
# ────────────────────────────────────────────────────────────────────────────

# Paste the ID from the historical Google Doc URL, or leave None to read from
# HTUparameters.ini (correct for live running; must be set manually for backtest).
document_id = "1s69sjSeKmk_OGPRKwd-iNUSa3FflV8YP9xEuHG6C5Vo"  # e.g. "1BxiMVs0XRA5nFMdKvBdBZjgmUUqptlbs74OgVE2upms"
# ────────────────────────────────────────────────────────────────────────────

runner = LiveTaskRunner(
    analyzer_group=analyzer_group,
    date_tag=date_tag,
    config_dir=scan_analysis_config.base_dir,
    image_config_dir=image_analysis_config.base_dir,
    document_id=document_id,
    gdoc_enabled=True,
)

runner.start()
try:
    while True:
        runner.process_new(
            base_directory=ScanPaths.paths_config.base_path,
            max_items=1,
            dry_run=False,
            rerun_completed=True,  # re-run analyses already marked done
            rerun_failed=True,
        )
        time.sleep(1)
finally:
    runner.stop()

2026-03-25 20:37:55,500  INFO      scan_analysis.gdoc_upload — Loaded logmaker_4_googledocs.docgen successfully.
2026-03-25 20:37:57,590  INFO      scan_analysis.live_task_runner — Started watching Z:\data\Undulator\Y2026\02-Feb\26_0212\analysis
2026-03-25 20:38:00,144  INFO      scan_analysis.task_queue — run_worklist: claiming scan=year=2026 month=2 day=12 number=4 experiment='Undulator' analyzer=Amp2Input priority=40 dry_run=False
2026-03-25 20:38:00,204  INFO      scan_analysis.task_queue — run_worklist: completed scan=year=2026 month=2 day=12 number=4 experiment='Undulator' analyzer=Amp2Input priority=40 display_files=None
2026-03-25 20:38:02,053  INFO      scan_analysis.task_queue — run_worklist: claiming scan=year=2026 month=2 day=12 number=5 experiment='Undulator' analyzer=Amp2Input priority=40 dry_run=False
2026-03-25 20:38:20,271  INFO      scan_analysis.task_queue — run_worklist: completed scan=year=2026 month=2 day=12 number=5 experiment='Undulator' analyzer=Amp2Input priority=40 display_files=['Z:\\data\\Undulator\\Y2026\\02-Feb\\26_0212\\analysis\\Scan005\\UC_Amp2_IR_input\\Array2DScanAnalyzer\\UC_Amp2_IR_Input_averaged_image_grid.png']
2026-03-25 20:38:22,999  INFO      scan_analysis.task_queue — run_worklist: claiming scan=year=2026 month=2 day=12 number=6 experiment='Undulator' analyzer=Amp2Input priority=40 dry_run=False
2026-03-25 20:38:24,385  INFO      scan_analysis.task_queue — run_worklist: completed scan=year=2026 month=2 day=12 number=6 experiment='Undulator' analyzer=Amp2Input priority=40 display_files=['Z:\\data\\Undulator\\Y2026\\02-Feb\\26_0212\\analysis\\Scan006\\UC_Amp2_IR_input\\Array2DScanAnalyzer\\UC_Amp2_IR_Input_average_processed_visual.png']
2026-03-25 20:38:26,258  INFO      scan_analysis.task_queue — run_worklist: claiming scan=year=2026 month=2 day=12 number=7 experiment='Undulator' analyzer=Amp2Input priority=40 dry_run=False

In [ ]: