How Steelflo Works: A Technical Overview of AI Steel Takeoff

Steelflo is an AI-powered structural steel takeoff tool that extracts every steel member label from a set of PDF construction drawings, locates each label on the page with a precise bounding box, and outputs a verified bill of materials ready for pricing, fabrication, or procurement. The entire process runs through a multi-stage extraction pipeline that combines deterministic text parsing with AI vision fallback, followed by a human verification step that delivers 95–99% accuracy on the final BOM.

This article documents exactly how the pipeline works, what data it produces, and where its current limits are.

The Extraction Pipeline

Steelflo's core is a multi-stage extraction pipeline. Each stage has a specific job, and the output of one feeds the next. Here is the full sequence from PDF upload to final results.

Stage 1: PDF Upload and Page Scanning

The user uploads a PDF through Steelflo's 6-step wizard. The file is stored in Supabase cloud storage and a takeoff record is created in the database. The worker service receives the PDF buffer and begins processing.

Stage 2: Text Extraction with pymupdf

Every page of the PDF is read by pymupdf (the Python binding for MuPDF). For each page, pymupdf extracts all text content along with the exact position of every text span in PDF-point coordinates. This is not OCR — pymupdf reads the embedded text layer directly, which means it works at 100% accuracy on any PDF with real text (as opposed to scanned/raster images).

The extraction returns raw text spans with their bounding box coordinates in the PDF coordinate system (origin at bottom-left, measured in points at 72 points per inch).

Stage 3: Standard Auto-Detection

Before running pattern matching, the pipeline needs to know which steel naming convention the drawings use. Steelflo scans all pages and counts signature pattern matches for each supported standard:

AISC patterns produce matches like W12X26, HSS6X6X1/4
BS/IS patterns produce matches like UC305*305*158, SHS220*220*6.0
AS/NZS patterns produce matches like 310UB40.4, 250UC89.5
EN patterns produce matches like HEA200, IPE300

The standard with the highest match count wins. A confidence score is computed based on how dominant the winning standard is relative to others. This auto-detection routes the entire drawing set to the correct pattern library without any user input.

Stage 4: Regex Pattern Matching

With the standard identified, the pipeline runs steel-specific regular expressions against every extracted text span. Each standard has its own pattern library:

AISC: W-sections, HSS (square, rectangular, round), angles (L), channels (C/MC), pipes, plates (PL), S-sections, WT-sections
BS/IS: 10 regex patterns covering UC, UB, SHS, RHS, CHS, PFC, PG, angles, and general H/I profiles. Also decodes structured member IDs (e.g., 01-STR-06-VBR018 maps to "Vertical Bracing")
AS/NZS: 11 regex patterns for the number-first format used in Australia and New Zealand (e.g., 310UB40.4, 250UC89.5, 150PFC, plus SHS/RHS/CHS)

When a regex matches, the pipeline records the normalized member type, raw display text, category (beam, hss, pipe, plate, channel, angle, or misc), and the exact bounding box coordinates from pymupdf. Every match becomes one detection object with a unique ID.

Stage 5: Vision AI Fallback

Some PDF pages contain steel labels that pymupdf cannot read as text. This happens with CAD-generated drawings that use vector fonts (polyline lettering), or pages where text is embedded as paths rather than characters. The pipeline identifies these pages automatically: any page with zero text-based steel labels but more than 10,000 vector paths is flagged as needing vision fallback.

Flagged pages are rendered as 3x resolution PNG images (e.g., 10,368 x 7,777 pixels for a typical structural drawing) and sent to Google's Gemini Vision AI. Gemini analyzes the image and returns bounding boxes for detected steel labels using normalized coordinates (0-1000 scale), which are then scaled to pixel coordinates.

This two-path approach — deterministic text extraction first, AI vision only where text extraction fails — keeps accuracy high and processing costs low. Most pages in a typical drawing set have readable text; only CAD vector-font pages need the vision fallback.

Stage 6: Steel Intelligence

After all detections are collected, the pipeline runs a steel intelligence layer:

AISC weight lookup: Each detected member type is matched against a validated database of 550+ AISC steel profiles. If found, the weight per linear foot (weight_plf) is attached to the detection.
Validation flags: Members that match the AISC database get aisc_validated: true. Members that don't match receive professional warnings (e.g., non-standard designation, possible typo).
Category assignment: Each detection is assigned to a category — beam, hss, pipe, plate, channel, angle, or misc — based on the member type prefix and standard.

The Detection Object

Each detection the pipeline produces is a structured object containing:

| Field | Description | Example | |-------|-------------|---------| | id | Unique identifier | a3f8c2d1-... | | member_type | Normalized designation | W10X15 | | member_type_display | Raw text as found on drawing | W10x15 | | category | Steel category | beam | | confidence | Detection confidence | 0.95 (pymupdf), varies (Gemini) | | bbox | Bounding box {x, y, w, h} | PDF points or pixel coords | | page | 0-indexed page number | 0 | | page_type | Drawing view classification | plan, elevation, detail | | detection_source | Which stage found it | pymupdf_text or gemini_vision | | coordinate_mode | Coordinate system | pdfPoints or image_pixels | | status | Review state | unreviewed | | weight_plf | AISC weight per linear foot | 15.0 | | aisc_validated | Found in AISC database | true |

Multi-Standard Support

Steelflo currently supports four structural steel naming standards. The pipeline auto-detects which standard a drawing set uses and routes to the correct pattern library.

| Standard | Regions | Example Designations | Pattern Count | Unit System | |----------|---------|---------------------|---------------|-------------| | AISC | United States | W12X26, HSS6X6X1/4, L4X4X1/4, C10X15.3, PL1/2X12 | 9 shape families | Imperial | | BS/IS | UK, India, Middle East | UC305305158, UB45719167, SHS2202206.0, CHS168.3*6.0 | 10 regex patterns | Metric | | AS/NZS | Australia, New Zealand | 310UB40.4, 250UC89.5, 150PFC, 100x100x6.0SHS | 11 regex patterns | Metric | | EN (Eurocode) | Europe | HEA200, IPE300, HEB160 | Stub (not yet production-tested) | Metric |

The auto-detection mechanism works by scanning all pages in the uploaded PDF and counting how many text spans match each standard's signature patterns. For example, if the pipeline finds 47 matches for AISC-style designations and 0 for BS/IS, it routes to the AISC pattern library with high confidence. Mixed-standard drawings (rare but possible on international projects) will route to whichever standard has the most matches.

For BS/IS drawings, the pipeline also decodes structured member IDs commonly used on Indian and Middle Eastern projects. A member ID like 01-STR-06-VBR018 is parsed into its components: 01 (zone), STR (structural), 06 (level), VBR (vertical bracing), 018 (sequence). This decoded information is attached to the detection as metadata.

The Verify Workflow

Steelflo does not assume its detections are perfect. Every takeoff includes a human verification step — Step 4 of the 6-step wizard — where the estimator reviews each detection against the source drawing.

The verify page displays:

Left panel: All detections grouped by member type and sorted by category (beams, HSS, pipe, plate, channel, angle, misc). Each group shows the count of occurrences.
Right panel: The original PDF rendered with bounding box overlays. Each box is drawn directly on the page at the exact coordinates from the pipeline output, linking the extracted data to the source drawing.

The estimator clicks through each detection. For each one, the bounding box highlights on the PDF so they can visually confirm the label matches the extracted data. They mark each detection as confirmed or rejected.

This workflow matters because the combination of AI detection and human review consistently outperforms either alone. The AI finds labels a human might overlook (especially on dense multi-page drawing sets), and the human catches edge cases the AI misreads. In testing, an experienced estimator named Billy counted 41 pieces and 17 member types from a 7-page US drawing set. Steelflo found all 17 of his types plus one he missed (W10X12), for a total of 53 individual label occurrences across 18 types.

Export and Downstream Outputs

After verification, Steelflo produces three export formats:

Cut List (CSV)

A bill of materials with every confirmed detection. Each row includes member type, category, measured length (if lengths were captured in Step 5), weight per linear foot, and total weight. This is the standard BOM format that fabricators use for pricing and purchasing.

Order Sheet with Nesting Optimizer

The order sheet groups confirmed members by type and runs a nesting optimization to minimize material waste. The optimizer uses a first-fit decreasing bin packing algorithm:

All cut pieces of the same member type are sorted longest-first
Each piece is placed on the first available mill-length stock bar that has room
A 1/4-inch kerf allowance is subtracted for each cut
Default mill lengths are assigned per category (e.g., 40 feet for W-sections, 24 feet for HSS), and the user can override per member type

The result is a visual layout showing each stock bar, which pieces fit on it, a utilization percentage, and waste color-coded by severity: green for under 8% waste, amber for 8-15%, red for over 15%.

Highlighted PDF

A copy of the original PDF with bounding boxes drawn over every confirmed detection. This serves as a visual record of what was counted and where it was found — useful for bid documentation and shop communication.

Real-World Performance

Steelflo has been tested against real structural drawing sets across multiple standards:

| Drawing Set | Standard | Pages | Detections | Unique Types | Notes | |------------|----------|-------|------------|--------------|-------| | Hutchings (US commercial) | AISC | 7 | 53 | 18 | Baseline test set. Includes W, HSS, pipe, plate, channel, angle. | | Jotun Convention Center (India) | BS/IS | Multi-page | 1,047 | Multiple | Large-scale project. UC, UB, SHS, RHS, CHS, PFC detected. | | WW Austral (AU commercial) | AS/NZS | Multi-page | 119 | Multiple | Number-first format (310UB40.4, 150PFC). | | BMW Penrith (AU industrial) | AS/NZS | Multi-page | 237 | Multiple | Industrial steel with SHS, RHS, CHS, UB, UC. |

Human vs. AI Comparison

On the Hutchings drawing set, the most thoroughly benchmarked test case:

| Metric | Billy (experienced estimator) | Steelflo | |--------|------------------------------|----------| | Pieces counted | 41 | 53 | | Unique member types | 17 | 18 | | Missed members | W10X12 (1 occurrence) | None (found all 17 of Billy's types + 1 extra) |

The difference in piece count (53 vs. 41) is expected: Steelflo counts every individual label occurrence, including the same member appearing on both plan and detail views. Billy, as an experienced estimator, mentally de-duplicates across views. Both counts are "correct" for their purpose — Steelflo's job is to find every label, and the verify step is where the estimator resolves duplicates.

What Steelflo Doesn't Do (Yet)

Transparency about limitations matters as much as documenting capabilities. Here is what Steelflo currently does not handle:

Scanned/raster PDFs require the Gemini Vision fallback path. If the entire drawing set is a scanned image (no embedded text layer), all pages go through vision AI, which is slower and less precise than text extraction.
Cross-view deduplication is manual. When the same beam appears on a framing plan and an elevation (learn more about reading structural drawings), Steelflo creates two detections. The estimator resolves this during verification. Automated cross-view dedup is on the roadmap but requires spatial reasoning the pipeline doesn't yet have.
Schedule parsing is not supported. Steel schedules (tables listing member marks with sizes, lengths, and quantities) are common on structural drawings. Steelflo's regex-based approach detects labels in free-form annotations but does not parse tabular data.
Design intent is human work. Steelflo extracts what is written on the drawing. It does not interpret design intent, check code compliance, or flag structural concerns. That remains the engineer's and estimator's job.
EN/Eurocode support is a stub. European steel designations (HEA, HEB, IPE) are recognized but the pattern library has not been validated against real European drawing sets.
No quantity multipliers. Annotations like "typical of 4" or "(x3)" are not parsed. Each bounding box equals one piece. Quantity interpretation happens during human verification.

Steelflo is built for structural steel estimators who want to move faster without sacrificing accuracy. The pipeline handles the tedious work of finding and cataloging every steel label across a drawing set, and the verify workflow ensures a human always has the final say. For more information or to try it on your own drawings, visit steelfloai.com.