Concurrency Guide

Sigil provides cooperative concurrency through an async runtime with lightweight tasks, channels, and non-blocking I/O.

The Async Runtime

All concurrent operations run within an async context created by with-async:

(import (sigil async))

(with-async
  (go (display "Task 1\n"))
  (go (display "Task 2\n")))

The with-async form creates a scheduler that runs tasks cooperatively. Tasks yield control at blocking operations (channel sends/receives, I/O, sleep), allowing other tasks to run.

Spawning Tasks

Use go to spawn concurrent tasks within with-async:

(with-async
  (go (begin
        (display "Starting task A\n")
        (sleep 0.1)
        (display "Task A done\n")))
  (go (begin
        (display "Starting task B\n")
        (sleep 0.05)
        (display "Task B done\n"))))
; Output:
; Starting task A
; Starting task B
; Task B done
; Task A done

Sleeping

The sleep function pauses a task for a specified number of seconds, allowing other tasks to run:

(with-async
  (go (begin
        (sleep 1)
        (display "One second later\n"))))

Outside of with-async, sleep blocks the entire program.

Yielding

Use yield to explicitly give other tasks a chance to run:

(with-async
  (go (let loop ((n 0))
        (when (< n 1000)
          (do-work n)
          (yield)  ; Let other tasks run
          (loop (+ n 1))))))

Async I/O

I/O operations are async-aware: when running inside with-async, reads from process pipes and sockets automatically yield to the scheduler while waiting for data.

Process Pipes

Reading from subprocess output is non-blocking in async context:

(import (sigil async)
        (sigil process)
        (sigil io))

(with-async
  ;; Spawn a slow process
  (let ((proc (process-spawn "slow-command" '())))
    ;; This read yields to the scheduler while waiting
    (go (let loop ()
          (let ((line (read-line (process-stdout proc))))
            (when (not (eof-object? line))
              (display line)
              (newline)
              (loop)))))
    ;; Other tasks can run while waiting for process output
    (go (other-work))))

The same read-line function works in both sync and async contexts:

• Sync context: Blocks until data is available
• Async context: Yields to scheduler, other tasks run while waiting

All standard I/O functions are async-aware:

• read, read-all — S-expression reading
• read-char, peek-char, write-char — Character I/O
• read-line, read-string, write-string — String I/O
• read-u8, write-u8, read-bytevector, write-bytevector — Binary I/O

Socket I/O

Socket operations also yield in async context:

(import (sigil async)
        (sigil socket))

(with-async
  (let ((sock (tcp-connect "example.com" 80)))
    (go (begin
          (socket-write sock "GET / HTTP/1.0\r\n\r\n")
          (display (socket-read sock 1024))))))

Checking Async Context

Use in-async-context? to check if code is running inside with-async:

(define (my-read port)
  (if (in-async-context?)
      (display "Running async\n")
      (display "Running sync\n"))
  (read-line port))

Port Predicates

Check if a port supports async I/O:

(pipe-port? port)   ; #t for process stdin/stdout/stderr
(async-port? port)  ; #t for ports that support async I/O
(port->fd port)     ; Get raw file descriptor (or #f)

Channels

Channels are the primary communication mechanism between concurrent tasks. Channel operations (channel-send, channel-receive) must be called within with-async.

Creating Channels

(import (sigil async)
        (sigil channels))

(with-async
  ;; Unbuffered channel - sends block until received
  (define ch (make-channel))

  ;; Buffered channel - can hold up to 10 items
  (define buffered-ch (make-channel 10))
  ...)

Sending and Receiving

;; Send a value (may block if unbuffered/full)
(channel-send ch "hello")

;; Receive a value (blocks until available)
(define msg (channel-receive ch))

;; Non-blocking receive
(define msg (channel-try-receive ch))
; Returns #f if no message available

Channel State

(channel-empty? ch)   ; No messages waiting
(channel-full? ch)    ; Buffer is full (buffered channels only)
(channel-closed? ch)  ; Channel has been closed

Closing Channels

(channel-close! ch)

;; Closed channels:
;; - Sends fail immediately
;; - Receives return remaining buffered items, then #f

Producer-Consumer Example

A complete example showing channels with with-async:

(import (sigil async)
        (sigil channels))

(with-async
  (define ch (make-channel 5))

  ;; Spawn a producer
  (go
    (let loop ((n 0))
      (when (< n 10)
        (channel-send ch n)
        (loop (+ n 1))))
    (channel-close! ch))

  ;; Consume in main task
  (let loop ()
    (let ((val (channel-receive ch)))
      (when val
        (display val)
        (newline)
        (loop)))))

Cooperative Scheduling

Sigil uses cooperative multitasking. Tasks yield control at:

• sleep calls
• Channel operations (channel-send, channel-receive)
• I/O operations on async ports (pipes, sockets)
• Explicit yield calls

Tasks that don't yield will block others:

;; Bad: never yields
(go
  (let loop () (loop)))  ; Blocks everything!

;; Good: yields periodically
(go
  (let loop ()
    (do-work)
    (yield)
    (loop)))

Common Patterns

Fan-Out (One to Many)

(define (broadcast ch workers)
  (let loop ()
    (let ((msg (channel-receive ch)))
      (when msg
        (for-each
          (lambda (w) (channel-send w msg))
          workers)
        (loop)))))

;; Create workers
(define worker-channels
  (map (lambda (_) (make-channel 10))
       '(1 2 3)))

;; Spawn workers
(for-each
  (lambda (ch)
    (go
      (let loop ()
        (let ((work (channel-receive ch)))
          (when work
            (process-work work)
            (loop))))))
  worker-channels)

;; Broadcast work
(broadcast main-channel worker-channels)

Fan-In (Many to One)

(define results (make-channel 100))

;; Spawn multiple workers that send to same channel
(for-each
  (lambda (task)
    (go
      (channel-send results (compute task))))
  tasks)

;; Collect results
(let loop ((count 0) (acc '()))
  (if (= count (length tasks))
      acc
      (loop (+ count 1)
            (cons (channel-receive results) acc))))

Select (Multiplexing Channels)

Use channel-select to handle multiple channels without blocking:

(define ch1 (make-channel 1))
(define ch2 (make-channel 1))

;; Non-blocking select on multiple channels
(channel-select
  ((recv ch1) => (lambda (val) (handle-ch1 val)))
  ((recv ch2) => (lambda (val) (handle-ch2 val)))
  (else (display "nothing ready")))

;; Can also select on sends
(channel-select
  ((send ch1 "hello") => (lambda () (display "sent to ch1")))
  ((recv ch2) => (lambda (val) (display val)))
  (else #f))

The channel-select macro tries each clause in order:

• (recv ch) succeeds if a value is available
• (send ch val) succeeds if the channel can accept a value
• else runs if no channel operation succeeds

Event-Driven Architecture

Channels work well for event systems:

(define event-bus (make-channel 100))

;; Event types
(define (make-event type data)
  (cons type data))

(define (event-type e) (car e))
(define (event-data e) (cdr e))

;; Publish
(define (publish type data)
  (channel-send event-bus (make-event type data)))

;; Subscribe (filter by type)
(define (subscribe types handler)
  (go
    (let loop ()
      (let ((event (channel-receive event-bus)))
        (when event
          (when (memq (event-type event) types)
            (handler event))
          (loop))))))

;; Usage
(subscribe '(user-login user-logout)
  (lambda (e)
    (log-event (event-type e) (event-data e))))

(publish 'user-login "alice")

Example: Chat Server

See examples/chat-server.sgl for a complete example using:

• A main event channel
• Multiple concurrent client handlers
• Broadcast to all connected clients

Key patterns from that example:

;; Accept connections in a loop
(go
  (let loop ()
    (let ((client (tcp-accept server)))
      (when client
        (spawn-client-handler client)
        (loop)))))

;; Handle each client concurrently
(define (spawn-client-handler client)
  (go
    (let loop ()
      (let ((msg (socket-read-line client)))
        (when (not (eof-object? msg))
          (broadcast-message msg)
          (loop))))))

Best Practices

1. Use Buffered Channels for Performance

;; Unbuffered: every send blocks
(define slow-ch (make-channel))

;; Buffered: sends don't block until full
(define fast-ch (make-channel 100))

2. Always Close Channels

;; Producer closes when done
(go
  (produce-items ch)
  (channel-close! ch))  ; Signal completion

;; Consumer checks for close
(let loop ()
  (let ((item (channel-receive ch)))
    (when item  ; #f means closed
      (process item)
      (loop))))

3. Avoid Shared Mutable State

;; Bad: shared variable
(define counter 0)
(go (set! counter (+ counter 1)))  ; Race condition!

;; Good: communicate via channels
(define counter-ch (make-channel))
(go
  (let loop ((count 0))
    (channel-send counter-ch count)
    (loop (+ count 1))))

4. Handle Errors in Tasks

(go
  (guard (err (else (log-error err)))
    (risky-operation)))

5. Limit Concurrency

;; Semaphore pattern - limit to N concurrent tasks
(define (make-semaphore n)
  (let ((ch (make-channel n)))
    (let loop ((i 0))
      (when (< i n)
        (channel-send ch #t)
        (loop (+ i 1))))
    ch))

(define (with-semaphore sem thunk)
  (channel-receive sem)  ; Acquire
  (let ((result (thunk)))
    (channel-send sem #t)  ; Release
    result))

(define sem (make-semaphore 5))
(go (with-semaphore sem (lambda () (heavy-work))))

Debugging Concurrent Code

1. Add logging: Print task IDs and timestamps
2. Start simple: Debug with one producer, one consumer
3. Check for deadlocks: Ensure channels are closed
4. Buffer size: Too small causes blocking, too large wastes memory