Introduction and Goals

The goal of this lab is to explore shared-memory multiprocessors through experiments. You will work with a dual-core system built from the same Sodor 5-stage core and L1 data cache you characterized in Lab 2, extended with a snooping bus that keeps the two caches coherent. You will write multi-threaded C programs, measure them with the caches’ hardware counters, and investigate how data-parallel workloads behave on a multi-core system under two cache coherence protocols: MI and MSI.

Students are encouraged to discuss the lab with one another. However, the directed portion must be completed and submitted individually. For the open-ended portion, students may work individually or in groups of two or three, and each group submits a single solution.

Graded Items

Both portions are graded by the Gradescope autograder:

Feedback is submitted separately:

  • Feedback: submitted through a Google Form

Feedback Portion

In order to improve the labs for the next offering of this course, we would like your feedback. As in Labs 2 and 3, fill out the Lab 4 feedback form. Every student should submit the form individually, including the questions about your team.

Feel free to write as much or as little as you prefer (a point will be deducted only if the form is left completely empty).

Background

The System

This lab uses two copies of the 5-stage Sodor core from Lab 2, each with its own L1 data cache. As in Lab 2, each cache is 4 KiB, 2-way set associative, with 32-byte lines (8 words), true LRU, write-back and write-allocate. Compared to Lab 2, the system introduces two new features:

  • Cache coherence: The two caches communicate over a shared snooping bus to maintain coherence (see Coherence).
  • Coherence counters: Each cache’s MMIO counter block includes additional counters to track coherence events (see Counters).

Coherence

The two caches share an atomic snooping bus: only one cache can use the bus at a time, and each transaction completes before the next begins. When both caches request the bus simultaneously, access is granted using round-robin arbitration.

Every bus transaction is broadcast to the other cache, which checks its tags and responds according to the transaction type:

Transaction Issued for What the other cache does
GetS Load miss (MSI only) downgrades its copy from M to S, supplying the data if dirty
GetM Store miss or any miss under MI invalidates its copy, supplying the data first if dirty
Upg Store to a line in S invalidates its copy; no data moves

The caches implement one of two protocols, selected by the simulator CONFIG:

  • MI (CS152Lab4MIConfig). Each cache line is either Invalid (I) or Modified (M). Without a Shared state, every cache miss (including a load miss) requests exclusive ownership and invalidates any copy held by the other cache.

  • MSI (CS152Lab4MSIConfig). Adds a Shared (S) state, allowing both caches to hold read-only copies of the same line. A store to a line in S issues an Upg transaction to invalidate the other cache’s copy.

Counters

The benchmarks use setStats(1) and setStats(0) to mark the beginning and end of the measured region. Results are printed in a table with one column per core, showing the events recorded by each core’s cache.

Counter Meaning
cycles Clock cycles
instret Instructions retired by this core
loads, stores Data accesses issued by this core
hits Data accesses completed without a bus transaction
load misses, store misses Cache misses
upgrades Upg transactions issued when storing to a line in S
writebacks Dirty lines this cache wrote back on eviction
invalidations Cache lines invalidated by the other core’s coherence requests
interventions Dirty cache lines supplied to the other core
bus-wait cyc Cycles spent waiting while the other cache owns the bus

Programming Environment

The system runs without an operating system. Both cores execute the same program, with each core invoking the following function concurrently:

void thread_entry(int cid, int nc);

Here, cid is the core ID (0 or 1), and nc is the total number of cores. Use cid to divide the workload between the two cores.

  • Memory. Each core has its own stack, so local variables are private to that core. Declare arrays and any data shared between cores as static. All benchmark arrays are aligned to 32-byte cache-line boundaries. To align your own data, use __attribute__((aligned(CS152_LINE_BYTES))).
  • Synchronization. Use barrier(nc) to synchronize the cores. Each core waits at the barrier until all cores have reached it.
  • Printing: Only one core should call printf() at a time to avoid interleaved output. Floating-point format specifiers are not supported. Keep printf() calls outside measured regions to avoid affecting performance counters.

Acknowledgments

This lab was originally developed for CS 152 at UC Berkeley by Christopher Celio, and builds on the multi-threaded software environment and coherence protocols that Andrew Waterman and Henry Cook (among others) developed for Rocket. The dual-core Sodor system, its coherent caches, and its snooping bus were developed by the course staff.

Table of contents